Experimentation is often seen as the realm of either engineering or marketing, but we believe that it plays a significant role in solving user experience and design problems as well.
What’s In an Experiment?
Design is about solving problems, whether that problem is solving a business goal, improving team productivity, bringing in new users and customers, or communicating a brand’s values. For each of these problems, there’s a methodology out there that promises to help guide the problem-solving process. But, we want to focus in on one way of solving problems that can be applied to any design problem: experimentation.
Before we jump straight in though, let’s take a moment to define what experimentation is (and what it isn’t). Experimentation is the structured, iterative, scientific method-driven process of arriving at the optimal outcome, given a set of parameters.
Now, that might sound like a bunch of words thrown together, but don’t worry, by the end of this post you’ll be an expert in what an experiment is, how it works, and how it will help you solve UX problems
The Three Principles of Proper Experimentation
As you can probably guess from the definition above, there are three keys principles to keep in mind whenever you run an experiment.
Principle #1: The Scientific Method
Most of us are familiar with the scientific method from grade school. There’s a hypothesis, a test, some analysis, and a conclusion. While not exactly the same, those pieces can be applied to UX experiments as well.
It starts with you creating a written template for every experiment you want to run. It should include a hypothesis, in the form of “This will change X to Y because of Z,” a brief overview of the resources required to run the experiment, a written explanation of your reasoning for why this experiment could be successful, and the risk of it failing. Every experiment you want to run should be written down in this template, providing a structure to your process of experimentation.
Doing all of this work for an experiment, whether it’s something big or something as small as changing the color of a button, is incredibly worthwhile. It helps clarify your thinking and helps in prioritizing what is worth focusing on.
Principle #2: Ruthlessly Prioritize
Once you’ve begun to document your experiments you’re going to find that there’s a long list of possible experiments you can run that are seemingly worth trying out. Which brings us to our second principle: you need to prioritize what you’re working on.
There are a few ways to approach prioritizing experiments, but we’ve found that the simplest way is typically the best.
Based on the experiments you defined, look for two pieces of information: how much work it’s going to take to roll out the experiment (in terms of hours of work or financial cost) and how big of an impact you think the experiment could have.
What you should be looking for are experiments that have a high ratio of return on your investment. Start at the top of the list and begin going through these experiments sequentially.
Principle #3: Iteration Is Key
If you only had the first two principles you’d have a nicely organized list of projects to work on, but you wouldn’t have a system of experimentation. That’s because proper experimentation requires iteration, week after week.
Iteration means taking the results of previous experiments and making use of any new learnings or insights to define future experiments. Learning from the hypotheses you’ve proven and disproven helps make all of your future hypotheses more likely to be successful and for you to get closer to finding the solution for your problem.
The Tools You’ll Need to Master
There are plenty of specialized tools that you can use to begin running experiments, but as far as we’re concerned there are two that you’re going to want to put at the center of your experimentation.
Optimization Software
Experimentation has become popular enough that there are specialized tools out there that help in running experiments, especially in the world of software. Two of our favorites are Optimizely and Google Optimize, which in broad terms offer the same feature set. The idea is that you instill a small snippet of code in your website, which enables you to quickly test changes to the website without having to actually write any code.
Want to have half of your visitors see one image and the other half see no image at all as you try to figure out what will push people to accomplish a certain task? That will take less than 2 minutes to set up in either tool.
The Common Spreadsheet
The second tool is one that we’re all familiar with, the common spreadsheet. Having a well-organized spreadsheet is key to running experiments, as you really do need to keep track of the experiments you’re proposing, ones you’ve run in the past, and ones you’re running currently. There are a bunch of ways that you can organize this spreadsheet, but this template has worked well for us in the past:
There are two key reasons why you’ll want to organize your experiments in a spreadsheet. The first is that it helps make sure you are keeping track of your learnings and applying them to future experiments since they’re all right next to each other. The second is that it helps you avoid running the same experiment twice, which is surprisingly common, especially when you’ve run hundreds of experiments.
The Role of Experimentation in UX Design and Development
As we’ve mentioned, like in all design, UX design is about solving specific problems. There is an infinite number of problems out there waiting to be solved, whether about communicating brand values, getting users to perform a specific action, making sure that team members follow protocols, or getting people to finish checking out at a store.
When it comes to user experience, experimentation helps you arrive at the solution for problems when your intuition isn’t able to provide you with direction. Instead of just going with a gut feeling, you can test a number of different options in lightweight experiments, see which ones seem to be working out, and then reinvest in the next round of experimentation.
Over time, you’ll have systematically disproven dozens of possible solutions and arrived at a solution that is proven with evidence to be superior. Not the perfect solution, mind you, but a solution that is better than all of the other ideas that have come along.
How Experiments Provide Non-Obvious Solutions
The last thing we want to note is that experimentation isn’t just about proving your idea right or wrong, but about arriving at solutions that aren’t at all obvious. You may think that you only need to run one or two experiments to prove that you’re right, but that would be missing the forest for the trees.
Instead, you should be ready to have all of your assumptions proven wrong and being pushed in a direction you weren’t expecting. That’s where the real value of experimentation lives, which is a value that we genuinely hope you’ll see firsthand in your upcoming work.
