What’s the fine line between test complexity and velocity? On one hand, simpler tests are faster. They produce faster results. More learnings and wins through velocity. You don’t risk losing a lot of time on test losers.
On the other hand, a small layer of complexity can make you cash out a lot more bang for your buck. Or, adding more variations can help you spot optimization opportunities you wouldn’t see otherwise.
In A to B episode 9, Stewart Ehoff, head of experimentation at RS Group, joined the host Shiva Manjunath to discuss how you can walk the fine line between test complexity and velocity.
We also prepared a bonus section: How to get engineering (resources) on your side. Enjoy.
Why (Maybe) You Should Deprioritize Complex Experiments
Complexity yields big lifts. Complex tests can bring outstanding results. But that's not always a rule. It always depends on the test and how you build it.
Shiva — “I tend to deprioritize complex tests in general unless there's a very good reason not to. Because the faster I build tests, the more insights I can generate. And likely quicker. Sometimes, there are tradeoffs. But for the most part, I deprioritize complex tests.”
Stewart — "Everything in experimentation principles leads us to believe that we should always focus on the fastest path to prove or disprove the hypothesis. In that regard, do as little as possible, so you don’t waste resources on unnecessary projects.”
Focus on deprioritizing complex tests in order to build and run more tests. Do as little as possible and don’t spend time on unnecessary projects.
Bonus tip: Complexity is also more than the number of lines of code. It can include how hard it is to track the data you need to measure against a particular outcome or hypothesis, or how many dependencies the experiment requires to run. In some instances, experiments become complex because you're asking a really difficult question and your hypothesis can be based on metrics that you as a business might not typically be geared up to collect.
Helpful Blueprints:
How to balance test velocity vs complexity? — a core graph for your testing program. Look to increase complexity and velocity. Test more on more things and in new ways with new skills and capabilities.
Should I run an A/B test? — A/B testing is expensive. This is a framework of objective questions to ask about running an A/B test or spending your calories elsewhere. Align and train your team to know when it’s to A/B test.
To Add or Not to Add Test Variation?
Stewart — “Another way to increase test complexity is to increase the number of variations that you deliver. I'm a big believer that you should always move away from A versus B and try to get a C or D an E in there if you can.
I remember this one test where we ran four variants. It was about introducing a new element on a product page, one of the highest-traffic pages on our site. All variations won against the control. However, the difference between the smallest and highest winner was almost by a factor of four or five (9% and 36% uplifts, respectively)
If we had done a simple A vs B and kept the complexity down, we would have ended up with a 9% increase on our primary metric, never discovering the variant that produced a 36% increase. If you found that 9% increase with an A/B test, you would just roll with it.
Obviously, this depends on your test power. Running multiple test variants increases engineering complexity. No matter how you look at it, you still get engineers coding more variance, more declarations of your WYSIWYG editor, more moving things around, etc.
But there’s a tradeoff between complexity and what you learn from a test. Some experimentation programs possibly go wrong by focusing too much on simple tests and less variance. If only they increased the complexity by a little, they would get a lot more bang for their buck.
Try thinking outside of the box. If you've come up with an A versus B, what other solutions could exist that might yield new insights? What could we learn more from this test? You can find a lot of value outside of the box.”
Shiva — “I tend to focus on a different path. Maybe an additional variation or two could be beneficial depending on the power of the experiment. Unfortunately, more variations also mean more chances for false positives or false negatives.
But I do agree with the principle that sometimes the ROI of adding one additional variation is greater than doing a whole other A/B test where you're adding control again and you're putting more traffic into the control again, instead of adding an additional variation at the start. If you have good statistical principles to control for it and the sample size, then you're probably in a better spot.
Regarding your example, I’d say that should never stop at a 9% increase. Not even at a 36% increase. Take the iterative approach to testing. Instead of doing four variations take a concept and say "All right, I'm going to test the scale of disruption and test incrementally pushing that concept further in that single test”. Continue to push forward and iterate on your concepts.
Strike the balance between test velocity and complexity. The more tests you run, the more learnings you get. But with a layer of complexity on them. The point is, don’t focus exclusively on quality or quantity. What happens if that complex test you needed a month to build doesn’t win? That’s a lot of wasted potential. On the other hand, simplify too much, and you don’t know what you’re leaving on the table.”
Running only A/B testing can take you down the path where you ignore other, potentially more profitable variants. This increases testing complexity and engineering resources though. But the key here is to ensure you aren’t focused too much on test simplicity. Think outside the box, and add a bit of complexity. One more variant is better than running one more A/B Test.
Also, strike a chord between complexity and velocity. More tests, but with a layer of complexity.
Helpful Blueprints:
Iterate Vs Move On — an intuitive visualization of how many iterations you need. This blueprint lets visualize how you prioritize your experiments. Its real power comes from combining the logic behind this blueprint with your own prioritization frameworks and unique business needs.
Is the Feature Worth the Squeeze?
Engineering and experimentation go hand in hand. With experiments, you can prove or disprove if investing more engineering time and resources (in that feature) is actually worth it.
Stewart — “We had instances where product managers made assumptions about lofty features that would take a lot of sprints to build. We challenged that and said: “Hey, let’s make a simple experiment and see if the feature is worth the sprints.”
When there’s merit, great, you double down on that. You invest and spend time on it. When there’s no merit, you kill the project early and move to more profitable initiatives.
Helpful Blueprint:
Validation Methods — Test and validate ideas with further research before releasing them into the wild. Decide which validation method to use for site changes and have visibility on different validation tools for functionality, copy, design, and more.
Engineers Are Humans Too
It’s hard to get dedicated resources for engineering. Product managers or owners often see experimentation as a big unknown.
Stewart — “Remember that engineers are humans too.
The issue happens because lots of C-level people consider engineering teams as feature factories. They create engineering departments for feature churning. Requirements come from the top of the business, get fed through product managers, and then you end up with engineers looking at a long, lofty roadmap that likely goes 12, 18, or 24 months into the future.
The super plan: We will win once we deliver all these features. That’s the kind of world engineers live in. Solving Jira ticket after Jira ticket. No engineer that I know wants to work like that.
Engineers actually want to know what the value is of the work they deliver. You can't imagine how getting it must feel to see, to spend 20, 30 weeks on building this big feature, and then you deploy it and you find out it's dropped your conversion rate by 10%. What a waste of time.
Experimentation is the answer.
You can say to engineers you don't have to build things that take 30 weeks, and then you release it and you figure out if it, you cross your fingers and pray. You can have that instant feedback from customers (with experimentation). Most importantly, you can have a seat on the table in terms of your ideas and what’s actually working for you as an engineer.”
Shiva — “I love this. Reminds me of when I work with UI and/or UX designers. Lots of times, the relationship between experimentation and UX/UI teams is seen as combative. Because the experimentation team is basically trying to test someone else’s hard work, sweat, and tears. Bringing quantitative data that shows someone else’s design sucks.
Sometimes, people see it like that. Just like you said, engineers are humans. But UX people are also humans. Of course, they come at testing their design with: “We will show you how great our design is”.
However, the relationship between UX designers and experimenters is more a partnership than rivalry. It’s a partnership because experimentation and UX teams are working together. “Hey, I want to test your idea and show that it has value with data. Show it has value on paper. So we can both prove to the boss that it works.”
You can apply the same principle to engineering. Lots of the time, they’re so stuck in the Jira ticket routine that they can’t the bigger picture. They’re so far removed from the process that they can’t see the results of their work. And everyone wants to see the results of their work.
Experimentation lets you see how the little piece you’ve built (coded) fits into the whole thing. It’s really cool seeing the sweat on your brow roll into the bigger thing. The experimentation team should approach engineers and say:
“We don't want you to work hard to build something that we don't even know if it's gonna work or not. We wanna chunk it down into something smaller so we could prove it's worth your valuable time to move it forward”.
Or, “Hey, remember that thing you helped me build. 13% lift in conversion rate. Let me buy you a drink.”
Optimizing Your Testing: Speed vs. Depth
Find the sweet spot between simple tests that yield fast learnings, and complex tests that unlock bigger wins. If you add some complexity to your testing, this can reveal key insights you'd otherwise miss. But keep things nimble enough to fail fast and pivot quickly. The key is optimizing both your test velocity and depth.
Don’t over-focus on complexity or velocity. More tests, more learnings for you. But add a layer of complexity to them. That way, you won’t lose resources on overly complex tests that may fail or on oversimplified testing where you don’t know what you’re leaving on the table.
.svg)
