Welcome to the first Speero Sideshow. Our new podcast is about the next step in experimentation: growth experimentation. Let’s stop pigeonholing experimentation into A/B testing and CRO, and turn experiments into a holistic strategic driver of growth and innovation in all business areas.
In this first episode, Jonny Longden (Chief Growth Officer at Speero) was joined by Makram Mansour, principal product manager at Intuit (and ex-LinkedIn), to talk about how successful experimentation is not about running individual A/B tests but about building a cohesive experimentation operating system.
Also, why if you don’t try a lot you won’t find golden opportunities and why learnings you don’t share don’t go anywhere.
TLDR:
- A strong problem statement is crucial and can lead to significant opportunities, such as a $1 million grant for developing wireless charging for hearing aids.
- Rapid prototyping and demonstration are effective ways to validate a business opportunity and secure investment.
- Making complex technology accessible to a wider audience, like hobbyists, can open up entirely new markets.
- A company's culture and leadership should not be afraid of failure, but rather see it as a necessary part of finding "golden opportunities".
- Building an in-house experimentation solution can be a competitive advantage, but only if the team has the right skills to execute it well and avoid frustrating internal users with a mediocre interface.
- Successful experimentation relies on a synergy of people, process, and technology. The right technology is useless without proper training and a supportive process.
- Sharing knowledge and learnings across teams is vital to prevent teams from repeating the same mistakes and to help them build on successful outcomes.
- Experimentation should be viewed as a coordinated program to achieve strategic goals, not just a series of individual tests.
- Companies should focus on pushing solutions to problems, not just pushing technology to people. This requires a dedicated effort to discover and understand customer problems, both known and unspoken.
- When A/B testing isn't possible, it's essential to find other forms of evidence to make decisions, such as observing customers through "follow me homes" and using low-fidelity prototypes.
It's Not About The Experiments, It's About The System
You're an experimentation program leader. You've been tasked with increasing growth, and you know experimentation is the way to do it. You've heard all the buzzwords: "fail fast," "test and learn," "data-driven decisions."
But here's the thing: you're not just running experiments. You're building an operating system. A system that can coordinate a massive, ambitious effort, where every team, every tool, and every process is working toward one strategic goal.
This isn't about running a few A/B tests on a landing page. That's like saying the Apollo missions were just about launching a few rockets. The Apollo program was a coordinated series of experiments, all managed to achieve a single goal: putting a man on the moon.
The individual experiments were important, sure, but the program itself was the real experiment. And the success came from the coordinated learning that happened across all the different parts of the program.
This is the shift in mindset we need to make. We need to stop thinking about isolated experiments and start thinking about the experimentation operating system. This is the only way to scale and get real, repeatable results.
The $1 Million Problem Statement
At the start of my career, when I was working at Intel, the company used my software to design high-speed circuits. I went from being a software developer to a software user. This gave me a different, customer-focused perspective.
Then I moved to National Semiconductor, where I got exposed to system-level design. We were always doing things the traditional way, but the people using our chips weren't thinking big. They couldn't see what was possible with new technology.
For example, we were working on wireless power charging. Now, everyone has this on their phone, but back then, we were just trying to figure out what else we could do with wireless charging. A business development person came to me and said, "What if we use this for hearing aids?".
The problem statement was this: people with hearing aids have a hard time changing the batteries every week, and the batteries are expensive. They last about 10 days, and you have to buy new ones.
So, a wireless, power-charged battery was a perfect fit for this problem. We quickly prototyped it with a small coil, a battery, and our IC.
We demonstrated the charging capability to one of the top hearing aid manufacturers. Then walked out of the meeting with a $1 million check to build the proof of concept.
This is where experimentation comes in different forms. It's not always about A/B testing. Sometimes it's about finding a problem statement and demonstrating a solution.
This connects directly to the Problem-Statement Focused Hypothesis Blueprint. This blueprint is all about grounding your experiment ideas in research and using problem statements as the bridge. It helps you ensure your tests are focused on real problems and not just random solutions.
You start with a test idea, but then you take a step back and ask, "What is the problem statement? Where is the evidence that this is a real problem?". This approach helps you move away from subjective, random testing and focus on what's important to the business and the customer.
How Simple UI Opened a Whole New Semiconductor Market
At Texas Instruments, I looked into the online design tools. The problem was that semiconductor design was like "black magic". It was really hard for most people to just use it.
But the market was changing. Hobbyists were getting into robotics and LED lights, and they needed to incorporate electronics.
Traditionally, we had to use middlemen and distributors to help the end consumers. We wanted to get rid of that and give consumers direct, easy access. The online design tools solved this.
Our users had a very simple interface. We handled all the complex math behind the scenes. Users could put in their requirements, get the results, and even do online simulations. This opened up a whole new market to the brand and made our products more accessible to everyone.
Rapidly Prototype and Demonstrate to Validate Decisions
When I was the experimentation platform PM at LinkedIn, someone from Instagram joined the brand, and the idea came up to do LinkedIn Stories. Instead of the executives just saying no, the mentality was to test it. We rapidly prototyped it and launched LinkedIn Stories.
We measured the metrics, and it wasn't as viral as we thought it would be. The use cases for LinkedIn were different from Instagram. So, it went away.
We also tested a business card scanner for the Japanese market because business cards are very important there. That also wasn't as viral as we hoped.
The real test for a company is whether the culture and leadership are afraid of failure or if they see it as a part of doing business. The most mature companies realize that 80-90% of ideas aren't going to work. But you still have to do them because that's how you find the golden opportunities.
This connects to the Iterate vs. Move On Blueprint. This blueprint helps you decide when to keep iterating on an idea and when to cut your losses and move on to something new. It's a great tool for making those tough decisions and ensuring you're not wasting time and resources on a dead end.
The Burden of Proof Is on You
At the end of the day, a company like Intuit is not in the business of building an experimentation platform. Many companies start with an external vendor solution. But you eventually hit a wall.
The tools have limits in terms of capacity, customization, and integration. For many valid reasons, we realized the vendor tools weren't scaling for our needs.
We made the decision to invest in building our own custom solution. When an experimentation platform becomes a competitive advantage, that's when you know you're doing it right. If you can do it well, you can fail and learn faster than everyone else.
But this decision comes with a huge challenge: you have to be able to execute. You need the right skills to build a solution that's on par with, or even better than, the vendor tools.
Otherwise, you'll frustrate your internal users who will wonder why they have to use a mediocre tool. This is where the challenge lies.
How Native Warehouses Let You Scale Experimentation
This is where the competitive advantage truly comes in. If you think about top-of-the-funnel activities, like before a visitor signs in, we have millions of people coming to our website.
Ideally, we want to know who they are, what their intent is, and what they're looking for. Then we can make it easy for them to find what they need, without them having to chase it. This is what personalization is all about.
But how do you personalize for guests you don't even know?. This is where experimentation and data are so important. If we're going to do a targeted offer, it needs to be connected to our pricing platform.
All these systems need to work together in synergy. If it takes a month to set up a single experiment, you're doing it wrong. The process needs to be fast so you can prove a hypothesis, get the learning, and then scale it.
This is a classic problem that can be solved with a data warehouse. The Using Data Warehouse vs. Not Using Data Warehouse Blueprint helps you see how bringing all your data together in one place can make decision-making better and your business more efficient.
When your tools aren't talking to each other, you can't scale. A data warehouse is what allows you to connect the dots and move quickly.
People, Processes, Tech, and the Weakest Link
The practice of experimentation is about three things: people, process, and technology. You can have the best technology in the world, but if your processes are bureaucratic, nothing will change. You have to change the processes to adapt to the experimentation system.
And then there's the people. If you give someone an advanced piece of machinery and don't train them on how to use it, you can't blame the system. It's like giving someone a fire truck and expecting them to know how to drive it.
This is not a toaster. These are complex systems that can cause major disasters if used incorrectly. You need to train your team properly.
This is especially true for online controlled experimentation, which has a lot of intricacies. So, I was managing a program to educate our workforce on proper experimentation techniques and best practices.
This is exactly what the How to Engage Community and Create a Culture Around Experimentation Blueprint is for. It gives you ways to experiment with how you engage and train your teams to make experimentation more accessible and enjoyable for everyone.
If You Don’t Spread the Learning, It’s Like You Didn’t Learn Anything
Having a Center of Excellence is important, but you can't just think the tool will solve all your problems. The other big problem is silos and "institutional memory". Many teams do similar things every year and don't share their knowledge.
At Intuit, we have multiple businesses like QuickBooks, Mailchimp, and TurboTax. As a Center of Excellence driver, I would see what was happening with experiments across the company. But the learning wasn't being shared as much as I wanted.
This is a huge problem because experimentation is all about learning from your failures and successes. If those learnings are only kept within a small team, you're not getting the full value. You're repeating the same mistakes and not building on what worked.
This is where you need a system for knowledge sharing. This is where the Experimentation Program RASCI Matrix Blueprint comes in handy. This tool visually represents who is responsible, accountable, supported, consulted, and informed about each activity.
It helps you define roles and responsibilities to increase efficiency and ensure everyone is aligned on how things work. This is a key part of the culture you need to build to spread learning and break down silos.
The Program Is the Experiment
Most businesses think of experimentation in terms of individual tests. But like the Apollo program, the real value is in thinking of the entire program as the experiment. The Apollo mission had a bold goal, and they achieved it through the coordinated management of a lot of different experiments.
The whole program was experimental, and a key driver of success was the coordination and sharing of learning. The teams had to work together very closely because the results of one experiment would affect another.
This is what an experimentation operating system is all about. It's the system and program management that brings everything together in pursuit of a strategic goal.
Don’t Push Tech on People, Push Solutions to Problems
Big companies need to be set up for growth. Lines of business are focused on short-term revenue and meeting their numbers. So who is looking at the two- or three-year roadmap? This is why a CTO office is so important.
They look at the cutting-edge technology coming out of universities and get it ready for the product teams to use. This is what Intuit’s "Futures" team does. Strong companies think both near and long-term.
But there's another side to this, handled by the emerging products group at Intuit. You can't just take a technology and push it on people. You have to push solutions to problems.
Our team is all about finding problems—known problems and unspoken problems. Once you have a problem, then you can start thinking about solutions.
When You Can’t A/B Test, Follow the Customers Home
What does testing look like when you can't run a statistically controlled A/B test? The answer is, there's still a lot you can do. It's about finding evidence. You can't just rely on gut feelings.
The way we do it is by talking to customers. We do "follow-me-homes," where we go and observe our target personas in their own environment without intervening.
This is how the founder of Intuit did it in the early days. When you do this many times, you start to see patterns and opportunities. This is what I call "evidence finding". Then you can get scrappy to validate a solution.
We use different techniques like user testing with Figma experiences. The most important thing is to know what you're testing. You need to list your "leap of faith assumptions" (LOFAs) and break them down into the smallest pieces to validate them.
This connects with the User Research Methods Framework blueprint. This blueprint helps you plan which research or validation method is best for your current business needs. It maps out different methods based on data type, effort, and value.
For example, a Heuristic Review has a low-value score because it's just an opinion, but it's a vital part of the process. On the other hand, Customer Interviews have a strong signal but require more effort. This framework helps you choose the right method to gather the evidence you need to make good decisions.
You Need the Right People, Processes, and Tech
It's clear that experimentation is so much more than just A/B testing. It's an operating system that requires the right people, processes, and technology to be successful.
The most mature companies realize that most ideas won't work, but they also know that without trying, they'll never find the golden opportunities.
To get there, you need to think about your experimentation program as a whole. You need to break down silos and make sure the learnings are shared across the organization.
And you need to be able to find evidence and validate your ideas, even when you can't run a traditional A/B test. This is how you build a system that can adapt, learn, and grow.
This is how you stop pushing technology on people and start pushing solutions to their problems. And that is how you win.
.svg)
