Internal experimentation days, experimentation summit, gathering, internal conference: so many names for the same concept. However you name it, successful summits have underlying patterns and tactics. Our goal in this article is to find them.
Ben Labay, Speero's CEO, interviewed four experimentation leaders from BBC, Circuit City, Delivery Hero, and Zalando about their experiences hosting internal experimentation summits, including all the how-tos, issues, and challenges you should know when you want to host your own.
Beyond exploring their stories and how they organized their summit, Speero has prepared key insights and a structured playbook on running your own experimentation gatherings.
Bonus free ungated file for you: A PDF checklist for hosting internal experimentation summits.
BBC’s Experimentation Day
Divya Isaiah came to BBC two years ago as a head of experimentation. Back then BBC already had show and tells, business newsletters talking about experiments, and an experimentation hub showing all the experiments. However, even though they had teams that experimented for 10 years, lots of experiments were on different maturity levels:
- Some teams didn’t know how to experiment at all (and needed a lot of support)
- Some teams were experimenting but were disillusioned by it (because they knew experimentation goes way beyond web optimization)
Divya and her team wanted an event that would re-introduce the central experimentation group, showcase existing and new capabilities, and get people excited about experimentation across both product teams and broader areas of the BBC.
The Look and Feel of the Summit
They ran a hybrid event with around 450 attendees total—60 to 70 in person in BBC’s London HQ, plus remote hubs nationwide. The schedule featured six speakers, each given roughly 25 minutes plus Q&A, with breaks for discussion (discussion breaks were vital due to all the info presented).
The summit was recorded (they’re BBC, after all) so that sessions could be shared in a shorter, bite-sized form to help people engage way after the summit was over.
“We also made a playbook for experimentation that was specific to the BBC,” Divya said; “In fact, one of the main reasons behind the summit was introducing this playbook to the whole company”.
Outcomes and Outputs
Divya’s team tracked several KPIs to see how BBC’s ‘Experimentation day’ went:
- Summit attendance (was the summit successful?)
- Playbook ‘reads’ (do people engage with the playbook?)
- Teams outside product reaching out to experiment (Did the whole org engage?)
The next iteration will zero in on more specific topics, such as platform-focused improvements, rather than covering/introducing every possible discipline.
They’re also playing around with inviting teams from other streaming platforms like Apple TV and Netflix to inspire: “Those types of teams have a lot happening in the back end of experimentation,” Divya said.
Zalando’s EXP Day
This story is from Marcel Toben, Head of Engineering for Economics & Experimentation at Zalando. At that time Zalando had an internal experimentation platform department that builds its capabilities and acts as its Center of Excellence, supporting all the different departments experimenting across the company.
When Zalando implemented Eppo and decided to roll it out to a wider audience inside the company, the brand realized a summit would be perfect for this. Also, create excitement around experimentation and broadcast improvements in their experimentation infrastructure.
Zalando wanted people across its departments—product management, applied science, partner services, and more—to align on experimentation’s value.
The Look and Feel of the Summit
Zalando’s first event was held at their headquarters in Berlin, drawing around 300 attendees in person and another 300 on a call. They booked their auditorium and convened for a half-day conference. The program included:
- Marcel talking about why Zalando experiments, the basis of testing, and more.
- A keynote from the experimentation platform team explaining why experimentation matters
- A guest talk on driving the culture of experimentation (their external speaker had Airbnb and Apple experience)
- Real-world use cases from pilot teams that showcased concrete experiment results—both successes and failures
- A leadership panel featuring senior vice presidents, VPs, and heads of product discussing how experimentation fits with broader business and fashion intuition
Outcomes and Outputs
The summit achieved its purpose: to transport the excitement and ambition of the experimentation platform team to the rest. The second goal—broadcasting the improvements in the platform and plans for the future—was also reached.
Attendees from less-involved departments became curious about using Eppo. Leadership’s public support also became a valuable reference point: teams regularly cited it when justifying new or expanded tests.
“The experimentation conference definitely helped scale our offering in the company and buy-in from the top management… this kind of top management alignment certainly had a big impact on the drive and vibe for experimenting” — Marcel Toben.
Delivery Hero’s Summit
Background and Goals
Iulian Vasilisca manages a central experimentation platform team at Delivery Hero— a multinational conglomerate present in 70 nations. The brand has numerous delivery apps across the world, including Foodpanda (Asia), Talabat (MENA), Hungryhouse (UK), and many others.
All these companies have their local teams and ambassadors responsible for experiments. However, a central experimentation platform in Berlin holds them all together, helps them scale, increases velocity, and runs deeper analysis via a native data warehouse platform.
Even though Delivery Hero holds monthly “Global Experimentation Council” meetings with experimentation ambassadors from these groups, the brand also wanted a summit to connect all disparate teams more deeply.
The Look and Feel of the Summit
They hosted a hybrid event in Berlin, inviting attendees from across the globe:
- Local experimentation “ambassadors” shared their challenges and specialties (example, one groups has more experience on loyalty experiments and shares this)
- Attendees brainstormed connected projects (example: a central experiments repository)
- Presentations sparked follow-up discussions on topics like switchback experiments, holdouts, and standardized metrics
Outcomes and Outputs
Because of Delivery Hero’s global footprint, the main benefit was to increase collaboration among different brands and share technical expertise, says Iulian.
“It was more about getting to know each other, doing some networking, and hopefully from those discussions, there would be some collaboration”—Iulian Vasilisca.
The summit enabled teams to locate each other more easily for specific questions like loyalty program experiments or special holdout designs. It also standardized how groups measure and define metrics.
“To give you an example, we had a group that built switchback experiments and they presented how they're doing it, and after that presentation, we had a few people that followed up and tried to see how they can scale this solution to other groups.” —Iulian Vasilisca
The next summit will niche down with dedicated topics such as holdouts and cross-brand impact reporting. “Imagine that all these delivery apps have their own data set but in the end we kind of have the same northstar metrics.”—Iulian Vasilisca
Circuit City’s Competition Summit
Kelly Wortham is here to share a story of when she was working for Circuit City, two recessions ago, back in 2008. “Do you remember Circuit City? They were like the red version of Best Buy way back in the day.”—Kelly Wortham
Back then, every quarter Circuit City organized ‘hands-on’, competitive research summits.
The Look and Feel of the Summit
During the morning, half of the team would visit Best Buy and A/B test it. Something like competitive user testing. The other half of the team would talk to customers in Circuit City’s physical stores.
In the afternoon, they’d run a hackathon based on this info to generate ideas on improving the customer experience, both offline and online. Around 20 people from paid, email, social media, and website teams were involved.
Ben Labay: “Every quarter this big cohort of people coming into Best Buy, didn't they figure that out?”
Kelly Wortham: “No, no, they never figured it out.”
Food and a sense of fun were essential, not just doing the work during the summit: “We always reported back what we learned and filled a hypothesis library,” Kelly said. Everyone pitched in, from analysts to channel owners.
Outcomes and Outputs
The best part of the summit was that it became a source of ideas for the teams. They had other idea sources, such as call centres and analytics data, but this way, they actually heard the customers and employees in retail. “This way, we kept our fingers on the pulse. Even though Circuit City went out of business, if there was a way to save it, we would’ve found it there” — Kelly.
The summit also let them break silos between email, social, and paid teams. The teams started to share data and learn from each other.
This way, they discovered surprising customer patterns—like how many gamers' “next big purchase” wasn’t more video games but expensive accessories like chairs and gaming TV. That insight directly changed Circuit City’s online marketing to gamers from games to accessories.
Key Insights from All Four Interviews
Key Insights and Learnings Across All Interviews
- Clarity of Goals: Nearly everyone highlighted the importance of articulating specific objectives. As Divya put it, “What are we trying to achieve out of it?”. This will help you avoid “just another event” syndrome.
- Leadership Buy-In: Whether it’s a Zalando executive panel or the BBC leadership on stage, having visible support from top stakeholders bolsters alignment and ensures the event’s long-term influence.
- Showcasing Real-World Success (and Failure): Sharing genuine experiences—from pilot experiments or even from competitor store visits—brings experimentation to life. Marcel noted that success stories and “not only success stories” are equally instructive.
- Cross-Team Collaboration: Summits unify previously siloed groups. Delivery Hero’s ambassadors found new partners for shared projects, while Circuit City’s channel owners discovered how to shift marketing emphasis.
- Celebration and Energy: Kelly stressed small touches like food, group awards, or fun run-of-show elements. This fosters enthusiasm and keeps people returning each time.
- Follow-Up and Measurement: Many teams introduced new resources (like the BBC’s internal playbook or Zalando’s new EPPO-based analytics platform). Track post-event analytics (visits, usage, additional experiments launched, requests for help) to prove success and refine future events.
A Structured Playbook for Hosting an Internal Experimentation Event
Below is a simple set of steps to guide your own “Experimentation Day” or summit. Adapt as needed to suit your organizational culture and scale.
Step 1: Define Your Goals and Success Measures
- Ask: Why do we need this event? Do we want to introduce a new experimentation platform, gain leadership alignment, or unify teams?
- Decide: How you will measure success (attendance, visits to new resources, number of new experiment ideas submitted, etc.).
Tip: “What are we trying to achieve out of it?” was crucial at the BBC, ensuring it wasn’t just an extra initiative with no clear outcomes.
Step 2: Secure Leadership Participation
- Invite Senior Leaders: Consider a panel discussion, executive keynote, or joint Q&A that signals top-level buy-in.
- Tip: At Zalando, the senior VP of product, chief economist, and others joined a panel “to show literally everybody” that top management backs experimentation.
Step 3: Curate Engaging Content
- Pick Your Themes: Decide whether to be broad (e.g., all aspects of experimentation) or more targeted (e.g., holdouts, platform best practices).
- Invite Speakers: Mix external experts with internal success stories—”not only success stories” but also learning moments.
- Tip: “In the next round, we will likely have…dedicated topics,” Iulian said, after realizing more focused discussions boost collaboration.
Step 4: Plan the Format (Hybrid or In-Person)
- Logistics: Choose a central venue, but consider hybrid. Make it easy for remote teams to join.
- Include Breaks: Give attendees space to digest info and discuss ideas informally.
- Tip: The BBC recorded the entire day and then “made sessions into bite-sized chunks,” so employees could revisit content later.
Step 5: Foster Cross-Functional Interaction
- Break Silos: Encourage engineering, data, product, marketing—everyone—to listen to each other’s case studies.
- Inspire Group Brainstorms: If possible, hold smaller workshops or breakout sessions.
- Tip: Circuit City’s approach had half the team gather real-world insights in the morning and reconvene in the afternoon, yielding cross-team aha moments.
Step 6: Make It Fun and Rewarding
- Food and Socializing: Offer a chance for people to mingle; small details (like lunch, coffee stations, or an end-of-day social) go a long way.
- Recognize Contributions: Celebrate experiment wins or give “most innovative test” awards.
- Tip: “Data, food, and awards” were Kelly’s three core elements for driving turnout and enthusiasm.
Step 7: Close the Loop and Follow Through
- Summarize Learnings: Publish a summary deck, internal newsletter, or short videos.
Share Resources: Link your experimentation playbook, templates, or newly introduced tools. - Track Metrics: Watch sign-ups, new experiment proposals, or visits to documentation.
Tip: Divya’s team tracked how many people accessed the BBC’s new experimentation playbook post-event to gauge real impact
Experimentation Summit Isn't Just Another Meeting
As you can see, summits can come in all shapes and sizes. Completely offline, completely online, or hybrid events staring outside speakers and leadership, involving from 20 to 600 people. The key is to define your goals, enlist leadership support, showcase real-world lessons, foster cross-team collaboration, and follow up with resources that help people move from inspiration to action.
Ultimately, experimentation days are a powerful way to align, excite, and teach the rest about experimentation.
All quotes and references are drawn from the interviews with Marcel Toben, Iulian Vasilisca, Kelly Wortham, and Divya Isaiah.
.svg)
