EXL 2025

Advanced topics in Practical A/B Testing Workshop 1: Ronny Kohavi

The course covers topics not usually covered in introductory courses, including Ronny's Maven class on A/B Testing and his book. These advanced topics include:
- P-values misconceptions,
- Power, and False Positive Risk
- Why you should always give positive results a haircut
- Success rate estimation, and success of ideas vs. experiments
- The cost of false positives
- TOST and early stopping – when and how to correctly peek
- Bayesian and Frequentist approaches
- CUPED – variance reduction
- Risks of unequal variants and rare events
- Long-term holdouts
- Simpson’s paradox
- Randomization strategies: front door or at trigger point, ramp-up
- Cultural issues: can I test that? OFAT or major redesigns

Experimentation Workshop 2: Various, MC'd by Craig Sullivan

This afternoon workshop will feature various short presentations from practitioners on their applications of AI within the practice and flywheels of AB testing, customer research, and experimentation.

Exact presenters and topics TBA, stay tuned.

We do know that the OG himself Craig Sullivan will MC the AI workshop session along with Iqbal Ali.

They've been working on AI 'playbooks' for experimentation and will lead some hands on sessions on these topics.
- discovery,
- problem exploration,
- problem statement,
- solution statement,
- ideation,- hypothesis,
- hyp/problem checker
‍
Additionally Andy McNaught from Onxmaps.com has been doing some fantasic applications of LLM models and testing, he'll be attending as part of this showcase.

Conference pre-party at Zilker Brewing Company and Taproom

Address: 1701 E 6th St, Austin, TX 78702

Ben Labay & Collin Crowell: Opening

Melissa Weiss: When experimentation isn’t love at first test

We are a group of passionate practitioners energized by experimentation but not every individual, team, or organization will embrace the craft as we have. Resistance arises when one’s product development ideals conflict with misconceptions or misapplications of experimentation. Changing perceptions is crucial to fostering a successful practice. I’ll share examples of navigating reluctance at the team and org level.

Carole Robin “Connect” Workshop: Grow Impact by Building Stronger, More Robust Relationships

The premise of this session is that people do business with people -- not products, strategies, or money -- and interpersonal competence is a proven determinant of professional success. In this interactive experiential workshop (talk then table session), participants will increase their interpersonal competence and learn how to more effectively influence and lead.

‘Connect’ Moderated table session #1

Topic: Connect session
A special moderated table session that extends Carole Robin’s content to allow hands on application of principles. Small groups will perform some ‘connect’ exercises, prompts coming from Carole.

‘Connect’ Q&A with Carole

Lunch and Networking

Moderated table session #2

Topic specific to each table
You will be hand-curated into groups by interests, experience, etc. Topics and prompts will vary.

Break

Lukas Vermeer: Bridging the gap: Integrating the Product Operating Model and the Experimentation Growth Model

In 'Bridging the Gap,' we compare the Product Operating Model (POM) and the Experimentation Growth Model (EGM), revealing their shared focus on customer-centricity, iterative development, and data-driven decision-making. Join us as we explore how integrating these models can drive sustainable success in today's digital landscape.

Stewart Ehoff: The Golden Thread - Connecting research, experimentation & business ROI

Speak to a researcher in any large organization and ask them their top frustration. Most will tell you it's action and impact as a result of the work they do. This problem goes far wider and deeper than just individual researchers getting their work stuffed into a drawer, it's a systemic problem with the way companies decide to solve problems, and validate that they've been solved. Join me in this talk to explore my thoughts on how to solve this big hairy business problem, by making research actionable, connected to your experimentation program, and most importantly, to the $$$ outcomes it drives.

Moderated table session #3

Topic specific to each table
You will be hand-curated into groups by interests, experience, etc. Topics and prompts will vary.

Break + Games

DINNER, GAMES, DRINKS AND CONNECTIONS UNTIL MIDNIGHT

Pure, unscripted fun. This is where the magic happens.