Before Circus Austin (our flagship AI+Experimentation conference), we asked everyone coming which table topics they wanted to spend two days on. The 222 people who answered cast 1,057 votes. AI workflow and automation took the most, with 307. Trust and governance came last, with 87.
The keynotes then spent most of the time on trust, guardrails, decision rules and memory.
Marty Cagan, who opened the main stage on day one, pointed back at that survey slide and told the room,
"I need to try to convince you, you're focused on the wrong thing."
Meaning he thinks we should NOT focus on AI workflows and automation.
Later in the day, Alexey Komissarouk showed an automated growth loop at Anthropic (with lots of very cool details I can't share publicly!). The automation people asked for is real, and it works. He showed us the future. Alexey showed how it moves the hard part of AI in experimentation somewhere else (and not where most of the survey votes were pointed).
My read after two days in the room is that when treatments are nearly free and most of them are bad, an experimentation program earns its keep by choosing well. Taste?
Choosing what gets traffic, what counts as a win, what goes into memory as fact, and which results deserve a "why." Selection is the job now.
Travis Brooks said on day two that if you do this right, your win rate probably goes down.
Below I go into this 'loop' step by step, then where the speakers disagreed, then what I'd think to do about it. Short talk summaries are at the end.
Who was in the room
Circus Austin ran October 19 to 21 in East Austin, co-produced by Speero and Datadog. 75%+ of the room ran experimentation in-house. 54% work at companies with 1,000+ people, 76% have 5+ years in the field, and 17% run 1,000+ tests a year.
Building is getting cheaper. The bottleneck is moving.
Travis Brooks runs product for Microsoft's experimentation platform (he did the same job before at Netflix). He calls it "treatment inflation". AI made treatments cheap, so now the constraint is traffic and people.
"If treatments become plentiful, if we can just go out and grab treatments from anywhere, what becomes the bottleneck? Actually, it's often users. It's often samples... In fact, it's you and me. Like we're all the constraint." Travis Brooks, Microsoft
Teams used to spend about 95% of their effort building a treatment and 5% testing it. Building got cheap and testing didn't, so testing is now about half the work. In his words, "the ROI in making experimentation simpler just went up."
Cagan said the same thing from the product side: "The hard part is not building. The truth is, the hard part was never building..."
Nils Stotz (Zalando) shared the line I heard most over the two days. He got it from his colleague Marcel, who said it at Circus London in May:
"AI has collapsed the cost of change. It has not collapsed the cost of being wrong."
Building is cheap. Being wrong isn't. So the work moves to everything around the build, which is the rest of this post.
The loop, step by step
Three speakers showed working loops: Alexey at Anthropic, Faith Shin and Joana Kogan at Wayfair, and Che Sharma (founder of Eppo, now Datadog). They look a lot alike. Agents now do most of the generating, building and running. Humans kept selection and the decision.
Side note and no shame pitch to you: This is the kind of loop we now set up for clients at Speero. We call it GXOS. We build the AI harness for your customer insight and experimentation programs (the agents, the skills, the shared memory and the decision rules), connected to your own data and tools. If you want help getting one running, send me a DM.
Ideas are cheap, mostly bad, and they run out
Travis showed that the average treatment in a program is negative.
"The average treatment is negative. This means most of your ideas aren't good. They mostly hurt, right? This is okay. This is true for everybody in the room." Travis Brooks, Microsoft
AI doesn't change that. Alexey asks the model for ideas grounded in every source he has (past tests, customer feedback, surveys) and expects most of them to be bad. "You're just trying to find the 10 to 20% that are actually worth running."
The good ideas also ran out. After the first few batches, the system started repeating itself. Che's fix is new inputs, like agents watching session replays and reading support tickets. Research that used to be optional becomes, in his words, "load bearing."
I like that one a lot. Research is now what feeds the loop. (I made a poster on exactly this: an AI setup for running customer insight programs.)
Selection is the job
If ideas are free and traffic isn't, the step that matters most is the one before the test. Every keynote had a version of it.
- Travis: pre-test selection. Offline evals and synthetic traffic filter ideas before they touch users.
- Alexey: throw out duplicates, ideas you already tried, and ideas too small to measure. Then prioritize with RICE, except Effort now means "stamps," the number of humans who have to approve the change.
- Wayfair: their agent, Gambit Assistant, checks every idea against past tests. In the demo, a user pushed back and it didn't budge. "No, I'm holding my stance. This has been neutral as well."
- Che: test where the metrics actually move. Some parts of a product only ever give you flat results. Spend your traffic on the parts that swing.
"I'd rather ship 2,000 lines of code that we can approve than a two-line change that I need to go talk to five other teams about, because the bottleneck is how many humans need to touch this before it goes out." Alexey Komissarouk, Anthropic
Travis went further. If treatments are cheap, stop powering tests for small wins. In the same time you can run 2 tests that can detect a 1% effect, or 10 tests that can detect about 2.2%. The 10 find the rare big wins faster.
"This means my win rate will go down. Booooo. The number of times I miss a win will go up. Booooo. I will call things wrong. I will say things are flat that aren't flat. Booooo. But big wins will go up." Travis Brooks, Microsoft
It only works if a treatment takes days to build. If it takes 3 months, don't.
Building is easy now. The cleanup isn't.
Building was the easy part of every loop on stage. The problems showed up after. Alexey said models are still bad at design, at fitting something new into a page without breaking it. Che described engineers getting "a fire hose of slop PRs in their face all day from people they don't recognize." His answer is to treat experiments more like software releases, with CI/CD, automated QA, and rollback when monitoring catches a problem.
Decide the rules before the result
Nils had the best 'story' of the event. A Zalando team tested an express checkout: one button, address and payment assumed. Conversion went up. So did support tickets and returns, and the margin hit ended up bigger than the gain. The checkout team said ship. The returns team said roll back.
"Two teams working in the same company looking at the exact same results, proven by an experiment, come to fundamentally different conclusions." Nils Stotz, Zalando
His fix isn't new. "Pre-registration". It's all about proper intake!
Set the primary metric, the guardrails and the thresholds before the test (returns and support tickets can't rise more than 1% or 2%). Agree on three outcomes up front: ship, stop or escalate. Leadership sets the rules, teams run against them, and the rules get reviewed on a schedule (his slide said quarterly). That review is the only time the argument reopens.
This matters more with agents.
"The agents that are doing this, they optimize ruthlessly. They do not care about dark patterns or anything like this if we do not tell them to." Nils Stotz, Zalando
The contract is how you tell them. If you don't write it down, the agent guesses, and my bet is it picks whatever metric is easiest to move. Che wants the goals written in plain sentences in the metric layer ("increase bookings this year while keeping support tickets constant"). Travis wants decisions to be rules-based, with one exception: "If it's unclear, punt it to a human."
BTW, this is the "metric driver" part of the GXOS setup we do for clients. We map which metrics drive which outcomes, then write the goals, guardrails and thresholds down where the agents can read them. That way the trade-offs are decided before a test runs. It's not only a proxy metric system, it's a self-maintaining governance of 'beliefs'.
One more from Zalando. Returns can land 30 days later, after the test is over. Keep watching your guardrails after the readout. Cohorts ftw.
Memory works both ways
Wayfair let anyone run a test. That worked, and then what people learned left with them.
"Teams can put real time and effort into an idea, build it, run the test, make the decision, but the learning and the decision that they made kind of moved on with them and we never captured it." Joana Kogan, Wayfair
Their AI assistant writes every result back into shared memory, so "test five starts higher than test one." Faith Shin added the warning.
"Compounding is neutral and it will really amplify whatever you feed it." Faith Shin, Wayfair
Bad tests become bad advice. So Wayfair filters weak tests out and weights results by how good the evidence is.
Get the why
"You can run 100 A/B tests, all you know is no, no, no, no. Yes, yes, no, no... But you don't really know why." Marty Cagan, SVPG
Cagan thinks that in about a year, every experimenter who's still doing well will be good at both qual and quant. (was a bit of a theme)
Aleks Bass and Leigh Smith from Typeform showed how. They ran a randomized study comparing a static video survey, an AI-moderated interview and a human-moderated interview. The AI interviews got 4.7x the words of the survey, took about half the time of a human interview, and people said they felt more able to be honest.
Now they run it inside A/B tests. A random slice of each arm gets an AI interview. A flat test used to just tell you no. Now it comes back with a reason.
Faith made a simple point. AI is great at spotting patterns in your data, like "people who used feature X bought more." But it can't tell you if feature X is why they bought. Maybe those people were going to buy anyway. You still need an A/B test to know that, and research to know why.
Trust
Katie Dove (Irrational Labs): "We are becoming a building society." We ship faster than anyone can use. Pendo's 2019 report found 80% of features are rarely or never used. Her three questions for any launch are will they try it, will they trust it, and will they come back. Her tips worked just as well when I swapped "customer" for "the PM who has to act on your agent's output."
Alexey said AI gets no grace at work. "The first time it gives you a bad idea, it loses trust." Wayfair built trust slowly. The agent lives in Slack, it started with one narrow job (test plans), and it shows which past tests drive each suggestion.
The problem nobody has solved yet is that every agent that needs a human creates a request.
"We moved the coding bottleneck, and we now have an approvals bottleneck." Alexey Komissarouk, Anthropic
Where the speakers disagreed
Some of these they argued on stage. Others are my read from putting the talks side by side.
1. Testing faster vs remembering right
Travis wants teams to run more tests with the same traffic. The lever is how small a win each test is built to catch. A test that can spot a 1% lift needs a lot of traffic. A test that only has to spot a 2.2% lift needs much less, so it finishes sooner and you can run more of them. The cost is that smaller wins slip through. In his words, you'll "say things are flat that aren't flat."
Faith wants an AI assistant that remembers every past result and uses it to judge new ideas. Her warning was that it repeats whatever you feed it, good or bad.
Put those two together and you get a problem. A test that could only catch a 2.2% lift comes back flat. The idea might still be worth 1%, the test just couldn't see it. But memory saves it as "tried it, didn't work." The next time anyone suggests that idea, the AI tells them to skip it.
The fix is simple. When you save a result, also save how big an effect the test was able to catch. Then the AI can tell the difference between "this didn't work" and "this didn't move things by more than 2.2%, so a smaller win is still possible."
This is part of the GXOS doctrine we set up for clients. Every test gets a power check at intake, so before it launches we know the smallest lift it can catch. When it finishes, the result is logged with that number (a precise flat is a different thing from an inconclusive one) and linked to the beliefs it was testing, the metric drivers from earlier. A belief only gets upgraded when a test actually settles it. So when the AI reads that memory later, it knows which ideas are dead and which just haven't been tested hard enough yet.
2. Where the human sits
At Anthropic, humans approve ideas at the start and the agent handles a lot after, including turning tests on and off. Alexey called that "a terrifying thing." At Wayfair, the agent drafts and configures, and humans make the ship call at the end.
"I like being able to make a decision that our customers are gonna see. I don't want to automate every single piece, but we'll find out." Joana Kogan, Wayfair
I think both are right. Put the human where the decision is hardest to undo.
3. Tools vs thinking
Che thinks most "culture" problems are really tool problems. Fix the setup and people will test. Cagan warned about "such a hunger for process and frameworks and tools as a way of abdicating thinking."
I think they're both right, they're just talking about different teams. Some teams barely test because getting a test live is a pain. Better tools help a lot there. Other teams test all the time but rarely stop to ask if they're working on the right problem. A new tool won't fix that one.
4. Small tests or big swings?
Ryan Lucht (Datadog) reopened this one. The old advice was to run lots of small, cheap tests. His slide said "Big swings are way cheaper now!" and then "But are external conditions still the same?"
Travis says go for bigger wins. Alexey says automate the small stuff so people have time for big swings. Cagan says use experiments to find a bigger mountain.
My take is that big swings got cheaper to build, but they still need the same traffic. So pick the parts of the product that actually move.
Ryan didn't pretend to have the answer either. His last two slides were "Big questions, no clear answers!" and then "Iterate in community." That was basically the brief for the whole event. Nobody in that room has this figured out yet, speakers included. The way through is comparing notes with other people hitting the same walls, and testing what they learned in your own program. That's why so much of Circus happens at the tables.
What I'd do in the next 12 months
Honestly, most of the speakers were presenting some form of the future. Maybe not more than 6 months. Coming soon to a program near you.
Alexey's prediction: "I would be shocked if we met here next year and you do not all have some version of the system running internally." I wouldn't bet against him. If he's right, the loop itself stops being an edge, and teams compete on everything around it.
- Report what you killed next to what won. Track ideas killed before testing, guardrail breaches, and tests stopped for setup problems. Stop leading with win rate. Program thinking ftw.
- Write the rules before an agent touches anything. Primary metric, guardrails, thresholds, and ship, stop or escalate. Review them on a schedule. Pre-registration ftw.
- Label every result with its power. So "flat" doesn't turn into "dead." Size&Value ftw.
- Treat approvals as capacity. Don't let every agent request land in everyone's Slack. Stop the slop grenades!
- Keep watching guardrails after the test. Returns and churn show up late. Driver metric setup!
- Add a qual arm to your A/B tests. Leigh's math is that 1% of a 10,000-person arm is 100 interviews. Regardless, it's powerful to pair VOC with your A/B.
- Feed the loop with research. Agents run out of good ideas on a prompt alone. Context engineering ftw.
- Start narrow. Put an agent in a Slack channel and ask it to pitch ideas. Wayfair started with test plans. Get one win, show the reasoning, then ask for more. It's also the first place we'll experience multi-player AI.
We call this whole set an experimentation operating system: the goal tree, the decision rules, the learning loop. What changed this year is that agents need to be able to read it too.
Which loop are humans in?
The Circus tagline is "join the humans in the loop." After Austin I'd ask which loop.
- Agents are taking the inner one: generate, build, run, analyze.
- The people on stage were holding the outer ones. What to sacrifice. Which problem to work on. Why something worked. Whether anyone trusts it. What counts as fact.
I don't know where that line sits a year from now. Alexey's guess for agents that only ask permission when they need it was "six months? Ask me next year."
The talks, one by one
Ryan Lucht, Datadog: opener
Ryan asked three questions. What is our job now, what are we working on, and where should we stop optimizing? AI has everyone doing everyone's job, he said, and experimentation leaders still own the standards, the learning loops and "epistemological sanity." He reopened the small tests vs big swings debate and closed with "Big questions, no clear answers. Iterate in community."
Marty Cagan, SVPG: Experimentation, the Product Model, and AI
Marty's first talk at an experimentation conference. In most companies, experimentation is QA at the end of the line, and AI is automating that whole factory. In the product model, experimentation is how you decide what to build. He wants experimenters in the room for that call, good at qual and quant, using evals for AI quality and A/B tests for value.
"My view... your highest order contribution is this. It's not telling us which treatment works best. It's figuring out what to build." Marty Cagan, SVPG
Nils Stotz, Zalando: It's the tradeoffs, stupid!
62 million customers, 600+ tech teams, and no easy wins left. Every metric you raise hurts another one. His answer is a contract set before the result, reviewed on a schedule and tied back to strategy. His warning for the AI era is that agents optimize ruthlessly and won't avoid dark patterns unless you tell them to.
"Strategy is what you're willing to sacrifice. But ultimately, the only way to do this is if you have an experimentation platform that can show you these trade-offs." Nils Stotz, Zalando
Alexey Komissarouk, Anthropic: Lessons learned from automating growth experimentation
Last year Alexey said he'd automate his job in six months. This year: "I am still employed in Anthropic, so it was not six months." He showed an AI loop running growth experiments where agents brainstorm, filter, build and analyze, and humans approve. Coding and analysis turned out easy. Running out of good ideas, design, trust and approvals turned out hard. He thinks most teams will build or buy one of these within a year.
"We moved the coding bottleneck, and we now have an approvals bottleneck." Alexey Komissarouk, Anthropic
Faith Shin and Joana Kogan, Wayfair: Building an agentic learning loop
Anyone at Wayfair can run a test, so the problem became losing what people learned. Gambit Assistant lives in Slack. It checks ideas against past tests, drafts plans, configures tests, analyzes results and writes them back to memory. Faith covered the hard parts: bad inputs, pattern vs proof, and adoption. Humans still decide what ships.
"Compounding is neutral and it will really amplify whatever you feed it." Faith Shin, Wayfair
Travis Brooks, Microsoft: Treatment inflation
When treatments are cheap, users and people become the constraint. His three rules are test more, test only the good ones, and look for bigger wins. Track two plots, your treatment distribution and your cumulative impact over time, and make the second one go up faster.
"When you make that cumulative impact over time plot, you just need to make it go up. Ideally, make it go up faster. That's your role." Travis Brooks, Microsoft
Che Sharma, Datadog: From early wins to experiment singularity
Che stepped in for John Cutler and built his talk the night before. He laid out a maturity curve from patchwork to continuous. Early on, nobody tests because nobody has won yet, so fix the diagnostics and get a winner. Later, stop babysitting single tests and run a program. At the agent end you need explicit goals, new idea inputs, modern CI/CD and a test catalog agents can read.
"It's okay for your experiments not to work, it's not okay to be slow." Che Sharma, Datadog
Katie Dove, Irrational Labs: Behavioral science, an AI builder's cheat code
We build faster than customers can notice. Her cheat sheet has three questions. Will they try it (defaults, make it obvious)? Will they trust it (comparisons, make it reversible, promise less)? Will they come back (cues, one decision that locks in future use, value that builds up over time)?
"I can get someone to do anything once, but to get them to do it again and again is a whole other story." Katie Dove, Irrational Labs
Aleks Bass and Leigh Smith, Typeform: Behavior is an outcome, experience is the mechanism
Adding "it depends" to their survey scales showed that at least 24% of answers were being forced into boxes. AI-moderated interviews beat static surveys on depth and honesty. Now a slice of every test arm gets one, so flat tests come back with a reason.
"I want to compel the world to collect more qualitative data, to know the humans before the machines take us all." Aleks Bass and Leigh Smith, Typeform
Posters and lightning talks
23 posters, 12 two-minute lightning talks, a cowbell, and Martin Tingley as MC. Ilya Izrailevsky from DoorDash won the room with a poster arguing AI should ship fewer wrong decisions. Across all 23, the pattern was small teams building memory, evals and human sign-off on top of tools like Airtable, Slack, n8n and Claude Code (we like that stack too). Synthetic users showed up as a pre-test filter more than once.
More on the posters and what I took from them in my LinkedIn post.
.svg)
