How ClickUp Was Finally Able to Trust Its Data

Olga Osadcha is the Director of Growth Products at ClickUp. It’s a B2B software for everything — from project management to docs, reporting, whiteboards, and more. Before ClickUp partnered with Speero, they faced several challenges:

• The testing program had no clear or standard process, it was a mess.
• The program was heavily tied/dependent on their A/B testing tool.
• They couldn’t trust their A/B testing tool. 
• Data was often lost, and it made them constantly question results, lowering test velocity.

During test validation, ClickUp continuously ran into issues and unexplainable mysteries with their A/B testing tool. ClickUp validates their tests in a couple of ways. Primarily through AA tests. 

Next, by checking if the traffic is the same in analytics and testing tools. ClickUp would often have a percentage of users they would lose in the tool, without knowing why.

‍

“Was it decay failing? Or other conditions? It was a constant mystery why a certain percentage of traffic was lost. This would slow down our testing velocity because now, we had to wait for longer to have the needed sample size.”

Olga Osadcha

‍

The third validation they did was to run several AA tests simultaneously and ensure the split was orthogonal: 

• One control and one treatment group in test 1. 
• One control and one treatment group in test 2.
• Then split the combination of both equally.

For a long time, ClickUp’s experimentation team ran this type of validation through their tool without realizing that whoever is bucketed into a treatment group in one test would always end up in the treatment group of the second one. At the same time, the control was a shared holdout group between all these different tests. 

ClickUp wasn’t able to see this happening. The tool simply wasn’t configured to run several concurrent tests. Despite this, its teams relied on the tool and its data to plan the next experiments and generate new ideas.

‍

“When we were analyzing the results, interpreting the data, and coming up with the next steps and validation… we constantly wondered… Does this actually work? Or is this just noise? Bad data?”

Olga Osadcha

Speero helped ClickUp solve all of this by decoupling their entire experimentation program from the tool. ClickUp was able to establish an experimentation program from the ground up.

This way, ClickUp developed a good habit of launching tests and interpreting them in a way that made sense. Ultimately, they were able to actually trust the results.

With a program decoupled from the original tool, ClickUp brought a different tool. It worked better for them and their setup. The new tool also lets them automate all the manual processes, (not possible before the decoupling). 

‍

“Even though we have a tool we can trust now, the CRO program itself doesn’t rely solely on that tool and the tool doesn’t dictate what we do. Basically, what we have now is a robust foundation and the new tool helped us speed up and be more efficient.”

Olga Osadcha

‍

‍

‍

ClickUp now loves its new tool. Why? They trust it. They aren’t losing traffic allocated to experiments. Even when they expected the new tool to fail… it didn’t.

‍

“We ran pretty extensive tests with it, and it worked well in situations where the prior tool would absolutely fail.”

Olga Osadcha

‍

The benefits go beyond the tool. These days all their data lives in a warehouse rather than in the tool. They can analyze the results on both sides. Lot less guesswork. Lot fewer reruns and manual tests.

‍

“I had a great experience working with Speero. We had a healthy cadence of check-ins and continuous progress. We had a good machine where we were communicating and making progress weekly.”

Olga Osadcha

‍