What A/B Test Developers Really Want in A/B Testing Tools (It’s Not Just Features)

In Q4 2024, Speero surveyed 105 A/B test developers on the A/B testing tools they use. We took at how performance, support, documentation, and pricing stack up in tool selection.

Everyone says devs are picky about tools, but that undersells it—they're ruthless. When deadlines are tight, stakeholders are circling, and the backlog never shrinks, no one's wasting time on clunky platforms or learning curves that feel like onboarding into a new job. 

This deep dive looks at exactly what developers say they care about most when choosing A/B testing tools—from UI and onboarding speed to code-level flexibility and ecosystem integration.

Some lean on familiarity to move faster; others go for feature-rich platforms that can handle SPA testing, server-side logic, and robust APIs. Many want it all, but is there a tool that gives it?

Bonus chart on top 3 favorite testing tools for test developers at the end of the article.

Easy-to-use, intuitive tools

‘Ease of use and intuitive UI’ was the most mentioned theme. Devs highlighted picking tools that are easy to use with a straightforward, user-friendly interface. A lot more devs said they value smooth visual editors or UI that “streamlines the workflow” for test setup.

• “VWO is my primary pick for ease of use and versatility.”
• “Adobe Target provides a form-based composer for streamlined test setup”
• “I chose Convert, Optimizely, and AB Tasty because of their exceptional UI/UX design, which streamlines the workflow and makes setting up and managing tests much more intuitive.”

Everyone is busy these days. Some developers said they prefer less complex tools with a quick learning curve:

• “Speed of onboarding and knowledge base availability are super important.”
• “Easy to understand.”
• “Intuitive operation and easy familiarisation.”

Familiarity and Community

15 devs mentioned getting tools because they have prior experience with them or because they are popular in their region/industry. Tasks keep coming, but resources don’t, so it's no wonder people take familiar tools with a lower learning curve and risk. 

• “(I take) familiar tools we’re partnered with.”
• “Maturity, flexibility, and developer experience.”
• “My extensive experience with Convert has made me confident in leveraging all its features.”

Others say that in their location, certain tools are “the main tools, so it’s easy to find resources and also get up and running quickly” (availability of talent, documentation, forums, etc.). Being an “industry standard” (like Optimizely) made a tool a safe choice for some.

• “Optimizely, basically an industry standard.”
• “In Australia, these are the main tools, so it's easy to find resources and also get up and running quickly when starting a new job.”

You change a company, I change a company. Your manager gives you new tasks. You jump into a new role. The managers of your manager change him for a new manager, and he has new tasks for you. The things-that-need-to-be-done never end. 

No wonder the majority of responses are about sticking with the tools devs already know or with ones everyone’s supposed to know. Or, if this is missing, they go with tools that are easy to learn and use. Most of the time, you just need to finish the task your boss told you to finish. Easy-to-use or learn tools seem like the fastest way for it.

Flexible, Dev-friendly Tools

Developer-friendly, flexible tools were mentioned 39 times, making them the second most popular theme for picking A/B test tools.

A/B test developers are often more experienced than their web dev counterparts, so I’m not surprised that they like tools that “just get out of my way and let me be as flexible as possible from a coding perspective.” 

The theme also reflects a desire for control and the ability to implement complex tests without tool-imposed limitations. Mentions included robust code editors, support for custom JavaScript/HTML, and overall developer-centric features:

• “I prefer working directly in Target, Optimizely, and Sitespect platforms. They also allow for quick web/frontend solutions and more technical API testing approaches for better performance and ultimate control over the experiences.”
• “Maturity, flexibility (when it comes to building radical/large-scale tests, e.g., on SPAs and challenging applications).”
• “Sitespect because it’s most flexible and feature-rich for developers.”

Rich Feature Set and Capabilities

Features and capabilities were mentioned 33 times, making them the 3d most popular theme. Devs appreciate tools with a wide range of features (A/B, multivariate, server-side testing, feature flags, personalization, and more):

• “VWO and ABTasty, are good - offer some great features and are importantly, cheap and negotiable.”
• “VWO: Used it for small personal projects as personalization tool and I'm happy with their features/widgets”
• “Optimizely…offers a flexible flag feature for easy toggling of tests.”

What else? The ability to support both client-side and server-side experiments and to handle SPA (Single Page Application) scenarios.

• “Utility code and SPA support in Optimizely, jQuery integration in Convert, and DataLayer targeting in AB Tasty.”
• Optimizely… compatible with Single Page Applications (SPA) via the Visual Experience Composer (VEC) and view-trigger capabilities.”
• “Eppo because of how it handles server side and data collection.”

Ecosystem Integration Capabilities

Although mentioned the least (18 times), devs still talk about favoring seamless integration, both with other marketing/analytics platforms like GA4 and the company tech stack.

• “ABLyft is also simple to connect to GA4 and Microsoft Clarity, which are crucial tools for most of my clients.”
• “Its seamless CRM integration create a powerful, developer-friendly testing environment that prioritizes fast iteration with high conversion impact.”
• “Adobe Target: I appreciate its seamless integration with the Adobe Experience Cloud, making it ideal for creating personalized user experiences at scale.”

Also, respondents valued open APIs and data export features that let them connect experiment data to their own systems.

• “Optimizely has a very good API with good functions from a dev point of view.”
• “I'm looking for a good API, full data export, a good text editor, speed, documentation and support.”

I believe this ties in with previous responses we saw about familiarity and easy-to-use tools, but also with powerful apps with rich features. Developers want freedom in their work and support for complex testing, but also tools that are easy to add to their tech stack and mind stack. Do things they know better, but quickly incorporate things they don’t.

Reliability and Performance

27 devs talked about tools that perform well, for a long time. If it’s stable, loads fast, and avoids issues like flicker, it's good to go.

• “VWO - works ok and has good options to avoid flicker.”
• “Ability to use code editors and flicker control.”
• “...more technical API testing approaches for better performance and ultimate control over the experiences.”

They also like having confidence in the tool’s data accuracy and stats– a few devs highlighted going for tools with strong statistical engines and trustworthy results.

Cost and Pricing Model

Pricing came up less frequently as a factor in tool selection. Typically, devs on budgets or from smaller companies favored cost-effective tools (or even their free tiers). Respondents said the same when their clients have lower budgets or need to build a business case for experimentation.

• “Price point and suitability for client requirements. Availability of useful features included in plan.”
• “...we enjoyed watching Target perform, and the flexibility that the tool offered way before anyone else had something comparable and affordable.”
• “VWO and ABTasty, are good - offer some great features and are importantly, cheap and negotiable.”

Support and Documentation

Good customer support and documentation were mentioned 26 times, but this doesn’t make them any less important. A robust knowledge base, responsive support team, and community were noted as things that “make life easier” when implementing a tool​. 

• “VWO and ABTasty, are good - offer some great features and are importantly, cheap and negotiable.”
• “They know what a developer requires, good support team, clean ui, customisable.”
• “Also documentation provided by their dev is also good.”

While not the top deciding factor for all, some did mention they picked tools known for reliable and timely support.

• “Presence of reliable and timely customer support without having to pay through the nose for it.”
• “Those are focused on server-side experimentation and feature toggle-first oriented, and also have very strong stats support.”

Conclusion

Most people think developers pick A/B testing tools based on raw power—advanced features, deep integrations, and complex targeting. And sure, those things matter. 

But when it comes down to what actually drives decisions, devs are shockingly pragmatic. They want tools that are intuitive, quick to onboard, and flexible enough to stay out of the way when code gets serious. 

They gravitate toward what they know, what works, and what doesn't waste their time. Ease of use, familiarity, and workflow fit trump theoretical functionality. And while everyone loves a rich feature set or clean integration with BigQuery, none of it matters if the tool is slow, confusing, or clunky under pressure.

The wild part? Cost barely shows up. Support and documentation don’t top the list either. The tools that win are the ones that meet devs where they are—tight deadlines, shifting roles, complex stacks—and make their work faster, not harder.

You don’t win over devs with the biggest feature list. You win with speed, clarity, and just enough control to let them break the rules when they need to.