Can you imagine running the experimentation program for a company that makes approximately two billion experimentation decisions per day? That’s the reality for Aaron Wroblewski, Senior Manager of Machine Learning at Zillow, who oversees experimentation, culture, platform, and science.
As Zillow’s first employee dedicated purely to experimentation, Aaron’s goal is to leverage experiments to measure what’s working for customers, and what’s not.
On Testing Insights, Aaron pulled back the curtain on the tools that Zillow uses today to run their experimentation program, and the long-term vision that he has for Zillow’s future.
In this interview, Aaron covers:
1. His thoughts on the “traditional” program tooling framework
2. What tooling looks like at Zillow
3. How third-party tools enhance Zillow’s flywheel
4. The tradeoffs to a hybrid approach vs. a single platform
What do you think of the “traditional” program tooling framework?
Aaron Wroblewski: “We have some different approaches that we're using [at Zillow] in the customer and the product area…for example, SEO experimentation is super important for Zillow. SEO is one of the largest sources we have of traffic. I think it's about 60% of our traffic is organic from Google, which is really wonderful. We're so thankful for that. And as competition continues to heat up in the space, we're having to fend off on a local basis some of our competitors,”
Zillow’s positioning means that experimentation is an extremely important lever. In the SEO space, they often do assignment by page or by market. This entails a number of hyper-local experiments on Zillow’s website which might be relevant to a specific section of a specific page.
Customers may or may not interact with these elements, but it’s something they test in order to highlight user-generated content or market content for search engines.
What tools facilitate your current approach?
Aaron Wroblewski “We've gone through cycles in our SEO testing. But the current approach is that we have some internal tooling for assignment. And we either do assignment manually in a notebook and upload that to our routing software. Or we use this on larger experiments, where we're just randomizing a large set of pages. For example, all of our “home details” pages, we might use our assignment service, which can randomize by any of our entities, user, page,”
On the backend, Zillow’s uses a combination of internal metrics and third-party metrics related to how they rank in different regions and in different languages.
Are there third-party solutions that are helping you do traditional user-based assignment types of testing?
Aaron Wroblewski: “Overall our experimentation strategy is moving from totally siloed—where different business lines and different organizations made their own choices—to a more consolidated hybrid strategy. In regards to SEO, if we're moving toward a situation where we think that we can get a lot of value out of buying tools for assignment. And we think that the value that we can get out of that is assignment is a relatively simple but high operations task. Especially for a site the size of ours, where we're doing more than 2 billion experimentation decisions a day. We think that by leveraging external tools for assignment, we can accomplish at least all of our current needs for the vast majority of our experiments across users, markets, and SEO,”
Zillow’s self-proclaimed “special sauce” is how they measure customer interaction and metrics, and are willing to constantly iterate based on that feedback.
Aaron finds that they are best positioned to do experiment analysis internally, and customize that in the way that works best for the individual experiment or business unit.
How are third-party tools explicitly enhancing your customer-focused flywheel?
Aaron Wroblewski: “One of the tools that we leverage is Optimizely. One of the key values that I see in Optimizely is this ability for optimized lead-to-support no-code or low-code deployment. And that has been something that's allowed our marketing team to iterate really heavily and make a lot of money, a lot of revenue, and a lot of conversions for Zillow in, for example, lead form pages, where small changes make a huge impact to customers’ ability to convert and for their motivation to convert. And Optimizely allows non-technical teams to visually make changes in a visual editor right to the page, and also allows us to, as those changes become operationalized, as we decide this is something we want to ship long-term, allows us to take that code, bring it internal, ship it within our internal stack, and continue to leverage the same non-technical user-controlled assignment and launch,”
Internally, Zillow is able to import all the data that Optimizely has about what users were assigned to and what results that they measured were, and enrich that with their own down-funnel data.
This helps them answer questions like:
- Did users end up becoming premier agent customers?
- Did users become partner customers of Zillow?
- Did users on a customer-facing page end up converting and selling their home to Zillow?
- Did users reach out to a Zillow agent?
Are there trade-offs to that hybrid approach, and is the future a single platform?
Aaron Wroblewski: “I think that longer-term, there's a lot of advantages of centralizing into a single platform. For one, if a marketer today tests something with Optimizely and it's a win, that's not the end of the road. We can't just launch that and expect that to live in Optimizely as an experiment launched at 100% forever. That raises a lot of operational concerns for us. One of which is the latency that those tools introduce as you include them in the pages. And we know latency is money, and we want to avoid that altogether. Because the more latency we apply to customers, the slower our sites perform, the more likely users are to abandon and not complete their tasks. But I will say it's a really great solution for us right now. We've got marketing teams with great ideas. We've got a company that's 2,000 engineers, only 25% engineering. So there's a lot of folks out there with great ideas that they want to test them and they want to be able to prioritize them in that limited engineering bandwidth that we have.”
In the short term, Aaron sees third-party tools like Optimizely as a very interesting trade-off, and one that makes sense for Zillow right now as they continue to experiment and foster an innovation mindset.
In the long term, Zillow is investing in new in-house solutions. One concept they are exploring is whether they can create an internal low-code/no-code experimentation platform that allows users to customize and edit react controls.
For Aaron, the experimentation platform itself requires an innovation mindset. It’s not something that can be built in a day, or that can remain static forever. Right now, the focus is on creating a sustainable pipeline of ideas, and enabling a full-stack toolset as Zillow’s experimentation program matures.
.svg)
