Experimentation / CRO

Experimentation and Testing Programs acknowledge that the future is uncertain. These programs focus on getting better data to product and marketing teams to make better decisions.

Research & Strategy

We believe that research is an integral part of experimentation. Our research projects aim to identify optimization opportunities by uncovering what really matters to your website users and customers.

Data and Analytics

90% of the analytics setups we’ve seen are critically flawed. Our data analytics audit services give you the confidence to make better decisions with data you can trust.

9 Lessons Learned from Running a CRO Agency

Run an agency? Offer conversion optimization services? Then you know it can be hard.I’ve been running an optimization agency for 6 years now. Here are my top lessons I’ve learned along the way.

1. Build a name, build an audience

It’s so much easier to run an agency business if you have a personal brand name and an audience. Think about it: you’re in the business of expertise. You’re not in the business of selling services, per se. Clients don’t pay you for your time, they pay for your expertise.So when a potential client thinks, “I need someone to come in and work on my site,” who will they respond to better: the one with a big audience and lots of social proof or someone that no one has heard of?

All things equal, there's less friction and uncertainty in going with the one that has the brand name. Cold outreach can work in getting clients, but in my experience, warm is 10x better. If leads come to you, then the sales call that follows is now about qualification. You’re in the position of power. It makes a massive difference.

If you have to sell them the idea that they should do CRO and that they should hire you, there’s way more friction.

How do you build an audience?

Today, there are so many ways, and the channel isn't the important part. There's blogging - my top choice, but also podcasting, YouTube, Snapchat, Instagram, etc. You can also get creative and organize communities online. It's easy to do that now with Facebook groups or Slack groups. Though, blogging has the benefit of SEO, which lasts a lot longer than other channels.

If you’re only working in a local market, you can easily host meetups and build events around what you do. Speaking at events works as well, and there are many that allow newer speakers, like Measure Camp, Product Camp, and local marketing meetups.

Relying only on referrals is a tough life. You’re not in control of the inputs or the outputs when you only rely on referrals for business, so if referrals stop coming, then what can you do? My advice: referrals are great, but have another lead generation channel as well. If you don’t know where to start building a name for yourself, guest post for the CXL blog.

2. Getting results is hard. Competition is intense.

If you read a lot of hacky blog posts, you might think this is easy. Change this button color, make it bigger, change your headline - 200% increase!It reality, it’s hard work. When running an optimization program, consider three metrics that contribute to success:

  • The number of experiments you run.
  • The percentage of winning tests.
  • The average uplift of successful experiments.

The first lever is all about resources and politics. How much traffic do you have to work with? How many designers and developers do you have working on the problem? How fast does the client allow you to work? These things are fairly limited in terms of their ceiling.

Once you max out on tests given traffic allotment, you can’t really push through more just because you want to.

However, the second two levers rely on expertise, and that’s where you can really bring value. Competition is tough, and getting harder all the time. You might think your competitors are not as smart as you are, but bear in mind that your client might not know this. Your client hears the same pitch from everyone, so how do they know you’ll deliver better results? And if you can’t deliver, know that you’ll be fired very fast.

In order to win in this business, you need to build your brand, but outside of that: Have the damn expertise. If you’re just winging it - “oh, I read a blog post or two so I know what I’m doing” - then you’re going to get fired. Sure, everyone has a bad month. Even the best agencies do. But word travels, and if you’re consistently bringing terrible results, you won’t be in business long.The best agencies invest in learning at all levels of their team, and they don't stop learning once they hit a basic threshold of knowledge.

3. Work with bigger businesses

Conversion optimization doesn't align with small business. For one, there are inherent limitations with traffic. If you don’t have the traffic to test, there’s not much we can do. Second, you can't bring as much ROI to small businesses.

Here's a quick exercise to prove this. When taking on a potential client, ask yourself, "If we improve their sales by 1%, what does that mean to their business?"If it's a company that does $1M in revenue per year, and we increase sales by 1%, that’s $10,000 extra revenue (who knows about profit?).

Maybe you can get more than one percent, but how long does it take you to get that lift? How much do you charge per month?

The ROI is really hard to achieve with this size of business.You may think, “oh hey, I charge $200 a month!” Well,

A) you’re an idiot

B) No company that makes over $1M a year is going to hire someone that charges $200 to do CRO. It’s just not the way the world works.

But if your company does $1B a year in annual sales, you’re going to achieve a crazy ROI with only a 1% increase. And with that crazy ROI comes happier clients, better testimonials, and more referrals.

As a ballpark, try not to do business with companies that do less than $10M a year in online revenue. Of course, there are exceptions, but that’s a good heuristic to follow.

In my experience, one optimization project manager can work with a maximum of 3-4 clients without sacrificing quality. Getting results is hard in the first place, so if you don’t have the focus, you won’t get results.

Also, think about it from a business perspective. There's not much margin on an agency business anyway, so how much do you think these clients should pay you for an engagement, given the assumption of 3-4 clients per project manager? Whether you’re working for a mom and pop shop with low revenue or a $500M a year ecommerce giant, the work you put into get results is the same. It’s just that for one of these, they can pay you more and they receive more ROI for your efforts. Why waste your life with the more stressful and frankly less valuable of the two types of clients?

4. Differentiation is hard. Branded methodology helps.

I hear it all the time from potential clients: “The other guys have the exact same pitch. What makes you different?”

If there’s no difference, why should they hire you? Of course, you can differentiate in nominal ways: better case studies, better testimonials, slicker landing pages, etc. And you can compete on price by being the cheapest. But it’s a race to the bottom: there’s always someone cheaper than you.

What’s the solution, then? Branded methodology.We have the PXL prioritization model and the ResearchXL conversion research framework - both widely adopted in the industry. It definitely helps to position us as the knowledge leader in the space.


Building frameworks also helps standardize the industry practices. In CRO, this is particularly important because of the loose association of “experts” that may or may not be doing things the right way. When there are rogue consultants promising big lifts and just throwing ideas at the wall and seeing what sticks, it’s bad for the industry. Models and frameworks give everyone a standard process, and it helps everyone (and gives you a brand boost).

5. Qualify

Lead quantity can be a vanity metric. We used to waste so much time and money on clients that weren’t qualified. It’s a huge issue. How do you eliminate that waste of time?

The easiest way is to improve lead qualification. One way is with lead scoring. There are so many tools out there for this that it’s easy to get lost, but the essence is that they score based on a series of inputs (company size, industry, visited specific pages, etc.). That way you can automate lead qualification in a way.

Another way is to filter out bad leads before they can even fill out the form. The best way we did this was by adding a form on our landing page that asks for their budget for the project. The lowest value is high enough that we eliminate the tire kickers and wastes of time:

6. Fire shitty clients. It’s not worth it.

No matter what stops you put in place, you’ll get a few clients from hell.

Clients from hell will suck the life out of you and your business. In building a conversion optimization agency, we’re chipping towards our dream and our vision, and a lot of that is daily grind. And to get to our goals, we need to enjoy the grind. When you have a horrible client, this chips away at that happiness and you sacrifice the long term goal for the short term cash flow. Sometimes this is necessary (you have to feed your family), but if you can avoid it, avoid it.

Signs of a client from hell:

  • They’re anal retentive about every little detail that goes into the contract or the proposal.
  • They want 17 meetings before they’re ready to move forward.
  • They try to haggle the pricing, hard

If we get a lead that comes in with the above signs, I usually drop it pretty quickly now. It's as simple as putting out an ultimatum: “Guys I don’t think this is working out. Either we change the way we’re working and you give us more freedom to do our stuff, or we part ways. If we can come to an agreement, that's great. Otherwise, no hard feelings."

7. Productize. Don’t sell hours.

Most people sell by hours. Don’t do that.You’re in the business of expertise. You sell your expertise, not hours.

Selling hours is a dick move to your clients. Why? People that are way better get the job done faster and more junior people take longer. In addition, it incentivizes you to take as long as possible to get the job done. You know, you get paid by the hour, so why hurry?

The hero of this story is value-based pricing. As long as the value is there, no one should care how many hours it takes you. Read The Consulting Bible, and start pricing your services based on the value you bring.The goal with our agency is 5-7 times ROI per client, so we plan out our value-based approach using that. That’s how we figure out our monthly flat retainer as well as our other project-based offerings.

To do this yourself, you can look at inputs. How many tests per month can you run? If it’s a low traffic site, you’re limited by statistics (another reason to work with bigger clients). Your monthly retainer should reflect a certain number of tests launched per month. Think about:

  • How many for desktop?
  • How many for mobile?
  • Analytics and measurement included?
  • Research and insights provided?

You can also productive other services. Do you provide one-time analytics or conversion audits? These are great, because you usually find a million problems and then you can productive the implementation/fix as well.

On another pricing note, I'd stay away from pure revenue share/performance-based pricing.

Companies that want you to do that instead of flat fee generally don’t have the money to be worth it. It's only the small businesses that want it - because they don't have the cash to pay you. No large enterprise has ever asked for it.

If they were a big enough company, the upside for you is too huge with revenue share. Avoid revenue share at all cost, mostly because of the fights that will inevitably happen. Once you get wins, your client won't want to pay that much and will start to find reasons why you shouldn't be paid that much. It often takes lawyers more time to agree on a language in the contract than the client engagement itself.

I have seen some successful implementations of hybrid models though: flat fee + performance bonuses. So, if you reach certain milestones. Stay away from pure performance though.

8. Training CRO folks takes a lot of time

If you’re going to hire CRO folks that have never done it before, expect a long ramp up period. I used to believe in the phrase, “hire for talent, train for skill.” You have young and ambitious people available for an affordable price. But I realized that the ramp up period was incredibly long. Why is that the case? To be good at CRO you need to know a lot about a lot:

  • Testing
  • Statistics
  • Psychology
  • Copywriting
  • Design
  • Analytics
  • Front End Development

That’s just the CRO-specific skills. It takes years to become decent at this. If you're client-facing at an agency, you also have to have account management, presentation, and project management skills. You need to be excellent at navigating political waters. Think of it like a graph, where on one axis is the quality of work and the other one involves the quality of the relationship. You want to be in the quadrant with the good work and good relationship, otherwise you won't be working with that client for long.

Your people need to be good at so many things. I’ve had many great analysts that were skilled at specific things, but when it comes time to talk to a client, things fall apart. It's hard to find a unicorn that is good at both sides of the equation.

Because of the difficulty in hiring, it's necessary to retain employees for longer, and also to hire experienced CRO people. Turnover and constantly training new employees will be too costly and too time consuming for your business. However, you can speed up your training. On-the job training works, but also formal training like what we offer at CXL Institute. We now have hundreds of companies sending their conversion optimization employees being trained there. It works.

9. Find mentors who have been there before

When I first started my agency, I was constantly reinventing the wheel because I didn’t know any better. Then I started going to conferences, and you know the value there isn’t necessarily the talks, it’s the people and the networking.

I would find the people who were running successful agencies and interview them. I asked what they were charging, what their processes were, etc. and we implemented the lessons at my agency. The greatest thing you can have: a one-on-one mentor, someone who has been there and is a master of the craft.


Running a conversion optimization agency is hard, make no mistake. But it’s rewarding. No matter what, you’ll make mistakes and learn from them. But do what you can to avoid them by learning from others.

Related Posts

Who's currently reading The Experimental Revolution?