We've created Zuko, our next-generation form analytics platform. Explore Zuko

Interviewing A CRO Pro: Peep Laja

Conversion Rate Optimisation

Peep Laja is best known for founding ConversionXL, a conversion optimisation agency in Austin, Texas. He’s straight-talking; in his own words, he will “call your bullshit”.

He embodies the philosophy of optimisation; guesses and hunches do not fly. Got an idea? Test it and bring him the data.

Highlights from the interview

  • Amateurs look for shortcuts. You have to be ready to bust your ass
  • Sample sizes are just maths – check out Peep’s advice to put paid to any debate on sample size ever.
  • “Some very well-known blogs are just so full of crap.” We need to protect the industry from wannabes.
  • ConversionXL is launching a new research product that will inject much-needed original studies into the industry.
  • Tickets to ConversionXL Live conference come with a bundle of CRO tools free, scroll to the end for details.
Peep Laja

Peep Laja, founder of ConversionXL

We started by discussing the hot topic of the day…


Al: Hey, Peep, how you doing today?
Peep: Great, thanks. The hot topic of the day has been the subject line of our newsletter. I wrote, “You’re Full of Crap”. It’s referring to the latest article I published about narrative fallacy. So I have some people giving me high fives and other people yelling at me.
Al: What do you expect the open rate to be?
Peep: I mean, based on today’s response, I would imagine it’s at least plus fifty percent of the normal rate. The results will be in in three days or so. We’ll see, maybe insulting people is the way to go.
Al: You don’t know until you test.
Peep: Exactly!

…before getting into the swing of the interview:

For those that don’t you, what’s an average day at ConversionXL?

Okay. So we are part agency, which is where the most of the money is coming from. So, today, we are 15 people. And most of the people work on the agency side doing enterprise conversion optimization. So working with companies that do at least, well, medium to large size, at least $10 million a year in online revenue.

We do done-for-you conversion optimization. So anything from data gathering to analysis to designing and coding treatments, and conducting post-test analysis. So the full circle of conversion optimization done for you without needing any resources besides money from the clients.

The rest of the team is content. Our blog is the most popular conversion blog in the world, in terms of monthly readers (220,000+).

ConversionXL team

The ConversionXL team

This year we’re also setting up a lab, ConversionXL Institute [Editor: So-called at time of interview].

We’ve already hired some people and we’re hiring more for the research lab. I’m going to do qualitative research, user research, eye-tracking and biometric stuff.

We’ll be hooking people (test participants) up with all the fancy state-of-the-art equipment, measuring their heart rates and if they’re smiling, measuring their facial expressions and all kinds of stuff when they’re using websites. And we’re going to start publishing original research starting from next year because there’s just so little of it out there.

Tell me a bit more about the new research branch of ConversionXL

This is going to be a new and exciting direction for us. I’m predicting it will outgrow the agency side of the business. We’re going to set up a subscription product. One will be a training plan.Right now, our course is a one-time fee of $1400.

In the new product we’ll also make the course available on a subscription basis and just add regular training content, CRO content and analytic stuff in there all the time. And then there’ll be a premium plan for original research.

Subscribers can take our insights from the lab and turn them into test ideas. We’re not recommending whatever we produce as a magic bullet that works for everybody, nothing does. There needs to be more original research out there though.

My prediction is that within the first year, we’ll match the agency’s revenue and then surpass it.

You talked about heart rates and facial expressions, are there any metrics or measurements that you would have liked to have tracked, but you can’t at the moment?

When it comes to biometric research, technology has advanced quite a bit. We still can’t read minds yet though!

Peep Laja

Peep at ConversioXL Live location

So I’ve got a tough question for you. What is the difference between growth hacking and CRO, and CRO with marketing?

Growth hacking is conversion optimization plus traffic acquisition.So conversion rate optimization – Actually, I hate the term rate optimization, so let’s just call it conversion optimization, because conversion rate optimization would imply that we’re optimizing for rate, which is not true. So conversion optimization, we just deal with the traffic that we already have.

In the full growth hacking spectrum, you also need to figure out how to acquire qualified traffic in the cheapest way possible.

Both of these fit under a bigger umbrella of marketing that has everything we need to know about how to get our product in the hands of the users. So marketing is just super, super broad. The actual product that we’re selling is also marketing. Whereas, in conversion optimization, we’re not really dealing with product improvement or figuring out a product market fit or any of that sort of stuff.

That’s a cool way of differentiating between them. So another tough question. What is a good sample size?

Well, this is one of those things where there is actually one right answer. So, you know, in marketing, there is a lot of grey area. But with sample sizes, it’s just math. It’s just statistics, so there is no debate. “I think it’s one hundred.” No, I think it’s two hundred.” Like, none of them matter because there is just one right answer.Just fire up any given sample size calculator and you can calculate it up-front based on your baseline conversion rates.

Let’s say it’s an ecommerce site and there’s a cart page. So usually the conversion rate for people who land on the cart page is, like, 30%. So that’s the baseline.

And then we have to predict an expected uplift and know the minimum uplift that we require. Let’s say 5% relative increase. Based on these numbers, we can calculate what is the required sample size.

Now, there are cases where if the sample size calculators, you know, it’s an exercise in algebra. This does not know what the actual truth is in your specific tests. So it might be that let’s say that you have a test that is 80% winner, and that’s also the lift that you predicted, so it will tell you that you need a very small sample size. So in those cases, when the sample size calculator tells you that you need a hundred or less than a hundred transactions per variation, then in those cases, I would say no. We need the experiment to run longer, like, 250, 350 conversions per variation to be confident.

What the sample size calculators don’t take into consideration is test duration. Let’s say it’s a big site, like Amazon, it would take a minute or two minutes to reach a couple of thousand transactions per variation for an A/B test?

So is the test done? Do we have adequate sample size now? No, we don’t, because it’s not a representative sample.

So then in order to ensure that the sample is not just big enough, but is also representative of our typical traffic, that means that we have to let the test run long enough so that it would involve every single week day, weekend, every traffic source, maybe our blog/newsletter publishing schedule, anything that might impact user behaviour. There might also be external factors, like pay day, phases of the moon, and so on.

What about weather? Do you think weather ever comes in as any kind of factor?

Oh, absolutely. I had a chat with the optimization people at Booking.com. They are noticing a crazy impact the weather has on bookings. And it’s like if it’s raining, if it’s bad weather, people are booking trips to Spain or whatever.

[Editor: programmatic advertising, which is growing in popularity, can be triggered by weather conditions.]

Illustrations of weather-specific activities

Ideally you’d test for at least two business cycles. So a length of a business cycle again depends on the business. It might be a month. It might be a week.

So usually when I start running tests, and let’s say that I have adequate sample size and statistical confidence at a two-week mark, after 14 days, I’ll let the test run for another two weeks, just to see if the result holds steady.

If I do it for multiple tests for the same site and I see that the confident winner at two weeks is still a confident winner at four weeks, then I know that I can start running shorter tests, even experimenting with just one week.

It really depends on how much transaction volume we have because in low traffic sites, if they can test at all – which they might not – As a ballpark, if you have less than one thousand transactions per month, then it’s very likely that you can’t A/B test at all. You’re not ready yet.

What’s your marketing stack? What tools do you use on ConversionXL?

The blog site is very simple, so not much, but here’s what we use:

  • Google Analytics with Google Tag Manager: to measure all kinds of interactions and event tracking.
  • Optimizely: run A/B tests on our own site.
  • Hotjar: for session replays and heat maps.
  • SumoMe: for social sharing.

In addition on the agency site we use:

  • Heap analytics: for event tracking.
  • Visual Website Optimizer (VWO): for testing experiments.

Why have you chosen VWO over Optimizely?

The agency site is a low-traffic site. So with the new Bayesian stats engine that VWO has, it just speeds up our testing process. Sometimes it’s not entirely scientific, but if you’re a low-traffic site, you deal with what you have.If you’re a low-traffic website, you’re taking a business risk each time you end a test. You might declare a test a winner or a loser, even though it’s actually false positive or false negative.

With VWO’s smart stats, they are able to give you a risk assessment earlier than Optimizely. So I don’t want to encourage people to end tests earlier or whatever but for us, VWO’s Bayesian model stats engine requires a lesser sample size to give you an optimization recommendation. We don’t know if variation is better, but we can say with confidence that it’s probably not worse.

Tell me more about your training course

We have a self-study course. Anybody can just sign up and teach themselves CRO, which is a really massive guide. If you print it out, it’s like five thousand pages.

I also run a coaching program twice a year where we use the same CRO courses to textbook, and then we’ll do a live optimization. So we’ll pick a test site, and during the 8-week coaching program, we’ll actually optimize a site week by week in a very structured process.

And so it’s two live sessions a week, so 16 live sessions and ongoing discussions. I’ve done six rounds of coaching now and it’s really, really terrific.

Peep Laja on stage

Peep presenting on conversion optimisation

I like teaching more than I do optimizing. It’s more fulfilling in a way. Of course, optimizing itself is also super-important. Like, you can’t be a teacher if you’re not a practitioner or you’re a fake. So I’m always working on actual sites at all times just to be always up to date and sharpening my own pencil.

The start of a community is all about sharing knowledge. Do you feel like for the guys and girls coming through at the moment, it’s important to keep sharing knowledge?

Oh, absolutely. I mean, true leaders create more leaders. So that’s one thing. And our industry is growing at a rapid pace. What we as optimization professionals can do and should do for our own industry is to make sure that everybody, including our competitors, actually know what they’re doing. Because otherwise what will happen is that the reputation of the whole industry will tank if there are a lot of these wannabes who think they know what they’re doing.

Of course there already are these wannabes who just optimize – Their whole optimization know-how is based on blog posts written by bloggers who never optimized anything in their life. It’s just the sad reality. It’s like so many very well-known blogs are just so full of crap.

It’s like SEO. SEO got a bad name because there were so many people doing it.

Yeah, exactly. High school kids think they are SEO people. As leaders of the SEO industry, we need to make sure that everybody is very well-educated. Make sure there’s some core competency, basic education about statistics and the process of optimization and all that stuff.

Because as you know, the sign of an amateur is thinking in terms of shortcuts, tactics, or copying market leaders and competitors. Anybody who is doing that, clearly does not know what he or she is doing.

When you were starting who was your hero?

In the very beginning, Craig Sullivan was a real help. He’s been the really super key ingredient of my professional development, like the training and mentorship he’s offered.

But it’s also been just a whole bunch of reading. I don’t even remember actually what or who I read when I got going. I think I also discovered that there was actually not that much great content available when I got started in conversion optimization, which is why I was able to create the most popular conversion blog rather quickly because there was just a huge lack of content on this topic.

Can you remember the bit of content you produced for the blog that set you on fire?

Oh, yeah. I have an article on pricing experiments, which has been shared so many times on social media. It got to number one on Hacker News and Tim Ferriss featured it on his Facebook and Twitter account. The traffic he sent took my blog down at a time! The post has forty thousand social shares or something. It’s just crazy.

What was the toughest bit of starting ConversionXL?

There was no true pain, but it’s been all hard work since the beginning. There is no shortcut to success if you’re not ready to bust your ass.

Peep’s reading list:

Peep is hosting ConversionXL Live, a big conference near Austin, Texas, March 30th to April 1st. The ticket includes accommodation and food in a private resort for the weekend, where all the speakers and attendees have a chance to network. Attendees will also be given an incredible bundle of CRO tools, for free.

image representing average form conversion rate uplift from using Formisimo From 0% to 24% Form Conversion Uplift Get the insight that increased form conversions by 24% (and rising)
Latest Insight
Improving Insurance Forms Reviewing the top 31 insurance forms