We've created Zuko, our next-generation form analytics platform. Explore Zuko

9 Optimisation Lessons From Craig Sullivan

A/B Testing

Earlier in the year I interviewed Craig Sullivan, aka Optimise or Die. I know Craig from the conference circuit and many of you will too. If you don’t, here are a few things you need to know:

  • He’s been in the optimisation business for 20 years.
  • He taught anyone worth their salt everything he knows, including ConversionXL’s Peep Laja.
  • You must be humble to be a true optimiser.
  • He loves analogies.
  • He swears a lot (I mean, I edited a lot out!)

Craig and I chatted at length, here are the best bits, the ultimate lessons from his experience as a conversion optimiser.

Craig Sullivan

#1 Being an optimiser comes from a natural obsession to understand what’s going on

Q. What was your first job and how did you get super-passionate about it?

I didn’t really fall into optimising. I guess it was just a natural confluence of several things kind of coming together. My earlier life was as a network engineer. When I went to John Lewis (where I worked between 1990 and 2004), I was doing security work, but also started on usability. And there was a guy who became my mentor. He took me under his wing and he did a bit of a Karate Kid on me. He got me hooked on looking for product defects.

I would go into the user testing and I would try and take someone through a website. And, of course, I’d be hopelessly biased and tell them what to do and things. And he would laugh at it and point out what was wrong and critique my work. And I would keep doing it. And after 100 hours in the lab, I thought we were getting kind of okay at doing the work. But that taught me then how careful I had to be when finding people to do that kind of work because I realised then how skilled it was. It looks easy on the surface, but not easy underneath.

I’m now obsessive about understanding the subconscious layer that we never see when we’re going through our lives. Take door handles. None of us will ever think anything about all the door handles we use over the course of a week: car doors, cupboard doors, dishwasher doors, toilet doors. Just go away for a week and think about all those doors that you use each time you use them. And you’ll suddenly discover, “I don’t think about this stuff, but now I do, I’m kind of noticing that some of these doors are hard and some of them are easy.” And that is the whole hidden, subconscious layer that’s there in web design that we’re tapping into when we do A/B testing.

#2 Share your knowledge, all of it

Giving away stuff for free always seems like a bad thing to do because there are a lot of people making money out of it and selling e-books and doing this, that and the next thing. But I actually get far more in value back from the community by giving away stuff freely than I ever imagined when I started doing it. (Editor: Oli Gardner said the same when we interviewed him).

The benefit of giving away that stuff is that you occasionally get to hear how that benefitted someone. I recently met a guy who told me, “I’ve started my own consultancy and built it up into a sizable little business and it’s all down to you giving me those two books years ago.” I was just walking around in a cloud for a couple of weeks.

#3 Realise your ignorance, you’ll be a better optimiser

Actually, from sharing that information, it tends to expose you to a lot of information that improves the quality of your ignorance.

By that I mean, most people start out not knowing at all about what they don’t know about. And over a period of time, they gradually learn a bit more about what they don’t know about. You progress from low-quality ignorance, which is, “I don’t really know what I know or don’t know,” versus high-quality ignorance, which is, “I’ve mapped out pretty successfully the shit that I know and the shit that I definitely don’t know.” And knowing where that line is, is really important for optimizers.

Good optimizers have a sense of humility, not just about the product they’re improving but about their personal opinions. (Editor: Craig has previously said of hiring an optimiser, “Hire the humble one”). That means actively subjugating themselves in favour of trying to figure out how the great, unwashed public are going to react or do things.

I think if you’ve ever had your product ripped to shreds in usability testing, if you’ve sat in the call centre and listened to people whine about something wrong on your website, then you’re probably feeling bad about your product. But that’s good. You need to be immersed in all the pain and misery as well as the delight in order to be able to understand and appreciate their viewpoint.

#4 CRO needs to be wielded by the right people

Q. How tough is it getting clients to really buy into the A/B testing CRO process? And is it easier now versus let’s say three or five years ago?

No, it’s equally as hard. I’ll make an analogy; if you were sitting in Ferrari’s technical area at the next Grand Prix, you wouldn’t see the race director of Ferrari down there with his hands in the engine and trying to figure out what all the engine schematics meant. But a web page has got this visual layer that belies the technical stuff and so everyone has an opinion on it, everybody wants to meddle.

It’s just as difficult to convince people these days as it was a few years. There’s so much ego and opinion and cherished notions in product design, they don’t actually go and get customer feedback. There are a lot of companies who have CRO tools gathering insights but they’re only ticking a box. There is a big difference between having form analytics and having a structured program to mine that form analytics data to then fix it in terms of conversion.

Q. Are we close to peak adoption of CRO?

I still think we’ve got a long way to go because the statistics are something that certainly in early iterations of the tools have been made quite hard for the layperson. So these guys have done a great job of democratising these tools. That’s great. But it’s a bit like you give out light sabres to allure teenage boys. You just know that people’s arms are going to get hacked off and they’re going to end up in A&E. So it obviously depends on the person holding the lightsabre. So if you did that, you would have a lot of stupid things happening. The same thing has happened to the A/B testing. It’s like, “Oh, yeah, just run this and you’ll know what to do.”

So, CRO has matured, and, yes, there are more people doing it, but everybody is on this hype cycle of A/B testing, which is a bit like the Gartner hype cycle. And you get a load of enthusiasm. People run off and do a load of testing. And then you get this kind of peak of stupid testing. And then people discover that they may not be testing in the right place.

I think what people like Peep (Laja) and myself and others are doing are trying to help people jump that gap and say, “Look, don’t end up in that trough where you’re questioning whether this stuff works or not.” Do it properly and then you’re actually in with a chance of making this transform your business or it being an effective agent of change in the way that people build products within your company.

#5 The dance of the P values: what you need to understand about statistical significance

Q. What are the major A/B test mistakes that you’ve made?

The big mistake I was making early on was not understanding that I needed to look at the confidence value in a test and plot it over time. When you do that you’ll see that it moves all over the shop. It’s a thing called the dance of the P values.

But it’s the thing that underpins what that value is, that thing moves around all the time. So 20 minutes into a test, it could come and say, “Oh, it’s hit 95% confidence. But that doesn’t mean that’s a representative sample of people across your purchase cycles. How can 22 minutes worth of traffic be analogous to a whole week of people coming to your site?

The weekend people are different. The people in the evenings are different. So you not only need to collect enough data, you also need to make sure that that data is actually representative of the audience. So it would be like taking a teaspoon of water out of a river and saying, “I will use this teaspoon to describe the entire yearly life cycle of this river,” which would just be bullshit because all you’re going to come up with is an approximation.

The first mistake I was making was not really having some sensible rules set-up for how long I was going to run a test, I should have been using a test calculator. You just give it the data and it will tell you how much time need to run the test. At the end of it, you stop the test and you analyse the data.

Remember, always do it on whole weeks. You don’t have five Saturdays in your sample, but only four Fridays otherwise you’re going to have biases in there. But that’s the mistake I was making, not having some sensible rules and understanding about testing.

Any decent analyst is always looking at A/B testing and trying to figure out all the stupid ways that they could bias the test or the data that they’re analysing.

#6 Measure everything (and know when to stop)

Loads of redesign projects fail because they’re not actually based around a thorough and detailed understanding of exactly what to throw away and exactly what to keep.

If you don’t understand the problem, you can’t find a solution. If you start doing a redesign without knowing what worked well in the old one and what didn’t work so well, what people liked and didn’t like, and what they find friction-free and fiction-filled, they make some stuff better that was worse before and they make some stuff worse that was actually okay before.

So what would normally happen is that some of the segment level performance might cancel out. So some users do better in the new home page and some users do worse. But since you never looked, you don’t know who was impacted.

The other thing is it could be that your change just wasn’t bold enough or it didn’t act on the problem. So the reason that people didn’t like the old home page is it didn’t have enough product on it or it didn’t have a search box or it didn’t have clear navigation.

A big mistake I made in my early days was hoping that the test would get better even when it wasn’t going my way. So this thing of, “Well, if I let it run a little bit longer, maybe it will change around or something.” And it never does. I’ve waited months for things to go my way. And the only time it’s happened is when I’ve waited so long that customer traffic coming into the site has changed. That’s no way to run a business.

#7 To learn the right stuff learn who the real practitioners are

Q. There’s a lot of advice out there, sometimes conflicting advice. How does someone new know where to start and what to trust?

There’s a load of bullshitters out there in the CRO industry. I would say there are two classes of people who talk about optimisation:

  1. There are people who do not do the work, but talk about it. They are trick ponies of the CRO and analytics world. They are people who are abstracted from the work. They may be interested in it. They may form opinions. They may create frameworks or digital guidelines or their top ten tips. But they don’t really fucking know when it comes down to it because their knowledge is superficial.
  2. And there are the practitioners, the people who are actually doing the work in a hands-on way. Follow them, since they will often be at the cutting-edge of tools and techniques that are rapid, that will make you more productive, that will augment your abilities as a CRO person because they’re doing it every day. So it’s not a theoretical or philosophical exercise, it’s actually something that you have to do in rapid, time-limited periods for clients. And I think that makes them a different class.

I highly recommend the ConversionXL course. There’s a lot of great and wide-ranging stuff in there. It’s a great way to get started in this.

And by the way, a UX-only approach is like a one-legged bar stool. You need UX analytics and split testing, then you’ve got enough solidity to run a proper product or optimisation program. I admire Luke W for blending those two things together. You should watch his presentation Mobile Design Essentials. These videos are pure gold. This is the best stuff I’ve heard on mobile for about 5 years.

#8 Don’t forget to optimise your web photos

There’s a whole arm of split testing that’s interesting around product images. There are different types of uses of imagery. If I want a product image of a jacket or a suit or something, I don’t really want to see the person’s face. I want to see the clothing item, right? But if I go to Autoglass because of my smashed windscreen, what I need from that image is a feeling of emotion, which is around solving a distress purchase. So we know that people have a number of fears.

So let’s say we do our UX research on the type of customers that would visit this page, like the Autoglass example. We know that people are worried about their insurance and claiming on that, so we need to sort that one out. They’ve also got other worries e.g. they’re away from home or they feel vulnerable as a female driver. So, the persuasive angle for that page is to reassure them that we’re going to sort it all out, we’re going to get someone there, and it’s all going to be dealt with at no fuss.

Understanding that emotional landscape helps craft some images to address it. In the Autoglass example, we discovered that using a strong female image in our western European markets always beat male images. And the images that we thought would work, the hunky guys carrying big pieces of glass around with rippling biceps, didn’t work because it was a mismatch with the emotional landscape of the customer when they arrived at that page.

What you need to work out is what kind of image can I use that will cause the emotional shift in the mind of the viewer that I understand is needed from doing my research?

Get the book, Usability of Web Photos (photoux.co.uk). It’s written by a friend of mine, James [Chudley]. It’s the only book that talks about the psychology of imagery for optimisation.

#9 Fixing bugs in the site could release huge revenue

What a lot of businesses fail to miss is the downside of not addressing website performance issues and defects in your product. Well, the answer is it could often be more than your entire IT budget is for the year.

I’ve got some real world examples where the bugs that we found in websites and fixed, the amount of money gained was considerably larger than their entire IT budget for everything for the year.

Now, if you’re sitting on a website and it takes $40 million a year, you’re probably thinking about how much it costs you to run. What you’re not probably thinking about is all the money you could be getting if you only went looking for it. So it could be earning $60 million a year if you were actually running it properly and got rid of the defects.

When I sit in a meeting and I’m showing people these bugs and they say things like, “Oh, so that’s what that looks like on mobile?” It just tells me they never looked.

Optimisation won’t be optional if you expect to compete

[Editor: I chose this as a great concluding point by Craig]

If people realise in the future that these competitive margins are so small, they’d realise the difference between surviving online and going out of business, albeit slowly like a zombie as not being able to compete with people. If your competitors A/B test and optimise, then they will be able to beat you and outspend you and outstrip your growth every day of the week. That’s what you have to guard against.