The #1 Problem With A/B Testing PPC Landing Pages & How to Fix It
The #1 Problem With A/B Testing PPC Landing Pages & How to Fix It

The #1 Problem With A/B Testing PPC Landing Pages & How to Fix It

“Let’s test it!”

If you work in marketing, you’ve heard this. A lot.

It’s a good idea. You can test things. You can make data-led decisions. That’s smart.

Many times, the landing page conversation and workflow will go something like this:

  • Create laser-focused landing pages for each of your priority PPC keywords
  • A/B test two (or more) variations of each landing page to figure out which is the top performer
  • As soon as you reach 95% statistical significance according to the landing page tool, promote one page to the champion and archive the lower performing variant
  • Create a new challenger landing page and re-start

Sounds good. But there’s one really important thing missing from the above, that many of the landing page tools don’t tell you… 

And that one thing makes a huuuuge difference to your ability to make decisions about which page to keep and which to kill…

A/B Testing Requires a Ton of Sample Size to be Accurate​​​​​​​

You need a lot of traffic to draw a reliable conclusion about confidence.

95% confidence at low sample sizes isn’t really 95% confidence. Not at all. The problem is a lack of statistical power.

“In plain English, statistical power is the likelihood that a study will detect an effect when there is an effect there to be detected.” – EffectSizeFAQs

Here’s an AB test size calculator by Online Dialogue. You might like to have a quick play with it to see what sample size is required for your next test.

Chances are, the calculator will show that you need a sample size in the tens of thousands to reach a level of confidence and minimise errors of Type I or Type II variety.

type 1 and type 2 errors

But PPC landing pages don’t get tens of thousands of visitors!

So can you trust the 95% confidence level so quickly output by the tools?

No. Your risk of a Type I or Type II error is too high.

You can note the confidence level, sure, but best not to rely on it. Without an adequate sample size, the tool may mislead.

Plus, there’s an opportunity here for something else that’s going to give you a deeper level of feedback about which page is performing the best.


Because you have a smaller number of visitors to your PPC landing pages, you can get way more granular in your measurement than is possible with a larger scale project. For example, for all visitors to your PPC landing pages, you can:

Take session recordings via tools like HotJar. Watch each user from the moment they land on your site, tracking their scrolling, mouse clicks, where they hover, where they hesitate, how they navigate forms, where they exit and anything else you would like to see.


Recruit user testers via tools like User Testing. “Behind every data point is the why”, and having a person in your target audience such as a current customer speak aloud as they move through your site is revealing.

If you’re recruiting from the pool of people at User Testing, you might like to ask for evalutations of your competitors’ websites too.

user testing

Get personal with Live Chat and bots. Intercom will give your website an unobtrusive chat box with loads of functionality beyond what you’d normally expect, such as the ability to make special offers to people based on their browsing behaviour. This can create a sales conversation while revealing insights into barriers that are holding your prospects back.

intercom example

Track engagement in detail with these 4 Google Analytics hacks:

Adding the above into your analysis, rather than just relying on 95% confidence and calling it, will give you a fuller picture about the real impact of your A/B tests and winning variants.

Conversion rate aside, you will also discover improvements to the user experience of your site, which is a win for the brand.

How we can help you

We can help you drive digital marketing performance by using the world’s most powerful digital marketing channels to full effect

How we can help you

Get a proposal

Would you like to see how Reef can improve your digital marketing campaigns? We welcome the opportunity to share our best ideas with you

Get a proposal