“Let’s test it!”
If you work in marketing, you’ve heard this. A lot.
It’s a good idea. You can test things. You can make data-led decisions. That’s smart.
Many times, the landing page conversation and workflow will go something like this:
- Create laser-focused landing pages for each of your priority PPC keywords
- A/B test two (or more) variations of each landing page to figure out which is the top performer
- As soon as you reach 95% statistical significance according to the landing page tool, promote one page to the champion and archive the lower performing variant
- Create a new challenger landing page and re-start
Sounds good. But there’s one really important thing missing from the above, that many of the landing page tools don’t tell you…
And that one thing makes a huuuuge difference to your ability to make decisions about which page to keep and which to kill…
A/B Testing Requires a Ton of Sample Size to be Accurate
You need a lot of traffic to draw a reliable conclusion about confidence.
95% confidence at low sample sizes isn’t really 95% confidence. Not at all. The problem is a lack of statistical power.
“In plain English, statistical power is the likelihood that a study will detect an effect when there is an effect there to be detected.” – EffectSizeFAQs
Here’s an AB test size calculator by Online Dialogue. You might like to have a quick play with it to see what sample size is required for your next test.
Chances are, the calculator will show that you need a sample size in the tens of thousands to reach a level of confidence and minimise errors of Type I or Type II variety.
But PPC landing pages don’t get tens of thousands of visitors!
So can you trust the 95% confidence level so quickly output by the tools?
No. Your risk of a Type I or Type II error is too high.
You can note the confidence level, sure, but best not to rely on it. Without an adequate sample size, the tool may mislead.
Plus, there’s an opportunity here for something else that’s going to give you a deeper level of feedback about which page is performing the best.
Granularity.
Because you have a smaller number of visitors to your PPC landing pages, you can get way more granular in your measurement than is possible with a larger scale project. For example, for all visitors to your PPC landing pages, you can:
Take session recordings via tools like HotJar. Watch each user from the moment they land on your site, tracking their scrolling, mouse clicks, where they hover, where they hesitate, how they navigate forms, where they exit and anything else you would like to see.
Recruit user testers via tools like User Testing. “Behind every data point is the why”, and having a person in your target audience such as a current customer speak aloud as they move through your site is revealing.
If you’re recruiting from the pool of people at User Testing, you might like to ask for evalutations of your competitors’ websites too.
Get personal with Live Chat and bots. Intercom will give your website an unobtrusive chat box with loads of functionality beyond what you’d normally expect, such as the ability to make special offers to people based on their browsing behaviour. This can create a sales conversation while revealing insights into barriers that are holding your prospects back.
Track engagement in detail with these 4 Google Analytics hacks:
- How far down each of your posts people are scrolling
- Which links in your posts get the most clicks
- How long people are really spending on your content (and why the metric ‘time on page’ is wrong by default)
- What content each specific user viewed before converting, in order
Adding the above into your analysis, rather than just relying on 95% confidence and calling it, will give you a fuller picture about the real impact of your A/B tests and winning variants.
Conversion rate aside, you will also discover improvements to the user experience of your site, which is a win for the brand.