A/B Testing Basics (and 4 Things You Should Never Do)

Renee Witko

In simple terms, an A/B test compares two versions of a web page to see which one performs better. (Let’s call them version A and version B.) The tester shows the experiment to users at random. The tester then uses statistical analysis to determine which version performed better.

Why You Should A/B Test

ab testing.jpg

Odds are you want your website visitors to convert from visitors to leads, then to customers.

A/B testing allows you to make the most out of your existing website traffic by minimizing guessing when it comes to your optimization efforts. Instead of guessing, you can make data-driven decisions to convert your website visitors into leads more effectively.


So, what can you test? The short answer is: Anything on your website that affects your visitors' behaviors, including:

  • Headlines
  • Sub-headlines
  • Paragraph copy
  • Testimonials
  • Call-to-actions
  • Images
  • Content near the fold

But, you can’t just test things to test them. You have to make sure the things you’re testing tie back to your goals and revenue. You also have to make sure your tests are statistically significant.  Here are 4 things to avoid when A/B testing:

4 Things You Should Never Do While A/B Testing

1. Pull the Plug Early

Running a test too long or not long enough can have consequences, including limited information or a sample size too small to actually learn anything.

To get a big enough sample size to see an impact, you need to run a website test for at least a week.

2. Test Too Many Small Things

Testing small elements, like buttons and forms, won’t produce the impact you need or, more importantly, result in any learning opportunities.

There are an unlimited amount of variables you can test on any of your web pages. It’s crucial you develop a hypothesis before you start testing. Before testing anything, ask yourself:

  • Why do I want to test this variable?
  • What is the expected outcome from testing this variable?

3. Ignore Positive Results

This one probably speaks for itself. If you have a test that’s statistically significant (one version performs better than the other), you need to implement your changes.

Don’t be the person who runs a test then does nothing with the results. It’ll just be a waste of time and potential revenue.

4. Not Test at All

Every test you run isn’t going to be a slam dunk, but every test will teach you something -- including what works and what doesn’t work. So fail ... then fail some more. It’ll only make you better at A/B testing.

I’ll leave you with this FunnelEnvy quote: “The truth is, nothing else offers the ROI potential of a good conversion rate optimization campaign.”

Happy testing! 

business growth 8 steps to improve marketing performance


More Interesting Reads

Inbound 2017: Affordable SEO Tips (Heck, These Are Free)

The way we navigate the internet evolves constantly. More importantly, search engines present the information we seek according to awesome technology that tailors the experience to us. Search engines...Read more

The Importance of User Intent for Search Engine Optimization

As a business or organization you understand the importance of generating traffic from search engines (hopefully). To get that traffic you obviously have to show up in search engines when your...Read more

Manufacturer Marketing ROI: 11+ Statistics Your Boss Will Love

The manufacturing industry is slowly coming to terms with the influence of the Internet. However, online marketing is throwing many small manufacturers for a loop. Some companies are...Read more