In-Depth Guide to A/B Split Testing Like a Pro

In-Depth Guide to A/B Split

Are your ads not performing well? Did you know that there are different strategies to see which can deliver the best results? What you can do to find out which ad can produce the best results is through conducting an A/B test. If you’re not familiar with it yet, or still struggling with conversion rate optimization, then you need to take a look at our A/B testing tutorial guide.

What is A/B Testing?

A/B testing or split testing is a method used by data-driven digital marketing agencies, like Propelrr, to test various versions of an ad, app, email, or website on similar/different conditions. The A/B testing statistics gathered will give marketers a clear picture of which version performs better over the other and use for future campaigns.

Where is A/B Testing most used?

This practice is used frequently to optimize marketing campaigns and ads. A good A/B testing analysis gives marketers the ability to find out words, phrases, images, videos, and other creative output that can produce the best results.

Why is it important?

Even with a good-looking website, and an attention-grabbing video/image, you can never be certain if that’s the best way to go to produce optimal results. Well, not without data-backed research. You can use multiple versions of an ad, video, image, or post to test various key performance indicators (KPI) that show the success level of your choices.

Benefits of A/B Testing

Split testing has a lot of benefits for businesses, websites, and marketers. Here are the biggest benefits that you can get out of A/B testing in marketing.

  • Easier analysis
    With this method, it’s easier for marketers to look at the KPI and compare it with each version of the creative. You can then easily determine which version performed better than the other and apply the variables that made it a “winner” to future campaigns.

  • Improved content
    The data you get from the A/B testing process can help marketers improve the content they produce. This testing method will weed out the things that don’t work for your creative so you can use the items that do to improve your content further.

  • Better conversion rates
    As mentioned, A/B testing methodologies can help you spot what works and what doesn’t. Applying this technique tends to deliver an improvement in conversion rates for your product or service.

How to Conduct an A/B Test?

Have we made you excited to try A/B testing? Don’t jump the gun just yet. You need to learn how to do it correctly. You’re in luck because this guide will help you do just that.

Here is your guide on how to conduct A/B testing:

  1. Know your objectives
  2. Know what to test
    1. Ad or Ad Copy
    2. Format
    3. Placement
    4. Audiences
    5. Landing Page
  3. Create a hypothesis
  4. Calculate sample size and duration
  5. Run a campaign for A/B Testing
  6. Analyze the results
  7. Declare the winner
  8. Optimize your ad and do it over again

Keep on reading to have an in-depth understanding of A/B testing for your ads online.

1. Know your objectives

What is the objective of your test? Do you want to know which creative works best for brand awareness, consideration, or acquisition? Even if you run multiple A/B testing, you need to understand why you are doing it in the first place. Without an objective to follow, your test will be useless. Decide if your objective is to raise brand awareness, consideration, or acquisition.

Once you know the objective, know your KPI. Find out which metric is needed to measure the success of your campaign. It can be impressions, clicks, click-through rate, conversion, reach, post engagement, etc.

2. Know what to test

A common mistake of marketers that are new to split testing is that they use different elements on each specimen with only one thing in common. Sadly, doing this will make it difficult to monitor your data to find out what works best.

It’s highly suggested that there should only be one element different in your test so it will be easier to find problems or strong points in your ad or creative—especially when running A/B testing on mobile apps. The following are the common things that are being tested:

A. Ad or Ad Copy

Thinking of running an ad on your social media accounts or other forms of media? Let’s break down how your A/B testing strategy should go.

  1. Ad Elements for Google Search Network

    ad elements for google search network

    Whenever you do a search query on Google, the results show headlines, descriptions, and other elements leading to a website or any other online resource. The same goes for Google’s search network ad, which shows on top or near the top of the first page of a search result.

    Testing your Google Ad Words copy will increase your conversions or site visits. That’s why A/B testing in Google analytics or its Search Network is crucial if you want more website visits, fewer bounce rates, etc.

    1. Headline 1
    2. Headline 2
    3. Headline 3
    4. Description 1
    5. Description 2
    6. Display URL (path)
    7. Call-to-action (CTA)


  2. Ad Elements for Google Display Network (Image Ad)

    ad elements for google display network.jpg

    Aside from ads in the search results, there are other places Google displays its ads such as text and banners spread across Gmail, websites, and apps.

    1. Headline
    2. Sub-headline
    3. Copies
    4. Value (i.e. 50 percent discount)
    5. Subject
    6. Background
    7. Common elements
    8. CTA


  3. Ad Elements for Facebook

    ad elements for facebook.jpg

    As of the second quarter of 2019, Facebook has a billion active monthly users worldwide. That’s why marketers are using this social media platform more often to show ads. Here are the elements you can play with when A/B testing on Facebook. Just take note of the guidelines for A/B testing Facebook ads—especially when it comes to images.

    1. Text description
    2. Image
    3. Headline
    4. Sub-headline
    5. Copies
    6. Value (i.e., 50 percent discount)
    7. Subject
    8. Background
    9. Common elements
    10. CTA


B. Format

In what form should your ad be? Find out the best way to deliver it to your audience through A/B split testing. Take a look at the formats that you can use.

  1. Video - Video ads are a natural storytelling medium displayed during or after a video stream or as a separate/independent post. It’s now considered a significant player in the world of advertising because of its ability to increase ad recall, purchase intent, and click-through-rate (CTR). How? Check out this Instagram case study.
  2. Static Image – In essence, it’s a graphic that’s used to promote a business or service. It may consist of elements such as business information, services, or products. When clicked, it usually takes users to a website.
  3. Carousel – This is a type of ad that showcases several images or video in a single ad to increase the chances of conversion or sale. It’s most popular in social media platforms such as Facebook or Instagram.

C. Placement

Where do you want your audience to see your ads? Placement is the location where an ad is shown. It can be as broad as an entire website or as specific as a certain area only.

  1. Location – It’s a select location on a page wherein ads appear.
  2. Day Parting – If you think that your ads will work better at certain times of the day, then you need to try Day Parting.
  3. Facebook Feed – These are ads that show up either on Facebook’s newsfeed, itself, or surrounding placements.
  4. Messenger – With a projected figure of 138.1-million users in 2022, ads are making its way to the messenger app. Ads are placed in the contact list to provide optimal visibility.
  5. Instagram – Once you’ve converted your personal Instagram account to a business profile, you’ll be able to run ads, as well as promote posts and stories.

D. Audiences

Who sees your ad? It’s crucial in any digital marketing strategy to know who your audience is and how to target them. Here are some methods for you to target your audience properly.

  1. Interest targeting – You select interest categories related to your business/product/service for your campaign. It then targets users whose interests broadly align with your business.
  2. Narrow Audiences – Most marketers would want to reach as many people as possible. But with online ads, quantity is better than quality. That’s why narrow audiences help marketers choose an audience who would most likely be interested in your offer.
  3. Lookalike Audiences – This is a way to expand your audience by reaching new people who are similar to your existing customers.

E. Landing page

ab test landing page

A landing page is a single web page with a purpose of capturing leads or to “warm-up” customers to a product you are trying to sell to them. Because it’s such an essential part of a website, it is highly recommended to do A/B testing on your landing pages.

To determine what part of the page to optimize first, you can use one of the A/B testing tools called heat mapping. It shows the location that has the most clicks on your webpage.

  1. Copy – The success of your landing page often depends on the written copy. If your copy fails to impress, nobody would want to buy your product no matter how well-designed the page is.
  2. Position of form – With only 50 milliseconds to make a good first impression, the position of your form is crucial. That’s why it’s important to run multiple landing page A/B testing for this.
  3. Button colors – There are plenty of ways and things to consider when choosing button colors. Most of this will depend on who your audience is and what you want them to feel when they see the button.
  4. Button placement – Another thing that needs landing page testing is the placement of the buttons. Moreover, it should have a strong CTA.
  5. Creatives – Should you use graphics, a photograph, or a video? Deciding the right creative for a landing page is equally important to the other things on this list since this will help capture the attention of your customer.
  6. Wireframe – Before publishing your landing page, a wireframe will give you a good picture of how the overall look will be. With this, you can decide which elements are needed and what else you need to add.
  7. Banner – Some marketers would choose to use a banner for the landing page. It has all the right elements needed to capture the attention of a customer, as well as a simple form to get the information needed.

3. Create a Hypothesis

In line with the objective of your test, you need to identify potential problems or questions that need to be answered by your test, also known as a hypothesis. It’s like predictive analysis, wherein you use the previous data to foretell the result of the test.

This isn’t done randomly. It requires looking at statistical data from multivariate testing such as web analytics, user tests, heat maps, customer feedback, and heuristic evaluation or ergonomic audit. Review performance data from the ads to gain insights and understand what has been achieved and what needs to be improved.

In addition, you can develop a hypothesis by identifying key campaign and/or elements that have a direct impact on the campaign performance. Make sure to prioritize hypothesis in terms of expected impact and difficulty of implementation.

Take a look at this simple yet effective syntax, so you have a clear idea how to come up with a sound hypothesis.

Changing an element from A to B will increase/decrease a KPI

But don’t mistake this as an all-in-one winning formula. You still need to take a good look at your data and KPIs, as well as your campaign’s objective to create a hypothesis that will help you succeed.

4. Calculate sample size and duration

To bein with, set a minimum sample size. According to the A/B testing case study by Adalysis, the minimum data recommendations for most companies are:

  1. Low traffic: 350 impressions or 300 clicks or 7 conversions.
  2. Mid traffic: 750 impressions or 500 clicks or 13 conversions.
  3. High traffic: 1,000 impressions or 1,000 clicks or 7 conversions.

A/B testing examples for advance analysis can be done through a ‘Paired T-Test’ or ‘Independent Sample T-Test’ or ‘Wilcoxon- Signed Rank Test’ or ‘Mann- Whitney U Test.’ Using these methods will help you sort through data to find out if there are any significant differences or not.

If there are significant differences, then you may use the data for comparison. If there is no disparity, you’ll need to set more data points. If the results still don’t produce significant differences, it means that both ads will produce the same result.

4. Run a campaign for A/B Testing

With most of the variables in play, it’s time for you to create a campaign for your A/B testing. Make sure that your ads for testing have the same settings selected such as budget and maximum cost-per-click (CPC) so it would be easy to know which ad is performing better in the same environment. In addition, you need to turn-off ad rotation for Google Ads so that your ads get the same “air time.” Using different settings will render different results, which is basically not part of the A/B testing best practices.

6. Analyze the result

Analyze ab test result

With the data on hand, you can now compare the results between your control and challenger ads. Of course, there are plenty of KPIs to compare the results with such as conversion, velocity, win rate, and projected versus actual spend. If your objective is ‘Increase Revenue’ and your KPI is conversion, then you must only compare conversions of the two ads. Other data is not needed.

7. Declare the winner

Based on the results and your objective, it should be easy to declare the winner among the ads that you’ve run through all those A/B testing tools like the heat map and click map. It can be based on the highest total conversions, the highest overall CTR, or the lowest cost per acquisition/result. It really depends on your KPI. If both ads have the same result, it means that you will have the same result with either ad. Thus, the control ad is the winner by default. The data you have from the winning ad can be used for website optimization, as well as email or social media campaigns.

8. Optimize your ad and do it over again

Just because you know which ad performed better means that’s the end of it. There’s always a way to optimize your ad further. After all, you’ve only tested one variable. Do it all over again then change another element and run another A/B test.

Key Takeaways

The A/B testing framework involves a lot of variables and patience to do it right and yield the desired results. Let’s take a look at the key takeaways you can get from this guide to A/B testing.

  1. Knowing your objectives. Having a clear objective is important to be successful in A/B testing. Find out if you want to raise brand awareness, consideration, or acquisition, among others.
  2. Testing the right variables or element. You need to know what to test based on your objectives. Make sure that you test one element, while the rest are practically similar or the same to produce the optimal results.
  3. Patience is key. The data you need can come quickly or after a while. Be patient.
  4. Use the data wisely. Compare the data and use it to improve your ad and campaigns. It can even be used to optimize your website and email campaigns.
  5. Repeat. There are more elements of an ad that can be tested to produce the best ad. Keep changing things until you find which elements will work for your audience.

A/B testing is a long process of trial and error. Use this guide to help you throughout your journey in searching for the right elements for your ad. It may take some time, but the rewards are immeasurable for those who wait.

Have anything else to add? Feel free to send us a message on Facebook, Twitter, or LinkedIn, and we’ll take care of things for you.


Notify of
Inline Feedbacks
View all comments

Never miss a beat.
Get regular updates in your inbox
on what's new on digital.