A/B Testing: Why, What, Where and How to do?

This article will provide an introduction to A/B testing — Why companies should do A/B tests, What exactly it is, Where we can run the tests and What can we A/B Test.

Content

1. Why Companies Should Be Testing?
2. What is A/B testing?
3. Where can we run A/B tests?
4. What can we A/B test?
5. How to get started?
6. Examples

Why Companies Should be Testing?

There are multiple benefits of A/B testing:

1. Increase Revenue or Conversions — Bing by Microsoft increased their revenue by 12% (100 million annually) by a simple A/B test which was earlier deprioritized by the Product Manager but was implemented by a Software Tester later. Thus, these tests impact the bottom line of business and increase the revenue.

2. Rapid Iteration — You can implement a new experience and whether it wins or loses, you can do it quickly.

3. Learn What Works — You win quickly and get gains to your business or you lose quickly and learn what doesn’t work for your audience

4. Uses Actual Site Visitors — Since the actual visitors of the site are being used to get the results, it makes sure that the data sense is not just random but tells actually what worked for a digital visitor and what didn’t.

5. Data Driven Approach — You’re letting your data and actual audience guide your decisions — investment, production and the things which put out to the world.

What is A/B testing?

Different Names — Same Meaning

1. Split Testing
2. Conversion Rate Optimization
3. Multi-Variate Testing
4. Landing Page Optimization
5. Digital Optimization
6. Online Experimentation
7. Growth Hacking

All of the above are different ways of saying that you’re running tests and you’re learning about your visitors with actual data on your actual site.

An AB test is an example of statistical hypothesis testing, a process whereby a hypothesis is made about the relationship between two data sets and those data sets are then compared against each other to determine if there is a statistically significant relationship or not.

AB Testing refers to a randomized experimentation process wherein two or more versions of a variable (web page, page element, etc.) are shown to different segments of website visitors at the same time to determine which version leaves the maximum impact and drive business metrics.

Essentially, A/B testing eliminates all the guesswork out of optimization and enables experience optimizers to make data-backed decisions. In A/B testing, A refers to ‘control’ or the original testing variable while B refers to ‘variation’ or a new version of the original testing variable.

The version that moves your business metric(s) in the positive direction is known as the ‘winner.’ Implementing the changes of this winning variation on your tested page(s) / element(s) can help optimize your website and increase business ROI.

The metrics for conversion are unique to each website. For instance, in the case of eCommerce, it may be the sale of the products. Meanwhile, for B2B, it may be the generation of qualified leads.

Where can we run A/B tests?

AB tests can be run on:

1. All Website experiences → Homepage, Landing Page, Site Content/Product Page, Conversion Flows, Signups, Checkouts
2. Paid Advertising → If you may be running paid ads to get more people to your site then that’s a great opportunity to run A/B tests to see if you have the Right Headlines, Right Titles, Right Image/Thumbnail
3. Mobile Apps → You can test User onboarding, User retention, User engagement rate, and for apps that feature transactions, every step along the sales funnel to maximize conversions. Examples of things to test include the number of forms required to sign-up for an account, the frequency of push notifications, the text used in calls-to-action (CTA), new features, and ad placement within your user interface.
4. Marketing Campaigns → As a company puts out marketing campaign, they can test how well does the campaign do relative to other campaigns.
5. Emails → Every time you send out an email and every visitor touch point like email, app or a website or an digital experience where they’re engaging with you is a great opportunity for AB testing.

What can we A/B test?

1. Site pages, Flow and Elements → It includes testing of pages and flow, pricing and headlines, videos and testimonials, social proof, etc. We can also test how much content to put and kind of elements visitors see in each experience.

1. Business Model → For example: What if the company provides free shipping and offer that through A/B test and then they can see how much more revenue did they make from free shipping and what was the trade off.
2. Backend Functionality and Algorithms → For Example: Flipkart may test when can they recommend different types of product or when can they target a person or genre with a particular kind of product.
3. New Products or Services → A company can test if they’re adding new products or services to the market, how will the products/services perform if introduced as part of an AB test. Thus, they can see the value or impact of the product/service and thus can evaluate the success in the short and long term about that product/service.

How to get Started?

1. Get a Testing tool → There are lots of tools out there. Depending on the size of business, budget, size of team, etc select a testing tool to get started.

2. Define Primary Success Metric → In order to do a good test, one must know what success metrics is being influenced. For example if one is trying to increase revenue per visitor as they are selling products then that’s the main success. Another example if one want to increase the conversion rates so you can get more people to apply for jobs on your portal then that’s the main success. Thus, one success should be chosen to influence so as to have a clear answer to each AB test.

3. Define Idea/Question to be tested → Decide what you want to test. Start with a high level question and then zoom in to get the results from there. Start simple, Don’t go crazy big!

4. Define Variations and Learning → Create the variations based on the question. ideally you’re creating your variations strategically first so you should know like this is my Problem statement and these are 3 or 4 variations that would provide solution to my Problem Statement. Thus, be very clear about the variation as to what will will you learn if let’s say variation 1 wins or when variation 2 wins. This should be known before the test is started.

5. Create the Variations → Involve the relevant teams/people to actually develop/code the variations like Creative Team, Frontend Developer, UI/UX team etc.

6. Measure and Activate → When the variation is ready to go live, we need to have some measurements in place like How does this test influence main success measure and make sure that it is being tracked by measurements so that our test result is clear and easy to learn.

Joke of the Day: 2 Software Testers went into a diner and ordered 2 drinks. Then they produced sandwiches from their tiffins and started to eat. The owner became quite furious and marched over there and shouted “You cannot eat your own sandwiches in here!!”

The testers looked at each other, shrugged their shoulders and exchanged sandwiches😂😏

7. Analyze the results → Once the test is complete, we need to see how each Variant did relative to others to see which one had the biggest impact.

8. Document Learning and Evangelize → Once analysis of results is done, document what is the learning and what were the recommended actions.

9. Ideation → After we have some learnings and recommendations from our previous tests, Now we can create some newer ideas from our previous learnings.

10. REPEAT → Repeat the steps again from 1 to 9.

Examples of A/B Testing

• Humana’s Banner Test: Humana tested two different banners on their homepage. A simpler design plus a stronger call-to-action (CTA) led to 433% more click-throughs. In the control variation, the CTA is not clear and in the test variation the CTA is clear.
• Hubspot’s Lead Conversion: Using an inline form for CTA led to 71% better conversion compared to a linking to a form on a separate page.
• MSN Home Page Search Box: The Overall Evaluation Criterion was the click-through rate for the search box and the popular searches links. Taller search box was compared with bigger search button. However, no remarkable differences were found.
• Electronic Arts: For their SimCity 5 game, they tested conversions between direct promotional banner and no banner. Surprisingly, the latter drove 43.4% more purchases.
• Coderwall: A simple title change improved Coderwall’s click-through rate (CTR) by 14.8% on Google Search. Over time, this resulted in improved ranking as well.