Plan & Measure›Pro· 30 min read

A/B Testing: Let Data Decide

Stop guessing which ad or button is better — show two versions, measure both, and keep the winner.

What you will learn

Explain what an A/B test is
Read an A/B test result and pick the winner
Know what to test and what makes a fair test

Opinions vs evidence

Should the button say “Buy now” or “Get yours”? People argue about this for hours. A better way is to let real customers decide. That is what an A/B test does.

In an A/B test, you make two versions of one thing — call them A and B — show each to half your audience, and see which gets better results. The winner is decided by data, not by the loudest opinion in the room.

A worked A/B test

A bakery tests two CTA buttons on its order page. Each version is shown to roughly the same number of visitors for one week.

Version	Button text	Visitors	Orders	Conversion rate
A	Order now	1,000	20	2.0%
B	Send me cake	1,000	35	3.5%

Version B wins clearly. Let us check the maths so you can read any test like this.

Comparing two button versions by conversion rate to pick a winner

Conversion rate = orders / visitors  x  100

Version A:   20 / 1000  x 100  =  2.0%
Version B:   35 / 1000  x 100  =  3.5%

Winner: B  (3.5% beats 2.0%)  ->  75% more orders for the same traffic

Note: Version B turned 3.5% of visitors into buyers versus 2.0% for A — that is 35 orders instead of 20 from the same 1,000 visitors. By simply changing the button text and measuring, the bakery gets 75% more orders at no extra cost.

What makes a fair test

Change only one thing at a time (just the button, not the whole page) — or you will not know what caused the change.
Give each version a fair, equal split of traffic.
Run it long enough to get enough visitors — a handful of clicks proves nothing.
Pick one metric to judge by (here, conversion rate).

Tip: Good things to A/B test: ad headlines, button text, images, email subject lines, and prices. Start with the part people see first — the headline or the main image usually has the biggest effect.

Watch out: Do not call a winner too early. If version A is ahead after only 30 visitors, that could easily be luck. Wait for a few hundred visitors per version before trusting the result.

Q. In an A/B test, why should you change only one thing between version A and B?

Answer: If you change several things at once, a difference in results could come from any of them. Changing one thing at a time tells you precisely what caused the improvement.

✍️ Practice

Version A of an email subject got 100 opens from 1,000 sends; version B got 150. Calculate each open rate and name the winner.
List three things on a landing page you could A/B test, and which one you would try first.

🏠 Homework

Design an A/B test for a gym ad: write two headline versions, say what single thing differs, and which metric you would use to pick the winner.