Improve›Core· 35 min read

A/B Testing: Pick the Winner

Show half your visitors version A and half version B, then keep whichever one gets more conversions.

What you will learn

Explain how an A/B test works
Read an A/B test result correctly
Avoid the common testing mistakes

Stop arguing, start testing

Your team argues: should the button say "Buy Now" or "Add to Cart"? Instead of guessing, you run an A/B test. You show version A to half your visitors and version B to the other half at the same time, then see which one gets more conversions. The visitors decide for you.

It is called A/B because there are two versions. The rule is simple: change only one thing between A and B, so you know exactly what caused any difference.

A worked example

The clothing store tests its checkout button. Version A is the current button; version B has new wording. Each version is shown to 5,000 visitors over two weeks.

Version	Button text	Visitors	Purchases	Conversion rate
A (current)	Buy Now	5,000	150	3.0%
B (new)	Get It Today	5,000	200	4.0%

Comparing the two versions and the size of the win

A: 150 / 5,000 = 3.0%
B: 200 / 5,000 = 4.0%

Improvement = (4.0 - 3.0) / 3.0 x 100 = 33% more conversions

Note: Version B converted at 4.0% versus 3.0% for A — a 33% improvement. With the same traffic, B brought 50 extra purchases. You make B the new default and the gain keeps compounding every month.

What you can A/B test

Headlines — "50% off everything" vs "Lowest prices this Diwali".
Buttons — wording, colour, or position.
Images — product photo vs lifestyle photo.
Email subject lines — to lift the open rate or CTR (CTR = click-through rate, the percentage of people who click).
Page layout — price at top vs reviews at top.

How an A/B test actually runs

You do not split visitors by hand. A testing tool (Google Optimize alternatives, VWO, or features built into your email and ads platforms) does it automatically:

You create two versions of one page or email — A and B.
The tool randomly shows A to half the visitors and B to the other half.
It quietly counts conversions for each version.
When enough people have seen both, you read the result and keep the winner.

The key word is random — each visitor is flipped a coin so the two groups are fair and comparable. That fairness is what lets you trust the result.

The mistakes that ruin tests

Mistake	Why it breaks the test
Changing two things at once	You cannot tell which caused the change
Stopping too early	A few clicks can look like a win by luck
Too few visitors	Small samples are unreliable
Testing during a one-off event	A sale or holiday distorts the result

Tip: Give a test enough people and enough time — usually a couple of weeks and at least a few hundred conversions per version. Tiny tests lie: 3 sales out of 10 is not proof of anything.

Watch out: A small difference might just be luck. If A gets 100 conversions and B gets 102, that is basically a tie. Only act when the winner is clearly and consistently ahead.

Q. Why should you change only ONE thing between version A and version B?

Answer: If you change two things at once and results improve, you cannot tell which change was responsible. Changing one thing keeps the test clean.

✍️ Practice

Version A of an email got 500 opens from 5,000 sends; version B got 650. Calculate both open rates and the winner.
Write two headline versions for a bakery Diwali offer that you could A/B test.

🏠 Homework

Pick one web page or email you have seen. Describe one A/B test you would run on it: state version A, version B, and the single thing you changed.