A/B Testing
Also known as: A/B Test, AB Testing, Split Testing, Bucket Testing
A/B testing, also called split testing, is an experiment that shows two versions of a page, ad, or email to randomly split audiences at the same time and measures which one better achieves a goal, such as sign-ups or purchases. Version A is the current control and version B is the variant with one change.
Key Takeaways
- An A/B test compares a control (A) with a variant (B) using randomly split, simultaneous traffic.
- Decide the goal metric, the change, and the sample size before the test starts.
- Statistical significance tells you a difference is unlikely to be chance; it does not tell you it is big enough to matter.
- A/B tests change one thing; multivariate tests change several elements at once and need far more traffic.
- For SEO-safe tests, Google advises no cloaking, rel=canonical on variant URLs, 302 redirects, and ending tests promptly.
How It Works
A test starts with a hypothesis tied to a metric, such as changing the headline to state the price will raise quote requests. Visitors are randomly assigned to the control or the variant, and both run at the same time so outside factors like day of the week or a promotion hit both equally. You decide the sample size before launch, run the test for whole business cycles, and only then read the result.
The result has two questions. Statistical significance asks whether the difference is unlikely to be random noise. Practical significance asks whether it is large enough to be worth shipping. A tiny lift can be significant with enough traffic and still not matter. Google defines A/B testing as testing two or more variations of a change, and multivariate testing as testing multiple types of changes at once to see the impact of each and how they interact (Google Search Central).
If the test changes URLs or content that Google crawls, follow Google's testing guidance: never show Googlebot something different from people, put rel=canonical on alternate URLs pointing to the original, use 302 rather than 301 redirects, and remove test elements once the test ends. I walk through the full process, including sample size math and QA, in my A/B testing guide for practitioners, and the CRO hub covers what to test first.
Why It Matters
It replaces opinion and guesswork with evidence from real user behavior. Instead of debating which headline or button works, you let visitors decide, which protects revenue and compounds small wins into meaningful conversion gains over time.
Example
A software company tests its pricing page. Version A keeps the current headline; version B adds a line naming the free trial length. Traffic splits evenly and randomly, and the team waits until the pre-planned sample size is reached across two full weeks. Version B produces more trial starts by a margin that is both statistically significant and large enough to matter, so it ships and the next test begins.
Common Mistake
Calling a winner too early. Ending a test before it reaches an adequate sample size and significance means you act on noise, not signal, and often ship a variant that performs no better than the original.
Frequently Asked Questions
What does A/B testing mean?
A/B testing means comparing two versions of something, like a web page or ad, by showing each to a random half of your audience at the same time and measuring which one performs better on a set goal. A is the original and B is the changed version.
What is an example of an A/B test?
An online store shows half its visitors a product page with the button text Add to Cart and the other half Buy Now, then compares purchase rates. Other common tests include headlines, hero images, form length, pricing layout, and email subject lines.
What is the difference between A/B testing and multivariate testing?
A/B testing compares versions that differ by one change. Multivariate testing changes several elements at once to measure each one's effect and how they interact, so it needs much more traffic to reach reliable results.
How long should an A/B test run?
Until it reaches the sample size you calculated before launch, and for whole weekly cycles so weekday and weekend behavior are both included. Stopping early because one version looks ahead is the most common way tests produce false winners.
Does A/B testing hurt SEO?
Not if you follow Google's guidance. Do not show Googlebot different content than users, add rel=canonical on variant URLs pointing to the original, use 302 redirects for redirect tests, and remove the test once it ends. Long-running tests can look deceptive.
What does statistical significance mean in an A/B test?
It means the measured difference between versions would be unlikely if there were truly no difference, given the confidence level you chose in advance. It does not measure how big or valuable the difference is, so check practical impact as well.