What is Split Testing (A/B Testing)?
Also known as: A/B testing, Bucket testing, Champion-challenger testing
What is split testing?
Split testing, also known as A/B testing, runs two versions of the same asset at the same time and measures which one converts better.
The asset can be anything: a landing page, an ad creative, an email subject line, a checkout flow, a pricing page. The principle is identical. Half the audience sees version A. Half sees version B. After enough conversions, the math tells you which version wins.
The test removes guesswork. Without split testing, marketing teams argue about copy and design. With it, the decision is data.
How a split test works
A split test runs in five steps.
1. Pick one variable
Change one thing. Just one.
If you change the headline and the button color and the hero image at the same time, the result tells you nothing about why one version won. You need a controlled experiment, not a redesign.
The variable should be the highest-impact element you can think of. Headlines and hero imagery move conversion rates more than fonts or colors.
2. Define the success metric
Pick the metric you'll judge by before the test starts. Common choices:
- Conversion rate. Sign-ups, purchases, lead form completions.
- Click-through rate (CTR). Useful for ads and emails.
- Revenue per visitor. When average order value matters as much as conversion.
- Bounce rate or time-on-page. When the test is about engagement, not action.
Choosing the metric after seeing results is called "p-hacking." It produces fake winners.
3. Calculate the sample size
Use a calculator like Optimizely's sample size calculator or Evan Miller's tool. Inputs:
- Current conversion rate (the baseline).
- Minimum detectable effect (the smallest improvement worth caring about, usually 5 to 20 percent relative).
- Statistical power (typically 80 percent).
- Significance level (typically 95 percent).
The output is the number of conversions per variant needed before you can trust the result.
4. Split traffic 50/50
Use a testing platform that randomly assigns each visitor to one variant. Google sunset Optimize in September 2023. The current standard tools are Optimizely, VWO, AB Tasty, Convert.com, and the experiment tools built into Meta Ads Manager and Google Ads.
Don't split manually. Manual splits introduce selection bias.
5. Wait. Don't peek.
Run the test until you hit your sample size and a full business cycle (typically 7 to 14 days for B2C, longer for B2B). Stopping early because one variant "looks ahead" produces false positives.
When the test ends, declare a winner only if it cleared the significance threshold.
Why split testing matters
Three reasons split testing is the foundation of performance marketing.
- It compounds. A 10 percent conversion rate lift on a landing page applies forever to every visitor. Five rounds of 10 percent lifts compounds to a 61 percent total improvement.
- It kills HiPPO decisions. "Highest paid person's opinion" is the worst basis for marketing choices. Split tests replace opinions with revenue numbers.
- It scales. Modern AI ad platforms can run dozens of micro-experiments across creatives, audiences, and placements at once. Manual decisions can't keep up.
A 10 percent lift compounded over five rounds becomes a 61 percent total lift. The math: 1.1 × 1.1 × 1.1 × 1.1 × 1.1 = 1.61.
Real-world example: A/B testing an ad creative
A SaaS company runs three Facebook ads with the same audience and budget. Each ad uses a different hook in the headline:
- Hook A: "Generate AI ad creatives in 30 seconds."
- Hook B: "Stop paying agencies $5,000 per month."
- Hook C: "Your competitors are running 20 ad variants. You're running 2."
Spend per ad: $200 over seven days. Total budget: $600.
Results after 7 days:
| Hook | Impressions | Clicks | CTR | Cost per lead |
|---|---|---|---|---|
| A (Speed) | 38,000 | 410 | 1.1% | $42 |
| B (Cost) | 41,000 | 980 | 2.4% | $19 |
| C (FOMO) | 39,500 | 1,140 | 2.9% | $14 |
Hook C wins. The team kills A and B, scales C to $5,000 weekly, then starts a new round of tests against C as the new baseline.
The lift from $42 to $14 cost-per-lead is a 67 percent reduction. On a $50,000 monthly ad budget, the same spend now produces 3,571 leads instead of 1,190. That's 2,381 extra leads per month from a single test round. The compounding effect of A/B testing across creative, audience, and landing page tests adds up to the 5x to 10x ROAS gaps that separate winning ad accounts from losing ones.
Split testing in ad platforms
Split testing inside ad platforms differs from website testing.
Meta Ads Manager has a dedicated A/B Test feature inside the campaign builder. Test variables include creative, audience, placement, and delivery optimization. The platform handles random assignment and significance.
Google Ads uses "Drafts and Experiments." A draft is a copy of a campaign with one or more changes. The experiment splits traffic between draft and control. Best for testing bid strategies, ad copy, and landing pages.
TikTok Ads Manager has Split Testing under the campaign budget settings. Tests creative, audience, or delivery type.
In an AI ad platform like Coinis, the testing loop is automated. The system generates 20 ad variants from a single product link. Each variant gets a small budget. The platform shifts spend toward the winners every 24 hours. The marketer sees the result, not the test setup.
Related terms
Frequently asked questions
Is split testing the same as A/B testing?
Yes. The two terms are interchangeable. Split testing is the older marketing term. A/B testing is what most software calls it. Both describe the same method: send half your traffic to one version and half to another, then measure which converts better.
How long should an A/B test run?
Until the result is statistically significant, usually 95 percent confidence. For a landing page with 1,000 daily visits and a baseline 2 percent conversion rate, that's typically 7 to 14 days. For ads with smaller audiences, 2 to 4 weeks. Stopping a test early because one version 'looks ahead' is the most common mistake in A/B testing.
What is the difference between A/B testing and multivariate testing?
A/B testing changes one element at a time (headline A vs headline B). Multivariate testing changes multiple elements at once and measures every combination. A/B is faster and easier to interpret. Multivariate finds compound effects but needs much more traffic to reach significance.
Can you A/B test ads in Meta and Google?
Yes. Both platforms have built-in experiment tools. Meta Ads Manager has 'Experiments' for A/B testing creatives, audiences, and placements. Google Ads has 'Drafts and Experiments' for campaign-level tests. Always run platform-native experiments instead of manual splits to avoid bias from algorithmic delivery.
What should you A/B test first?
Test the elements with the largest expected impact. Headlines and hero images on landing pages. Hooks and thumbnails in video ads. Subject lines in emails. Skip button colors and font sizes until everything bigger is dialed in.