People search for an A/B test sample size calculator when a new page is ready and the question is how many visitors each version needs.
The sample size calculator uses the normal approximation for a change in a proportion. NIST’s Engineering Statistics Handbook publishes that method, with a z for confidence and a z for power. The two-variant form on this page uses both conversion rates.
n = (z alpha + z beta) squared × (p1 × (1 - p1) + p2 × (1 - p2)) / (p1 - p2) squared
p1 and p2 are the two rates written as decimals. n is visitors per variant. The page rounds n up to the next whole visitor. Total visitors are that count times the number of variants. Traffic is assumed to split evenly.
The field labels name the z values this page is built around. Two-sided 95% significance is 1.96. 80% power is 0.84. Their sum is 2.80. Squared, that is 7.84. Every example below keeps those two z values unless a row says otherwise. Changing the z is a different test. These rows change the rates and the variant count.
Stopping early, extra looks at the dashboard, and a holdout after the test sit outside the figure. The formula assumes you wait for the count. Revenue from a visitor count you already have is the conversion funnel calculator and the funnel post.
Fig 1
From two rates to a headcount
The gap sits in the denominator
Square the sum of the two z values, multiply by both variances, and divide by the squared gap. Then multiply by the variant count.
Example 1: 20% to 30%
Inputs: baseline 20%, expected 30%, z alpha 1.96, z beta 0.84, variants 2.
p1 = 0.20. p2 = 0.30. The gap is 0.10.
Variance: 0.20 × 0.80 + 0.30 × 0.70 = 0.16 + 0.21 = 0.37
n = 7.84 × 0.37 / 0.01 = 2.9008 / 0.01 = 290.08
The page rounds up to 291 visitors per variant.
Total: 291 × 2 = 582
Absolute lift: 30 - 20 = 10 percentage points, which the page prints as 10%.
Calculator result
A/B test sample size
- Baseline conversion rate
- 20%
- Expected conversion rate
- 30%
- Significance z (two-sided 95% is 1.96)
- 1.96 z
- Power z (80% power is 0.84)
- 0.84 z
- Variants including control
- 2 variants
291
visitors per variant
- Total visitors
- 582
- Absolute lift
- 10%
- Variants
- 2
If the new page does convert at 30%, the variant’s 291 visitors produce 291 × 0.30 = 87.3 orders. The control’s 291 produce 291 × 0.20 = 58.2 orders. The difference on this sample is 29.1 orders. Using the $80 ticket from the funnel post’s worked example, 29.1 × 80 = $2,328. That dollar line is the ticket times the order gap. It is not part of the sample-size result. The sample-size page stops at visitors.
At 200 visitors a day, split across two variants, the test needs 582 / 200 = 2.91 days. Two hundred a day is a number you type. It is not a traffic benchmark.
Quoting 291 as the whole test leaves out the control. The pair is 582.
Example 2: cut the lift in half, then keep cutting
Same baseline, 20%. Same z values. Two variants.
Expected 25%. The gap is 0.05.
Variance: 0.20 × 0.80 + 0.25 × 0.75 = 0.16 + 0.1875 = 0.3475
n = 7.84 × 0.3475 / 0.0025 = 2.7244 / 0.0025 = 1,089.76, which rounds up to 1,090 per variant.
Total: 1,090 × 2 = 2,180
A 5 point lift needs 1,090 per variant. The 10 point lift in Example 1 needed 291. The gap was cut in half and the sample rose by much more than double, because the gap is squared.
Expected 22%. The gap is 0.02.
Variance: 0.16 + 0.22 × 0.78 = 0.16 + 0.1716 = 0.3316
n = 7.84 × 0.3316 / 0.0004 = 6,499.36, rounded up to 6,500 per variant.
Total: 6,500 × 2 = 13,000. Lift 2%.
Expected 21%. The gap is 0.01.
The page returns 25,551 per variant and 51,102 total. Lift 1%.
At 2,000 visitors a day, 51,102 / 2,000 = 25.551 days. At 200 a day, the same test is 51,102 / 200 = 255.51 days. The formula did not change. The calendar did, because you divided the same headcount by a different daily number.
Calculator result
A/B test sample size
- Baseline conversion rate
- 20%
- Expected conversion rate
- 22%
- Significance z (two-sided 95% is 1.96)
- 1.96 z
- Power z (80% power is 0.84)
- 0.84 z
- Variants including control
- 2 variants
6,500
visitors per variant
- Total visitors
- 13,000
- Absolute lift
- 2%
- Variants
- 2
Example 3: a 20% relative lift is not a 20 point lift
People say “a 20% lift” and type 40 into the expected box. On a 20% baseline, a 20% relative lift is 20 × 1.20 = 24%. A 20 point lift is 40%.
20% to 24%, two variants, same z values: 1,678 per variant, 3,356 total. Lift on the page: 4%.
20% to 40%: variance is 0.16 + 0.40 × 0.60 = 0.16 + 0.24 = 0.40
n = 7.84 × 0.40 / 0.04 = 3.136 / 0.04 = 78.4, rounded up to 79 per variant.
Total: 79 × 2 = 158. Lift 20%.
1,678 / 79 = 21.24. The relative reading needs about 21 times the visitors per variant. Both readings use the words “20%.” Only one of them is a 20 point gap. The page’s lift line is the difference of the two rates you type, in percentage points.
Example 4: the same 10 points, higher on the scale
Inputs: baseline 40%, expected 50%, same z values, 2 variants.
Variance: 0.40 × 0.60 + 0.50 × 0.50 = 0.24 + 0.25 = 0.49
n = 7.84 × 0.49 / 0.01 = 384.16, rounded up to 385 per variant.
Total: 385 × 2 = 770. Lift 10%.
Example 1 was also a 10 point lift, and it needed 291 per variant. The rates here sit closer to 50%, where p × (1 - p) is larger. 0.49 / 0.37 = 1.324, and 290.08 × 1.324 is about 384, which is the unrounded n before the page rounds up. A 10 point lift is not one sample size. It depends on where the two rates sit.
Example 5: a third variant, then a fourth
Keep 20% and 30%, and the same z values. Set variants to 3.
Per variant stays 291. The page does not shrink n because you added a variant. Total visitors: 291 × 3 = 873.
Four variants on the same rates: 291 × 4 = 1,164 total. Per variant is still 291.
Calculator result
A/B test sample size
- Baseline conversion rate
- 20%
- Expected conversion rate
- 30%
- Significance z (two-sided 95% is 1.96)
- 1.96 z
- Power z (80% power is 0.84)
- 0.84 z
- Variants including control
- 3 variants
291
visitors per variant
- Total visitors
- 873
- Absolute lift
- 10%
- Variants
- 3
A different test, four variants, baseline 30%, expected 33%:
The page returns 3,756 per variant and 15,024 total. Lift 3%.
The third and fourth variants multiply the per-variant count. They do not divide it.
Fig 2 · 1/2
Per variant, then the whole test
20% to 30%, z of 1.96 and 0.84.
Visitors
- Per variant
- Total
| Category | Per variant | Total |
|---|---|---|
| 2 variants | 291 | 582 |
| 3 variants | 291 | 873 |
| 4 variants | 291 | 1164 |
Each variant still needs 291 visitors when you add a third or a fourth.
The test total is 582, then 873, then 1,164.
Example 6: a checkout moving from 12% to 14%
Inputs: baseline 12%, expected 14%, same z values, 2 variants.
The page returns 4,430 per variant and 8,860 total. Lift 2%.
A signup page moving from 8% to 10% is also a 2 point lift, lower on the scale. The page returns 3,207 per variant and 6,414 total.
Same 2 point gap, two different counts. Example 2’s 20% to 22% needed 6,500 per variant. The variance term changes with the rates, so “2 points” is not a single row in a sample-size table.
Example 7: one point on a small rate
Inputs: baseline 3%, expected 4%, same z values, 2 variants.
The page returns 5,292 per variant and 10,584 total. Lift 1%.
Baseline 5% to 6%: 8,146 per variant, 16,292 total. Also a 1 point lift.
Baseline 2% to 2.5%: the gap is half a point. The page prints the lift as 0.5%, with 13,791 per variant and 27,582 total.
Example 2’s 1 point lift, from 20% to 21%, needed 25,551 per variant. The 3% to 4% test needs 5,292. The gap is one point in both rows. The variances are smaller when both rates are near zero, and the sample follows.
Example 8: a 0.3 point lift on a 1.5% baseline
Inputs: baseline 1.5%, expected 1.8%, same z values, 2 variants.
The page returns 28,269 per variant and 56,538 total. The lift prints as 0.3%.
If the page sees 300 visitors a day, 56,538 / 300 = 188.46 days. Three hundred a day is a number you type for the division. A busier page shortens the calendar. It does not shrink the 56,538.
Fig 3 · 1/6
Visitors per variant as the gap shrinks
Baseline stays at 20%. z values stay at 1.96 and 0.84.
Visitors per variant
- Per variant
| Point | Per variant |
|---|---|
| 20 pp | 79 |
| 10 pp | 291 |
| 5 pp | 1090 |
| 4 pp | 1678 |
| 2 pp | 6500 |
| 1 pp | 25551 |
A 20 point lift needs 79 visitors per variant. A 10 point lift needs 291.
A 4 point gap needs 1,678. A 2 point gap needs 6,500. A 1 point gap needs 25,551.
Example 9: the two rates are the same
Inputs: baseline 20%, expected 20%, same z values, 2 variants.
The gap is zero. The page stops and asks for a conversion change, because the formula divides by that gap. There is no visitor count to round up.
A test that hopes to “see what happens” still needs an expected rate in the box. Type the smallest lift you care about detecting. Example 2 is that exercise for a 20% baseline. One point needs 25,551 per variant. Ten points need 291.
Example 10: days, once the count is known
The formula returns visitors. Days are that count divided by visitors per day, which you supply.
| Test | Total visitors | Visitors per day | Days |
|---|---|---|---|
| 20% to 30%, 2 variants | 582 | 200 | 2.91 |
| 20% to 22%, 2 variants | 13,000 | 200 | 65 |
| 20% to 22%, 2 variants | 13,000 | 400 | 32.5 |
| 20% to 21%, 2 variants | 51,102 | 2,000 | 25.551 |
| 1.5% to 1.8%, 2 variants | 56,538 | 300 | 188.46 |
| 20% to 30%, 3 variants | 873 | 200 | 4.365 |
13,000 / 400 = 32.5. 873 / 200 = 4.365. Doubling daily traffic cuts the days in half. It does not change n. Adding a variant raises the total, so the same 200 a day takes longer: 4.365 days instead of 2.91.
Fig 4 · 1/4
Total visitors on four tests
Two variants, z of 1.96 and 0.84, except the third bar.
Total visitors
20% to 30%
582
40% to 50%
770
3 variants
873
20% to 25%
2,180
The default pair needs 582 visitors.
A 10 point lift from 40% to 50% needs 770. Three variants of the default need 873. A 5 point lift from 20% needs 2,180.
The results side by side
| Example | Baseline | Expected | Variants | Per variant | Total | Lift |
|---|---|---|---|---|---|---|
| 1. Defaults | 20% | 30% | 2 | 291 | 582 | 10% |
| 2. Half the lift | 20% | 25% | 2 | 1,090 | 2,180 | 5% |
| 2. Two points | 20% | 22% | 2 | 6,500 | 13,000 | 2% |
| 2. One point | 20% | 21% | 2 | 25,551 | 51,102 | 1% |
| 3. Relative 20% | 20% | 24% | 2 | 1,678 | 3,356 | 4% |
| 3. Twenty points | 20% | 40% | 2 | 79 | 158 | 20% |
| 4. Higher rates | 40% | 50% | 2 | 385 | 770 | 10% |
| 5. Third variant | 20% | 30% | 3 | 291 | 873 | 10% |
| 5. Fourth variant | 20% | 30% | 4 | 291 | 1,164 | 10% |
| 5. Four, smaller lift | 30% | 33% | 4 | 3,756 | 15,024 | 3% |
| 6. Checkout | 12% | 14% | 2 | 4,430 | 8,860 | 2% |
| 6. Signup | 8% | 10% | 2 | 3,207 | 6,414 | 2% |
| 7. Three to four | 3% | 4% | 2 | 5,292 | 10,584 | 1% |
| 7. Five to six | 5% | 6% | 2 | 8,146 | 16,292 | 1% |
| 7. Half a point | 2% | 2.5% | 2 | 13,791 | 27,582 | 0.5% |
| 8. Email-sized | 1.5% | 1.8% | 2 | 28,269 | 56,538 | 0.3% |
| 9. No gap | 20% | 20% | 2 | none | none | 0 |
Every row uses z alpha 1.96 and z beta 0.84. “None” on the last row is the page refusing the division.
Open the A/B sample size calculator with the rate you have and the rate you hope to detect. Then divide the total by the visitors you get in a day. Orders and revenue from a count you already collected belong on the funnel calculator. The ticket on those orders is the average order value calculator. How long a paid customer takes to repay acquisition cost is the payback calculator.