Ab Test Sample Size Calculator

Sample Size Estimates Under Varying Conditions
Baseline Rate (%)	MDE (%)	Significance (α)	Power (1-β)	Sample Size Per Variant
5.0	10.0	0.05	0.80	3,842
5.0	15.0	0.05	0.80	1,717
5.0	10.0	0.01	0.80	6,636
5.0	10.0	0.05	0.90	5,123
10.0	10.0	0.05	0.80	3,842
10.0	20.0	0.05	0.80	1,717

What is an A/B Test Sample Size Calculator?

An A/B test sample size calculator is a tool used by marketers, product managers, data analysts, and UX designers to determine the appropriate number of participants or observations needed for an A/B test. Running an A/B test involves splitting your audience into two or more groups, showing each group a different version (A or B) of a webpage, app feature, or marketing campaign, and then measuring which version performs better based on a specific metric (like conversion rate, click-through rate, or revenue). The sample size calculator helps ensure that you gather enough data to make a statistically valid conclusion about which version is superior. Without sufficient sample size, you risk drawing incorrect conclusions due to random chance, leading to poor business decisions.

Who should use it? Anyone planning or executing A/B tests. This includes:

Digital Marketers: Optimizing landing pages, ad copy, email campaigns.
Product Managers: Testing new features, UI changes, onboarding flows.
UX/UI Designers: Evaluating design variations for usability and engagement.
Data Analysts: Ensuring the validity of experimental results.
E-commerce Managers: Improving conversion rates and average order value.

Common misconceptions:

“Bigger is always better”: While more data is generally good, unnecessarily large sample sizes waste time and resources. The calculator helps find the *optimal* size.
“We need to run the test for a fixed time (e.g., 2 weeks)”: The duration of a test is determined by traffic volume and the required sample size, not an arbitrary deadline.
“Sample size doesn’t matter if the difference is huge”: Even large differences can be due to chance with small sample sizes. Statistical significance accounts for this.
“We can just use the same sample size for all tests”: Sample size requirements vary significantly based on baseline rates, desired effect size, and confidence levels.

A/B Test Sample Size Formula and Mathematical Explanation

The core of an A/B test sample size calculator relies on statistical formulas derived from hypothesis testing. Specifically, for A/B tests measuring conversion rates (a common scenario), the formula is based on the two-proportion z-test. The goal is to find the sample size per group (N) required to detect a specific difference between two proportions (p1 and p2) with a given level of confidence and power.

A common formula for sample size per group (N) is:

N = [ (Z_α/2 * sqrt(p̄(1-p̄) * 2)) + (Z_β * sqrt(p1(1-p1) + p2(1-p2))) ]^2 / (p1 – p2)^2

Where:

N: Sample size required for *each* group (variant A and variant B). The total sample size is 2N.
p1: The expected conversion rate for the variation (Control Group Rate + Minimum Detectable Effect).
p2: The expected conversion rate for the control (Baseline Conversion Rate). Let’s correct this in the formula derivation for clarity: usually p1 is the expected rate for variant A and p2 for variant B. Let’s redefine for clarity:

N = [ (Z_α/2 * sqrt(2 * p̄ * (1-p̄))) + (Z_β * sqrt(p1*(1-p1) + p2*(1-p2))) ]^2 / (p1 – p2)^2

Let’s redefine the variables for clarity, aligning with the calculator’s inputs:

Variable	Meaning	Unit	Typical Range / Source
p_baseline	Baseline conversion rate (Control group)	Proportion (e.g., 0.05 for 5%)	0.01 – 0.99 (Based on historical data)
MDE	Minimum Detectible Effect (Absolute or Relative)	Proportion or Percentage	0.01 – 0.50 (What you aim to detect)
p_variation	Expected conversion rate for the variation	Proportion	p_baseline + MDE_absolute or p_baseline * (1 + MDE_relative)
α (Alpha)	Statistical Significance Level (Type I Error Rate)	Probability (e.g., 0.05)	0.01, 0.05, 0.10 (Confidence Level = 1 – α)
β (Beta)	Type II Error Rate	Probability (e.g., 0.20)	(1 – Power)
Z_α/2	Z-score for two-tailed test at significance level α	Standard Score	Approx. 1.96 for α = 0.05, 2.576 for α = 0.01
Z_β	Z-score for one-tailed test at power level (1-β)	Standard Score	Approx. 0.84 for Power = 0.80, 1.28 for Power = 0.90
p̄ (p-bar)	Average proportion under the null hypothesis	Proportion	(p_baseline + p_variation) / 2
N	Required sample size per group (variant)	Count	Calculated value

Simplified Calculation Approach: Many calculators use approximations or simplified versions. The calculator above uses a common approximation that calculates the pooled variance and then solves for N. It assumes a two-tailed test and calculates the necessary Z-scores based on the inputs.

The calculator first determines p_variation based on the baseline rate and MDE (assuming MDE is relative). Then it calculates p̄. Finally, it computes N using the appropriate Z-scores.

Practical Examples (Real-World Use Cases)

Example 1: E-commerce Checkout Button Color Test

An e-commerce site wants to test if changing the “Add to Cart” button color from blue (control) to green (variation) increases the purchase conversion rate.

Current Baseline Conversion Rate: 3.5%
Minimum Detectible Effect: They want to detect at least a 15% relative increase (meaning the green button needs to achieve at least 3.5% * 1.15 = 4.025% conversion rate).
Statistical Significance: 95% (α = 0.05)
Statistical Power: 80% (1-β = 0.80)

Calculation using the tool:

Inputting these values yields a required sample size of approximately 2,580 participants per variant.

Interpretation: To be reasonably confident (95% sure) that any observed difference is real and not due to chance, and to have a good chance (80%) of detecting a 15% improvement if it exists, they need at least 2,580 visitors to the page showing the blue button and 2,580 visitors to the page showing the green button. If their daily traffic is 500 visitors, this test would need to run for about 11 days (5160 / 500 ≈ 10.32 days).

Example 2: SaaS Sign-up Form Headline Test

A SaaS company wants to test a new headline on their landing page to see if it improves free trial sign-ups.

Current Baseline Conversion Rate: 10%
Minimum Detectible Effect: They want to detect at least a 10% relative increase (meaning the new headline needs to achieve at least 10% * 1.10 = 11% conversion rate).
Statistical Significance: 95% (α = 0.05)
Statistical Power: 90% (1-β = 0.90) – They want higher confidence in detecting a true effect.

Calculation using the tool:

Inputting these values yields a required sample size of approximately 7,688 participants per variant.

Interpretation: Because they require higher power (90% vs 80%) and are aiming to detect a smaller relative effect (10% vs 15% in the previous example), the required sample size is significantly larger. Running this test would require approximately 15,376 visitors in total (7,688 per variant). If their traffic is 1,000 visitors per day, the test would run for roughly 16 days.

How to Use This A/B Test Sample Size Calculator

Using the calculator is straightforward. Follow these steps:

Determine Your Baseline Conversion Rate: This is the current performance of your control version. Use historical data from your analytics platform (e.g., Google Analytics). Express it as a percentage (e.g., 5%).
Define Your Minimum Detectible Effect (MDE): What is the smallest improvement you care about detecting? This is crucial. A smaller MDE requires a larger sample size. You can think of this in relative terms (e.g., a 10% lift) or absolute terms. The calculator typically uses relative MDE.
Set Statistical Significance (Alpha): This is your tolerance for a false positive (Type I error – concluding there’s a difference when there isn’t). The standard is 95% confidence (α = 0.05). Lowering this (e.g., to 0.01 for 99% confidence) increases the required sample size.
Set Statistical Power (1 – Beta): This is your ability to detect a real difference if one exists (avoiding a false negative, Type II error). The standard is 80% power (β = 0.20). Increasing power (e.g., to 90% or 95%) increases the required sample size.
Click “Calculate Sample Size”: The calculator will instantly provide the required sample size needed for *each* variant (A and B).

How to read results: The primary result (“Required Sample Size Per Variant”) tells you the minimum number of unique users or sessions needed for each version you are testing. The intermediate results show the calculated values for the expected conversion rate of the variation.

Decision-making guidance: Once you have the sample size, divide your total traffic by this number to estimate how long the test needs to run. If the required duration is too long for your business needs (e.g., longer than a typical campaign cycle), you might need to reconsider your MDE (aiming for a larger effect) or accept a lower level of statistical confidence or power.

Key Factors That Affect A/B Test Sample Size Results

Several factors influence the sample size calculation. Understanding these helps in setting appropriate parameters:

Baseline Conversion Rate: Lower baseline rates generally require larger sample sizes to detect the same relative or absolute lift. Detecting a 10% relative lift on a 1% baseline (0.1% absolute) is harder than on a 10% baseline (1% absolute).
Minimum Detectable Effect (MDE): This is arguably the most significant factor. The smaller the effect you want to be able to detect, the larger your sample size needs to be. Aiming to detect a 5% lift requires far more data than aiming to detect a 50% lift.
Statistical Significance (Alpha): Increasing your confidence level (e.g., from 90% to 95% or 99%) requires a larger sample size. This reduces the risk of a false positive but demands more data to be certain.
Statistical Power (1 – Beta): Increasing your power (e.g., from 80% to 90% or 95%) also increases the sample size. This reduces the risk of a false negative (missing a real effect) but requires more observations.
Type of Test (One-tailed vs. Two-tailed): The standard A/B test assumes a two-tailed test (checking if B is better OR worse than A). If you have a strong hypothesis that B can *only* be better (or only worse), a one-tailed test requires a smaller sample size, but it’s less common and riskier in practice. Our calculator uses the standard two-tailed approach.
Segmentation and Traffic Split: While not directly in the core formula, consider your traffic volume and how you plan to split it. If you’re testing more than two variants (A/B/C testing), the sample size per variant calculation needs adjustment. Also, if you plan to analyze results for specific segments (e.g., mobile vs. desktop users), you’ll need sufficient sample size *within each segment*.
Variability of the Metric: While conversion rates are common, some tests might measure continuous variables (e.g., average order value). These require different formulas (often based on standard deviation) and sample size calculations, which can differ significantly.

Frequently Asked Questions (FAQ)

What is the difference between relative and absolute MDE?
A relative MDE is a percentage increase over the baseline (e.g., a 10% relative increase over a 5% baseline means you want to detect a new rate of 5.5%). An absolute MDE is a fixed difference (e.g., detecting any difference of 0.5 percentage points, regardless of the baseline). The calculator typically assumes relative MDE.

How long should I run my A/B test?
You should run the test until you reach the calculated sample size per variant. Running for a fixed time (like 1 week) can be misleading if you haven’t hit your sample size target or if you’ve captured a full business cycle (e.g., weekdays vs. weekends).

What happens if I stop the test early?
Stopping early, especially if you see a significant result, risks making a decision based on chance (violating your statistical significance). This is known as the “peeking problem” and leads to inflated false positive rates. Always aim for your target sample size.

Can I use different sample sizes for variants A and B?
Standard A/B testing assumes equal sample sizes for valid statistical comparison. While unequal sample sizes are possible with advanced methods, it’s generally recommended and simpler to aim for an equal split of traffic and thus equal sample sizes.

What if my website traffic is very low?
Low traffic makes it difficult to achieve adequate sample sizes quickly. You might need to: accept a larger MDE, run the test for a longer period, test during peak traffic times, or explore multi-armed bandit testing approaches which allocate more traffic to winning variants sooner.

Does the calculator account for seasonality or day-of-week effects?
The core sample size formula itself doesn’t explicitly model seasonality. However, the recommendation to run tests over full business cycles (e.g., 1-2 weeks) helps mitigate these effects by averaging them out. Ensure your historical baseline data also reflects typical patterns.

What’s the practical difference between 80% and 90% statistical power?
80% power means you have an 80% chance of detecting a real effect of your specified MDE. 90% power increases this chance to 90%. Higher power reduces the risk of a false negative (Type II error) but requires a larger sample size. For critical tests or when missing an effect is very costly, higher power is preferred.

Can I use this calculator for metrics other than conversion rates?
This specific calculator is designed for conversion rates (proportions). Metrics like average revenue per user, time on site, or click-through rates (if framed as a proportion) may require different formulas and calculators, often involving standard deviation rather than just proportions.

Related Tools and Internal Resources

Why A/B Testing is Crucial for Growth

Learn about the fundamental benefits of A/B testing and how it drives data-informed decisions.
Ultimate Guide to Conversion Rate Optimization (CRO)

Discover strategies and tactics to improve your website’s performance and user experience.
Understanding Statistical Significance in Experiments

Deep dive into what statistical significance means and why it matters for A/B tests.
Conversion Rate Calculator

Calculate conversion rates from raw visitor and conversion data.
Top 10 A/B Testing Mistakes to Avoid

Learn from common pitfalls to ensure your A/B tests are effective and reliable.
A/B Testing vs. Multivariate Testing

Understand the differences between these testing methodologies and when to use each.

A/B Test Sample Size Calculator