Meta-Analysis Calculator For A/B Tests

apples
For meta-analyzing and combining two or more a/b tests together.
Good for apples, red, green, orange-like, pink or simply honey crisps. Your call.
Test 1
/
vs
/
3.30%
0.2956
1.046
Your A/B Test
/
vs
/
 
 
 
Calculate
Meta Inverse Variance-weighted Method Total z-score 1.051 | Sum of weights 1,014.75 | Total sample 20,321
Stouffer Method: value 1.063 | p-value 0.2878 | z-score 1.063
+3.30%
0.2932
1.051

What Next?

The effect is likely small and possibly positive. We'd need more data to detect effects smaller than ±13.3%.

+8.6%
80% −8.6%
80%
+10%
90% −10%
90%
+11.2%
95% −11.2%
95%
+13.3%
99% −13.3%
99%
0%
  • 99% power to detect effects of ±13.3% or larger
  • 95% power to detect effects of ±11.2% or larger
  • 90% power to detect effects of ±10% or larger
  • 80% power to detect effects of ±8.6% or larger

α = 0.05 (two-sided)

QUICK SUMMARY OF APPROACH
  1. We take an inverse variance-weighted (precision-weighted) average of the lifts from each experiment to estimate the "true" lift that's assumed common to all experiments. This is also the maximum-likelihood estimate.
  2. Because we're working with RELATIVE lifts, we estimate the variance of the lift using the standard approach derived using the delta method.

CREDITS
A big thanks to the following indivduals for making this little project possible:
Ron Kohavi for guidance and input on weighing of p-values and inspring this project with his own meta-analysis spreadsheet.
Simon Dhala from whom I took the cumnormdist PHP function.
Tyler Buffington for multiple discussions and input about various meta analysis approaches.
Ryan Thomas for finding that the initial p-values were not being properly calculating upon the initial release.

Thanks,
Jakub Linowski