What effects can you expect from A/B tests?

Actively updated insights from 650 A/B tests :)

Small or Big

The smaller experiments are all sourced from GoodUI and are typically more "isolated" towards a single pattern (with occasional confounding). The bigger experiments are ones which combined as many multiple and positive probability changes into a single variation. These have been sourced from larger optimization projects we've designed and a/b tested over the years.

Adding, Replacing or Removing

Desktop or Mobile

By Change Types

By Screen Types

By Metric Types

Statistical Power

Share of tests with 80% power at each MDE (608 of 650 tests with binary metric data)

Power is the chance a test would detect a true relative effect of the given size (minimum detectable effect, MDE) as significant, using a two-tailed test at α = 0.05. It's calculated from each test's visitor counts and control conversion rate on its primary metric. Tests without visitor data or with continuous metrics (such as revenue) are excluded.