It’s surprisingly hard to find an online sample size calculator that has all the components needed for calculating sample size. So I put together one for proportion metrics (conversion rate, click-through rate), as always, for you and for me :)

Download the Tableau workbook here.

Components of this sample size calculator:

  • Baseline rate: the higher the baseline rate, the less sample size is needed to detect a certain MDE.
  • Relative MDE: the minimal detectable effect relative to the baseline rate the given sample size can detect. MDE indicates practical significance, what’s the lift that’s needed to make a business decision. 1% increase in conversion rate for Amazon has much higher practical significance than a 1% increase in conversion rate for an early-stage start-up.
  • Ratio of arm 2 to arm 1: not every test is 50–50 split, if you need to create a global holdout group it usually is 1% — 10% of the total sample.
  • P-value: the probability of detecting the difference by chance if there’s one.
  • Power: the probability of not committing a type-II error — the ability to detect a difference if there’s one.
  • Runtime vs. MDE: when making sample size recommendations, it’s always helpful to think as your stakeholder. What do they care about? How long do we need to run the test to detect a difference that we want to detect?