Online controlled experiments (or A/B testing, multivariate testing) are one of the most effective ways to make causal inferences, to scientifically validate most ideas. As they grew more and more popular, setting up and interpreting experimentation results have become must-have skills for analysts and popular interview questions.

Even though I am quite experienced with experimentations, I still sometimes get confused about the exact definitions of common terms like 'statistical significance'. So I put together this cheat sheet to help you (and me) to remember the right interpretation of those experimentation results.

P.S. If you want to learn online experimentation, I highly recommend the book Trustworthy Online Controlled Experiments (A Practical Guide to A/B Testing), written by industry practitioners, it has a great mix of theory and practical advices that most other books lacks.

Join the club to see the video tutorial of this dashboard and many more exclusive content.