I will spare you an entire post on why running controlled experiments is the gold standard for data-driven decision making or how to run them (so many books and posts on this). If you are reading this, you would already know that controlled experiments (or A/B testing, multivariate testing) are one of the most effective ways to make causal inferences, to scientifically validate most ideas.

In the early stage of a business, it is okay to base decisions on HiPPO (Highest paid person’s opinion) and past experiences, given the lack of data. But as a company scale, running experiments can help significantly reduce the cost of trying new ideas, fail fast (on a small scale), and pivot fast.

If your team is aligned on shared goal metrics, and you have the resources and infrastructure to run online controlled experiments, you are ready to foster a culture of experimentation! The three proven tips below would hopefully help you achieve it.

Show, don’t tell.

The goal of running experiments is to use these scientifically proven results in decision makings. This means as analysts, our job is not to present the findings to stakeholders and hope they would use it someday if they feel like it; but to convince them, gradually gain their trust and get them into the habit of using experimentation results in their decision making.

Convincing is hard, especially when the results are contradictory to people’s world-views, which frequently happens with A/B testings (and that’s the point of running tests). I suggest starting small, something requires low effort and you have high confidence in. So it doesn’t require much effort from your team, and you have confidence it would improve performance to help you prove your point.

For example, start by a/b testing subject lines. When your team is arguing about if adding emojis to subject lines would improve open rate, cut it, and end the argument with an A/B testing proposal. Review the experiment and interpret results together with stakeholders to ensure transparency throughout the process. When you’ve found a result you are confident in based on the stats, implement it on a larger scale, and share the performance improvement.

Once your stakeholders start seeing the positive impact on some of the small things, you can gradually work with them on testing and implement higher-cost and higher-impact things. With their buy-in, you are not long from empowering the team to innovate and share experimental ideas to improve key metrics. Ship fast, fail fast, pivot fast.

Effect size! Effect size! Effect size!

We all know statistical testing results need to be statistically significant. But when evaluating an A/B testing result, statistical significance only indicates if the treatment had an effect or not, but it doesn’t indicate the magnitude of the effect, the effect size.

The lift in performance from treatment not only is important in deciding the sample size (so you have enough statistical power to detect the expected effect), but it can also make or break your team’s trust in experimentations.

In business settings, in most cases, implementing the treatment on a larger scale requires some cost. Therefore, it is extremely important to set expectations on the magnitude of the lift in performance. For example, if the $/user is expected to increase by 0.1% — 0.5% (confidence interval of the effect size), will it justify the $50k cost spent implementing the treatment? If not clearly communicated, stakeholders might be disappointed in the result because of false expectations and already invested effort and money.

Two questions you should always ask your stakeholders before A/B testing:

• What’s the baseline and how big the effect you wish/expect to see with the treatment?
• Does the improvement need to pass a certain threshold for you to be able to justify the cost of making the change?

Too good to be forgotten — document everything

This is extremely important for keeping the experimentation culture in your organization but rarely mentioned, that is, keep the institutional memory by creating documentations of all experimentation designs, results, and learnings.

Besides everything else, the document will clearly show all the work you’ve done, (your blood and sweat) and the contributions you’ve made through experimentations. It’s definitely a handy material for your promotions and raises 😉.

And for the purpose of bringing experimental culture to your company…this is your internal marketing material!! Share the page with the entire organization. Because even if some people were not involved in the tests, the learnings from experimentations are still helpful for them.

Once they’ve seen all the documented examples of performance improvement through experimentations vs. purely based on intuitions, any sensible people would be coming running after you to ask for your help on setting up experiments for them.

Experimentation is a mindset, a team sport, a company culture. You cannot do it alone. Hope these 3 tips can help you bring your team on board!