My learnings from A/B testing

Why did A/B testing gain so much popularity in the internet industry? There are 3 main reasons…

1. Its relatively easy to iterate on versions
2. Scale required for this statistical method is available
3. Segments catered are truly diverse

As a PM when you are building products, you have to deal with assumptions, opinions, and gut feelings — not only from people around you but also from yourself. For over 2 decades now A/B testing, or split testing, has been used in the internet industry to separate theory (assumptions, opinions and gut feelings) and empirical evidences. However, just as with other methodologies, A/B testing is not a panacea and definitely has limitations. This article aims to briefly cover these based on my learnings. Disclaimer: I have used chatGPT to polish this article however learnings are my own.

A/B testing my catch

Cheap as it may be A/B testing is still not free, A/B testing will consume resources on design, development and your own bandwidth so don’t try to learn everything through A/B testing. A/B testing is NOT a substitute for…

1. Initial research: Before you arrive at testing experiences make sure you have done all the initial research. Customer reviews, clickstream data and competition benchmarking will give you ton of insights and will help you build more conviction in the design of your experience.
2. Prioritisation: Don’t let A/B testing be about choosing best of 2 evils. Its not about which of A or B better resonate with users, but to reach to those A and B you have to cut a lot of Cs, Ds and X, Y, Zs.
3. Design language: My biggest learning has been that the coherence matters more than coolness, so don’t push your designer to build an experience to A/B test, that eventually will never go live.
4. Learning from others mistake: Don’t waste resources when the learning is clearly available else where. Of course its a definite possibility that others settings may not apply to your product, but do a careful review before concluding. You may end up tweaking your experience based on those learnings.
5. Good story telling: Even if you are able to see the results in the A/B testing you should have the ability to back the results up with connected metrics on user journey.

Now that you have sorted the above points, you must also figure following things before you start A/B testing…

1. Establish clear baseline and goals of the experiment: Before you start your experiment its important to know the “current” state, and the goal your experiment aims to achieve on this baseline. Make sure seasonality or other external factors are accounted for.
2. Experiment design: Its important to get the following factors of experiment design right a) hypothesis b) audience c) success metrics d)Guard-rail metrics e)exposure % and f) duration.
3. Infrastructure to collect data: You need robust tracking system to capture user interactions and behaviors. Preferably everything automated so you can view the impact of your experiments in (Near) real-time. Standard and custom (experiment specific) visualizations are required to tell effective story.
4. Close look at metrics: While statistical significance can help you make a go, no-go decision, don’t forget to look at confidence intervals, margin of error. Its also important to corroborate with qualitative evidence as well.
5. Documenting and Evangelizing: Before you close the experiment as either success or failure its important to document and evangelize this within your team and stakeholders. Helps you find the blindspots. Its also a good practice to convey the results with certain frequency to give more visibility to stakeholders.

A/B testing is a continuous learning process, of the experiments you launch its likely that more than 70% experiments won’t yield the results you thought they would. Its important to iterate fast and document your learnings so you can get better at it. If you have other thoughts please write them in comments or leave a dm on linkedin — will be happy to learn.