For: DPG Media – Kiosk.nl
Project Goal
The primary goal of this project was to quickly get A/B test results of previously conducted A/B tests. An A/A test concluded multi- exposure on a lot of the historical A/B tests, so I decided to create an automated analysis to minimize time spent. This project involved cleaning and preparing raw data, addressing sample ratio mismatches, and applying Bayesian statistical methods to determine which variant performs better. The analysis was conducted at an overall level and further segmented by device type.
How it was built
The project started with data preprocessing using pandas to clean and prepare the data. This included dropping irrelevant columns, renaming columns for clarity, and identifying and removing users who had seen both variants to avoid contamination. Following data preparation, a sample ratio mismatch (SRM) test was conducted using Chi-squared to check for imbalances in the test groups. We then implemented a Bayesian A/B testing method, calculating the probability that the variation outperforms the control for various metrics (trial, subscription, and total conversions). The core of the analysis involves generating posterior distributions using Beta distributions based on conversion data for each variant. These distributions were then sampled to calculate the probability of the variation outperforming the control, as well as the expected uplift. Finally, the results are visualized using Plotly for clear interpretation, and presented in a tabular format using the ‘tabulate’ library.
Technologies used
- pandas: For data manipulation and analysis.
- scipy.stats: For statistical calculations (t-tests, beta distributions, chi-squared test).
- tabulate: For presenting data in tabular format.
- plotly: For creating interactive visualizations.
- numpy: For numerical operations.