Non-Normal Data in AB Testing

Solution & Explanation

Understanding the Problem

In this scenario, you are dealing with non-normal data distributions in an AB test for Uber Fleet. The challenge is to determine which variant performs better despite the data not fitting a normal distribution. Traditional parametric tests like the t-test, which assume normality, are not suitable here.

Approach

To address this, you can employ several techniques that do not rely on the assumption of normality:

1. Non-Parametric Tests

   • Mann-Whitney U Test: Also known as the Wilcoxon rank-sum test, this is a non-parametric test used to compare differences between two independent groups. It is ideal for small sample sizes and non-normal distributions, as it evaluates whether one group tends to have higher values than the other.
   • Kolmogorov-Smirnov Test: This test compares the cumulative distributions of two data sets to determine if they differ significantly. It is another non-parametric option that does not assume normality.

2. Bootstrapping

   • Resampling Technique: Bootstrapping involves repeatedly sampling from the data with replacement. By calculating the desired statistic (e.g., mean, median) for each sample, you can build a distribution of the statistic. This method helps estimate confidence intervals and test hypotheses without relying on normality.
   • Convergence to Normality: With enough resamples, the sampling distribution of the statistic will approximate normality, allowing for more traditional analysis methods.

3. Effect Size Measurement

   • Cohen's d or Rank-Biserial Correlation: These metrics measure the magnitude of differences between groups. Even if statistical significance is not achieved, a large effect size may indicate practical significance.

4. Gather More Data

   • Increasing Sample Size: If feasible, increasing the sample size can help achieve a more reliable estimate of the true effect. Larger samples tend to stabilize variance and can sometimes approximate normality due to the Central Limit Theorem.

5. Log Transformation

   • Transforming Data: Applying a log transformation can sometimes normalize skewed data, allowing for parametric tests. Ensure to check the transformed data for normality before proceeding.

Criteria for Determining the Winning Variant

• Statistical Significance: Use non-parametric tests to determine if differences between variants are statistically significant.
• Practical Significance: Evaluate the effect size to assess the real-world impact of differences between variants.
• Confidence Intervals: Analyze bootstrapped confidence intervals. Minimal overlap between intervals for the two variants suggests one may be superior.
• Holistic Evaluation: Consider additional business metrics or qualitative insights that may affect the decision beyond statistical analysis.

By combining these approaches, you can make a robust decision on which variant performs better, even in the presence of non-normal data distributions.