ETA Range Display

Solution & Explanation

Experiment Design

1. Objective Clarification:

   • Goal: Determine if displaying an ETA range (e.g., 3-7 minutes) instead of a precise ETA (e.g., 5 minutes) improves user engagement, reduces ride cancellations, and increases conversion rates.
   • Key Metrics:
     • Primary Metric: Conversion rate (number of rides booked/number of rides searched).
     • Secondary Metrics: Ride cancellation rate, app downtime, and user satisfaction based on post-ride surveys.
     • Guardrail Metrics: Weekly active users (WAU), daily active users (DAU), and app uninstallation rate.

2. Hypothesis Formulation:

   • Null Hypothesis (H0): Displaying an ETA range has no effect on conversion rates.
   • Alternative Hypothesis (H1): Displaying an ETA range increases conversion rates.

3. Experimental Setup:

   • Randomization: Randomly assign users into control and treatment groups using user IDs to ensure unbiased results.
   • Sample Size Calculation: Determine the minimum sample size needed using power analysis, setting alpha (0.05) and beta (0.2) levels, and estimating the minimum detectable effect (MDE) based on historical data.
   • Experiment Duration: Calculate based on sample size and average daily rides, ensuring at least a week to account for weekly seasonality and avoiding major holidays or events.

4. Implementation:

   • Control Group: Receives the current precise ETA.
   • Treatment Group: Receives the ETA range.
   • Rollout Strategy: Gradual rollout (e.g., 20% of users initially) to monitor system performance and handle unexpected issues before full deployment.

5. Data Collection:

   • Monitor key and guardrail metrics continuously throughout the experiment.
   • Ensure data integrity with sanity checks, e.g., checking for sample ratio mismatches and maintaining consistent app performance across groups.

Analysis & Evaluation

1. Statistical Testing:

   • Use a two-sample z-test to compare conversion rates between control and treatment groups.
   • Evaluate the p-value against the significance level (0.05) to determine statistical significance.

2. Result Interpretation:

   • Statistical Significance: If p-value < 0.05, reject H0 and conclude that the ETA range has a significant impact on conversion rates.
   • Practical Significance: Assess if the observed effect size is large enough to justify implementation from a business perspective.

3. Additional Considerations:

   • User Segmentation: Analyze results across different user demographics (e.g., new vs. returning users) to identify any segment-specific effects.
   • Feedback Loop: Gather qualitative feedback from users to understand perceptions of the ETA range and potential areas for improvement.

4. Decision Making:

   • If results show both statistical and practical significance, recommend rolling out the ETA range feature to all users.
   • Consider potential trade-offs, such as increased complexity in ETA calculations, and balance against the benefits observed.

5. Post-Experiment Monitoring:

   • Continuously track key metrics to ensure sustained performance and user satisfaction post-launch.
   • Plan for iterative improvements based on ongoing data analysis and user feedback.