Market Opening Notifications

Solution & Explanation

Identifying Statistically Significant Metrics

To determine which metrics from the experiment are statistically significant, we need to evaluate the p-values provided for each metric. A common threshold for statistical significance is a p-value of less than 0.05, indicating that there is less than a 5% chance that the observed effect is due to random variation.

However, given the multiple metrics tested, we should apply a correction for multiple comparisons to control the overall Type I error rate. A conservative approach is the Bonferroni correction, which divides the significance threshold by the number of tests (12 metrics in this case), resulting in a corrected threshold of 0.00417 (0.05/12).

Metrics with p-values < 0.00417 (Bonferroni-corrected significance level):

• Daily_Sessions/User: p-value = 0.0022

Metrics with p-values < 0.05 (standard significance level):

• D1_Revenue/User: p-value = 0.0475
• D1_Retention: p-value = 0.0495
• D14_TimeSpent/ActiveUser: p-value = 0.0433

Criteria for Significance

• Bonferroni Correction: To adjust for multiple comparisons, ensuring the overall significance level remains at 0.05.
• Standard Threshold: A p-value less than 0.05 is often used in practice when considering individual tests.

Recommendation for Robinhood

The decision to implement push notifications across Robinhood's entire user base should consider the following:

1. Objective Alignment:

   • If the primary goal is to increase daily user engagement, as indicated by the significant increase in Daily_Sessions/User, then rolling out the notifications could be beneficial.
   • If the focus is on boosting revenue or retention, the evidence is weaker, with p-values only slightly below the 0.05 threshold.

2. Cost-Benefit Analysis:

   • Consider the cost of sending notifications against the potential benefits in user engagement and revenue.

3. User Experience:

   • Assess whether users find the notifications beneficial or intrusive, possibly affecting user satisfaction.

4. Further Testing:

   • Conduct additional experiments focusing on specific user segments (e.g., new vs. seasoned users) to understand differential impacts and refine the strategy.

5. Monitoring Post-Implementation:

   • If implemented, continue monitoring key metrics to ensure sustained benefits and identify any adverse effects.

In conclusion, while the experiment shows promising results in increasing user engagement, Robinhood should carefully weigh the goals and potential impacts before deciding to fully implement the notifications. Further testing and analysis are advisable to optimize the strategy and ensure alignment with business objectives.