Experiments for Social Network Features

Requirements Clarification & Assessment

1. Objective Understanding:

   • Feature: Evaluate the 'Close Friends' feature on Instagram Stories.
   • Goal: Increase user engagement by encouraging more story posts.
   • Hypothesis:
     • Null (H0): The feature does not affect story posting frequency.
     • Alternative (H1): The feature increases story posting frequency.

2. Network Effects Consideration:

   • Challenge: Standard A/B testing may not capture network effects due to interconnected user interactions.
   • Definition: Network effects occur when a user's behavior change affects their network, leading to potential biases.

3. Metrics Identification:

   • Primary Metric: Average number of stories posted per user per day.
   • Guardrail Metric: Time spent on Instagram per user per day.

4. Experiment Parameters:

   • Significance Level (α): Typically set at 0.05.
   • Power: 80% to detect a minimum detectable effect (MDE).
   • Sample Size: Calculated based on desired MDE and power.

5. Unit of Randomization:

   • Proposed Solution: Use community or cluster-based randomization instead of individual user randomization to minimize network effect bias.