N. Alagoz, A. Mahfoudhi-Maximizing marketplace experimentation: switchback design for small samples

Switchback design helps increase precision in experiments with small sample sizes and subtle effects by alternating treatment between groups over time periods

Most experiments (90%) show no significant impact, making it critical to test ideas quickly and efficiently to find what actually works

Key advantages of switchback over traditional A/B testing:

• Reduces cross-sectional interference between treatment/control groups
• Helps handle autocorrelation in time series data
• Can increase precision by 2-4x compared to A/B testing
• Allows shorter experiment runtime for same statistical power

Important considerations when implementing switchback:

• Length of treatment periods
• Frequency of switches between conditions
• Cost of switching treatments
• Potential temporal interference effects
• Carryover effects between periods

Particularly valuable for:

• Two-sided marketplaces (e.g., Uber, DoorDash)
• Industries with high stakes but low sample sizes (airlines)
• Contexts where actions of one group can affect others
• Situations requiring detection of small effect sizes (1-6% changes)

Design phase is critical - requires careful advance planning of:

• Population splits
• Duration of experiment
• Data collection requirements
• Treatment period lengths
• Analysis approach