Danial Senejohnny - Causal Effect Estimation in Practice: Lessons Learned from E-commerce & Banking

When dealing with causal inference in practice, randomized A/B testing is considered the gold standard but often not feasible in real business settings

Key challenges in causal effect estimation:

• Lack of proper randomization
• Difficulty finding comparable control groups
• No simultaneous contact of treatment/control groups
• Time-varying effects and conversions
• Multiple confounding variables

For creating control groups:

• Use simple rule-based methods initially
• Match control group characteristics to treatment group
• Ensure overlap in covariate distributions (positivity)
• Validate with domain experts

Important assumptions to verify:

• Exchangeability between groups
• No interference between groups
• Consistency in treatment application
• Proper handling of confounding variables

Practical recommendations:

• Start with simpler KPIs before complex ones
• Account for appropriate timeframes for different metrics
• Use multiple methods to validate results
• Consider survival analysis for time-varying conversions
• Avoid conditioning on collider variables

For evaluation:

• Compare results across multiple causal inference methods
• Validate findings with domain experts
• Acknowledge that true unbiased effect is unknown in practice
• Consider both long-term and short-term KPI impacts

Specific considerations for marketing campaigns:

• Account for email availability and permissions
• Consider product category similarities
• Watch for interference from other campaigns
• Allow sufficient time for conversion effects