Emeli Dral - More like this: monitoring recommender systems in production | PyData Global 2023

Monitoring recommender systems in production requires a combination of online and offline metrics, evaluating performance, data quality, and observability to drive business decisions.

Key takeaways

When monitoring recommender systems in production, consider both online and offline metrics. Online metrics track immediate effects, while offline metrics analyze long-term consequences.
Start with business metrics, such as revenue, and use online proxy metrics to connect to them.
When debugging issues, analyze individual user behavior and consider data quality and data drift.
It’s essential to have a dashboard that displays both online and offline metrics, enabling quick analysis and decision-making.
When selecting metrics, consider the trade-off between accuracy and comprehensiveness.
Offline metrics can be calculated in batch mode, reducing the need for online calculations.
When monitoring recommender systems, prioritize observability and alerting systems.
Use a hierarchical approach to metric selection, focusing on business metrics first, then online quality metrics, and finally offline metrics.
For offline metrics, consider using a batch mode to reduce processing time.
Monitor both model quality and service health to ensure overall system reliability.
When debugging issues, use both online and offline metrics to understand the root cause.
Implement additional tools, such as data quality monitoring and alert systems, to support observability and decision-making.
Monitor multiple metrics, including accuracy, novelty, diversity, and serendipity, to gain a comprehensive understanding of recommender system performance.
When optimizing recommender systems, prioritize metrics that correlate with business metrics.
Consider using synthetic user data to simulate user interactions and evaluate recommender system performance.
When monitoring recommender systems, prioritize observability and alerting systems to detect issues early.
Consider using a multi-arm bandit system to balance exploration and exploitation.

Emeli Dral - More like this: monitoring recommender systems in production | PyData Global 2023

More talks