Malte Tichy - Paradoxes in model training and evaluation under constraints | PyData Global 2023

When dealing with constrained demand (e.g., limited stock capacity), using sales data as an approximation for unconstrained demand introduces systematic bias in model training and evaluation

The relationship between unconstrained demand forecasts and actual sales depends heavily on the capacity constraints - mean sales will always be smaller than mean unconstrained demand when capacity limits exist

Selecting/filtering data based on whether capacity was hit creates additional bias by only looking at extreme outcomes that aren’t representative of the full demand distribution

Proper evaluation requires:

• Looking at predictions grouped by forecast probability rather than outcomes
• Using forward-looking metrics instead of backward-looking analysis
• Considering the full probability distribution rather than just point forecasts

Statistical techniques like censored/truncated distributions should be used to properly model demand under constraints rather than simplifying to sales data

The optimal capacity setting involves balancing waste against stockouts - maximizing data science effectiveness often conflicts with operational constraints

Evaluation should be done by bucketing predictions into similar capacity hit probabilities and comparing actual vs predicted hit rates

Real-world complications like uncertain stock levels, theft, and other factors make constraint modeling more complex but the fundamental statistical principles still apply

Commercial packages often support proper constraint modeling but open source options for censored/truncated distribution analysis are currently limited

Using shortcuts like equating sales to demand leads to systematically biased models that underforecast and increase stockouts over time