Backtesting & Calibration
Walk-forward evaluation of predicted sales, margins, and 90% bands vs realized outcomes. Metrics include MAPE / WAPE / sMAPE / RMSE / bias, coverage & Winkler interval score, Diebold-Mariano vs naive, calibration regression (α, β, R²), reliability diagram, and PIT histogram.
Slice
N observations
260
MAPE
16.2%
WAPE
15.4%
RMSE (units)
73.3
Bias (ME)
-12.21
Theil's U
0.812
90% Coverage
85.0%
Winkler score
265.5
sMAPE
15.9%
Margin MAE
1.6%
Margin bias
-0.2%
DM vs naive
t=-3.16, p=0.002
Calibration regression y = α + β·ŷ
Intercept α
3.49
Slope β
1.027
R²
0.863
Points on the 45° line ⇒ predictions match realized means within each decile. β < 1 ⇒ over-prediction at the top; α > 0 ⇒ systematic under-prediction floor.
PIT histogram (interval calibration)
A flat histogram ⇒ the 90% bands are well calibrated. U-shape ⇒ bands too narrow; hump in the middle ⇒ bands too wide.
Per-SKU accuracy
| ASIN | N | MAPE | RMSE | Bias | Coverage | Margin MAE |
|---|---|---|---|---|---|---|
| B08N5WRWNW | 52 | 17.2% | 64.9 | 1.2 | 84.6% | 1.5% |
| B07FZ8S74R | 52 | 18.4% | 27.4 | -5.8 | 80.8% | 1.6% |
| B07HFPV3H8 | 52 | 18.3% | 128.4 | -23.2 | 84.6% | 1.8% |
| B00FLYWNYQ | 52 | 13.3% | 32.4 | -13.4 | 84.6% | 1.3% |
| B0BR01ABCD | 52 | 13.7% | 66.0 | -19.8 | 90.4% | 1.7% |