Loading the track record…
Loading the track record…
How good are the probabilities? Every number here is a proper score on forecasts made with only the information available before the session, compared with simple baselines. The three groups below mean different things, so they are never mixed.
Encouraging on past races, not yet proven on live ones.
Source: the 2026 test write-up (docs/results/2026-test.md, 14 Nov 2026). That test was of the setup version 1 was built from; version 1 adds practice pace from car data, which improved podium odds on the same races and left win odds about the same (docs/results/2026-telemetry.md). “Clearly” means the 95% interval of the difference excludes zero. The tables below give each version's scores against the simple baselines (lower is better).
For every driver and every race we ask: what probability did the forecast give to what actually happened? Log loss is the average of −log(that probability). A confident correct call scores near 0; a confident wrong call is punished hard. Lower is better.
“Win” scores P(win) for every driver (winner or not); “podium” scores P(top 3). Brier score is the mean squared error of the same probabilities (also lower is better). ECE (expected calibration error) checks whether events given 30% happen about 30% of the time.
RPS (ranked probability score) looks at the whole predicted finishing order, not just win or podium: it compares the forecast chance of finishing in each position with where the driver actually finished, so being close (forecast P3, finished P4) costs less than being far off. 0 is perfect, 1 is the worst possible; lower is better.
Forecasts made and committed to the ledger before each session, then scored after it. This is the real evidence, and it starts small: 0 races scored so far. Below 10 scored races the numbers are descriptive only: no verdict either way.
Nothing scored yet
A walk-forward replay of 2026 rounds 1–20 with acceptance criteria fixed before any 2026 result was looked at. The model never trained or tuned on these rounds, but they have since been scored twice, so they are no longer a fresh test set.
Scores version 1 of the model (live-v1), not version 2, which makes today's forecasts.
run 20261114-074930-phase5-telemetry-2026 · config telemetry (5df53b0fe48e) · seasons 2026
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 20 | 0.0821 (best) | 0.1958 (best) | 0.0243 (best) | 0.0634 (best) | 0.0260 (best) | 0.1115 (best) |
| Starting grid | 20 | 0.1074 | 0.2247 | 0.0278 | 0.0656 | 0.0318 | 0.1338 |
| Championship order | 20 | 0.1267 | 0.2633 | 0.0368 | 0.0875 | 0.0464 | 0.1386 |
| Same as last race | 20 | 0.1355 | 0.3077 | 0.0371 | 0.0870 | 0.0283 | 0.1604 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.045 (35% lower) | −0.068 (26% lower) | −0.020 | −0.027 (20% lower) |
| Starting grid | −0.025 (24% lower) | −0.029 (13% lower) | −0.006 | −0.022 (17% lower) |
| Same as last race | −0.053 (39% lower) | −0.112 (36% lower) | −0.002 | −0.049 (31% lower) |
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 20 | 0.1165 (best) | 0.2301 (best) | 0.0361 (best) | 0.0762 (best) | 0.0324 | 0.1162 (best) |
| Championship order | 20 | 0.1267 | 0.2633 | 0.0368 | 0.0875 | 0.0464 | 0.1386 |
| Same as last race | 20 | 0.1355 | 0.3077 | 0.0371 | 0.0870 | 0.0283 (best) | 0.1604 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.010 (8% lower) | −0.033 (13% lower) | −0.014 | −0.022 (16% lower) |
| Same as last race | −0.019 (14% lower) | −0.078 (25% lower) | +0.004 | −0.044 (28% lower) |
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 6 | 0.0545 (best) | 0.1737 (best) | 0.0149 (best) | 0.0583 | 0.0358 (best) | 0.1011 |
| Starting grid | 6 | 0.0651 | 0.1744 | 0.0190 | 0.0571 (best) | 0.0438 | 0.0971 (best) |
| Championship order | 6 | 0.1280 | 0.2377 | 0.0371 | 0.0825 | 0.0873 | 0.1092 |
| Same as last race | 6 | 0.1385 | 0.2422 | 0.0387 | 0.0770 | 0.0740 | 0.1092 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.073 (57% lower) | −0.064 (27% lower) | −0.052 | −0.008 (7% lower) |
| Starting grid | −0.011 (16% lower) | −0.001 (about equal) | −0.008 | +0.004 (4% higher) |
| Same as last race | −0.084 (61% lower) | −0.068 (28% lower) | −0.038 | −0.008 (7% lower) |
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 6 | 0.1084 (best) | 0.2355 (best) | 0.0311 (best) | 0.0846 | 0.0685 (best) | 0.1116 |
| Championship order | 6 | 0.1280 | 0.2377 | 0.0371 | 0.0825 | 0.0873 | 0.1092 (best) |
| Same as last race | 6 | 0.1385 | 0.2422 | 0.0387 | 0.0770 (best) | 0.0740 | 0.1092 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.020 (15% lower) | −0.002 (1% lower) | −0.019 | +0.002 (2% higher) |
| Same as last race | −0.030 (22% lower) | −0.007 (3% lower) | −0.005 | +0.002 (2% higher) |
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 20 | 0.1358 | 0.2425 (best) | 0.0375 (best) | 0.0813 (best) | 0.0329 | 0.0756 (best) |
| Championship order | 20 | 0.1343 (best) | 0.2651 | 0.0386 | 0.0866 | 0.0269 | 0.0938 |
| Same as last race | 20 | 0.1506 | 0.3052 | 0.0399 | 0.0917 | 0.0268 (best) | 0.1354 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | +0.002 (1% higher) | −0.023 (9% lower) | +0.006 | −0.018 (19% lower) |
| Same as last race | −0.015 (10% lower) | −0.063 (21% lower) | +0.006 | −0.060 (44% lower) |
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 6 | 0.1294 (best) | 0.2299 (best) | 0.0375 (best) | 0.0741 (best) | 0.0682 | 0.0675 (best) |
| Championship order | 6 | 0.1403 | 0.2398 | 0.0410 | 0.0809 | 0.0560 (best) | 0.0917 |
| Same as last race | 6 | 0.1472 | 0.2652 | 0.0409 | 0.0778 | 0.0650 | 0.1059 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.011 (8% lower) | −0.010 (4% lower) | +0.012 | −0.024 (26% lower) |
| Same as last race | −0.018 (12% lower) | −0.035 (13% lower) | +0.003 | −0.038 (36% lower) |
Scores version 2 of the model (live-v2), the version making today's forecasts.
run 20261115-222738-promo-recheck-driver-drift · config driver-drift (28a4508f891c) · seasons 2023–2025
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 70 | 0.0860 (best) | 0.2069 (best) | 0.0253 (best) | 0.0639 (best) | 0.0093 (best) | 0.1109 (best) |
| Starting grid | 70 | 0.1148 | 0.2399 | 0.0331 | 0.0673 | 0.0102 | 0.1337 |
| Championship order | 70 | 0.1248 | 0.2725 | 0.0368 | 0.0852 | 0.0206 | 0.1490 |
| Same as last race | 70 | 0.1442 | 0.3211 | 0.0399 | 0.0943 | 0.0241 | 0.1644 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.039 (31% lower) | −0.066 (24% lower) | −0.011 | −0.038 (26% lower) |
| Starting grid | −0.029 (25% lower) | −0.033 (14% lower) | −0.001 | −0.023 (17% lower) |
| Same as last race | −0.058 (40% lower) | −0.114 (36% lower) | −0.015 | −0.054 (33% lower) |
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 70 | 0.1075 (best) | 0.2542 (best) | 0.0316 (best) | 0.0805 (best) | 0.0234 | 0.1230 (best) |
| Championship order | 70 | 0.1248 | 0.2731 | 0.0368 | 0.0853 | 0.0206 (best) | 0.1490 |
| Same as last race | 70 | 0.1442 | 0.3211 | 0.0399 | 0.0943 | 0.0241 | 0.1644 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.017 (14% lower) | −0.019 (7% lower) | +0.003 | −0.026 (17% lower) |
| Same as last race | −0.037 (25% lower) | −0.067 (21% lower) | −0.001 | −0.041 (25% lower) |
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 18 | 0.0858 (best) | 0.1705 | 0.0254 (best) | 0.0500 | 0.0248 | 0.1069 (best) |
| Starting grid | 18 | 0.0928 | 0.1570 (best) | 0.0289 | 0.0470 (best) | 0.0209 (best) | 0.1134 |
| Championship order | 18 | 0.1302 | 0.2770 | 0.0367 | 0.0844 | 0.0399 | 0.1489 |
| Same as last race | 18 | 0.1563 | 0.3442 | 0.0414 | 0.0992 | 0.0416 | 0.1646 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.044 (34% lower) | −0.107 (38% lower) | −0.015 | −0.042 (28% lower) |
| Starting grid | −0.007 (8% lower) | +0.014 (9% higher) | +0.004 | −0.006 (6% lower) |
| Same as last race | −0.071 (45% lower) | −0.174 (50% lower) | −0.017 | −0.058 (35% lower) |
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 18 | 0.1184 (best) | 0.2468 (best) | 0.0329 (best) | 0.0758 (best) | 0.0392 (best) | 0.1250 (best) |
| Championship order | 18 | 0.1302 | 0.2770 | 0.0367 | 0.0844 | 0.0399 | 0.1489 |
| Same as last race | 18 | 0.1563 | 0.3442 | 0.0414 | 0.0992 | 0.0416 | 0.1646 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.012 (9% lower) | −0.030 (11% lower) | −0.001 | −0.024 (16% lower) |
| Same as last race | −0.038 (24% lower) | −0.097 (28% lower) | −0.002 | −0.040 (24% lower) |
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 70 | 0.1175 (best) | 0.2567 (best) | 0.0356 (best) | 0.0805 (best) | 0.0268 | 0.1180 (best) |
| Championship order | 70 | 0.1314 | 0.2840 | 0.0383 | 0.0885 | 0.0276 | 0.1411 |
| Same as last race | 70 | 0.1600 | 0.3291 | 0.0413 | 0.0963 | 0.0226 (best) | 0.1519 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.014 (11% lower) | −0.027 (10% lower) | −0.001 | −0.023 (16% lower) |
| Same as last race | −0.043 (27% lower) | −0.072 (22% lower) | +0.004 | −0.034 (22% lower) |
| Model | Rounds | Win log loss | Podium log loss | Brier win | Brier podium | ECE podium | RPS |
|---|---|---|---|---|---|---|---|
| Model (ratings + race simulation) | 18 | 0.1488 (best) | 0.2402 (best) | 0.0405 (best) | 0.0772 (best) | 0.0324 (best) | 0.1090 (best) |
| Championship order | 18 | 0.1527 | 0.2764 | 0.0409 | 0.0853 | 0.0514 | 0.1319 |
| Same as last race | 18 | 0.1750 | 0.3481 | 0.0437 | 0.1028 | 0.0373 | 0.1456 |
| Against | Δ win log loss | Δ podium log loss | Δ ECE podium | Δ RPS |
|---|---|---|---|---|
| Championship order | −0.004 (3% lower) | −0.036 (13% lower) | −0.019 | −0.023 (17% lower) |
| Same as last race | −0.026 (15% lower) | −0.108 (31% lower) | −0.005 | −0.037 (25% lower) |
Does “30%” happen about 30% of the time? Each forecast probability is put in one of ten bins (0–10%, 10–20%, …); each dot is a bin, at the bin's average forecast (across) and how often the event actually happened (up). On the dashed diagonal the forecasts are perfectly calibrated; above it, events happened more often than forecast (too cautious); below it, less often (overconfident). Bigger dots hold more forecasts; small ones are noisy.
Model only, pooled over every (round, driver), the same rows as the scores above. Descriptive: no model decision is taken from these bins. The development group was tuned on, so expect it to look better.
Held-out test
| Bin | n | Mean forecast | Happened |
|---|---|---|---|
| 0%–10% | 396 | 1.0% | 0.8% |
| 10%–20% | 16 | 14.6% | 6.3% |
| 20%–30% | 4 | 24.3% | 50.0% |
| 30%–40% | 5 | 35.3% | 60.0% |
| 40%–50% | 6 | 43.8% | 33.3% |
| 50%–60% | 4 | 54.8% | 50.0% |
| 60%–70% | 7 | 64.9% | 71.4% |
| 70%–80% | 1 | 70.5% | 100% |
| 80%–90% | 1 | 82.4% | 100% |
| 90%–100% | 0 | — | — |
Development (tuned on)
| Bin | n | Mean forecast | Happened |
|---|---|---|---|
| 0%–10% | 1255 | 0.9% | 0.8% |
| 10%–20% | 47 | 14.0% | 17.0% |
| 20%–30% | 20 | 25.1% | 25.0% |
| 30%–40% | 10 | 35.8% | 20.0% |
| 40%–50% | 13 | 45.2% | 53.8% |
| 50%–60% | 8 | 56.4% | 50.0% |
| 60%–70% | 11 | 65.5% | 63.6% |
| 70%–80% | 23 | 74.5% | 73.9% |
| 80%–90% | 11 | 83.3% | 90.9% |
| 90%–100% | 0 | — | — |
Live