Counting how often the favourite won is the easiest way to grade football forecasts, and the least informative. It treats a 34% pick and an 80% pick the same, and it ignores everything a forecast said about the other two results. That is why our track record leads with two scores that grade the whole forecast: the ranked probability score (RPS) and log loss, with the Brier score alongside. This guide works through all three on one match.
The example match
Take a forecast of 50% home win, 28% draw, 22% away win. We will score it three times: once as if the home side won, once for a draw and once for an away win. In each case the result is written as 1 for what happened and 0 for what did not, so a home win is (1, 0, 0).
| Result | RPS | Brier | Log loss |
|---|---|---|---|
| Home win | 0.149 | 0.377 | 0.693 |
| Draw | 0.149 | 0.817 | 1.273 |
| Away win | 0.429 | 0.937 | 1.514 |
For all three scores, lower is better and 0 would be a perfect forecast. The rest of this guide explains where each number comes from.
The Brier score: squared distance from what happened
The Brier score takes each of the three probabilities, subtracts what actually happened (1 or 0), squares the difference and adds the three together.
Brier score
Brier = (p_home − o_home)² + (p_draw − o_draw)² + (p_away − o_away)²
If the home side wins: (0.50 − 1)² + (0.28 − 0)² + (0.22 − 0)² = 0.25 + 0.0784 + 0.0484 = 0.377.
If it is a draw: (0.50 − 0)² + (0.28 − 1)² + (0.22 − 0)² = 0.25 + 0.5184 + 0.0484 = 0.817.
The Brier score runs from 0 (all the probability on the right result) to 2 (all of it on a wrong one). It punishes confident misses more than cautious ones, which is what you want. But it has one blind spot in football.
The blind spot: football results have an order
Home win, draw and away win are not three unrelated options. A draw sits between the two wins. If the home side wins, a forecast that leaned towards the draw was closer to the truth than one that leaned towards an away win.
Compare two forecasts for a match the home side won:
- Forecast A: 50% home, 30% draw, 20% away.
- Forecast B: 50% home, 20% draw, 30% away.
Both get exactly the same Brier score, 0.38, because the Brier score does not care which wrong outcome got the extra 10%. Forecast A was clearly the better one.
The ranked probability score fixes that
RPS compares running totals instead of single outcomes. First it looks at “home win” on its own, then at “home win or draw” together, and adds up the squared gaps. With three outcomes it is divided by 2, so it runs from 0 to 1.
Ranked probability score (three outcomes)
RPS = ½ × ( (P₁ − O₁)² + (P₁ + P₂ − O₁ − O₂)² ) P₁ = p_home, P₂ = p_draw · O₁, O₂ = 1 or 0 for what happened
For our example, the running totals of the forecast are 0.50 (home) and 0.78 (home or draw).
- Home win, running totals (1, 1): ½ × ((0.50 − 1)² + (0.78 − 1)²) = ½ × (0.25 + 0.0484) = 0.149.
- Draw, running totals (0, 1): ½ × ((0.50 − 0)² + (0.78 − 1)²) = 0.149.
- Away win, running totals (0, 0): ½ × (0.50² + 0.78²) = ½ × (0.25 + 0.6084) = 0.429.
Notice that this forecast scores the same for a home win and a draw. It put 78% on “not an away win”, and both of those results are on that side. An away win, the result it rated least likely and furthest from its lean, costs almost three times as much.
Back to forecasts A and B: A scores 0.145 and B scores 0.170. RPS rewards A for leaning the right way, which is why the football forecasting literature has argued for it as the main score for match results.
Log loss: how surprised the forecast was
Log loss only looks at the probability given to the result that actually happened, and takes its negative natural logarithm.
Log loss
Log loss = − ln(p of the actual result)
A home win costs −ln(0.50) = 0.693; a draw costs −ln(0.28) = 1.273; an away win costs −ln(0.22) = 1.514. The curve gets steep near zero: giving the actual result only 5% costs 3.0, and 1% costs 4.6. That makes log loss the best detector of over-confidence, so we report it next to RPS.
What is a good score?
One match tells you very little; these scores only mean something averaged over hundreds of matches. Two reference points help:
- A forecast of exactly one third for each result scores an RPS of about 0.28 when either side wins and 0.11 on a draw. Over a season it does far worse than any sensible model.
- A better benchmark is the base rate: always forecasting the league’s long-run share of home wins, draws and away wins. On the matches we have graded so far it averages an RPS of 0.2414.
Good models in the big European leagues typically average an RPS somewhere around 0.19 to 0.21, depending on the league and season. Our published forecasts currently average 0.193 over 72 graded matches. Differences in the third decimal place matter over thousands of matches but are often noise over a few hundred, which is why the track record shows a 95% interval next to every average.
How we use these scores
- Every locked forecast is graded on all three scores after full time, once two sources agree on the result.
- The track record leads with RPS and log loss, and shows Brier and the hit rate alongside.
- Our AI analysts are graded with the same scores and compared with the model only on the matches they covered. They only get a say in the published numbers if they beat the statistical model under rules fixed in advance (see the methodology).
Want the full definitions in one place? See RPS, Brier score and log loss in our glossary.
