# Who grades the forecast?

A forecast's accuracy only means something when it is measured on data the forecast never saw, by something other than the thing that made it.

Published 2026-10-20 by Brokyl.

Every forecasting tool will tell you how accurate it is. The more useful question is: accurate on what, and according to whom? A forecast that is scored on the same history it was built from, by the same system that built it, will almost always look better than it really is.

## Fitting the past is easy

Give a model enough freedom and it can trace any history almost perfectly. It learns the trend and the seasonality, and then it keeps going and learns the noise too: the one-off promotion, the late delivery, the month a store was closed. None of that repeats, so none of it helps the next forecast. Worse, it actively hurts it.

Try it below. Each step to the right gives the curve more freedom to bend.

*Figure: Monthly demand over four years. A curve is fitted to the first 36 months (filled dots) and extended over the last 12 (hollow dots), which it never saw. As the model gets more flexible, the error on history keeps falling while the error on the next 12 months rises sharply.*

At low flexibility the curve misses some of the history but follows demand into the future reasonably well. Turn it up and the error on history shrinks month after month, which is exactly what a self-reported accuracy number would celebrate, while the forecast for the months that actually matter falls apart.

## Score on what the model never saw

The fix is old and simple: hold some data back. Build the forecast on the history up to a cut-off, then measure it on the months after. This is called backtesting, and it is the only accuracy number worth reporting before real values arrive. Done properly, it is repeated over several cut-offs so one lucky window does not decide anything.

Once a forecast is live, the real test starts: the actuals come in, and the forecast either held up or it did not.

## Don't let the forecaster grade itself

This matters even more with AI. A language model asked how well its own forecast did will happily produce a confident, plausible answer. That is not measurement. In Brokyl, when real values arrive, a deterministic script, not the AI, scores every forecast version against them. The agent then explains those numbers in plain language. The thing that made the forecast never grades its own work.

A few questions worth asking of any forecast, whoever or whatever made it:

- Was the accuracy measured on data the model was trained on, or on data it never saw?
- Over how many periods and cut-offs, not just the best one?
- Who or what computed the score, and can you rerun it yourself?
- Is every version of the forecast kept, so you can compare what was predicted with what happened?

If the answers are clear, the accuracy number is worth something. If they are not, it is a description of the past dressed up as a promise about the future.