A forecast must allow an unambiguous calculation
“Business will have a difficult time next year” cannot be checked cleanly: the statement has no threshold, date or source for the outcome. A useful educational forecast is: “With a probability of 70%, the published annual revenue of a hypothetical company will exceed the prior-year figure on the same perimeter.” Then state the currency, report, resolution date and treatment of restatements. The important practice is how the statement is recorded, not a prediction about a particular security.
Forecast-evaluation methodology distinguishes checking one forecast, comparing several approaches and combining forecasts. These tasks require suitable metrics: the error of a point estimate and the quality of an event probability are not the same thing. Research on professional forecasters also emphasizes that a single point estimate does not by itself communicate the author’s uncertainty.[1][2]
A journal that is difficult to rewrite after the fact
Each record needs a creation date, wording, probability, basis, cutoff date and rule for determining the outcome. Do not delete the initial record when updating it. New information may reasonably change the probability, but that is a new version with its own date. If a forecast is cancelled because the reporting perimeter changed, preserve the reason for cancellation instead of silently removing an inconvenient observation.
An empty table is often more useful than a complicated model. Separate an assumption about what will happen from a decision about what to do: a high probability of revenue growth does not mean that a share price is attractive. Probability can be checked independently of investment performance. The question of price and risk belongs to [expected-value decisions](/guides/expected-value-decisions/), and is not settled by confidence in one event.
A small Brier-score calculation
For a binary event, take the squared difference between the probability and the outcome: one means that the event occurred and zero means that it did not. The average of these squares is the Brier score; a lower value means a smaller squared error. In four fictional events, the probabilities are 0.8, 0.6, 0.3 and 0.1, while the outcomes are 1, 0, 1 and 0. The errors are 0.04, 0.36, 0.49 and 0.01. Their average is 0.225.
A constant 50% forecast on the same outcomes would give 0.25. There is a difference in this toy table, but four observations do not establish a durable advantage. In particular, they should not be treated as independent if all events concern one macroeconomic shock. Before comparing numbers, verify that the event lists are the same: an author who selected only easy questions was solving a different task.
Calibration and useful discrimination
Calibration means that stated probabilities correspond to observed frequencies in a comparable group of forecasts. For example, fourteen of twenty events stated at 70% occurred. That is consistent with a 70% level, but it does not prove accuracy in the next series: the group is small and may be dependent. Uncertainty intervals and grouping rules defined in advance are needed.
A forecast can be reasonably calibrated yet not very informative if it always reports the base rate. Therefore also test whether the approach distinguishes events that occur more often from those that occur less often. Do not artificially increase confidence for a more dramatic separation: an overconfident wrong probability receives a large penalty. Combining several opinions also does not guarantee improvement, especially when they use the same underlying model.
Analyze the error instead of holding a forecaster contest
After the question is resolved, ask whether the data, the causal model or the level of confidence was wrong. Separately note whether the event itself was formulated correctly. Do not assess a probability from one outcome: an event with a 10% chance should sometimes occur. However, systematically turning “unlikely” events into frequent ones is a reason to revisit the model and the quality of the underlying information.
In primary research, look for the sample, loss function and updating rules, not only an accuracy ranking. For your own observations, fix a simple baseline model in advance and keep the full journal. The important result of this practice is a more honest description of uncertainty, not a licence to forecast prices with guaranteed profit. Verification of the reliability of source publications continues in the [financial source card](/guides/source-verification-finance/).
Check before starting the journal
Run a trial resolution of the question: could two readers independently identify the same outcome from the specified document? If one chooses the initial publication and the other a later correction, the instruction is insufficient. Clarify it before the result becomes known. This work may seem less interesting than assigning a probability, but it is what makes later evaluation meaningful.
Record the base rate separately and state how it was obtained. It must be available on the forecast date and relate to similar cases, rather than being calculated from a future sample. Only then can you compare your estimate with a simple benchmark. The definitions of [calibration](/glossary/kalibrovka-prognoza/) and the [Brier score](/glossary/ocenka-brayera/) answer different questions: matching frequencies and measuring squared error. A good journal preserves both aspects without substituting either for causal analysis.
Sources and boundaries of the check
[1] [Forecast Evaluation and Combination](https://www.nber.org/papers/t0192). Comparison of forecasts, including probabilistic forecasts; the educational calculations here are our own.
[2] [Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters](https://www.nber.org/papers/w11978). A point estimate does not disclose uncertainty.