What to record
Calibration checks whether probabilistic assessments are consistent with subsequent frequencies. If, among a set of comparable events assigned 70% in advance, about seven out of ten occur, that is consistent with the probability. But a small group does not provide a precise test by itself.[1]
Record the initial probability, date, unambiguous definition of the event and the source used to resolve the question. Keep updates separately rather than replacing an unsuccessful early forecast with a later corrected version.
Do not confuse it with informativeness
A constant statement of the base rate can be calibrated while doing almost nothing to distinguish events. Therefore, alongside calibration, it is useful to assess whether higher probabilities are assigned to outcomes that actually occur more often.
Twenty dependent questions about one crisis are not twenty independent observations. Disclose errors, uncertainty intervals and grouping rules. The workflow for a forecast log is described in [probabilistic forecast evaluation](/guides/forecast-calibration/); calibration is not a promise of a profitable investment decision.
Primary material
[1] Diebold, Lopez, Forecast Evaluation and Combination; forecast-evaluation methods. [Forecast Evaluation and Combination](https://www.nber.org/papers/t0192)