How a random winner appears
Backtest overfitting occurs when parameter, period or rule choices are tailored to features of one historical record. Among many attempts, a random winner can look convincing. The original paper by Bailey and co-authors examines the probability of this error and the limits of an ordinary holdout test in investment problems.[108](#source-cv-backtest)
An illustrative probability model
In a separate educational statistical model, twenty independent tests with a five-percent probability of false success each produce about a 64.15% probability of at least one false success. This is not the probability that a real strategy will fail: its attempts may be dependent and may have a different error structure.
How to test more honestly
For a review, retain all variants, the selection criterion, the order in which data were viewed and realistic costs. Retuning after seeing the holdout segment turns that segment into part of the training data. Stability across neighboring parameters is useful but does not guarantee future returns. An economic explanation can also be invented after a result is found, so it is better to record hypotheses and constraints in advance.