An attempt is more than a new algorithm
Changing the period, asset set, threshold, trading frequency or exclusion rule also creates a new attempt. Once the result has been seen, changing the method uses historical information. Renaming the strategy does not erase the earlier search. Keep a journal of rejected variants and hypotheses; without it, one published model appears preselected even when it was the result of search.
Why many tests find accidental winners
In a fictional statistical problem, each independent test has a 5% false-positive probability. For 20 independent tests, the probability of at least one false positive is 1 − 0.95²⁰, approximately 64.15%; for 100 it is approximately 99.41%. These are not probabilities that a particular trading system will fail: independence and error rate are deliberately stipulated. Real alternatives may be dependent, so the percentages cannot be transferred without a matching model and data.
How a test period becomes contaminated
Repeatedly viewing a holdout period and changing the strategy makes that period part of tuning. Randomly shuffling time observations can also break dependencies and information order. Time order matters in financial series, but one calendar split does not solve every problem. The article’s comparison procedure has its own assumptions and is not a certificate of future profit.[108](#source-cv-backtest)
Test fragility and execution realism
Inspect neighboring parameters. If only one threshold works and a small change destroys the result, investigate robustness. Vary realistic costs, execution delay, available instruments and start date. Do not hide deterioration by selecting the most convenient report. Economic explanation helps but does not replace statistical testing; write the mechanism in advance and name an observation that could disprove it. SEC materials distinguish actual post-selection results from hypothetical history.[27](#source-market-performance)
A report should contain the whole search, selection criterion, untouched test, costs and limitations. Lost search history is material uncertainty. Lack of visible overfitting does not guarantee future return because market regimes can change.
Check the success metric itself
If a strategy first loses on return, then is judged by drawdown, individual years and finally a specially chosen ratio, the victory criterion changed. Record the metric and constraints before comparison and preserve reasons for later changes. More indicators do not solve selection: each metric can become another angle for choosing a favorable result.
Rule families and explicit specifications
The following are artificial research specifications, not trading advice. Only closes through time t are available; an order can occur no earlier than the next observation. A threshold and exit are fixed before the control sample. The trend rule uses simple averages of the last 3 and 5 closes; for 10, 9, 8, 9, 10, 11, the fifth averages are 9 and 9.2 and the sixth 10 and 9.4. The event is known at the sixth observation, not at the future peak.
A range rule uses prior 20 closes, excluding current: L=min, U=max, z=(P−L)/(U−L), with U=L unclassified. z≤0.1 is a fictional event; with L=80, U=100, P=82, z=0.1. Measure the next five-day change, not recovery until success. A triple screen must define weekly, daily and hourly filters, forbid an incomplete current week and compare with both inactivity and the first screen alone. Missing hourly data means it cannot honestly be called tested.
Position lifecycle, spreads and portfolio limits
Define states—out, awaiting execution, in position, awaiting exit and re-entry quarantine—plus order lifetime, partial fills and event priority. A sample exit is the first of five observations, a 3% fall or a 4% rise; re-entry requires a new crossing and two full observations out. Thresholds do not guarantee execution: a gap can produce a larger loss, and if both thresholds occur in one bar show both possible outcomes. Ten similar signals are not ten independent risks; do not increase size after a loss.
For two fictional contracts S=F₁−hF₂. With multiplier 10, h=1, prices 100 and 96 moving to 103 and 98, ΔS=1 and gross pair result is 10 monetary units. Both legs incur costs and asynchronous execution adds risk; with different multipliers, quote subtraction is not a monetary result. Test fundamental, seasonal and technical variants as separate searches, preserve every failure and do not treat one illustrative trade as validation.