The Gold Strategy You Kept Was One of Many You Tested

Count the choices that produced the winner

Imagine testing three lookback lengths, four exit distances and two session filters on the same gold history. That gives 24 candidate rules. If you select the strongest return from that grid, the displayed result answers which candidate ranked first in that sample. Its evidential weight depends partly on the 23 chances you gave other candidates to win. Dropping losing configurations from the experiment log conceals that choice.

For an illustration with independent tests, assume every candidate has a 5% probability of a false positive under its null model. Across 24 such tests, the chance of at least one false positive is 1 minus 0.95 raised to the 24th power, approximately 71%. Trading rules often share observations, features and trades. That dependence breaks the independence assumption, so 71% is an arithmetic example, not an estimate of the false discovery rate for any CyGuru or CY66 rule.

Notice when the holdout becomes part of development

Suppose a rule was chosen on two years of observations and a third year was set aside. The third-year score is disappointing. You inspect its losses, remove one session and improve the result. The change might be economically sensible, but that year supplied information for choosing it. Calling the revised score out-of-sample would conceal the role those observations played.

Bailey, Borwein, López de Prado and Zhu study selection across investment backtests and develop a method for estimating backtest overfitting. Their work helps explain why holding back one sample gives limited protection if researchers repeatedly use the score to change the strategy. I would keep the original third-year result and date the revised rule as a new candidate. Its next test needs observations that did not inform the revision.

Source notes: [1] Bailey, Borwein, López de Prado and Zhu — The Probability of Backtest Overfitting

Create a register before opening the results

For each candidate, record a version identifier, the precise decision rule, available data dates, assumed costs and the score used for selection. Keep candidates rejected on sight once their results were viewed. Record the number of discretionary rounds as well as the size of any initial grid. A researcher who tried nine further adjustments after the first 24 has a larger search history than the spreadsheet tab labelled “24.”

Before the final evaluation, freeze how ties will be resolved, define a minimum observation count and specify what will count as an inconclusive result. Check for training trades whose outcomes extend into the evaluation period. Cut or isolate these overlaps according to a rule set in advance. Otherwise information from the evaluation window can enter through the outcome labels even if the input timestamps look clean.

Publish the denominator next to the selected result

My research note would name the full candidate count, show the distribution of development scores and state how often the evaluation period was inspected. It would report the untouched result, its sample size and the friction assumptions. If the last test already influenced a redesign, I would label it exploratory and record the date from which fresh observations can accumulate. This article describes an experiment design. It contains no verified trading result, selected signal or system win rate.