Would a meaningless gold signal look just as convincing?

Make the comparison capable of embarrassing the model

A useful research control should be able to challenge the preferred explanation. If a gold indicator claims to identify favourable moments, ask whether a deliberately uninformative schedule with similar opportunities would produce a comparable nominal outcome. That is a different question from whether the indicator’s selected examples look impressive on a chart.

This article proposes an audit design. No placebo experiment has been run here, and no CY66 improvement, win rate or statistical significance is claimed. The goal is to define a comparison before results are known so that a later test cannot quietly weaken the control when the real model struggles to outperform it.

Specify the null before randomizing anything

An independent shuffle of every observation may destroy patterns that are important to the question. A control intended to remove signal timing should not accidentally remove the session structure, holding horizon or clustering that the original setup experiences. The relevant design depends on what is being tested. A placebo for timing is not automatically a valid placebo for direction or stop placement.

Berkowitz and Kilian’s Federal Reserve research paper on time-series bootstrapping emphasizes model selection and the importance of block size. It does not validate a particular gold experiment. I use it as methodological context for a narrower principle: resampling choices carry assumptions. Calling a comparison random does not make those assumptions harmless or its result automatically interpretable.

Source notes: [1] Jeremy Berkowitz and Lutz Kilian — Recent Developments in Bootstrapping Time Series, Federal Reserve FEDS

Keep the opportunities comparable

My proposed protocol begins with a frozen set of eligible sessions and a documented outcome rule. Retain the same coverage requirements, maximum holding horizon and treatment of unresolved setups. Then define a control that removes the information under investigation while preserving the features necessary for a fair comparison. For example, a timing study might consider alternative eligible times within predeclared session strata, subject to explicit overlap restrictions.

That example is a design candidate, not a universally valid randomization test. It would need justification for the dataset and hypothesis. If the control gets opportunities the real strategy could never take, or avoids periods the real strategy must face, the comparison answers a different question. A reviewer should be able to reconstruct both opportunity sets before seeing the score.

Prevent the scoring rule from choosing the winner

Write the primary outcome in advance. A nominal TP1-before-SL measure should use identical price-side definitions, frozen levels, expiry windows and same-bar ambiguity handling in both groups. Report unresolved and unavailable cases as separate counts rather than quietly discarding more inconvenient observations from one side. An additional outcome can be informative, but it should not replace the primary one merely because it looks better.

Repeatability also requires retaining the random seed, algorithm version and the complete set of control runs. A single lucky or unlucky placebo is not a distribution. I would inspect how the conclusion changes under defensible alternative design choices and report that sensitivity. If many outcomes, strata and control methods were tried, the breadth of that search belongs in the record.

Interpret failure and success symmetrically

If the real model resembles its controls, the result could mean it adds little under this design, the test lacks information, or the control failed to isolate the intended feature. Diagnose those possibilities without declaring victory. If the real model differs, inspect leakage, coverage and design choices before calling the difference an edge. A large contrast can be a sign of a broken comparison as well as a useful discovery.

Even a well-supported difference in nominal OHLC outcomes is not a verified execution result. It omits broker fills, commissions, spread, slippage and P&L. The Research Lab’s value is to preserve a reproducible question and its evidence boundaries. A control makes that work stronger when it is designed to compete fairly with the model, rather than to decorate a conclusion already chosen.