No failures were observed. How much uncertainty remains?

Begin with what the zero actually counts

A research file may contain no failures because a process performed consistently, because the sample was short, because failures were never recorded, or because only favourable cases were chosen for publication. Those possibilities produce the same visible count and very different evidence. Before calculating anything, define a failure, identify every eligible observation and explain how an unresolved observation is handled.

For a gold system, failure could mean a specific reference stop occurring before a specified target, a data-delivery failure, or a negative broker trade after costs. These are different outcomes. A target message does not establish net profitability, and a delivery failure does not establish a losing setup. A statistical report needs one operational definition and a record showing that all eligible observations were checked against it.

A small hypothetical experiment makes the limitation visible

Consider an invented experiment with 30 independent trials, each having the same unknown probability p of a defined failure. Assume every trial is observed completely and none fails. This is a teaching example, not a count of CyGuru signals. Under the binomial model, the probability of seeing zero failures is (1 − p) raised to the power 30. Even if the true failure probability were 5%, zero failures would occur about 21.46% of the time.

The exact one-sided 95% upper confidence limit after zero failures in n trials is 1 − 0.05 raised to the power (1/n). For n = 30 it is about 9.50%; for n = 100 it is about 2.95%. These are calculations from the stated model, not estimates of a trading system. The upper limit is not a guaranteed ceiling on future failures. The confidence procedure has a repeated-sampling coverage interpretation when its assumptions hold.

Source notes: [1] NIST CSRC · Binomial distribution · [2] NIST/SEMATECH · Confidence intervals for a proportion

Why a perfect-looking interval can be a calculation error

A familiar normal approximation uses the observed proportion and its estimated standard error. At zero observed failures, that plug-in standard error can collapse to zero. The resulting zero-width interval would suggest certainty precisely where a small sample leaves considerable uncertainty. NIST’s statistical handbook discusses exact binomial intervals for small failure counts or small samples; its Dataplot documentation distinguishes one-sided from two-sided limits.

The distinction between one-sided and two-sided matters here. The calculation above allocates the full 5% tail probability to the upper bound. Substituting 2.5% would answer a different interval question and give a different number. A reproducible notebook should save the tail convention and confidence level beside the count, rather than leave readers to infer them from a rounded percentage.

Source notes: [2] NIST/SEMATECH · Confidence intervals for a proportion · [3] NIST Dataplot · Exact binomial confidence limits

Thirty alerts may not be thirty independent trials

NIST’s binomial definition requires independent trials with the same probability. Gold setups overlapping one price move do not automatically meet that condition. Several timeframes can respond to the same shock; a copied alert can repeat one identity; a model revision can change the process halfway through a sample. Enlarging the row count under those circumstances can make an interval narrower without adding equivalent independent information.

My first audit would group shared signal identities and mark overlapping exposure windows before choosing a statistical model. I would also preserve engine versions, inclusion rules and observation cutoffs. Grouping is not a magic cure: treating each day as independent still needs justification, and a large shift in volatility can change outcome probabilities across days. If the assumptions cannot be supported, present descriptive counts and their limitations rather than attach an exact-looking confidence claim.

Source notes: [1] NIST CSRC · Binomial distribution

Selection cannot be repaired by a confidence formula

A collection of selected successful case studies is useful for inspecting how evidence was retained, whether original levels were frozen and which timestamps are missing. It supplies no denominator for an overall failure rate. The mathematical examples above must never be applied to that collection as though it were a complete sample. Unknown or unobserved outcomes belong in a separate unresolved category; excluding them requires an explicit reason that a reader can evaluate.

For a prospective study I would set the sample boundary and stopping rule in advance, collect all eligible outcomes, and retain contradictory evidence. Repeatedly checking the interval and stopping when it looks reassuring changes the sampling procedure. Until a suitable full record exists, the useful conclusion is narrower: the observed sample contains no recorded failures under a stated definition. This article reports no actual trade results, independent signal trials, broker executions, costs or future-profit probability.