Two reviewers agree on the gold evidence. Is their agreement informative?
Agreement is useful only after defining what was classified, which records were reviewed and how often each verdict occurred. Two matching labels are not an independent verification of truth.
On this page6 sections · 2 source notes

- Publish the cross-tabulation of independent decisions before reconciliation.
- High agreement can coexist with poor detection of rare but important evidence problems.
Review the rule before celebrating the verdict
A gold evidence archive may ask reviewers to classify a record as eligible, ineligible or unresolved. If two people usually agree, that is encouraging only after the task is defined. Did they independently inspect the same source packet? Did they use the same rule version? Could one see the other’s decision? A percentage without those conditions is a weak description of the process.
My proposed review design freezes the packet, the observation cutoff and the eligibility rules before classification begins. Reviewers record both the verdict and the evidence supporting it. Reconciliation comes afterwards. This order preserves the distinction between initial reproducibility and the final editorial decision; a discussion that produces consensus cannot retrospectively establish independent agreement.
The common category can dominate the score
Jacob Cohen’s original work on nominal classifications introduced kappa as a way of comparing observed agreement with an agreement level implied by the reviewers’ category frequencies. In his later first-person account, he explained why a high raw percentage can be misleading when reviewers heavily favour the same category. The reference concerns measurement reliability; applying it to gold evidence is an editorial research proposal.
A hypothetical archive makes the issue visible. Suppose two reviewers independently classify 100 records, each marking 80 eligible and 20 ineligible. Their joint table contains 70 both eligible, 10 eligible only according to the first, 10 eligible only according to the second and 10 both ineligible. Raw agreement is 80%. The marginal-frequency agreement term is 0.8² plus 0.2², or 68%; kappa is (0.80−0.68)/(1−0.68), equal to 0.375.
Source notes: [1] Jacob Cohen — A Coefficient of Agreement for Nominal Scales · [2] Jacob Cohen — Citation Classic retrospective on agreement
Keep the table alongside the coefficient
The hypothetical reviewers disagree on 20 records. A single coefficient cannot tell an editor whether those disagreements concern missing activation events, revised price levels or contradictory stops. Nor does it show the operational consequence of a mistaken decision. The cross-tabulation and the reason codes should therefore remain visible even when a summary statistic is reported.
I would avoid declaring a universal pass threshold for this application. The consequences of publishing an unsupported result differ from those of retaining a valid candidate for more investigation. A quality process needs to understand that asymmetry. Agreement is one diagnostic of how consistently a rule is applied, not a substitute for assessing what the rule permits.
Do not force uncertainty into a favourable category
An unresolved category is useful when the evidence packet genuinely lacks a required field. Forcing every record into yes or no can create artificial certainty and conceal a data-access problem. Define unresolved conditions in advance, such as unavailable original levels or unclear event ordering, rather than allowing the label to become a discretionary holding area.
If almost every record belongs to one category, report that prevalence plainly. A sample assembled from only easy cases says little about difficult cases. Conversely, deliberately oversampling ambiguous records can be useful for training but should not be presented as the agreement rate of the entire production archive. Keep the sampling purpose and population claim aligned.
Use disagreements to repair specific instructions
After the independent pass, classify the disagreements themselves. Some may reveal ambiguous wording; others may expose missing evidence, inconsistent source interpretation or an overlooked exception. The remedy should match the cause. Adding another paragraph to a policy does not repair a missing timestamp, while collecting more timestamps does not resolve an undefined eligibility condition.
When a rule changes, version it and decide which previous records need another review. Preserve the initial verdicts alongside the revised outcomes so the quality record remains auditable. I would also keep a small set of difficult examples for future reviewer training, with explanations of why the evidence supports the assigned result. These examples should test the rule, not reward memorising a preferred answer.
Agreement does not establish market truth
Two reviewers can consistently apply a flawed rule or share the same mistaken source. Independence of classification is therefore narrower than independence of evidence. The source chain still needs its own checks: original identity, level history, activation and event ordering, followed by matching OHLC replay where available. Reviewer agreement cannot create information that the packet does not contain.
This article reports no actual agreement score, signal outcome or verified broker trade. Every number above is an invented teaching example. Its purpose is to make evidence review itself reviewable: retain the packet, preserve independent decisions, publish the disagreement structure and document reconciliation. A credible research process should be able to explain not only its final answer but why another careful reader might initially have reached a different one.
Sources, dates & evidence limits
Sources checked 15 November 2026 at 04:10:13 UTC / 15 November 2026 at 08:10:13 Dubai. Source observation and release dates are separate from this article’s publication date. Linked data may later be revised.
- Jacob Cohen — A Coefficient of Agreement for Nominal Scales ↗April 1960; Educational and Psychological Measurement 20(1), 37–46
Original methodology paper; bibliographic record freshly checked. The gold application and numerical table are original illustrative analysis.
- Jacob Cohen — Citation Classic retrospective on agreement ↗20 January 1986; author account dated 18 November 1985
First-person explanation of category frequencies and kappa. No universal acceptance threshold is imported into gold-signal review.
Follow the reading path
How to Backtest a Gold Trading Strategy Without Fooling Yourself →Gold Trading & Research →Research Lab: test the evidence →CyGuru trading books: continue your study →Research Lab · inspect the evidence standard →Gold Terminal · reported signal records →Sunday Gold Brief · separate market preparation →Follow new Gold Intelligence articles →All 15 November 2026 articles →Search the Gold Intelligence library →Back to article contents ↑Original educational analysis. Illustrative calculations are not historical performance. Interpretation is identified in the text; no trade outcome is established by this article. Editorial standards →



