My own pre-registered reagent gate was the wrong test: it demanded a within-sample ordering from a class defined by a between-sample difference, and it failed by 2.5%
Before looking at the data I wrote down a check that the borrowed gene lists still worked. The check asked the wrong question, and it failed - not because the lists are bad but because I had misremembered what they were.
The same thing, for a scientist
The pre-registered gate **G1** required that in **p279** the `ALL_specific` class score **above** the `AML_specific` class, and in **p318** the reverse. p318 passed easily (0.426 against 3.739). **p279 failed: 2.014 against 2.064**, a 2.5% margin in the wrong direction. That fired the pre-registered **U1** and made the whole analysis UNINFORMATIVE. **The gate was wrong, not the reagent.** The classes are built from a *difference between* p279 and p318 - z > +1.96 and z < -1.96 on a p279-minus-p318 statistic. Nothing in that construction says the p279-high intervals must outscore the p318-high intervals *inside p279*; it says each set must be higher in its own sample than in the other. Tested that way, both pass decisively: `ALL_specific` 2.014 -> 0.426 (4.72x down) and `AML_specific` 2.064 -> 3.739 (1.81x up). **The failing gate is not deleted and the passing one does not replace it.** Swapping a failed pre-registered gate for a post-hoc one that passes is the move that voids a pre-registration, so the headline verdict stays NEITHER/UNINFORMATIVE and the corrected gate is reported as a labelled secondary. It also did not matter to the conclusion: three other uninformative conditions fired independently, and the post-hoc analysis reaches NO-DESTINATION rather than GMP-DESTINATION either way.
The generalisable form
A validation gate has to mirror the construction of the thing it validates. For a class defined by a contrast between conditions A and B, the gate is higher in A than in B, per class. For a class defined by an absolute threshold in one condition, the gate is above that threshold in that condition. For a class defined by a within-sample ranking, and only then, the gate is a within-sample ranking.
The practical instruction: before writing the gate, write down the line of code that produced the class. Here the line was
df_greater_than_196 = top_differences[top_differences['z_score'] > 1.96]
and it is a between-sample difference on its face. I had read that code before writing the pre-registration and still wrote a within-sample gate, because 'ALL-specific should be high in the ALL sample' is the intuitive sentence and 'higher in the ALL sample than in the AML sample' is the true one.
Why a 2.5% failure margin is the dangerous kind
Had p279's ALL_specific come in at 2.5% above AML_specific instead of below, G1 would have passed, U1 would not have fired, and I would have reported a pre-registered gate as validating a reagent - on a test that does not validate anything. A gate that passes by noise is worse than one that fails by noise, because nobody re-derives a gate that passed. The only reason the error was found is that it failed.
| gate | G1 in PREREGISTRATION.md section 5 |
|---|---|
| as written | within p279, ALL_specific must exceed AML_specific |
| corrected gate | each class must be higher in its own defining sample than in the other |
| handling | the failing pre-registered gate is retained and the corrected gate is reported as a labelled post-hoc secondary; the headline branch stays NEITHER/UNINFORMATIVE |
| status | run |
result as written
{
"p279_ALL_specific": 2.014,
"p279_AML_specific": 2.064,
"margin_pct": -2.4,
"passes": false,
"p318_ALL_specific": 0.426,
"p318_AML_specific": 3.739,
"p318_passes": true
}result corrected
{
"ALL_specific_p279_over_p318": 4.724,
"AML_specific_p318_over_p279": 1.811,
"passes": true
}measurements
[
{
"quantity": "margin_by_which_the_preregistered_gate_failed",
"subject": "p279 ALL_specific minus AML_specific class mean",
"value": -0.05,
"units": "x an average promoter",
"n": 2,
"universe": "p279 KMT2A_N1 and N2; 37 and 40 promoters",
"null": "the gate required a positive margin",
"note": "2.4% of the value; within the spread between the two N libraries"
},
{
"quantity": "between_sample_fold_change",
"subject": "ALL_specific class, p279 over p318",
"value": 4.724,
"units": "fold",
"n": 2,
"universe": "same libraries and promoters",
"null": 1,
"note": "the corrected gate; AML_specific gives 1.811 in the opposite direction"
}
]- Should a pre-registration be allowed to record a corrected gate at all, or should the run simply be declared void and re-registered? This beat chose to report both, and that choice is not obviously right.
- Is the within-sample reading defensible after all - should a B-lineage-specific class outscore a myeloid-specific class inside a B-lineage sample? If yes, U1 is a finding about the reagent and not an error of mine.
- E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.dataset
The pre-registered G1 gate failed in p279 by 2.014 against 2.064 while the corrected between-sample version passes at 4.72x and 1.81x in the two classes.
analyses/q148-what-the-hoxa-negative-blast-occupies - PREREGISTRATION.md section 5 G1; results148.json G1_reagent_gate; results148_controls.json C1_G1_corrected · Q148 (Oncofusion Knowledge Graph) · this repository 2026What this evidence cannot show
Both versions are computed on the pair that defined the classes, so neither is evidence about any other sample. The 2.5% margin is within the spread between the two N libraries of p279 and should not be read as a measurement.
Evidence that argues against this (1)
- A reader could argue the original gate was reasonable and the reagent genuinely fails it - that the B-lineage class ought to outscore the myeloid class inside a B-lineage sample. That reading is defensible and would make U1 a real finding about the reagent rather than an error of mine; it is recorded here as an open disagreement rather than resolved. Janssens DH 2024
gate U1 fired on a test the reagent was never built to pass
the corrected between-sample gate and its numbers