All three pre-registered branches came out of the same numbers under four defensible floors - the expressed/silent binary was measuring the ruler, not the gene
also called q149 threshold sensitivity · the verdict tracks the threshold - IRX1 case
We had to decide whether a gene was on or off. Depending on which reasonable cut-off we used, the same data said on, off, or undecidable. So the on/off question was not answerable, and we reported the comparisons that do not need a cut-off instead.
The same thing, for a scientist
The q149 pre-registration named three branches - **SILENT (A)**, **EXPRESSED-BUT-UNOCCUPIED (B)**, and **NEITHER** - with numeric firing conditions on the fraction of non-AFF1 KMT2A-r samples called IRX1-ON. The floor was pre-registered as **max of a ten-gene silent-tissue panel in the same sample**, chosen specifically so it could not be tuned to IRX1's distribution. **Re-scored under four floors built from that same panel, the branch changes in every cohort:** | cohort | max (pre-reg) | p95 | median | min | |---|---|---|---|---| | St Jude paediatric AML, non-AFF1 n=163 | 14.7% -> **NEITHER** | 22.7% -> **NEITHER** | 64.4% -> **B** | 88.3% -> **B** | | CCLE haematopoietic lines, non-AFF1 n=5 | 0% -> **A** | 0% -> **A** | 20% -> **NEITHER** | 20% -> **NEITHER** | | BeatAML adult, non-AFF1 n=17 | 5.9% -> **A** | 35.3% -> **B** | 11.8% -> **NEITHER** | 35.3% -> **B** | | GSE68720 infant ALL, non-AFF1 n=26 | 11.5% -> **NEITHER** | 11.5% -> **NEITHER** | 15.4% -> **NEITHER** | 26.9% -> **B** | **BeatAML alone yields A, B and NEITHER from 17 samples and one panel.** No floor here is indefensible: the maximum is the most conservative, the median is the panel's central tendency, the 95th percentile is the standard choice for a background panel, and the minimum is the most permissive. A reader given only the pre-registered row would have received `NEITHER, A, A, NEITHER` and would have had no way to know the rest existed. **The rule this project already holds is `if the verdict tracks the threshold, you are measuring the ruler`** - written after the same numbers gave 0, 4 and 4 hits under three imported noise floors. **This is the second instance, and it is worse, because here the floors are not imported from other work but all derived from one deliberately pre-registered panel.** Internal consistency is no protection. **What we did instead.** Report the quantities that do not need a floor: absolute anchors with the same anchor applied to the null (IRX1 > 0, > 1, > 2, > 4 log2), the within-cohort AFF1-versus-non-AFF1 **contrast** (stable under all four floors), rank tests on the continuous values with their tie structure printed, and per-sample values named for every group below n = 5.
Why the floor moves so much, and the diagnostic to run first
Because the panel was not silent. In aml_stjude_2024 the ten-gene panel's own distribution is:
| gene | median | p95 | max |
|---|---|---|---|
| KRT1 | -1.13 | +4.13 | +7.38 |
| GFAP | -4.52 | +0.80 | +8.61 |
| ALB | -3.01 | -0.46 | +4.53 |
| CDX2 | -3.01 | -0.18 | +4.38 |
| MYOG | -5.84 | -3.75 | -0.57 |
| SFTPC | -5.80 | -3.57 | -0.52 |
IRX1's own cohort median is -3.31, so KRT1, ALB and CDX2 all sit ABOVE it. A max floor is therefore set by whichever contaminant runs hottest in that library, and the spread between the panel's max and its min is larger than the spread between IRX1's on and off states. The floor was doing most of the work.
The pre-registration named exactly this as a withdrawal condition (section 7: "the silent panel is not silent: if KRT1 or CDX2 is high in leukaemia samples, the floor is wrong"). It fired. That is the pre-registration working, not failing.
The diagnostic, cheap and first: before using a background panel as a floor, print the panel's own per-gene distribution against the target gene's distribution in the same samples. If any panel member's median exceeds the target's median, the panel is not a floor for that target.
The related instrument failure, which is MY panel and not the resource
aml_ohsu_2022 (BeatAML) returned values for only 2 of the 10 silent genes - KRT1 and CDX2 - which are precisely the two worst behaved. The first version of this analysis therefore scored 0 of 671 BeatAML samples and would have reported BeatAML as unusable.
That is my gene panel failing against a restricted profile, not BeatAML lacking the data. Recorded as an instrument failure with the two available genes named, and the resulting BeatAML numbers are flagged as resting on a two-gene floor. Similarly INS has no value in aml_stjude_2024, so that cohort's floor is nine genes, not ten.
The general form: a fixed panel is a hidden dependency on the profile's gene coverage. Print how many panel members the profile actually returned, per cohort, beside every number derived from it.
| stable under all four | the AFF1-versus-non-AFF1 CONTRAST in GSE68720 (AFF1 rate exceeds non-AFF1 rate under max, p95, median and min) |
|---|---|
| script | analyses/q149-is-irx1-expressed-in-non-aff1-kmt2a-r/stats149.py |
| absence class | measured-not-analysed |
silent panel
[ "ALB", "INS", "MYH7", "CRYAA", "RHO", "SFTPC", "KRT1", "MYOG", "GFAP", "CDX2" ]
panel members returned per cohort — the numbers behind the chart
{
"aml_stjude_2024": 9,
"ccle_broad_2019": 10,
"aml_ohsu_2022": 2,
"GSE68720_GPL570": 10
}panel members absent named
{
"aml_stjude_2024": [
"INS"
],
"aml_ohsu_2022": [
"ALB",
"INS",
"MYH7",
"CRYAA",
"RHO",
"SFTPC",
"MYOG",
"GFAP"
]
}branch by threshold
{
"aml_stjude_2024_nonAFF1_n163": {
"max": "NEITHER",
"p95": "NEITHER",
"median": "B",
"min": "B"
},
"ccle_broad_2019_nonAFF1_n5": {
"max": "A",
"p95": "A",
"median": "NEITHER",
"min": "NEITHER"
},
"aml_ohsu_2022_nonAFF1_n17": {
"max": "A",
"p95": "B",
"median": "NEITHER",
"min": "B"
},
"GSE68720_nonAFF1_n26": {
"max": "NEITHER",
"p95": "NEITHER",
"median": "NEITHER",
"min": "B"
}
}measurements — the numbers behind the chart
[
{
"quantity": "n_distinct_branches_from_one_cohort",
"subject": "aml_ohsu_2022 non-AFF1 KMT2A-r, four floors of one panel",
"value": 3,
"units": "pre-registered branches",
"n": 17,
"universe": "IRX1 ON rates 0.059 / 0.353 / 0.118 / 0.353 under max / p95 / median / min",
"null": 1,
"note": "SILENT_A, EXPRESSED_B and NEITHER all obtainable from 17 samples and one panel"
},
{
"quantity": "proportion_IRX1_ON_range_across_floors",
"subject": "aml_stjude_2024 non-AFF1 KMT2A-r",
"value": 0.736,
"units": null,
"n": 163,
"universe": "0.147 at max to 0.883 at min, same 163 samples, same panel",
"null": null,
"note": "the null moves too: 0.334 to 0.943 in the non-KMT2A-r comparator, which is why only the contrast is usable"
},
{
"quantity": "silent_panel_median_above_IRX1_median",
"subject": "aml_stjude_2024 ten-gene silent panel",
"value": 3,
"units": "genes",
"n": 9,
"universe": "KRT1 -1.13, ALB -3.01, CDX2 -3.01 all above IRX1 cohort median -3.31",
"null": 0,
"note": "the panel is not silent; this is the mechanism of the threshold sensitivity"
}
]- A trial eligibility criterion of the form 'IRX1-positive' is not implementable from these data: the same patient is positive or negative depending on a background panel nobody has standardised, and in the infant ALL cohort the non-AFF1 positive population is 3, 3, 4 or 7 of 26 children under four reasonable cut-points.
- What IS a defensible expression floor for a low-expressed developmental transcription factor in bulk leukaemia RNA? A spike-in, an ERCC control or a matched non-expressing sorted population would each give one, and none of the four public profiles used here carries any of them.
- How many of this corpus's expression on/off calls rest on an unstated threshold? The rule now has two instances and no sweep.
- E1moderate confidence 3 of 5: One strong study, or several smaller consistent ones.dataset
Re-scoring the same IRX1 expression values under four floors derived from one ten-gene silent panel - its maximum, 95th percentile, median and minimum - yields all three pre-registered branches across cohorts, including all three from the single 17-sample BeatAML arm.
Integrative analysis of drug response and clinical outcome in acute myeloid leukemia · Bottomly D, Long N, Schultz AR, et al. · Cancer Cell 40:850-864.e9 2022What this evidence cannot show
The BeatAML arm rests on a TWO-gene floor because only KRT1 and CDX2 of the ten panel members are present in that profile, and those two are the least silent - so its threshold sensitivity is exaggerated relative to the other cohorts. BeatAML is also ADULT AML and its KMT2A partner annotation names only MLLT3. The four floors are not independent of each other; they are order statistics of one panel, which is why the demonstration is about the panel and not about floors in general.
Evidence that argues against this (1)
- A critic could say only the pre-registered floor counts and the sweep is post-hoc licence to pick a verdict. That is the right worry, and the answer is that the sweep is reported as a REFUSAL to pick - the headline is NEITHER, which is the pre-registered floor's own answer in the primary cohort, and the sweep is why no branch is claimed. If the sweep had been used to select branch B it would be exactly the error it warns about. Bottomly D et al. 2022
the pre-registered binary verdict for this cohort is threshold-dependent; the contrast is not
why the AML arm is reported with absolute anchors rather than the pre-registered floor
a non-expressed-locus panel worked for occupancy; the transcript analogue of it did NOT - three of its ten genes sit above IRX1's cohort median
the 2x2 on thresholded calls is used precisely because the rate is threshold-dependent