A fusion-versus-rest expression difference can be pure morphology composition - I published one at p below 1e-12 and it dissolved inside every FAB stratum
also called q149 composition confound · Simpson's paradox in a fusion-vs-rest contrast
We found that children with a KMT2A fusion had much less IRX1 than other children with leukaemia, and the statistics looked overwhelming. It was an illusion: KMT2A leukaemias are mostly one cell type, and that cell type has low IRX1 whether or not there is a fusion.
The same thing, for a scientist
**The wrong claim, which I wrote down inside this beat before checking it.** In `aml_stjude_2024`, IRX1 transcript in 163 non-AFF1 KMT2A-rearranged paediatric AML samples has median **-5.199** against **-2.597** in 707 non-KMT2A-rearranged samples - Mann-Whitney two-sided **p ~ 0** (z = -7.13), permutation two-sided **p = 0.0001** over 20,000 shuffles. The pre-registered on/off rate was 14.7% against a cohort null of 33.4%. The natural reading, and my first one, was *KMT2A-rearranged AML is IRX1-depleted*. **The positive control made it look safe.** HOXA9 in the very same samples moves the other way - median 6.759 against 2.036, two-sided p ~ 0 - so the cohort demonstrably can report a KMT2A-r-specific difference, and the IRX1 result was not an instrument failure. **A working positive control does not make a pooled contrast interpretable.** **What it actually was.** Restricting the null to the same morphology destroys it entirely: | FAB | KMT2A-r IRX1 ON | non-KMT2A-r IRX1 ON | |---|---|---| | M7 | 13 / 20 = **65.0%** | 54 / 97 = **55.7%** | | M5 | 2 / 96 = **2.1%** | 3 / 52 = **5.8%** | **Within each stratum the two arms agree.** The pooled difference exists because **96 of 171 non-AFF1 KMT2A-r cases are FAB M5** - the lowest-IRX1 morphology in the cohort - while the comparator arm is spread across M1, M2, M4, M6 and M7 where IRX1 rates run 20-56%. The fusion arm's composition, not the fusion, produced the p-value. **This is Simpson's paradox, and this project already owns the rule that catches it** - `restrict the null with the population`. The rule was written after a subgroup made a mean rise 2.9x and the null rose with it. Here the null did not rise; it inverted the conclusion.
Why this instance is worth a concept entry and not just a footnote
Three features made it hard to see, and all three recur:
- The effect size was large and the p-value was extreme. Nothing about
p ~ 0invites a second look. The DepMap base-rate failures in this project were caught because the numbers were unremarkable; this one was caught only because a separate question - is the IRX1-high subgroup megakaryoblastic, as Nagel predicts? - happened to require the cross-tab. - A valid positive control was present and passed. HOXA9 moved the right way with the same samples, the same profile and the same test. That licenses the instrument, and nothing more. A control that shows the assay works does not show the comparison is fair.
- The confound is the disease's own biology. KMT2A-rearranged AML is monocytic-skewed; that is a real and well-known property of the disease, not a sampling accident. So the composition imbalance is not a defect of the cohort that could be fixed by better sampling - it is structural, and any fusion-versus-rest contrast in this disease will carry it.
The practical rule. Before reporting any fusion-versus-rest, mutation-versus-wild-type or subtype-versus-rest expression difference: find the strongest categorical covariate in the cohort (morphology, lineage, cell of origin, institution), tabulate the outcome within each of its levels, and print the per-stratum table beside the pooled statistic. If the pooled difference is not present in any stratum, the pooled statistic is a fact about composition.
What survives, stated narrowly
The surviving claim is in ana-q149-irx1-in-kmt2a-r-paediatric-aml-is-a-megakaryoblastic-lineage-property-not-a-fusion-property: IRX1 expression in paediatric AML is a FAB-M7 property, present at indistinguishable rates with and without a KMT2A rearrangement.
What does NOT survive: any statement of the form KMT2A-rearranged AML has low IRX1 as a fusion-associated fact. It has low IRX1 because it is mostly M5.
And note the asymmetry in the damage. Had I reported the pooled result, it would have argued for the beat's branch (A) SILENT reading and for the brief's expectation. The artefact pointed the comfortable way. This project's standing observation is that filters and confounds tend to return the comfortable answer, and this is another instance.
| withdrawn claim | non-AFF1 KMT2A-rearranged paediatric AML is IRX1-depleted relative to other paediatric AML |
|---|---|
| withdrawn because | 96 of 171 KMT2A-r cases are FAB M5, the lowest-IRX1 morphology; within M5 and within M7 the KMT2A-r and non-KMT2A-r arms agree |
| absence class | measured-not-analysed |
pooled statistic that misled — the numbers behind the chart
{
"median_kmt2ar": -5.1989,
"median_other": -2.5973,
"mann_whitney_z": -7.1348,
"mann_whitney_p_two_sided": 0,
"permutation_p_two_sided": 0.0001,
"permutation_p_one_sided_less": 0.00005
}positive control that passed anyway
{
"gene": "HOXA9",
"median_kmt2ar": 6.7592,
"median_other": 2.036,
"p_two_sided": 0.0001,
"direction": "higher in KMT2A-r, as expected"
}measurements — the numbers behind the chart
[
{
"quantity": "proportion_IRX1_gt0_difference",
"subject": "non-AFF1 KMT2A-r vs non-KMT2A-r paediatric AML, POOLED",
"value": -0.2451,
"units": null,
"n": 878,
"universe": "aml_stjude_2024, 171 KMT2A-r vs 707 non-KMT2A-r, IRX1 log2>0",
"null": 0,
"note": "0.117 minus 0.362; this is the ARTEFACT. Within FAB M7 the difference is +0.093 and within M5 it is -0.037"
},
{
"quantity": "proportion_IRX1_gt0_difference",
"subject": "non-AFF1 KMT2A-r vs non-KMT2A-r, WITHIN FAB M7",
"value": 0.093,
"units": null,
"n": 117,
"universe": "aml_stjude_2024 FAB M7 only, 20 KMT2A-r vs 97 non-KMT2A-r",
"null": 0,
"note": "0.650 minus 0.557; opposite sign to the pooled difference"
},
{
"quantity": "share_FAB_M5",
"subject": "non-AFF1 KMT2A-r paediatric AML",
"value": 0.561,
"units": null,
"n": 171,
"universe": "aml_stjude_2024, FAB recorded for 157 of 171",
"null": null,
"note": "96 of 171; M5 is the lowest-IRX1 morphology in the cohort in BOTH arms, which is the whole mechanism of the artefact"
}
]- A biomarker derived from a pooled fusion-versus-rest contrast will select patients by morphology while claiming to select them by fusion. Here that would have enrolled M7 children under a KMT2A-r label and excluded the M7 children without the rearrangement who have the same IRX1 status.
- How many of this corpus's fusion-versus-rest expression claims have ever been stratified on morphology or lineage? A sweep would say whether this is one instance or a class.
- Is the monocytic skew of KMT2A-rearranged AML a consequence of the fusion, in which case the pooled contrast is the biologically right one and the stratified one over-controls? Nothing here decides it and the answer changes which statistic to quote.
- E1moderate confidence 3 of 5: One strong study, or several smaller consistent ones.dataset
In 887 paediatric AML cases, the pooled IRX1 difference between non-AFF1 KMT2A-rearranged and non-KMT2A-rearranged samples (medians -5.199 vs -2.597, Mann-Whitney two-sided p ~ 0, permutation two-sided p = 0.0001) is absent within FAB strata: M7 is 65.0% versus 55.7% IRX1-positive and M5 is 2.1% versus 5.8%, while 96 of 171 KMT2A-rearranged cases are M5.
A new genomic framework to categorize pediatric acute myeloid leukemia · Umeda M, Ma J, Westover T, et al. · Nat Genet 56:281-293 2024What this evidence cannot show
My analysis of their deposited data, not their result. FAB is recorded for 787 of 887 samples and missing for 14 of the 171 KMT2A-r cases, which are reported in a NEVER-RECORDED row rather than scored. Stratification onto six morphologies at these n means several strata are individually underpowered; the argument rests on M7 and M5, which carry 116 of the 171 KMT2A-r cases between them. The stratified rates use an absolute log2>0 anchor rather than the pre-registered per-sample floor, because that floor's verdict was itself threshold-dependent.
Evidence that argues against this (1)
- Stratifying on FAB can also destroy a real effect if the fusion CAUSES the morphology, which for KMT2A-rearranged AML is arguable - the monocytic skew is part of the disease. On that reading the pooled comparison is the biologically meaningful one and the stratified one over-controls. The answer here does not turn on it, because the direction within M7 and within M5 both agree with the non-fusion arm rather than differing from it - but the objection is real and applies to any use of this rule. Umeda M et al. 2024
the withdrawn pooled claim and the surviving stratified one are the same measurement
- related to →IRX1 fusion occupancy is confined to KMT2A::AFF1: 0 of 18 samples across five other partners
the pooled artefact pointed toward that entry's reading; the stratified result does not
an expression subgroup can also be a MORPHOLOGY subgroup wearing a fusion label
the FAB field in this study is what exposed the composition artefact