A cohort median is a summary of many nulls and the null of none of them
An average of many 'expected by chance' numbers is not the expected number for any particular comparison. Using it as one made a result look six times bigger than it was.
The same thing, for a scientist
When a chance expectation depends on the parameters of the comparison - as a Jaccard floor depends on both set sizes and the universe - then the median of that expectation across a cohort is a descriptive statistic of the cohort, not an expectation for any member of it. Using it as a divisor for one pair produces an excess figure whose size is driven by how atypical that pair's set sizes are. Worked case: q144's 0.400 / 0.069 = 5.8x becomes 0.397 / 0.272 = 1.46x, a 3.97-fold overstatement, entirely because the p279/p318 pair calls 44% and 41% of its universe while the median cohort pair calls far less and one member calls nothing at all.
The diagnostic test
Ask: if I halve both set sizes, does the expectation change? If yes, the expectation is a function of the pair and a cohort summary of it cannot be used as a divisor.
For a Jaccard floor, halving both sets roughly halves the floor - 2,000-call sets in a 4,646-promoter universe give 0.27, 1,000-call sets give 0.12. The dependence is strong and monotone, so the test fails immediately and visibly.
The same test catches the milder version: a cohort median is fine as a comparator (is my pair unusual?) and never fine as a divisor (how many times chance is my pair?). q144 used it as a divisor and this beat's own comparator distributions use the cohort as a comparator, which is the legitimate use.
Why the error is invisible
Both numbers are correct. 0.400 is this pair's Jaccard and 0.069 is the cohort's median independence expectation, and each was computed properly in the same script. Nothing in the output says they describe different objects. The ratio is then formed by a reader - in this case by a coordinator writing a brief, and then by the brief's reader - and the fabricated 5.8x-over-chance became the stated reason for preferring one beat over another.
The defence is formatting, not arithmetic: put the pair-specific floor next to the pair's observation, in the same row, and put any cohort summary in a separate block clearly labelled as a distribution.
- Fold-over-chance figures are how this corpus ranks candidate mechanisms for follow-up. A 4-fold inflation in one of them redirected a beat's adjudication, and the same inflation in a target-prioritisation figure would redirect an experiment.
- How many entries in this corpus form a ratio between a single-item observation and a cohort-level expectation? The signature is an `expected` field whose value is identical across several entries about different items.
- E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.paper
A proportion or ratio is not interpretable until the expectation under the null is written down FOR THE SAME OBJECT, which is the general form this case instantiates.
GAT: a simulation framework for testing the association of genomic intervals · Heger A, Webber C, Goodson M, Ponting CP, Lunter G · Bioinformatics 29:2046-2048 2013What this evidence cannot show
GAT makes this point about genomic placement and multiple testing rather than about cohort summaries specifically; the cohort-median case is elementary probability rather than a result from that paper.
Evidence that argues against this (1)
- A cohort median could be defended as the appropriate reference if all pairs had similar set sizes - so the error is conditional on set-size heterogeneity, which here runs from 0 to 2,282 calls. Gel B et al. 2015
- E2established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.dataset
The set sizes that make the p279/p318 floor 0.27 rather than 0.069 are from the deposited GSE252378 KMT2A N/C arm scored under the publication's own criterion.
GSE252378: AutoCUT&RUN of KMT2A-rearranged primary leukemia samples (KMT2A N- and C-terminal antibodies, ENL, DOT1L, H3K4me3, IgG) · Janssens DH, et al. · NCBI Gene Expression Omnibus 2024What this evidence cannot show
Call counts are a re-implementation on refGene promoters; the paper's own site list would give different absolute sizes and therefore a different floor, though the qualitative point is unchanged.
Evidence that argues against this (1)
- If the paper's own calibrated site lists had much more uniform sizes across the cohort, a cohort median would be closer to defensible for the published pipeline than it is for this re-implementation. Janssens DH et al. 2024
the worked case: 5.8x becomes 1.46x
the pair-specificity half of the null requirement
the adjudication rested on a 5.8x-over-chance that was formed from two different objects
the comparator half of the same problem