The whole-peak-set overlap metric cannot tell a MEN1 ChIP from a DOT1L ChIP in the same untreated cells (Jaccard 0.357, inside the 0.310-0.487 range of all 45 sample pairs) - so on the same data it scores cycle 83's threat as SURVIVING while the pre-specified locus set scores it as FALLING
also called GSE149183 bulk-overlap null · the metric decides the answer
One popular way of comparing these experiments turns out to be useless here: it cannot even tell apart two experiments that measured two different proteins in untreated cells. Using it gives the opposite answer to the careful method, on exactly the same data.
The same thing, for a scientist
This is the control that decides which of this beat's two answers to believe, and it was run because the pre-registration named 'neither branch fires' as a live outcome. **The negative control.** MEN1 DMSO and DOT1L DMSO are two ChIPs of two different proteins in the same untreated MOLM13 culture. Their approximate Jaccard is **0.357**. Across all 45 pairwise comparisons among the ten factor ChIPs the statistic spans only **0.310 to 0.487**, and the MEN1-DMSO-versus-MEN1-VTP-50469 value is **0.339** - *lower* than the different-protein control, and inside the same narrow band as every other pair [E1]. **So a difference in peak-set overlap of this size does not distinguish a drug effect from a change of antibody.** Any claim built on bulk peak-set overlap in this deposit is measuring peak-calling behaviour near the q < 0.01 threshold, not occupancy. **And the two metrics genuinely disagree on this dataset.** Pre-registered decision, same ten files, same day: | site set | VTP-50469 loss | largest DOT1L-inhibitor loss | verdict | |---|---:|---:|---| | all 2,362 DMSO MEN1 peaks | 0.737 | 0.696 (compound 11) | **O1 - threat SURVIVES** | | top 500 by fold-enrichment | 0.642 | 0.532 | O1 - threat SURVIVES | | top 250 | 0.488 | 0.328 | O1 (recovery) / **O2** (median FE) | | top 100 | 0.280 | 0.110 | **O2 - threat FALLS** | | 19 pre-specified canonical target peaks | 1.000 | 0.263 | **O2 - threat FALLS, p = 0.0003** | **The gradient is monotone and it is the signal-to-noise gradient.** The full peak set is ~2,000 weak broad calls whose recovery is a coin flip; as the site set is restricted to stronger and then to biologically pre-specified sites, VTP separates from the DOT1L inhibitors and keeps separating [E1]. **This is almost certainly the shape of cycle 83's alarm.** A metric that puts VTP at 0.737 and a DOT1L inhibitor at 0.696 reads as 'a DOT1L inhibitor does it too'. The two numbers are 6% apart and neither of them means anything, because the same metric puts two different proteins 0.357 apart in untreated cells.
Why this is reported rather than quietly dropped
The pre-registration fixed O3 - neither branch fires, and here is what I would do as a named third outcome, precisely so that a mixed result could not be resolved after the fact by picking the metric that gave the tidier answer. The recorded outcome is: O1 on the bulk metrics, O2 on the restricted and pre-specified ones, and a negative control showing the bulk metric is not a measurement. The O2 is believed because the O1's instrument fails a control that the O2's instrument passes - not because O2 is the preferred answer.
What would have been detectable, since this is partly a negative
With n = 1 and no measurable floor, the only effect size this deposit can defend is presence versus absence of a peak call at a pre-specified locus. It detected a change from 19/19 to 0/19 at p = 0.0003. It could not have detected, and says nothing about, a two-fold reduction in menin occupancy that left the peaks callable - which is exactly the size of effect the DOT1L inhibitors might be having and which would require the spike-in and the replicates this deposit lacks.
| script | analyses/menin-inhibitor-against-dot1l-inhibitors-in-one-deposit/run.py |
|---|---|
| status | run |
| geo | GSE149183 |
| jaccard men1 dmso vs dot1l dmso | 0.357 |
| jaccard men1 dmso vs men1 vtp | 0.339 |
| detectable effect | presence vs absence of a peak call at a pre-specified locus; a sub-threshold reduction that left peaks callable is NOT detectable in this deposit |
| rows dropped by any filter | 0 |
jaccard range all 45 pairs
[ 0.31, 0.487 ]
verdict by site set
{
"all_2362_dmso_men1_peaks": "O1 - threat SURVIVES",
"top500": "O1 - threat SURVIVES",
"top250": "O1 on recovery, O2 on median fold-enrichment",
"top100": "O2 - threat FALLS",
"19_pre_specified_target_peaks": "O2 - threat FALLS, p=0.0003"
}vtp loss by site set — the numbers behind the chart
{
"all": 0.737,
"top500": 0.642,
"top250": 0.488,
"top100": 0.28,
"target_loci": 1
}largest dot1l inhibitor loss by site set — the numbers behind the chart
{
"all": 0.696,
"top500": 0.532,
"top250": 0.328,
"top100": 0.11,
"target_loci": 0.263
}median fold enrichment all ten samples
[ 2.79, 3.4 ]
- If a bulk occupancy-overlap statistic cannot separate two antibodies, it cannot serve as a pharmacodynamic endpoint in a trial either - and occupancy-based PD endpoints for menin inhibitors are exactly what the field is reaching for. The endpoint has to be locus-specified in advance.
- How many claims in this corpus rest on bulk peak-set or bulk-bin overlap between two ChIP libraries without a different-antibody negative control? The control costs one extra comparison when both antibodies are in the deposit, and this is the first time the corpus has run it.
- Does the same discrimination failure hold for binned-coverage comparisons, which this project uses far more often than peak-set overlap? The GSE149183 .tdf tracks would answer it for this deposit and were not downloaded.
- E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.dataset
Across the ten GSE149183 factor ChIPs, approximate Jaccard of peak sets spans 0.310-0.487 for all 45 pairs; MEN1-DMSO vs DOT1L-DMSO (different proteins, no drug) is 0.357 and MEN1-DMSO vs MEN1-VTP-50469 (same protein, drug) is 0.339. The pre-registered decision gives O1 on the full DMSO peak set and O2 on the top-100 and on the pre-specified locus set.
analyses/menin-inhibitor-against-dot1l-inhibitors-in-one-deposit (PREREGISTRATION.md, run.py, results.json) - this repository, re-analysis of the ten deposited GSE149183 MACS2 peak tables · Oncofusion knowledge graph 2026What this evidence cannot show
Approximate Jaccard here is (peaks of i hitting j + peaks of j hitting i) / (n_i + n_j), an overlap-count statistic, not a base-pair Jaccard. A base-pair measure would give different absolute values; the comparison that matters is between pairs under one definition, and every pair is computed the same way.
Evidence that argues against this (1)
- A high MEN1-DOT1L overlap is not purely artefactual - menin and DOT1L genuinely co-occupy many KMT2A-fusion sites, so some of the 0.357 is real biology. That is why the control is stated as 'this metric cannot DISCRIMINATE', not as 'these peak sets are noise'. The discrimination failure is what invalidates its use as a drug-effect readout, and that holds either way. Oncofusion knowledge graph 2026
- E2established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.dataset
The deposited peak sets are broad and near-threshold: median called peak length is of the order of 1-2 kb with MACS fragment-size estimates of 822-1112 bp, 1,101-2,362 peaks per sample, and median fold-enrichment 2.79-3.40 in every one of the ten samples.
NCBI GEO GSE149183 - MEN1, DOT1L and H3K79me2 ChIP-seq in MOLM13 and RS4;11 under DMSO, VTP-50469, EPZ5676, compound 10 and compound 11; 31 samples, Homo sapiens; SubSeries of GSE149185, BioProject PRJNA627612, SRA SRP258055 · NCBI Gene Expression Omnibus 2020What this evidence cannot show
Median fold-enrichment being near-identical (2.79-3.40) across all ten samples is itself a sign that the bulk of each peak set sits at the detection threshold, where membership is governed by sampling rather than by occupancy.
Evidence that argues against this (1)
- None; this is a description of the deposited files. NCBI Gene Expression Omnibus 2020
shows the bulk-overlap metric that would have given the opposite answer cannot distinguish two antibodies in untreated cells
the bulk metric puts VTP at 0.737 loss and a DOT1L inhibitor at 0.696 - the shape of cycle 83's alarm, produced here by an instrument with a failing negative control
both entries are about the same failure: a peak-level statistic that moves for reasons other than occupancy