The 'sixteen percent of GEO sample labels contradict their own metadata' finding is a one-line off-by-one: 2,280 of 2,280 titles were attributed to the wrong sample, and with the record boundary fixed the rate falls from 96 of 586 comparable pairs (16.4%) to 7 of 656 (1.1%) - and all seven, read by hand and named here, are tokeniser artefacts rather than contradictions
A previous analysis in this project concluded that one archived experiment in six is labelled inconsistently with its own description, and the worst dataset two in five; the code that found this was reading each experiment's name off the next experiment's record. With the bug fixed the rate is 7 in 656 (1.1%), and reading all seven by hand shows each is a wording mismatch in our own matching code rather than a real contradiction.
The same thing, for a scientist
`analyses/does-this-deposit-agree-with-itself/run.py` keys each sample record on the line `!Sample_geo_accession`. In a GEO SOFT record the field order is `^SAMPLE = GSM...`, then `!Sample_title`, then `!Sample_geo_accession`. **The title line therefore arrives before the accession line, and is written into the previous sample's record.** Every sample is scored with the NEXT sample's title against its own characteristics. Measured directly: **2,280 titles belong to the following sample; 0 are correctly attributed.** Re-running the unchanged audit logic with the record boundary anchored on `^SAMPLE =` takes the headline from **96 disagreements in 586 comparable pairs (16.4%) to 7 in 656 (1.1%)**. **1.1% is the measured rate and it is what should be quoted.** Reading all seven by hand is a separate, clearly-labelled step and it finds each to be a tokeniser artefact of the kind the original script's own `U2` clause forbids reporting - `MLL1c` against `MLL1 (c terminus)`, and `SEM_0umEPZ5676` against `DMSO`, where *zero micromolar* EPZ-5676 IS the vehicle. **So: 1.1% by the instrument, 0 of 7 surviving a hand read.** Both numbers are stated because the second depends on a judgement the first does not. **Independently reproduced by the coordinator on 2026-09-23**, who reached the same 7/656 and the same 0 antibody and 0 treatment contradictions in GSE159608, and who additionally found that the same artefact produced a false `SEM_AFF1_1`-carries-EPS15 example in GSE252378 (0 of 44 on a corrected re-run).
The seven survivors, named in full, because a count without identities is a claim
| GSM | series | title | characteristics | verdict |
|---|---|---|---|---|
| GSM5696327 | GSE173599/GSE173600 | HCB1_ChIP_MLL1c_DMSO_REP1 | antibody: MLL1 (c terminus) | identical; MLL1c vs MLL1 (c terminus) |
| GSM5696328 | GSE173599/GSE173600 | HCB1_ChIP_MLL1c_dVHL_REP1 | antibody: MLL1 (c terminus) | identical |
| GSM2212263 | GSE83671 | SEM_0umEPZ5676_Rep1 | treated with: DMSO for 7 days | identical; the title says zero micromolar EPZ-5676, which is the vehicle |
| GSM2212264 | GSE83671 | SEM_0umEPZ5676_Rep2 | treated with: DMSO for 7 days | identical |
| GSM2212265 | GSE83671 | SEM_0umEPZ5676_Rep3 | treated with: DMSO for 7 days | identical |
(GSM5696327 and GSM5696328 are counted twice because GSE173599 and GSE173600 both contain them - itself worth noting, since it means the 'series' denominator double-counts.) The original script's U2 says 'a mismatch I manufacture by failing to normalise is my defect, not the deposit's'. All seven are exactly that. They are listed rather than rounded away: the measured rate is 7/656 = 1.1%, and the claim that none is genuine rests on reading the five distinct records above, not on a threshold. 1
Why the error was invisible
Three things hid it, and each is generalisable.
- The shifted title is a real title of a real sample in the same experiment, so every 'contradiction' looked plausible - an H3K4me3 title against an H3K4me1 characteristics is exactly what a sloppy submission looks like. Nothing in the output looked malformed.
- The first sample in every file silently dropped out (its
curwas stillNone), which shrank the denominator by 32 without any message. A parser that drops records quietly is a parser whose denominator you cannot trust. - The corroborating 'third vote' was computed from the same shifted string. The audit reported that the parenthesised original filename inside each title sides with the title 39 to 0, then correctly noted the filename might not be independent and measured 156/156 agreement. Both figures are properties of the shifted title compared with its own embedded filename - self-consistent by construction whatever the shift. The check built to adjudicate the error could not see the error, because it was downstream of it. 1
What this does and does not overturn
It overturns the contradiction count and every conclusion drawn from it. It does not overturn the design of the original analysis, which is good: it pre-registered A1/A2/A3 and four ways of being uninformative, and U2 is precisely the clause that disposes of the seven survivors. The failure was not in the reasoning; it was one line beneath it. That is the uncomfortable part - a well-pre-registered analysis with a correct uninformativeness clause still shipped a false headline, because pre-registration disciplines the inference and not the parse.
| script | analyses/sem-arm-identity-from-signal/parser_offbyone_demo.py |
|---|---|
| status | run |
| genuine after tokeniser review | 0 |
| titles shifted by one | 2280 |
| titles correctly attributed | 0 |
| the fix | parse() must key the record on '^SAMPLE = GSM...', not on '!Sample_geo_accession', because '!Sample_title' precedes the accession line in a GEO SOFT record. |
| samples checked | 2312 |
| series checked | 32 |
| headline to quote | 7 of 656 comparable pairs (1.1%) after the boundary fix; 0 of those 7 survive a hand read |
| independently reproduced by | coordinator, 2026-09-23, same 7/656 and same 0/0 for GSE159608 |
| also corrected upstream | GSE252378: the 'SEM titled AFF1 carries EPS15' example was the same artefact; 0 of 44 antibody disagreements on a corrected run |
as published — the numbers behind the chart
{
"disagree": 96,
"comparable": 586,
"pct": 16.4
}with boundary fixed — the numbers behind the chart
{
"disagree": 7,
"comparable": 656,
"pct": 1.1
}- This entry is itself a count produced by code we wrote. The independent check is the five live GEO fetches in E2, and a reader who doubts the 2,280 should start there rather than with our script.
- Correcting a headline downward is not automatically progress: if there IS a mislabelled deposit somewhere, this correction makes the project less likely to look for it. The right reading is 'the instrument was broken', not 'GEO is clean'.
- The seven residuals are called artefacts on a hand read of five distinct records. Is 'MLL1c vs MLL1 (c terminus)' unambiguously agreement, or is there a submission in which MLL1c means something else? One record's wording is not a rule.
- How many other analyses in this repository parse GEO SOFT by field rather than by record boundary? A grep for '!Sample_geo_accession' as a record key would bound it.
- The corrected audit finds zero contradictions in 32 series. Is that because submitters are careful, or because the 32 cached series are dominated by two careful labs? A random draw of GEO series would answer it and would be a genuinely new measurement.
- Can a parser be made to fail loudly when it drops a record? Here 32 first-samples vanished with no message; a assert that every '^SAMPLE' line produced a record would have caught the whole thing.
- E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.dataset
Running the original parser and a boundary-corrected parser over the same 32 cached GEO SOFT dumps gives 2,280 shifted titles and 0 correct ones, and moves the disagreement count from 96/586 to 7/656.
sem-arm-identity-from-signal: parser_offbyone_demo.py and results_parser_offbyone.json (this repository) · Q140-sem-menin-arm · Oncofusion knowledge graph 2026What this evidence cannot show
The demonstration imports and runs the original `audit()` function unchanged, so the only difference between the two numbers is the record boundary. It is our own code checking our own code; the independent confirmation is E2, which does not use either parser.
Evidence that argues against this (2)
- The corrected rate of 0 genuine contradictions is an upper-bounded claim about the 32 cached series only, and about the antibody and treatment axes only. It does not license trusting GEO titles generally. Janssens DH et al. 2021
- The coordinator reproduced the corrected 7/656 independently on 2026-09-23 with their own fix. That is confirmation of the arithmetic, not of the hand-read verdict on the seven residuals - two independent runs of the same corrected logic share any remaining tokeniser defect, which is precisely what the seven residuals are. Janssens DH et al. 2021
- E2established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.dataset
Fetched live from GEO on 2026-09-23, GSM5139712 is titled SEM_H3K4me3_EPZ5676_AutoCUTnTag_Rep2 and its characteristics read 'with 30 uM EPZ-5676 for 72hrs'; GSM5139718 is titled SEM_IgG_DMSO_AutoCUTnTag_Rep2 with 'cut&tag antibody: IgG'; GSM4835705 is titled K562_IgG_AutoCUTnTag with 'cell type: K562'. All agree.
GSE159608: Automated CUT&Tag profiling of chromatin heterogeneity in mixed-lineage leukemia (419 samples, hg19 fragment BEDs) · Janssens DH, Meers MP, Wu SJ, Babaeva E, Meshinchi S, Sarthy JF, Ahmad K, Henikoff S · NCBI Gene Expression Omnibus 2021What this evidence cannot show
Five records were checked live by direct HTTP fetch of the GEO SOFT view, independently of any cached file and of both parsers. These are exactly the five records the earlier analysis named as its worked examples of contradiction.
Evidence that argues against this (2)
- This is a single live re-fetch by the same agent that diagnosed the bug, on the same day, with no second party checking the records. The coordinator verified the same three accessions independently against raw line numbers and agreed - but that is two readings of one file, not an independent measurement, and the original error survived exactly this kind of confirmation.
- Three agreeing samples do not establish that all 2,312 agree. The strong claim rests on the corrected script's aggregate, which was written by the author of the original bug and has been run by nobody else; the hand-read of three records bounds the failure mode, not the rate.
The 16.4% headline is an off-by-one; corrected, 7 of 656 pairs (1.1%) and none survives a hand read
The corrected counts come from the demonstration in the defect report
The deposit's sample labels are sound, so the SEM inventory in that entry can be read at face value