Oncofusion

start / graph / ana-a-wrong-pmid-is-undetectable-here-because-no-field-holds-the-cited-papers-title

Analysisstrong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.

Testing whether every citation points at the paper it claims: 314 scoreable blocks, 242 scoring below 0.3 on title overlap, and every low scorer read is CORRECT - because this project's citation title is a descriptive sentence by house style, which means a PMID pointing at the wrong paper is undetectable by any automated check over the corpus alone

also called a wrong PMID is undetectable here · the citation schema stores no paper title · M3 - the metric measured house style · precision zero, and the negative is the finding

Last cycle this project confirmed every one of its 1,908 cited paper identifiers points at a real paper. That is not the same as pointing at the RIGHT paper, and this project has twice cited an identifier that led somewhere real and entirely unrelated. Testing for that turns out to be impossible with what is stored: the project's notes record a description of why a paper matters, never the paper's own title, so there is nothing to compare against. The fix is cheap and is now specified.

The same thing, for a scientist

**Statement.** Cycle 118 found 1,908 of 1,908 cited PMIDs resolve to a real paper, and its own counter-evidence named the weakness: *"it tests that an identifier exists, not that it is the right one — and the two documented failures in this project were both PMIDs that resolved perfectly to the WRONG paper."* This runs that check. **`M3` fires — neither pre-registered outcome — and the structural finding is the output.** | | | |---|---:| | citation blocks carrying both a PMID and a title | **314** | | scoring below 0.3 on title overlap | **242 (77%)** | | of those read, actually wrong | **0** | **Every low scorer is a correct citation whose stored `title` is a descriptive sentence.** `protein-eps15-domains` stores *"EPS15 / AF1p — domain architecture, and the one coordinate nobody has published"* against a live *"A novel gene, AF-1p, fused to HRX in t(1;11)…"* — the citation is right; the stored title was never meant to be the paper's. **So the consequence is structural: THE CITATION SCHEMA HAS NO FIELD HOLDING THE CITED PAPER'S TITLE.** There is nothing for a wrong PMID to disagree with. **Both documented failures here had to be caught by a human reading the abstract**, and so did three agent near-misses in the last four cycles — a *Trypanosoma brucei* paper, an asciminib paper and a cheese-whey biohydrogen paper. **The fallback detector failed too, and its failure is reported rather than buried.** Asking whether the live paper carries *any* subject vocabulary flags **15 of 314 — and all 15 are correct citations** whose titles simply lack the words in my list (*"A proteome-scale map of the human interactome network"*, *"Missense mutations in PML-RARA…"*). **Precision zero, no true positives.** It is not usable as written. **Its negative is still worth having:** nothing in the committed record points at an obviously unrelated paper. Combined with cycle 118's 100% resolution, the committed citation record survives both checks that can currently be run on it. **THE FIX IS CHEAP AND SPECIFIED.** `citation-index.json` already holds all 1,908 live titles. **Storing the live title beside each PMID converts an undetectable error class into a one-line diff** — and makes every future citation self-checking.

Go deeper
My first version of this check was broken, and its own U3 said how

The first run reported 89.1% below 0.3 and would have read as an alarming defect rate. It was scoring entry titles and analogy headings against whichever PMID appeared first anywhere in the same file, because the walker yielded every dict carrying a title — including the node's root dict.

U3 named the assumption before the run — "one PMID per block is assumed" — and it was violated for most blocks. Restricting to dicts carrying the citation schema, with the PMID in their own scalar fields, cut 1,339 blocks to 314 and the low-scorer rate from 89.1% to 77%.

The pre-registration caught this, not vigilance. Writing U3 down before running is what made an implausible number readable as an instrument failure rather than a finding — the same pattern as the U3 in cycle 109 that fired unnoticed because it was checked the wrong way.

What a 77% low-overlap rate does and does not mean

It means the metric is wrong for this corpus, not that the corpus is wrong. Token overlap presumes the stored string is an attempt at the paper's title. Here it is an attempt at why the paper matters, which is arguably the more useful thing to store — it is just not checkable.

And the check has low sensitivity by construction (U2): two unrelated papers on the same topic share many tokens, so a high score never proves correctness. Any count this produces is a lower bound, which is why 0 wrong out of 242 read is reassuring rather than conclusive.

The honest summary is that this corpus currently has no automated defence against a wrong PMID, has never had one, and has caught both historical instances by reading. That is a worse position than the 100% resolution rate suggested and a better one than it would be if the reading discipline were not working.

Facts and figures
questiondoes every citation point at the paper the entry claims?
scoreable citation blocks314
blocks with title but no pmid26707
low scorers read and found wrong0
the structural findingthe citation schema has NO field holding the cited paper's title, so a PMID pointing at the wrong paper is undetectable by any automated check over the corpus alone; both documented failures were caught by reading
the fixcitation-index.json already holds all 1,908 live titles; storing the live title beside each PMID makes every future citation self-checking
my own instrument failurethe first version scored 1,339 blocks at 89.1% below 0.3 by comparing entry titles against unrelated PMIDs from the same file; U3 had named the assumption before the run
absence classmeasured-not-curated
bite backThis entry reports that the committed citation record passes both checks that can be run on it. Both checks are weak - one has zero precision, the other zero sensitivity to same-topic errors - and a reader taking 'passes' as 'clean' would be drawing a conclusion the entry's own caveats forbid.
overlap distribution
>=0.71.000>=0.7: 1>=0.53.000>=0.5: 3>=0.328.0>=0.3: 28>=0.940.0>=0.9: 40<0.3242.0<0.3: 242
overlap distribution — the numbers behind the chart
{
  ">=0.9": 40,
  ">=0.7": 1,
  ">=0.5": 3,
  ">=0.3": 28,
  "<0.3": 242
}
off domain detector
{
  "flagged": 15,
  "of": 314,
  "true_positives": 0,
  "precision": 0,
  "reading": "not usable as written; its negative is informative but weak"
}
Still open
  1. Store the live title beside each PMID. The index already holds all 1,908; this converts an undetectable error class into a one-line diff and is the single cheapest hygiene improvement available.
  2. Check that each cited paper SUPPORTS the claim it is attached to. That is the thing that matters and no check here touches it; abstract-level indexing plus a claim-to-abstract comparison is the next rung.
  3. 26,707 blocks carry a title with no PMID and are outside this audit entirely - including every DOI-only and URL-only citation. Resolving the 3,440 DOIs would bring most of them in scope.
  4. Widen or abandon the off-domain vocabulary. At 15 flags and zero true positives it is currently worse than nothing, because a reader may mistake its silence for evidence.
Every source behind this entry (1)
  1. E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis

    Across 2,603 committed node files, 314 dicts carry this project's citation schema (a title plus at least one of authors/venue/url/type) together with a PMID in their own scalar fields. Comparing each stored title against the live PubMed title by token containment gives 40 blocks at or above 0.9, 1 at or above 0.7, 3 at or above 0.5, 28 at or above 0.3 and 242 below 0.3. Reading the low scorers shows stored titles that are descriptive sentences rather than paper titles, with correct underlying citations. A detector asking whether the live paper title carries any subject vocabulary flags 15 of 314, all of which are correct citations. An earlier version of the same script scored 1,339 blocks with 89.1% below 0.3 because it compared entry-level titles against unrelated PMIDs from the same file.

    What this evidence cannot show

    TOKEN OVERLAP IS NOT MEANING AND THIS CHECK HAS LOW SENSITIVITY BY CONSTRUCTION. Two unrelated papers on the same topic share many tokens, so a high score never proves a citation correct - it only fails to detect an error. Every count here is therefore a LOWER bound on wrong citations, and '0 wrong of 242 read' means 0 DETECTED. The 242 were read by the coordinator at the level of stored-title versus live-title plausibility, NOT by checking that each paper supports the claim it is attached to, which is the thing that actually matters and was not done. The off-domain vocabulary list is the coordinator's and its 15 flags are all false positives, so its own precision is zero and its negative result is correspondingly weak. Only PMID-bearing citations are scoreable: 26,707 blocks carry a title without a PMID and are entirely outside this audit, including every internal analysis and every DOI-only or URL-only citation.

    Evidence that argues against this (3)
    • The reassuring half of this entry rests on a reading I performed on my own work, with no second reader and no record of which 242 were examined in what depth. 'Every low scorer read is correct' is a claim about my attention, and the entry presents it with the same confidence as the counts, which are machine-produced and reproducible. The two are not equally reliable.
    • Diagnosing this as a schema problem makes it someone else's fault. An equally available reading is that storing a descriptive title is correct and the real gap is that nobody re-reads citations after writing them - in which case adding a title field creates a second thing to maintain and drift out of date, and the historical failures would STILL have been caught only by reading. The fix proposed here is cheap precisely because it is mechanical, which is not the same as being the right fix.
    • This is the third consecutive coordinator beat about the project's own instruments rather than about leukaemia, and the second in two cycles whose headline is a defect in a tool built the cycle before. The corpus gained one entry about citation hygiene while two agents did the biology. An honest accounting is that the coordinator has been auditing rather than researching for some time now.
Connected to