A checker for the error the citation checker cannot see
also called verify-claims · claim-abstract overlap · claim support
Our existing checker catches a citation pointing at the wrong paper. It cannot catch the opposite mistake, where the paper is right and the sentence attached to it says something the paper does not. This scores every claim against the abstract it cites.
The same thing, for a scientist
`verify-citations.py` compares stored title to real title and is blind by construction to a **correct PMID with a correct title attached to a claim the paper does not make** — the error counter-sweeps kept finding by hand, where an entry cites a paper's own caveat while contradicting its headline. `scripts/verify-claims.py` scores each claim against its abstract with IDF weighting over the abstract corpus, for the same reason the title checker needed it: every paper here shares 'leukaemia' and 'KMT2A'. **2,913 claims scored, median 0.687, 89 below a 0.12 floor.**
The control that says the scorer works
ana-reading-depth found that about a third of paper-citing items declare in their caveats that only the abstract was read. Those claims were written FROM the abstract, so they must score high if the scorer measures what it says. They do:
| band | n | median | 10th percentile | below floor |
|---|---|---|---|---|
| declared abstract-only | 704 | 0.778 | 0.512 | 2 |
| not declared | 2,209 | 0.647 | 0.232 | 87 |
The separation is the calibration. An instrument that scored noise would not put the abstract-written band a seventh of a point above the rest, and would not concentrate 87 of 89 floor-breaches in the band that draws on full text. It also means the two remaining low-scoring declared-abstract-only claims have no innocent explanation and should be read first — down from six, because four of the original six were never abstract-only at all (see the correction below).
What its first use found, and it is not a wrong citation
The FA-dependency hypothesis carried: humans with biallelic FA-gene loss develop KMT2A-rearranged leukaemia, so the requirement cannot be absolute. It ranked near the bottom of the queue.
The citation is correct and the claim is supported — the paper is a boy with Fanconi anaemia who developed a KMT2A-rearranged leukaemia. The low score is a faithful paraphrase in different vocabulary, exactly the false positive the script's own note predicts.
Reading it found something else. He had early pre-B ALL with a normal karyotype, was treated with 140 mg/m² of doxorubicin — a topoisomerase II inhibitor — and relapsed 38 weeks later as AML with t(11;16) MLL-CBP. The KMT2A rearrangement is therapy-related, arising by the standard topoisomerase II route, which is if anything easier in a DNA-repair-deficient child. So the case shows a person lacking FA function can ACQUIRE a KMT2A rearrangement, and not that FA function is dispensable for maintaining one — which is what the dependency claim is about. That entry now says so.
The instrument did its job by being wrong in a useful direction: it could not tell a paraphrase from a drift, and the reading it forced was worth doing anyway.
Correction: four of the six anomalies were the classifier, not the claims
Corrected 2026-09-19. The classifier behind this number had no negation handling: a caveat reading “Full text read (PMC4523210), not abstract only” contains the literal words “abstract only” and was counted as a declaration of shallow reading. 165 of 888 matches (18.6%) were negations of exactly this kind. The pattern now requires that no phrase asserting the full text WAS read appears, with a negative lookbehind so that “not read in full” is not itself cancelled. Found by reading the top item of the reading queue this number produced.
This entry's calibration control is built on the declared-abstract-only band, so the correction moves it — and moves it the right way:
| band | n | median | 10th pct | below floor |
|---|---|---|---|---|
| declared abstract-only (before) | 826 | 0.779 | 0.501 | 6 |
| declared abstract-only (after) | 704 | 0.778 | 0.512 | 2 |
| not declared (after) | 2,209 | 0.647 | 0.232 | 87 |
Four of the six floor-breaches disappeared, and that is exactly what a correct guard should do. An item whose agent read the full text writes a claim containing full-text detail, which by construction scores low against the abstract alone. Those items were never anomalies; they were mislabelled. The entry had called the six the ones with no innocent explanation. Four had the most innocent explanation available.
The separation between bands survives and is slightly cleaner, so the instrument's calibration claim stands. What does not survive is the confidence with which this entry named its own outliers.
- Read the two declared-abstract-only claims still below the floor. Four of the original six were classifier errors, which is the innocent explanation the entry said they lacked.
- Entailment, not overlap, is the measure this should be. A claim that inverts an abstract's meaning while reusing its words scores high here, and that is the failure the counter-sweeps actually kept finding.
- 33 PMIDs have no retrievable abstract at all, so 76 claims cannot be checked by any automated means and belong in the unreadable register.
- E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.analysis
scripts/verify-claims.py scores 2,913 paper-citing claims against their abstracts with IDF weighting over the abstract corpus: median 0.687, 10th percentile 0.281, 89 below a 0.12 floor. Abstracts were unavailable for 76 items across 33 of 1,156 PMIDs.
Claim-support check: IDF-weighted overlap between every claim and the abstract it cites · Oncofusion knowledge graph (internal tooling) 2026What this evidence cannot show
Run on 2026-09-22 over 858 entries. The floor is a chosen threshold, not a calibrated one; it was set to make the queue readable rather than from any error rate.
Evidence that argues against this (2)
- A low score is a reading queue and not a defect list, and the first item read was a false positive. Claims legitimately state what an abstract omits — methods, a figure, a number from results — and a faithful paraphrase in different vocabulary scores low. Any use of this output that skips the reading step reproduces the error it exists to catch. Oncofusion knowledge graph (internal tooling) 2026
- It is also blind in the direction that matters most. A claim that copies an abstract's vocabulary while reversing its meaning — citing a paper's caveat against its headline, which is the failure that motivated this — scores HIGH, because overlap is not entailment. The script catches drift in wording and not drift in direction. Oncofusion knowledge graph (internal tooling) 2026
- E2strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis
The declared-abstract-only band scores median 0.778 with 6 of 704 below floor, against median 0.647 and 77 of 2,209 for items not so declared — the separation predicted if the scorer measures claim-to-abstract correspondence.
Claim-support check: IDF-weighted overlap between every claim and the abstract it cites · Oncofusion knowledge graph (internal tooling) 2026What this evidence cannot show
The bands are defined by self-declaration, which `ana-reading-depth` shows is unevenly applied across cycles, so the comparison carries that unevenness.
Evidence that argues against this (1)
- The separation is partly circular by construction rather than evidence about the graph: a claim written from an abstract will share that abstract's vocabulary whether or not it represents the paper faithfully. The control shows the scorer responds to the thing it measures; it does not show that the thing it measures is support. Oncofusion knowledge graph (internal analysis) 2026
same blind spot from the other side: that checker scores evidence[].claim and never reads counter_evidence[].note, which is where all 76 truncations and all four imported facts lived
a third checker for a defect class the existing ones cannot see
grades and claim support are the two axes on which an evidence item is judged
same failure class one level down: the checker ran and was too weak, rather than being absent