Two quality checkers audited: one was skipping a quarter of its input, the other did not exist and failed its own defining case twice before it worked
also called checker audit cycle 59 · quoted-retraction check · accession coverage gap
We checked our own checking tools. One was quietly ignoring a quarter of what it was supposed to look at, and the other had to be rebuilt twice before it could catch the mistake it was written to catch.
The same thing, for a scientist
`verify-accessions.py` only examined entries carrying BOTH `data.uniprot` and `data.gene_symbol`, skipping 56 of 226 accession-bearing entries in silence; widened, all 226 check clean. `check-quoted-retractions.py` is new: it finds wording an entry says it retracted still standing as a live assertion elsewhere in the same entry - the cycle-58 failure that `check-stale-corrections.py` cannot see. Its first two designs scored 0 precision and then 0 recall on the case that motivated it; the working version matches on content-word overlap and pins its thresholds with a selftest that reproduces that case. Current graph: 17 hits, 3 in `claim` fields, and **all three adjudicated as topic overlap with claims that are already correct** - no surviving retracted claim in the graph.
The accession checker was skipping a quarter of its input
The selection line was if acc and sym. An entry carrying a UniProt accession but not asserting which gene it is was dropped without a word, and the run still printed a clean checked 159 mismatches 0.
56 of 226 accession-bearing entries - a quarter - had never been checked, including every phase-2 target dossier and twenty-three gene-* entries carrying an explicit length_aa that could have been verified at any time.
Widened, the checker now reports what it actually looked at:
checking 226 accessions against UniProt
170 with a gene_symbol asserted by the entry
23 with the symbol inferred from a `gene-` id
33 resolved and length-checked only (no symbol to check against)
checked 226 mismatches 0 not found 0
The 56 were all correct. That is the boring outcome and it is worth stating plainly: the defect was in the instrument, not in the data. What changed is that the clean report is now earned rather than assumed, and it says which of the three grades of check each accession received.
The new check, and the two ways it failed first
Cycle 58 found mech-breakpoint-position-consequences carrying a correction section that read "This entry originally stated that roughly half of KMT2A::AFF1 cases break upstream of AFF1 exon 1" while an evidence claim in the same entry still said "about half of KMT2A::AFF1 cases break upstream of AFF1 exon 1 producing no reciprocal allele". check-stale-corrections.py cannot see this: it looks for absence assertions near a correction, on the assumption that a correction fixes the claim it sits beside.
First design: match quoted wording exactly. Precision 0 of 7 on claim fields - every hit was a trial-registry quotation or UniProt's protein name for SIRT1, text a correction quotes as a source rather than as its own retracted words.
Second design: require the quote to follow the retraction phrase. Precision improved and recall went to zero on the case the check exists for. Run against that entry as it stood at commit b4fadcd, it reported nothing. The reason is that the retraction paraphrases - "roughly half" against "about half", "producing a spliced fusion with no reciprocal allele" against "producing no reciprocal allele". Exact substring matching is the wrong instrument for text a human rewrote.
Working design: content-word overlap. Take the clause after the retraction phrase, quoted or not, reduce it to content words keeping every number and identifier, and flag any sentence elsewhere in the entry covering at least 60% of them on at least 5 words. The thresholds are pinned by a selftest() that reproduces the cycle-58 case in the exact wording both halves had, plus a benign case it must not fire on. The script refuses to run if the selftest fails, so a future edit cannot quietly return it to the state where it reported a clean graph while missing the only defect of its kind.
What it finds now, and what it cannot tell you
17 hits across 1,080 entries; 3 in claim fields; 0 real defects. All three were read:
| entry | verdict |
|---|---|
fusion-bcr-abl1 | already fixed, and better than the retraction - the claim correctly splits 2HYY (human, P00519) from 1IEP (murine, P00520), which is what the correction was about |
risk-menin-pocket-occupancy-endocrine-late-effects | different content - retracted clause is femur growth-plate closure, surviving sentence is parathyroid hormone and phosphate |
mech-breakpoint-position-consequences | fixed in cycle 58; the corrected claim shares vocabulary with the clause it replaced, which is unavoidable |
The honest limitation is in that last row. A claim corrected in place necessarily talks about the same subject in the same words as the claim it replaced, so this check cannot separate still wrong from now right about the same thing. Only a human read does that, and it takes seconds - which is the same bargain check-stale-corrections.py offers and the reason both are triage rather than verdict.
Three false-positive classes were removed on principle rather than by tuning: fields whose names mean they hold retracted text (withdrawn_claims), sentences that are themselves corrections, and previously claimed / previously reported used adjectivally about the literature - "the previously claimed favourable prognosis of t(9;11)" is the field describing published history, not the entry retracting itself. A determiner in front is the tell.
- Work the `check-stale-corrections.py` queue: 81 candidates, 40 before their correction. It needs reading, not tooling, and it has been deferred for several cycles.
- How many corrections in this graph are written WITHOUT a self-referential phrase? Those are invisible to this check, and their number bounds its recall.
- Should a corrected `claim` field be required to carry an inline marker, so that a reader who meets only the claim knows a correction exists? That would make the defect structurally impossible rather than detectable after the fact.
- E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis
verify-accessions.py examined only entries carrying both `data.uniprot` and `data.gene_symbol`, silently skipping 56 of 226 accession-bearing entries. Widened to resolve every accession, all 226 check clean with 0 mismatches and 0 not found.
verify-accessions.py - UniProt accession verification across the graph, widened to entries carrying an accession without an asserted gene symbol · Oncofusion knowledge graph (internal tooling) 2026What this evidence cannot show
Run 2026-09-19. The 33 entries with no symbol to check against get the weakest grade of check - the accession resolves and the length matches where a length is recorded, but nothing asserts which protein the entry means, so a plausible wrong accession would still pass. Twenty-three more have the symbol inferred from a `gene-<symbol>` id, which is an assertion the id makes rather than one the entry makes deliberately.
Evidence that argues against this (1)
- The widened run found nothing, so the practical value here is bounded: this corrected an instrument, not a fact. The argument for doing it anyway is the NOMO-1 incident - a cell-line accession that resolved to an endometrial carcinoma from a 71-year-old - which is exactly the failure an unchecked accession hides. Oncofusion knowledge graph (internal tooling) 2026
- E2strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis
A check for retracted wording still asserted elsewhere in the same entry failed twice before working: matching quoted text exactly gave 0 of 7 precision on claim fields, and requiring the quote to follow the retraction phrase gave 0 recall on the case the check was written for, because that retraction paraphrases rather than quotes.
check-quoted-retractions.py - finding retracted wording that is still asserted elsewhere in the same entry · Oncofusion knowledge graph (internal tooling) 2026What this evidence cannot show
Recall was measured against a single case - the cycle-58 entry as it stood at commit b4fadcd - because it is the only confirmed instance of this defect in the graph's history. **A recall estimate from one example is not a recall estimate.** The content-word threshold of 0.60 was chosen to clear that one case at 0.71 and has no other calibration behind it.
Evidence that argues against this (2)
- The second design's failure is the more instructive one and it argues against the whole family: if corrections paraphrase, then any string-matching check will have unknown recall, and a clean report from it means very little. The check earns its place by pinning one known case in a selftest, not by any claim to completeness. Oncofusion knowledge graph (internal tooling) 2026
- `check-stale-corrections.py` already reports 81 candidates, 40 of them sitting before their correction, and those have not been read. Building a second checker before working the first one's queue is a defensible choice only because this one finds a different defect class - but it does not reduce that backlog by one. Oncofusion knowledge graph (internal tooling) 2026
- E3moderate confidence 3 of 5: One strong study, or several smaller consistent ones.analysis
Across 1,080 entries the check reports 17 survivals, 3 of them in evidence `claim` fields, and all three are topic overlap with claims that are already correct. No retracted claim currently survives anywhere in the graph.
check-quoted-retractions.py - finding retracted wording that is still asserted elsewhere in the same entry · Oncofusion knowledge graph (internal tooling) 2026What this evidence cannot show
'Already correct' is this coordinator's own adjudication of three entries, two of which were corrected by this coordinator in the preceding cycle. **That is a self-assessment and should be read as one.** The check also only sees entries whose corrections are worded self-referentially; a correction written as a plain restatement, with no phrase like 'this entry previously said', is invisible to it.
Evidence that argues against this (1)
- A zero from an instrument whose recall is unknown is weak evidence of absence. The stronger statement available from this run is the negative one: **63 entries in this graph retract their own earlier wording**, which is a measure of how often facts here turn out to be wrong, and it is larger than the number of defects the check found. Oncofusion knowledge graph (internal tooling) 2026
third instance of a checker reporting a number nobody had read to ground
a third checker for a defect class the existing ones cannot see
reproduces this entry's cycle-58 defect as the selftest that pins the thresholds
same failure class: a checker that ran, reported clean, and was too weak to mean it
adversarial self-check discipline applied to a new checker before trusting its output
both measure whether a quality number is reporting what its name implies
another instrument whose flags mostly dissolved on reading
the author check added there is what made group clustering possible