Oncofusion

start / graph / ana-two-checkers-that-were-not-checking

Analysisunrated confidence 0 of 5: unrated

Two quality checkers audited: one was skipping a quarter of its input, the other did not exist and failed its own defining case twice before it worked

also called checker audit cycle 59 · quoted-retraction check · accession coverage gap

We checked our own checking tools. One was quietly ignoring a quarter of what it was supposed to look at, and the other had to be rebuilt twice before it could catch the mistake it was written to catch.

The same thing, for a scientist

`verify-accessions.py` only examined entries carrying BOTH `data.uniprot` and `data.gene_symbol`, skipping 56 of 226 accession-bearing entries in silence; widened, all 226 check clean. `check-quoted-retractions.py` is new: it finds wording an entry says it retracted still standing as a live assertion elsewhere in the same entry - the cycle-58 failure that `check-stale-corrections.py` cannot see. Its first two designs scored 0 precision and then 0 recall on the case that motivated it; the working version matches on content-word overlap and pins its thresholds with a selftest that reproduces that case. Current graph: 17 hits, 3 in `claim` fields, and **all three adjudicated as topic overlap with claims that are already correct** - no surviving retracted claim in the graph.

Go deeper
The accession checker was skipping a quarter of its input

The selection line was if acc and sym. An entry carrying a UniProt accession but not asserting which gene it is was dropped without a word, and the run still printed a clean checked 159 mismatches 0.

56 of 226 accession-bearing entries - a quarter - had never been checked, including every phase-2 target dossier and twenty-three gene-* entries carrying an explicit length_aa that could have been verified at any time.

Widened, the checker now reports what it actually looked at:

checking 226 accessions against UniProt
  170 with a gene_symbol asserted by the entry
   23 with the symbol inferred from a `gene-` id
   33 resolved and length-checked only (no symbol to check against)

checked 226  mismatches 0  not found 0

The 56 were all correct. That is the boring outcome and it is worth stating plainly: the defect was in the instrument, not in the data. What changed is that the clean report is now earned rather than assumed, and it says which of the three grades of check each accession received.

The new check, and the two ways it failed first

Cycle 58 found mech-breakpoint-position-consequences carrying a correction section that read "This entry originally stated that roughly half of KMT2A::AFF1 cases break upstream of AFF1 exon 1" while an evidence claim in the same entry still said "about half of KMT2A::AFF1 cases break upstream of AFF1 exon 1 producing no reciprocal allele". check-stale-corrections.py cannot see this: it looks for absence assertions near a correction, on the assumption that a correction fixes the claim it sits beside.

First design: match quoted wording exactly. Precision 0 of 7 on claim fields - every hit was a trial-registry quotation or UniProt's protein name for SIRT1, text a correction quotes as a source rather than as its own retracted words.

Second design: require the quote to follow the retraction phrase. Precision improved and recall went to zero on the case the check exists for. Run against that entry as it stood at commit b4fadcd, it reported nothing. The reason is that the retraction paraphrases - "roughly half" against "about half", "producing a spliced fusion with no reciprocal allele" against "producing no reciprocal allele". Exact substring matching is the wrong instrument for text a human rewrote.

Working design: content-word overlap. Take the clause after the retraction phrase, quoted or not, reduce it to content words keeping every number and identifier, and flag any sentence elsewhere in the entry covering at least 60% of them on at least 5 words. The thresholds are pinned by a selftest() that reproduces the cycle-58 case in the exact wording both halves had, plus a benign case it must not fire on. The script refuses to run if the selftest fails, so a future edit cannot quietly return it to the state where it reported a clean graph while missing the only defect of its kind.

What it finds now, and what it cannot tell you

17 hits across 1,080 entries; 3 in claim fields; 0 real defects. All three were read:

entryverdict
fusion-bcr-abl1already fixed, and better than the retraction - the claim correctly splits 2HYY (human, P00519) from 1IEP (murine, P00520), which is what the correction was about
risk-menin-pocket-occupancy-endocrine-late-effectsdifferent content - retracted clause is femur growth-plate closure, surviving sentence is parathyroid hormone and phosphate
mech-breakpoint-position-consequencesfixed in cycle 58; the corrected claim shares vocabulary with the clause it replaced, which is unavoidable

The honest limitation is in that last row. A claim corrected in place necessarily talks about the same subject in the same words as the claim it replaced, so this check cannot separate still wrong from now right about the same thing. Only a human read does that, and it takes seconds - which is the same bargain check-stale-corrections.py offers and the reason both are triage rather than verdict.

Three false-positive classes were removed on principle rather than by tuning: fields whose names mean they hold retracted text (withdrawn_claims), sentences that are themselves corrections, and previously claimed / previously reported used adjectivally about the literature - "the previously claimed favourable prognosis of t(9;11)" is the field describing published history, not the entry retracting itself. A determiner in front is the tell.

Still open
  1. Work the `check-stale-corrections.py` queue: 81 candidates, 40 before their correction. It needs reading, not tooling, and it has been deferred for several cycles.
  2. How many corrections in this graph are written WITHOUT a self-referential phrase? Those are invisible to this check, and their number bounds its recall.
  3. Should a corrected `claim` field be required to carry an inline marker, so that a reader who meets only the claim knows a correction exists? That would make the defect structurally impossible rather than detectable after the fact.
Every source behind this entry (3)
  1. E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis

    verify-accessions.py examined only entries carrying both `data.uniprot` and `data.gene_symbol`, silently skipping 56 of 226 accession-bearing entries. Widened to resolve every accession, all 226 check clean with 0 mismatches and 0 not found.

    What this evidence cannot show

    Run 2026-09-19. The 33 entries with no symbol to check against get the weakest grade of check - the accession resolves and the length matches where a length is recorded, but nothing asserts which protein the entry means, so a plausible wrong accession would still pass. Twenty-three more have the symbol inferred from a `gene-<symbol>` id, which is an assertion the id makes rather than one the entry makes deliberately.

    Evidence that argues against this (1)
    • The widened run found nothing, so the practical value here is bounded: this corrected an instrument, not a fact. The argument for doing it anyway is the NOMO-1 incident - a cell-line accession that resolved to an endometrial carcinoma from a 71-year-old - which is exactly the failure an unchecked accession hides. Oncofusion knowledge graph (internal tooling) 2026
  2. E2strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis

    A check for retracted wording still asserted elsewhere in the same entry failed twice before working: matching quoted text exactly gave 0 of 7 precision on claim fields, and requiring the quote to follow the retraction phrase gave 0 recall on the case the check was written for, because that retraction paraphrases rather than quotes.

    What this evidence cannot show

    Recall was measured against a single case - the cycle-58 entry as it stood at commit b4fadcd - because it is the only confirmed instance of this defect in the graph's history. **A recall estimate from one example is not a recall estimate.** The content-word threshold of 0.60 was chosen to clear that one case at 0.71 and has no other calibration behind it.

    Evidence that argues against this (2)
    • The second design's failure is the more instructive one and it argues against the whole family: if corrections paraphrase, then any string-matching check will have unknown recall, and a clean report from it means very little. The check earns its place by pinning one known case in a selftest, not by any claim to completeness. Oncofusion knowledge graph (internal tooling) 2026
    • `check-stale-corrections.py` already reports 81 candidates, 40 of them sitting before their correction, and those have not been read. Building a second checker before working the first one's queue is a defensible choice only because this one finds a different defect class - but it does not reduce that backlog by one. Oncofusion knowledge graph (internal tooling) 2026
  3. E3moderate confidence 3 of 5: One strong study, or several smaller consistent ones.analysis

    Across 1,080 entries the check reports 17 survivals, 3 of them in evidence `claim` fields, and all three are topic overlap with claims that are already correct. No retracted claim currently survives anywhere in the graph.

    What this evidence cannot show

    'Already correct' is this coordinator's own adjudication of three entries, two of which were corrected by this coordinator in the preceding cycle. **That is a self-assessment and should be read as one.** The check also only sees entries whose corrections are worded self-referentially; a correction written as a plain restatement, with no phrase like 'this entry previously said', is invisible to it.

    Evidence that argues against this (1)
    • A zero from an instrument whose recall is unknown is weak evidence of absence. The stronger statement available from this run is the negative one: **63 entries in this graph retract their own earlier wording**, which is a measure of how often facts here turn out to be wrong, and it is larger than the number of defects the check found. Oncofusion knowledge graph (internal tooling) 2026
Connected to