Oncofusion

start / graph / ana-the-year-check-would-have-caught-seventy-percent-of-the-errors-this-project-recorded

Analysismoderate confidence 3 of 5: One strong study, or several smaller consistent ones.

Validating the wrong-paper year check against this project's own recorded identifier errors - 16 of 23 would have been flagged, 7 would have passed silently on a shared year, and the figure is biased optimistic because every pair in the set is an error somebody caught

also called the year check catches 70% of recorded errors · V1, weakly · the validation set is the corpus's own confessions · caught errors are not escaped errors

A check built last cycle compares the year each note records against the year of the paper its link actually opens, to catch a working link pointing at the wrong paper. This asks whether that check would have caught the mistakes this project has already made and written down. It would have caught about seven in ten — and the three it would have missed are the dangerous kind, where the wrong paper happens to share a year with the right one.

The same thing, for a scientist

**Statement.** The cycle-123 year check reported 92.0% exact agreement across 16,420 citation blocks and bounded *"right identifier, wrong paper"* from above. **Its own counter-evidence said the bound was untested:** > *"the two documented historical failures in this project would need to be re-checked to know whether this instrument would have caught either of them — that test was not run and would have been the honest validation."* **A caveat that names a measurable gap is an unexecuted task, and this project has paid for that six times. Executed the cycle after writing it.** **The validation set is the corpus's own confessions.** This project records near-misses in prose and in `identifier_errors_caught_in_this_beat` fields — *"the DOI this agent would have written from recall, `10.1101/gad.1594107`, is wrong; the fetched value is `10.1101/gad.1574507`"*. **Each is a (wrong id, right id) pair with ground truth attached, written down before anyone thought of building a checker.** | | | |---|---:| | recorded confessions carrying two identifiers | 23 | | **would have been caught** (years differ by 2+) | **16 (70%)** | | **would have passed silently** | **7** | **`V1` fires — validated, weakly.** Examples it catches are stark: `gene-chaf1b` has a 20-year spread, `gene-xpo1` 13, `ana-two-unrelated-menin-inhibitors-converge` 14. The ones it misses are quiet: `mechanism-ctcf-primes-repressive-chromatin` recorded a wrong DOI whose paper was published in **the same year** as the right one — `10.1186/s13148-019-0758-2` against `10.1186/s13148-019-0764-4`, both 2019, **adjacent identifiers in the same journal and volume.** 🔴 **`U1` bites hard and it runs the wrong way.** Every pair here is an error a human **caught**. An error that survives review is *more likely* to be a plausible same-year substitute — **exactly the case this instrument cannot see.** So 70% is an **upper** bound on the true catch rate, and the residual risk is concentrated in precisely the errors most likely to be in the corpus now.

Go deeper
The misses are a recognisable class, and it is the dangerous one

The seven silent passes are not random. The clearest is two adjacent DOIs in the same journal, volume and year — s13148-019-0758-2 and s13148-019-0764-4, six apart in the publisher's sequence. A digit slip inside one journal-year is invisible to a year comparison by construction, and it is also the easiest transcription error to make.

🔵 That gives the next check its specification, for free: compare the resolved authors against the entry's stored authors string. Two adjacent papers in one journal-year almost never share a first author, and doi-index.json already caches the resolved title for 3,424 DOIs — so the fetch cost is zero and only a surname comparison is missing.

🟠 What this does NOT license: a 70% catch rate on caught errors says nothing reliable about the rate of uncaught errors in the committed corpus, which remains unmeasured and is the number anyone actually wants.

Scoring my own instrument on a set I assembled

U3 was pre-registered: I wrote the pair extractor, I knew what the instrument does, and I scored it myself. The extractor was written before any pair was scored, which is worth something and is not the same as blind.

The honest reading of the whole exercise is that the corpus turned out to contain a usable validation set nobody designed — 23 labelled errors, produced as a by-product of a discipline adopted for a different reason. That is the most reusable thing here, and it generalises: any project that records its corrections in a machine-readable place accumulates a test set for free.

🔴 It is also a small set. n = 23, with a wide interval, and no figure from it should be quoted without the n beside it.

Facts and figures
questionwould the cycle-123 year check have caught this project's own recorded identifier errors?
verdictV1 - validated, weakly, and the figure is biased optimistic
pairs with two resolvable years23
would have been caught16
catch rate PCT70
would have passed silently7
the miss classa digit slip inside one journal-year - mechanism-ctcf-primes-repressive-chromatin recorded 10.1186/s13148-019-0758-2 for 10.1186/s13148-019-0764-4, both 2019, adjacent in the same journal's sequence
why the bias runs optimisticevery pair is an error a human CAUGHT; an error that survives review is likelier to be a plausible same-year substitute, the case this instrument cannot see
the next check is specified for freecompare resolved AUTHORS against the stored authors string - two adjacent papers in one journal-year almost never share a first author, and the resolved records are already cached
the reusable findingthe corpus contained a usable validation set nobody designed: 23 labelled errors produced as a by-product of a discipline adopted for another reason
absence classmeasured-not-curated
bite backThis entry validates an instrument at 70% and the number is an upper bound on caught-error-like failures, not a statement about the corpus. The rate of UNCAUGHT wrong-paper citations in the committed graph remains completely unmeasured, and that is the number a reader will assume this entry provides.
Still open
  1. Add the authors axis. It is specified, the resolved records are cached, and it is the only proposed check that addresses the miss class - a digit slip inside one journal-year, which no year comparison can see.
  2. Measure the rate of UNCAUGHT wrong-paper citations. Every number in this project's citation hygiene work is about errors somebody already found; nobody has estimated what is still in the corpus, and that is the number that matters.
  3. Record confessions in a structured field rather than in prose. 23 pairs were recoverable by text heuristic; an `identifier_corrections` field would make the validation set exact and would let any future checker exclude them automatically.
  4. Re-run this against the PMID side. The set here is dominated by DOI pairs and the same question applies to the PMIDs, where this project's two documented wrong-paper failures actually occurred.
Every source behind this entry (1)
  1. E1moderate confidence 3 of 5: One strong study, or several smaller consistent ones.analysis

    Scanning every committed node file for sentences in which the corpus records an identifier it got wrong - matching phrases including 'is wrong', 'were wrong', 'from recall', 'written from memory', 'by inference', 'the fetched record', 'the true DOI' and 'identifier_errors' - and keeping only those carrying two or more distinct identifiers of the same kind, yields 23 pairs whose years both resolve against PubMed esummary or the Crossref REST API. Of these, 16 have a spread of two years or more between the wrong and the right identifier and would therefore have been flagged by the cycle-123 year check, and 7 have a spread below two years and would have passed silently. Among the misses, mechanism-ctcf-primes-repressive-chromatin records 10.1186/s13148-019-0758-2 against 10.1186/s13148-019-0764-4, both published in 2019 in the same journal. Among the catches, gene-chaf1b shows a 20-year spread, ana-two-unrelated-menin-inhibitors-converge 14 years and gene-xpo1 13 years.

    What this evidence cannot show

    THESE ARE CAUGHT ERRORS, NOT ESCAPED ONES, AND THE BIAS RUNS OPTIMISTIC. Every pair in this set was noticed by a human before it was committed; an error that survives review is more likely to be a plausible same-year substitute, which is exactly the case the instrument cannot see, so 70% is an UPPER bound on the true catch rate. n IS 23, with a wide interval, and no figure should be quoted without it. I WROTE THE EXTRACTOR, I KNOW WHAT THE INSTRUMENT DOES AND I SCORED IT MYSELF - the extractor was written before any pair was scored, which is not the same as blind. The pair extraction is a text-window heuristic and will both miss confessions phrased differently and occasionally pair a wrong identifier with an unrelated one mentioned nearby; pairs whose years did not both resolve were excluded rather than guessed. A two-year threshold was used because the cycle-123 check found 18 of its 22 flags were legitimate online-versus-print splits.

    Evidence that argues against this (2)
    • A 70% catch rate sounds like a pass and the more useful reading is the opposite: the 30% it misses is a COHERENT CLASS - a digit slip inside one journal-year - which is both the easiest transcription error to make and completely invisible to this check. The instrument is weakest exactly where the error is most likely, and reporting the headline without that structure would be misleading.
    • The set may not be independent of the instrument in the way this entry implies. These confessions were written by agents following a brief that told them to verify identifiers by fetching, so the errors that got recorded are the ones that fetching catches - which correlates with having different metadata, including a different year. The validation set and the instrument may share a bias, and nothing here rules that out.
Connected to