Oncofusion

start / graph / ana-the-absence-recheck-is-unvalidated-because-its-only-test-case-is-one-it-cannot-solve

Analysisstrong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.

Re-checking all 60 bounded-absence claims against the repaired citation index returns zero strong candidates - and that null is not reassurance, because the one absence known to be false ranks first for the wrong reason and its actual refutation is a deposit the method cannot see

also called R3 - the absence re-check is unvalidated · a null from an untested instrument · zero candidates is not a clean bill of health · the same shape as the 100% resolution rate

Two of this project's 'nobody has done this' claims turned out to be wrong last cycle. This checked the other fifty-eight against every paper the project cites, and found no strong matches — which sounds like good news. It is not: the one claim we already know was wrong is one this method is incapable of catching, so a clean result from it means very little.

The same thing, for a scientist

**Statement.** After cycle 120, at least two absence claims in this graph were false and **nobody knew the rate**. This measures what can be measured: all **60** `absence-` entries scored against the live PubMed titles of all **4,000** PMIDs this corpus cites, with each entry's own citations excluded so an absence cannot be refuted by its own evidence. | | | |---|---:| | absence entries scored | **60** | | with a candidate at >= 0.5 subject overlap | **0** | | 0.3 - 0.5 | **2** | | below 0.3 | **58** | **`R3` fires: neither pre-registered outcome, and the finding is that the instrument is unvalidated.** The one absence here *known* to be false — `absence-no-menin-inhibitor-occupancy-dataset-in-an-infant-derived-kmt2a-aff1-line` — **ranks first of 60**. That looks like validation and is not. Its top candidates share the words *infant, inhibitor, model* with a **BET-inhibitor mouse paper**, which has nothing to do with why it was false. **Its actual refutation was GSE159608, a deposit, and `U4` recorded before the run that this method cannot see deposits at all.** **So for the single case with a known answer, the method gets the rank right for the wrong reason and surfaces nothing usable. A null result from an instrument that provably cannot detect the one failure it was tested against is not evidence that the absences are sound** — it is the same shape as cycle 118's *100% of PMIDs resolve*, a confident number on an axis nobody tested. **The verdict named the fix and the fix was executed the same cycle** rather than recorded as an open question: see `ana-this-corpus-cites-504-deposits-and-a-paper-index-cannot-see-any-of-them`. It surfaced **GSE252378** — the infant menin-inhibitor experiment — at a score of **0.333**, **below any threshold that would have flagged it automatically.**

Go deeper
What a worklist is for, and why no count of false absences appears here

U1 was pre-registered: a title match is a CANDIDATE, never a refutation. A paper about single-cell KMT2A does not refute "no single-cell demonstration of X in Y". So this entry deliberately reports no number for how many absence claims are false, because any such number would be wrong in both directions.

U2 is why the false-positive rate is high: most absences in this graph are deliberately narrow. "No menin readout at 5 µM / 72 h" is untouched by a paper about menin readouts. Topic-level matching cannot see a dose bound, a line bound or an assay bound, and every one of the top-ranked candidates read this cycle failed for exactly that reason.

U3: an absence refuted by a paper nobody here cites is invisible. Every figure above is a lower bound on unsafe absences, on a corpus that is not the literature.

The pattern this makes, counting from cycle 113
cycleinstrumentwhat it could not see
113type fieldscontent — understated coverage by a median of 19.5x
115two marginsthe join
117entry ids and prosecitations
118prose-only citation regex73.7% of citations, in the schema's own field
121paper-only absence checkdeposits

Each instrument was built as the fix for the previous one, and each inherited the shape. The consistent tell is not carelessness: it is that every one of these tools reported a confident number on an axis its author had not tested, and in four of five cases the author's own pre-registration or caveats named the untested axis in advance.

🔵 What is different this time is only that the clause was believed. R3 and U4 were written before the run, the null came back, and the null was reported as unvalidated rather than as reassurance. That is the whole of the improvement, and it is worth stating plainly because it is small.

Facts and figures
questiondoes this corpus already cite the refutation of its own absence claims?
verdictR3 - neither, and the method is unvalidated
absence entries60
papers scored against4000
candidates at 0 50
candidates 0 3 to 0 52
below 0 358
why no false absence count is givenU1 - a title match is a candidate, never a refutation; any count would be wrong in both directions
the fix executed same cycleana-this-corpus-cites-504-deposits-and-a-paper-index-cannot-see-any-of-them
and what it foundGSE252378 at score 0.333 - below any threshold that would have flagged it automatically, found by reading the worklist rather than by the ranking
absence classmeasured-not-curated
bite backThis entry refuses to say how many of the 60 absence claims are false, which is the number a reader wants. That refusal is defensible on U1 but it also means the project still does not know the rate, which was the question that prompted the work, and a future cycle may read 'unvalidated' as 'checked' simply because an entry exists.
control
{
  "id": "absence-no-menin-inhibitor-occupancy-dataset-in-an-infant-derived-kmt2a-aff1-line",
  "rank": 1,
  "of": 60,
  "best_score": 0.333,
  "shared_tokens_with_top_candidate": [
    "infant",
    "inhibitor",
    "model"
  ],
  "actual_refutation": "GSE159608, a GEO deposit",
  "reading": "right rank, wrong reason; U4 recorded before the run that deposits are out of scope, so this is not evidence the method works"
}
Still open
  1. Read the 58 low-scoring absences by hand, in tranches. That is the only method available that is not itself unvalidated, and it is the only way the rate of false absence claims will ever be known.
  2. Build a validation set. Two known-false absences exist and one is out of scope; deliberately planting known-refutable absences, or re-checking the historical corrections log, would give the first honest measure of any of these instruments' recall.
  3. Match absences against deposit SAMPLE records rather than series titles. Sample text is cached for 32 of 102 series and both known failures lived at sample level.
  4. Decide whether briefs should assert gaps at all. Five instruments in nine cycles have each been built to make novelty claims safe, and each was itself wrong; the alternative - state what is known and ask for an extension - has been proposed three times and never adopted.
Every source behind this entry (1)
  1. E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis

    Scoring all 60 bounded-absence entries in the corpus against the live PubMed titles of the 4,000 PMIDs this corpus cites, by containment of each absence's subject vocabulary in each paper title, with stopwords and project-ubiquitous terms removed and with each absence's own cited PMIDs excluded, gives 0 absences with a candidate at 0.5 or above, 2 between 0.3 and 0.5 and 58 below 0.3. The single absence in the set known to be false ranks first with a best score of 0.333, on candidates sharing the tokens infant, inhibitor and model with an antileukemic BET-inhibitor mouse study, while its actual refutation is GEO series GSE159608, which contains no PubMed title and is outside the method's scope by construction.

    What this evidence cannot show

    THE HEADLINE NULL IS NOT A RESULT ABOUT THE ABSENCES, IT IS A RESULT ABOUT THE METHOD. n=1 known-false case is not a validation set by any standard, and that one case is out of scope by construction, so the method has effectively never been tested against a case it could solve. A TITLE MATCH IS NOT A REFUTATION and no count of false absences is given here for that reason. Most absences in this graph are deliberately narrow and topic-level matching cannot see a dose, cell-line or assay bound, so the false-positive rate is high by construction and every top-ranked candidate read this cycle failed for that reason. The stopword and ubiquitous-term lists are the coordinator's and were tuned while the results were visible, which is a degree of freedom that was not controlled. Absences refuted by papers this corpus does not cite are invisible entirely.

    Evidence that argues against this (2)
    • Declaring the instrument unvalidated is safe and self-flattering: it costs nothing, cannot be wrong, and reads as rigour. The less comfortable possibility is that the 58 absences scoring below 0.3 really are mostly sound, in which case this entry has talked itself out of a genuine reassurance and the project will re-do this work later. Nothing here distinguishes those two readings, and the entry does not claim to.
    • The companion deposit index found the one thing this analysis missed, which makes a tidy narrative in which pre-registration caught a defect and the fix worked the same day. The deposit index scored that same case at 0.333 - also below any automatic threshold - so it too required a human to read past the ranking. Two indexes that both need manual reading of a noisy worklist have not automated this check; they have relocated it.
Connected to