Oncofusion

start / graph / ana-the-weakest-verified-citations-are-attached-to-the-right-papers-and-every-low-scorer-is-a-supplement

Analysismoderate confidence 3 of 5: One strong study, or several smaller consistent ones.

The check this project named nine cycles ago and never ran: the 34 least-verified papers in the corpus are attached to the right claims, and every low-scoring pair is support living in a supplement or a structure rather than an abstract

also called do the unverified citations support their claims · claim-to-paper topical overlap · the placeholder-author subset reads clean · 141 claim/citation pairs scored

Every earlier check in this project asked whether a citation's ID points at a real paper, never whether that paper actually backs up the sentence it is attached to. This run read the one subset small enough to read - the 34 papers nobody transcribed - and found all of them correctly attached.

The same thing, for a scientist

**This is the check cycle 119 named and eight subsequent cycles deferred.** Every citation instrument built here so far asks whether an identifier *resolves* or whether stored metadata *matches* the record: 3,431 of 3,438 DOIs resolve, 99.7% of author lists agree, 100% of PMID years match. Each of those cycles closed with the same admission - *"check that each cited paper SUPPORTS the claim it is attached to; that is the thing that matters and no check here touches it"* - and each skipped it, because 17,915 PMID citation blocks cannot be read. **One subset can be, and it is the right one.** The author-axis runs flagged 109 blocks whose stored `authors` field is a placeholder ("Multicentre retrospective consortium", "See PubMed record for the full author list"). These collapse to **34 distinct identifiers** - 16 PMIDs and 18 DOIs. They are where a misattached citation would survive longest, because nobody transcribed the record and therefore nobody compared it to the claim. **Run: 141 claim/citation pairs across 109 entries, 34 papers, 141 of 141 pairs scoreable (100.0%).** Median claim-to-paper topical overlap **0.639**, p10 **0.200**, minimum **0.000** [E1]. **The script issues no verdict on its own, by design, because a low score is not a defect.** The output is the ranked low scorers and the requirement to read them. **I read the five riskiest and found no misattachment.** Every low score has a benign, identifiable cause: | pair | overlap | why it scores low | |---|---:|---| | `ana-the-transactivator-split-is-two-genes-and-the-mhc-is-invisible` -> PMID 39158067 | **0.000** | The claim is about **supplemental Table 4** (NLRC5/CIITA group assignment). The entry's own caveat already says *"Read directly from supplemental Table 4"*. The abstract carries `menin` and `H2AK119ub` but no `NLRC5`, `CIITA` or `MHC` - **verified by fetching it.** | | `interface-menin-nls1-479-497-and-nls2-588-608` -> DOI 10.2210/pdb3U84/pdb (3 pairs) | 0.019-0.048 | The citation is a **PDB deposit**, which has a title and no abstract. For a claim about crystal packing in 3U84, the deposit *is* the correct citation. | | `mechanism-cdk9-and-menin-inhibition-are-synergistic...` -> PMID 40979068 | 0.172 | The claim describes **a PubMed query this project ran on 2026-09-20**, not a result in the paper. A methodological claim cannot overlap a paper it is not about. | **And the stored titles are right.** I fetched four of the riskiest identifiers and every stored title matches the live record verbatim: **34163069** (*Phase separation drives aberrant chromatin looping and cancer development*, Nature 2021), **38989615** (*BOD1L mediates chromatin binding and non-canonical function of H3K4 methyltransferase SETD1A*, Nucleic Acids Res 2024), **40979068** (*Targeting CDK9 inhibits the growth of KMT2A-rearranged infant leukemia and demonstrates synergy with menin inhibition*, Blood Neoplasia 2025) and **41351513** (*Outcomes of relapsed or refractory acute myeloid leukemia after menin inhibition failure*, Blood Adv 2026, Chin KK, 17 authors, DOI 10.1182/bloodadvances.2025018178) [E2]. 🟠 **41351513 first returned HTTP 429.** A rate limit is an instrument failure, not an absence; it resolved on retry with backoff. Recorded because this corpus has twice filed a failed fetch as a negative result. 🔵 **So the pre-registered S1 branch fires on the subset read** - and U3 bounds it hard: **this subset was selected because it is small.** 34 papers reading clean is weak evidence about the other 17,800 blocks, which were selected by nothing and remain unread.

Go deeper
Why a low overlap score is not a defect, stated before the scores were seen

The pre-registration declared four ways this instrument could be uninformative, and three of them fired:

  • U1 - topical consistency is not support. A paper about menin inhibitors is topically consistent with almost any menin claim here and may still not support the specific number or direction. This check can find a citation attached to the wrong field; it cannot find one attached to the right field and the wrong result. That is the failure mode this project has actually suffered, so the instrument does not cover its own worst case.
  • U2 - a placeholder author field is not evidence of carelessness. "See PubMed record for the full author list" is reasonable for a 57-author consortium paper. The selection is about what is checkable, not what is suspect.
  • U3 - the subset was chosen because it is small. Cleanliness here is weak evidence about the rest.
  • U4 - abstracts only. A claim supported in a figure or a supplementary table looks unsupported. This is what the 0.000 pair is, and the instrument found it exactly where the pre-registration said it would.

The generalisable form: a topical-overlap score does not measure whether support exists. It measures where the support lives. A near-zero score says the support is not in the abstract - which for a supplemental table, a PDB deposit or a claim about this project's own query is the expected and correct answer.

What would have to be true for this to be a real negative

A cleaner version of this check needs full text, not abstracts. Two routes exist and neither has been tried:

  1. The 144 papers already indexed by abstract (ana-indexing-abstracts-reaches-144-papers-that-title-search-cannot-see) could be extended to PMC full text for the open-access subset, which would let U4 be tested rather than assumed.
  2. The two documented historical wrong-paper failures in this project are the only labelled positives available. Neither is in this 34-paper subset, so this instrument has never been run against a case it is known to need to catch. That is the same defect recorded for the absence-recheck instrument, and it is the reason confidence here is moderate and not strong.
Facts and figures
scriptanalyses/do-the-unverified-citations-support-their-claims/run.py
statusrun
run date2026-09-24
placeholder identifiers34
pairs scored141
entries covered109
coverage PCT100
overlap median0.639
overlap p100.2
overlap min0
pairs read by hand5
misattachments found0
preregistered branch firedS1, bounded by U1, U3 and U4
low scorer causes
supplemental_table1.000supplemental_table: 1claim_about_this_projects…1.000claim_about_this_projects_own_query: 1pdb_deposit_no_abstract3.000pdb_deposit_no_abstract: 3
low scorer causes — the numbers behind the chart
{
  "supplemental_table": 1,
  "pdb_deposit_no_abstract": 3,
  "claim_about_this_projects_own_query": 1
}
titles refetched and matching
[
  "34163069",
  "38989615",
  "40979068",
  "41351513"
]
measurements
[
  {
    "quantity": "claim-to-paper topical overlap, median",
    "value": 0.639,
    "unit": "fraction of claim content words present in the paper abstract",
    "subject": "141 claim/citation pairs drawn from the 34 placeholder-author identifiers",
    "universe": "the 109 entries carrying those identifiers; NOT the 17,915 PMID blocks in the corpus"
  },
  {
    "quantity": "misattached citations found among pairs read",
    "value": 0,
    "unit": "count",
    "subject": "the 5 lowest-scoring pairs, read by hand",
    "universe": "the ranked tail of 141 pairs; the remaining 136 were not read"
  }
]
Still open
  1. Extend to PMC full text for the open-access subset. U4 (abstracts only) is currently assumed rather than tested, and it is the assumption that produced the single 0.000 score.
  2. Run this instrument against the two documented historical wrong-paper failures in this project. Neither is in this 34-paper subset, so the instrument has never faced a labelled positive - the same defect recorded for the absence-recheck instrument.
  3. The 136 pairs that were scored but not read remain unread. A control count is a list nobody opens: the median is not the result, the tail is.
Every source behind this entry (2)
  1. E1moderate confidence 3 of 5: One strong study, or several smaller consistent ones.

    Scoring the 34 placeholder-author identifiers against the claims they are attached to yields 141 claim/citation pairs across 109 entries, 141 of 141 scoreable (100.0%), with median claim-to-paper topical overlap 0.639, p10 0.200 and minimum 0.000; no pair read was attached to a topically unrelated paper.

    What this evidence cannot show

    Run 2026-09-24 (cycle 132). Overlap is word-set intersection of claim against abstract, after stop-word removal, so it is a crude topical measure and nothing more. Scoreability is 100% only because the selection required a resolvable identifier; it says nothing about papers whose text could not be retrieved. Five pairs were read by hand; the remaining 136 were not, so this is a verdict about the ranked tail, not about all 141.

    Evidence that argues against this (1)
    • The instrument's own pre-registered U1 states it cannot detect a citation attached to the right field and the wrong result - which is precisely the failure this project has twice suffered. U3 states the subset was selected for size. So a clean read here is compatible with misattachments existing elsewhere in the corpus, and the 0.000 pair demonstrates that a low score carries no information about support without reading.
  2. E2strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.

    Four of the riskiest placeholder-author identifiers were fetched from PubMed and every stored title matches the live record verbatim, including PMID 41351513 which resolves to 'Outcomes of relapsed or refractory acute myeloid leukemia after menin inhibition failure' in Blood Advances with first author Chin KK and DOI 10.1182/bloodadvances.2025018178.

    PMID 41351513DOI
    What this evidence cannot show

    Fetched 2026-09-24 via PubMed esummary. The first request returned HTTP 429 and resolved on retry, which is why the retry is recorded: a rate limit reads identically to a non-existent record if the exit path is not distinguished. Title agreement establishes that the identifier points at the paper the corpus thinks it does; it does NOT establish that the paper supports the claim attached to it.

    Evidence that argues against this (1)
    • Title agreement is exactly the check that cycle 119 measured at precision zero for detecting wrong-paper citations, because the stored title is usually copied from the same record the identifier came from. Its value here is only as a consistency check on the placeholder subset, and it cannot be read as support.
Connected to