The grading rule: confidence grades the claim, not the source - so one paper carrying several grades is correct, and only a three-step spread is worth reading
also called confidence grading rule · established versus strong
Our confidence labels describe how strongly a particular statement is supported, not how good the paper is. So the same paper can support one statement firmly and another weakly, and that is not a mistake.
The same thing, for a scientist
Ledger priority 4 asked for a rule deciding when a census paper is cited as `established` versus `strong`, noting that `find-contradictions.py` had reported the pairwise version for several cycles without anyone acting. Measured: **1,356 PMIDs carry a graded evidence item, 668 are cited more than once, and 228 of those (34%) carry more than one grade** - but **189 of the 228 are adjacent grades**, which the rule below says is expected rather than wrong. The stated rule: **confidence grades the evidence item's claim, not the source**; a census count from its own cohort is `established`, a generalisation from that cohort is one grade lower, and a bibliographic claim about what an abstract contains can be `established` because it is checkable by reading the abstract.
The rule
1. The grade attaches to the claim, not to the paper. A strong paper can support a weak inference and a weak paper can establish a bibliographic fact. It follows that one PMID carrying different grades in different entries is the normal case, not a defect - and find-contradictions.py's "moderate vs established" lines are therefore a prompt to look, not a report of an error.
2. For a census, registry or archive paper, three grades in descending order:
| what is being claimed | grade |
|---|---|
| a count within the reported cohort - "of 3,401 patients, 1,233 were AFF1" | established |
| a generalisation from that cohort to the disease - "AFF1 is the commonest partner in infant ALL" | strong - the archive has a referral pattern |
| an inference beyond the reported numbers - a trend, a mechanism, a switch point | moderate or weak |
3. A bibliographic claim can be established. "The abstract does not name the fusions used" is checkable by reading the abstract, so it is established - provided it is worded so that it is obviously about the paper and not about the biology. An entry that grades "the paper reports X" as established is asserting that the report exists, which is true and supports nothing about X.
4. A three-step spread on one paper is a flag to read. Eight papers carry grades from weak to established. Reading them, most are rule 2 working correctly: the recombinome is established for "in 3,401 patients the partner counts are..." and weak for "infant patients show a preference that switches at about school age". Same paper, different claims, correctly different grades.
What the measurement actually showed, against what the ledger implied
The ledger's wording - "39 of the 60 widest-reach sources are graded inconsistently" - reads as a defect list. Measured across the whole graph:
| count | |
|---|---|
| distinct PMIDs carrying a graded evidence item | 1,356 |
| cited more than once | 668 |
| carrying more than one grade | 228 (34%) |
| ...of which adjacent grades (spread 1) | 189 |
| ...spread 2 | 31 |
...spread 3, weak to established | 8 |
So the actionable list is 8 papers, not 228 - and on reading, most of the 8 are correct. The recombinome alone is cited 112 times with 54 established, 51 strong, 6 moderate and 1 weak, and that distribution is what rule 2 predicts for a census cited across many different kinds of claim.
One pattern was worth hunting and turned out to be tiny. A sweep for evidence claims that merely restate their citation's title found 311 items - but almost all are legitimate, because a title that states a finding, restated, is the finding. Filtering to claims that assert the existence of a report - "The paper reports that..." - leaves 2 items, both in ana-unstated-system-*-unresolved entries, and both are worded as bibliographic claims about what an abstract contains. Under rule 3 they are correctly graded, and no regrade was applied.
That is the third instrument this session that flagged far more than it should have. The pattern is consistent enough to be worth naming: a similarity or consistency score identifies candidates, and reading them is what separates defect from design.
- Apply rule 2 deliberately to the recombinome's citations rather than checking that they happen to fit. 6 of the 112 are `moderate` and 1 is `weak`; those 7 are where the rule would bite if anyone disagreed with them.
- Should `find-contradictions.py` stop reporting confidence splits as contradictions? Under rule 1 they are expected, and the report has been carried for several cycles as though it were a defect queue.
- The 31 papers with a two-step spread were not read. They are the band where the rule is most likely to be wrong in either direction.
- E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.analysis
Across the graph, 1,356 distinct PMIDs carry a graded evidence item and 668 are cited more than once; 228 of those carry more than one confidence grade, but 189 of the 228 differ by a single step and only 8 span weak to established.
Measuring confidence-grade consistency per cited paper across the graph · Oncofusion knowledge graph (internal analysis) 2026What this evidence cannot show
A count within this graph on 2026-09-20, so `established` under rule 2 - it is a fact about the corpus, not about the world. Grades are read from the evidence item's `confidence` field; items with no grade or a non-standard string are excluded, and counter-evidence citations are not graded and so are not counted.
Evidence that argues against this (1)
- A spread of one step is treated here as expected, which is an assumption the measurement cannot test. If graders were simply inconsistent at the margin - `strong` and `moderate` used interchangeably - the distribution would look identical. **The rule makes adjacent spread acceptable by fiat; it does not demonstrate that the adjacent cases are principled.** Oncofusion knowledge graph (internal analysis) 2026
- E2strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.journal-article
The KMT2A recombinome is the graph's most-cited paper at 112 graded citations, distributed as 54 established, 51 strong, 6 moderate and 1 weak - the distribution the grading rule predicts for a census cited across counts, generalisations and inferences.
The KMT2A recombinome of acute leukemias in 2023. · Meyer C, Larghero P, Almeida Lopes B, et al. · Leukemia 2023What this evidence cannot show
That the observed distribution matches what the rule predicts is weak evidence that graders were following the rule, since the rule was written after seeing the distribution. It is offered as a consistency check, not a validation.
Evidence that argues against this (1)
- 112 citations from one paper is the concentration finding from `ana-how-many-laboratories-is-this-graph-standing-on` restated at the level of a single source. A grading rule makes the citations individually defensible and does nothing about the graph's dependence on one diagnostic archive. Meyer C et al. 2023
the same separation applies to a counter-note: the grade belongs to the claim, and the counter-note's citation belongs to the counter-note, not to its parent
THE COMPANION RULE ON HOW TO READ A QA PERCENTAGE: confidence grades the claim rather than the source, and likewise counter-argument coverage counts notes rather than arguments.
the rule, applied to 20 citations
the same concentration seen per source rather than per group: one paper cited 112 times
another instrument whose flags mostly dissolved on reading
a recombinome-cited entry whose grades the rule governs
grades and claim support are the two axes on which an evidence item is judged