Oncofusion

start / graph / ana-the-independence-metric-does-not-predict-support

Analysisunrated confidence 0 of 5: unrated

The 82.9% population scored twice as many defect-shaped notes as the same-source one, and reading six of them says the comparison is confounded by our own DepMap numbers

also called counter-note support by population · does independence predict support · 82.9% is not a quality measure

We asked whether counter-arguments that cite a second paper are better supported than ones reusing the first. The scores say they are worse - but reading the flagged cases shows most are quoting our own calculations alongside a paper, which the test cannot tell from a misattributed fact.

The same thing, for a scientist

The counter-evidence sweep found six defects among 77 same-source notes and ended on the question it could not answer: **would attaching a second identifier have fixed them, or merely moved them into the 82.9%?** This scores both populations with the same instrument. | population | n | median support | <=0.25 | defect-shape | rate | |---|---:|---:|---:|---:|---:| | same-source | 286 | 0.219 | 60.8% | 16 | **5.6%** | | different-source (the 82.9%) | 287 | 0.230 | 53.3% | 31 | **10.8%** | **The support-score distributions are nearly identical** - median 0.219 against 0.230 - which is the first result and the solid one: **citing a different paper does not predict better support.** The independence metric is silent about quality in both directions. **The defect-shape rate is twice as high in the 82.9% population, and I do not believe it.** Reading six of the 31 flagged rows shows a systematic confound the instrument cannot see.

Go deeper
The confound, found by reading the rows rather than the rate

The second axis flags a note carrying specific factual tokens absent from the cited abstract - the shape of an external result imported under a citation that does not carry it. In the different-source population, much of what it flags is this project's own numbers:

  • hyp-pause-axis-asymmetry-kmt2a-r E1: "The ordering within NELF tracks pan-essentiality almost perfectly: NELFB (-1.41) > NELFA (-1.04) > NELFCD (-0.62)" - DepMap gene effects from our own runs, in a note that cites a paper for the argument they bear on.
  • gene-ice1 E3: the same shape, with -1.30, -0.61 and -0.78.
  • protein-ewsr1-domains E3: 29.87, 1.89, 94.17, 1.65.

Those numbers cannot appear in any abstract, because no paper reports them. The note is doing something legitimate and common in this graph - opposing a literature claim with an internal measurement - and the instrument scores it as an unsupported import.

Why the confound falls on one side. A same-source note is written while reading one paper and argues about that paper. A different-source note is written to bring in something else, and in this graph that something else is often a DepMap number rather than another paper. The populations differ in what kind of content they carry, so the rate difference is not evidence that one is worse.

This is the seventh time in this project that a score produced candidates and only reading separated defect from design - and the first time it happened to a comparison the coordinator built to check someone else's work.

What the run does establish

One solid result and one dead prediction.

Solid: the two populations have near-identical support-score distributions (median 0.219 against 0.230, means 0.229 against 0.251). Whatever the 82.9% measures, it does not predict how well a counter-note is supported by the paper it cites. The sweep's conclusion that the metric is orthogonal to quality survives this test, now from the other direction.

Dead: the prediction written into the script before it ran - if the defect is caused by inheriting a citation, the different-source population should be cleaner, because attaching one is a deliberate act. It is not cleaner on the raw rate, and the raw rate is confounded, so the prediction is untested rather than refuted. Recording it as a failure to test rather than as a finding either way.

The fix for a future run is specific: before counting a token as an unsupported specific, check whether it appears in this graph's own analyses/*/results.json. A number this project computed is not an unattributed import; it is an internal citation that the schema currently has no way to express, which may be the more interesting problem.

A schema gap this exposed

A counter-note opposing a literature claim with a DepMap number has no correct citation to attach. The convention that has grown up is to cite the paper being argued with, which is exactly the pattern the sweep identified as the defect shape - and here it is the right thing to do with no better option available.

Some entries do attach an internal analysis as a type: analysis citation with a GitHub URL, and that is the better pattern. It is not applied consistently, and nothing enforces it.

The cheap improvement: when a counter-note's argument rests on an internal number, cite the analysis that produced it. That makes the note auditable, distinguishes it from an unattributed import, and would remove most of the 31 flagged rows here without changing a single argument.

Still open
  1. Before counting a token as an unsupported specific, check whether it appears in this graph's own analyses/*/results.json. That one change would remove most of the 31 flagged rows and turn the comparison into something interpretable.
  2. Calibrate the scorer against known-good and known-bad notes at scale. The six repaired notes from the sweep are the only ground truth this project has, and six is not a calibration.
  3. Read the remaining 25 flagged different-source rows and report the yield, so the 10.8% is either explained away or confirmed.
  4. Decide whether a counter-note resting on an internal number should cite the analysis that produced it, and if so, enforce it - the pattern exists in some entries and nothing requires it.
Every source behind this entry (3)
  1. E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis

    Scoring 286 same-source and 287 different-source counter-notes against the abstracts they cite gives near-identical support distributions - median 0.219 against 0.230, mean 0.229 against 0.251 - so citing a different paper does not predict better support.

    What this evidence cannot show

    Run 2026-09-20, seed 20260920, sampled 300 from each population (650 same-source and 4,371 different-source notes carry a PMID). **Strong, not established**: the scorer is a word-overlap heuristic, notes legitimately paraphrase figures and methods absent from abstracts, and the different-source sample is drawn from a population fifteen times larger so carries more uncertainty per row. The comparison is the result; no individual score is. Run while two agents were editing the graph, so it is a snapshot.

    Evidence that argues against this (1)
    • Near-identical medians could also mean the instrument has little resolution at all - if most notes score between 0.1 and 0.4 regardless of quality, then finding no difference between populations is weak evidence rather than a null result. Nothing here calibrates the scorer against known-good and known-bad notes at scale; the only calibration is the two hardcoded strings in its selftest. Oncofusion knowledge graph (internal) 2026
  2. E2moderate confidence 3 of 5: One strong study, or several smaller consistent ones.database

    The different-source population shows a defect-shape rate of 10.8% against 5.6% for same-source, but reading six of the 31 flagged rows shows the flagged tokens are frequently this project's own DepMap gene-effect values quoted inside a note that cites a paper for the argument, which no abstract can contain.

    What this evidence cannot show

    **Moderate, and the grade is the point.** Six rows of 31 were read, which is enough to establish that the confound exists and not enough to quantify how much of the difference it explains. The remaining 25 were not read. The correct statement is that **the rate difference is uninterpretable as it stands**, not that the populations are equivalent.

    Evidence that argues against this (1)
    • The confound explanation is convenient for the instrument's author and should be treated as such. A quicker and less flattering reading is that the second axis simply detects numerals, and any population whose notes contain more numerals scores worse. That reading also predicts the result and requires no argument about what kind of content each population carries. Oncofusion knowledge graph (internal analysis) 2026
  3. E3moderate confidence 3 of 5: One strong study, or several smaller consistent ones.analysis

    A counter-note that opposes a literature claim with an internally computed number has no correct citation to attach under the current schema, and the convention that has grown up is to cite the paper being argued with, which is indistinguishable from the defect pattern the counter-evidence sweep identified.

    What this evidence cannot show

    An observation about this repository's conventions from reading flagged rows, not a systematic audit of how often internal numbers are cited well or badly. Some entries do attach the internal analysis as a `type: analysis` citation with a URL, so the better pattern exists and is simply not enforced; the frequency of each is unmeasured.

    Evidence that argues against this (1)
    • Citing an internal analysis inside a counter-note also has a cost this does not weigh: it makes the graph's opposition partly self-referential, and this project has already found that internal analyses sharing one API URL collapsed into a single apparent source in its own load-bearing audit. Oncofusion knowledge graph (internal) 2026
Connected to