Two measurement defects confirmed: a denominator summed over seven baits made the pan-fusion core look 7.5x cleaner than it is (corrected fold 1.06, p 0.91), and my own gap report counts as coverage in the join that measures the gap
also called q185 · a gap report counts as coverage · the seven-bait denominator
Two ways of measuring turned out to be broken. The first divided a one-column number by a seven-column number, which made one group of proteins look seven times cleaner than another purely by arithmetic; corrected, the two groups are identical. The second is subtler and is mine: I measured how many proteins the project had never discussed, then wrote a report discussing them - and the next run of the same measurement counted that report as coverage. Writing about a gap closes it, by that measure.
The same thing, for a scientist
🔴 **1. The seven-bait denominator, found by Q180 and king-verified.** `analyses/mll-fusion-interactome-distinctiveness/run.py` line 156 computes `signal = sum(_counts(r[c+1]) for c in BAIT_COL.values())` — spectral counts summed over **all seven baits** — while the numerator is the **single** `neg_ctrl` column. A pan-fusion prey is non-zero in seven bait columns and a singleton in one, so **the denominator is ~7x larger for the core by construction** [E1]. | | as published | per-bait denominator | |---|---:|---:| | pan-fusion core | 0.00232 | 0.01625 | | singleton tail | 0.01723 | 0.01723 | | fold | **7.42** | **1.06** | | two-sided permutation p | 0.0745 | **0.9102** | **The tell was inside the entry's own text.** It wrote *"they also carry 20-fold more spectral counts (mean 315.6 vs 15.7)"* — and **20.16 = 7 × 2.88 exactly**, where 2.88 is the real abundance difference and 7 is the summation. The artefact was quoted as the justification for correcting it. 🟠 **And the raw direction runs the other way.** Pan-fusion proteins carry non-zero control signal **more** often than singletons — **23.8% vs 12.0%** — which the entry reported and then normalised away with the broken denominator [E1]. **Actioned:** E3 withdrawn and downgraded to `weak`, `background_check` split into `_WITHDRAWN` and `_CORRECTED`, `open_questions[0]` **re-opened** from RESOLVED, and the script annotated at the line so it now computes and prints **both** denominators. What survives untouched: 32 of 42 pan-fusion proteins have zero control signal, so there is still no positive evidence the core *is* background. 🔴 **2. A gap report counts as coverage — my own defect.** q179 measured how many of the 110 AF4-specific partners the corpus had ever mentioned, by word-boundary content search, and reported **15 mentioned / 95 never**. Q180 re-ran it and got 21/89, tracing the difference to `ana-q178` — *the gap report itself*. Re-measured today over 3,067 entries [E2]: | join | mentioned | never | |---|---:|---:| | counting everything | **31** | 79 | | excluding q178/q179/q180 — the artefacts written *about* the gap | **22** | **88** | **Nine symbols are 'covered' only by entries created to discuss their absence.** The measure is reflexive: **running it and publishing the result changes it in the direction of closure**, and a beat dispatched to fix a gap closes it on this metric whether or not it writes a single substantive entry. 🔵 **Q180's own 16 entries make this worse, not better, and that is the point.** Its report said the six discrepant symbols were mentioned *only* by q178; by the time I checked, q180's declines mention them too, so excluding q178 alone now changes nothing. **The contamination grows with every cycle that works on the gap.**
What a self-excluding coverage join should look like
Any corpus metric of the form "how many X does this corpus discuss" must exclude:
- entries whose subject is the metric — gap reports, coverage audits, read-queues;
- entries produced by the beat dispatched in response to the metric;
- enumeration entries that list a screen's rows wholesale, which this corpus writes routinely and which guarantee a mention for anything in any curated screen.
q178 already reported that 135 of 135 IntAct prey were 'mentioned somewhere' and flagged in its own counter-evidence that this was "partly an artefact of how this corpus works: it writes large enumeration entries that list every row of a screen". That caveat named category 3 and missed categories 1 and 2 — including itself. The entry warning about mention-inflation was itself inflating the next measurement.
The honest figures for the AF4-specific tail are 22 mentioned and 88 never, and any future re-measurement must state its exclusion list or it is not comparable.
Why these two defects are the same defect
Both are a denominator that includes the thing being measured.
In the first, the denominator sums across baits, so a protein's presence in many baits — the very property that defines it as pan-fusion — inflates its own denominator and makes it look clean. In the second, the corpus's discussion of a gap enters the corpus, so the act of measuring absence reduces measured absence.
Neither is detectable from the output: both produce plausible numbers moving in the expected direction. The first was caught by asking why is 20.16 such a round multiple of 7; the second by an agent re-running a figure it had been handed and noticing it had changed. Both were found by re-deriving rather than re-reading, which is the fourth time that has been the operative move in this session.
| script | analyses/q180-is-af4-specificity-a-depth-artefact/control_denominator.py (re-run by the King); fresh content join, cycle 159 |
|---|---|
| status | executed |
| corrects | ana-the-index-fusion-interactome-is-not-distinctive-and-two-earlier-answers-were-artifacts |
| bug location | analyses/mll-fusion-interactome-distinctiveness/run.py line 156 |
| script annotated | true |
| verdict | both defects are a denominator that includes the thing being measured; both were found by re-deriving rather than re-reading |
| e1 e2 unaffected | true |
| script output also corrected | analyses/mll-fusion-interactome-distinctiveness/results.json still asserted 'Normalised for signal the core is cleaner, not dirtier' after the entry was corrected; the script now emits _WITHDRAWN keys plus a CORRECTED_per_bait block and prints both denominators |
as published — the numbers behind the chart
{
"pan_fusion": 0.00232,
"singletons": 0.01723,
"fold": 7.42,
"p": 0.0745
}corrected — the numbers behind the chart
{
"pan_fusion": 0.01625,
"singletons": 0.01723,
"fold": 1.06,
"p": 0.9102
}denominator inflation — the numbers behind the chart
{
"published_fold": 20.16,
"true_abundance_fold": 2.88,
"ratio": 7
}raw nonzero control
{
"pan_fusion": "10/42 (23.8%)",
"singletons": "68/568 (12.0%)"
}af4 tail mentions
{
"counting_everything": 31,
"excluding_self_reference": 22,
"never_mentioned_corrected": 88,
"n_symbols": 110,
"q179_as_published": "15 mentioned / 95 never"
}measurements — the numbers behind the chart
[
{
"quantity": "fold",
"subject": "control-per-unit-signal, core vs tail, AS PUBLISHED",
"universe": "42 pan-fusion and 568 singleton Skucha preys",
"value": 7.42
},
{
"quantity": "fold",
"subject": "control-per-unit-signal, core vs tail, CORRECTED per-bait",
"universe": "42 pan-fusion and 568 singleton Skucha preys",
"value": 1.06
},
{
"quantity": "count",
"subject": "AF4-specific symbols mentioned only by gap-report or gap-beat entries",
"universe": "110 AF4-specific Skucha partners",
"value": 9
},
{
"quantity": "count",
"subject": "AF4-specific symbols never mentioned, self-reference excluded",
"universe": "110 AF4-specific Skucha partners",
"value": 88
}
]- Audit every other corpus-coverage figure for self-reference. This project publishes coverage numbers routinely - interactome coverage, PI-dimension coverage, absence-class coverage - and none of them states an exclusion list. Any that counts mentions has the same defect.
- Decide whether a reasoned DECLINE counts as coverage. It should, and a bare gap-list mention should not, but the join here treats them identically. Distinguishing them needs an entry-level judgement the current measure cannot make.
- RESOLVED same cycle. The other two answers the parent entry calls artifacts do NOT rest on the seven-bait denominator: it appears only in the background-check block feeding E3, while E1 (rank-based selection) and E2 (equal-n distinctiveness) are computed from ranked/maxp/unique_counts and never reference neg_ctrl. The parent entry's title and thesis stand. Verified by reading every use of the `signal` and `prof` variables in the script.
- Replace 'mentioned' with 'makes a claim about'. Going from 31 to 22 swaps one weak measure for a less weak one; neither says whether a protein was thought about.
- E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.software
analyses/mll-fusion-interactome-distinctiveness/run.py line 156 sums spectral counts over all seven bait columns into the denominator while the numerator is the single neg_ctrl column at index 17. Re-running the reproducer independently reproduces the published figures exactly - pan-fusion 0.00232, singletons 0.01723, fold 7.42, two-sided permutation p 0.0745 - and with a per-bait denominator gives pan-fusion 0.01625, singletons 0.01723, fold 1.06, p 0.9102 over 20,000 draws at seed 180. Mean denominators are 315.6 vs 15.7 as published (20.16-fold) and 45.1 vs 15.7 per bait (2.88-fold); 20.16 / 2.88 = 7.00. Raw, unnormalised: 10 of 42 pan-fusion proteins (23.8%) have non-zero control signal against 68 of 568 singletons (12.0%).
What this evidence cannot show
The per-bait denominator is one defensible correction among several - dividing by the maximum single-bait count, or by the count in the bait under consideration, would give different numbers. What is not in doubt is that the published denominator mixes a seven-column sum with a one-column numerator.
Evidence that argues against this (2)
- Correcting this removes the negative-control column as a discriminator entirely (p 0.91), which also removes an argument that could have been used AGAINST the shared core. The correction is neutral in direction, not favourable: it makes the core neither cleaner nor dirtier than the tail, and anyone wanting to argue the tail is contaminated has lost this axis too.
- The entry's headline - that there is no evidence the shared core is bead or tag background - was never dependent on the normalised figure, so a reader could reasonably say the withdrawal changes nothing that mattered. The counter is that open_questions[0] explicitly upgraded the shared fraction from 'upper bound on shared biology' on the strength of the 7.5x, and that upgrade is now reversed.
- E2strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.software
A word-boundary content join over 3,067 corpus entries for the 110 AF4-specific Skucha partners returns 31 mentioned and 79 never mentioned counting every entry, and 22 mentioned and 88 never when entries with ids beginning ana-q178, ana-q179 or ana-q180, and entries in the q180 folder, are excluded. q179 as published reported 15 and 95; Q180 re-running it reported 21 and 89 and attributed the difference to ana-q178, the gap report itself. Nine symbols are mentioned only by entries whose subject is the gap or by the beat dispatched to address it.
What this evidence cannot show
The three published figures (15/95, 21/89, 31/79) were measured at three different corpus states hours apart and are not directly comparable; the exclusion comparison 31 vs 22 is the only like-for-like pair here. The exclusion list is by id prefix and folder, so a self-referential entry with an unrelated id would not be caught.
Evidence that argues against this (2)
- Excluding a beat's own entries is not obviously right. If q180 genuinely examined and declined a protein with cited evidence, that IS coverage in any useful sense - it is exactly what the corpus is for. The reflexivity problem is real for a GAP REPORT that merely lists symbols, and much weaker for a decline entry that adjudicates one. Treating both the same way understates coverage.
- 'Mentioned' was always a weak bar and this entry's correction does not fix that. Going from 31 to 22 replaces one poor measure with a slightly less poor one; neither tells anyone whether a protein has been thought about. The useful statistic would be entries that make a claim about the protein, which nothing here measures.
withdraws E3's 7.5-fold cleaner-per-unit-signal figure: the denominator summed seven bait columns against a one-column numerator; corrected fold 1.06, p 0.9102
the gap report itself counts as coverage in the join that measures the gap; 9 AF4-specific symbols are mentioned only by entries written about their absence
third instrument defect of the cycle found by re-deriving rather than re-reading
answers q185's first open question: 37.8% of the named-but-unwritten queue is the project's own controls, registered collisions or an alias-filed entry
second of q185's coverage-figure audits: absence_class is carried by 529 entries of which only 69 are absences