Oncofusion

start / graph / ana-only-one-scored-criterion-in-the-corpus-carries-a-fragile-count-threshold-and-it-is-uncashed

Analysisestablished confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.

Cycle 76 warned that count thresholds in falsifiers may not transfer across set sizes. Sweeping all 2,301 entries found 5 scored criteria with a count threshold and exactly ONE that is both fragile and live - hyp-q117's uncashed PEAK ARM, now amended before it runs

also called the count-threshold sweep · how many falsifiers have the 20-of-52 problem · the alarm that deflated itself

Last cycle I found a pre-registered test whose pass/fail answer flipped depending on whether you read its "20 out of 52" as a raw number or as a percentage, so this time I searched all 2,301 records for tests written the same way and found that almost none are: of roughly 7,600 places a numeric condition could hide, only five are real scored tests containing a count, and four of those are already safe. Exactly one had the flaw, and because it has not been run yet it could simply be rewritten to state the number both ways at once - so my alarm was much bigger than the problem, and the useful result is one repair made before anything depended on it.

The same thing, for a scientist

**Statement.** The cycle-76 registry row `an absolute count threshold in a falsifier does not transfer to a differently-sized set` describes a real defect and a rare one. A corpus-wide sweep of `data.falsifiers`, `data.predictions`, `data.criteria` and `sections[].body_md` over all 2,301 node files - **7,647 scanned field instances** - matched **35** clauses of threshold shape, of which **only 5 sit in a scored-criteria field** and **30 are narrative prose** in section bodies ("more than 80 partners", "fewer than 200,000 persons", "at least 15 min") that are descriptions, not criteria. **Of the 5 scored criteria, 4 are already sound:** | entry | clause | why it is safe | |---|---|---| | `hyp-an-encode-only-archive-bound-systematically-undercounts-...` | "at least 15 percentage points" | already a proportion | | `hyp-the-isr-collapse-...` falsifiers[3] | "more than 1 log2" | a magnitude, not a set-membership count - a false positive of the classifier, reported rather than hidden | | `hyp-q117` falsifiers[0], CpG ARM | "at least 20 of 52" (38.46%) | names its denominator, and is scoped to the murine 52-bin set where both readings agree; the ambiguity is latent, not active, and this arm is CASHED so it is not amended | | `hyp-hp1-gamma-holds-a-sub-threshold-constitutive-foothold-...` falsifiers[1] | "below the 50th in at least 20 of the 31 archived BRCA1 tracks" (64.52%) | the denominator is **fixed by the archive** - there are 31 BRCA1 tracks and that number is a property of the resource, not of a result set, so no larger set can ever be substituted | **The one live defect.** `hyp-q117` falsifiers[1], REDISTRIBUTION PEAK ARM: *"if a MACS2 narrow-peak call ... returns more than 200 treated-only peaks at q < 0.05"*. This fixes no denominator, and MACS2's total yield varies with read depth, q cutoff and input - none of which the clause constrains - so the same biology could pass or fail depending on the size of the call. It is marked NOT YET RUN, which is what makes amendment legitimate rather than post-hoc. **Amended 2026-09-23 before running, without relaxing or tightening it:** score BOTH (a) more than 200 treated-only peaks at q < 0.05 AND (b) treated-only peaks exceeding 5% of the DMSO peak count in the same call; **if (a) and (b) disagree the arm reports UNDECIDED and does not get to pick the clause that fires**; report the DMSO peak count either way so anyone can re-score it. The 5% figure matches the original number's intent rather than its arithmetic - at Q117's observed scale the gained set was 52 bins against 1,228 held, or 4.2%. **The methodological point is about my own instrument, not the corpus.** The first version of this sweep reported **452 findings** and ranked `gene-nop2` top at an impossible "15882.14% (4447 of 28)". It was matching every `N of M` in prose, because I had written a regex for a lexical shape and described it as selecting a criterion. Requiring an explicit threshold modifier (`at least`, `more than`, `fewer than`, `at most`) cut 452 to 35 and the scored subset to 5. **Selecting a format while claiming to select a property is now the sixth instance of that error in this project's record**, and it was mine, in the very analysis written to check other people's thresholds.

Go deeper
What was walked, and what that excludes

Stated up front because three checkers in this project have walked a filename while reporting on a field.

Walked: data.falsifiers (635 instances), data.predictions (648), data.criteria / data.pre_registered_criteria (0 present), and sections[].body_md (6,364). Total 7,647 field instances across 2,301 node files, 2,301 of which parsed. 2,217 entries carry at least one scanned field; 84 carry none, and those 84 are invisible to this sweep.

Not walked, and therefore not cleared: thresholds stored in analyses/*/PREREGISTRATION.md files on disk rather than in node fields. Q118's own criterion G1 ("53 against a threshold of 50") lives there, not in a node, so the sweep cannot see the very class of threshold that motivated it. That is the largest gap in this analysis and it is not small: 162 analysis directories exist. A second pass over PREREGISTRATION.md files is the obvious next step and was not run.

Also not detected by construction: a threshold expressed in words ("a handful", "most of"), a threshold split across two sentences, and a threshold in a field name rather than a value.

Why a fixed denominator is the thing that matters, not the form

The hp1-gamma clause and the q117 CpG clause look identical in shape - both are at least 20 of N - and they are not equally safe, for a reason that has nothing to do with how they are written.

at least 20 of the 31 archived BRCA1 tracks has a denominator that is a property of a public archive. There are 31 tracks. No future run can hand this clause a set of 209 tracks, so the raw-count and proportion readings can never diverge.

at least 20 of 52 gained bins has a denominator that is a result of the analysis. The gained set was 52 in mouse and 209 in human, and that is exactly how the divergence appeared.

So the prescription in the cycle-76 registry row needs a qualifier: write count clauses as proportions when the denominator is itself an outcome; a raw count is fine when the denominator is fixed by the instrument or the archive. That distinction is the actual finding here, and it is more useful than the blanket rule I wrote last cycle.

The caveat above is now closed, and it was 32x too large

The section before this one calls the unwalked analyses/*/PREREGISTRATION.md files "the largest gap in this analysis" and sizes it as "162 analysis directories exist". run_prereg.py walked them in the same cycle. Only 5 of the 162 directories carry a PREREGISTRATION.md at all - 1,493 lines in total - so the unwalked population was five files, not 162. I sized a gap by the population it could have lived in rather than by the population it does live in, which is the same shape as every other denominator error in this record, committed this time inside a caveat.

What the second pass found: 3 clauses of threshold shape, 2 in a criteria context, and both are the SAME clause in q112-signal-level-reread - "fewer than 10 tracks of that protein pass the C1 gate". It is a bare count whose denominator (the arm's track count) is not fixed by the clause, so it is the same defect as the PEAK ARM.

And q112 caught it itself, before running. Its own pre-registration says the clause "cannot be met by the ENCODE arm alone, which holds only 7 BRCA1 and 4 CBX3 tracks", records it "before running, as a defect in the corpus pre-registration's own arithmetic", notes the n=76 figure assumed a ChIP-Atlas arm that was not included, and resolves it by reporting per-arm track counts and treating the clause as satisfied-by-construction rather than pretending otherwise. That is the behaviour the cycle-76 registry row asks for, executed roughly six cycles before the row was written. My "discovery" was a rediscovery, and the earlier instance was handled better than I handled mine, because it was caught before the run rather than after a different beat tripped over it.

A separate finding from the same pass, more interesting than the thresholds. The 5 pre-registered directories are exactly q112, q114, q116, q117, q118 - the five consecutive ChIP-signal beats. Pre-registration in this project is not a patchily-applied rule with 157 delinquents; it is a practice that began at q112 and has been used by every hypothesis-testing signal beat since, without exception. The other 157 directories are largely scripts and audits that test no pre-specified hypothesis. That is a fact about method adoption worth knowing before anyone proposes enforcing pre-registration corpus-wide.

Facts and figures
node files walked2301
parsed as dicts2301
entries carrying a scanned field2217
entries carrying none and therefore uninspected84
scanned field instances7647
clauses of threshold shape35
in a scored criteria field5
narrative prose in section bodies30
scored criteria already sound4
scored criteria fragile and live1
the live onehyp-q117-menin-inhibition-evicts-the-fusion-without-redistributing-it-to-a-recognisable-programme, data.falsifiers[1], REDISTRIBUTION PEAK ARM
first pass false positive count452
first pass top ranked absurditygene-nop2, '4447 of 28' = 15882.14% implied proportion
refines the cycle 76 registry rowwrite count clauses as proportions WHEN THE DENOMINATOR IS ITSELF AN OUTCOME; a raw count is sound when the denominator is fixed by the instrument or the archive, as hyp-hp1-gamma's 31 BRCA1 tracks are
absence classmeasured-not-analysed
absence class noteThe thresholds in 162 analyses/*/PREREGISTRATION.md files exist on disk, in this repository, and were not analysed by this sweep - which walks node fields only. This is the cheapest identifiable gap in this analysis: the data is local, needs no network, and is the file type most likely to carry a pre-registered count. Not never-measured (the files are right here) and not literature-absence-bounded (no external literature is involved).
caveat sizing errorthis entry sized its own largest gap as '162 analysis directories' when the actual unwalked population was 5 files - 32x too large. An overstated caveat is a deflating claim, and deflating claims in this project get waved through without the check a positive claim gets
by field
data.criteria0.000data.criteria: 0data.falsifiers635.0data.falsifiers: 635data.predictions648.0data.predictions: 648sections[].body_md6,364sections[].body_md: 6364
by field — the numbers behind the chart
{
  "sections[].body_md": 6364,
  "data.predictions": 648,
  "data.falsifiers": 635,
  "data.criteria": 0
}
n of m clauses with implied proportions
{
  "hyp-hp1-gamma-...-falsifiers[1]": "20 of 31 = 64.52%, denominator FIXED by the archive, safe",
  "hyp-q117-...-falsifiers[0] CpG ARM": "20 of 52 = 38.46%, denominator is an OUTCOME, latent but cashed and untouched"
}
instrument failures in building this
[
  "ROOT resolved one directory too high and the sweep reported 0 node files walked; caught only because the script prints its population first, and now guarded by an assert that refuses to report a zero",
  "a literal 0x08 backspace byte written into the regex by a Bash heredoc made the pattern match nothing while looking correct in a diff - the same failure documented in scripts/is-symbol-covered.py's own construction, and the reason the fix went through the Write tool",
  "the first regex selected a lexical shape (any 'N of M') while the entry described it as selecting a criterion, inflating 35 to 452"
]
preregistration pass
{
  "analysis_dirs": 162,
  "carrying_a_PREREGISTRATION_md": 5,
  "which_ones": [
    "q112-signal-level-reread",
    "q114-suv39h1-at-the-kmt2a-locus",
    "q116-h3k9-axis-at-the-locus",
    "q117-gained-fusion-occupancy",
    "q118-menin-at-the-human-index-fusion"
  ],
  "lines_scanned": 1493,
  "clauses_of_threshold_shape": 3,
  "in_a_criteria_context": 2,
  "distinct_clauses": 1,
  "the_clause": "q112-signal-level-reread: 'fewer than 10 tracks of that protein pass the C1 gate' - a bare count whose denominator is the arm's track count, which the clause does not fix",
  "already_self_flagged_before_running": true,
  "how_q112_handled_it": "recorded in its own pre-registration, before running, as a defect in the corpus pre-registration's arithmetic (n=76 assumed a ChIP-Atlas arm that was not included); resolved by reporting per-arm track counts and treating the clause as satisfied-by-construction for the ENCODE-only arm",
  "consequence_for_this_entry": "the corpus-wide answer is ONE live fragile threshold - hyp-q117's PEAK ARM, amended this cycle - plus ONE historical instance that the beat itself caught and neutralised before running",
  "what_is_still_unwalked": "the 157 analysis directories with no PREREGISTRATION.md. They hold no pre-registered criteria to be fragile, but a threshold could sit inside a run.py as a literal - that is a third pass and it was not run"
}
Still open
  1. CLOSED 2026-09-23 in the same cycle: the PREREGISTRATION.md pass ran. 5 files of 162 directories, 1 distinct fragile clause, already self-flagged by q112 before running.
  2. Do the 84 entries carrying none of the scanned fields hold thresholds somewhere else in their structure, or do they genuinely have no criteria?
  3. Should the PEAK ARM's clause (b) have been derived rather than chosen? The 5% figure matches the intent of '200' at Q117's observed scale, but a derivation from the expected false-positive rate of a MACS2 call at q < 0.05 would be defensible where a judgement call is not.
  4. Third pass: a threshold can sit as a literal inside a run.py rather than in prose. 157 analysis directories have no pre-registration but do have code. Not run.
Every source behind this entry (1)
  1. E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.analysis

    Across all 2,301 node files and 7,647 scanned falsifier, prediction, criteria and section-body instances, 35 clauses carry a count threshold; 5 of those sit in a scored-criteria field and 30 are narrative prose. Of the 5, one - hyp-q117's uncashed REDISTRIBUTION PEAK ARM, 'more than 200 treated-only peaks at q < 0.05' - fixes no denominator and is the only instance that is both fragile and live.

    analyses/count-threshold-transferability/run.py and results.json, this repository; run 2026-09-23 against the working tree at 2,301 node files · Oncofusion knowledge graph, coordinator · internal analysis, reproducible from the repository 2026
    What this evidence cannot show

    The classification is LEXICAL and the script deliberately labels nothing as wrong - every one of the 35 was read by hand before this entry was written, which is how the 30 prose cases were separated from the 5 criteria. Two further limits matter more than that one. FIRST, thresholds stored in analyses/*/PREREGISTRATION.md are NOT walked, and Q118's G1 criterion (53 against a threshold of 50) lives there - so the sweep is blind to the file type most likely to hold a pre-registered count, across 162 analysis directories. The '5 scored criteria' figure is therefore a count over NODE FIELDS ONLY and must never be quoted as a corpus-wide total. SECOND, 84 of 2,301 entries carry none of the scanned fields and are uninspected. Thresholds in words rather than digits, split across sentences, or encoded in a field name are undetectable by construction.

    Evidence that argues against this (3)
    • The strongest counter-argument is that this analysis deflates an alarm I raised one cycle earlier, and a sweep run by the person who raised the alarm is the weakest possible instrument for retiring it. The specific risk is the one named in the caveats: because PREREGISTRATION.md files are unwalked, a reassuring node-field count could coexist with many fragile thresholds in the analysis directories. The honest reading is that this sweep bounds the problem in NODE FIELDS and says nothing about the 162 analysis directories.
    • My own first pass at this sweep returned 452 findings and a top-ranked clause of '4447 of 28' - a 15,882% implied proportion, which is arithmetically impossible for a subset count and was pure regex noise. That the corrected figure is 35 is a fact about a regex that was wrong twice (it also carried a literal 0x08 backspace byte that silently made the pattern match nothing, producing a clean-looking zero). A reader is entitled to treat a number from an instrument with that history as provisional, and the raw findings list is in results.json for exactly that reason.
    • Amending an uncashed falsifier is defensible but not free: it means the PEAK ARM will be scored against a criterion written after the beat that motivated the change, by the same coordinator, and the 5% figure in clause (b) was chosen by judgement about intent rather than derived. A stricter practice would have retired the arm and written a new falsifier with a new id, leaving the original visibly unscored.
Connected to