start / graph / concept-q117-a-pre-registered-verdict-can-flip-on-a-deviation-introduced-for-an-unrelated-reason

ExplainerestablishedQ117-gained-fusion-occupancy

Pre-registration protects against moving the goalposts and does not protect against this: a blacklist added to fix an unrelated artefact moved a pre-registered count from 42 to 52 across a threshold of 50, flipping the verdict from FAIL to PASS without anyone touching the threshold

also called Q117 threshold fragility · the deviation that decided the result

We wrote down in advance what would count as a positive result, then fixed an unrelated data problem, and the fix alone turned the answer from no to yes. Writing the rules down first stops you from bending them, but it does not stop a reasonable cleaning step from deciding the outcome for you.

The same thing, for a scientist

Q117 pre-registered three criteria for calling a gained-occupancy set, all required. The size criterion was **>= 50 bins**. The first run returned **42** and G1 failed. An **ENCODE mm10 blacklist** was then added because one 2 kb bin was found carrying 13.7% of an unrelated track; the blacklist is independently justified, was applied to every track identically, and had nothing to do with the gain criterion. The same pipeline then returned **52** and G1 passed [E1]. **Nobody changed a threshold and the verdict changed anyway.** The threshold was 50, the observation moved from 42 to 52, and the distance between fail and pass was two bins out of 1,228. Three things follow, and they are the transferable content: 1. **Report the observation's sensitivity to every deviation, not just its value.** This beat's entry records both 42 and 52 and says which processing state produced each. A reader who is given only 52 has been given a number whose stability they cannot judge. 2. **A threshold near the observation is a coin toss dressed as a decision.** When `|observed - threshold| / observed` is a few percent, the honest verdict is *unresolved*, whatever the criterion says. That should be written into the pre-registration as a declared indifference band, and Q117's was not. 3. **Pre-register the processing, not only the statistics.** Thresholds, nulls and effect sizes were all pre-registered here. Blacklisting, bin size, the mappability filter and the input reference were not; and the blacklist turned out to be the parameter that mattered. The complementary observation, which is the reassuring half: **the other two criteria did not move.** G2 went 28.5 -> 29.9 percentile points against a threshold of 10, and G3's Jaccard was 0.0 both times. **A criterion measured as a magnitude was robust; a criterion measured as a COUNT past a cutoff was not.** Prefer effect sizes to counts when writing a pre-registration.

Go deeper

Why this is not an argument against pre-registration

Without the pre-registration, the sequence would have been: run the analysis, get 42, notice the blacklist problem, fix it, get 52, and report 52 with no record that 42 ever existed. The pre-registration is what makes the fragility visible, because the threshold was fixed in advance and the deviation had to be written into a DEVIATIONS section rather than folded into the method.

What the episode shows is the boundary of the protection. Pre-registration constrains the analyst's freedom over the statistic. It does not constrain the analyst's freedom over the data the statistic is computed on, and in a re-analysis of somebody else's deposit that second freedom is the larger one.

The rule, in a form that can be applied next time

Declare an indifference band with the threshold. "G1 fires at >= 50; between 40 and 60 the result is unresolved." Two lines, written before looking, and it would have made this beat's verdict unresolved rather than pass.
Pre-register the pre-processing. Bin size, blacklist, mappability filter, input reference, chromosome set. Each is a fork and each can move a count.
Re-run every pre-registered criterion under every deviation and print the matrix, not just the final value.
Prefer a standardised effect size to a count past a cutoff. The count moved 24%; the effect size moved 5%.

Facts and figures

domain	research method
rule	Declare an indifference band with every pre-registered threshold; pre-register the pre-processing forks as well as the statistics; re-run every criterion under every deviation and print the matrix; prefer standardised effect sizes to counts past a cutoff.
deviation that caused it	ENCODE mm10 blacklist v2, added after one 2 kb bin was found carrying 13.7% of a BCOR track

worked example

{
  "criterion": "G1, number of top-0.1% treated bins below the 90th percentile of control",
  "threshold": 50,
  "before_deviation": 42,
  "after_deviation": 52,
  "verdict_before": "FAIL",
  "verdict_after": "PASS",
  "relative_move": "24% in the observation, across a threshold 19% away from the first value",
  "robust_criteria": {
    "G2_points_over_null": [
      28.5,
      29.9
    ],
    "G2_threshold": 10,
    "G3_jaccard": [
      0,
      0
    ],
    "G3_threshold": 0.25
  }
}

How this could bite us back

A result that flips on a defensible processing choice will flip back for the next person who makes a different defensible choice. If such a result reaches a decision - which experiment to fund, which combination to take forward - the decision inherits the fragility without inheriting the record of it, because the deviations section is the first thing that gets dropped when a finding is summarised.

Still open

How many of this project's stored analyses report a count past a cutoff rather than an effect size, and how many of those counts sit within 20% of their threshold? That is a corpus sweep nobody has run and it would identify every result in the graph that is one defensible deviation away from reversing.
Should a pre-registration in this project be required to name its pre-processing forks explicitly? Every beat that re-analyses a public deposit faces the same set - bin size, blacklist, mappability, input reference - and none of them is currently pre-registered by convention.

Every source behind this entry (2)

E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.dataset
In one beat, adding the ENCODE mm10 blacklist for an unrelated reason moved a pre-registered count from 42 to 52 against a fixed threshold of 50, flipping the verdict, while the two magnitude-based criteria in the same pre-registration moved from 28.5 to 29.9 percentile points (threshold 10) and stayed at Jaccard 0.0 (threshold 0.25).
Q117 pre-registration and both result files: analyses/q117-gained-fusion-occupancy/PREREGISTRATION.md, results-v1-no-blacklist.json, results.json · Iqbal Lab oncofusion knowledge graph 2026
What this evidence cannot show
This is a single episode in a single beat, not a survey. Whether count-based criteria are generally more deviation-fragile than magnitude-based ones is an assertion supported here by one comparison with n = 3 criteria, and the mechanism offered - that a count near a cutoff has a discontinuous response - is a mathematical observation rather than an empirical finding. Both result files are on disk so the claim is checkable.
Evidence that argues against this (1)
- The blacklist is not obviously a neutral deviation with respect to the gain question. Blacklisting removes 6% of bins, which changes the universe, which changes every percentile, which changes the top-0.1% membership directly. Someone could argue the two are not unrelated at all and that the deviation should have been anticipated. That reading makes the episode less about luck and more about an unstated fork - which is the same conclusion. Amemiya HM et al. 2019
E2established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.dataset
The deviation was introduced because one 2 kb bin, chr2:98,666,000, carries 13.7% of one deposited track's entire coverage, which is an artefact independent of the gain question.
NCBI GEO GSE260742, deposited .tdf tracks re-binned to 2 kb; blacklist artefact load 1.65-1.87% in batch 1 against 14.4-45.6% in batch 2 · NCBI Gene Expression Omnibus 2026
What this evidence cannot show
The artefact is in the BCOR, RING1B and input tracks, not in the FLAG tracks the gain criterion is computed on - FLAG puts 1.65-1.73% of its coverage in blacklist regions. The blacklist therefore reached the gain criterion only through the shared bin universe and the input-based mappability filter, which is precisely why the interaction was not anticipated.
Evidence that argues against this (1)
- If the input filter had not been used to define the universe, the blacklist would have touched the FLAG analysis much less and the flip might not have happened. The fragility is partly a consequence of coupling the FLAG universe to a batch-2 input track, which was itself a forced choice because all four matched FLAG-arm inputs carry no deposited file. NCBI Gene Expression Omnibus 2026

Connected to

related to →Asked for the first time whether a KMT2A fusion GAINS chromatin sites under a menin inhibitor: all three pre-registered criteria fire, and the set they fire on is 52 low-amplitude bins reaching 6x the genome median, 16 of them with no gene within 20 kb and 1 of 52 on a CpG island where 66% of the fusion's real sites are
the verdict that flipped is that entry's G1 criterion, 42 bins before the deviation and 52 after
depends on →Two processing batches inside one GEO series: the FLAG and KDM2B tracks put 1.7-1.9% of their coverage in ENCODE blacklist regions and the BCOR, RING1B and input tracks put 14-46%, with a single 2 kb bin at chr2:98,666,000 carrying 18.1% of the BCOR SGC0946 track on its own
the deviation was the blacklist added for the artefact that entry documents
← supportsFifty-three 2 kb bins gain MLL1-N-reactive signal under VTP-50469 in RS4;11, and the same 53 coordinates gain 1.77-fold in MOLM13 against a class-matched null of 1.01 with the MOLM13 input flat at 1.01 - so there IS a small, reproducible, input-controlled redistribution, and it carries no CpG-island signature
G1 fires at 53 against a threshold of 50, inside a declared indifference band of 40-65 - the same deviation-fragile count Q117 watched move from 42 to 52
← related toA plateau criterion of the form 'the last interval contributes at most 10% of the total' fires when the last interval is NEGATIVE, so it reports a plateau for a series that reversed - and the guard that caught it was a separate monotonicity check written for a different reason
the sibling failure of a pre-registered verdict: there a deviation changed the verdict, here the threshold's sign convention would have produced the opposite verdict and a separate guard caught it