The bigWig refusal of cycle 57 is overturned: pybigtools installs on this box under Python 3.12, reads GEO tracks by HTTP range request without downloading them, and passes the control loci the refusal itself demanded
also called the tooling correction · pybigtools works, pyBigWig does not · the refusal was right about the tool and wrong about the conclusion
Last cycle this project correctly established that the standard Python library for reading these genome signal files would not build on this Windows machine, and correctly refused to guess. A different library, built in Rust with ready-made Windows packages, installs in seconds and reads the same files straight off the internet without downloading them.
The same thing, for a scientist
`absence-h3k9me3-at-the-kmt2a-locus-was-assayed-and-never-reported` records, in a cycle-57 king amendment, that reading GEO bigWigs on this host is blocked because *"pyBigWig FAILS TO BUILD on this Windows host... deeptools is absent, and the UCSC binaries bigWigSummary and bigWigAverageOverBed are not on PATH"*, and that a hand-rolled BBI parser was deliberately refused. **Every one of those statements is still true, and the conclusion drawn from them is not.** `pybigtools` 0.3.0 - the Python binding to BigTools, a maintained Rust reimplementation of the BBI format - installs with prebuilt wheels [E1]. It fails under the system CPython **3.14** (no wheel exists for 3.14, so pip falls back to `cargo rustc` and the build fails - the same symptom the refusal describes for pyBigWig), and succeeds under the uv-managed **CPython 3.12.13 already present on this machine** [E1]. It also accepts an **HTTPS URL** in place of a path and issues range requests against the BBI index, so **nothing is downloaded**: this beat read 19 tracks totalling 2.9 GB on the server and transferred a few hundred kilobytes [E1]. **And it passes the validation the refusal demanded**: over the H19/IGF2 imprinting control region - the textbook CTCF site - HCT116 CTCF reads 7.78x its genome mean against input 1.02x, while the GAPDH promoter reads 0.97x against input 2.37x and a chr2 gene desert 0.41x against input 0.92x [E2].
What was refused, and what the refusal got right
The cycle-57 amendment set three conditions, any one of which would make the compute safe: (a) pyBigWig on a Linux host, e.g. the Explorer cluster; (b) the UCSC bigWigSummary binary on PATH; (c) a hand-rolled reader validated against a positive and a negative control locus first.
It was right on every factual point. Re-checked on this machine: import pyBigWig -> ModuleNotFoundError; bigWigSummary, bigWigAverageOverBed, bigWigToBedGraph, bigWigInfo, samtools and bedtools all absent from PATH; C: 26 GB free, D: 14 GB free 1. And it was right to refuse the hand-rolled parser - the reasoning that an R-tree indexing error would return a confident wrong value at the breakpoint cluster is exactly correct, and that is the failure this beat's headline number would have had.
What it missed is that the option space was three items long when it is four. There exists a maintained third-party implementation with prebuilt Windows wheels, so the choice was never between a broken C extension and a parser written from scratch. pybigtools is option (d): someone else's validated binary parser. The generalisable form: when a refusal rests on 'the tool will not build', the missing step is a search for a second implementation of the same format, not a better argument for refusing.
The recipe, so nobody re-derives it
# the system CPython on this box is 3.14 and has NO pybigtools wheel -> cargo build -> fails
# use the uv-managed 3.12 that is already installed:
PY312="D:/Users/ashenoy00000/AppData/Roaming/uv/python/cpython-3.12.13-windows-x86_64-none/python.exe"
"$PY312" -m venv <scratch>/venv312
<scratch>/venv312/Scripts/python.exe -m pip install pybigtools # ~seconds, prebuilt wheel
Reading, with the two API traps that cost this beat a round trip each:
import pybigtools, numpy as np
f = pybigtools.open("https://ftp.ncbi.nlm.nih.gov/geo/samples/GSM3714nnn/GSM3714940/suppl/GSM3714940_ChIP_seq_CTCF_HCT116.bw")
f.chroms() # {'chr11': 135006516, ...} -> READ THIS FIRST, it is the build check
f.info()["summary"] # basesCovered, sum, mean, min, max, std -> the depth normaliser, free
v = f.values("chr11", 118359328, 118359475, missing=0.0) # per-base array
v = np.nan_to_num(v, nan=0.0) # STILL needed; uncovered positions come back NaN
binned = f.values("chr11", 0, n*147, bins=n, summary="mean", missing=0.0) # zoom-level scan, fast
Trap 1: summary= is ignored unless bins= is set. f.values(c, s, e, summary="mean", exact=True) returns the full per-base array, not a scalar. This beat's first readout produced arrays where it expected means and the tables were nonsense.
Trap 2: uncovered bases are NaN, not 0. The library warns about this once and the default changed between versions. A mean over an interval with gaps silently becomes a mean over covered bases only, which inflates a sparse region's value - for a break track, where most of the genome is empty, that is a large error in the wrong direction. Pass missing=0.0 and nan_to_num.
Trap 3, in the data not the library: GSE210480's bigWigs declare every chromosome length as hg19 + exactly 100,000 bp. chr1 249,350,621; chr2 243,299,373; chr11 135,106,516; chrX 155,370,560 - all exactly 100 kb above hg19, which is a HOMER-style padded chrom.sizes. The coordinates inside are hg19 coordinates and are safe to query; only the declared lengths are inflated. The danger is build detection: for chr11 the padded length is 19,894 bp from hg38 and 100,000 bp from hg19, so a 'nearest known build' heuristic run on chr11 alone picks hg38 and is wrong, which at this locus would displace a 147-bp query by about 20 kb. Always check several chromosomes and look for a constant offset 2.
What this unblocks, named
- This beat's own headline, which is the proof of concept: 19 tracks across three studies read at base resolution, with chr11-wide nulls, in one session - see
ana-kmt2a-exon-11-element-is-a-top-0-05-percent-etoposide-dsb-site. - The Q102 H3K9me3 question, which was the entry the refusal lives in.
GSM6430988_CTRLsiKDM3B_H3K9me3_rep1.bigWigopens and reports 25 chromosomes 2. The eight H3K9me3 samples, the eight H3K9me1, the eight H3K9me2 and the four CTCF samples of GSE210480 are all readable by the recipe above. That analysis is now a compute job with no blocker, and its acceptance test - reproduce the published H3K9me1 depletion across the KMT2A BCR - is runnable. This beat did not run it, because that entry belongs to another beat; the King should dispatch it. - Gray 2023's own CTCF summit, the single inferential link
concept-ctcf-at-kmt2a-fragile-and-guarded-is-one-elementflagged as settleable-but-unread. Four CTCF bigWigs in GSE210480, one query each, hg19 coordinates. - Every 'no peak calls were deposited' caveat in this graph becomes cheaper to work around, because a coverage track plus a chr-wide rank is a peak call for a single named interval.
| script | pybigtools 0.3.0 in a venv from the uv-managed CPython 3.12.13; scripts in the beat scratchpad |
|---|---|
| status | run |
| result summary | The cycle-57 bigWig refusal is overturned by a fourth option it did not consider: pybigtools, a maintained Rust implementation with prebuilt Windows wheels. It fails under the system CPython 3.14 (no wheel, cargo build exits 101) and installs under the uv-managed CPython 3.12.13. It reads GEO tracks by HTTP range request with no download. It passes the positive control (H19 ICR: CTCF 7.78x genome mean, input 1.02x) and the negative controls (GAPDH promoter CTCF/input 0.41; chr2 desert 0.30). GSE210480's H3K9me3 tracks open, so the Q102 analysis has no remaining blocker - and its bigWigs carry an hg19+100,000 padded chrom.sizes that defeats naive build detection, dangerously so on chr11. |
| tool | pybigtools 0.3.0 |
| python | CPython 3.12.13 (uv-managed); the system CPython 3.14 does not work |
| data transferred | a few hundred kilobytes, against 2.9 GB of server-side files |
inputs
[ "pybigtools 0.3.0 from PyPI", "GSE129526 GSM3714939/40/41 for the control loci", "GSE210480 GSM6430988 header, as a readability probe for the Q102 question" ]
api traps
[ "summary= is ignored unless bins= is set; f.values(c,s,e,summary='mean',exact=True) returns a per-base array, not a scalar", "uncovered bases return NaN, not 0; pass missing=0.0 and np.nan_to_num or a sparse region's mean is silently inflated", "GSE210480 bigWigs declare hg19 chromosome lengths plus exactly 100,000 bp; on chr11 that is closer to hg38 than to hg19, so nearest-build detection picks the wrong build" ]
- Overturning a refusal makes the next refusal cheaper to disbelieve, which is the wrong lesson. The cycle-57 refusal was factually correct on every point it asserted and correct to decline a hand-rolled parser; what it missed was one search. The lesson is 'enumerate implementations', not 'refusals are usually wrong'.
- A route that is now cheap will get used a lot, and this beat has already demonstrated how easy it is to produce a confident, wrong table from it - the first readout pass returned per-base arrays where it expected means, and a second pass returned zero for every token because a loop overwrote its own output file. Both were caught by implausibility, not by a check.
- Does pybigtools agree numerically with pyBigWig or bigWigSummary on the same file and interval? One SSH command to the Explorer cluster would establish it, and it is the only validation the cycle-57 refusal asked for that is still missing.
- How many other entries in this graph carry a costed refusal whose blocker was 'the tool will not build'? Each is a candidate for the same treatment, and the pattern - search for a second implementation before refusing - generalises past bigWig.
- Is the hg19+100,000 padding in GSE210480 a HOMER makeUCSCfile artefact, and does it appear in other HOMER-derived bigWigs this project might read? If so it is a class of trap, not one series' quirk.
- E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.database
On this Windows host as of 2026-09-22, pyBigWig is absent and unimportable, deeptools is absent, and bigWigSummary, bigWigAverageOverBed, bigWigToBedGraph, bigWigInfo, samtools and bedtools are all absent from PATH, with 26 GB free on C: and 14 GB free on D:; pybigtools 0.3.0 fails to install under the system CPython 3.14 because no wheel exists and the maturin/cargo source build exits 101, and installs successfully from a prebuilt wheel under the uv-managed CPython 3.12.13 at D:/Users/ashenoy00000/AppData/Roaming/uv/python/cpython-3.12.13-windows-x86_64-none; it accepts an HTTPS URL in place of a file path and serves interval queries by HTTP range request against the BBI index, so nineteen GEO tracks totalling about 2.9 GB on the server were read without downloading any of them.
BigTools / pybigtools - a Rust library and Python bindings for the BigWig and BigBed (BBI) formats; installation and remote-read behaviour verified on this host 2026-09-22 · Huey JD, Kaplan N (BigTools project) · PyPI / GitHub 2024What this evidence cannot show
Version-specific and host-specific: pybigtools 0.3.0, CPython 3.12.13, Windows Server 2022. A future pybigtools release may add a 3.14 wheel, in which case the venv step becomes unnecessary; equally, a future release may change the values() signature again, as it already changed the NaN default. The remote-read path depends on the GEO endpoint honouring HTTP range requests, which ftp.ncbi.nlm.nih.gov over HTTPS currently does; a mirror that does not would force a full download. Nothing here validates pybigtools' numerical agreement with pyBigWig or the UCSC binaries on the same file, because neither is available to compare against - the validation is against biological expectation (E2), not against a reference implementation.
Evidence that argues against this (1)
- The absence of a cross-implementation numerical check is a real gap and it is the one thing the cycle-57 refusal would still object to. A reader that silently mis-handled zoom levels or the R-tree would produce internally consistent, biologically plausible numbers - which is exactly what BigTools' own zoom-level path is used for in the binned chr11 scan here. The mitigation used is that the headline percentile was computed from per-base values in the region and from binned values genome-wide, and the region's value is consistent between the two paths; a stronger check would be to run the same query on the Explorer cluster with pyBigWig, which is one SSH command away and was not done. NCBI Gene Expression Omnibus 2024
- E2established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.dataset
The reader passes the positive and negative control loci that the cycle-57 refusal required: over the H19/IGF2 imprinting control region at GRCh37 chr11:2,020,000-2,024,000, the canonical CTCF-bound element, HCT116 CTCF ChIP reads 7.78x its file's genome mean and RAD21 11.34x against matched input 1.02x, whereas the GAPDH promoter at GRCh37 chr12:6,643,000-6,644,000 reads CTCF 0.97x against input 2.37x and a chr2 gene desert at GRCh37 chr2:20,000,000-20,010,000 reads CTCF 0.41x against input 0.92x; and GSE210480's H3K9me3 track GSM6430988_CTRLsiKDM3B_H3K9me3_rep1.bigWig opens and reports 25 chromosomes whose lengths are each exactly hg19 plus 100,000 bp (chr1 249,350,621; chr2 243,299,373; chr11 135,106,516; chrX 155,370,560).
Control-locus reads of GEO GSM3714939/GSM3714940/GSM3714941 and a header read of GEO GSM6430988, with pybigtools 0.3.0 over HTTP range requests, 2026-09-22 · Q107 beat; data from Canela et al. 2019 (GSE129526) and Gray et al. 2023 (GSE210480) · NCBI GEO 2026What this evidence cannot show
The controls are enrichment-direction checks, not calibration: they show the instrument reads high where CTCF is known to bind and low where it is not, which is what the refusal asked for, but they do not establish that an absolute value is correct. The H19 ICR window is 4 kb and the CTCF sites within it are a subset, so 7.78x understates the local peak. The GAPDH promoter is an imperfect CTCF negative - promoters carry some CTCF genome-wide - which is why the input-corrected ratio (0.41) is quoted alongside. The GSE210480 header read establishes readability and the padded chrom.sizes only; no signal value from that series was read or reported here.
Evidence that argues against this (1)
- One control result argues for caution rather than confidence: the matched input track reads 2.37x its genome mean at the GAPDH promoter and 2.67x at KMT2A exon 11. Both are highly mappable, high-GC, open regions and the input is doing what input does - which means every fold-over-genome-mean figure in this beat is inflated at exactly the kind of region the beat is interested in. The fold-over-input and the rank-based statistics are not affected, and they are the ones the conclusions rest on; but any raw x_gmean number quoted out of context from this beat will be about 2x too high. Q107 beat; data from Canela et al. 2019 2026
the tooling that made the readout possible, with two API traps and a padded chrom.sizes trap named
the H3K9me3 tracks open; that analysis has no remaining blocker
overturns the cycle-57 costed refusal by a fourth option it did not consider: a maintained Rust implementation with prebuilt Windows wheels