Oncofusion

start / graph / ana-q118-the-deposited-processed-table-quotes-every-value-above-999

Analysisestablished confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.Q118-menin-at-the-human-index-fusion

The deposited GSE127507 processed signal table is an Excel export in which every value at or above 1,000 is a quoted comma-separated string, so a plain numeric parse silently zeroes 34,955 of 386,298 cells - and those cells are the entire top of all eighteen distributions

also called the quoted-thousands trap · 9% of a processed table is unparseable and it is the top 9%

The ready-made results table in this public dataset came out of a spreadsheet, so every number over 999 is stored with a comma inside quotation marks. A normal number-reading program turns exactly those into zero - and those are all the strongest signals, which is the only part anyone wants.

The same thing, for a scientist

`GSE127507_ML2_MOLM13_RS411_ROSE_TSS-associated_DOT1L_MENIN_MLL1_ChIPseq_signal.txt.gz` is the only series-level processed file in the deposit and the obvious starting point for anyone re-using it: 21,461 RefGene TSS windows x 18 samples, hg19, input-subtracted. **It is an Excel export.** Values below 1,000 are bare integers. **Values at or above 1,000 are written as a quoted, comma-separated string** - the literal cell content is `" 1,015 "` - and zero/negative is the token `-` [E1]. | | | |---|---:| | cells in the table | 386,298 (21,461 x 18) | | cells that `float()` refuses | **34,955** | | as a fraction | **9.05%** | | distinct unparseable tokens | 5,909 | | every one of them | a value >= 1,000 | **The failure is aligned with the analysis.** A parser that treats an unparseable cell as zero - which is the natural defensive choice - removes precisely the top of every distribution. Measured on the first probe run, before the fix: every one of the eighteen columns reported a maximum of **993-999**, medians were depressed by 20-60%, and the column totals were 12-83% too low. **`MOLM13_DOT1L_DMSO`'s true total is 34,279,592 and the broken parse gave 3,321,331 - a ten-fold error, in the column with the most signal.** Nothing about the broken output looks broken. Eighteen columns with maxima clustered at 999 reads like a saturating detector or a capped export, which is a plausible property of a browser-signal table. **It was caught only because 999 as a maximum in all eighteen columns is too tidy**, and the check that settled it was printing the distinct unparseable tokens rather than counting them. **The fix is `s.strip().strip('"').strip().replace(',', '')`.** The lesson is the family, not the string: *a deposited processed table is a file that came out of somebody's spreadsheet, and the cells that got reformatted are the large ones.*

Go deeper
The general form, and the three siblings this graph already holds

This is the same shape as failures this brief already documents: the empty <ArticleTitle>, the PMC record with no <body>, the esearch returning a well-formed count: 0, and the supplementary .xlsx that is a valid zip containing a zero-byte file. The shape to distrust is well-formed, correctly named, and quietly missing its content.

What makes this one worse than those is that the loss is proportional to the value. An empty title is visibly empty. A table whose top 9% of cells are zero still has 91% of its cells right, still sums, still ranks, still plots - and every rank statistic computed on it is wrong in the same direction, at the extreme where every claim lives.

Two operational rules that would have caught it in one line each:

  1. Count the cells your parser refused, and print a sample of them. Not the count - the strings. Five minutes.
  2. Check the maximum of every column against the number of digits it could hold. All eighteen maxima being 993-999 in a table of ChIP signal is a statement about the parser, not the data.
Facts and figures
scriptanalyses/q118-menin-at-the-human-index-fusion/roseA.py (n_cells_needing_quoted_thousands_fix)
statusrun
result summary9.05% of cells are quoted comma-separated strings, all >= 1,000; zeroing them caps all eighteen column maxima at 993-999 and loses 12-83% of each column total.
absence classmeasured-not-curated
absence class noteThe gap this token describes is that the values >= 1,000 are PRESENT in the deposited file but structurally invisible to the obvious numeric join - published, and absent from the data structure anyone would join against. It does not describe a missing measurement; every number is there.
inputs
[
  "GSE127507_ML2_MOLM13_RS411_ROSE_TSS-associated_DOT1L_MENIN_MLL1_ChIPseq_signal.txt.gz, 975,809 bytes"
]
Still open
  1. Do the other three subseries of GSE127508, and other Armstrong-lab GEO depositions, carry the same Excel formatting in their processed tables? One `zgrep` per file would say, and it was not run.
  2. Are there deposited processed tables in this project's other re-analyses that were parsed with a zero-on-failure fallback? A sweep of this repository's analysis scripts for that pattern was not run and is cheap.
Every source behind this entry (1)
  1. E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.dataset

    34,955 of the 386,298 numeric cells (9.05%) in the deposited GSE127507 ROSE signal table are quoted comma-separated strings, all of them values at or above 1,000, across 5,909 distinct tokens; a parse that zeroes them caps every one of the eighteen column maxima at 993-999 and understates MOLM13_DOT1L_DMSO's total by ten-fold.

    What this evidence cannot show

    The count is of cells this beat's parser initially refused and is exact for this file. Whether other GEO series from the same submitter carry the same formatting was not checked. The ten-fold understatement figure is for one column; the range across the eighteen is 12-83% of the total lost.

    Evidence that argues against this (1)
    • A parser that RAISES rather than zeroing would fail loudly and this would never have been a hazard, so the defect is partly in the defensive choice rather than in the file. A reader who uses pandas.read_csv with thousands=',' gets the right answer without noticing there was a problem - so the trap only catches hand-written parsers, which is what this project writes. NCBI Gene Expression Omnibus 2019
Connected to