Scientific foundation
Three questions behind the work.
The research is organised around questions, not products. Each question has a frozen protocol, public data and an honest statement of what is and is not known.
Question 1 · RNA observability
What determines whether RNA can be reliably observed?
Research Manuscript submitted to NAR Genomics and Bioinformatics. Not yet peer-reviewed.
“Does experimentally measured in-vivo RNA structure improve prediction of workflow-specific transcript observability beyond abundance, sequence, transcript architecture, identifiability and callability?”
RNA-sequencing is a measurement process, not a transparent window on biology. A transcript that is hard to observe is not necessarily absent, and workflows that disagree do not tell you which one is right.
RNA structure is a plausible contributor: molecules fold, and folding can affect biochemical access. But an association between structure and sequencing behaviour could simply reflect the stronger, known explanations. So the study asked a narrow question: does independently measured in-vivo structure add information after those explanations are accounted for?
Design
- System
- K562 cells; 15,999 transcripts; Ensembl 91 (GRCh38)
- Workflows
- Illumina short-read RNA-seq vs Oxford Nanopore direct-RNA; quantified with Salmon 1.9.0
- Endpoint
- Five workflow-specific classes: BOTH, ILLUMINA_ONLY, DIRECT_RNA_ONLY, NEITHER, INDETERMINATE
- Comparator
- Strongest locked structure-blind model: abundance, sequence composition, transcript architecture, identifiability, assay callability
- Addition
- One prespecified structure feature from GEO GSE132099, pooled K562 in-vivo icSHAPE
- Evaluation
- 5 held-out folds grouping same-gene and similar sequences; paired cluster bootstrap; negative-control permutations; rules locked before structure access
NOT_SUPPORTED; practical classification: NEGLIGIBLE. Values are exact, from the public repository.| Structure-blind macro-F1 | 0.54890983706729 |
|---|---|
| With structure, macro-F1 | 0.5498834635530492 |
| Difference | +0.0009736264857591603 |
| Positive folds | 4 of 5 |
| Strict-callability cohort | 15,114 transcripts, +0.0007399110130869024 |
| Decision | NOT_SUPPORTED · NEGLIGIBLE |
What we found
Within the tested K562 setting and the prespecified workflow-specific endpoint, transcript-wide experimental RNA structure provided negligible incremental predictive value beyond the other variables considered.
What this does not mean
It does not mean RNA structure does not matter. It does not generalise to other cell types, workflows, endpoints or position-level structure effects, none of which were tested.
Why it matters for the business
Measurement reliability depends on observable properties of the molecule and the workflow. Those limits should be modelled explicitly, and a candidate property should earn its place with evidence.
Question 2 · Variant evidence adjudication (CrossCall)
How should computational systems reason about sequencing disagreement?
In development Experimental research. No model has been evaluated and no performance is claimed.
CrossCall studies whether calibrated models with explicit abstention can adjudicate disagreements between paired Illumina and Oxford Nanopore germline small-variant call sets. It is not a variant caller and it is not a clinical tool.
Scope: GRCh38 chromosomes 1–22, germline SNVs and small indels (1–49 bp), paired Illumina and Oxford Nanopore call sets for the same sample.
Protocol design
- Evidence
- Illumina and Oxford Nanopore call-level evidence for the same candidate episode
- Features
- 42 approved features: call evidence from each platform plus local reference context
- Preprocessing
- Frozen before modelling
- Baseline protocol
- Frozen before modelling
- Extraction
- Deterministic, audited feature extraction
- Decision layer
- Selective prediction with explicit abstention
- Model results
- None yet. No performance has been measured or claimed.
Evidence states
BOTH_SUPPORTEDBoth submitted calls are truth-equivalent.ILLUMINA_SUPPORTEDThe Illumina call is truth-equivalent; the ONT call is not.ONT_SUPPORTEDThe ONT call is truth-equivalent; the Illumina call is not.NEITHER_SUPPORTEDAt least one workflow made a call, but neither interpretation is truth-equivalent.
Model action
ABSTAIN
An uncertainty-driven model action emitted instead of a support state. The model declines to answer a case that could, in principle, be answered.
Benchmark condition
UNASSESSABLE
A dataset-labelling condition: truth equivalence cannot be established reliably (no-call, excluded allele class, benchmark boundary, comparator ambiguity). Excluded from supervised labels.
Benchmark uncertainty and model abstention are different things and are never merged. One is a property of the data; the other is a decision the model makes.
Question 3 · OmicsEdge Evidence Atlas
How can benchmark evidence remain inspectable and provenance-rich?
Released v1.0.0 Released 2026-10-01 as OmicsEdge Benchmark Evidence Registry v1.0.0, DOI 10.5281/zenodo.23085673. 20 / 20 pre-specified Phase 2 release-validation checks passed (release, protocol and data-integrity criteria; not clinical or model-performance validation).
Most benchmarks answer: how well did this call set perform? The Evidence Atlas asks: what empirical evidence exists for this biological variant across experiments, technologies, benchmark runs and truth versions, and exactly where did it come from?
It preserves run-scoped events and observations rather than collapsing them into a score, and separates the biological variant from the benchmark event. A variant can accumulate evidence from many runs without pretending those runs are the same.
The Atlas is an evidence and provenance resource. It is not a predictive model. It is scientifically independent of the variant adjudication work above: it uses no part of it and is not an input to it.
Selection without looking
Sources and the 50-window Phase 2 interval panel were frozen before any benchmark outcome was inspected; zero query outcomes were used during panel selection. chr20, used in the Phase 1 proof of concept, was deliberately excluded from Phase 2.
Sources
| Sample | Technology | Platform | Pipeline | Coverage | Experiment | Phase |
|---|---|---|---|---|---|---|
| HG002 | Illumina | NovaSeq 6000 | RN-Illumina | 35x | exp-pfda-v2-60z59-hg002 | Phase 1A · chr20 |
| HG002 | Oxford Nanopore | PromethION R9.4 | PEPPER-DeepVariant (ONT) | 47x | exp-pfda-v2-ru88n-hg002 | Phase 1A · chr20 |
| HG003 | Illumina | NovaSeq 6000 | RN-Illumina | 35x | exp-pfda-v2-60z59-hg003 | Phase 2C · panel |
| HG003 | Oxford Nanopore | PromethION R9.4 | PEPPER-DeepVariant (ONT) | 85x | exp-pfda-v2-ru88n-hg003 | Phase 2C · panel |
| HG004 | Illumina | NovaSeq 6000 | RN-Illumina | 35x | exp-pfda-v2-60z59-hg004 | Phase 2C · panel |
| HG004 | Oxford Nanopore | PromethION R9.4 | PEPPER-DeepVariant (ONT) | 85x | exp-pfda-v2-ru88n-hg004 | Phase 2C · panel |
All runs: hap.py 0.3.15 with vcfeval 3.12.1 against GIAB v4.2.1 on GRCh38. Public precisionFDA Truth Challenge V2 submissions. Read from the frozen Evidence Atlas bundle.
How the Phase 2 panel was chosen
Ten 25 kb windows were selected for each of five GIAB v3.6 stratifications, by coordinates alone. These describe window selection. The v1 release deliberately assigns no per-variant context label, because a window's selection context is not the same as the context at each base.
| Window-selection stratification | GIAB v3.6 file |
|---|---|
| Not in difficult regions | GRCh38_notinalldifficultregions.bed.gz |
| Homopolymer | GRCh38_AllHomopolymers_ge7bp_imperfectge11bp_slop5.bed.gz |
| Tandem repeat | GRCh38_AllTandemRepeats.bed.gz |
| Low mappability | GRCh38_lowmappabilityall.bed.gz |
| Segmental duplication | GRCh38_segdups.bed.gz |
Lineage of one record
Each Evidence Atlas record keeps the chain from public source to biological variant.
Visual vocabulary
- Filled observed: a call or observation is present
- Outline not observed, or no event for that run
- Split technologies disagree for the same sample and variant
- Solid line exact relationship
- Dashed line unresolved relationship
01 · Source benchmark
Truth set artifact-truth-hg003-grch38-v421
- query call set: artifact-query-60z59-hg003
- benchmark regions: artifact-phase2-frozen-panel
02 · Version
hap.py 0.3.15 · vcfeval 3.12.1
- runtime image d63b963a6cb0…
- reference: artifact-reference-grch38-no-alt
03 · Sample
HG003
- experiment exp-pfda-v2-60z59-hg003
04 · Platform
Illumina NovaSeq 6000
- pipeline RN-Illumina · 35x
05 · Event
chr8:3,989,216 T>C (1-based, GRCh38)
- event-run-phase2c-hg003-60z59-grch38-v421-chr8-record-1226
- benchmark run run-phase2c-hg003-60z59-grch38-v421
06 · Observations
truth TP · query TP · GT 0/1 · match gm
- obs-run-phase2c-hg003-60z59-grch38-v421-1226-truth
- obs-run-phase2c-hg003-60z59-grch38-v421-1226-query
07 · Atlas record
EXACT_NORMALIZED_ALLELE → biological variant
- event-variant-link:sha256:7bb5…
- variant:sha256:00e903d4b12125d074b…
- 4 linked events across 2 samples and 2 technologies
What the release deliberately does not generate
- No caller ranking
caller_ranking_disabled - No clinical interpretation
clinical_interpretation_disabled - No consensus generation
consensus_generation_disabled - No per-variant genomic-context labels
per_variant_genomic_context_disabled - No reliability scoring
reliability_scoring_disabled - No representation-equivalence inference
representation_equivalence_disabled - No technology ranking
technology_ranking_disabled - No trust scoring
trust_scoring_disabled
Technical provenance
- Release archive
- results/phase2e/omicsedge_phase2_final_results.tar.gz
- SHA-256
- 186cbed74fa718a728f7a24b43d58118a308165fae385eb95cc4cb4444788ca3
- Web bundle
- schema 1.0.0 · search index 5aa6736173ed1782… · served byte-identical at
/data/evidence-atlas/v1/ - Repository
- github.com/omicsedgebio/omicsedge-benchmark-evidence
Limitations recorded in the release: four admitted public query artifacts plus Phase 1 HG002 evidence, not a representative survey of platforms, callers, chemistries or populations; GIAB v4.2.1 is treated as a comparator reference, not perfect biological truth; evidence aggregation does not establish biological or clinical truth.
Methodology
How the work is done.
- Freeze before looking. Protocols, data selection and decision rules are locked before outcomes are inspected.
- Report negative results. The RNA structure result is negligible, and says so.
- Separate uncertainty types. Benchmark uncertainty, unresolved identity and model abstention are distinct and labelled.
- Checksum everything. Releases are manifest-verified and independently re-extracted.
- Use public data. Every dataset here can be obtained and reanalysed by anyone.