Scientific foundation

Three questions behind the work.

The research is organised around questions, not products. Each question has a frozen protocol, public data and an honest statement of what is and is not known.

  1. Q1 What determines whether RNA can be reliably observed?
  2. Q2 How should software reason about sequencing disagreement?
  3. Q3 How can benchmark evidence stay inspectable? Released: Evidence Atlas

Question 1 · RNA observability

What determines whether RNA can be reliably observed?

Research Manuscript submitted to NAR Genomics and Bioinformatics. Not yet peer-reviewed.

“Does experimentally measured in-vivo RNA structure improve prediction of workflow-specific transcript observability beyond abundance, sequence, transcript architecture, identifiability and callability?”

RNA-sequencing is a measurement process, not a transparent window on biology. A transcript that is hard to observe is not necessarily absent, and workflows that disagree do not tell you which one is right.

RNA structure is a plausible contributor: molecules fold, and folding can affect biochemical access. But an association between structure and sequencing behaviour could simply reflect the stronger, known explanations. So the study asked a narrow question: does independently measured in-vivo structure add information after those explanations are accounted for?

Design

System
K562 cells; 15,999 transcripts; Ensembl 91 (GRCh38)
Workflows
Illumina short-read RNA-seq vs Oxford Nanopore direct-RNA; quantified with Salmon 1.9.0
Endpoint
Five workflow-specific classes: BOTH, ILLUMINA_ONLY, DIRECT_RNA_ONLY, NEITHER, INDETERMINATE
Comparator
Strongest locked structure-blind model: abundance, sequence composition, transcript architecture, identifiability, assay callability
Addition
One prespecified structure feature from GEO GSE132099, pooled K562 in-vivo icSHAPE
Evaluation
5 held-out folds grouping same-gene and similar sequences; paired cluster bootstrap; negative-control permutations; rules locked before structure access
Change in held-out macro-F1 when adding measured RNA structurePrimary cohort: +0.0010, 95% paired cluster-bootstrap interval −0.0023 to +0.0041. Strict-callability cohort: +0.0007, interval −0.0034 to +0.0048. Both intervals cross zero. The negative-control permutation 95th percentile, +0.0015, is above the observed increment. Locked decision: NOT_SUPPORTED; practical classification: NEGLIGIBLE.−0.004−0.0020+0.002+0.004negative-control 95th pct +0.0015Δ macro-F1 (structure − structure-blind)Primary cohort15,999 transcriptsfold deltas+0.0010Strict callability15,114 transcripts+0.0007
Adding measured structure changed pooled held-out macro-F1 by +0.0010. Both bootstrap intervals cross zero and the increment sits below the negative-control threshold. Locked decision: NOT_SUPPORTED; practical classification: NEGLIGIBLE. Values are exact, from the public repository.
Exact locked values
Structure-blind macro-F10.54890983706729
With structure, macro-F10.5498834635530492
Difference+0.0009736264857591603
Positive folds4 of 5
Strict-callability cohort15,114 transcripts, +0.0007399110130869024
DecisionNOT_SUPPORTED · NEGLIGIBLE

What we found

Within the tested K562 setting and the prespecified workflow-specific endpoint, transcript-wide experimental RNA structure provided negligible incremental predictive value beyond the other variables considered.

What this does not mean

It does not mean RNA structure does not matter. It does not generalise to other cell types, workflows, endpoints or position-level structure effects, none of which were tested.

Why it matters for the business

Measurement reliability depends on observable properties of the molecule and the workflow. Those limits should be modelled explicitly, and a candidate property should earn its place with evidence.

Public repository, protocol and claims register

Question 2 · Variant evidence adjudication (CrossCall)

How should computational systems reason about sequencing disagreement?

In development Experimental research. No model has been evaluated and no performance is claimed.

CrossCall studies whether calibrated models with explicit abstention can adjudicate disagreements between paired Illumina and Oxford Nanopore germline small-variant call sets. It is not a variant caller and it is not a clinical tool.

Scope: GRCh38 chromosomes 1–22, germline SNVs and small indels (1–49 bp), paired Illumina and Oxford Nanopore call sets for the same sample.

Protocol design

Evidence
Illumina and Oxford Nanopore call-level evidence for the same candidate episode
Features
42 approved features: call evidence from each platform plus local reference context
Preprocessing
Frozen before modelling
Baseline protocol
Frozen before modelling
Extraction
Deterministic, audited feature extraction
Decision layer
Selective prediction with explicit abstention
Model results
None yet. No performance has been measured or claimed.

Evidence states

  • BOTH_SUPPORTEDBoth submitted calls are truth-equivalent.
  • ILLUMINA_SUPPORTEDThe Illumina call is truth-equivalent; the ONT call is not.
  • ONT_SUPPORTEDThe ONT call is truth-equivalent; the Illumina call is not.
  • NEITHER_SUPPORTEDAt least one workflow made a call, but neither interpretation is truth-equivalent.

Model action

ABSTAIN

An uncertainty-driven model action emitted instead of a support state. The model declines to answer a case that could, in principle, be answered.

Benchmark condition

UNASSESSABLE

A dataset-labelling condition: truth equivalence cannot be established reliably (no-call, excluded allele class, benchmark boundary, comparator ambiguity). Excluded from supervised labels.

Benchmark uncertainty and model abstention are different things and are never merged. One is a property of the data; the other is a decision the model makes.

Question 3 · OmicsEdge Evidence Atlas

How can benchmark evidence remain inspectable and provenance-rich?

Released v1.0.0 Released 2026-10-01 as OmicsEdge Benchmark Evidence Registry v1.0.0, DOI 10.5281/zenodo.23085673. 20 / 20 pre-specified Phase 2 release-validation checks passed (release, protocol and data-integrity criteria; not clinical or model-performance validation).

Most benchmarks answer: how well did this call set perform? The Evidence Atlas asks: what empirical evidence exists for this biological variant across experiments, technologies, benchmark runs and truth versions, and exactly where did it come from?

It preserves run-scoped events and observations rather than collapsing them into a score, and separates the biological variant from the benchmark event. A variant can accumulate evidence from many runs without pretending those runs are the same.

The Atlas is an evidence and provenance resource. It is not a predictive model. It is scientifically independent of the variant adjudication work above: it uses no part of it and is not an input to it.

Selection without looking

Sources and the 50-window Phase 2 interval panel were frozen before any benchmark outcome was inspected; zero query outcomes were used during panel selection. chr20, used in the Phase 1 proof of concept, was deliberately excluded from Phase 2.

The full release Open the Evidence Explorer

11,553benchmark events
23,106observations
4,278biological variants
11,223exact identity events
60unresolved events
270not evaluated for identity

Sources

SampleTechnologyPlatformPipelineCoverageExperimentPhase
HG002IlluminaNovaSeq 6000RN-Illumina35xexp-pfda-v2-60z59-hg002Phase 1A · chr20
HG002Oxford NanoporePromethION R9.4PEPPER-DeepVariant (ONT)47xexp-pfda-v2-ru88n-hg002Phase 1A · chr20
HG003IlluminaNovaSeq 6000RN-Illumina35xexp-pfda-v2-60z59-hg003Phase 2C · panel
HG003Oxford NanoporePromethION R9.4PEPPER-DeepVariant (ONT)85xexp-pfda-v2-ru88n-hg003Phase 2C · panel
HG004IlluminaNovaSeq 6000RN-Illumina35xexp-pfda-v2-60z59-hg004Phase 2C · panel
HG004Oxford NanoporePromethION R9.4PEPPER-DeepVariant (ONT)85xexp-pfda-v2-ru88n-hg004Phase 2C · panel

All runs: hap.py 0.3.15 with vcfeval 3.12.1 against GIAB v4.2.1 on GRCh38. Public precisionFDA Truth Challenge V2 submissions. Read from the frozen Evidence Atlas bundle.

How the Phase 2 panel was chosen

Ten 25 kb windows were selected for each of five GIAB v3.6 stratifications, by coordinates alone. These describe window selection. The v1 release deliberately assigns no per-variant context label, because a window's selection context is not the same as the context at each base.

Window-selection stratificationGIAB v3.6 file
Not in difficult regionsGRCh38_notinalldifficultregions.bed.gz
HomopolymerGRCh38_AllHomopolymers_ge7bp_imperfectge11bp_slop5.bed.gz
Tandem repeatGRCh38_AllTandemRepeats.bed.gz
Low mappabilityGRCh38_lowmappabilityall.bed.gz
Segmental duplicationGRCh38_segdups.bed.gz

Lineage of one record

Each Evidence Atlas record keeps the chain from public source to biological variant.

Visual vocabulary

  • Filled observed: a call or observation is present
  • Outline not observed, or no event for that run
  • Split technologies disagree for the same sample and variant
  • Solid line exact relationship
  • Dashed line unresolved relationship
  1. 01 · Source benchmark

    Truth set artifact-truth-hg003-grch38-v421

    • query call set: artifact-query-60z59-hg003
    • benchmark regions: artifact-phase2-frozen-panel
  2. 02 · Version

    hap.py 0.3.15 · vcfeval 3.12.1

    • runtime image d63b963a6cb0…
    • reference: artifact-reference-grch38-no-alt
  3. 03 · Sample

    HG003

    • experiment exp-pfda-v2-60z59-hg003
  4. 04 · Platform

    Illumina NovaSeq 6000

    • pipeline RN-Illumina · 35x
  5. 05 · Event

    chr8:3,989,216 T>C (1-based, GRCh38)

    • event-run-phase2c-hg003-60z59-grch38-v421-chr8-record-1226
    • benchmark run run-phase2c-hg003-60z59-grch38-v421
  6. 06 · Observations

    truth TP · query TP · GT 0/1 · match gm

    • obs-run-phase2c-hg003-60z59-grch38-v421-1226-truth
    • obs-run-phase2c-hg003-60z59-grch38-v421-1226-query
  7. 07 · Atlas record

    EXACT_NORMALIZED_ALLELE → biological variant

    • event-variant-link:sha256:7bb5…
    • variant:sha256:00e903d4b12125d074b…
    • 4 linked events across 2 samples and 2 technologies
A real record, end to end. The variant the frozen release selects in its own deterministic product query. Every line is read from the Evidence Atlas v1 bundle. Open it in the Evidence Explorer.

What the release deliberately does not generate

  • No caller ranking caller_ranking_disabled
  • No clinical interpretation clinical_interpretation_disabled
  • No consensus generation consensus_generation_disabled
  • No per-variant genomic-context labels per_variant_genomic_context_disabled
  • No reliability scoring reliability_scoring_disabled
  • No representation-equivalence inference representation_equivalence_disabled
  • No technology ranking technology_ranking_disabled
  • No trust scoring trust_scoring_disabled
Technical provenance
Release archive
results/phase2e/omicsedge_phase2_final_results.tar.gz
SHA-256
186cbed74fa718a728f7a24b43d58118a308165fae385eb95cc4cb4444788ca3
Web bundle
schema 1.0.0 · search index 5aa6736173ed1782… · served byte-identical at /data/evidence-atlas/v1/
Repository
github.com/omicsedgebio/omicsedge-benchmark-evidence

Limitations recorded in the release: four admitted public query artifacts plus Phase 1 HG002 evidence, not a representative survey of platforms, callers, chemistries or populations; GIAB v4.2.1 is treated as a comparator reference, not perfect biological truth; evidence aggregation does not establish biological or clinical truth.

Methodology

How the work is done.

  • Freeze before looking. Protocols, data selection and decision rules are locked before outcomes are inspected.
  • Report negative results. The RNA structure result is negligible, and says so.
  • Separate uncertainty types. Benchmark uncertainty, unresolved identity and model abstention are distinct and labelled.
  • Checksum everything. Releases are manifest-verified and independently re-extracted.
  • Use public data. Every dataset here can be obtained and reanalysed by anyone.