gffbase 0.3.0

gffbase 0.3.0

on Python PyPI

A performance and robustness release: ingest rebuilt around a streaming Rust producer, relation queries rewritten around what DuckDB can index, loops over a live iterator prefetched, and the noisy files real annotation sources publish read as they were meant.

pip install --upgrade gffbase

abi3 wheels for Linux (x86_64, aarch64), macOS (Intel, Apple Silicon) and Windows (x86_64) cover CPython 3.10 through 3.14; no Rust toolchain is needed. No database migration is needed: 0.3.0 reads and writes the same schema as 0.2.x.

Faster, in far less memory

Against gffutils 0.14 (one canonical run on the installed wheel, exhaustive validation, correctness signatures matching on every row):

Against 0.2.1 (ingest only, same machine and cores): GENCODE v49 GFF3 537 s and 9.7 GiB peak → 192 s and 2.8 GiB; RefSeq 347 s and 6.9 GiB → 119 s and 2.0 GiB; MANE 36 s and 1.7 GiB → 15 s and 1.0 GiB. Databases are about half the size they were.

• Ingest is rebuilt. The Rust parser streams its input instead of decompressing the whole file first; a Rust producer resolves ids and duplicates and hands each batch to DuckDB as Arrow with no Python object per record; indexes are single-column, built over sorted input, one at a time; the closure is built level by level. Ingest into a file runs under a 512 MB DuckDB budget, raised only for the step that needs more.
• children() and parents() are 3–25× faster per call: one single-table lookup, then the features by key, instead of a join DuckDB answered by scanning the whole closure.
• Loops are prefetched. A children(), parents() or db[id] call on a feature an open gffbase iterator holds is answered, with its neighbours', by one query — nested loops too — so the canonical gffutils loops run 4–21× faster than calling per id, 1.4–3.5× behind gffutils 0.14. Answers are identical to the per-call path (a Hypothesis state machine interleaving streams, calls and writes holds them to it). A scattered call on an id from your own list is still a DuckDB statement; batch those with children_batched.
• DuckDB uses at most 8 threads by default (GFFBASE_THREADS overrides): more bought no time and cost memory.

Noisy files, read as meant

Byte-order marks, lone-CR line endings, whitespace-only lines, . in column 9, Parent=a,, spaced =; CDS-only and AUGUSTUS GTF; gene_id without transcript_id; UCSC GTFs with one transcript id on several contigs; Liftoff _1/_2 ids; FlyBase's gzipped tar; NUL bytes; truncated gzip (GFFFormatError(kind="ReadError") in both engines). Each is on the new Noisy and non-standard files page, and every change to what a query returns is declared in tests/parity/deviations.toml. Nine more annotation sources — Ensembl mouse, NCBI E. coli, FlyBase, WormBase, UCSC knownGene and ncbiRefSeq, a T2T Liftoff annotation — are ingested and fully validated by the test suite.

Behaviour you may notice

• Inside a loop over a gffbase iterator, children(), parents() and db[id] may be answered from a prefetch. Writes through FeatureDB invalidate it; SQL run directly on db.conn inside such a loop is not seen by that loop's prefetched answers.
• The ingest connection's DuckDB memory limit is managed by gffbase unless pragmas sets one; the handle you get back uses DuckDB's default.
• DuplicateIDError names both lines; the CLI reports a user error in one line and exits 1; the dialect is voted per key, weighted by attribute count, as gffutils does.

The full list is in the changelog; tuning advice is on the new Memory and performance tuning page.

How this release was verified

• Qualified in CI on Linux, macOS and Windows across Python 3.10–3.14 on both spatial index paths, with a five-corpus correctness gate, the gffutils parity suite, the pure-Python engine without the extension, and the minimum supported dependencies.
• Each rebuilt path is held to the one it replaced by an equivalence oracle: the Python ingest loop (producer), the row-by-row duplicate pass, the recursive closure query, the closure join (relations) and the per-call path (prefetch).
• The benchmark table above is generated from one canonical run on the installed wheel, with exhaustive validation, and publishes a ratio only where both engines' correctness signatures match.
• The binaries attached here are byte-identical to the files on PyPI.

Docs: https://khchao.com/gffbase/ · Changelog: CHANGELOG.md · PyPI: https://pypi.org/project/gffbase/0.3.0/