NVIDIA/ncore v20.0.0-alpha.1

on GitHub

Highlights

First pre-release of NCore 20, for testing in downstream projects before the 20.0 release. Install with pip install --pre nvidia-ncore or nvidia-ncore==20.0.0a1. The public reader / writer APIs of components are unchanged; APIs below the components (stores, zarr access) changed, and may still change based on feedback until 20.0.

zarr-python 3 support (zarr-python 2 remains supported)

• NCore works with zarr-python 2.x and 3.x (>= 3.1.4); the matching implementation is selected at import time.
• Component stores can be written in zarr format v3 (zarr_format=3, --zarr-format 3 in the data converters; requires zarr-python 3). zarr format v2 stays the default and is readable by all NCore and zarr-python versions.
• Data written with zarr-python 2 and 3 is equivalent (byte-identical chunk records).

Faster reads

Frame reads of the test sequence (local .itar archives, all frames of all sensors) compared to NCore 19.8:

• Readers cache opened groups / arrays (node_cache_size), local archives are read via positional reads, and reads of complete single-chunk arrays (e.g., encoded images) return their chunk record without copies.
• With zarr-python 3, chunks are decompressed directly into the output array instead of via zarr-python's codec pipeline: multi-chunk arrays read 1.4x to 3.7x faster, multiple large chunks up to ~17x (concurrent, in-place decompression).
• Opening a sequence takes ~70 ms with zarr-python 2 and ~300 ms with zarr-python 3, which constructs all nodes of the consolidated metadata up front.

Asynchronous reads

• All component reader methods reading array data have *_async variants (e.g. get_frame_data_async, get_frame_ray_bundle_data_async). Awaiting reads concurrently overlaps their I/O and decompression without blocking the event loop; on remote storage (e.g. s3:// archives), they fetch exact byte ranges concurrently: 3x to 4.5x faster than synchronous reads for random frame access at 20 ms request latency (simulated).
• Synchronous reads can fetch the same way with SequenceComponentGroupsReader(..., itar_remote_range_reads=True), so thread pools over the synchronous APIs fetch remote frames concurrently (opt-in, the buffered block cache suits sequential reading).

Breaking changes (APIs below the components)

• IndexedTarStore is replaced by itar.IndexedTarArchive (records of an archive, independent of zarr-python) and stores.IndexedTarZarrStore (a zarr hierarchy in an archive). Other zarr-python stores are used via stores.ZarrStoreAdapter (e.g., ZarrStoreAdapter.directory(path)).
• stores.consolidate_compressed_metadata / open_compressed_consolidated are replaced by Store.consolidate() / Store.open_root_group().
• Component readers / writers access their data via nodes.Group / nodes.Array (self._group), the wrapped zarr-python objects via .zarr. Array reads return read-only arrays. Groups also provide zarr-python style read access (group[path], get, iteration, len, keys, items, name).
• JsonLike moved from ncore.impl.data.types to ncore.impl.data.json_types, and accepts any mappings / sequences (e.g., read-only mappings, tuples) as values to write; documents to write (attributes, generic meta data of writers) are typed as JsonDocumentLike. Values as read are typed as Json / JsonDocument (dicts / lists).
• BaseDataConverterConfig requires zarr_format.
• Node metadata records of .itar archives are appended when closing the archive, so archives are no longer byte-identical to archives written by NCore 19.8 (their records are; NCore 19.8 reads them).

See the storage and access documentation (zarr versions and formats, asynchronous reads, cloud storage) for details.

Changes

➕ Added

• (data) range reads of remote archives for synchronous reads (888532a) - @janickm
• (data) asynchronous component reader API (8c2bf4e) - @janickm
• BREAKING (data) zarr format v3 stores (2658b05) - @janickm
• BREAKING (data) support zarr-python 3 (8756ac8) - @janickm
• BREAKING (data) typed node API for component groups (3f06b8c) - @janickm

🪲 Fixed

• (data) read single poses with their stored dtype (1de0192) - @janickm
• type narrowing in ncore_export_ply and the argoverse2 converter test (348262a) - @janickm
• (kitti) keep lidar point timestamps integer (5bf8f9f) - @janickm

⚡ Performance

• (data) direct chunk reads with zarr-python 3 (52d7e8e) - @janickm
• (data) node cache for component readers (a65383a) - @janickm
• (data) cache decoded label types and schemas in CameraLabelDescriptor.from_dict (1813046) - @janickm
• (data) load label and point-cloud reader data on first use (fe6eaf2) - @janickm

🔄 Changed

• BREAKING (data) implement IndexedTarArchive independently of zarr-python (eacc8fa) - @janickm

⚙️ CI

• publish the documentation built by build-and-test (ed2d016) - @janickm
• refresh the Bazel disk cache on every push (09a3665) - @janickm
• remove merge queue triggers (3f52089) - @janickm
• write release notes into the release commit body (ccc650a) - @janickm
• release from main and release branches when the version changes (e8aba41) - @janickm

🏗️ Build

• don't compress the repacked patched torch wheel (dc91c12) - @janickm
• add python 3.12 / 3.13 test environments (5b5fc02) - @janickm

🎨 Style

• remove unused type: ignore comments (4953168) - @janickm

🧪 Tests

• (sensors) share the lidar model between tests of a parameterization (c490c63) - @janickm
• run tests with a single OpenMP thread (2d4263f) - @janickm
• make tests independent of scipy / numpy versions (f2d59b0) - @janickm

Full diff: v19.8.0...v20.0.0-alpha.1