NVIDIA/ncore v20.0.0-beta.5

on GitHub

Highlights

Fifth beta of NCore 20, deriving the validity of OpenCV pinhole projections from the camera's radial distortion instead of fixed limits, and adding write APIs to the storage package. Install with pip install --pre nvidia-ncore or nvidia-ncore==20.0.0b5. Compared to NCore 19.8, see the beta.4 release notes and the earlier pre-releases they link.

OpenCV pinhole projection validity (since beta.4)

• OpenCVPinholeCameraModel previously invalidated rays whose rational radial distortion factor d was outside fixed limits (0.8, 1.2). These stand in for the distorted radius r d(r) no longer being one-to-one in r, but don't follow from the coefficients: they rejected valid rays in the periphery of cameras with strong barrel distortion, cut off pincushion distortions, and could accept rays beyond the distortion's extremum.
• Valid rays are now bounded by the first extremum of the distorted radius (or a pole of d), derived from the radial coefficients by the new static method OpenCVPinholeCameraModel.compute_max_normalized_radius_squared(radial_coeffs) (numpy only, also usable by other implementations of the model). Projections and unprojections of rays beyond it are invalid.
• The new constructor argument max_normalized_radius_squared (default None: derived; math.inf: no bound) overrides the bound. It is a setting of the model instance, not part of the camera model parameters, so stored data is unchanged. Distortion-free models keep projecting like ideal pinholes.
• Tangential and thin-prism distortions are not taken into account (as before).

Storage write and read APIs (since beta.4, ncore.data.stores)

All of them work the same with zarr-python 2 and 3 and both zarr formats, and write records byte-identical to Group.create_array.

• Array.read(selection, *, out=None) / read_async(..., *, out=None) read into a preallocated buffer and return it (validated before any read: dtype and shape of the selection, C-contiguous, writeable). zarr-python 3 direct reads decode the chunks straight into out.
• Group.create_empty_array(name, *, shape, dtype, chunks, compression, fill_value=None, attributes) creates an array with an explicit chunk grid and no stored chunks, written by Array.write(selection, data) (basic selections). Chunks never written are not stored and read as the fill value (default: the one create_array writes for the dtype). Indexed tar archives are write-once per chunk: rewrites raise the new RecordExistsError (a ValueError).
• EncodedArray.encode(data, *, dtype, chunks="single", compression, zarr_format=2) encodes the chunk records of an array without a store (safe on worker threads, e.g. to compress the arrays of a group in parallel), stored by Group.create_encoded_array(name, encoded, *, attributes).
• Store.open_root_group(consolidated="auto") opens from the consolidated metadata record if the store has one, and from the node metadata records otherwise (an index lookup for indexed tar archives).
• BloscCompression, EncodedArray and RecordExistsError are exported from ncore.data.stores; fill values are typed as FillValue.

Upgrading from beta.4

• Valid flags of OpenCV pinhole projections change for rays between the former limits and the distortion's extremum (now valid) and for rays beyond the extremum but within the former limits (now invalid). Cameras with moderate distortion are typically unaffected within their images (e.g., no change for the Waymo camera of the tests); a camera with strong barrel distortion (k1 = -0.1, f = 800 px, 1920x1280) gains valid rays in its periphery. Unprojections of image points whose rays lie beyond the extremum are invalid as well (previously only the convergence of the undistortion was checked). Pass max_normalized_radius_squared to bound the rays explicitly.
• ncore.data.stores.is_consolidated_metadata_key(key) is now Store.is_consolidated_metadata_key(key); the module-level function is removed.
• Store.open_root_group(consolidated=...) accepts only True, False and "auto"; other values, including integers and None, raise ValueError (previously they were used for their truth value).
• Writes through hierarchies opened for reading raise ValueError with both zarr-python versions: with zarr-python 3, hierarchies opened from consolidated metadata now use a read-only view of the store (as unconsolidated opens already did). Writable stores of types without read-only views raise ValueError when opened from consolidated metadata.

For upgrading from NCore 19.x, see the beta.3 release notes.

Changes since v20.0.0-beta.4

➕ Added

• (data) export BloscCompression and EncodedArray from ncore.data.stores (bb30302) - @janickm
• (data) open hierarchies from consolidated metadata if stored (f224965) - @janickm
• (data) create arrays from chunks encoded on other threads (8fc4f14) - @janickm
• (data) create fill-valued arrays and write selections of them (d4114e8) - @janickm
• (data) read arrays into preallocated output buffers (9dc561a) - @janickm
• BREAKING (sensors) derive the OpenCV pinhole projection validity from its radial distortion (ffad25e) - @janickm

🔄 Changed

• BREAKING (data) move is_consolidated_metadata_key to Store (5175459) - @janickm

Changes since v20.0.0-beta.4: v20.0.0-beta.4...v20.0.0-beta.5
Full diff: v19.8.0...v20.0.0-beta.5