iskorotkov/avro v2.34.0

on GitHub

Release notes — v2.34.0

Highlights

• Nanosecond-precision timestamps — Avro 1.12 timestamp-nanos and local-timestamp-nanos across encode, decode, the generic reader, the type resolver, schema parsing, and code generation.
• Float and double decoded per spec on big-endian hosts — Reader.ReadFloat/ReadDouble reinterpreted wire bytes in host byte order, so every float and double decoded byte-swapped on s390x. Even a same-host write-read round trip produced garbage.
• OCF zip-bomb hardening — new caps on declared block size and post-decompression size, bounded block reads, and a fix for a data race on shared zstd codecs.
• soe.Codec.Encode returns caller-owned buffers — two in-flight single-object encodings from one codec could alias, so the first silently became the second.

Features

• feat: add nanosecond-precision timestamp logical types (#1) — adds timestamp-nanos and local-timestamp-nanos with int64 overflow guards and exact error bounds. Also fixes local-timestamp decoding to reinterpret wall-clock components in the local zone instead of subtracting the zone offset, correcting values around DST transitions, and aligns the generic reader with the codec path.
• feat(ocf): add WithMaxBlockBytes and WithMaxDecompressedBlockBytes (#17, #28) — bound the declared compressed size and the post-decompression size of a data block. Both default to disabled, so existing readers are unaffected until opted in.
• feat: reject empty union schemas (#17) — NewUnionSchema now errors on a union with no members, per the Avro spec.
• feat(soe): add AppendEncode (#37) — Codec.AppendEncode and TypedCodec.AppendEncode take a caller-supplied destination buffer, matching the shape twmb/avro and linkedin/goavro use for single-object encoding.

Fixes

• fix: decode float and double as little-endian per Avro spec (#31) — Reader.ReadFloat/ReadDouble reinterpreted wire bytes via unsafe.Pointer in host byte order while the spec mandates little-endian and the writer already encoded little-endian. Decode now lives in build-tagged files: known little-endian architectures keep the verbatim unsafe fast path, while big-endian and unknown architectures use explicit binary.LittleEndian, so new ports are correct by default. A test-bigendian CI job runs the suite under GOARCH=s390x.
• fix(soe): return an owned buffer from Codec.Encode (#37) — Encode did append(c.header, data...) on a header shared by every call. The 10-byte header rounds up to a 16-byte size class, so a payload fitting the 6 spare bytes did not reallocate and shared its backing array with the previous result. The same defect made a Codec unsafe to share between goroutines. BuildHeaderForFingerprint now also returns an exactly-sized header, closing the same aliasing for callers that frame payloads from an exported header.
• fix(ocf): cap decompressed block size and harden shared zstd codecs (#28) — closes the zip-bomb amplification vector: io.LimitReader plus post-check for deflate, varint DecodedLen rejection for snappy, and zstd.WithDecoderMaxMemory for owned decoders. readBlock now reads the compressed payload through a bounded chunked buffer instead of make([]byte, attacker-size), validates the sync marker before decompressing, and rejects negative record counts. Also removes a per-call Reset(nil) that raced when a zstd decoder or encoder was shared across goroutines.

Performance

Six hot-path optimisations land in this release. Allocation behaviour is unchanged: B/op and allocs/op are identical between v2.33.1 and this release across all 42 shared sub-benchmarks.

• perf(codec): specialize scalar array encoder (#24) — scalarArrayEncoder[T] mirrors the existing scalar decoder for arrays of primitives, skipping per-element ValEncoder dispatch and UnsafeGetIndex. Measured on Apple M4 Pro against the generic array encoder: geomean −42.7% ns/op, +65.8% B/s, allocations unchanged.
• perf(reader): batched scalar int/long array decode (#25) — replaces the per-element ReadInt/ReadLong loop with a batched drain of the read buffer: inline 1-byte peek for the dominant small-value case, 2x unroll for instruction-level parallelism, and a continueVarint fallback for wider values.
• perf(writer): inline fast paths + binary.AppendUvarint in encodeInt (#19) — replaces the per-byte append loop with 1-byte and 2-byte inline fast paths plus a stdlib binary.AppendUvarint slow path. The 2-byte path covers the common range for Avro IDs, array lengths, and small zigzagged values.
• perf(encode): TextAppender fast-path with Writer scratch buffer (#21) — detects encoding.TextAppender (Go 1.24+) and appends into a reusable scratch buffer instead of relying on each MarshalText call to allocate. Maps keyed by time.Time, net.IP, netip.Addr, *big.Int and friends drop from O(N) to O(1) allocations per Marshal.
• perf(union): skip defer + typeConverters dispatch when none registered (#22) — unionNullableDecoder.Decode set up a defer and a sync.Map lookup on every call even with no custom converters registered. Now gated behind a monotonic atomic.Bool on TypeConverters.
• perf(record): cache isPointer in struct field chain (#18) — embedded struct fields walked a chain resolving a reflect2.Type interface per non-terminal step. The pointer check is now computed once in describeStruct.

Security and CI

• doc: add security guidelines (#14, @klajok) — adds SECURITY.md, documenting the OCF caps, per-codec coverage, the shared-decoder OOM caveat, and the unbounded OCF header metadata map.
• test(fuzz): add fuzz tests (#17) — fuzz targets for schema parsing, the decoder, and the OCF reader, run in CI on amd64 and 386.
• ci: harden workflows (#20) — actions pinned to commit SHAs with explicit permissions.
• ci: add benchmark regression workflow (#15).

Notes for users

• Additive API only: Codec.AppendEncode, TypedCodec.AppendEncode, ocf.WithMaxBlockBytes, ocf.WithMaxDecompressedBlockBytes, TypeConverters.HasAny. Nothing was removed or re-typed, and ocf.SnappyCodec stays source-compatible despite gaining an unexported field.
• Behaviour change: NewUnionSchema now rejects a union with no members, and schemas containing [] fail to parse. The Avro spec requires at least one member, so such schemas were always invalid, but code that constructed them will now see an error.
• Correctness: anyone running on a big-endian host (s390x) should upgrade — float and double decoding was wrong before this release.
• soe users get the Encode aliasing fix with no code change. Encode now allocates one buffer per call, where the previous behaviour reused the codec's array. Hot paths that want the old allocation count should move to AppendEncode with a reused buffer.

Full changelog: v2.33.1...v2.34.0