github SaveTheRbtz/zstd-seekable-format-go pkg/v0.10.0

4 hours ago

Breaking Changes

Reader, Writer, and Encoder are now concrete types, not interfaces.

// Before
var r seekable.Reader
r, err = seekable.NewReader(rs, dec)

// After
var r *seekable.Reader
r, err = seekable.NewReader(rs, dec)

NewWriter now returns *Writer; ConcurrentWriter was removed. Use *Writer directly for Write, WriteMany, and Close.

// Before
var w seekable.ConcurrentWriter
w, err = seekable.NewWriter(dst, enc)

// After
var w *seekable.Writer
w, err = seekable.NewWriter(dst, enc)

Decoder and NewDecoder were removed. Use NewSeekTable for byte-oriented seek-table inspection.

// Before
d, err := seekable.NewDecoder(seekTableFrame, dec)
entry := d.GetIndexByDecompOffset(off)
size := d.Size()

// After
table, err := seekable.NewSeekTable(seekTableFrame)
entry, ok := table.EntryByDecompressedOffset(off)
size := table.Size()

Reader seek-table inspection moved from Decoder-style methods to Reader.SeekTable.

// Before
entry := d.GetIndexByID(id)
n := d.NumFrames()

// After
table, err := r.SeekTable()
entry, ok := table.EntryByID(id)
n := table.NumFrames()

The env subpackage was removed. Environment interfaces and frame metadata now live in the main package.

// Before
import "github.com/SaveTheRbtz/zstd-seekable-format-go/pkg/env"

func (e customEnv) GetFrameByIndex(entry env.FrameOffsetEntry) ([]byte, error)

// After
func (e customEnv) GetFrameByIndex(entry seekable.FrameOffsetEntry) ([]byte, error)

FrameOffsetEntry field names changed to full names.

// Before
entry.CompOffset
entry.DecompOffset
entry.CompSize
entry.DecompSize

// After
entry.CompressedOffset
entry.DecompressedOffset
entry.CompressedSize
entry.DecompressedSize

Reader and writer option names were expanded.

// Before
seekable.WithRLogger(logger)
seekable.WithREnvironment(env)
seekable.WithWLogger(logger)
seekable.WithWEnvironment(env)

// After
seekable.WithReaderLogger(logger)
seekable.WithReaderEnvironment(env)
seekable.WithWriterLogger(logger)
seekable.WithWriterEnvironment(env)

Option types were renamed.

// Before
func custom(o *seekable.rOption) {} // or seekable.wOption usage

// After
var ro seekable.ReaderOption
var wo seekable.WriterOption

WithWriteCallback now receives full frame metadata instead of only compressed size.

// Before
seekable.WithWriteCallback(func(size uint32) {
    written += uint64(size)
})

// After
seekable.WithWriteCallback(func(entry seekable.FrameOffsetEntry) {
    written += uint64(entry.CompressedSize)
})

NewReader, NewWriter, and NewEncoder now use the renamed option types and return concrete pointers.

// Before
r, err := seekable.NewReader(rs, dec, seekable.WithRLogger(logger))
e, err := seekable.NewEncoder(enc, seekable.WithWLogger(logger))

// After
r, err := seekable.NewReader(rs, dec, seekable.WithReaderLogger(logger))
e, err := seekable.NewEncoder(enc, seekable.WithWriterLogger(logger))

Performance

Benchmarks were run on linux/amd64 with go1.26.4-X:nodwarf5 on an Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz:

go test -run '^$' -bench . -benchmem -benchtime=500ms -count=3 -timeout=30m .

Values below are medians across 3 runs. v0.8.3 is the baseline. The seek-table and reader-cache benchmarks were copied/adapted into the v0.8.3 and v0.9.0 snapshots only for this experiment; older reader-cache rows use the pre-v0.10 built-in single-frame cache.

Area v0.10.0 vs v0.8.3
Writer 1.31x geomean faster over 10 rows
Seek table build 14.49x geomean faster over 3 rows
Decompressed-offset lookup 2.04x geomean faster over 18 rows
Frame-ID lookup 2,499x geomean faster over 21 rows
Reader frame cache Uniform: 1.77x (FIFO_MaxFrames=10000)
Zipf_s=1.2_v=1: 3.87x (SieveK16_MaxFrames=10000)
Gaussian_mu=mid_sigma=5600: 2.11x (SieveK16_MaxFrames=10000)

Writer

Benchmark v0.8.3 baseline v0.9.0 v0.10.0
Write 128 B 1.65 us, 77.4 MB/s, 518 B, 4 allocs 1.57 us, 81.4 MB/s, 517 B, 4 allocs (1.05x vs baseline) 1.36 us, 94.2 MB/s, 457 B, 2 allocs (1.22x vs baseline)
WriteMany 128 B 4.63 us, 27.6 MB/s, 796 B, 6 allocs 4.24 us, 30.2 MB/s, 632 B, 6 allocs (1.09x vs baseline) 4.17 us, 30.7 MB/s, 632 B, 6 allocs (1.11x vs baseline)
Write 4096 B 8.26 us, 496.0 MB/s, 9.4 KiB, 4 allocs 8.28 us, 494.5 MB/s, 9.4 KiB, 4 allocs (1.00x vs baseline) 7.09 us, 577.3 MB/s, 9.3 KiB, 2 allocs (1.16x vs baseline)
WriteMany 4096 B 7.51 us, 545.4 MB/s, 9.6 KiB, 6 allocs 7.13 us, 574.8 MB/s, 9.6 KiB, 6 allocs (1.05x vs baseline) 6.06 us, 675.6 MB/s, 9.6 KiB, 6 allocs (1.24x vs baseline)
Write 16384 B 22.14 us, 739.9 MB/s, 37.4 KiB, 4 allocs 20.46 us, 800.9 MB/s, 37.4 KiB, 4 allocs (1.08x vs baseline) 18.06 us, 907.1 MB/s, 37.3 KiB, 2 allocs (1.23x vs baseline)
WriteMany 16384 B 13.81 us, 1186.3 MB/s, 37.5 KiB, 6 allocs 13.40 us, 1222.7 MB/s, 37.6 KiB, 6 allocs (1.03x vs baseline) 8.26 us, 1984.5 MB/s, 37.6 KiB, 6 allocs (1.67x vs baseline)
Write 65536 B 70.61 us, 928.1 MB/s, 152.1 KiB, 4 allocs 64.37 us, 1018.1 MB/s, 152.1 KiB, 4 allocs (1.10x vs baseline) 57.24 us, 1145.0 MB/s, 152.0 KiB, 2 allocs (1.23x vs baseline)
WriteMany 65536 B 38.44 us, 1705.0 MB/s, 152.3 KiB, 6 allocs 37.95 us, 1726.7 MB/s, 152.3 KiB, 6 allocs (1.01x vs baseline) 20.84 us, 3144.2 MB/s, 152.3 KiB, 6 allocs (1.84x vs baseline)
Write 1048576 B 1.86 ms, 564.1 MB/s, 4.92 MiB, 14 allocs 2.05 ms, 510.4 MB/s, 4.92 MiB, 14 allocs (0.90x vs baseline) 1.65 ms, 635.2 MB/s, 4.92 MiB, 12 allocs (1.13x vs baseline)
WriteMany 1048576 B 1.21 ms, 865.0 MB/s, 4.92 MiB, 16 allocs 1.04 ms, 1010.7 MB/s, 4.92 MiB, 16 allocs (1.17x vs baseline) 836.17 us, 1254.0 MB/s, 4.92 MiB, 16 allocs (1.45x vs baseline)

Seek Table Build

Frames v0.8.3 baseline v0.9.0 v0.10.0
16K 6.83 ms, 1.37 MiB, 24,004 allocs 6.49 ms, 1.37 MiB, 24,004 allocs (1.05x vs baseline) 565.90 us, 512.0 KiB, 2 allocs (12.1x vs baseline)
128K 61.66 ms, 11.00 MiB, 191,940 allocs 54.41 ms, 11.00 MiB, 191,940 allocs (1.13x vs baseline) 4.03 ms, 4.00 MiB, 2 allocs (15.3x vs baseline)
1M 525.22 ms, 88.00 MiB, 1,535,428 allocs 474.72 ms, 88.00 MiB, 1,535,428 allocs (1.11x vs baseline) 31.91 ms, 32.00 MiB, 2 allocs (16.5x vs baseline)

Seek Table Lookup By Decompressed Offset

Frames Case v0.8.3 baseline v0.9.0 v0.10.0
16K First 243.30 ns, 48 B, 1 alloc 237.30 ns, 48 B, 1 alloc (1.03x vs baseline) 69.41 ns, 0 B, 0 allocs (3.51x vs baseline)
16K Middle 254.80 ns, 48 B, 1 alloc 243.80 ns, 48 B, 1 alloc (1.05x vs baseline) 72.46 ns, 0 B, 0 allocs (3.52x vs baseline)
16K Last 308.20 ns, 48 B, 1 alloc 284.40 ns, 48 B, 1 alloc (1.08x vs baseline) 77.08 ns, 0 B, 0 allocs (4.00x vs baseline)
16K MissPastEnd 6.31 ns, 0 B, 0 allocs 5.79 ns, 0 B, 0 allocs (1.09x vs baseline) 23.10 ns, 0 B, 0 allocs (0.27x vs baseline)
16K Sequential 345.50 ns, 48 B, 1 alloc 326.40 ns, 48 B, 1 alloc (1.06x vs baseline) 158.70 ns, 0 B, 0 allocs (2.18x vs baseline)
16K PseudoRandom 778.70 ns, 48 B, 1 alloc 698.70 ns, 48 B, 1 alloc (1.11x vs baseline) 311.90 ns, 0 B, 0 allocs (2.50x vs baseline)
128K First 281.30 ns, 48 B, 1 alloc 271.80 ns, 48 B, 1 alloc (1.03x vs baseline) 77.17 ns, 0 B, 0 allocs (3.65x vs baseline)
128K Middle 293.70 ns, 48 B, 1 alloc 278.40 ns, 48 B, 1 alloc (1.05x vs baseline) 83.81 ns, 0 B, 0 allocs (3.50x vs baseline)
128K Last 339.00 ns, 48 B, 1 alloc 326.70 ns, 48 B, 1 alloc (1.04x vs baseline) 82.07 ns, 0 B, 0 allocs (4.13x vs baseline)
128K MissPastEnd 5.97 ns, 0 B, 0 allocs 5.84 ns, 0 B, 0 allocs (1.02x vs baseline) 22.92 ns, 0 B, 0 allocs (0.26x vs baseline)
128K Sequential 388.00 ns, 48 B, 1 alloc 373.10 ns, 48 B, 1 alloc (1.04x vs baseline) 171.70 ns, 0 B, 0 allocs (2.26x vs baseline)
128K PseudoRandom 1.29 us, 48 B, 1 alloc 1.18 us, 48 B, 1 alloc (1.10x vs baseline) 521.20 ns, 0 B, 0 allocs (2.48x vs baseline)
1M First 338.60 ns, 48 B, 1 alloc 326.60 ns, 48 B, 1 alloc (1.04x vs baseline) 83.42 ns, 0 B, 0 allocs (4.06x vs baseline)
1M Middle 299.70 ns, 48 B, 1 alloc 298.50 ns, 48 B, 1 alloc (1.00x vs baseline) 92.21 ns, 0 B, 0 allocs (3.25x vs baseline)
1M Last 387.80 ns, 48 B, 1 alloc 383.60 ns, 48 B, 1 alloc (1.01x vs baseline) 90.99 ns, 0 B, 0 allocs (4.26x vs baseline)
1M MissPastEnd 5.98 ns, 0 B, 0 allocs 5.80 ns, 0 B, 0 allocs (1.03x vs baseline) 22.80 ns, 0 B, 0 allocs (0.26x vs baseline)
1M Sequential 416.00 ns, 48 B, 1 alloc 433.50 ns, 48 B, 1 alloc (0.96x vs baseline) 177.40 ns, 0 B, 0 allocs (2.34x vs baseline)
1M PseudoRandom 1.98 us, 48 B, 1 alloc 1.88 us, 48 B, 1 alloc (1.05x vs baseline) 1.03 us, 0 B, 0 allocs (1.92x vs baseline)

Seek Table Lookup By Frame ID

Frames Case v0.8.3 baseline v0.9.0 v0.10.0
16K First 179.03 us, 0 B, 0 allocs 188.24 us, 0 B, 0 allocs (0.95x vs baseline) 30.11 ns, 0 B, 0 allocs (5,946x vs baseline)
16K Middle 88.95 us, 0 B, 0 allocs 90.15 us, 0 B, 0 allocs (0.99x vs baseline) 30.37 ns, 0 B, 0 allocs (2,929x vs baseline)
16K Last 61.12 ns, 0 B, 0 allocs 61.21 ns, 0 B, 0 allocs (1.00x vs baseline) 30.17 ns, 0 B, 0 allocs (2.03x vs baseline)
16K MissNegative 5.55 ns, 0 B, 0 allocs 5.51 ns, 0 B, 0 allocs (1.01x vs baseline) 12.87 ns, 0 B, 0 allocs (0.43x vs baseline)
16K MissPastEnd 180.56 us, 0 B, 0 allocs 190.34 us, 0 B, 0 allocs (0.95x vs baseline) 12.97 ns, 0 B, 0 allocs (13,922x vs baseline)
16K Sequential 163.46 us, 0 B, 0 allocs 169.42 us, 0 B, 0 allocs (0.96x vs baseline) 30.96 ns, 0 B, 0 allocs (5,280x vs baseline)
16K PseudoRandom 89.25 us, 0 B, 0 allocs 92.10 us, 0 B, 0 allocs (0.97x vs baseline) 32.14 ns, 0 B, 0 allocs (2,777x vs baseline)
128K First 2.25 ms, 0 B, 0 allocs 2.30 ms, 0 B, 0 allocs (0.98x vs baseline) 30.01 ns, 0 B, 0 allocs (74,890x vs baseline)
128K Middle 1.03 ms, 0 B, 0 allocs 946.46 us, 0 B, 0 allocs (1.08x vs baseline) 30.01 ns, 0 B, 0 allocs (34,202x vs baseline)
128K Last 73.40 ns, 0 B, 0 allocs 71.84 ns, 0 B, 0 allocs (1.02x vs baseline) 30.07 ns, 0 B, 0 allocs (2.44x vs baseline)
128K MissNegative 5.56 ns, 0 B, 0 allocs 5.55 ns, 0 B, 0 allocs (1.00x vs baseline) 12.76 ns, 0 B, 0 allocs (0.44x vs baseline)
128K MissPastEnd 2.26 ms, 0 B, 0 allocs 2.31 ms, 0 B, 0 allocs (0.98x vs baseline) 12.79 ns, 0 B, 0 allocs (176,952x vs baseline)
128K Sequential 2.29 ms, 0 B, 0 allocs 2.28 ms, 0 B, 0 allocs (1.00x vs baseline) 30.89 ns, 0 B, 0 allocs (74,196x vs baseline)
128K PseudoRandom 1.05 ms, 0 B, 0 allocs 1.02 ms, 0 B, 0 allocs (1.03x vs baseline) 36.39 ns, 0 B, 0 allocs (28,963x vs baseline)
1M First 21.57 ms, 0 B, 0 allocs 20.20 ms, 0 B, 0 allocs (1.07x vs baseline) 30.19 ns, 0 B, 0 allocs (714,405x vs baseline)
1M Middle 10.54 ms, 0 B, 0 allocs 11.90 ms, 0 B, 0 allocs (0.89x vs baseline) 30.22 ns, 0 B, 0 allocs (348,769x vs baseline)
1M Last 88.65 ns, 0 B, 0 allocs 85.68 ns, 0 B, 0 allocs (1.03x vs baseline) 30.02 ns, 0 B, 0 allocs (2.95x vs baseline)
1M MissNegative 5.70 ns, 0 B, 0 allocs 5.57 ns, 0 B, 0 allocs (1.02x vs baseline) 12.85 ns, 0 B, 0 allocs (0.44x vs baseline)
1M MissPastEnd 20.02 ms, 0 B, 0 allocs 20.04 ms, 0 B, 0 allocs (1.00x vs baseline) 12.82 ns, 0 B, 0 allocs (1,561,576x vs baseline)
1M Sequential 19.83 ms, 0 B, 0 allocs 20.29 ms, 0 B, 0 allocs (0.98x vs baseline) 31.07 ns, 0 B, 0 allocs (638,171x vs baseline)
1M PseudoRandom 10.19 ms, 0 B, 0 allocs 10.38 ms, 0 B, 0 allocs (0.98x vs baseline) 47.23 ns, 0 B, 0 allocs (215,833x vs baseline)

Reader Frame Cache

Distribution Cache Version Median result vs v0.8.3 SingleFrameBuiltIn
Uniform SingleFrameBuiltIn v0.8.3 3.74 us, 0.00% hit, 388 B, 9 allocs baseline
Uniform SingleFrameBuiltIn v0.9.0 3.77 us, 0.00% hit, 388 B, 9 allocs 0.99x
Uniform FIFO_MaxFrames=10000 v0.10.0 2.11 us, 3.88% hit, 129 B, 3 allocs 1.77x
Uniform LRU_MaxFrames=10000 v0.10.0 2.12 us, 3.87% hit, 129 B, 3 allocs 1.77x
Uniform SieveK16_MaxFrames=10000 v0.10.0 2.14 us, 3.88% hit, 144 B, 3 allocs 1.75x
Zipf_s=1.2_v=1 SingleFrameBuiltIn v0.8.3 2.07 us, 5.21% hit, 390 B, 8 allocs baseline
Zipf_s=1.2_v=1 SingleFrameBuiltIn v0.9.0 2.19 us, 5.21% hit, 390 B, 8 allocs 0.94x
Zipf_s=1.2_v=1 FIFO_MaxFrames=10000 v0.10.0 561.90 ns, 87.63% hit, 16 B, 0 allocs 3.68x
Zipf_s=1.2_v=1 LRU_MaxFrames=10000 v0.10.0 565.00 ns, 89.41% hit, 14 B, 0 allocs 3.66x
Zipf_s=1.2_v=1 SieveK16_MaxFrames=10000 v0.10.0 533.90 ns, 90.78% hit, 14 B, 0 allocs 3.87x
Gaussian_mu=mid_sigma=5600 SingleFrameBuiltIn v0.8.3 2.62 us, 0.01% hit, 388 B, 8 allocs baseline
Gaussian_mu=mid_sigma=5600 SingleFrameBuiltIn v0.9.0 2.59 us, 0.01% hit, 388 B, 8 allocs 1.01x
Gaussian_mu=mid_sigma=5600 FIFO_MaxFrames=10000 v0.10.0 1.34 us, 45.20% hit, 73 B, 2 allocs 1.95x
Gaussian_mu=mid_sigma=5600 LRU_MaxFrames=10000 v0.10.0 1.35 us, 47.20% hit, 71 B, 2 allocs 1.95x
Gaussian_mu=mid_sigma=5600 SieveK16_MaxFrames=10000 v0.10.0 1.24 us, 53.17% hit, 70 B, 1 alloc 2.11x

What's Changed

Full Changelog: pkg/v0.9.0...pkg/v0.10.0

Don't miss a new zstd-seekable-format-go release

NewReleases is sending notifications on new releases.