Breaking Changes
Reader, Writer, and Encoder are now concrete types, not interfaces.
// Before
var r seekable.Reader
r, err = seekable.NewReader(rs, dec)
// After
var r *seekable.Reader
r, err = seekable.NewReader(rs, dec)NewWriter now returns *Writer; ConcurrentWriter was removed. Use *Writer directly for Write, WriteMany, and Close.
// Before
var w seekable.ConcurrentWriter
w, err = seekable.NewWriter(dst, enc)
// After
var w *seekable.Writer
w, err = seekable.NewWriter(dst, enc)Decoder and NewDecoder were removed. Use NewSeekTable for byte-oriented seek-table inspection.
// Before
d, err := seekable.NewDecoder(seekTableFrame, dec)
entry := d.GetIndexByDecompOffset(off)
size := d.Size()
// After
table, err := seekable.NewSeekTable(seekTableFrame)
entry, ok := table.EntryByDecompressedOffset(off)
size := table.Size()Reader seek-table inspection moved from Decoder-style methods to Reader.SeekTable.
// Before
entry := d.GetIndexByID(id)
n := d.NumFrames()
// After
table, err := r.SeekTable()
entry, ok := table.EntryByID(id)
n := table.NumFrames()The env subpackage was removed. Environment interfaces and frame metadata now live in the main package.
// Before
import "github.com/SaveTheRbtz/zstd-seekable-format-go/pkg/env"
func (e customEnv) GetFrameByIndex(entry env.FrameOffsetEntry) ([]byte, error)
// After
func (e customEnv) GetFrameByIndex(entry seekable.FrameOffsetEntry) ([]byte, error)FrameOffsetEntry field names changed to full names.
// Before
entry.CompOffset
entry.DecompOffset
entry.CompSize
entry.DecompSize
// After
entry.CompressedOffset
entry.DecompressedOffset
entry.CompressedSize
entry.DecompressedSizeReader and writer option names were expanded.
// Before
seekable.WithRLogger(logger)
seekable.WithREnvironment(env)
seekable.WithWLogger(logger)
seekable.WithWEnvironment(env)
// After
seekable.WithReaderLogger(logger)
seekable.WithReaderEnvironment(env)
seekable.WithWriterLogger(logger)
seekable.WithWriterEnvironment(env)Option types were renamed.
// Before
func custom(o *seekable.rOption) {} // or seekable.wOption usage
// After
var ro seekable.ReaderOption
var wo seekable.WriterOptionWithWriteCallback now receives full frame metadata instead of only compressed size.
// Before
seekable.WithWriteCallback(func(size uint32) {
written += uint64(size)
})
// After
seekable.WithWriteCallback(func(entry seekable.FrameOffsetEntry) {
written += uint64(entry.CompressedSize)
})NewReader, NewWriter, and NewEncoder now use the renamed option types and return concrete pointers.
// Before
r, err := seekable.NewReader(rs, dec, seekable.WithRLogger(logger))
e, err := seekable.NewEncoder(enc, seekable.WithWLogger(logger))
// After
r, err := seekable.NewReader(rs, dec, seekable.WithReaderLogger(logger))
e, err := seekable.NewEncoder(enc, seekable.WithWriterLogger(logger))Performance
Benchmarks were run on linux/amd64 with go1.26.4-X:nodwarf5 on an Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz:
go test -run '^$' -bench . -benchmem -benchtime=500ms -count=3 -timeout=30m .Values below are medians across 3 runs. v0.8.3 is the baseline. The seek-table and reader-cache benchmarks were copied/adapted into the v0.8.3 and v0.9.0 snapshots only for this experiment; older reader-cache rows use the pre-v0.10 built-in single-frame cache.
| Area | v0.10.0 vs v0.8.3 |
|---|---|
| Writer | 1.31x geomean faster over 10 rows |
| Seek table build | 14.49x geomean faster over 3 rows |
| Decompressed-offset lookup | 2.04x geomean faster over 18 rows |
| Frame-ID lookup | 2,499x geomean faster over 21 rows |
| Reader frame cache | Uniform: 1.77x (FIFO_MaxFrames=10000) Zipf_s=1.2_v=1: 3.87x (SieveK16_MaxFrames=10000) Gaussian_mu=mid_sigma=5600: 2.11x (SieveK16_MaxFrames=10000) |
Writer
| Benchmark | v0.8.3 baseline | v0.9.0 | v0.10.0 |
|---|---|---|---|
| Write 128 B | 1.65 us, 77.4 MB/s, 518 B, 4 allocs | 1.57 us, 81.4 MB/s, 517 B, 4 allocs (1.05x vs baseline) | 1.36 us, 94.2 MB/s, 457 B, 2 allocs (1.22x vs baseline) |
| WriteMany 128 B | 4.63 us, 27.6 MB/s, 796 B, 6 allocs | 4.24 us, 30.2 MB/s, 632 B, 6 allocs (1.09x vs baseline) | 4.17 us, 30.7 MB/s, 632 B, 6 allocs (1.11x vs baseline) |
| Write 4096 B | 8.26 us, 496.0 MB/s, 9.4 KiB, 4 allocs | 8.28 us, 494.5 MB/s, 9.4 KiB, 4 allocs (1.00x vs baseline) | 7.09 us, 577.3 MB/s, 9.3 KiB, 2 allocs (1.16x vs baseline) |
| WriteMany 4096 B | 7.51 us, 545.4 MB/s, 9.6 KiB, 6 allocs | 7.13 us, 574.8 MB/s, 9.6 KiB, 6 allocs (1.05x vs baseline) | 6.06 us, 675.6 MB/s, 9.6 KiB, 6 allocs (1.24x vs baseline) |
| Write 16384 B | 22.14 us, 739.9 MB/s, 37.4 KiB, 4 allocs | 20.46 us, 800.9 MB/s, 37.4 KiB, 4 allocs (1.08x vs baseline) | 18.06 us, 907.1 MB/s, 37.3 KiB, 2 allocs (1.23x vs baseline) |
| WriteMany 16384 B | 13.81 us, 1186.3 MB/s, 37.5 KiB, 6 allocs | 13.40 us, 1222.7 MB/s, 37.6 KiB, 6 allocs (1.03x vs baseline) | 8.26 us, 1984.5 MB/s, 37.6 KiB, 6 allocs (1.67x vs baseline) |
| Write 65536 B | 70.61 us, 928.1 MB/s, 152.1 KiB, 4 allocs | 64.37 us, 1018.1 MB/s, 152.1 KiB, 4 allocs (1.10x vs baseline) | 57.24 us, 1145.0 MB/s, 152.0 KiB, 2 allocs (1.23x vs baseline) |
| WriteMany 65536 B | 38.44 us, 1705.0 MB/s, 152.3 KiB, 6 allocs | 37.95 us, 1726.7 MB/s, 152.3 KiB, 6 allocs (1.01x vs baseline) | 20.84 us, 3144.2 MB/s, 152.3 KiB, 6 allocs (1.84x vs baseline) |
| Write 1048576 B | 1.86 ms, 564.1 MB/s, 4.92 MiB, 14 allocs | 2.05 ms, 510.4 MB/s, 4.92 MiB, 14 allocs (0.90x vs baseline) | 1.65 ms, 635.2 MB/s, 4.92 MiB, 12 allocs (1.13x vs baseline) |
| WriteMany 1048576 B | 1.21 ms, 865.0 MB/s, 4.92 MiB, 16 allocs | 1.04 ms, 1010.7 MB/s, 4.92 MiB, 16 allocs (1.17x vs baseline) | 836.17 us, 1254.0 MB/s, 4.92 MiB, 16 allocs (1.45x vs baseline) |
Seek Table Build
| Frames | v0.8.3 baseline | v0.9.0 | v0.10.0 |
|---|---|---|---|
| 16K | 6.83 ms, 1.37 MiB, 24,004 allocs | 6.49 ms, 1.37 MiB, 24,004 allocs (1.05x vs baseline) | 565.90 us, 512.0 KiB, 2 allocs (12.1x vs baseline) |
| 128K | 61.66 ms, 11.00 MiB, 191,940 allocs | 54.41 ms, 11.00 MiB, 191,940 allocs (1.13x vs baseline) | 4.03 ms, 4.00 MiB, 2 allocs (15.3x vs baseline) |
| 1M | 525.22 ms, 88.00 MiB, 1,535,428 allocs | 474.72 ms, 88.00 MiB, 1,535,428 allocs (1.11x vs baseline) | 31.91 ms, 32.00 MiB, 2 allocs (16.5x vs baseline) |
Seek Table Lookup By Decompressed Offset
| Frames | Case | v0.8.3 baseline | v0.9.0 | v0.10.0 |
|---|---|---|---|---|
| 16K | First | 243.30 ns, 48 B, 1 alloc | 237.30 ns, 48 B, 1 alloc (1.03x vs baseline) | 69.41 ns, 0 B, 0 allocs (3.51x vs baseline) |
| 16K | Middle | 254.80 ns, 48 B, 1 alloc | 243.80 ns, 48 B, 1 alloc (1.05x vs baseline) | 72.46 ns, 0 B, 0 allocs (3.52x vs baseline) |
| 16K | Last | 308.20 ns, 48 B, 1 alloc | 284.40 ns, 48 B, 1 alloc (1.08x vs baseline) | 77.08 ns, 0 B, 0 allocs (4.00x vs baseline) |
| 16K | MissPastEnd | 6.31 ns, 0 B, 0 allocs | 5.79 ns, 0 B, 0 allocs (1.09x vs baseline) | 23.10 ns, 0 B, 0 allocs (0.27x vs baseline) |
| 16K | Sequential | 345.50 ns, 48 B, 1 alloc | 326.40 ns, 48 B, 1 alloc (1.06x vs baseline) | 158.70 ns, 0 B, 0 allocs (2.18x vs baseline) |
| 16K | PseudoRandom | 778.70 ns, 48 B, 1 alloc | 698.70 ns, 48 B, 1 alloc (1.11x vs baseline) | 311.90 ns, 0 B, 0 allocs (2.50x vs baseline) |
| 128K | First | 281.30 ns, 48 B, 1 alloc | 271.80 ns, 48 B, 1 alloc (1.03x vs baseline) | 77.17 ns, 0 B, 0 allocs (3.65x vs baseline) |
| 128K | Middle | 293.70 ns, 48 B, 1 alloc | 278.40 ns, 48 B, 1 alloc (1.05x vs baseline) | 83.81 ns, 0 B, 0 allocs (3.50x vs baseline) |
| 128K | Last | 339.00 ns, 48 B, 1 alloc | 326.70 ns, 48 B, 1 alloc (1.04x vs baseline) | 82.07 ns, 0 B, 0 allocs (4.13x vs baseline) |
| 128K | MissPastEnd | 5.97 ns, 0 B, 0 allocs | 5.84 ns, 0 B, 0 allocs (1.02x vs baseline) | 22.92 ns, 0 B, 0 allocs (0.26x vs baseline) |
| 128K | Sequential | 388.00 ns, 48 B, 1 alloc | 373.10 ns, 48 B, 1 alloc (1.04x vs baseline) | 171.70 ns, 0 B, 0 allocs (2.26x vs baseline) |
| 128K | PseudoRandom | 1.29 us, 48 B, 1 alloc | 1.18 us, 48 B, 1 alloc (1.10x vs baseline) | 521.20 ns, 0 B, 0 allocs (2.48x vs baseline) |
| 1M | First | 338.60 ns, 48 B, 1 alloc | 326.60 ns, 48 B, 1 alloc (1.04x vs baseline) | 83.42 ns, 0 B, 0 allocs (4.06x vs baseline) |
| 1M | Middle | 299.70 ns, 48 B, 1 alloc | 298.50 ns, 48 B, 1 alloc (1.00x vs baseline) | 92.21 ns, 0 B, 0 allocs (3.25x vs baseline) |
| 1M | Last | 387.80 ns, 48 B, 1 alloc | 383.60 ns, 48 B, 1 alloc (1.01x vs baseline) | 90.99 ns, 0 B, 0 allocs (4.26x vs baseline) |
| 1M | MissPastEnd | 5.98 ns, 0 B, 0 allocs | 5.80 ns, 0 B, 0 allocs (1.03x vs baseline) | 22.80 ns, 0 B, 0 allocs (0.26x vs baseline) |
| 1M | Sequential | 416.00 ns, 48 B, 1 alloc | 433.50 ns, 48 B, 1 alloc (0.96x vs baseline) | 177.40 ns, 0 B, 0 allocs (2.34x vs baseline) |
| 1M | PseudoRandom | 1.98 us, 48 B, 1 alloc | 1.88 us, 48 B, 1 alloc (1.05x vs baseline) | 1.03 us, 0 B, 0 allocs (1.92x vs baseline) |
Seek Table Lookup By Frame ID
| Frames | Case | v0.8.3 baseline | v0.9.0 | v0.10.0 |
|---|---|---|---|---|
| 16K | First | 179.03 us, 0 B, 0 allocs | 188.24 us, 0 B, 0 allocs (0.95x vs baseline) | 30.11 ns, 0 B, 0 allocs (5,946x vs baseline) |
| 16K | Middle | 88.95 us, 0 B, 0 allocs | 90.15 us, 0 B, 0 allocs (0.99x vs baseline) | 30.37 ns, 0 B, 0 allocs (2,929x vs baseline) |
| 16K | Last | 61.12 ns, 0 B, 0 allocs | 61.21 ns, 0 B, 0 allocs (1.00x vs baseline) | 30.17 ns, 0 B, 0 allocs (2.03x vs baseline) |
| 16K | MissNegative | 5.55 ns, 0 B, 0 allocs | 5.51 ns, 0 B, 0 allocs (1.01x vs baseline) | 12.87 ns, 0 B, 0 allocs (0.43x vs baseline) |
| 16K | MissPastEnd | 180.56 us, 0 B, 0 allocs | 190.34 us, 0 B, 0 allocs (0.95x vs baseline) | 12.97 ns, 0 B, 0 allocs (13,922x vs baseline) |
| 16K | Sequential | 163.46 us, 0 B, 0 allocs | 169.42 us, 0 B, 0 allocs (0.96x vs baseline) | 30.96 ns, 0 B, 0 allocs (5,280x vs baseline) |
| 16K | PseudoRandom | 89.25 us, 0 B, 0 allocs | 92.10 us, 0 B, 0 allocs (0.97x vs baseline) | 32.14 ns, 0 B, 0 allocs (2,777x vs baseline) |
| 128K | First | 2.25 ms, 0 B, 0 allocs | 2.30 ms, 0 B, 0 allocs (0.98x vs baseline) | 30.01 ns, 0 B, 0 allocs (74,890x vs baseline) |
| 128K | Middle | 1.03 ms, 0 B, 0 allocs | 946.46 us, 0 B, 0 allocs (1.08x vs baseline) | 30.01 ns, 0 B, 0 allocs (34,202x vs baseline) |
| 128K | Last | 73.40 ns, 0 B, 0 allocs | 71.84 ns, 0 B, 0 allocs (1.02x vs baseline) | 30.07 ns, 0 B, 0 allocs (2.44x vs baseline) |
| 128K | MissNegative | 5.56 ns, 0 B, 0 allocs | 5.55 ns, 0 B, 0 allocs (1.00x vs baseline) | 12.76 ns, 0 B, 0 allocs (0.44x vs baseline) |
| 128K | MissPastEnd | 2.26 ms, 0 B, 0 allocs | 2.31 ms, 0 B, 0 allocs (0.98x vs baseline) | 12.79 ns, 0 B, 0 allocs (176,952x vs baseline) |
| 128K | Sequential | 2.29 ms, 0 B, 0 allocs | 2.28 ms, 0 B, 0 allocs (1.00x vs baseline) | 30.89 ns, 0 B, 0 allocs (74,196x vs baseline) |
| 128K | PseudoRandom | 1.05 ms, 0 B, 0 allocs | 1.02 ms, 0 B, 0 allocs (1.03x vs baseline) | 36.39 ns, 0 B, 0 allocs (28,963x vs baseline) |
| 1M | First | 21.57 ms, 0 B, 0 allocs | 20.20 ms, 0 B, 0 allocs (1.07x vs baseline) | 30.19 ns, 0 B, 0 allocs (714,405x vs baseline) |
| 1M | Middle | 10.54 ms, 0 B, 0 allocs | 11.90 ms, 0 B, 0 allocs (0.89x vs baseline) | 30.22 ns, 0 B, 0 allocs (348,769x vs baseline) |
| 1M | Last | 88.65 ns, 0 B, 0 allocs | 85.68 ns, 0 B, 0 allocs (1.03x vs baseline) | 30.02 ns, 0 B, 0 allocs (2.95x vs baseline) |
| 1M | MissNegative | 5.70 ns, 0 B, 0 allocs | 5.57 ns, 0 B, 0 allocs (1.02x vs baseline) | 12.85 ns, 0 B, 0 allocs (0.44x vs baseline) |
| 1M | MissPastEnd | 20.02 ms, 0 B, 0 allocs | 20.04 ms, 0 B, 0 allocs (1.00x vs baseline) | 12.82 ns, 0 B, 0 allocs (1,561,576x vs baseline) |
| 1M | Sequential | 19.83 ms, 0 B, 0 allocs | 20.29 ms, 0 B, 0 allocs (0.98x vs baseline) | 31.07 ns, 0 B, 0 allocs (638,171x vs baseline) |
| 1M | PseudoRandom | 10.19 ms, 0 B, 0 allocs | 10.38 ms, 0 B, 0 allocs (0.98x vs baseline) | 47.23 ns, 0 B, 0 allocs (215,833x vs baseline) |
Reader Frame Cache
| Distribution | Cache | Version | Median result | vs v0.8.3 SingleFrameBuiltIn |
|---|---|---|---|---|
| Uniform | SingleFrameBuiltIn | v0.8.3 | 3.74 us, 0.00% hit, 388 B, 9 allocs | baseline |
| Uniform | SingleFrameBuiltIn | v0.9.0 | 3.77 us, 0.00% hit, 388 B, 9 allocs | 0.99x |
| Uniform | FIFO_MaxFrames=10000 | v0.10.0 | 2.11 us, 3.88% hit, 129 B, 3 allocs | 1.77x |
| Uniform | LRU_MaxFrames=10000 | v0.10.0 | 2.12 us, 3.87% hit, 129 B, 3 allocs | 1.77x |
| Uniform | SieveK16_MaxFrames=10000 | v0.10.0 | 2.14 us, 3.88% hit, 144 B, 3 allocs | 1.75x |
| Zipf_s=1.2_v=1 | SingleFrameBuiltIn | v0.8.3 | 2.07 us, 5.21% hit, 390 B, 8 allocs | baseline |
| Zipf_s=1.2_v=1 | SingleFrameBuiltIn | v0.9.0 | 2.19 us, 5.21% hit, 390 B, 8 allocs | 0.94x |
| Zipf_s=1.2_v=1 | FIFO_MaxFrames=10000 | v0.10.0 | 561.90 ns, 87.63% hit, 16 B, 0 allocs | 3.68x |
| Zipf_s=1.2_v=1 | LRU_MaxFrames=10000 | v0.10.0 | 565.00 ns, 89.41% hit, 14 B, 0 allocs | 3.66x |
| Zipf_s=1.2_v=1 | SieveK16_MaxFrames=10000 | v0.10.0 | 533.90 ns, 90.78% hit, 14 B, 0 allocs | 3.87x |
| Gaussian_mu=mid_sigma=5600 | SingleFrameBuiltIn | v0.8.3 | 2.62 us, 0.01% hit, 388 B, 8 allocs | baseline |
| Gaussian_mu=mid_sigma=5600 | SingleFrameBuiltIn | v0.9.0 | 2.59 us, 0.01% hit, 388 B, 8 allocs | 1.01x |
| Gaussian_mu=mid_sigma=5600 | FIFO_MaxFrames=10000 | v0.10.0 | 1.34 us, 45.20% hit, 73 B, 2 allocs | 1.95x |
| Gaussian_mu=mid_sigma=5600 | LRU_MaxFrames=10000 | v0.10.0 | 1.35 us, 47.20% hit, 71 B, 2 allocs | 1.95x |
| Gaussian_mu=mid_sigma=5600 | SieveK16_MaxFrames=10000 | v0.10.0 | 1.24 us, 53.17% hit, 70 B, 1 alloc | 2.11x |
What's Changed
- cmd/zstdseek: bump pkg to v0.9.0 by @SaveTheRbtz in #237
- go: enable race detector in tests by @SaveTheRbtz in #239
- pkg: validate seek table entry count by @SaveTheRbtz in #240
- pkg: support zero-sized frames and improve perf by @SaveTheRbtz in #238
- pkg: handle short ReaderAt reads by @SaveTheRbtz in #241
- pkg: replace Decoder with SeekTable by @SaveTheRbtz in #242
- tests: simplify pkg coverage by @SaveTheRbtz in #244
- pkg: define reader post-close errors by @SaveTheRbtz in #245
- pkg: release writer index on close by @SaveTheRbtz in #246
- ci: add typos check by @SaveTheRbtz in #247
- go: remove toolchain directives by @SaveTheRbtz in #248
- ci: scan workflows with typos by @SaveTheRbtz in #249
- pkg: simplify checksum casts by @SaveTheRbtz in #250
- pkg: improve godoc comments by @SaveTheRbtz in #251
- pkg: export SeekTable type by @SaveTheRbtz in #252
- pkg: improve godoc comments by @SaveTheRbtz in #253
- writer: stop after failed frame writes by @SaveTheRbtz in #254
- writer: return WriteMany cancellation errors by @SaveTheRbtz in #243
- reader: match writer frame size limit by @SaveTheRbtz in #255
- ci: harden workflow token permissions by @SaveTheRbtz in #256
- ci: pin workflow actions by @SaveTheRbtz in #257
- ci: publish OpenSSF Scorecard results by @SaveTheRbtz in #258
- pkg: standardize lifecycle finalization by @SaveTheRbtz in #259
- pkg: rename public option API by @SaveTheRbtz in #261
- cmd: bump package dependency by @SaveTheRbtz in #262
- cmd: replace zap with slog by @SaveTheRbtz in #263
- pkg: return concrete types by @SaveTheRbtz in #264
- pkg: expose reader seek table by @SaveTheRbtz in #265
- readme: streamline package overview by @SaveTheRbtz in #267
- docs: polish package godoc by @SaveTheRbtz in #266
- pkg: expand writemany write callback by @SaveTheRbtz in #268
- pkg: preserve reader offset on eof by @SaveTheRbtz in #269
- pkg: short-circuit empty reader reads by @SaveTheRbtz in #270
- pkg: stop writemany after cancellation by @SaveTheRbtz in #271
- pkg: bound footer parse errors by @SaveTheRbtz in #273
- cmd: bump zstdseek pkg dependency by @SaveTheRbtz in #272
- pkg: add pluggable frame cache by @SaveTheRbtz in #274
- cmd: bump zstdseek pkg dependency by @SaveTheRbtz in #275
- cmd: bump pkg to v0.10.0-rc.1 by @SaveTheRbtz in #276
- pkg: avoid disabled debug log allocations by @SaveTheRbtz in #277
- pkg: compact seek table index entries by @SaveTheRbtz in #281
- cmd: bump pkg to v0.10.0-rc.2 by @SaveTheRbtz in #282
Full Changelog: pkg/v0.9.0...pkg/v0.10.0