Gravity v1.6.2
Release: v1.6.2
Full Changelog: v1.6.1...v1.6.2
Related pinned components:
- gravity-reth: rev
acc45884(fix(rocksdb): flush batch data into storage to make sure stage is completed, gravity-reth#340)
A short two-PR patch release. Both items target single, narrowly-scoped reliability issues observed in production.
Highlights
- RocksDB durability fix for low-traffic state-trie databases: state-trie
commit()previously returned without anfdatasync(), soaccount_db/storage_dbcould go 2,000+ blocks without syncing — a power-failure window that would cause large trie data loss and a state-root mismatch panic during consensus re-execution on restart. - Sentinel 100% CPU busy-spin fixed. The
linemux0.3.0 upstream bug (jmagnuson/linemux#57) silently spun six sentinel processes at 100% CPU for 21 days on production with all log-based alerting disabled. Switches to a Galxe fork with a 6-line patch.
What's Changed
Bug Fixes
- fix(sentinel): switch to patched linemux fork to stop 100% CPU busy spin by @nekomoto911 in #728
- Sentinel's
linemuxdep is bumped to a Galxe fork (Galxe/linemux@0194e022) that fixes the upstream busy-spin whenwatched_filesbecomes empty. The patch is six lines (unwrap_or_continue!chain → explicitmatch); filed upstream as jmagnuson/linemux#76 (issue) + jmagnuson/linemux#77 (PR), but we don't gate prod recovery on upstream review cadence. - Defense in depth:
Reader::next_linenow handlesOk(None)withexit(2)so systemd restarts the process if the patched linemux ever does report an exhausted stream. Ships a systemd unit template atbin/sentinel/systemd/sentinel.service(Restart=always) so the exit→restart path is real. - Regression test in the fork (
regression_no_busy_spin_on_drained_watched_files) constructs the exact production state (Inner::readersnon-empty +MuxedEvents::watched_filesempty +StreamState::Events) and assertsnext_line()resolves toOk(None)within 500 ms. Reverting the patch causes the test to hang (busy-spin), confirming the test catches a regression.
- Sentinel's
Dependencies
- chore(deps): bump gravity-reth to acc45884 (rocksdb durability fix) by @nekomoto911 in #730
- Bumps
gravity-reth41c0b709 → acc45884. Range is exactly one commit ahead (ahead 1, behind 0); this bump introduces gravity-reth#340 and nothing else.
- Bumps
gravity-reth changes since v1.6.1
The greth pin moved from 41c0b709 to acc45884 — exactly one commit:
#340 fix(rocksdb): flush batch data into storage to make sure stage is completedby @AshinGauprovider_rw.commit()calledrocksdb::DB::write()withoutfdatasync(). Withwal_bytes_per_sync=4MB, the low-trafficaccount_db/storage_dbcould go 2,000+ blocks without syncing, risking large trie data loss on power failure and a state-root mismatch panic during consensus re-execution after a crash.- The fix:
- All
write()calls incommit_view()usewrite_opt(sync=true)— everycommit()isfdatasync-durable before returning. - Disable pipelined writes and drop
wal_bytes_per_sync(redundant with per-commit sync). - Recovery always re-checks every stage checkpoint (no early return), repairing only stages that are behind.
- All