github ObolNetwork/lido-charon-distributed-validator-node v0.3.16
on GitHub

45 minutes ago

Obol Logo

This is LCDVN's v0.3.16 release, a security and maintenance release for the current fork. It is not Gloas-ready: Gloas support ships with the v0.4 line and Charon v2.0.0. It gives each deployment its own random Engine API secret, hides validator key shares from the Charon container, fixes false positives in grafana/security-check.sh, and updates Charon to v1.11.4, lido-dv-exit to v0.6.2 and the client images to their latest patch releases. If you are coming from a release before v0.3.15, read the v0.3.15 release notes first, as they apply to this release too. Feedback is welcome and appreciated, please use github issues or discord if you have trouble with this release.

Update

# Save your local .env / custom overrides
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.3.16
# Re-apply your stashed changes
git stash apply
# Restart the node. This recreates the execution and consensus clients with a new Engine API secret.
docker compose up -d
# Check your node (may need sudo)
./grafana/security-check.sh

Run docker compose up -d soon after the checkout, and don't restart single containers in between: an execution or consensus client still running from before the update can't authenticate to one that has the new secret. docker compose up -d recreates both, and they reconnect with the new secret.

If you had changed jwt/jwt.hex, git stash apply reports a conflict on it (deleted by us), because this release stops tracking that file. Before docker compose up -d, choose one:

  • Keep your secret: git rm --cached jwt/jwt.hex. The file stays in place, git now ignores it, and jwt-init loads it.
  • Use a generated secret instead: git rm --cached jwt/jwt.hex && rm jwt/jwt.hex.

Changes

Engine API secret generated per deployment

Previous releases shipped the same jwt/jwt.hex to every operator, so the secret protecting the Engine API (port 8551) between the execution and consensus clients was public. By default this stack publishes port 8551 on 127.0.0.1 only, so the Engine API was reachable from the host itself and from containers on the same Docker network. If you set NETHERMIND_IP_ENGINE or EL_IP_ENGINE to a non-loopback address, anyone who could reach that port could use the public secret to send commands to your execution client. Check that setting, and if you changed it, make sure a firewall restricts the port.

  • A new one-shot jwt-init service generates a random secret on first start and stores it in a jwt Docker volume. Only the execution and consensus clients mount it, read-only. jwt/jwt.hex is no longer tracked in git.
  • Nothing to do on upgrade. The next docker compose up -d generates the secret and recreates both clients with it. It stays the same across restarts and docker compose down/up. It changes only if you remove the volume (docker compose down -v).
  • The old public secret is ignored if it is still in jwt/jwt.hex.
  • To use your own secret, for example with an execution client outside this stack, put it in jwt/jwt.hex as 64 hex characters and run docker compose up -d --force-recreate. Anything else that talks to the Engine API with the old secret needs the new one.
  • To print the secret in use: docker compose run --rm jwt-init cat /jwt/jwt.hex.

Validator key shares hidden from Charon

Charon never reads the key shares in .charon/validator_keys, so the Charon container now gets an empty, read-only mount in their place. If Charon were ever compromised, the key shares wouldn't be exposed through it. The validator clients and lido-dv-exit keep their own mounts and are unaffected.

grafana/security-check.sh false positives fixed

  • No more false reports of deleted datasources. This stack recreates its Prometheus and Loki datasources every time Grafana starts, so v0.3.15 reported deleted data_source rows on any node restarted a few times. The check no longer counts them. Deleted users or API keys are still reported.
  • Dashboards on recent Grafana versions. Grafana 12 and later store dashboards and folders differently. The check now also reports dashboards and folders that this stack doesn't provision, including deleted ones.
  • Files you added under grafana/, such as your own dashboards, are listed for you to review instead of being counted as findings.

Validator ejector oracle allowlist updated

The mainnet .env.sample.mainnet adds the rotated oracle keys of P2P (0x3f0dAF0066533bE244dBeF17fDD21fb1F24F41e0) and MatrixedLink (0x0194Ab59A22b6787ef7529954142B269824551d2) to VE_ORACLE_ADDRESSES_ALLOWLIST. The validator ejector only accepts reports signed by allowlisted keys. Your .env is not updated automatically: copy the VE_ORACLE_ADDRESSES_ALLOWLIST line from .env.sample.mainnet into your .env.

Version bumps

  • Charon v1.11.3 → v1.11.4: DKG robustness fixes, secret redaction in stack-sniping telemetry, and a fix for beacon node refreshes stopping after the first request.
  • lido-dv-exit v0.6.1 → v0.6.2: backs off instead of retrying every slot when the cluster lock isn't published to the Obol API.
  • Nethermind 2.2.0 → 2.2.2, Reth v2.8.0 → v2.8.1
  • Lighthouse v8.3.1 → v8.3.3, Teku 26.10.1 → 26.10.3, Lodestar v1.50.0 → v1.50.1, Prysm v7.3.0 → v7.3.1, Nimbus v26.10.1 → v26.10.2
  • Commit-Boost v0.11.0 → v0.11.1
  • Grafana 13.2.4 → 13.2.5, Alloy v1.21.1 → v1.21.3

Known issues

  • Running a DKG in this folder after starting the stack. Starting the stack before the DKG makes Docker create an empty .charon/validator_keys/ folder for the new Charon mount. charon dkg then stops with data directory not clean. Run the DKG before docker compose up, as the README describes, or delete the empty .charon/validator_keys/ folder first.

What's Changed

Full Changelog: v0.3.15...v0.3.16

Check out latest releases or
releases around ObolNetwork/lido-charon-distributed-validator-node v0.3.16

Don't miss a new lido-charon-distributed-validator-node release

NewReleases is sending notifications on new releases.

Get notifications