ObolNetwork/lido-charon-distributed-validator-node v0.3.1 on GitHub

Summary

This release introduces Charon v1.9.2. This release is a strongly recommended update for Node Operators sending logs to Obol that are members of clusters that have been instructed to begin their upgrade to the Charon 1.9.* versions. If you are a node operator sending logs to Obol in a cluster still on v1.8.2, please update to LCDVN release v0.2.17 instead.

Operators sending metrics but not logs are unaffected by this patch Charon release.

Operators running Prysm VC must add disable_duties_cache to their CHARON_FEATURE_SET_ENABLE= environment variable when updating to this release.

Breaking changes

If you are updating to this release from v0.2.14 or newer, (and you have set your ALERT_DISCORD_IDS= environment variable as requested in earlier updates), there are no additional action items to update to this release beyond the normal flow described below.

Operators upgrading from a version older than v0.2.14 should note that there are breaking changes to monitoring that require environment variable adjustment for those running v0.2.9 and earlier. Details below:

Note

Recent releases adjusts how metrics are sent to Obol. Since recent updates, you no longer have to keep stashing and unstashing changes on prometheus/prometheus.yml every update. It is now generated at runtime by ./prometheus/run.sh, and the required variables are injected into the file from your .env. If you're updating from an older LCDVN version. The steps you need to take to handle this change are described below.

If you currently have a locally modified prometheus/prometheus.yml, make a backup copy of it outside of the repository before upgrading (just in case). Ensure you record the value of PROM_REMOTE_WRITE_TOKEN from the authorization.credentials section of that file.
This token must now be provided via your .env file instead of being defined in prometheus.yml. The variable PROM_REMOTE_WRITE_TOKEN is present (commented out) in all .env.sample.* files. To configure it, copy the variable to your own .env, uncomment it and set your token, for example:

PROM_REMOTE_WRITE_TOKEN=obolH7d...

We have added a dedicated way to set discord IDs on your deployment, allowing you to be @'d directly by our Obol Agent if there is an issue with your node. Add (uncomment) ALERT_DISCORD_IDS= in your .env, and specify one or more (with comma separation) IDs. To get the ID that corresponds to your discord account; Enable developer mode on discord with User Settings > Advanced. Then right click on a user's profile picture or name and select Copy ID to get a unique 18-digit number that represents their account. If you previously changed your CHARON_NICKNAME to your discord ID, you can set that back to a human friendly name for your node.
If you have any other custom modifications to your original prometheus.yml, compare it against prometheus.yml.example, which is now used as the base template for generating the final configuration. Any additional fields or customizations should be added to prometheus.yml.example.
With your environment variables set, and any other modifications ported to the prometheus.yml.example file; run docker compose up -d as normal. Confirm that your metrics are being received on the Obol Grafana. Once you have verified that the new Prometheus setup is working as expected, you can safely delete your backup of the old prometheus.yml.

Important

To maximise compatibility of environment variable interpolation across operating systems, changes have been made between this version and v0.2.3.

Please set (uncomment) the following .env vars if you haven't already, or your charon, mev-boost, and charon-dv-exit-sidecar may not start properly.

(Copying the .env.sample.mainnet file afresh and making any modifications you may have made, might be the least disruptive way to pick up these changes.)

CHARON_BEACON_NODE_ENDPOINTS=http://${CL}:5052
CHARON_EXECUTION_CLIENT_RPC_ENDPOINT=http://${EL}:8545
VE_BEACON_NODE_URL=http://${CL}:5052
VE_EXECUTION_NODE_URL=http://${EL}:8545
LIDO_DV_EXIT_BEACON_NODE_URL=http://${CL}:5052
CLUSTER_NAME="Your cluster name here"
CLUSTER_PEER="Your peer name here"

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Copy your PROM_REMOTE_WRITE_TOKEN and put it in .env after stashing
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.3.1
# You should no longer need to apply any stashed changes if you followed the instructions above
# Restart the node
docker compose up -d

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

Update charon to v1.9.2 by @OisinKyne in #288

Full Changelog: v0.3.0...v0.3.1