Observability 🔍

The Wormhole node now exposes a readiness check and detailed Prometheus metrics, by default on [::1]:6060 (configurable via --statusAddr). See OPERATIONS.md for details.

All metrics are documented. Here's an example output from node of a five-node cluster: https://gist.github.com/leoluk/29bda9e98c9cbb771de8c251e9063bd8

Once we gain more operational experience with Wormhole, we expect a common set of best practices on alerting and service level objectives/indicators to emerge.

Improvements

Native polling Solana watcher

guardiand is now using GetProgramAccounts with a server-side filter to poll for new lockup accounts on Solana. This improves lockup observation reliability and mitigates solana-labs/solana#9909. Lockups will now always be processed, even if the node or network was down - this completely eliminates the risk of missing lockups.

As a side-effect of this, lockup observation on Solana is now implemented natively in guardiand thanks to the awesome https://github.com/dfuse-io/solana-go library, removing a lot of complexity. The separate Rust agent is still required to submit messages to Solana.