malbeclabs/doublezero contributor-rewards/v0.7.0

on GitHub

Changes in 0.7.0

• feat(contributor-rewards): the worker retries a failed merkle root posting 3 more times, one minute apart, before it reports the failure. The 3 retries share one grace-period wait, so grace_period_max_wait_seconds still limits the total wait
• feat(contributor-rewards): the metro prices, which set the priority of the multicast demands, come from the new metro_prices settings and no longer from the MetroHistory accounts of the shred subscription program. The defaults are 100 for FRA and AMS, 60 for NYC, LON, SIN and TYO, and 30 for every other city, and the snapshot stores the prices of its epoch. Every city that is not in the table gets the default price, so a city whose price on chain was not 30 changes its Shapley share (malbeclabs/infra#3302)
• feat(contributor-rewards): the settings rpc.solana_read_url and rpc.solana_write_url become one setting, rpc.solana_url (DZ__RPC__SOLANA_URL), which the worker uses for reads and writes. Replace the two old settings with it. Use a Solana testnet URL when network is testnet, and a mainnet-beta URL when network is mainnet-beta
• feat(contributor-rewards): add calculate-rewards --shapley-from-ledger, which reads the epoch's shapley output record from the DZ Ledger instead of running the Shapley calculation, then posts the merkle root built from that record. The flag implies --skip-device-telemetry, --skip-internet-telemetry, --skip-reward-input and --skip-shapley-output, so no record on the ledger is rewritten and a failed merkle root posting retries without the 20 minute calculation (#4469)
• feat(contributor-rewards): add offchain/crates/contributor-rewards/Dockerfile, which builds the worker into a linux/amd64 image for the Kubernetes CronJob. A person builds it from the repository root with docker buildx build --platform linux/amd64 -f offchain/crates/contributor-rewards/Dockerfile . and pushes it to ECR (malbeclabs/infra#3302)
• feat(contributor-rewards): the leader schedule of a DoubleZero epoch adds up the slot counts of every Solana epoch that the DoubleZero epoch overlaps, where it used the one Solana epoch that holds the first timestamp. Solana epochs are now shorter than DoubleZero epochs, so the one epoch described only the first part. The snapshot stores the epochs in leader_schedule.solana_epochs, and an older snapshot without them still loads
• feat(contributor-rewards): add snapshot --leader-schedule-file, repeatable, which reads the leader schedule of one Solana epoch from a JSON file with solana_epoch and schedule_map instead of fetching it, and adds the files up. It makes a snapshot for an epoch that the RPC no longer holds a leader schedule for. The command refuses files that do not cover exactly the Solana epochs that the DoubleZero epoch overlaps
• feat(contributor-rewards): add scheduler start --once, which runs one tick and exits, with a failure code when a step of the tick failed, so a Kubernetes CronJob can run the scheduler and retry a failed run (malbeclabs/infra#3302)
• feat(contributor-rewards): the scheduler checks the chain for an epoch's device telemetry, internet telemetry, reward input and shapley output records and its merkle root before it builds the snapshot, so a run with nothing to do ends without fetching the epoch. When a snapshot for the epoch is already in storage, it loads that snapshot instead of fetching again, so a retry posts the same root as the first attempt. To build a fresh snapshot, delete the stored one (malbeclabs/infra#3302)
• feat(contributor-rewards): with access_key_id and secret_access_key both empty or unset, the S3 client loads credentials from the AWS default credential chain, such as an instance role or a pod role. A configured pair works as before, and a config with only one of the two keys is an error (#4418)
• fix(contributor-rewards): checking whether a snapshot exists in S3 returns false for a missing object. It matched "NotFound" in the error text, but the SDK prints only "service error", so a missing snapshot was an error
• feat(contributor-rewards): the scheduler creates the next epoch's Distribution account each tick, moved from doublezero-solana-validator-debt initialize-distribution. Each tick finalizes the rewards of the epoch before the one it processes, which locks that root one epoch after it was posted. The program requires a finalized debt calculation first, so the scheduler finalizes it with zero debt in the same transaction. It skips the finalize step with a warning until the program's minimum_epoch_duration_to_finalize_rewards allows the epoch, so that setting must be 2 or lower. The rewards accountant key signs both steps, which is the debt accountant on mainnet-beta. The scheduler no longer writes off debt or sweeps tokens
• feat(contributor-rewards): add the example catch_up_distributions, which creates the Distribution of each DoubleZero epoch from the program's next epoch through --through-dz-epoch. The scheduler creates a distribution only when the program's next epoch equals the ledger's last completed epoch, so this example catches up when the program is behind. It waits for the program's initialization grace period between two creations, and it needs the debt accountant key. Run it with cargo run --release -p doublezero-contributor-rewards --example catch_up_distributions -- -um -k <KEYPAIR> --through-dz-epoch <EPOCH>, and add --dry-run to simulate the first creation
• fix(contributor-rewards): the scheduler no longer pays each DoubleZero epoch's rewards from its Distribution account once the swap program sweeps that epoch's tokens, and it no longer posts the Slack message for that payment. The distribute-rewards command and its Slack table are removed too. The monthly steps do not change. The state file drops last_distributed_epoch and consecutive_distribution_failures, and a state file that still has them loads without error
• fix(contributor-rewards): distributing rewards adds to its compute budget what the program spends deriving each recipient's ATA, 1,500 compute units for each bump past the first. Measured against the program, a recipient costs about 10,250 at its ATA's first bump, so the flat 12,500 per recipient ran out for a contributor whose recipients' ATAs derive at low bumps, and every retry failed the same way. Epoch and monthly payments share this budget (malbeclabs/infra#2696)
• feat(contributor-rewards): with scheduler.enable_monthly_distribution set, which it is not by default, the scheduler finalizes the previous UTC month's contributor rewards and distributes the last three months'. Once the previous month is collected and a DoubleZero epoch has ended after it, the scheduler recalculates its shares, stores the leaves in a monthly_rewards record on the DoubleZero Ledger keyed by month and merkle root, and posts and finalizes that root in one transaction. It distributes each unpaid leaf with DistributeMonthlyContributorRewards for three months, so a contributor that sets up its recipients late is still paid. Any other month with work left is for a manual run. Each tick warns about it, and a month left unfinalized also counts the tick as failed (malbeclabs/infra#2696)
• feat(contributor-rewards): add distribute-monthly-rewards --year --month --solana-url --dz-url [--dry-run], which finalizes and distributes one month by hand with the scheduler's own steps, signed with the rewards accountant's keypair. It takes both RPCs as flags and reads no config file. It covers a month the scheduler left for a manual run (malbeclabs/infra#2696, malbeclabs/infra#3211)
• feat(contributor-rewards): with scheduler.enable_monthly_distribution set, the scheduler records each DoubleZero epoch's end, the slot and block time of its last block, in an epoch_end record on the DoubleZero Ledger. calculate-monthly-rewards reads these records before block times, so selecting a month no longer depends on the RPC's block history, which on testnet is about 18 days (malbeclabs/infra#2696)
• feat(contributor-rewards): add calculate-monthly-rewards, which averages the posted shares of every DoubleZero epoch whose last block falls in a UTC month into contributor shares that total exactly 1,000,000,000, the input monthly contributor distributions need. It refuses a month that has not closed, has no epochs, or has an epoch whose shapley output record does not match the merkle root posted to its Distribution. It is read-only (malbeclabs/infra#2692)
• test(contributor-rewards): mark the three keypair tests #[serial]. They set and remove REWARDER_KEYPAIR_PATH, which is process-wide state, so one removing it while another sat between set_var and the read made the read fail. It never fired under cargo nextest, which gives each test its own process, and became reachable when this crate joined a workspace whose CI runs cargo test
• test(contributor-rewards): fix the Shapley golden tests. assert_close now treats both values as equal once each is below 1e-6, so the seven per-city entries that are cancellation noise (values around 1e-12) no longer sit below their own comparison gate. UPDATE_GOLDEN now regenerates only when set to 1, not on any set value, so a stale exported UPDATE_GOLDEN=0 can no longer make both tests pass while silently overwriting the goldens. The fixture and goldens rename from mn-beta to mainnet-beta, matching the no-abbreviations rule
• test(contributor-rewards): extend the Shapley golden to per-city outputs. The aggregate can hide drift when two cities move in opposite directions, so the per-city values are pinned as well
• test(contributor-rewards): pin the aggregated Shapley output for the committed mainnet-beta fixture with a golden file, the crate's first test covering reward values. Drives PreparedData::from_snapshot, the same path the scheduler uses for snapshots. Structure (operator set, ordering, counts) is asserted exactly. Values use a 1e-12 relative tolerance, because bit-identical floating point is not guaranteed across architectures and an exact gate would go permanently red on a CI architecture change. Regenerate deliberately with UPDATE_GOLDEN=1 cargo test -p doublezero-contributor-rewards
• test(contributor-rewards): add a committed mainnet-beta snapshot fixture (tests/goldens/mainnet-beta-epoch-129-trimmed.json) and the script that produces it (tests/goldens/make-fixture.py), for a future Shapley golden test. The only previously committed snapshot (testnet) produces zero reward for every operator, so no golden test can assert a real value against it. Exact Shapley computation is O(2^n) in the operator count, so the fixture keeps only the 4 contributors with the most devices and the 6 cities with the most surviving devices among them, which keeps the network connected and the computation fast. It is not a byte-size trim of the full topology (malbeclabs/infra#2392)
• fix(contributor-rewards): migrate the access pass status value Expired to ExpiredDeprecated when loading a snapshot captured before doublezero-serviceability PR #3831 renamed that enum variant. Without the migration, such a snapshot fails to deserialize (malbeclabs/infra#2392)
• fix(contributor-rewards): the scheduler no longer writes a snapshot it cannot use. A failed leader-schedule fetch was warned and discarded, so an unusable snapshot overwrote the epoch's canonical S3 key and the tick then failed reading it back with "Missing leader schedule". Both producers now propagate the fetch error, and the scheduler validates before saving, which also covers --dry-run, where nothing validated at all. Scheduler failures log the full cause chain, and every EpochFinder RPC error is stripped of its request URL, in the retry logs and in the error it propagates, since that URL carries the mainnet-beta read endpoint's API key into journald and Loki (malbeclabs/infra#2372)
• fix(contributor-rewards): resolve the Solana epoch for a timestamp from real block times instead of dividing wall clock by a hardcoded 400ms slot duration. The old estimate drifted about 30k slots per day of lookback and picked the wrong epoch near a boundary, and no fixed constant survives the SIMD-0525 rollout. That epoch selects the leader schedule rewards are computed against, so the search now errors rather than returning a wrong answer: a backfill older than the endpoint's ledger retention fails on the ingestor::demand path instead of silently mis-estimating (malbeclabs/infra#2317)
• fix(contributor-rewards): snapshot validates before writing. It warns and continues when the leader schedule cannot be fetched, but every consumer rejects a snapshot without one, so the command exited 0 having written an unusable file under the canonical name and a snapshot then export-shapley chain failed a step late. Pre-existing, but reachable now that resolving the Solana epoch depends on block-time reads (malbeclabs/infra#2317)
• migrate to Solana 3.0: workspace solana-* crates and solana-sdk move to the 3.0 line, solana-program-test to 3.0.12, and the doublezero SDK git-deps repin from client/v0.27.1 to the #3830 merge revision (malbeclabs/infra#1853)
• release artifact now builds as a static x86_64-unknown-linux-musl binary (malbeclabs/infra#1853)
• TLS for HTTP clients moves from openssl to rustls; trust roots are the bundled webpki Mozilla set plus the host OS certificate store, so OS-installed private CAs remain trusted (malbeclabs/infra#1853)
• refactor(contributor-rewards): use the shared Wallet memo helpers from solana-client-tools in place of the local RELAY_MEMO_CU constant
• fix(contributor-rewards): backfill User.feed_pk (zero pubkey) in the snapshot compat migration so snapshots captured before doublezero#4030 still deserialize (malbeclabs/infra#1853)
• fix(contributor-rewards): handle a serviceability account that surfaces as a decode Err per-type instead of aborting the whole fetch mid-loop. AccessPass (reward-neutral) decode failures are warned and skipped; every other, reward-bearing type fails the epoch loudly after warning each bad account, since a partial snapshot would feed Shapley a shrunk graph and freeze a skewed merkle. Decode failures increment a serviceability_decode_errors metric (labeled by account type) and are surfaced as DecodeErrors= in the completion log (#403)

Install

This release has no packages. Build the worker image from this tag with docker buildx build --platform linux/amd64 -f offchain/crates/contributor-rewards/Dockerfile . from the repository root.