Description
This is a patch release addressing a parachain block production halt that occurred on Kusama on November 25.
The runtime was fixed, and an additional runtime patch was embedded into the parachain chain specification to ensure successful execution of block #9091612, completing the migration.
Post-Mortem
Incident Tracking
A full timeline of events and technical discussion is available in the issue:
paritytech/polkadot-sdk#10428
Background
The runtime upgrade to version 4.0 introduced a major redesign of the parachain block production mechanism.
Previously, the parachain used a simple open relay consensus, where any participant could propose a block.
Starting from runtime v4.0, a new Aura-based consensus algorithm adapted for parachains was introduced.
This new model requires a predefined set of authorities — a dedicated group of nodes authorized to produce blocks.
The initial list of authorities was planned to be set during the runtime upgrade via the OnRuntimeUpgrade hook.
The runtime was tested on a fork of the chain using Chopsticks and prepared for release.
What Went Wrong
On the live network, it was later discovered that OnRuntimeUpgrade is not executed atomically.
Instead, the upgrade process occurs in two phases:
- The runtime structure changes first (including the installation of the new validation function)
- The migration logic runs in the following block
This sequencing caused a critical deadlock:
- v3.x collators were unable to produce blocks because they did not support the new runtime structure
- v4.0 collators could not start producing blocks because the authorities set had not yet been initialized
As a result, the parachain became stuck midway through the upgrade, with no collator version capable of proposing a valid block.
Mitigation & Recovery
Resolving the issue required intervention at the relay chain level.
A patched runtime was prepared that temporarily allowed blocks authored by any address in cases where the authorities field was not yet initialized. This made it possible to produce a block and complete the migration.
The situation was further complicated by the fact that, in order to replace the runtime from the relay chain side, the parachain first had to be unlocked.
This operation requires root privileges and therefore involved a Kusama governance vote, which significantly extended the recovery time.
We would like to express our sincere gratitude to the Kusama community and the Parity team for their support and responsiveness throughout this process.
Once the lock was removed, the patch was applied successfully, allowing the migration block to be executed and block production to resume.
Lessons Learned & Preventive Measures
To prevent similar incidents in the future, we are taking the following steps:
- Move from Chopsticks-based testing to Zombienet-based testing
- Validate runtime upgrades in environments that more closely resemble real network conditions
- Explicitly test non-atomic upgrade paths and intermediate runtime states
Summary
This incident highlighted an important edge case in parachain runtime upgrades involving consensus transitions.
The applied patch resolved the immediate issue, and the lessons learned will directly improve the safety and reliability of future upgrades.
Upgrade priority: CRITICAL
Runtimes
Robonomics
✨ spec_version: 40
🏋 Runtime Size: 774570
🗜 Compressed: Yes
🎁 Metadata version: 14
🗳️ sha256: 0x26d974c0aeaae53839930d18b11041a59c2d4a5eb3116ee0f2b3f9e3ce0fca66
🗳️ blake2-256: 0x359b99b8a5137b69c11df9c143f8a43d208e6c4d525a85e6f65683bb42a54c3a
🗳️ proposal (authorizeUpgrade): 0x280b013620dd22a5803496a3658fe57a3395d8315a47b11fe07005e6da0340ed
📦 IPFS: QmS1jSeihoTqQd8gS8SeTgy7hBC7KvuNKatPSngggjxaW2
Build Info
WASM runtime built using rustc 1.88.0 (6b00bc388 2025-06-23)