Polygon’s zkEVM rollup entered an emergency state on March 23 after the chain stopped processing blocks. Final week, Polygon launched an evaluation of the outage that took its zkEVM down.
Polygon reported late Saturday that its zkEVM sequencer had failed, attributing the trigger to a reorg of the Ethereum mainnet. The postmortem states that the synchronizer for Polygon zkEVM mishandled the reorg, and the chain began processing batches of transactions with incorrect timestamps.
A blockchain reorganization, or reorg, occurs when a community discards blocks from a earlier model and adopts a brand new chain of blocks because the correct historical past of transactions. This will happen when completely different components of the community briefly disagree on latest transaction order or validity. The community finally converges on a single model of the reality, selecting the chain of blocks with probably the most computational work (or, in some consensus mechanisms, probably the most stake backing it) as the proper sequence. Reorgs assist to make sure that the community stays safe and constant, although it may result in momentary uncertainties in transaction histories.
Learn extra: Polygon zkEVM prover reaches Kind 1 standing
Polygon zkEVM’s sequencer, which assembles and broadcasts batches of transactions to the community nodes, performs a vital function within the ecosystem. It’s not solely liable for ordering transactions but in addition assigns them timestamps, important for sustaining a coherent historical past of transactions on the community. Nonetheless, in the course of the incident, the sequencer didn’t detect this reorg on the Ethereum mainnet.
This led to the processing of transactions based mostly on outdated data, including an incorrect world exit route — Polygon zkEVMs cryptographic methodology to make sure Ethereum layer-1 and Polygon’s layer-2 settlement — to the subsequent transaction block. The resultant mismatch precipitated inaccuracies, disrupting the community’s potential to take care of a constant document of the community’s ledger.
In the course of the confusion, plenty of transactions have been processed by the community as no-operation, with the community appearing “as in the event that they weren’t there,” the autopsy stated. Synchronizing these empty transactions was a part of what made the community’s downtime prolong so long as it did.
All-told, it took round 14 hours for the rollup to reopen for withdrawals, and round 4,000 transactions might have been affected by the outage, Polygon stated.
The outage solely affected Polygon zkEVM, which is in so-called mainnet beta and subsequently may have bugs and availability points, in accordance with Polygon’s help web page. The Polygon PoS layer-1 and chains utilizing Polygon’s chain improvement equipment (CDK) weren’t impacted.
Polygon’s zkEVM, quick for zero-knowledge Ethereum Digital Machine, is a layer-2 that makes use of zero-knowledge proofs to confirm transactions with the Polygon-operated sequencer earlier than settling batches of transactions on Ethereum. StarkWare and zkSync are constructing comparable merchandise. That is all in distinction to widespread layer-2s like Arbitrum and Optimism, which operate as optimistic rollups and don’t use zero-knowledge proofs.
Learn extra: StarkWare’s zero-knowledge prover Stwo comes out of stealth
Shortly after the outage started, Polygon’s zkEVM went into an emergency state that halted the community and let an improve to the community’s prover and verifier be pushed by alongside a repair to maintain the sequencer from malfunctioning throughout a reorg.
Upon finishing this improve, the emergency state was deactivated, which means any additional updates to the system are topic to a 10-day timelock earlier than taking impact.
This occasion marked the primary time Polygon’s zkEVMs emergency state operate was used, and doing so required approval from the community’s Safety Council. When activated, this state halts sure operations, together with the updating of transactions and the functioning of the blockchain bridge, to guard customers’ belongings.
The council operates as a 6/8 multisig. Polygon says two of the council members are from Polygon Labs, however the identities of the Safety Council members should not publicly disclosed.
A 2023 weblog put up stated the group is “made up of extremely respected members of the Ethereum neighborhood” and will probably be dismantled as soon as Polygon zkEVM is able to ditch its “coaching wheels.”
The outage and its remediation factors to lingering centralization points throughout Ethereum layer-2 networks, in accordance with Jarrod Watts, a developer relations engineer at Polygon Labs:
“The worst case state of affairs is an improve is carried out that steals consumer funds,” he stated on X. “This type of malicious improve is one thing that’s attainable on nearly each [layer-2] right now.”
However Watts added, “over time, IMO the chance of malicious upgrades to steal consumer funds on [layer-2s] will probably be zero or very near zero!”
Polygon zkEVM has struggled to realize complete quantity locked (TVL) and utilization, and has but to learn from Ethereum’s Dencun arduous fork. The chain is slated so as to add help for the “blobspace” of EIP-4844 in a couple of month’s time, following its “Feijoa” improve.