At block number 12,244,000 the Berlin hard fork was activated on Ethereum. Among many changes, it paves the ground for the upcoming London hard fork which will revamp how fees work on Ethereum with the activation of EIP-1559.
Hard forks change a protocol’s rules in backward incompatible ways. As such, they are always fraught with danger: in the worst case scenario, the network splits in two chains, one following the old rules and another one following the new ones. A compounding risk factor is the presence of multiple implementations of the protocol. Each one must implement the new rules in the exact same way, as well as activate them at the exact same time. Any disagreement between implementations results at best in a partial network halt, at worst in a network split.
294 blocks after the Berlin hard fork activated, the second most popular implementation of the Ethereum protocol, OpenEthereum (ex-Parity) refused to accept new blocks. According to Etherscan, around 16% of all nodes were OpenEthereum (and another 15% Parity).
Immediately, many exchanges and services (including Coin Metrics) either became effectively disconnected from the Ethereum network, or stopped broadcasting transactions out of caution.
Around 5 hours after the stoppage, a fix was released by the OpenEthereum team and most of the affected services were back online 9 hours after the incident started.
At first glance, the number of transactions mined on Ethereum was barely, if at all, affected by the sudden disappearance of OpenEthereum nodes.
(approximate stoppage window shown in red)
The impact of the stoppage is slightly more visible when looking at fees paid. Usually, Ethereum fees increase between 2PM-8PM UTC. That increase was less marked the day of the outage.
(approximate stoppage window shown in red)
The best way to quantify the impact of the OpenEthereum stoppage is to look at how many distinct accounts created transactions:
(approximate stoppage window shown in red)
It usually hovers around 15,000 unique accounts creating transactions per hour, but dropped to around 9,000 during the outage. Knocking down 16% of nodes dropped the number of active network participants by 40%!
Some ERC-20 tokens were also affected by the outage. USDC was one of the most affected tokens, with a noticeable drop in transaction originators during the outage window:
(approximate stoppage window shown in red)
Some other ERC-20 tokens were not as affected, like UNI:
(approximate stoppage window shown in red)
Thankfully, no block was mined that would have split the network in two. Furthermore, there was no decrease in mining activity. This is an indication that miners were not running OpenEthereum.
(approximate stoppage window shown in red)
The stoppage highlighted the specialized uses of the multiple Ethereum clients. Miners seem to be running non-OpenEthereum, probably Geth. Wallet providers like Ledger, as well as some analytics companies like ourselves, favour OpenEthereum as it provides out-of-the-box block and transaction tracing capabilities.
This makes reducing the importance of an implementation to the number of nodes running it moot. Knocking down 16% of nodes resulted in a reduction of 40% in accounts originating transactions (either because the operators behind these accounts were running OpenEthereum, or out of caution in case there was a chain split).
This incident is bound to revive the discussion on the risk/benefit ratio of having multiple implementations of protocols. It is also interesting to consider if a similar situation could arise on Bitcoin given the recent debate around the activation of the Taproot softfork.
