Partnered With:

How do you view the Infura service crash and its impact?

Read this article in 16 Minutes

We should give more trust and respect to client developers.

Original title: "How to view the Infura service crash and its impact?" "

Original source: Ethereum enthusiasts

The incident

On the afternoon of November 11, 2020, Beijing time, Infura, a well-known node service in the Ethereum community, was exposed to an API service error, which led to the collapse of multiple services built on Infura , or the front end displays incorrectly.

As far as Infura itself is concerned, it can be understood as a public Ethereum node, which will receive requests and return certain services, such as helping to forward transactions , For example, check whether a certain transaction has been uploaded to the chain, or the status of a certain account. In fact, as long as you deploy an Ethereum node, you can provide the same service as Infura. But its particularity is that most of Infura's services are free, so many services (including exchanges) have chosen to rely on Infura to broadcast the status of the Ethereum blockchain to themselves, eliminating the trouble of deploying their own nodes .

Because of this, Infura made a mistake, which theoretically would spread widely. During the divergence of the incident, some people even threatened that "Ethereum will fork (or A fork is taking place)". The reason is that on two different block browsers (Etherscan and Blockchair), two different blocks are displayed for the same block height (but the blocks after these two blocks, the two browsers display the same of).

But obviously, Ethereum has not forked at all. In fact, the subsequent blocks displayed by both block explorers are the same, which means that the miners who produced the block (at least most of the miners) did not use two different blocks as the parent block to continue mining , and neither reject each other's blocks. Theoretically, only when the nodes that produce blocks use different consensus rules (so they will reject the blocks produced by each other) and occupy a certain amount of computing power, can a fork be formed.

In fact, it was quickly discovered that this was because Infura was not running the latest version of the Geth client, which was triggered by some particular transaction A bug in the client side caused it to crash. The same is true for Blockchair. So soon someone came out and called on everyone to upgrade the Geth client as soon as possible.

Blockchair team's Nikita Zhavoronkov@nikzh published Twitter, explaining the cause and effect of the incident:

1. A code change by an Ethereum developer caused the split of the Ethereum blockchain that day, and the split began at block height 11234873;

1. p>

2. Service providers who did not update their clients, including Blockchair and Infura, were victimized and left on a chain composed of a small number of people (the chain produced 30 blocks within 2 hours)

3. Technically, this means an "unannounced hard fork"

4. The fix is to upgrade the geth client and run debug. setHead(11234872)

He also said that this incident should never be underestimated, and should be considered as the most important event on the Ethereum blockchain after The DAO incident. Serious accident.

It's really strange why there is a bug that only causes the historical version of the software to crash before a certain time, but the current version does not crash? Doesn't this mean that the consensus rules of different versions of the geth client are actually different, that is, at some point a consensus rule change ("hard fork") that is not backward compatible occurred? In addition, a crash of Infura caused a large area of service failure. Does this mean that Infura has become a "single point of failure" source?

Reason

For the above two questions, the leader of the Geth client team ;Péter Szilágyi@peter_szilagyi has responded.

-Technically, it can indeed be said that an "undisclosed hard fork" occurred, but this was only because the developer fixed a bug that had been dormant for more than two years, and because of fear that the public disclosure of this bug would lead to attacks on Ethereum, so A silent repair was selected.

-People should not despise Infura is not using the latest Geth client. From an operator's point of view, it is rational not to keep up with the latest version of the software. For services that depend on Infura, you have handed over this right yourself, not someone else prohibiting you from running the node, so there is nothing to complain about.

Peter's response also elicited mixed reactions. A member of the Monroe community said that in 2017, they were Choose to fix bugs silently. Of course, Some people also think that it is right to choose silent repair, but at least large infrastructure should be notified Providers, if contacted, can greatly reduce the damage caused by this vulnerability.

At 5:34 am on the 12th Beijing time, Peter released "Geth v1.9.17 client-side post-mortem report", which locates the source of the problem: Geth v1.9.7 released on November 7, 2019 incorrectly implements EIP-211; John Youngseok Yang reported the issue on July 15, 2020, and the Geth team fixed it in the v1.9.17 update on July 20. This fix makes the Geth client consistent with other Ethereum clients (such as Besu, Nethermind) when executing transactions involving relevant rules, but it makes the v1.9.17 version inconsistent with the historical version of Geth.

As Peter said, this process is not at all to introduce a consensus rule that the Ethereum community does not know or disagree with, it is only necessary because of writing a bug Fix bugs. Unless you call writing a bug a "hard fork", there is no reason to call fixing a bug a "hard fork" (Nikita obviously disagrees, saying that it happened twice here, not once, hard fork cross).

Secondly, how to release a fix is actually not simple. Ethereum’s hard fork coordination also takes a long time. If a bug with serious danger is disclosed, it is difficult to guarantee that no one will try to attack during the upgrade process of each node. As a client developer, he thinks more about the security of the Ethereum network than the security of a certain service. Also, they don't take the same silent fixes for all bugs, many of them are fixed publicly.

Team Optimism on the 12th at 7:11am Jing is hiring for Optimism@jinglanW came out disclosed more information: They They copied the code base of the Geth client 6 months ago to research and develop the Optimistic Virtual Machine. During the process, they found a mysterious bug and fixed it, but they have been unable to locate its source; they always thought, The bug might be related to a custom improvement the team introduced, but on the 11th they started to suspect that the bug was in the old geth client, not because they introduced some improvement. So after looking at the node distribution shown on ethernodes.org (and finding that the vast majority of nodes had been upgraded), they decided to test the bug on the mainnet. So there is the following thing.

So, in fact, it was the Optimism team who found a bug and rashly decided to test whether the bug existed on the mainnet, plus the Geth team had previously I chose to fix the bug silently, which made some nodes that were not upgraded in time go wrong.

How to understand and look at this matter?

As far as the cause of the matter is concerned, this is because the client team chose to quietly fix a bug that has been sleeping for a long time. While many people think that the geth team can reduce disruption by contacting infrastructure providers, I still believe here that we should give more trust and respect to client developers. I believe that the Geth client team has a reason for doing this. They know that most of the nodes are using their own software, and they also considered the sleeping time of the bug, so they chose to fix it silently. From the perspective of hindsight, of course, it would be better to notify the large infrastructure provider in advance, and the damage will be less. But is such nitpicking justified? Why don't services that depend on Infura assume that Infura can crash?

I admit I'm not being fair here, but to be more fair, many people have already said it. I just want to show my respect to the geth client team here. I'd like to share my impressions with them, as they've provided a lot of proof-of-work in the past. They deserve everyone's respect.

There is of course room for improvement in the implementation of silent repair measures, and we should also learn from the experiences of the Monero and Bitcoin communities. But if you only want to condemn the geth team, or even speculate on them with conspiracy theories, that is a greater injustice.

About "whether Infura has become a source of single point of failure", there are also simple answers and complex answers. The short answer is no, because like Peter said, no one ever prohibits you from deploying nodes, it's just that many providers choose to outsource themselves. Infura is not a single point that has to be passed through at the design level. It's just that it's probably the largest node service provider for a variety of reasons.

But the complex answer is that the resource consumption of Ethereum nodes is relatively large, which is indeed an underestimated problem. The operation of the Ethereum protocol requires each node to fully execute the transactions contained in the block, and the execution of the transaction must take out the data from the state data and write the result after completion. This process will involve a large number of hard disk random reads and writes. Moreover, as the volume of state data expands, the efficiency requirements for reading and writing will also increase. The "state expansion" problem that was hotly discussed in the past few years has not yet been resolved on the current Ethereum. The threshold for running a node is high, and the number of nodes is naturally small. From a goodwill point of view, if the operating threshold of Ethereum nodes is lowered, I believe that more people will build their own nodes (after all, it is safer) instead of relying on Infura.

But the solution to this problem also depends on the wisdom of Ethereum client developers and researchers. Statelessness can be said to be the ultimate solution to the problem of state expansion. Before the ultimate solution becomes feasible, we still need client developers to contribute more efficient clients to us.

So, one thing did happen, and it did expose some problems, pointing out the direction of our learning and progress. But solving these problems cannot be done without our understanding and respect for different groups in the community. Stay away from conspiracy theories, malicious and clever ridicule, figure out the root cause of the problem, think about its essence and improvement plan. What we do defines who we are.