NewsFlash Articles Data Fundraising BlockBeats Pro

The first five years of MEV: Starting from before the birth of Flashbots.

Read this article in 27 Minutes

Specialized MEV is predictable MEV, and predictable MEV is useful MEV.

Original Title: "MEV: The First Five Years"

Original Author: James Prestwich

Original Translation: aididiaojp.eth, Foresight News

Five years ago, I wrote an article titled "Miners Aren't Your Friends" which mentioned MEV and Ethereum. At that time, we didn't call it MEV yet. Phil Daian and his colleagues named it "Miner Extractable Value" in their groundbreaking paper "Flash Boys 2.0" a year later. We planned to publish a follow-up article with Python code for front-running trades on EtherDelta and other mainnet DEXs. However, I got involved in building HTLC and other cross-chain tools, and the article was delayed and eventually shelved.

Writing about MEV today, five years later, feels a bit surreal. It's like we're once again standing at the beginning of a long bear market, amidst the ruins of a speculative frenzy. Just like last time, I firmly believe that things will be different this time around. We will learn from our mistakes and create something better.

This is an article about the history of MEV. It is not without bias, in fact, it is a bit of a ramble and even a bit self-indulgent. I am just a supporting character in this story, but in my blog, I will tell it from my perspective.

MEV is the foundation of protocol design. Every serious practitioner should understand MEV, make plans for MEV, and have strong opinions on MEV. From the current perspective, it is easy to forget that our understanding of MEV is brand new. Its roots are as ancient as cryptocurrency, but the terminology, frameworks, and tools are still being invented. The past two years have completely changed our understanding of MEV, but it is easy for people to forget that MEV started from scratch.

MEV Story Before

The discussion of MEV can be traced back to the early 2010s when Bitcoin was researching "fee sniping". "Fee sniping" was later promoted to the EVM state model and evolved into the "time bandit" attack described in the Flash Boys 2.0 paper. Bitcoin implemented the first consensus layer MEV mitigation measure (100 block coinbase maturity rule) and also the first client layer MEV defense (adding a time lock of 1 block to node wallet transactions in December 2014).

Source: 0xB10C, Fee Sniping Relief

The competition for status and the state transitions that have been submitted but not yet confirmed have generated MEV. Since Bitcoin basically has no shared state that can be competed for, and Bitcoin state transitions are strictly specified, MEV is usually limited to fee sniping and other types of double-spending attacks. In other words, Bitcoin miners have little MEV to extract without directly attacking the consensus mechanism. This makes Bitcoin an uninteresting blockchain in MEV research and not worth spending time on.

MEV's Birth (2018-2019)

Like any good relationship, MEV requires two things: competition and commitment. First, it requires people to compete for control over certain blocks. MEV needs people to want the state it desires and be willing to pay for it. Second, it requires commitment before execution. MEV needs time to update disputed states that have been submitted but not yet executed. When users sign and broadcast transactions, they commit to competing for control over that state. The lag between commitment and execution is where MEV comes in. It captures users' attention, pushes their commitment to the worst acceptable outcome, and allows extractors to pocket the difference.

DEX is closely related to MEV, and they have a perfect competitive secret: everyone wants to trade in the market, and each transaction can compete with every other transaction. The self-matching order book DEX designed by EtherDelta in 2016 allowed miners to take a step ahead in trading, but no one really used it for anything. EtherDelta designed several on-chain automated CLOBs, which looked crazy at the time and definitely bizarre in hindsight. Although we knew that it would eventually cost Gas, the cost of one unit of Gas in mid-2018 was about 1/400 of today's price and 1/10,000 of last year's price. It wasn't until 2020 that we had an instinctive understanding of the Gas price market.

The launch of DAI (now SAI) in 2017 introduced a liquidation mechanism to DeFi. Liquidations introduce a significant but infrequent amount of MEV. Due to the advantage of extractors being a certain percentage of outstanding debt, abnormal CDPs make up a large portion of MEV related to liquidations. Over the past few years, we have observed cheating through oracles to win liquidations and extreme gas auctions to win abnormal MEV based on their own protocols. Abnormal MEV can also be caused by hacker attacks, NFT airdrops, and other abnormal events. Because it cannot be divided into small data packets, abnormal MEV has a huge impact on protocol operation. Protocol design must consider rare but potentially large malicious incentives.

On the other hand, DEX tends to fluctuate with external market fluctuations, and the back-and-forth movement of the market creates flow MEV. The characteristic of flow MEV is more frequent and smaller MEV data packets. AMMs are particularly interesting because their use is inseparable from MEV. They exist to track external markets and achieve this by giving value to extractors, who push constant functions towards "fair" pricing. In fact, they maintain efficiency by extracting MEV through "arbitrageurs". The first wave of AMMs (first Bancor, then Uniswap) began to gain traffic in their order books in the mid-to-late 2018s, corresponding to an increase in MEV and causing great attention to be paid to this topic. Of course, as early as 2018, the size of Uniswap's liquidity pool was only a few million, and the assets were much smaller.

At the time, there was almost no MEV research, and of course, we didn't even call it MEV. We mostly referred to it as frontrunning, borrowing the name from traditional financial trading, namely "DEX frontrunning" or "liquidation frontrunning". We knew it wasn't just applicable to order books. We theoretically understood back-running and sandwich attacks, and we realized that if it was profitable for extractors, transactions could be delayed or reverted.

The veteran DeFi may have quietly built the environment needed to create MEV, but no one really cares. The state is not ideal enough, so there is no return. Because no one has practical experience in this area, we don't even know what it will look like when the extraction arrives.

Flashbots Emerges (2019-2020)

With the release of Flash Boys 2.0 in April 2019, MEV research began to gain popularity. It is the foundational paper for protocol and mechanism designers and will forever be a part of standard curriculum, but without Tina, I think it may have ended there.

Tina and I met in the office of a certain VC in early 2018. We were introduced through a mutual friend, and our sales pitches were at odds with each other. I was promoting Summa's HTLC variant, while we both realized the limitations of cross-chain protocols for two online participants (and the market's aversion to complexity), and we believed it would eventually be cancelled. Tina was promoting a social chain game about ranching (unrelated to the later CoW Swap). We were both interested in mining derivatives and had some common ideas. We caught every hackathon during the bear market. She has a talent for choosing forward-thinking ideas and building teams around them. Her hackathon project became CarboClan, which built honeylemon on top of the idea of mining derivatives.

In the second half of 2019, Tina began organizing events. She hosted a conference themed around defi.wtf in just one week and co-hosted with Osaka DevCon. The second conference, macro.wtf, began just three weeks later. Pirate Ships started shortly after that. The name "Pirate Ship" came from a joke about flash loans and flash mints that we made at EthDenver in 2019. Tina hosted salon-style gatherings in San Francisco, New York, and other places. You can visit this ship and find a group of unconventional researchers, engineers, operators, scammers, and other crypto enthusiasts. Some people have resident researchers, some have one or more themes, and some only stay in Puerto Rico for a week. Pirate Ships has been running throughout the COVID-19 pandemic since the end of 2019, attracting the most active crypto nomads.

Compared with the Flash Boys 2.0 paper, the emergence of Flashbots can be attributed to Pirate Ships, which was born in a permanent salon. MEV Ship was established in the middle of 2020 and naturally expanded from personal space to digital collective. Other Flashbots founders and managers such as Stephen, Phil, and Alex were pulled into this ship (you can still find some hints about this in the Flashbots documentation). Eventually, the Ship was officially renamed Flashbots, donning a cute robot emoticon and embarking on its path to immortality.

One whole year is a long process. Flashbots Research (formerly known as MEV Ship) made its debut and pushed MEV to the center of Ethereum discussions. DeFi Summer has started, but we don't know what it is yet. The release of mev-explore and mev-inspect provided rough numbers for MEV available on-chain, suddenly making it tangible and no longer just a theory. With the announcement of the Flashbots Auction that fall, the emergence of MEV extraction tools was imminent.

MEV Ship laid the foundation for Flashbots and established a collaborative research and experimental culture that drives organizational development. Over the years, the clear mission defined by this culture has shaped the narrative of MEV. While the productization of MEV extraction is inevitable, the culture and spirit of Flashbots are not.

Professionalization (2021-present)

Of course, the aura of MEV Ship and Flashbots has attracted many talents in the encryption industry. Just three months after the announcement, the Flashbots Auction (MEV-geth and Flashbots Relay) was launched in January 2021. MEV-geth appeared at the beginning of Ethereum's rapid growth, and the integration of MEV also increased significantly.

Before the Flashbots Auction, some searchers extracted MEV by broadcasting transactions at specific gas prices to the txpool. Obviously, this was unreliable and inefficient. It required searchers to view, simulate, extract, and broadcast transactions in the block range from the txpool. The fee semantics before EIP-1559 and the general unreliability of the txpool made this process complex. Assuming blocks were sorted by fees, frontrunning transactions would select fees to attempt immediate execution before the target. Abnormal MEV transactions would only pay astonishing gas prices. As searchers used a public transaction pool, real-time auctions were highly competitive.

The relative simplicity of Flashbots Auction has driven its adoption. It can be boiled down to allowing searchers to submit a single RPC endpoint for their bundle to be included in a block exactly as specified, with recovery resulting in the bundle being removed, creating a streamlined interface for specialization. Searchers seek out MEV, capture it, and pin it to a high-quality bundle, which is then passed on to miners to include on-chain. Flashbots Auction quickly defined extraction.

As we saw in mev-boost, the later version of Flashbots Auction separates "builders" from "producers" (a common term for miners and stakers). I first heard about the proposal for builder-producer separation (PBS) at EthBerlin 2019. At the time, Will was researching ETH2.0 and taught me a lot about the latest designs for state witness production and stateless Ethereum. I don't remember what we called it then, but it was conceptualized as a protocol-authorized specialized node with additional responsibilities. Stateless Ethereum died during the COVID-19 period. MEV PBS, instantiated in mev-boost, exists outside the protocol and has become a market-based specialization rather than a protocol-specified one.

Today, we have a relatively mature MEV supply chain. Searchers mine the transaction flow of MEV, including competing peaks and traffic. Searching requires skills and high specialization, which are kept secret because they need to maintain an advantage over other searchers. Builders accept data packets from searchers and build them into blocks. Searchers and builders have a symbiotic relationship, with builders relying on searchers to extract, and searchers relying on builders to honestly include the data content.

The builder purchases the right to include their block in the main chain from the proposer. The proposer has the right granted by the protocol to add blocks to the chain and chooses to pay for the most MEV. Users create MEV, searchers extract it, searchers pay the builder, and the builder pays the proposer.

When we discuss this topic, we need to be aware that bundle has an unexpected side effect: they can cause searchers to inadvertently subsidize the transaction fees of their target. When a searcher includes a user transaction in a bundle, that user transaction can be confirmed earlier and more reliably than other means. The MEV portion extracted by the searcher is paid to the supply chain of builders and proposers. This effectively converts some of the extracted MEV into transaction fees for the bundle. Essentially, the extracted MEV pays "shadow fees" to the proposers. The priority of a transaction is determined first by its MEV and second by its protocol fee.

Extracting accidentally broke EIP-1559, and non-bundled transactions use gas fees on top of the base fee. On the other hand, bundling is still an effective first-price auction, with the base fee being a fixed fee rather than a gas fee. This means that there are now two interdependent fee markets, one for shadow markets for MEV transactions and one for regular markets for all other transactions. We predicted these shadow fee markets before 1559 (and all other fee mechanisms), but what we didn't anticipate was the use of the MEV market track. Since extraction is done behind the scenes, users benefit from the secondary market, even if they don't know it exists.

From the perspective of mechanism design, MEV-driven PBS may not be "safe" or "incentive-compatible". So far, Flashbots MO has relied on the honest assumption to fill the gap in incentive mechanisms. Then, they established these assumptions as market norms. Why don't builders split bundles and adopt MEV? Why don't builders include recovery packages? Because violating these market norms will cause searchers to stop using relays. We bet that iterative games will be more profitable than bundle MEV games ending in a grab. In the long run, I am not sure if it can support MEV and vertical integration.

We can temporarily ignore the incentive incompatibility of auctions, and having market regulations is sufficient. Mechanism design is subject to market pressure, and profit compression is the only relevant issue in today's MEV supply chain. As proposers have the exclusive right to choose blocks, builders must compete on price. They are forced to give up more and more MEV shares to proposers and take more from searchers for payment. The MEV profit margin has collapsed, and proposers occupy the largest share.

Fundamentally, the proposer extracts rent and the builder and searcher have no choice but to purchase from no one else. The proposer has been granted exclusive rights to select the next block and can exercise that right unsupervised. To quote a rather famous politician, "The proposer has got this thing, and it's got a golden value, and they're not going to give it up for nothing." Rod, like the proposer, has been granted something of great value by a mysterious protocol, not by moral or natural right, but by the rules of the protocol, which can be changed by humans. In any case, the proposer is rewarded for blockchain expansion, and the builder pays the proposer a fee to expand it in a specific way.

Before the extraction of MEV, sorting was a perfunctory role, and the overall subsidy of inflation was the main incentive. Therefore, the right was distributed for free, which is why proposers do not pay any fees for this right. It used to be almost worthless. Now, the sorting right includes a lot of MEV value, and this asymmetric power relationship distorts the MEV supply chain. Research in this field is ongoing, and we are likely to see proposed protocol mechanisms to address this issue.

Sunny advocates threshold encryption for transactions in flight. Encrypted transactions prevent proposers from knowing the MEV value of a transaction before it is sorted. Personally, I do not believe that MEV will be eliminated. I am interested in designs that include MEV in fork selection rules, selling sorting rights to the highest bidder, and retaining most of the extracted value for redistribution within the protocol.

MEV: The Next Five Years

So, what does this leave us with? Five years ago, MEV was just a curiosity, a topic of conversation over dinner among researchers, discovered accidentally at the bottom of a bottle after all the real business was done. MEV was waiting for the right combination of DeFi activity and practical experience. As we enter the bull market of 2021, Flashbots has seized the attention of MEV research and has been on the rise ever since.

Now, we are holding a MEV symposium, bringing together searchers and developers to discuss MEV-aware protocol design. The market structure has been determined to be configured with three roles and is beginning to test its limits. Although people still respect the ideals of Pirate Ship, extraction on Ethereum has become professionalized. Looking to the future, MEV will not be defined by trends or research, the story of MEV now belongs to profitability.

Throughout its history, people have referred to MEV as dangerous or evil. The word "theft" has been widely spread. I think it is unreasonable to make value judgments. MEV is a fact, and no amount of philosophical thinking, research, or bumpkin Gödel will change that. That's why I'm glad to see competitive MEV extraction supply chains. Specialized MEV is predictable MEV, and predictable MEV is useful MEV. We are not smart enough to plan the market in advance, so let's use it where it is practical.