The Takeaway:

A clever hustle in Ethereum's mempools enabled attackers to steal $8.3 million from MakerDAO users on Black Thursday, according to research published Wednesday.

To recap: The price of ether (ETH) plummeted on March 12 and the Ethereum network was congested by a flood of attempted transactions. As investors fled to fiat, ETH's price sunk low enough to trigger liquidations of the collateral held on the MakerDAO lending platform. These programmatic liquidations enabled attackers to walk away with $8.3 million in ETH, for free, shorting borrowers and MakerDAO itself. 

The congestion, though, was key and completely intentional, according to Blocknative, a company focused on studying action in blockchain mempools.

The new research suggests March’s “Black Swan” event for Ethereum may have actually been a sophisticated plan to cash in on a global sell-off fueled by COVID-19 concerns.

"The entire affair meant [the attackers] were able to achieve over 1,000 zero-bid auctions … and collect that underlying value with almost no out-of-pocket expense," Blocknative CEO Matt Cutler told CoinDesk in an interview.

At the heart of Blocknative's work is mempools: the temporary storage on every Ethereum node where transactions wait to get mined and finalized. 

In mid-March, mempools got congested with useless transactions on purpose, Blocknative said, as part of a plan to win zero-bid auctions for ETH on MakerDAO under just these conditions.

Indeed, the Maker Foundation wrote as much in its post-mortem published in April:

(The Maker Foundation referred CoinDesk to the above blog post and declined to comment further for this story.)

Obviously, many Ethereum users will wonder whether the drop in ETH price itself was somehow manufactured, but that question is outside the scope of Blocknative's investigation. The attackers could have been poised to opportunistically take advantage of a dramatic drop in ETH's price; whether the price drop itself was manufactured remains unknown.

That said, Blocknative did find what appears to be a March 8 test run of the attack's mechanics, a fact the research firm doesn't describe in its report. 

"It is an interesting coincidence that the test and the attack were within just four days of each other," Cutler told CoinDesk. "[But] we don't have any evidence that this is anything other than opportunistic."

Either way, the attackers took advantage of some very subtle insights about both Ethereum and MakerDAO. "They basically exploited some techniques that had never been seen before," Cutler said.

More on those techniques later. First, we need to cover a few basics about MakerDAO and Ethereum.

MakerDAO is known as the creator of dai (DAI), the decentralized stablecoin currently beloved by yield farmers. DAI is created with debt. Users put ETH or other crypto-assets up as collateral on the Maker platform to then withdraw a portion of the value of those assets in the form of brand-new DAI.

To get back their collateral, users must repay the DAI they borrowed plus whatever interest the loan has accrued (in MakerDAO parlance this is the "stability fee," but it's just a variable interest rate). MakerDAO enforces the DAI price by liquidating collateral if its value falls below the minimum threshold to maintain proper collateralization. For ETH, that's 150%, but most users put in a lot more ETH than the minimum.

So, if ETH were at $200 and the user posted 1 ETH to borrow 100 DAI, they won't get liquidated unless ETH drops below $150.

But on Black Thursday, ETH's price fell almost $100, from $193, so that triggered a lot of liquidations.

Liquidations can be done by anyone, by the way, with bots called "Keepers." MakerDAO itself runs a Keeper, but a few other unknown entities do as well.

Keepers win liquidations through an auction (described step-by-step in plain language by CoinList), so different Keepers bid to close the loan, and on Black Thursday, those auctions only lasted 10 minutes, or a few dozen Ethereum blocks.

The idea is that these auctions should (and normally have) resulted in users getting back their collateral minus however much they owed, plus the stability fee and the liquidation fee (it's the last part that hurts). But that's not what happened this time.

Borrowers got nothing and, in fact, MakerDAO got paid back much too little DAI, and the whole system was undercollateralized.

Ethereum is a blockchain, which means it’s always gathering up transactions and miners are competing to compose blocks of those transactions, encrypt them, break the encryption and then prove their work to the rest of the miners to win a block reward.

Transactions aren't real until they are in a mined block. And there are usually more transactions out there waiting to get into a block than there is room for more transactions. Those delayed transactions wait in what's called the "mempool." 

Mempools are one of those things that most people don't really need to think about most of the time, except they become really important when situations get urgent: like when the price of ETH is falling off a cliff.

"When you most need to be sure that things are happening are happening in an orderly fashion," Cutler said, "is when things are least reliable."

This is the whole point of Blocknative. The firm keeps a detailed account of mempools all over the world, studying what it calls "value in motion." Blocknative helps its customers decide if they need to be more aggressive in things like gas payments when things are going crazy. Mempool data is "value in motion;" finalized blockchain data is value at rest.

Crucially, miners cannot process a new transaction if the prior transaction hasn't gone through. Every transaction on Ethereum from a wallet gets a number, and 515 won't go through if 514 hasn't (this is tracked by the transaction "nonce," in Ethereum-speak). This sequential reality turns out to be the key to the attack.

Blocknative has been keeping mempool data for Ethereum going back to early 2018 (also its testnets and for the Bitcoin network as well). The firm decided to take a look at the mempool data to see what happened around March 12.

Blocknative found that an unusually high proportion of the mempool was clogged by transactions with very low gas prices on them. 

Usually this proportion isn't very high because users actually want their transactions to go through, so they will monitor gas prices and set them at levels that are likely to get picked up by a miner. But that's not what was happening on March 12. There were loads of transactions in the pool that had low gas prices on them. Too many.

This allowed the attackers to submit "zero bids" in MakerDAO's collateral auctions with strong gas prices attached – knowing full well they could likely win those auctions against well-intentioned Keeper bots who couldn't get their bids through.

Blocknative describes something called "Hammerbots." These would be bots designed to craft transactions precisely for the purpose of clogging the mempool. 

"The bots hammered the mempool with transactions that were never intended to be finalized. These 'Hammerbots' consumed mempool resources by issuing extremely high rates of replacement transactions without any corresponding increase in gas," Blocknative wrote on its blog.

These transactions were additionally designed with a lot of pointless operations that could be shifted and changed easily to vary the hash, but appeared to serve no real purpose.

"These particular transactions, they would be particularly good at consuming mempool resources," Chris Meisl, a Blocknative co-founder, told CoinDesk.

So that's the first problem: Congestion made it hard for borrowers on MakerDAO to add more collateral and it made it hard for Keepers to get bids through. 

"This resulted in anomalous mempool conditions, which would ultimately favor certain transactions," the Blocknative post reports.

But there was another crucial observation the attackers appear to have made about Keepers: they didn't seem to be checking to see if transactions were getting through.

"When you do transactions on an account or address on Ethereum, they have to be ordered," Meisl said.

As we wrote above, if a nonce is missing in a blockchain's record, miners can't take later transactions until one with the prior nonce comes through. So a later transaction will get stuck, even if it has a very high gas price attached, until the prior one goes through.

This had a bizarre upshot. From the Blocknative blog post: 

In short: The attackers knew Keepers would fail to get their first bids through and it would result in subsequent bids "probabilistically" (in Cutler's words) getting stuck. And it worked often enough.

The open-source code that MakerDAO published for Keeper bots didn't have measures to check for stuck transactions.

This created a potential gap that allowed the attacker to submit a bid with a strong gas price but a 0 DAI bid for the collateral, starting that short 10-minute auction clock ticking.

"While automated trading systems are often designed to programmatically increase the gas price of transactions, many such trading systems do not handle nonce gaps well – if at all," the Blocknative post warns.

In 1,462 cases, the Keepers failed to notice that their bids were getting stuck in the mempools, the attackers won the bid, stealing millions of dollars in ETH and nearly forcing an emergency shutdown on MakerDAO.

MakerDAO has since extended the auction time to six hours. Blocknative has opened its data set of mempool activity for members of the community to study further. 

The blog post notes:

In this case, however, the attackers studied Maker's Keeper code and realized it was possible to know what the real Keepers didn't.