Our ETH Account Got Hacked For $20! How did this happen?
Last week, one of our Geth nodes used for prototyping and our old development environment was scammed. We lost about ~20$ worth of ETH to a random botnet — take a look!
How was this possible?
We had a Geth client with the Ethereum blockchain for early development and prototyping use. Our setup was really simple — just an AWS EC2 instance open to the world with the Geth client installed. Imported accounts were protected by passwords and those accounts had small amounts of funds in them — the bare minimum for new contract deployments and transaction fees, but that was not enough to make them secure.
The routine for sending a new transaction to the blockchain is unlocking the account in the Geth client, submitting the transaction and locking the account. There is a gap of a few milliseconds, during which the account is unlocked between these 3 requests (each action is a separate cURL request from the node.js app to Geth client).
Every day at midnight our exchange rate Oracle updates the PBL/USD rate in the smart contract. The attacker was able to replace this rate update transaction with sending the whole account balance by using the same nonce and higher gas price. Since the account was already unlocked, the transaction went through.
Step by step of what happened
- To change data in a smart contract, you need to submit a transaction to the Ethereum network with the function and the new data. This, namely updating PBL/USD rates, is something we did every midnight.
- You do this from your account. Accounts are imported in the Geth client, but are encrypted, and therefore password protected. To submit a transaction, you need to unlock (decrypt) that account first with your password, and at that point it is no longer encrypted on the Geth client, until it is locked again.
- When you submit a transaction, it is not written into the blockchain right away! First it lands in an unconfirmed transaction pool and then miners pick transactions from this pool beginning with those with the highest transaction fee, and mine them while they are at the pending pool.
- It is possible to overwrite your transaction by using the same nonce (which, to put it simply, is the transaction number) and a higher gas price.
- This is exactly what the hacker did! They created a new transaction with the same nonce and a higher gas price, meaning that the miners prioritized their transaction over ours!
- How did they get our funds? They replaced our “update PBL/USD rate” transaction, with a different one — “send all funds to address X”!
How to prevent this?
For starters, having Geth exposed to the world is a terrible idea! It should be behind firewalls or other security mechanisms at all times to ensure you’re the only one to have access to it. For our live infrastructure we actually use 2 firewalls for our Geth clients. The first firewall is their location — within our private VPC network without any direct link to the outside. Our second firewall is an internal load balancer that grants access to the Geth client only for specific microservices that should have access. Even if one of our microservices gets compromised, it has to be right one to interact with Geth client.
In our case, we prepare transactions and submit them through the node.js app to the Geth client, which has the imported account. A much better approach would be to sign the transaction with the private key in the node.js app and submit the signed transaction to Geth. This means that there is no need for having an account on Geth, which needs to be unlocked.
And, of course, don’t store a large amount of funds in a hot wallet. If we had a considerable sum stored in that specific wallet, then this whole experience could have been severely more costly. Our service ETH wallets only have $100 or less worth of funds at any given time. The Oracle account has even less, since transactions cost only a few cents a day. Because of this, the attacker was only able steal ~20$ worth of Ether and we had a good reason to do some spring cleaning for our old infrastructure and leftovers from prototyping phase.
Curious about the Microservice Architecture we use for our publishing cryptoproject Publica ? Learn more here ! Or find out why you should turn to Scandiweb for help on your blockchain projects !
Our ETH Account Got Hacked For $20! How did this happen? was originally published in Scandiweb Stories on Medium, where people are continuing the conversation by highlighting and responding to this story.