Short story:Somebody hacked my backup machine with pool data hosted on Linode and steal
3094 BTC ("hot" coins ready for payouts). Cold backup was not affected in any way by this hack.
It looks that also user database has been compromised. Although passwords are stored in SHA1 with salt, I
strongly recommend to change your password on the pool
immediately.
Robery of Bitcoins has no impact to pool users, I'm covering the loss from my own income (although it means that many months of my work is wasted
).
Long story + evidence:This morning I received SMS from pool monitoring that BTC balance went under expected amount, so I started investigating what happen. I saw that there was transaction moving 3094 BTC out of the pool wallet (
http://blockexplorer.com/tx/34b84108a142ad7b6c36f0f3549a3e83dcdbb60e0ba0df96cd48f852da0b1acb) few minutes ago. I watched the logs and it didn't look like server has been compromised in any way.
Then I found that two of my Linode machines has been restarted half a hour ago, too, and root passwords has been changed. I changed passwords to new one and found that there was malicious activity on the machines. Then I discover that passwords were changed over Linode Manager (Linode web management), because there was record about password change in Host Job queue (last activity done over Manager). This also explains why attacker restarted machines, because it's necessary to apply this change from Manager.
I reported accident to Linode staff and asked for log of recent logins to Manager. To my surprise, there were only my own log attempts and last login before the attack was on 08/02/2012! I reported to Linode that something is going wrong, because I has been using strong password for my Linode Manager (because I know it's basically backdoor to my machines) and I didn't use this password on different places.
Full log of support ticket is
here.
I'm still waiting what they'll find, but expect they'll try to hide any issue on their side and they will definitely reject to pay 3000 BTC for this attack :-/.
PlusFew hours ago another guy contacted me that his Linode machine has been attacked and his coins was moved to the same wallet, asking me if I know what happen (because he found that 1Mining2 address is mine). We found that our issues are the same - changed password in Manager, stolen coins & Linode staff is telling they have no security issue on their side. Heh.
It looks like attackers found some vulnerability of Linode Manager and used it to infiltrate Linodes with running bitcoind (we both had bitcoind running on the machine), to gain maximum profit for the less rush (it does not look that so much machines has been hacked, at least I didn't find anything on twitter etc). It looks like attackers were interested only in Bitcoins, because they leave Namecoins untouched, although they have the same chance to steal them.
From the attacker's wallet it looks there were more people affected by this Linode hack, maybe they'll know anything more?
ConclusionThere's no reason to think that pool itself was hacked. I changed all passwords everywhere (mainly to database), moved coins to new wallet and everything is working fine. Backup machine didn't contain keys for accessing pool server, so there's no need to reinstall pool to another machine. I'm covering all financial loss from my own money, to keep pool users out of this stupid issue.
