Looks like my BTC wallet was hacked

[Tommo_Aus]

I logged into my server this morning but it was running slowly, so I thought I'd restart it and see if that helped, which it didn't. Turned out to be some connectivity issues as far as I could see, the server would be available for 40 seconds, then drop out for 20 seconds and be back up again. It wasn't a DDOS as looking at the 24 hours server statistics from my web host there wasn't any spike in network traffic.

These issues eventually subsided so I went to process the daily TomCoin payouts where I transfer BTC from other sources to the one wallet and make payments according to share contributions (its a BTC payout multipool). The pool has a 0.01 BTC minimum payout meaning the wallet has some residual funds that carry over from previous days, checking the incoming BTC transactions for the payouts I noticed the wallet was lower on funds than I'd expect. I checked the transactions and noticed these two:

{
        "address" : "1EEERRbx4v6TNxgHJNthgroKBQLhehgdRt",
        "category" : "send",
        "amount" : -2.00000000,
        "fee" : 0.00000000,
        "confirmations" : 31,
        "blockhash" : "0000000000000000cf924f2bf8543fd4448b741be87c3faaa769dbf92d95d37b",
        "blockindex" : 34,
        "blocktime" : 1392593790,
        "txid" : "23ad0f3424c038b00f8b4113edf8b9d2725a38b20f2b63ba05e84359e5ae7262",
        "time" : 1392592307,
        "timereceived" : 1392592307
    },
    {
        "address" : "1EEERRbx4v6TNxgHJNthgroKBQLhehgdRt",
        "category" : "send",
        "amount" : -0.83000000,
        "fee" : -0.00010000,
        "confirmations" : 31,
        "blockhash" : "0000000000000000cf924f2bf8543fd4448b741be87c3faaa769dbf92d95d37b",
        "blockindex" : 298,
        "blocktime" : 1392593790,
        "txid" : "c673db0fe09107b9ef3239571fbd5718fdc38691ff4badeb1b4d52fbc31a08fb",
        "time" : 1392592452,
        "timereceived" : 1392592452
    }

They occurred perhaps half an hour before I restarted the server, no payout jobs were running at this time and I didn't perform any manual transfers during this time. This address has never mined with my pool.

I've been able to find the txid's in the blockchain explorer, strangely the 2 BTC transaction above doesn't match the blockchain explorer, instead its listed as 2.39021875 BTC.

I've looked through the server logs and the only successful logins are from myself, although yet again I have various failed login attempts from Chinese IP addresses. Unfortunately I can't get anything useful from the BTC wallets debug.log as it starts fresh each time the wallet starts, and seeing as I restarted the server I had to restart the wallet.

I'm guessing that's it for the BTC, I accept there probably isn't any chance of recovering it as the transactions can't be rolled back, but what I haven't figured out yet is how it happened. All I can think of so far is someone cracked a random 45 character wallet password but the probability is so low it shouldn't even be a possibility, and in any case a lot more of us would be in trouble if password that long are being quickly cracked.

Is there a way I can track the funds to see where they're used or find out more about the transactions? I can't see where I should go from here if anywhere, and I'm worried it could happen again. Just another fun part of being a pool operator I guess.

[sentriclecub]

Worth ruling out the obvious--did anyone have access to your machine while you were asleep like an ex-gf or ex-roommate.

I too have about 1.5 btc in an online wallet so I'm very interested in my own btcoinage.

Please keep us updated. I've subbed to this thread.

[Tommo_Aus]

Nope, I'm the only one with access to the server. I've been through the access logs and each entry matches when I've logged in so I'm pretty sure I can rule that out.

I'd like to be able to say it was possible because of xxxxx and now I've closed that hole, but I still don't know how it happened.

[bananas]

The people who work in the hosting company?

[Tommo_Aus]

Its always a possibility but seems pretty far fetched that someone in the hosting company would do it, word would travel pretty fast if that was the case and they wouldn't be in business for long.

[fcmatt]

Do you allow rpc access to bitcoin from all remote ips or just trusted subnets like localhost and your workstation?

[Tommo_Aus]

I used to allow all but after today I've changed it to localhost, its a pain as I remote to the wallet from a number of different machines, some with dynamic IP's so I guess I'll need to work out another way to go about my business. I would've thought such a long password would be secure, this particular wallets only been in service for a couple of weeks.

If the password was indeed hacked, I guess by brute force, it seems pretty incredible it was done within such a sort period of time given similar passwords I've tried in password calculators estimate over 100 years to break the code. Not to mention I'd surely notice a huge spike in network activity if such an attempt was made.

[fcmatt]

I used to allow all but after today I've changed it to localhost, its a pain as I remote to the wallet from a number of different machines, some with dynamic IP's so I guess I'll need to work out another way to go about my business. I would've thought such a long password would be secure, this particular wallets only been in service for a couple of weeks.

If the password was indeed hacked, I guess by brute force, it seems pretty incredible it was done within such a sort period of time given similar passwords I've tried in password calculators estimate over 100 years to break the code. Not to mention I'd surely notice a huge spike in network activity if such an attempt was made.

i know this might be tinfoil hat territory but is the server hosted some place where an employee or someone else can sniff packets?

Otherwise how does one explain the high load except by brute forcing? and brute forcing might not use as much bandwidth as you think.. but it would create really high load on the server.

too bad the logs are gone.. that would be the best hint.

[Tommo_Aus]

Yeah I wish the logs were still there, or that logs were appended to after a wallet restart rather than a new file being created.

The servers at one of your standard web hosts, so I guess it'd be as susceptible as any other dedicated server hosted in a DC. I did a brief top to check processes before I restarted, I didn't notice anything out of the ordinary, this would've been before the transfers occurred.

[Tommo_Aus]

Well after locking down the pools wallets to localhost it happened again, this time all of the other wallets were emptied so I guess its pretty much the end of my pool. Well done you bastard whoever you are.

At least this time I have the logs, there wasn't any SSH accessed gained, looks to be RPC to the wallet. Not really sure what debug data could be of use or where I can go from here, its useless continuing as it'll just keep happening until I figure out how its happening.

[fcmatt]

Well after locking down the pools wallets to localhost it happened again, this time all of the other wallets were emptied so I guess its pretty much the end of my pool. Well done you bastard whoever you are.

At least this time I have the logs, there wasn't any SSH accessed gained, looks to be RPC to the wallet. Not really sure what debug data could be of use or where I can go from here, its useless continuing as it'll just keep happening until I figure out how its happening.

What hosting provider are you with? Is it a dedicated server only can can get into via password or is their an isp control panel?

Share pieces of the logs and your config please.

[Tommo_Aus]

I'm with Versaweb, its a dedicated server so I'm the only one who accesses it. What sort of logging information should I post?

[fcmatt]

I'm with Versaweb, its a dedicated server so I'm the only one who accesses it. What sort of logging information should I post?

Well post your config file first. Remove or scramble your passwd but at least demonstrate how tough it was.

Then hopefully your debug.log has timestamps. Find out what time couns were stolen and paste a good 1000 line chunk of it with the withdraw in the middle that stole your coins.

Did every coin daemon have same passwd?

I assume running up to date linux with patches? What software were u running on pool? Home made?

[Tommo_Aus]

I'm running on Ubuntu 12.04.3 LTS, latest patches installed. BTC config file example with jumbled user and password:

rpcuser=blahblah
rpcpassword=P9xOA2ewIjgJaoA7RyWK6RJ8D6fnh8A5AEZvAheGLDbO
rpcallowip=localhost
rpcport=9170
port=9171
daemon=1
server=1
listen=1
noirc=0
maxconnections=30

I've copied 1000 lines from the debug log to http://tompool.org:81/btclog.out - the transaction takes place around line 420. This is the transaction from the wallet:

    {
        "account" : "Main",
        "address" : "1EEERRbx4v6TNxgHJNthgroKBQLhehgdRt",
        "category" : "send",
        "amount" : -0.52100000,
        "fee" : -0.00010000,
        "confirmations" : 38,
        "blockhash" : "000000000000000051d2e759c63a26e247f185ecb7926ed7a6624bc31c2a717b",
        "blockindex" : 156,
        "blocktime" : 1392660808,
        "txid" : "b64fc823455f24566a2de3827caf1f1080bf0e5d72ffa49ea19cf5e6dd289927",
        "time" : 1392660930,
        "timereceived" : 1392660930
    }

The pool runs MPOS behind the scenes but the front end is a custom site, however this BTC wallet has no link between MPOS or the website, its basically a holding place for daily BTC payouts from TomCoin. I use sendtoaddress not sendfrom so I know the transaction wasn't done by any of the software I've written for the pool.

The wallets use the same rpcpassword.

[lithod02]

Tommo, what open ports where to the world on that box?

[fcmatt]

Tommo, what open ports where to the world on that box?

And not only that you have forum software on the server. Can the uid running the webserver read bitcoins config file? What version of vanilla is that? Do you have a config file in www root that attacker could read to get passwd info? Or anywhere else on the server? Cause i am leaning towards that vector. Check www logs during timestamp of sendfrom.

[Tommo_Aus]

Most are open but secured with long passwords, I need the ports to be open to communicate with other TomPool servers.

No the uid can't read the bitcoin config file, they're under differen't users.

[fcmatt]

Most are open but secured with long passwords, I need the ports to be open to communicate with other TomPool servers.

No the uid can't read the bitcoin config file, they're under differen't users.

I edited post. More questions above.

[Tommo_Aus]

Its version 2.0.18.8 of vanilla, I don't have the root login details anywhere on the server. I'll start to work through the www logs, but its very very long and verbose, I'm not really sure what I'm looking for.

What I'm yet to understand is the RPC queries can only be done from local host, but there is not evidence to suggest someone else logged into the server to perform these queries.

[fcmatt]

Its version 2.0.18.8 of vanilla, I don't have the root login details anywhere on the server. I'll start to work through the www logs, but its very very long and verbose, I'm not really sure what I'm looking for.

What I'm yet to understand is the RPC queries can only be done from local host, but there is not evidence to suggest someone else logged into the server to perform these queries.

Imagine I somehow exploit vanilla forum to upload a php file. This php file allows me to run commands and get output as the www user (whatver it is, nobody, www, apache, etc..)

I then search for any file that contains sensitive info. I find your bitcoin password in a www config file. Or I find it in a /tmp folder, or a user dir with permits that allow read, etc...

I then make a command as www to do a rpc command that runs on the server itself as 127.0.0.1

But before all that I examine your website, have friends in the hacking community, and see this:

http://vanillaforums.org/discussion/25668/dec-2013-security-update-2-0-18-10-and-2-1b2

You are running

"<script src="/tompoolforum/js/library/jquery.gardenmorepager.js?v=2.0.18.8" type="text/javascript"></script></head>"

2.0.18.8 I imagine. word on the street is that you have a problem. the update checker in the forum. i checked. you still have that code in your forum. the new version removes it.

I could spend more time and figure it out.. as in the actual exploit.. but that is my best guess right now without having root on the server for an hour.