Bitcoin Forum

I logged into my server this morning but it was running slowly, so I thought I'd restart it and see if that helped, which it didn't. Turned out to be some connectivity issues as far as I could see, the server would be available for 40 seconds, then drop out for 20 seconds and be back up again. It wasn't a DDOS as looking at the 24 hours server statistics from my web host there wasn't any spike in network traffic.

These issues eventually subsided so I went to process the daily TomCoin payouts where I transfer BTC from other sources to the one wallet and make payments according to share contributions (its a BTC payout multipool). The pool has a 0.01 BTC minimum payout meaning the wallet has some residual funds that carry over from previous days, checking the incoming BTC transactions for the payouts I noticed the wallet was lower on funds than I'd expect. I checked the transactions and noticed these two:

{
        "address" : "1EEERRbx4v6TNxgHJNthgroKBQLhehgdRt",
        "category" : "send",
        "amount" : -2.00000000,
        "fee" : 0.00000000,
        "confirmations" : 31,
        "blockhash" : "0000000000000000cf924f2bf8543fd4448b741be87c3faaa769dbf92d95d37b",
        "blockindex" : 34,
        "blocktime" : 1392593790,
        "txid" : "23ad0f3424c038b00f8b4113edf8b9d2725a38b20f2b63ba05e84359e5ae7262",
        "time" : 1392592307,
        "timereceived" : 1392592307
    },
    {
        "address" : "1EEERRbx4v6TNxgHJNthgroKBQLhehgdRt",
        "category" : "send",
        "amount" : -0.83000000,
        "fee" : -0.00010000,
        "confirmations" : 31,
        "blockhash" : "0000000000000000cf924f2bf8543fd4448b741be87c3faaa769dbf92d95d37b",
        "blockindex" : 298,
        "blocktime" : 1392593790,
        "txid" : "c673db0fe09107b9ef3239571fbd5718fdc38691ff4badeb1b4d52fbc31a08fb",
        "time" : 1392592452,
        "timereceived" : 1392592452
    }

They occurred perhaps half an hour before I restarted the server, no payout jobs were running at this time and I didn't perform any manual transfers during this time. This address has never mined with my pool.

I've been able to find the txid's in the blockchain explorer, strangely the 2 BTC transaction above doesn't match the blockchain explorer, instead its listed as 2.39021875 BTC.

I've looked through the server logs and the only successful logins are from myself, although yet again I have various failed login attempts from Chinese IP addresses. Unfortunately I can't get anything useful from the BTC wallets debug.log as it starts fresh each time the wallet starts, and seeing as I restarted the server I had to restart the wallet.

I'm guessing that's it for the BTC, I accept there probably isn't any chance of recovering it as the transactions can't be rolled back, but what I haven't figured out yet is how it happened. All I can think of so far is someone cracked a random 45 character wallet password but the probability is so low it shouldn't even be a possibility, and in any case a lot more of us would be in trouble if password that long are being quickly cracked.

Is there a way I can track the funds to see where they're used or find out more about the transactions? I can't see where I should go from here if anywhere, and I'm worried it could happen again. Just another fun part of being a pool operator I guess.

Worth ruling out the obvious--did anyone have access to your machine while you were asleep like an ex-gf or ex-roommate.

I too have about 1.5 btc in an online wallet so I'm very interested in my own btcoinage.

Please keep us updated. I've subbed to this thread.

Nope, I'm the only one with access to the server. I've been through the access logs and each entry matches when I've logged in so I'm pretty sure I can rule that out.

I'd like to be able to say it was possible because of xxxxx and now I've closed that hole, but I still don't know how it happened.

The people who work in the hosting company?

Its always a possibility but seems pretty far fetched that someone in the hosting company would do it, word would travel pretty fast if that was the case and they wouldn't be in business for long.

Do you allow rpc access to bitcoin from all remote ips or just trusted subnets like localhost and your workstation?

I used to allow all but after today I've changed it to localhost, its a pain as I remote to the wallet from a number of different machines, some with dynamic IP's so I guess I'll need to work out another way to go about my business. I would've thought such a long password would be secure, this particular wallets only been in service for a couple of weeks.

If the password was indeed hacked, I guess by brute force, it seems pretty incredible it was done within such a sort period of time given similar passwords I've tried in password calculators estimate over 100 years to break the code. Not to mention I'd surely notice a huge spike in network activity if such an attempt was made.

i know this might be tinfoil hat territory but is the server hosted some place where an employee or someone else can sniff packets?

Otherwise how does one explain the high load except by brute forcing? and brute forcing might not use as much bandwidth as you think.. but it would create really high load on the server.

too bad the logs are gone.. that would be the best hint.

Yeah I wish the logs were still there, or that logs were appended to after a wallet restart rather than a new file being created.

The servers at one of your standard web hosts, so I guess it'd be as susceptible as any other dedicated server hosted in a DC. I did a brief top to check processes before I restarted, I didn't notice anything out of the ordinary, this would've been before the transfers occurred.

Well after locking down the pools wallets to localhost it happened again, this time all of the other wallets were emptied so I guess its pretty much the end of my pool. Well done you bastard whoever you are.

At least this time I have the logs, there wasn't any SSH accessed gained, looks to be RPC to the wallet. Not really sure what debug data could be of use or where I can go from here, its useless continuing as it'll just keep happening until I figure out how its happening.

What hosting provider are you with? Is it a dedicated server only can can get into via password or is their an isp control panel?

Share pieces of the logs and your config please.

I'm with Versaweb, its a dedicated server so I'm the only one who accesses it. What sort of logging information should I post?

Well post your config file first. Remove or scramble your passwd but at least demonstrate how tough it was.

Then hopefully your debug.log has timestamps. Find out what time couns were stolen and paste a good 1000 line chunk of it with the withdraw in the middle that stole your coins.

Did every coin daemon have same passwd?

I assume running up to date linux with patches? What software were u running on pool? Home made?

I'm running on Ubuntu 12.04.3 LTS, latest patches installed. BTC config file example with jumbled user and password:

rpcuser=blahblah
rpcpassword=P9xOA2ewIjgJaoA7RyWK6RJ8D6fnh8A5AEZvAheGLDbO
rpcallowip=localhost
rpcport=9170
port=9171
daemon=1
server=1
listen=1
noirc=0
maxconnections=30

I've copied 1000 lines from the debug log to http://tompool.org:81/btclog.out - the transaction takes place around line 420. This is the transaction from the wallet:

    {
        "account" : "Main",
        "address" : "1EEERRbx4v6TNxgHJNthgroKBQLhehgdRt",
        "category" : "send",
        "amount" : -0.52100000,
        "fee" : -0.00010000,
        "confirmations" : 38,
        "blockhash" : "000000000000000051d2e759c63a26e247f185ecb7926ed7a6624bc31c2a717b",
        "blockindex" : 156,
        "blocktime" : 1392660808,
        "txid" : "b64fc823455f24566a2de3827caf1f1080bf0e5d72ffa49ea19cf5e6dd289927",
        "time" : 1392660930,
        "timereceived" : 1392660930
    }

The pool runs MPOS behind the scenes but the front end is a custom site, however this BTC wallet has no link between MPOS or the website, its basically a holding place for daily BTC payouts from TomCoin. I use sendtoaddress not sendfrom so I know the transaction wasn't done by any of the software I've written for the pool.

The wallets use the same rpcpassword.

Tommo, what open ports where to the world on that box?

And not only that you have forum software on the server. Can the uid running the webserver read bitcoins config file? What version of vanilla is that? Do you have a config file in www root that attacker could read to get passwd info? Or anywhere else on the server? Cause i am leaning towards that vector. Check www logs during timestamp of sendfrom.

Most are open but secured with long passwords, I need the ports to be open to communicate with other TomPool servers.

No the uid can't read the bitcoin config file, they're under differen't users.

I edited post. More questions above.

Its version 2.0.18.8 of vanilla, I don't have the root login details anywhere on the server. I'll start to work through the www logs, but its very very long and verbose, I'm not really sure what I'm looking for.

What I'm yet to understand is the RPC queries can only be done from local host, but there is not evidence to suggest someone else logged into the server to perform these queries.

Imagine I somehow exploit vanilla forum to upload a php file. This php file allows me to run commands and get output as the www user (whatver it is, nobody, www, apache, etc..)

I then search for any file that contains sensitive info. I find your bitcoin password in a www config file. Or I find it in a /tmp folder, or a user dir with permits that allow read, etc...

I then make a command as www to do a rpc command that runs on the server itself as 127.0.0.1

But before all that I examine your website, have friends in the hacking community, and see this:

http://vanillaforums.org/discussion/25668/dec-2013-security-update-2-0-18-10-and-2-1b2

You are running

"<script src="/tompoolforum/js/library/jquery.gardenmorepager.js?v=2.0.18.8" type="text/javascript"></script></head>"

2.0.18.8 I imagine. word on the street is that you have a problem. the update checker in the forum. i checked. you still have that code in your forum. the new version removes it.

I could spend more time and figure it out.. as in the actual exploit.. but that is my best guess right now without having root on the server for an hour.

been speaking to tommo on tompool.org and ive thought of one possible way that the coins were taken.

The server has a mirrored raid setup so it would be easy for the hosting provide to take out the second set of HDDs and set them put in a secondary system and make the commands to send the coins. It seems very simple but it could have been done.

the server would have logs in /var/log/messages saying the server noticed HDs removed.
you cannot yank hard drives without the server knowing. well in most cases i know of. raid controllers are pretty
verbose when it comes to drives just being pulled.

Lets say that's it, would there be anything I could check to confirm this happened?

show us www logs right around the rpc call was made. do you see any POSTS or strange files being called by the webserver?

give us 100 lines before and after the time the rpc call was made.

and i have to admit the faster you feed us info the better.

true.. true.. but the company might be dodgie... setting up things so if they did want to do anything they could.

I'll grab it ASAP, unfortunately some real life bits and pieces have to be attended to before I can grab this info.

i admit the idea has merit but i have not encountered a single linux box in my career that did not complain loudly in the logs that disks were removed. raid controller bios has no option to disable that stuff. that would mean a custom linux distro was installed to disable it? doubtful.

but going through message files looking for the disks being removed is pretty simple.. depending on how far back you wish to look. log rotation might only allow several days to look back.

what would concern me more is some automated backup process they offer that is automagically installed on the server for the customer or he is using iscsi from the server to a large storage box. THEN the isp could view the backups of that or even connect to the storage box themselves.

I've been looking at the www logs, this gives me a suspicious feeling unless its a complete coincidence...

BTC stolen at these times (in PST, which is the servers timezone):

Feb 16 2014 15:11:47
Feb 16 2014 15:14:12
Feb 17 2014 10:15:30

Apache logs:

94.231.83.139 - - [16/Feb/2014:15:11:46 -0800] "POST /tompoolforum/cache/Smarty/2/send.php HTTP/1.1" 200 1711 "http://tompool.org:81/tompoolforum/cache/Smarty/2/index.php" "Mozilla/5.0 (Windows NT 5.1; rv:27.0) Gecko/20100101 Firefox/27.0"
94.231.83.139 - - [16/Feb/2014:15:14:11 -0800] "POST /tompoolforum/cache/Smarty/2/send.php HTTP/1.1" 200 1714 "http://tompool.org:81/tompoolforum/cache/Smarty/2/index.php" "Mozilla/5.0 (Windows NT 5.1; rv:27.0) Gecko/20100101 Firefox/27.0"
77.109.138.42 - - [17/Feb/2014:10:15:30 -0800] "POST /tompoolforum/cache/Smarty/2/send.php HTTP/1.1" 200 1714 "http://tompool.org:81/tompoolforum/cache/Smarty/2/index.php" "Mozilla/5.0 (Windows NT 6.1; rv:10.0) Gecko/20100101 Firefox/10.0"

Both IP's are located in Switzerland according to a Google search, the Smarty/2/send.php doesn't exist in the folder anymore. Am I onto something?

Yes. Disable forum software. It has too many holes to run on the pool server.

Find out what file they read to get rpc passwd and username. They had to.

There seems to be a common flow of requests, it goes:

77.109.138.42 - - [17/Feb/2014:10:11:20 -0800] "GET /tompoolforum/cache/Smarty/2/index.php HTTP/1.1" 200 3170 "http://tompool.org:81/tompoolforum/cache/Smarty/2/send.php" "Mozilla/5.0 (Windows NT 6.1; rv:10.0) Gecko/20100101 Firefox/10.0"

... then moments later ...

77.109.138.42 - - [17/Feb/2014:10:11:25 -0800] "POST /tompoolforum/cache/Smarty/2/send.php HTTP/1.1" 200 1952 "http://tompool.org:81/tompoolforum/cache/Smarty/2/index.php" "Mozilla/5.0 (Windows NT 6.1; rv:10.0) Gecko/20100101 Firefox/10.0"

Problem being these files don't exist anymore so I don't know what's in them.

Interesting, I've also found a call to the file tompoolforum/library/lib.php, but I don't see this file in the vanilla forums project git. Viewing the file it looks like this:

<?php
$o="QAEAOzh3b3cKDQAjYnV1aHVYdWIAAHdodXNuaWAvMC48Cg1HdGIAAHNYamZgbmRYdnJoc2J0WHUAgHJ pc25qYi83AfFoZVh0c2Z1JABzLwDRI2oBkSc6J2J/d2toY2IAEy8gJ ... (about 40 thousand characters more) ... RsbGxsbGxsbGwpOw=="));return;?>

If I load the page the source looks like this:

<style type="text/css">
   input {font:11px Verdana;BACKGROUND: #FFFFFF;height: 18px;border: 1px solid #666666;}
</style>
<form method="POST" action="">
   <span style="font:11px Verdana;">Password: </span><input name="password" type="password" size="20">
   <input type="hidden" name="doing" value="login">
   <input type="submit" value="Login">
</form>

It renders as a password input field with a Login button.

that is not surprising that the files are gone. also the IP address is a tor exit node. helping people keep their privacy and assisting hackers all day long. essentially the hacker can somehow upload a file to your cache directory which probably has permissions allowing write to the webserver. they they post some info to it to call the rpc command.

i think you found the problem. to learn anymore would probably require i to access your server but i do not think either of us want that. your best bet is to completely remove the forum software and if you want forum software again run it on a totally different server.. a throw away box.

that is probably a base64 encrypted php chunk/file that the attacker used to run commands on your server.
they do a POST to it logging in and then running a command. i have seen them before. quite common.
seeing that is contains that much data means it could be a whole webpage of commands for the script kid to run.

feel free to post the whole thing here. lets convert it to ascii

Yep I'm quite convinced the forum was the weakness, I've taken the forum out so that'll be the back door gone at least, unless they built something else in while they were there.

Now I need to work out if I get back up again or call it a day.

well they could only write to places where www could write.. and only run commands from the www dirs... so that limits what they could have done. but keep in mind that some are clever. they could have run a script that starts a process that listens on a high port.. then deleted the file.. it would stick around until a reboot.

it is prob in your best interest to move the www dir to a backup place in /blah. then move over file by file of things you trust that were not able to be written to by www.. and rebuild the www back.

Dunno. You were evolving your pool really well though. I was impressed. So sorry to hear about this!

Wow this sucks!  Have only been mining with Tom for a few days and was happy to be working with an Aus local pool. Oh well if you come back Tom I will point my miners back to you pool for sure, it was just so convenient mining and getting paid in btc each day.

I highly appreciate the work you have done on the pool and I would be eager to see it develop in future. Is there any way to send you some alt coins as donation. I am sure they many would support this idea so maybe if we all pull together we can make up for the loss, you take a few days off to clear your head and start afresh?

We love tompool!

Just to say I literally just started mining on your pool in the last day or two and I am very sorry to hear what happened.

As far as I can see you are one of the very few pools offering SHA profitability switching and payouts in BTC, so I would love to see the pool back online.

Don't let the bastards win.

I have been mining off and on for a while and really liked the interface and can tell you put a lot of work into it.  Don't let those bastards win.  I for one have faith that you are telling the truth and will defend Tom Pool's honor if anyone posts any negative bullshit on this or any other forum.  Just Zero everyone's account out and start over.  I'm down with sending you some coin as a donation to help you rebuild.  Just let the members know what we can do to help. 

Mod_Security would most likely have prevented files from being uploaded and executed, which it sounds like happened here.

:(

Perhaps you could reach an arrangement with somebody like fcmatt, to do some private security consulting for you.  At the very least it would be a second layer, a second set of experienced eyes keeping watch on that side of your site.  I always feared right from the start the possibility of thieves coming against you as you became higher profile.

Sorry for your troubles Tommo - I'd be happy for a zeroed account and a fresh start!

I am a user of your pool. This is very unfortunate.  I have also been following the hack of the TomCoin portion for the past few days.  Please copy all of suspicious PHP source and make a pastebin.  I should be able to decrypt it and give you more information regarding the attack on your pool.

Thanks and Best of Luck!

What would be a good place to upload it? I could put it in a zip file.

I will be happy to donate my miners on your pool for a day to help you build up your pool again. 38gh for a day. Let start a donation to Tom pool! Any else would like to make a miner donation to Tom pool?

Heh many thanks but no need, I'll just get things back up and more secure then hope some miners return :)

I've removed the forum software, changed passwords, reviewed files in the web directories and blocked a heap of ports (about the only ones open now are for the web server, mining (stratum) and SSH access).

TomCoin is now up, should have the other pools and multipools up soon. I'm just taking this as an opportunity to load balance between the servers as running so many wallets on my server with lower disk I/O is hurting its performance.

tommo, i am glad to see you up and running again.

perhaps i can message you in the future if i have any curious questions about your experiences running a multicoin pool?

Good news.  My miner switched back automatically when the pool came back online.

Thanks guys, I've just brought the individual pools and multipools back online, hopefully some smooth sailing from here!

Don't hesitate to contact me fcmatt, thanks for all your help identifying the point of entry you were spot on, I'd be glad to return the favour :)

http://www.zippyshare.com/ would work nicely. They don't have any ads or compulsory registration.

Sorry to hear that. I wonder cause hacking is really a big problem - what is the best way to protect from hacks?

Tom Pool is up and running again!!!!!

Have happily kept 30GH/s on tompool the entire time. Thankfully I am auto-withdrawing altcoins to cryptsy and not utilizing TomCoin.  Really glad to see the project wasn't closed due to the hack.

I've uploaded the lib.php file to here:

http://www8.zippyshare.com/v/82893458/file.html

I'd greatly appreciate if someone could take a look at it just to make sure there isn't anything else I should worry about.

Checked my BTC wallet today and found another transaction, luckily it wasn't significant as everything else had been emptied out earlier... little shit ('scuse my French) going back for more. I did a file contents search and found another instance of it in my web dir under a different name. I'd moved all of the other wallets to another server earlier on so access was only available to the one wallet. I've since deleted this last instance of the craplication.

You really need to remove everything from www dir and put back only what you know. Or list all files looking for newer dates, file creation times, files created by www instead of root or your normal user you edit files with,etc They tend to leave multiples of these backdoors for this exact reason.  i failed to mention this to you but i did say start www from scratch or review every file.

Yeah I checked through the files and creation dates, somehow I missed this one.

Even tho I'm late...

A good file sharing site is:  mega.co.nz

Security suggestion: Implement 2 Factor Auth (now that the backdoor is known and gone) and mod_security to prevent hackers uploading files if they find a new backdoor.

Just my 2c worth...
Any money the hacker is watching this thread and found out about your pool from this forum.

Paranoid much? :)

Sucks to hear you got hacked tommo, good to hear you are persisting and this has been a worthwhile learning experience. At least only 10k or so of coins were stolen so there is no real need for clients to come after you. You don't need legal or financial problems to be added to your grief.

Been a while since this happened but does this mean the coins are still sitting in the same address?

https://blockchain.info/address/1EEERRbx4v6TNxgHJNthgroKBQLhehgdRt

Really sucks seeing the coins there in limbo in someone else's account.

Looks that way to me.

Is anyone good at tracing BTC transactions? Its been a while but I thought I'd revisit this and I've seen the BTC moved in January:

https://blockchain.info/address/1EEERRbx4v6TNxgHJNthgroKBQLhehgdRt

Wouldn't mind knowing where it went, if its gone to an exchange there might be something to trace.

The coins eventually landed up here https://blockchain.info/address/1FsVcdeHbpvUVT3gjeuVR2ZSDnpcsJMsLL
If you Google for the address, you will get several links. It seems to be an exchange address.

Have you figured out the source of the hack? Was is definitely the forum software?

How what use that would be? If he ran the coins through a exchange, it becomes as difficult as trying to guess coins after being ran by a mixer.. it's an headache trying to figure out who owns what anymore.
I would say OP downloaded some dodgy wallet from some altcoin.

 

yeah best way to keep a wallet safe is dedicate a pc to the blockchain and nothing else.

I use an old mobo/cpu from gpu mining back in 2012 .  I run my wallet on that and nothing else.