We thwarted a hacking attempt on Bitcoin Video Casino and learned a lot.

[BitcoinVideoPoker]

We thwarted a hacking attempt on Bitcoin Video Casino and learned a lot. We wanted to share our experience so that other site operators can be prepared.

On Monday Sep 9th, an attempt to hack into Bitcoin Video Casino was made. We initially noticed something was amiss because our server had become unreachable for several hours. When our server eventually came back online again, we noticed that the machine had been restarted. Our server has had a 100% uptime since we launched (~185 days ago) and this is the first time it was restarted.

We snooped around a bit and found an open login by a user on /dev/tty1, a local controlling terminal. That's very bad. All logins should only be from psuedo-terminals (ptyX). More looking around found a script hiding in /etc called "bitcoin.sh" and a crontab entry meant to constantly restart the script. The script was nothing but a phone home to this intruder's Amazon EC2 instance and wasn't very clever, but it did the job it was intended to. Unfortunately for the hacker, he left his private key on our servers.

We took all of the standard precautions when building our website: offline cold storage of most Bitcoins, closing down open ports, disabling password logins (ssh keys only!), but most importantly we decided to use full disk encryption, with the disk decryption key encrypted with GPG.

The downside to full disk encryption is that our site cannot come up automatically after a reboot. That's fine, really, we never reboot the machine anyway. So once the would-be hacker restarted the machine, he had no way to access our database, our hot wallet, or our game source code as they reside on the encrypted disk. The hacker couldn't even mount the encrypted disks, as the decryption key isn't stored on the server.

We had a long talk with our hosting provider on how this happened. Nobody should be able to log into our machine locally (the /dev/tty1 part). We thought that someone local to our hosting provider's data center was trying to break in, but as we pressed for answers, we learned that a KVM device was installed that allowed remote access into our machine one day prior to the hack. The evidence that the device was plugged in was present in the kernel log ('dmesg'). While the machine was rebooted, the hacker modified /etc/sudoers and changed an account password so that he could have access after the reboot. Later, auth.log would verify his activity.

So why did someone working at the datacenter install a KVM on our machine when we didn't request it? It turns out that our hosting provider's billing software was actually where the hack began. An order was fraudulently placed on our account to install the KVM through his hacking of HostBill, the software our hosting provider uses. It's a particularly popular piece of software, so it's likely other hosting providers could be vulnerable.

Anyway, the long story short is that we didn't lose any of our Bitcoins or our game software because of one smart move: full disk encryption. We highly recommend to all you developers out there to use full disk encryption if you have sensitive data to protect on remote servers. Full disk encryption doesn't help thwart hot attacks while the site is live, but it does help tremendously when hackers could possibly get physical control of your remote-hosted machine.

Note: We also posted this to Reddit, so all of the redditors on here can see the post at http://www.reddit.com/r/Bitcoin/comments/1nicl9/we_thwarted_a_hacking_attempt_on_bitcoin_video/

[Zaih]

Very well done on avoiding any losses. Most of that terminology goes over the top of my head, but it definitely sounds like you have some sound security. Hopefully this can be use to everyone else!

[masterluc]

I'll leave it here

http://threatpost.com/ipmi-protocol-bmc-vulnerabilities-expose-thousands-of-servers-to-attack

[casinobitco]

Nice work guys, We appreciate the insight and have similar protocols setup on our side, but can do better!

Threads like these help us all collaborate and our really beneficial for the entire Bitcoin Gaming community, instead of have cut-throat disagreement and showmen ship  with one another!

Cheers!

[ThomasDB]

Thanks for sharing this and great work!

[BobMarley]

wow way over my head

[saif313]

Thanks for sharing this and great work!

very helpful for all users of community and many can saved themselves in future attacks like this

[Hfleer]

We thwarted a hacking attempt on Bitcoin Video Casino and learned a lot. We wanted to share our experience so that other site operators can be prepared.

On Monday Sep 9th, an attempt to hack into Bitcoin Video Casino was made. We initially noticed something was amiss because our server had become unreachable for several hours. When our server eventually came back online again, we noticed that the machine had been restarted. Our server has had a 100% uptime since we launched (~185 days ago) and this is the first time it was restarted.

We snooped around a bit and found an open login by a user on /dev/tty1, a local controlling terminal. That's very bad. All logins should only be from psuedo-terminals (ptyX). More looking around found a script hiding in /etc called "bitcoin.sh" and a crontab entry meant to constantly restart the script. The script was nothing but a phone home to this intruder's Amazon EC2 instance and wasn't very clever, but it did the job it was intended to. Unfortunately for the hacker, he left his private key on our servers.

We took all of the standard precautions when building our website: offline cold storage of most Bitcoins, closing down open ports, disabling password logins (ssh keys only!), but most importantly we decided to use full disk encryption, with the disk decryption key encrypted with GPG.

The downside to full disk encryption is that our site cannot come up automatically after a reboot. That's fine, really, we never reboot the machine anyway. So once the would-be hacker restarted the machine, he had no way to access our database, our hot wallet, or our game source code as they reside on the encrypted disk. The hacker couldn't even mount the encrypted disks, as the decryption key isn't stored on the server.

We had a long talk with our hosting provider on how this happened. Nobody should be able to log into our machine locally (the /dev/tty1 part). We thought that someone local to our hosting provider's data center was trying to break in, but as we pressed for answers, we learned that a KVM device was installed that allowed remote access into our machine one day prior to the hack. The evidence that the device was plugged in was present in the kernel log ('dmesg'). While the machine was rebooted, the hacker modified /etc/sudoers and changed an account password so that he could have access after the reboot. Later, auth.log would verify his activity.

So why did someone working at the datacenter install a KVM on our machine when we didn't request it? It turns out that our hosting provider's billing software was actually where the hack began. An order was fraudulently placed on our account to install the KVM through his hacking of HostBill, the software our hosting provider uses. It's a particularly popular piece of software, so it's likely other hosting providers could be vulnerable.

Anyway, the long story short is that we didn't lose any of our Bitcoins or our game software because of one smart move: full disk encryption. We highly recommend to all you developers out there to use full disk encryption if you have sensitive data to protect on remote servers. Full disk encryption doesn't help thwart hot attacks while the site is live, but it does help tremendously when hackers could possibly get physical control of your remote-hosted machine.

Note: We also posted this to Reddit, so all of the redditors on here can see the post at http://www.reddit.com/r/Bitcoin/comments/1nicl9/we_thwarted_a_hacking_attempt_on_bitcoin_video/

Wow, an interesting read there.  Good thinking on your part with the security measures in place, think we've seen enough BTC hacking.

[Herbert]

Thanks for sharing that experience! And grats for keeping your coins safe

[monbux]

Congratz on the save.
Do you have any suspicions of who did this?

[markjamrobin]

Nice work guys, We appreciate the insight and have similar protocols setup on our side, but can do better!

Threads like these help us all collaborate and our really beneficial for the entire Bitcoin Gaming community, instead of have cut-throat disagreement and showmen ship  with one another!

Cheers!

*Cough* Betcoin.tm Debacale *Cough*

[deepceleron]

We thwarted a hacking attempt on Bitcoin Video Casino and learned a lot. We wanted to share our experience so that other site operators can be prepared.

On Monday Sep 9th, an attempt to hack into Bitcoin Video Casino was made. We initially noticed something was amiss because our server had become unreachable for several hours. When our server eventually came back online again, we noticed that the machine had been restarted. Our server has had a 100% uptime since we launched (~185 days ago) and this is the first time it was restarted.

We snooped around a bit and found an open login by a user on /dev/tty1, a local controlling terminal. That's very bad. All logins should only be from psuedo-terminals (ptyX). More looking around found a script hiding in /etc called "bitcoin.sh" and a crontab entry meant to constantly restart the script. The script was nothing but a phone home to this intruder's Amazon EC2 instance and wasn't very clever, but it did the job it was intended to. Unfortunately for the hacker, he left his private key on our servers.

We took all of the standard precautions when building our website: offline cold storage of most Bitcoins, closing down open ports, disabling password logins (ssh keys only!), but most importantly we decided to use full disk encryption, with the disk decryption key encrypted with GPG.

The downside to full disk encryption is that our site cannot come up automatically after a reboot. That's fine, really, we never reboot the machine anyway. So once the would-be hacker restarted the machine, he had no way to access our database, our hot wallet, or our game source code as they reside on the encrypted disk. The hacker couldn't even mount the encrypted disks, as the decryption key isn't stored on the server.

We had a long talk with our hosting provider on how this happened. Nobody should be able to log into our machine locally (the /dev/tty1 part). We thought that someone local to our hosting provider's data center was trying to break in, but as we pressed for answers, we learned that a KVM device was installed that allowed remote access into our machine one day prior to the hack. The evidence that the device was plugged in was present in the kernel log ('dmesg'). While the machine was rebooted, the hacker modified /etc/sudoers and changed an account password so that he could have access after the reboot. Later, auth.log would verify his activity.

So why did someone working at the datacenter install a KVM on our machine when we didn't request it? It turns out that our hosting provider's billing software was actually where the hack began. An order was fraudulently placed on our account to install the KVM through his hacking of HostBill, the software our hosting provider uses. It's a particularly popular piece of software, so it's likely other hosting providers could be vulnerable.

Anyway, the long story short is that we didn't lose any of our Bitcoins or our game software because of one smart move: full disk encryption. We highly recommend to all you developers out there to use full disk encryption if you have sensitive data to protect on remote servers. Full disk encryption doesn't help thwart hot attacks while the site is live, but it does help tremendously when hackers could possibly get physical control of your remote-hosted machine.

Note: We also posted this to Reddit, so all of the redditors on here can see the post at http://www.reddit.com/r/Bitcoin/comments/1nicl9/we_thwarted_a_hacking_attempt_on_bitcoin_video/

Let me rearrange this so it is readable:
1. Hosting service uses software package "HostBill"
2. Hacker hacks that software and places an order for a remote access device to be installed.
3. A remote access device installed by ISP allows hacker to see the machine console and reboot
4. rebooting allows single-user mode, where files can be edited and a new user account or password inserted
5. hacker now has root, but the gaming platform is on encrypted media without a password the hacker knows.

You are lucky that the hacker couldn't think of anything interesting to do; however that machine is not 100% secure unless it can be image-restored or reloaded. An intrusion detection system would have alerted to any system changes or the downtime. The hacker's goal may not have been to steal Bitcoins, it may have been to discover the site owner's identity or that of players or to log credentials.

[knowitnothing]

however that machine is not 100% secure unless it can be image-restored or reloaded.

This needs to be emphasized. Your server has been compromised, plus your ISP has been compromised. You cannot assume that all is good, or that all will be good ever again with this combo machine + ISP.

Also, while I recommend people going for disk encryption too, please do not forget to make proper backups and do not forget to test for restoration plans.

[BitcoinVideoPoker]

Thanks everyone for your kind words. We hope that this has been helpful!

What was the hosting company?

It’s one of the few hosting companies out there that accepts Bitcoins. We are trying to not put a negative spin on this post by passing blame around. We only want other developers to be aware of what recently happened to us, so they can better protect themselves.

With that being said, if anyone has recommendations for fast and reliable hosting companies that take Bitcoin, please let us know!

Congratz on the save.
Do you have any suspicions of who did this?

No idea who did this. Probably just some random guy that thought he could steal some Bitcoins.

Let me rearrange this so it is readable:
1. Hosting service uses software package "HostBill"
2. Hacker hacks that software and places an order for a remote access device to be installed.
3. A remote access device installed by ISP allows hacker to see the machine console and reboot
4. rebooting allows single-user mode, where files can be edited and a new user account or password inserted
5. hacker now has root, but the gaming platform is on encrypted media without a password the hacker knows.

You are lucky that the hacker couldn't think of anything interesting to do; however that machine is not 100% secure unless it can be image-restored or reloaded. An intrusion detection system would have alerted to any system changes or the downtime.

Thanks for the summary. Yes that is how it went down.

After we collected evidence, we trashed everything on the server and started fresh.

Do you have any recommendations for such an intrusion detection system? Anything that we can add to our servers to make them more secure would be great.

[SpaceJelly]

If it's your own hardware that's co-located and if you're sure that you'll never need access locally then simply fill all the ports with epoxy glue. That'll stop any KVM mischief again!

If you do need access in the future, then most motherboards have an unused USB header, so just open the case and stick one of the USB expanders on there so you can plug a keyboard in.

however, this does destroy the resale value of the server but if you're like me I run em till they're too slow then scrap em!

[Mooshire]

If it's your own hardware that's co-located and if you're sure that you'll never need access locally then simply fill all the ports with epoxy glue. That'll stop any KVM mischief again!

If you do need access in the future, then most motherboards have an unused USB header, so just open the case and stick one of the USB expanders on there so you can plug a keyboard in.

however, this does destroy the resale value of the server but if you're like me I run em till they're too slow then scrap em!

But this is the internet, and what is this "local" you speak of. Everything goes through the Internet.

[knowitnothing]

If it's your own hardware that's co-located and if you're sure that you'll never need access locally then simply fill all the ports with epoxy glue. That'll stop any KVM mischief again!

If you do need access in the future, then most motherboards have an unused USB header, so just open the case and stick one of the USB expanders on there so you can plug a keyboard in.

however, this does destroy the resale value of the server but if you're like me I run em till they're too slow then scrap em!

But this is the internet, and what is this "local" you speak of. Everything goes through the Internet.

locally -> Physical access, in person, does not go through the Internet.

[Mooshire]

If it's your own hardware that's co-located and if you're sure that you'll never need access locally then simply fill all the ports with epoxy glue. That'll stop any KVM mischief again!

If you do need access in the future, then most motherboards have an unused USB header, so just open the case and stick one of the USB expanders on there so you can plug a keyboard in.

however, this does destroy the resale value of the server but if you're like me I run em till they're too slow then scrap em!

But this is the internet, and what is this "local" you speak of. Everything goes through the Internet.

locally -> Physical access, in person, does not go through the Internet.

What is this "in person" you speak of? I do not exist in the physical realm, only in the Internet.

[EzCheese]

Thanks everyone for your kind words. We hope that this has been helpful!

What was the hosting company?

It’s one of the few hosting companies out there that accepts Bitcoins. We are trying to not put a negative spin on this post by passing blame around. We only want other developers to be aware of what recently happened to us, so they can better protect themselves.

With that being said, if anyone has recommendations for fast and reliable hosting companies that take Bitcoin, please let us know!

Congratz on the save.
Do you have any suspicions of who did this?

No idea who did this. Probably just some random guy that thought he could steal some Bitcoins.

Let me rearrange this so it is readable:
1. Hosting service uses software package "HostBill"
2. Hacker hacks that software and places an order for a remote access device to be installed.
3. A remote access device installed by ISP allows hacker to see the machine console and reboot
4. rebooting allows single-user mode, where files can be edited and a new user account or password inserted
5. hacker now has root, but the gaming platform is on encrypted media without a password the hacker knows.

You are lucky that the hacker couldn't think of anything interesting to do; however that machine is not 100% secure unless it can be image-restored or reloaded. An intrusion detection system would have alerted to any system changes or the downtime.

Thanks for the summary. Yes that is how it went down.

After we collected evidence, we trashed everything on the server and started fresh.

Do you have any recommendations for such an intrusion detection system? Anything that we can add to our servers to make them more secure would be great.

Something sounds very fishy here and it sounds like your hosting site has some major security flaws.

1. How large is this datacenter, is it some mom and pop shop or is it a larger company?  I ask this because when I walk into my datacenter I would have NO idea what each server in there does.  Unless they have a big sign on your cage (next question), then how would this guy know you're a bitcoin casino?  He would have to bypass the firewall, then the IPS to find out your servers real IP on the network and physical location

A)  The hosting company is crooked and this is common there
B)  An employee has gone rouge and has access to all the local infrastructure documentation, which would pin point DNS name/IP to your server location
C)  The rouge employee has an outside man who he sells info too or is a cohort in the operation. 

If it's a mom and pop shop I would do a security check and see what information they have on your server rack or cage. 

2.  Physical security: are your servers inside a lockable cage or do they have open physical access?  What kind of KVM was installed: wired/wireless?  If the servers are open, you need to pay a few more bitcoins to have it inside a cage or lockable rack.  If the KV was wired did the wire lead too?  If it was wireless, who has access and where can they access it?  Based on the last question, is there video surveillance so you can see who was accessing the kvm/equipment?

A)  Security physical lockable cage/rack if not done already
B)  Do a security sweep and see if there are any physical weaknesses (holes, opening, etc..)
C)  Verify video if they have any -- Verify kvm used and where it was used at
D)  One of your own admins or someone who you know or admins know that has access to your physical location of the servers gave up knowledge. 

Recommendations: 

1.  I would change hosting sites if all possible.  Yeah they accept bitcoins, but what would of happened if they took all your coins?  Just depends on what you're comfortable with. 

2.  Install an IP camera on your server/rack that takes video or pictures 24 hours a day.  This will help ward off any physical access and also give you video evidence in case they do not have it. 

3.  Based on their remote hands/ticketing system, I would put a complete hold on all orders unless they have verbal communication from your or any of your trusted admins.  If they do not allow this then LEAVE ASAP.   

4.  I would also look at your own admins and who ever knows any inside knowledge of your physical servers/passwords/logins. 

I hope this helps, PM me if you have any more questions, this was a quick response.   

[BitcoinVideoPoker]

EzCheese thanks for the thorough response and recommendations. Much appreciated!