Linode compromised and it's effect on TradeHill

[Jered Kenna (TradeHill)]

From tradehillblog.com


Dear Clients,

Yesterday, March 1st, one of our hosting providers was compromised. Linode.com contacted us several hours ago via email stating:



" As a valued customer, the security of your account is our top priority.  Today we suffered from an unauthorized access of our system that resulted in eight customer accounts being compromised. Unfortunately, your account was one of the accounts targeted.

You should begin a compromised system recovery procedure immediately -- we recommend a complete redeployment -- on your  Linodes. "




It appears that hackers were targeting accounts owned by Bitcoin businesses.

We built TradeHill with security as a priority and make use of multiple data centers to protect core infrastructure. Due to our system architecture design it appears that no user data was compromised, and all wallet files are safe, however we are in the process of a more thourough audit and will provide updates when more information is available.

We are taking TradeHill offline at this point as a precaution and appreciate your patience.

Updates can be found  here and on twitter via @tradehill and @jeredkenna

Regards,

Jered Kenna
Chief Executive Officer
TradeHill

[1714988730]

1714988730

[deepceleron]

It would seem the attacker used backdoor administration access that was not logged (and not publicized as being present) to reset shadow passwords and gain access. They were quite quick in withdrawing funds from slush mining pool, bitcoin faucet, and bitcoinica, but I wouldn't rule out any kind of compromise or future wallet emptying, as it seems that many think this came from inside Linode themselves using tools only their personnel would have access to on any reasonably administered system. It would be wise to go as far as considering your entire VPS file system cloned and logged into with root access, then think of what the intruder might do with the data.

[Jered Kenna (TradeHill)]

It would seem the attacker used backdoor administration access that was not logged (and not publicized as being present) to reset shadow passwords and gain access. They were quite quick in withdrawing funds from slush mining pool, bitcoin faucet, and bitcoinica, but I wouldn't rule out any kind of compromise or future wallet emptying, as it seems that many think this came from inside Linode themselves using tools only their personnel would have access to on any reasonably administered system. It would be wise to go as far as considering your entire VPS file system cloned and logged into with root access, then think of what the intruder might do with the data.

I hesitate to speak too soon and we're going through it right now but it looks fine. The idea was that Linode could be compromised without compromising TradeHill and without compromising the servers where everything happens. We built the exchange to protect against things like this and the engineer that laid it out did fine job. I wasn't enjoying paying the extra data centers every month but I certainly don't regret the decision now.

Linode is done though, they were great to work with and we'll see what happens.

Jered

[zby]

It is reassuring to see you posting here - is TradeHill customer service still working?

[Jered Kenna (TradeHill)]

It is reassuring to see you posting here - is TradeHill customer service still working?

Thanks. Yes customer service is still working we're just swamped trying to get all the funds out and deal with a lot of individual requests.
When the banks simultaneously closed our accounts it created a huge backlog. We'll get to the emails as fast as we can. Almost everything has been processed at this point and is under control. We'll have the Linode situation sorted soon as well.

-Jered

[Etlase2]

Good work, TradeHill. In the interest of protecting from future events like this, would you disclose what you did that made your accounts immune to this attack?

[Technomage]

This is 10 points for Tradehill. I hope you launch Bitcoin.com soon, I'm interested in that site. You've earned the trust of the community and we're excited for what's coming.

[zby]

It is reassuring to see you posting here - is TradeHill customer service still working?

Thanks. Yes customer service is still working we're just swamped trying to get all the funds out and deal with a lot of individual requests.
When the banks simultaneously closed our accounts it created a huge backlog. We'll get to the emails as fast as we can. Almost everything has been processed at this point and is under control. We'll have the Linode situation sorted soon as well.

-Jered

What is the ETA for dealing with the requests?

[Jered Kenna (TradeHill)]

It is reassuring to see you posting here - is TradeHill customer service still working?

Thanks. Yes customer service is still working we're just swamped trying to get all the funds out and deal with a lot of individual requests.
When the banks simultaneously closed our accounts it created a huge backlog. We'll get to the emails as fast as we can. Almost everything has been processed at this point and is under control. We'll have the Linode situation sorted soon as well.

-Jered

What is the ETA for dealing with the requests?

We're currently dealing with them but we are being very careful and doing them all manually.
I'd estimate a few days at most we'll be caught up and they shouldn't take more than a day after that point.
We don't want to screw anything up considering everything that has happened.
We've also been dealing a lot with our lawsuit against Dwolla.
See tradehillblog.com for more info on that or the other post in this forum.

-Jered

[Michael_S]

I haven't seen my funds yet, and no reply from TH since 16 Feb 2012.

Generally I fear that my TH funds (EUR and USD) are gone forever (hope I am wrong), because I do not trust TH any more.

Why? --> See here.

[casascius]

SOmething I thought hilarious, I read the TradeHill complaint (lawsuit) on Scribd.com, which plastered it with banner ads for Linode "hosting as little as 65 cents a day" between every page.  Not just once, but the majority of the ad spots between the 19 pages went to Linode.

[Jered Kenna (TradeHill)]

I haven't seen my funds yet, and no reply from TH since 16 Feb 2012.

Generally I fear that my TH funds (EUR and USD) are gone forever (hope I am wrong), because I do not trust TH any more.

Why? --> See here.

I'm pasting this from the other thread. We take this very seriously.

Regarding the withdrawals the site will be back up very soon and either have an interface or provide withdrawals via email.
Send an email to info@tradehill.com now if you like and I'll process it as soon as we finish recovering from the Linode hack.

About the email with the information CC'd. We take privacy very seriously. This was an isolated incident where one employee made a mistake and sent out an email to several customers using CC instead of BCC. It's not excusable and the employee has been reprimanded. Despite this only happening once with over 100,000 emails sent I want to make it very clear that it was not our policy and I will take responsibility for this. It was not our intention and I would like to seriously apologize to the people effected.

SOmething I thought hilarious, I read the TradeHill complaint (lawsuit) on Scribd.com, which plastered it with banner ads for Linode "hosting as little as 65 cents a day" between every page.  Not just once, but the majority of the ad spots between the 19 pages went to Linode.

That is funny as hell.

-Jered

[ctoon6]

It scares me that websites dealing with the amount of money you guys deal with even have the remote consideration of using these types of services (clouds and vps). simply a joke and a very good reveal of the security taken by all these companies.

[Jered Kenna (TradeHill)]

It scares me that websites dealing with the amount of money you guys deal with even have the remote consideration of using these types of services (clouds and vps). simply a joke and a very good reveal of the security taken by all these companies.

We've never kept any wallet files or records on Linode. I'm not as technical as the guys that set it up so I don't want to comment on specifics and make an incorrect statement.

That said it was essentially a decoy and it's compromise didn't cost TradeHill anything other than time and if we were still up with our engineers it would have been insignificant. It worked exactly as intended. We were very concerned about these types of attacks and others which is what we looked at when designing it. I'll see someone more qualified on the subject wants to speak but they have moved on to other projects and are no longer working with us.

-Jered

[coiningz]

It looks to me like you did in fact lose bitcoins because of Linode.  Why else are you stalling to process refunds from Tradehill?  Did you really lose all of your coins?

The timing of Linode getting hacked, Tradehill's refusal to manually process withdrawals, and the lawsuit against Dwolla all leads me to believe you have nothing left to pay withdrawals with, and you sued Dwolla both as misdirection and as your last hope of recovering the lost funds. 

Am I wrong?  Prove it by either putting Tradehill.com back online so people can withdraw through automated means (why hasn't this been done yet?) or manually processing every withdrawal request in your queue.

[Melbustus]

I wouldn't mind an update... I still have some BTC as well as USD in my Tradehill account.

[Jered Kenna (TradeHill)]

I wouldn't mind an update... I still have some BTC as well as USD in my Tradehill account.

I cab move all the coins to an address I announce ahead of time to show that they're still under my control.
Instead of bringing the site up and paying for all the servers we're going to do withdrawals via email.
There aren't very many accounts left with funds in them and it should be straightforward.

The goal was to have this done this morning but it is taking longer than anticipated.
I'll have an update soon. My apologies on this taking far longer than expected.
It was somewhat of a perfect storm. We were almost completely wrapped up when this hit along with Paxum ceasing to do Bitcoin business.
I'm also busy moving to a new house so I've just given up on sleep.

Jered

[Melbustus]

I wouldn't mind an update... I still have some BTC as well as USD in my Tradehill account.

I cab move all the coins to an address I announce ahead of time to show that they're still under my control.
Instead of bringing the site up and paying for all the servers we're going to do withdrawals via email.
There aren't very many accounts left with funds in them and it should be straightforward.

The goal was to have this done this morning but it is taking longer than anticipated.
I'll have an update soon. My apologies on this taking far longer than expected.
It was somewhat of a perfect storm. We were almost completely wrapped up when this hit along with Paxum ceasing to do Bitcoin business.
I'm also busy moving to a new house so I've just given up on sleep.

Jered

Thanks for the update - much appreciated. I can wait... Tradehill's reputation to date has been rock solid, so as long as you guys are responsive on this board, I'm happy. I understand things are crazy over there right now - hope you guys get all this behind you soon. 

[coiningz]

I wouldn't mind an update... I still have some BTC as well as USD in my Tradehill account.

I cab move all the coins to an address I announce ahead of time to show that they're still under my control.
Instead of bringing the site up and paying for all the servers we're going to do withdrawals via email.
There aren't very many accounts left with funds in them and it should be straightforward.

Then do so with haste.  Every day you hold those funds is a day that your former customers can't have use of them.  The best way you can prove to the world that you didn't lose all your coins is to send them back to everyone who's been waiting a week to withdraw.

[Yankee (BitInstant)]

Attn BTC and USD Holders on TradeHill:

I can confirm that the funds are safe. Gareth and myself are working hard, in addition to our own work, in helping Jered and his team get everything back up to speed and get your funds out.

For all those who think Jered is delaying, he is not. We are all on different timezones, ranging from CA, to NY to the UK and Jered has been staying up late to make sure everything is worked out.

If my word means anything, as far as I know (and I've seen) your funds are safe and should be ready for withdrawal ASAP.

We are all start-ups here and not mega corporations. We've never had to deal with situations like this, so contingency plans are difficult to create. We ask that you cut us some slack.

Thank You

Charlie Shrem, CEO
Bitinstant LLC