Bitcoin Forum

From tradehillblog.com


Dear Clients,

Yesterday, March 1st, one of our hosting providers was compromised. Linode.com contacted us several hours ago via email stating:



" As a valued customer, the security of your account is our top priority.  Today we suffered from an unauthorized access of our system that resulted in eight customer accounts being compromised. Unfortunately, your account was one of the accounts targeted.

You should begin a compromised system recovery procedure immediately -- we recommend a complete redeployment -- on your  Linodes. "




It appears that hackers were targeting accounts owned by Bitcoin businesses.

We built TradeHill with security as a priority and make use of multiple data centers to protect core infrastructure. Due to our system architecture design it appears that no user data was compromised, and all wallet files are safe, however we are in the process of a more thourough audit and will provide updates when more information is available.

We are taking TradeHill offline at this point as a precaution and appreciate your patience.

Updates can be found  here and on twitter via @tradehill and @jeredkenna

Regards,

Jered Kenna
Chief Executive Officer
TradeHill

It would seem the attacker used backdoor administration access that was not logged (and not publicized as being present) to reset shadow passwords and gain access. They were quite quick in withdrawing funds from slush mining pool, bitcoin faucet, and bitcoinica, but I wouldn't rule out any kind of compromise or future wallet emptying, as it seems that many think this came from inside Linode themselves using tools only their personnel would have access to on any reasonably administered system. It would be wise to go as far as considering your entire VPS file system cloned and logged into with root access, then think of what the intruder might do with the data.

I hesitate to speak too soon and we're going through it right now but it looks fine. The idea was that Linode could be compromised without compromising TradeHill and without compromising the servers where everything happens. We built the exchange to protect against things like this and the engineer that laid it out did fine job. I wasn't enjoying paying the extra data centers every month but I certainly don't regret the decision now.

Linode is done though, they were great to work with and we'll see what happens.

Jered

It is reassuring to see you posting here - is TradeHill customer service still working?

Thanks. Yes customer service is still working we're just swamped trying to get all the funds out and deal with a lot of individual requests.
When the banks simultaneously closed our accounts it created a huge backlog. We'll get to the emails as fast as we can. Almost everything has been processed at this point and is under control. We'll have the Linode situation sorted soon as well.

-Jered

Good work, TradeHill. In the interest of protecting from future events like this, would you disclose what you did that made your accounts immune to this attack?

This is 10 points for Tradehill. I hope you launch Bitcoin.com soon, I'm interested in that site. You've earned the trust of the community and we're excited for what's coming.

What is the ETA for dealing with the requests?

We're currently dealing with them but we are being very careful and doing them all manually.
I'd estimate a few days at most we'll be caught up and they shouldn't take more than a day after that point.
We don't want to screw anything up considering everything that has happened.
We've also been dealing a lot with our lawsuit against Dwolla.
See tradehillblog.com for more info on that or the other post in this forum.

-Jered

I haven't seen my funds yet, and no reply from TH since 16 Feb 2012.

Generally I fear that my TH funds (EUR and USD) are gone forever (hope I am wrong), because I do not trust TH any more.

Why? --> See here (https://bitcointalk.org/index.php?topic=67284.msg788463#msg788463).

SOmething I thought hilarious, I read the TradeHill complaint (lawsuit) on Scribd.com, which plastered it with banner ads for Linode "hosting as little as 65 cents a day" between every page.  Not just once, but the majority of the ad spots between the 19 pages went to Linode.

I'm pasting this from the other thread. We take this very seriously.

Regarding the withdrawals the site will be back up very soon and either have an interface or provide withdrawals via email.
Send an email to info@tradehill.com now if you like and I'll process it as soon as we finish recovering from the Linode hack.

About the email with the information CC'd. We take privacy very seriously. This was an isolated incident where one employee made a mistake and sent out an email to several customers using CC instead of BCC. It's not excusable and the employee has been reprimanded. Despite this only happening once with over 100,000 emails sent I want to make it very clear that it was not our policy and I will take responsibility for this. It was not our intention and I would like to seriously apologize to the people effected.



That is funny as hell.

-Jered

It scares me that websites dealing with the amount of money you guys deal with even have the remote consideration of using these types of services (clouds and vps). simply a joke and a very good reveal of the security taken by all these companies.

We've never kept any wallet files or records on Linode. I'm not as technical as the guys that set it up so I don't want to comment on specifics and make an incorrect statement.

That said it was essentially a decoy and it's compromise didn't cost TradeHill anything other than time and if we were still up with our engineers it would have been insignificant. It worked exactly as intended. We were very concerned about these types of attacks and others which is what we looked at when designing it. I'll see someone more qualified on the subject wants to speak but they have moved on to other projects and are no longer working with us.

-Jered

It looks to me like you did in fact lose bitcoins because of Linode.  Why else are you stalling to process refunds from Tradehill?  Did you really lose all of your coins?

The timing of Linode getting hacked, Tradehill's refusal to manually process withdrawals, and the lawsuit against Dwolla all leads me to believe you have nothing left to pay withdrawals with, and you sued Dwolla both as misdirection and as your last hope of recovering the lost funds. 

Am I wrong?  Prove it by either putting Tradehill.com back online so people can withdraw through automated means (why hasn't this been done yet?) or manually processing every withdrawal request in your queue.

I wouldn't mind an update... I still have some BTC as well as USD in my Tradehill account.

I cab move all the coins to an address I announce ahead of time to show that they're still under my control.
Instead of bringing the site up and paying for all the servers we're going to do withdrawals via email.
There aren't very many accounts left with funds in them and it should be straightforward.

The goal was to have this done this morning but it is taking longer than anticipated.
I'll have an update soon. My apologies on this taking far longer than expected.
It was somewhat of a perfect storm. We were almost completely wrapped up when this hit along with Paxum ceasing to do Bitcoin business.
I'm also busy moving to a new house so I've just given up on sleep.

Jered

Thanks for the update - much appreciated. I can wait... Tradehill's reputation to date has been rock solid, so as long as you guys are responsive on this board, I'm happy. I understand things are crazy over there right now - hope you guys get all this behind you soon. 

Then do so with haste.  Every day you hold those funds is a day that your former customers can't have use of them.  The best way you can prove to the world that you didn't lose all your coins is to send them back to everyone who's been waiting a week to withdraw.

Attn BTC and USD Holders on TradeHill:

I can confirm that the funds are safe. Gareth and myself are working hard, in addition to our own work, in helping Jered and his team get everything back up to speed and get your funds out.

For all those who think Jered is delaying, he is not. We are all on different timezones, ranging from CA, to NY to the UK and Jered has been staying up late to make sure everything is worked out.

If my word means anything, as far as I know (and I've seen) your funds are safe and should be ready for withdrawal ASAP.

We are all start-ups here and not mega corporations. We've never had to deal with situations like this, so contingency plans are difficult to create. We ask that you cut us some slack.

Thank You

Charlie Shrem, CEO
Bitinstant LLC

*bump* and watching...

I got some assets on tradehill in the form of US$, does that mean I'm screwed?

Thanks for the note. I do understand, and do not envy, the position they're in. Being involved in several startups myself, I understand that things can be extremely hectic, and they may just not have the human resources to handle the work in a timely fashion.

That said, I *do* think we should be getting updates in this thread, ideally daily, but certainly no less frequent than every two days. Just quick regular status updates would be great. People with money tied up will quickly lose confidence and move to freak-out mode if they think Tradehill personnel have gone dark.

Ok, waited 2 weeks now since the Linode issue. What's the last state of getting my 586 Euro back, that I transfered via Paxum into the TradeHill account at 3. of January? (ID 37652)

Can anyone provide an update?

http://tradehillblog.com/

The last post there was March 6th. We need more current info than that. Jared posted an update in this thread on the 11th. I would hope he'd be willing to give quick updates here (or *somewhere*) every day or two...

The silence is not good.

Here's another poster with (what I would consider) big money in TH that he wants back 
 
http://www.reddit.com/r/Bitcoin/comments/r52e5/had_a_couple_hundred_btc_in_tradehill_and_hadnt/ 
 
TradeHill: luckily I didn't have anything in there when you went down, but for the sake of others who did, please update everyone.

At least prove the ownership of funds by doing the coin move like you said you could, that would provide an 80% reduction in anxiety for all concerned.

I've been working non stop going through requests getting coins and fiat back to people.
It's mostly all cleared up but please send an email to info@tradehill.com if yours hasn't been addressed or returned recently.

I'm the only one working at TradeHill for now and I'll be answering info@tradehill if you send an email.
I'm doing my best to get this done but it is all being done manually and recovering from the Linode hack was a lot of work.
I appreciate everyone's patience and my biggest concern is getting everyone's funds back, and shutting down legitimately so I can resume later with trust.
This will also go a long way for Bitcoin.

I'm going to transfer 20,000 BTC to 1yvnLFMNRDRGKJ2dMqprzD68mtKga8uoJ right now to show that I still possess a large quantity and didn't lose them when Linode was compromised.

I'll work on being easier to contact. I've also got a lot going on in my life right now but want to wrap this up before moving on to a new project or direction.
I've been involved with Bitcoin for a long time and don't intend to move away from it now. I've never been so passionate about something in my entire life.
 
Regarding the Dwolla lawsuit: itwon't enter the next phase most likely for another 2 weeks give or take. They have been served and we're awaiting their decision.

Regards,
Jered Kenna

Thank you, will inform you about my case.

I hope you guys get your money back, they are nothing more than crooks for doing that, and I will not be satisfied until justice is properly served to them.

I apologize for any preconceived notions about you guys, you are needed in this market, weather i like it or not, it needs competition, and you guys supply just that.

Hello Jered, I've not yet get any answer. Please keep in mind, that I wasted many hours on reminding you with tickets and forum messages, I get tired. What's the situation now? (My deposit ID 37652 from 3. of January, 586 Euro from Paxum)

SUCCESS SUCCESS SUCCESS

My problem has been solved now. Jered transfered my missing 586 EUR as BTC back to me. Thank you very much.

My impression is that he want to solve all issues with residual money in TradeHill accounts. I do not really know why this is so complicated. Stay calm and remind him from time to time.

SUCCESS SUCCESS SUCCESS

I been sent all my btcs (several k) and usd yesterday. Thank you Jered! Your trust is rock hard.