Bitcoin Forum

Short story:

Somebody hacked my backup machine with pool data hosted on Linode and steal 3094 BTC (http://blockexplorer.com/address/1NRy8GbX56MymBhDYMyqsNKwW9VupqKVG7) ("hot" coins ready for payouts). Cold backup was not affected in any way by this hack.

It looks that also user database has been compromised. Although passwords are stored in SHA1 with salt, I strongly recommend to change your password on the pool immediately.

Robery of Bitcoins has no impact to pool users, I'm covering the loss from my own income (although it means that many months of my work is wasted  ::) ).

Long story + evidence:

This morning I received SMS from pool monitoring that BTC balance went under expected amount, so I started investigating what happen. I saw that there was transaction moving 3094 BTC out of the pool wallet (http://blockexplorer.com/tx/34b84108a142ad7b6c36f0f3549a3e83dcdbb60e0ba0df96cd48f852da0b1acb) few minutes ago. I watched the logs and it didn't look like server has been compromised in any way.

Then I found that two of my Linode machines has been restarted half a hour ago, too, and root passwords has been changed. I changed passwords to new one and found that there was malicious activity on the machines. Then I discover that passwords were changed over Linode Manager (Linode web management), because there was record about password change in Host Job queue (last activity done over Manager). This also explains why attacker restarted machines, because it's necessary to apply this change from Manager.

I reported accident to Linode staff and asked for log of recent logins to Manager. To my surprise, there were only my own log attempts and last login before the attack was on 08/02/2012! I reported to Linode that something is going wrong, because I has been using strong password for my Linode Manager (because I know it's basically backdoor to my machines) and I didn't use this password on different places.

Full log of support ticket is here (http://pastebin.com/UW7iT5fj).

I'm still waiting what they'll find, but expect they'll try to hide any issue on their side and they will definitely reject to pay 3000 BTC for this attack :-/.

Plus
Few hours ago another guy contacted me that his Linode machine has been attacked and his coins was moved to the same wallet, asking me if I know what happen (because he found that 1Mining2 address is mine). We found that our issues are the same - changed password in Manager, stolen coins & Linode staff is telling they have no security issue on their side. Heh.

It looks like attackers found some vulnerability of Linode Manager and used it to infiltrate Linodes with running bitcoind (we both had bitcoind running on the machine), to gain maximum profit for the less rush (it does not look that so much machines has been hacked, at least I didn't find anything on twitter etc). It looks like attackers were interested only in Bitcoins, because they leave Namecoins untouched, although they have the same chance to steal them.

From the attacker's wallet it looks there were more people affected by this Linode hack, maybe they'll know anything more?

Conclusion

There's no reason to think that pool itself was hacked. I changed all passwords everywhere (mainly to database), moved coins to new wallet and everything is working fine. Backup machine didn't contain keys for accessing pool server, so there's no need to reinstall pool to another machine. I'm covering all financial loss from my own money, to keep pool users out of this stupid issue.

Wow.  I'm sorry to read about this slush.  


I applaud you for covering this out of pocket.  Another demonstration of why I'm glad to be mining in your pool.

Hopefully Linode comes clean...

Man, that's a huge loss.  Thanks again Slush for everything you do, you have a donation coming your way from me.  It wont be much, but I'll do what I can at least to help out...

Three things for everyone to learn from this:

#1, use cold storage as preemptive damage control.  Congratulations on being the first high-profile case to get this right.  :)

#2, don't store high value wallets on a public-facing server.  It's much better to keep your wallet on a machine in another secure location, poll for any required sends, sanity check them, and then send them to the network.

#3, Slush just earned 3094 honor points.

Is it positively confirmed that it is a linode issue and not an exploit for bitcoind?

There's no way how to "learn" linode's username and password to login into Linode Manager from machine itself. And attacker obviously used Linode Manager to change root password. So - yes, it isn't bitcoind issue.

The most interesting point of the whole hack is that Linode don't have any log about login to Manager by the attacker, which indicate that they used some vulnerability of Manager itself.

FYI:

The Bitcoin Faucet bitcoind's are both running on a Linode VPS, which was mysteriously restarted 14 hours ago.  The 5 bitcoins in the main-net Faucet's wallet were stolen, also; I'll shutdown the Faucet website, do NOT donate any coins to the Faucet donation address, it is controlled by the thief.

Transaction ID:  14350f6f2bda8f4220f5b5e11022ab126a4b178e5c4fca38c6e0deb242c40c5f
... if you want to start watching where the coins end up.

This is extremely disturbing. Wonder who else was stolen from. Sounds like it was well-planned.

Wow, thats going to be an interesting one to figure out ...

I can't remember, does MtGox block stolen coins from deposit?

Gavin, thank you for info. It's the same time when my linodes were restarted (it was around 7 am UTC). Did you contacted Linode about this issue? Looks like they're still rejecting any problems on their side...

just changed my password, thanks for the heads up.

do you have a donation address?

You can donate to 18pmHDP5fx4A9Tpo69V1KEXWUQyK7EvT9C . Thank you for your support!

digital: thank you, too :-)

His full address from the firstbits is:

Edit: nevermind, see above post

I've already sent along what I could spare...

Woops, guess I was a little late on that one...

digital, you're correct, it's my general "donation" address, but I created the new one to track donations to pool funds...

It exists now at an informal level, but I expect the "tainted coins" stigma will decrease over time.  Right now we have a high percentage of relatively fresh coins, but just like fiat, after they've been in circulation for some time it will be taken for granted that some percentage of it has been involved in some kind of scam.

If you trace the coins forward, it looks like they are going through some sort of laundering/mixing process as we speak.

The downside is this would destroy fungibility.  I'm not eager to see that happen.

The idea of reputation is intriguing, but realistically that will just mean people will pay for premium laundry services that can provide freshly-mined coins.  Mining could become unusually profitable for a while.  :)

I am against anything that could potentially put coins into limbo and add even a hint of centralization to the mix.

plus, there is no way I would trust any organization to decide how "tainted" my coins were ... it sounds like it could be ripe for abuse

Not to throw petrol on this absolute fkup(and it does seem linode is to blame), isnt there a way to manage autopayouts with encrypted wallets so that if your wallet gets accessed its still highly encrypted and unspendable(atleast within the next couple of billion years before its cracked)

isn't this something the new bips can help with

Your software has to know the encryption key in order to make the payouts.

In this particular case it may have helped - if the key was stored only RAM (Slush would have had to type it in every reboot) it would have been wiped when the server was rebooted.  On the other hand, if the attackers get access without rebooting they can grab the key out of RAM and decrypt the wallet.

Oh yeah, no doubt about that :)

I'm curious, How is this handled in the "real world" now with currency?

I'm under impression, you are the first (or one of the very few) people who were hacked and decide to cover the loss from their own pocket. Now I'm happy we have at least the 2%.

So we can see that all linode bitcoin users were affected - if I were you I would contact everyone else affected and send a letter to the company demanding to cover the losses or have a class action lawsuit. At least that's what I would do but I am not a lawyer/what's their ToS/on what terms you were using their service,etc, but I wish you good luck.

The reboot is what's throwing me on this whole thing ... I've got to go read the timeline again, it wouldn't make sense to me to reboot the machine (potentially alerting the server admin ) if you were able to comprise a linode node at the level that has been suggested.

edit: nvm, its clearly explained in the OP. though why a node would need a reboot after a password change is beyond me

Getting access to the Linode admin UI doesn't give access to the server itself.  You can view the console, but you just get the login prompt.  You still need the server's password to log in.

To reset the password the server has to be shut down so that /etc/shadow can be modified.  At that point they could just go in and grab the data, but they most likely used Linode's password changer to minimize the downtime to a few seconds to help prevent getting caught.

A reboot wouldn't be required if they got access to the Linode hosts, but it doesn't sound like that was the case here.  I'm guessing the exploit is in their web-based server management.

No one needs to prevent it, and the data is not all right there in the chain, the most relevant piece in this case is in this thread.

Thefts are not usually known in the first minutes after they happen. It will be trivial to switch the coins before they get the taint. Someone else will hold the bag (and they'll be kindly informed after it is too late by your spiffy taint client).

This is by far one of the scariest things about the process.  Considering Slush and the Faucet were compromised at roughly the same time, it points to the flaw being in Linode's administrative control panel.  A -very- scary situation, considering Linode is one of the largest VPS providers around.

Sorry to hear that guys. I only hope Gavin manages to achieve consensus and use his available resources to have that multisig feature implemented.

This is not the right thread for this, we should move.

The person is not known unless 100% of bitcoin services ID customers.

Linode confirmed that it was their fault, see bottom of pastebin (http://pastebin.com/UW7iT5fj).

So far it looks like superadmin account of Linode Manager leaked, which also explains why there was no login attempt to my account, although there was job for restart & password change.

A classic example of why we need ps2h

With p2sh Slush could have had one key on the server and a second key on an independent device (with third key kept always offline on paper as failsafe).  If he makes payments in batches he could even keep the second device offline outside payment windows and route signing through vpn or tor to provide further hardening.

ps2h is needed to provide not just "stupid user protection" but enterprise grade security solutions.

Wow, quite the attack.  I'd go straight after Linode with a lawsuit.  

Are they going to cover your losses ? This is a substantial amount of money involved.

There may be something in their EULA to protect them against this type of lawsuit

Ask them to cover your losses.

EULA's aren't the end-all that companies make them out to be though.  Even if they say "we will not be held liable for blah blah blah", doesn't mean that a court won't hold them liable.

https://www.linode.com/tos.cfm

I wouldn't expect any different from inexpensive hosting.  No one would take on that kind of liability without a large markup.

It might be in their interests to take responsibility for damages for PR reasons, but I don't think they have a mandate (ethically or legally) to cover $15k of consequential damages for a customer using a $50-100/month service.

I also would not jump on them for admitting fault.  There are way too many companies out there that try to cover everything up when they screw up.  Linode should be commended for providing a prompt and honest answer right from the top brass.

I suggest asking nicely, not with a lawyer's letterhead.

Shows a major weakness in linode I'd say. Other linodes were hit as well.  I would be saying goodbye to linode. Since they seem to be short on details we can't conclude anything, except that they're system is flawed. They need to have failsafes in place.

What could you have done to prevent this?  Would an encrypted wallet prevented this?  Multiple wallets?  It may help a lot of people to discuss how to make it harder at least.

Stealing BTC might become more profitable than mining or maybe it already is, the crook had to give up an 0day possibly?

Would be nice to see linode present an in-depth analysis if they can't cover any of your losses.

It all depends on how slush manages this ordeal, worst case I would atleast want to get partial damages reimbursed.

the joys of having a superadmin account

Slush already said he'd cover it from his own pocket.  No loss to anyone mining with him.

So how about not having those. 

The fact that they have a super admin account that isn't restricted to whitelisted IPs is amazing.  Even my small startup (not even public outside of bitcoin forums/BTC Guild) doesn't allow administrator logins from anywhere other than my office and my home.

This.

Superadmin account + sa in hands of 3rd party who accepts no liability = :(

A nice gesture on their part, in addition to fixing the vulnerability and explaining exactly how they've done so, would be to accept bitcoin as payment for their service.   

Well, either Linode compensates adequately for this or they will have a serious boycott campaign on their hands. If they compensate and promise to fix their systems their reputation might be saved, otherwise it will go down the drain.

I'll be surprised if they offer to cover any losses ... imaginary money and all that hooey

Wow. Watching this.

oh yeah, and eric schmidt thinks p2p currency is illegal too!

It's a bit disturbing that bitcoinica was also down at this time

3 high profile bitcoin sites all down at the same time. (19:00 UTC) - -- EDIT Ignore: I misread the 07:00 UTC in OP)

[2nd EDIT: 020212-03:12 UTC right sentiment, wrong reason.]


marked

Well that's easy to resolve.
Give them a new slush address and tell them transfer in the same imaginary money that was lost.

+1

Wow, I was actually just looking into moving some of my hosting and linode was where I had chosen.  I guess I will have to rethink that.  I'll probably go with EC2 since it seems Amazon takes security quite seriously, but EC2 is noticably more expensive.

Actually, the concept being described is completely against the Bitcoin design.

The design is of course that when a transaction occurs, it cannot be reversed.
End of story.

As soon as that is no longer true you have destroyed the Bitcoin design.
It is no longer decentralised - someone now has power over it.

If 'some' central authority has the power to deem coins good or bad, then you may as well just dump Bitcoin.

Unfortunately sometimes people hack into other people's access security information and are able to steal what is protected by that information.
That certainly does not mean we should consider giving power of Bitcoin to anyone in any manner whatsoever.
That is purely a knee-jerk reaction to the problem - and should never be done.

Of course everyone has the ability to track down the path of the coins and then possibly confront the perpetrator and request them to return the coins.
However, giving that power to any particular person or group to decide is ludicrous.

If that is what you want - then go visit SolidCoin2.0 and stay away from Bitcoin.

also it appears that bitcoinica is hosted at rackspace:

http://whois.domaintools.com/50.56.4.62

A while ago I decided to track down the 'allinvain' stolen coins and see where they ended up.  It turned out that by mid February they were distributed to over 100,000 different addresses, including 8 of my own addresses.  I'm guessing somebody did a very good job of laundering them.  Either that, or this is just the natural way that bitcoins are passed around.

http://bitcoin.stackexchange.com/a/2900/659 is where I posted my findings.

I love that post, thanks for taking the time to do that

Great analysis.  A good example of why we can't blacklist coins.

Dan

The security of EC2 is dependent on what applications you install on it.
If you install an application that has a security vulnerability that gives access to the wrong information then you are no better off.

However, the fact that Linode has an administrative "backdoor" into their system that they put in place with crap level security, certainly suggests anyone would be a fool to EVER use them again.
I'd probably even say that just having an administrative "backdoor" into their system that they put in place means you should not trust them.

Hopefully comments similar to that will spread across the internet and that will be the end of Linode.

QFT

Big irony is that my previous login into Linode Manager (those on 08/02/12) was because I created backup machine and moved pool backup here from my home server. I had some connectivity issues at home and sometimes backup didn't finished properly, so I decided to move backup to standalone machine to make it "safer".

I heard a polish exchange lost their money there because they didn't backup the wallet somewhere else.

It isn't possible to design large automated systems with no kind of management built in. It just can't be done. Certainly it would be possible for Amazon or any other provider to do something similar - shut down the VPS, modify /etc/shadow, and boot up again having given themselves access.

The question is whether they can keep it secure. Good passwords, token based authentication, minimal number of users with access, fine-grained permissions properly applied, etc. Not to mention blatantly obvious things like only allowing access from certain IP addresses. Not doing so is inviting disaster, as per Linode.

i.e. trust a "central authority".
As soon as a large % of Bitcoin people trusted such a central authority I would sell my bitcoins and go find a true decentralised secure currency.
Bitcoin would no longer be that.

And right there you have pointed out the obvious flaw in the whole idea.

It is possible to keep your wallet outside of those large automated systems though.  I don't put my wallet on a system that anyone but me has a access to. Problem solved.

Can you say "Linode employee"?

That's your perp...

I'm glad you've decided that your predictions are infallible.

As I said:

Just coz you have an idea - doesn't mean is isn't flawed - step back and read it again.
I'll explain it if you really can't see it.

Did some work on this before:
http://anonymity-in-bitcoin.blogspot.com/2011/07/bitcoin-is-not-anonymous.html


Have you seen this SVG we made, linked to from that page?  
https://sites.google.com/site/btcanalysis/AllegedTheftBlogVersion.svg?attredirects=0&d=1

You'll need to open it in something that renders SVGs well - I use Google Chrome.

If you mouse over the graphics, you can see the addresses.

The node: 104741, as we number them (this corresponds to http://blockexplorer.com/address/12RyZB4odBmdenN6TPukb1ZR29DHKgMHuJ - the nodes in the SVG have clickable links to blockexplorer) (the node is in the top-middle of our diagram; but chrome etc will let you search the SVG to find it, by the number 104741), is where those coins you found, break off the main flow of funds, which we continued to track.  We only rendered the principal flow of coins; our code follows the 2K of coins, and ignores the flow of ~20coins that break off, that you mention; 10 or so hops later, that small flow arrives at the address you mention.

If you are interested in this sort of thing, check that diagram out.
I'm biased, but I think it does a pretty good job of allowing us to unravel bitcoin flows.

Hey slush, I'm sorry this happened. I'm sending a few coins to you, for your hard work and the decision to cover the losses!

Just a small detail, but:

If I send tainted coins to an address which is already holding clean coins, the two transaction outputs don't "mix".  They stay separate - some tainted, some clean.  The mixing only happens when I combine several outputs to make a new transaction, then each of the outputs is tainted with:

output_taint = sum(input_n * input_n_taint) / sum(input_n)

Is just an idea but maybe offering to the miners the option to donate a percent of their choice from their earnings for the next x days/weeks to help the pool recover would also help

Linode knows about bitcoin because we spoke to an employee from there about witcoin  months ago. They had asked us to take down a particular post about their policy of shutting down a site for content they didnt agree with. They are fully aware of it. I hate to say it but "rogue employee" comes to mind. Boycott unless they fix it.

Thank you Slush for being so man-up about the situation.  Also, I think Linode should be responsible for a portion of the lost.  We should send Linode this thread.   Definitely "Rogue Employee" come to mind at Linode.

No, I did use the term "backdoor" for a reason :)

They didn't acknowledge the access that had been through their "backdoor" until later.
It didn't show up in the logs or slush's information he had.
The first reply from Linode, didn't acknowledge the "administrative" access at all. (re: slush's pastebin)
i.e. they hide the access ...

That's why

I agree that if a thief is willing to go through the trouble, they could launder bitcoins, so as to make them very hard to track; possibly even impractically hard to track.


I don't have a strong opinion on whether 'marking' coins - basically, trying to keep some record of which 'coins' were stolen (or rather, which fraction of the balance at a particular address was 'stolen') would have the effect of reducing the incentive to steal bitcoins.


I do like the idea that even if a small fraction of bitcoin users will refuse to accept 'marked' bitcoins, then this instantly decreases the value of 'marked' bitcoins; and this can possibly have a knock on effect; so that 'marking' doesn't have to be total, or centralised, for it to have an impact.

But I also see the argument that it would be very difficult to know who to trust to maintain lists of marked bitcoins; there seems to be some centralization inherent in the idea of lists of bad coins.

And there would be conflicts of interest; the more bitcoins that get marked as 'stolen', the more valuable non-stolen bitcoins become; anyone with a lot of bitcoins would be incentivised to have other bitcoins 'marked'.

There is also the fundamental difficulty of establishing whether bitcoins that are alleged to be stolen, were actually stolen, or not.  I'm not talking about any specific case here.
If its possible to pay for goods, with Bitcoins, and then later declare the bitcoins used in the payment to be stolen, and hence marked, you mess with the way bitcoin handles non-repudiation.
It would also totally change the setup of services, which have user accounts layered on top of the bitcoin protocol - they don't have a direct mapping from individual users, to bitcoin addresses - while they can be considered to be outside the bitcoin system, in some sense, 'marking' would not work well with them, in practice.

Like all these economic things, it's very hard to reason about the effects of such a system, so I've no strong opinion on it.

The fact that the complete history of a balance is stored, and publicly available, allows you to think about doing interesting things like this, in a way thats hard in other setups.
It might be interesting, if, instead of balances, there were specific 'coins' in the protocol (at the moment, balances lose their individual identities, when they pass through a transaction) - that would allow 'marking' to be done properly - maybe such a system could support 'marking' in some decentralised fashion, and be more resistant to theft.  I don't know, but interesting to think about.

While I think that a marked coin or tainted coin might be harder to spend with some users, there is still a whole segment of users that don't care either way, so the spice will still flow.

Just sad that some criminal made 10k quid from your honest work just like that.

By the looks of it the attack was directed. A random hacker who happens to run into a wallet most probably wouldn't have known what to do about it immediately.

It was very likely a Linode employee who knew about your operation...

I'd look into running servers with hot wallets to .onion sites so hackers don't even know where and how is it stored at all. When I was looking into the betting business I was told of colocation services with flat insurance for intrusion... but it was a nightmare of regulations and very limiting, and expensive as well. Running a server from home with a good redundant connection would probably be a sensible solution considering the amount of money at stake. At worst... well your hot wallet is offline for a while.

Another takeaway is that bitcoin services hosted remotely should keep their on-line wallets encrypted.  Then they can't be stolen after a reboot because the server will require manual entry of encryption password.

Not true.  Each transaction into an address is a separate coin, and they are redeemed separately when you spend them.  They only mix when multiple coins are redeemed at the same time.

If you mean completely individual, non-mixing coins, I don't think there's a practical way to do it with a Bitcoin-like cryptocurrency.  The blockchain would become huge.

How will you be sure they "fixed it" unless they disclose the full vulnerability?
So, as you can never be sure, I say "boycott unless they greatly compensate Slush for the loss" - "I'm sorry" just doesn't cut it.

Linode does not owe you anything, especially an 'estimated value' of your Bitcoins.

Terms of Service exist for a reason, even if it was their fault (which I somehow doubt, given their track record)

Have a more secure system in place next time.

They already disclosed that it was a support login that did it. What else do they need to disclose?

@JeffK just crawl back under the rock you were since Jan 9, 2012.
Interesting that you came back just to say that...

The attacker went outside his secure system and gained root access.  There's not much you can do about that except for not using a hosting service which allows attackers root access to your files.

How about encrypting the wallet ?

Their track record? This last statement tells me you didn't read the thread. The access was from one of Linode's administrative accounts. Therefore the track record is not good... A more secure system would involve not using linode since the access came from them..

Please read the thread before commenting, otherwise you make yourself look foolish.

Thralen

Linode already acknowledged that it's their fault.  BTW I have a mat I'd love to sell you. It has conclusions on it and you can jump to them.

Ah yes, a goddamn pastebin surely is proof

Some people on this forum trust slush quite a bit.  What is his motivation to lie about this?

Another 10k+ BTC from Bitcoinica :-/.
https://bitcointalk.org/index.php?topic=66961

The proof that if it happens again, criminal charges will be taken against the offender, and the victim will be compensated - basically, a secure SLA.
For example. Was that too hard?

Cause if you don't need anything else and are satisfied with their reply as it is, then you have very minimal requirements with people who have responsibilities over your assets and it's people like you who endanger everyone else.

Dasse....

Hell if I know, I'm just saying that the proof is very shaky, I'll wait for a statement from Linode before I think they actually screwed up, but given this community's history for having 'trusted people' disappear with funds, I don't know how much the opinion of 'some people on this forum' matters.

Bitcoin raises web hosting to a new level.  Yes, there are juicy non-bitcoin targets out there such as credit cards and personal data.  But there is nothing like bitcoin for a hacker thief.  Once you steal them, you can wait to use them, something that does not work as well with credit cards.  You can mix them, something you can not do with credit cards.  You can even lay down false tracks by sending them to peoples public addresses.  

Now you have 'data' that is pretty much worth a years (or more) salary for a typical sysadmin.  An employee of a webhost can take it and if they know what they are doing, they can be much 'safer' then stealing credit card information.  Right now the only crime is unauthorized access and data theft, not all of the other crimes that go along with credit card fraud that could involve massive jail time.  I am not saying if caught they would not go to jail, but laws have not caught up to bitcoin.  

I would not trust any shared host (VM or not) that has access to your data for a wallet over $1000.  The only way to do this is with encrypted disks that are setup or encrypted by the customer with no host access of any kind.  No 'control panel" based hosting.  

Do you see any other proof for the opposition posted, in addition there are corroborating reports from others as to the same thing occurring to them nearly simultaneously. Therefore the concept of admin access used for the crime is far more feasible. So we have proof of a sort vs. your opinion. Exactly why would be believe your opinion over even the slightest shred of proof?

Thralen

Here is some more 'proof' for you. Although you're liable to dismiss this in the same manner as the other:

https://bitcointalk.org/index.php?topic=66961

Thralen

do these incidents not bode well for online clients like Electrum or Blockchain.info?

even with encrypted user generated private keys, they can be stolen by the server when opened to sign tx's.

That corroborates the current theory (Linode admin leak).

What are you trying to prove with that link that is contrary to a Linode admin leak?

I think he may be trying to "set JeffK straight" as they say...

Unfortunately this is very hard to achieve in real world. For example, I cannot use any housing here in Prague because of stupidly poor connectivity to abroad. Then it really don't matter if the provider is VPS or not, because technically there must be somebody who have physical access to the server instead of me. I'm hosting the pool in France - it's standalone server, but there is still software KVM (because *I* need to reach the server anytime) and there are probably tens of sysadmins with physical access to server.

So it happen today in Linode, but it can happen everywhere else tomorrow. So choosing server provider for services where you don't have thousands of dollars monthly to protect your own server room is like playing russian roulette.

Hm, please read my previous post. I don't think that VPS containers itself are huge security risk. As you see now, virtualization wasn't the reason for the hack, but it was supporting tool which is in some form in every hosting company, even for unmanaged servers (yes, I'm even paying extra fee for software KVM).

I do agree that it is hard to find options in some areas.  In Baltimore we have a few 'rack space' rental places that will allow you to drop in a server that you have physically set up and nobody has access to online.  Sure, they could get to it physically but that kind of attack is quite different if disks are encrypted.  (and yes, I know it is POSSIBLE to break into those as well but you do need to take the machine offline to do it)

Lol, psy deleted his post immediately ;)

In the transaction related to your incident, one of the destination addresses had 25k BTC or so... by the looks of it the perp has amassed a lot of bitcoins and I bet there were many legit wallets in Linode with legit transactions so he can also use these to launder his money.

It's a lot of money to launder, though. We're talking about 1/4 million US$ or so.

Beware of big mining contract purchases in ferroh or GPUMax (or others) during the next few days.

Since they are a company with real money on the line, they are probably doing an investigation before they make any statement, period.

Yeah, I deleted it because I wasn't even trying to attack you nor did I wished to derail the thread.

Was just replying to you now to say: colocation with encrypted disks?

I understand if you tell me it's expensive, but the alternative is worse, as we all see now.

PS: I don't have any bitcoind facing the web so it's easy for me to stay safe.
Those guides about setting up hidden services are really helpful when one wants to setup a secure server.

Sorry Slush, hope you didn't got mad with me. I'm really in pain with this situation. I was already in pain when it was only you and Gavin, much more now that Bitcoinica even lost more than both of you together.

Yes, I have issues with people that I've never seen contribute meaningfully to something trying to tear apart people that I know have contributed to that thing. In this case, Bitcoin being the thing and Slush (as someone with major contributions to it) being 'attacked' and being, in essence, called a liar. I tend to jump to the defense of what I believe in at those points. Therefore I posted the link to the other major breach that was only tangentially mentioned and linked to in this thread as additional proof, seeing if he'd decide to call Zhou (as well as Slush) a liar by continuing his current stand.

Thralen

Can someone explain how the encrypted wallet was compromised? The attacker found the wallet's password in the source code / config file somewhere?

Maybe because it wasn't encrypted?
I don't remember any of them saying the wallets were encrypted.
Maybe I'll need to re-read the thread(s)...

It's also terribly unfair to attack one of the longest standing most reputable providers without any real statement on their part, and it's doubly unfair to demand they pay back what was allegedly "lost" on the service, since they aren't required by law or their TOS to hold backups of your data for you.

zhoutong didn't provide transaction id of the robbery like slush did

Backups are not really the issue here.

@JeffK Full disclosure request:

What is your relationship with Linode?

It is "hosting something of value on an unencrypted server that is irreplaceable" then?

http://blockchain.info/tx-index/2873808/0268b7285b95444808753969099f7ae43fb4193d442e3e0deebb10e2bb1764d0 -- may be it.

Not saying that the host did anything wrong....  but the problem is not the lack of backups....

It is one backup too many.

Two year customer last month, never had problems but I've been around the Bitcoin community long enough to be suspicious of people who "lose" bitcoins or have them "stolen"

Ok, you're going to be suspicious of Gavin, the bitcoinica guy, and Slush?

Normally I would agree with you but in this case Slush (and Zhoutong who's btc also were stolen) said they will cover the losses out of their own pocket.

And I find suspicious that after being inactive since Jan 9th 2012 you came back today...

I'm half a noob when it comes to exactly how the blockchain can be used to track transactions, but my understanding is that since we have the hash that stole the coins, even if he tries to wash them can't we see at least where big chunks will go?   can we track this money through the block chain?

Aside from covering the losses themselves, both Slush and Zhoutong have been operating honorably and openly for some time.  This is not at all like MyBitcoin which was red flagged by plenty of people as a likely scam long before it went down.

could be, if it's the only 10 grand that moved lately, will wait for zt confirm

funny I was wondering the very same thing. I don't get why anyone would steal bitcoin since when you go to "cash out" it could conceivably be red-flagged - then again they could do small amounts  BUT STILL what thief wants to sit there and do $50 cash out at a time ? can anyone explain this?

he is suspicious of Linode itself.

I'm just a big Linode fan and I don't think it's fair that people are posting shit on reddit and hacker news calling them insecure before we have any statement from them, and when they have a good history of being one of the longest providers of VPSs I've known of, and has always goven me good customer service and free upgrades.


I think it's only fair to give them a chance to respond first. If I was actually a Linode employee, I'd probably be working on that response instead of posting here.

Silk Road has the best laundry and you sure ain't gonna get their help, much less you'll get help from the drug dealers to whom those coins will be delivered ultimately.

Partially this too, I'd rather hear their word on it, than some guys who are posting negative things all over before Linode posts their "Yup, it was our fauly" or "That never happened" response

So you prefer for all other business that host with them to stay relaxed and wait for their turn to be majorly ass-pounded while Linode crafts a response? Is that it?








...moron...

So the moment you move to your next host, and some customer posts something about them being insecure with very flaky 'proof', you will jump ship to another provider immediately?





...moron...

But Linode's vice president already confirmed it was security issue of Linode. He even posted the same explanation to me and Gavin. I cannot say for myself, but I think Gavin is one of most honest people around.

Yes I would. Better safe than sorry!

Better to have to re-enter the encryption password any time the server goes down, than to have no encryption at all. Linux servers never crash of their own accord anyway, unless there is something major wrong.

ah got it you mean the jack-ass's that do this actually spend the bitcoins ? LOL   still even silkroad should be like oh look at that guy buying $1,000 $5,000 worth of ecstasy gee ..

Well then I'm overly shocked the rest of us haven't gotten some kind of notification.

Regardless, I find it hard to believe that a hacker who supposedly has access to all of the Linodes uses that ability to hijack a few bitcoins. i also don't believe there is any legal precedent at all that would require them to cover the Bitcoins in question.

Don't bother replying to JeffK. He already implied you are a liar by questioning your paste of that conversation.

Server never "opens" the key.  The signing is done client side.  While you could have funds stolen it would be because of malware on your computer.  There is nothing on the server to steal.

JeffK, ain't it incredible how some people can value things you deem worthless?

Looks like Linode has issued a status update (http://status.linode.com/):

If you call 13000+ BTC a "few coins", then please send me few coins back. I bet that bitcoins are the most valuable information across Linode servers at all.

Shit, this guy knows his stuff. Check out the transaction size of the 25k transaction:
http://blockchain.info/tx-index/2893660/d9804de366aa4c2a01565c3a3c8aa2ea20baafc276dc875f80b9044841205333
Size:   1337 (bytes)

I guarantee that isn't a coincidence.

A "few" bitcoins? troll much?  Looks like at least 4 major bitcoin sites/wallets were hit.  There may be dozens more.  At least 12K BTC were taken in a few minutes.  Could easily be double that.   We are talking six figures in USD, better than most armed bank robberies and a lot safer. You find it "hard to believe" a hacker or dishonest employee would use a foolishly unprotected super admin account to acquire $100K in irrevocable funds for a few minutes of "work"?

I think an additional measure would be for services to broadcast transactions from their hot wallets strictly behind proxies (as simple as connecting it to a single, separate bitcoind without a wallet hosted somewhere else?), wherever they are hosted.  That way attackers can't figure out the ip address of your hot wallet just by lurking in #bitcoin.

What's that transaction? Who got jacked out of 25k BTC?

Interesting. There's remaining question - how attacker found that exactly those eight accounts are running bitcoin services without scanning whole database? It just confirms my opinion that they compared linode database with list of IPs with running bitcoind, but technically they had access to all linode boxes, if they wanted.

that would be the thief counting his coins in a single stash, seen live as it happened...

It uses the terms "credentials" and mentions that he had to gain individual access to eacher account, so it wasn't a superuser account

right... anyone trying to follow the bits?

Blackmail linode... Get money for yourself plus publicity for bitcoin...

http://blockchain.info/tx-index/2893660/d9804de366aa4c2a01565c3a3c8aa2ea20baafc276dc875f80b9044841205333

Help?

Remind me why linode should pay you back for your own fuck up? If you're too lazy to search around and to then use a respectable host with reasonable security measures then its your own problem if you lose your own money. It's no different to if I change my gold into fiat dollars, put it into a government backed bank who then goes bust.

WTF http://blockchain.info/address/0c767fd66d57a601838213fe5da3b20681a85db4

99K Bitcoins???? 1 hop away from the 25k transaction? holly SH************************

Or is that a Bitcoinica or Slushs' address? I can't get my head to understand all those inputs and outputs.

Slush never asked or demanded that Linode pay him back so how about you just fuck off for a while?

You obviously missed the part about the coins leaving and coming back to the same address.

BTC received != BTC total

It is sad that you have no option of hosting at home, Slush. I always figured it would be stupid to think private keys hosted anywhere else are not compromised and thus as long as they have not yet been stolen to assume it is mostly because there is not yet enough value in them to bother stealing them yet.

I have never considered hosting my private keys anywhere other than a site I physically control and know who else (if anyone) has physical access to. Hence, at home or in some kind of locked bunker no-one else has keys to.

Is there really no way you can get your own home hooked up to the net?

-MarkM-

Dude even says he doesn't expect the company to cover this which kinda implied he hoped they would in the first place.

I'm late to the party. None of my bitcoind Linodes have been compromised...yet. Come and get 'em...all my coins are hot now.

I obviously get lost whenever I see more than 2k Bitcoins /me drools

Is quoting Paul not alloed here? I thought everyone was pretty libertarian? or was it that I had a Carl Marks quote next to it.

I guess it was mostly the 'highest profile' targets that got hit, which explains Gavin getting chosen (although I always thought the faucet kept a rather low amount of coins in it at any time to a roughly equal inflow/outflow of coins or the fact that it used to run empty often

refer to the section written by piuk himself:  http://bitcoin.stackexchange.com/questions/2240/what-are-the-risks-of-using-strongcoin-com-as-an-online-wallet

I can't believe the hacker!

Don't even let off 5 Bitcoins...  :(

If you think about it that is pretty low - attack the free bitcoin faucent wtf?  ???

It was just for confirming he had access to all of Linode.  They said only 8 accounts were accessed (presumably those running bitcoind), so one question is, who were the other 5 and did they have any coins in their wallet?

Also, why 25k BTC?  That's the exact same number allinvain lost.  allinvain had a bit more than 25k in his wallet, but the thief only stole 25k even and let him keep the rest.

I have root access.  I log in, modify bitcoind to send a copy of the plaintext password in a file somewhere the next time they type it, then reboot their system.  They log back in, type their password, and I get their BTC.  It's very hard to protect against an attacker with root access.  P2SH would help, of course.

Yea, that is a reason to remain 'low profile'. But the faucet...yea, that just doesn't make sense. 5, 20 or 100 coins, grabbing from the faucet will hurt the end game.

Now we are getting somewhere. Hacker works for the CIA? Or, more likely, hacker works for a large bank or collection of banks? Stealing from the faucet is terrorism, plain and simple. Call the federales.

I've seen a fair bit of traffic since I got into bitcoin talking about encrypting one's wallet if it's used for backup, etc. The initial articel I read indicating Linode was used only to hold a copy of the wallet but in reading the posts it sounds like it was the live wallet used to make transactions on the running systems, I guess I'm curious regarding which it was.

The last few replies mention allinvain and CIA  - anyone seen allinvain?  hmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm

Couple of ways to look at it. One Allinvain worked for the CIA and wanted to make it look like there was a "huge bitcoin" loss or two the  CIA off'd Allinvain since nobody has heard from him in what like a thousand years? Or taken him to the brig off at sea....

Thieving is the lowest of all sins.  

These are all the transactions with outputs of 2500 BTC or more in the time period we're looking at:


The Bitcoinica 10k is certainly in that 06:50:07 block - it was a busy block indeed!  http://blockexplorer.com/b/169179

no, he's been posting regularly over in the Hardware section in the Ztex thread i believe.

https://bitcointalk.org/index.php?topic=66979.0 -- They posted some of their "suspicious" TX Id's

aaaand the selling begins... http://mtgoxlive.com

 For sure a shared host can be less trusted than a dedicated server but . . . if the datacenter manager ( or employee ) is compromised, the thief can reboot in rescue mode, acces the disk, change root password . . . and the result will be the same . . . cold storage and therefore delayed withdraws ( manually validated once / day by the pool or exchange admin ) seem to be the only safe answer to me . . .

Come on, stop spreading FUD. There is NO WAY IN HELL that the guy can cash out so quickly. Think of daily withdrawal limits, ID verification, coin tracing, and so forth.

My guess? Disheartened noobs cashing out because of loss of faith in the system. All the more coins for me!

Yeah, it's more likely market panic.

The price is dropping  ???  Not going to lie I got a little shaken also ...uggh...

This too shall pass.

But in the mean time, I am vexed!  >:(

Buy!

yeah i never said it was the stolen coins that were being sold.

Markets can remain irrational for longer than I can remain solvent. :'(

Yes this is a problem sometimes :(

Lesson learned: private keys (wallet.dat) are just that: private. Once you put them out there, cloud, webserver, hosting server, email, etc, THEY ARE NO LONGER PRIVATE.

Can we move along now?

Please do not mix things. There are two separate issues:
1. - the security of the server that your client is talking to.
2. - the security of the software running on your computer.

1. It is completely impossible for an Electrum server to steal your coins, because transactions are signed locally. The only way to steal your coins would be to compromise your own computer.
2. Someone gaining access to the server that distributes the software could insert malicious code in the software that is being distributed. (the tar.gz or .zip file, or the executable). Such an attack would only affect the users who downloaded and installed software between the time of the attack and the time where the attack is discovered. This kind of attack is possible for any type of bitcoin client (even the official one). It is mitigated by scrutiny from the community.

Please understand that the situation is very different if you use a web wallet such as blockchain.info. If you use a web wallet, points 1 and 2 are not distinct; an attacker who gains control of the server will modify the javascript code that is sent by the server. The danger is amplified by the fact that your web browser will update the javascript code running on your computer everytime you use the service, and not just when you decide to upgrade your client. Thus, if the server is compromised, then the attacker can quickly replace the javascript code running in the web browser of all clients, and do whatever they want.

I think it's more likely that Linode has a staffer into bitcoins that used the command line tools from the host VM manager to halt the systems, modify the shadow file and bring them up and steal the coins than it is that the Linode user's management tool was compromised.

Linode, if we can believe what they've said, didn't see any management UI activity in the logs at the time the reboots occurred. This is more consistent with someone using a tool outside of the normal logged events, such as the native VM tools, rather than the UI being broken into.

Yes, Eligius' better than the traditional pool, on that point.

If, and I believe in it, it was a staffer, I just fully hope that Linode has logged all such attemps and will identify the attacker and will try hard to force him to return the stolen funds. If he somehow managed to bypass the logs, or hacked the Linode, then Linode should end immediately as whole, this is unacceptable.

Hopefully, this doesn't encourage other VPS/service/host providers to decline service to any potential future bitcoin sites. If slush/bitcoinica successfully convice Linode to compensate them in some significant way, then the lesson for other hosts is that "bitcoin losses will hurt or kill us". In any event, I bet every major host is double checking their TOS and reminding their clientele that they don't cover "imaginary webzone dollar" losses.

This

If facebook's employee administrator panel was hacked into and someone stole facebook credits from users, would they say "fuck you it's imaginary money"?

I hope you're being sarcastic and not an uneducated twat who has never heard of digital commodities, intellectual properties and suing for damages.

Well Facebook has complete control over their own currency and could easily mitigate such issues. Bitcoin is a different animal of a different color on a different planet. This isn't a data redundancy issue, nor an intellectual property issue. This is storing, backing up and restricting access to unique digital information that once accessed and used, is no longer valuable to anyone anywhere ever again (particularly the victim). I can steal the secret formula for Coca Cola, but that doesn't prevent Coca Cola from continuing to produce and sell their beverage. I can pirate a movie, but that doesn't mean the original copy is unviewable (in the vast majority of cases). I can login and delete all of your live data, but you still likely have backups. I can't keep my wallet in a safe and have the ability to double spend my illicitly accessed bitcoin (outside of exceedingly unlikely circumstances).

I do not believe that I am a twat.

PS: All money is imaginary.

Don't see how Linode can get out of compensating (at least in the form of 5 free years of hosting or something) without implying "we're just not a secure-enough service for you to put sensitive data on. Don't put data on our servers unless you're hosting non-interactive web-pages with cute little kittens, or protect your data like Fort Knox because there's no telling when it'll be compromised, either by our staff or our irresponsibility/incompetence."

Exactly - like the difference  between stealing a shirt or pirating a movie . Both might cost the same but the 1st is a 100% loss to the store and the other...well, the pirate probably wouldn't have bought it anyway so no real loss.

You are trying to preach a libertarian ideal without accepting that the US legal system is not libertarian. Bring it back down to earth now.

In a court of law, what Linode did was actionable. That is the only point that needs be made.

P.S. I don't think you're a twat and I typically agree with you, but this point smells of agenda.

Yea, so you agree then? Linode should be held responsible since it had nothing to do with customer security and was indistinguishable from an inside job...

A court of law and physical reality don't always agree, I'll give you that. I'm hopeful that all parties involved will work together to determine what can be done to mitigate the losses, but this is an unfortunate collision between the purity of mathematical and physical reality and legal opinion, (assuming it even gets that far), and opinion will never trump reality.

You missed  - on eligius, added bonus:
The coins you receive are virgin whereas with most pools you potentially could get mixed/old coins.

You can sacrifice them to please internet gods.

Fair enough. ^^

weren't you the one that brought up the whole concept of taint recently?

virgin coins have 0% taint.

They're not associated with any past transactions so have better anonymity.

P2pool is another one.

1)  BTC / block chain / block explorer is awesome as we can literally see where the money goes.  If anyone does any transaction with any of these funds, assuming you would ever really follow this enough to have a computer look for one of the hashes on this trail of tears, then please post everything about it here.

2)  Ok, so I'm a master criminal, and I hacked the lol-tastic Linoodle security web tool, and I steal the 40k BTC off all the BTC business sites hosted there - so I have ~ $160k USD and i'm an asshole so I'd like to get some cash now.  (also note homeboy is certainly reading this thread) You pretty much need to sell any reasonable amount on Gox.  If they are smart they will lay low and not make any more transactions for a while.  But, at some point, those coins are going to have to make it to Gox.  we should ask them, really fucking nicely, to do all they can to make sure those coins don't get turned into cash on their xchange.  Tradehill too.  If you can get enough of the exchanges, even down to the small ones, to get on board with this and someone write some code to follow the block chain until it gets to Gox.  Might be able to get some more clues.

just some thoughts.

definitely clubs.

Firstly, it looks like we're looking at 50K+ BTC.

Secondly, we need the homeboy to get either lazy or impatient. I don't want to be giving ideas but certainly these coins don't have to ever make it to any exchange if he's determined enough...

It's even more likely they never will. People who already had that amount could just be recouping losses of selling their legitimate coins. We're not looking for a poor hacker here, we're looking for someone who already had a lot of coins to begin with. A business maybe. Bitcoinica would be the first person to suspect tbh (although I don't have reason to believe it was Zhou).

Operator of Silk Road?

Coincidently with this incident I went to check the road, and guess what...

I was reading the slashdot story on this today and got a chuckle ... they served a linode ad embedded in the article about a linode exploit.

i thought it was funny :)

Irony.

Satoshi is back!!

Yep. Just reclaiming his property.

Wait! Are these addresses connected with some that Satoshi owned? ( I know that I can check, I just want an easy answer ;D )

Anyway, it can be interesting to see who with a good knowledge of Bitcoin isn't posting on the forum during the last 2/3 days :)
( posting somewhere in the forum after my message isn't a good way to avoid the scanning ;D )

That forces the majority of Linode customers, who don't host large-value websites, to subsidize those who do. To provide coverage for exceptional and consequential losses, Linode would have to obtain much more expensive insurance and raise their rates to cover it. There's certainly room in the market for such a service, but I don't see why Linode should be forced to provide it, and their customers forced to pay for it, if they don't wish to.

If you put leave your $50,000 Rolex watch in the pocket of a coat you put in the coat check of your local restaurant, you can't expect them to be responsible for it. It's just too costly to provide a service suitable for that type of high-value item. Use a safety-deposit box, where you pay for that level of security.

Bitcoins in a hot wallet are simply too valuable and too easy to steal. Putting them on a cheap hosting account is equivalent to checking the Rolex at a restaurant.

our gov't stores gold at Fort Knox (allegedly) or in the basement of the FRBNY inside vaults with security guards, etc.

our banks store their fiat cash in vaults with similar heavy security.

Bitcoin cash needs to be stored in a likely manner.

Now this would be interesting.  Wild speculation here.. but SR could've been hosting their online-wallet at linode and may have been one of the other 5 linode accounts accessed.

IMO the only way in court you might successfully win damages is if you showed they were negligent regarding their security. I think that would be pretty hard. You'd probably have to show they were aware of the vulnerability or open "customer service portal" and disregarded it. Or maybe they knew an employee was involved in malicious accesses but ignored it. In either case it would probably require an inside whistle blower. So far there haven't been indications that negligence occurred.

So what is the latest has the 43,000 bitcoins left the wallet yet ?

What exactly do you mean by "left the wallet"?

From...?

Did you expect Linode to announce it openly or for anyone in the community to know that without a formal investigation?

I was referring to the off-chance that Linode knew about their hacker and he works there at Linode, and they're just covering it up. They mentioned something about a policy change due to this incident. Covering their asses through insurance or profits doesn't change anything for existing customers. I guess you misunderstood. Anyway, it doesn't matter what I think, what matters is if a court of law sees Linode as being responsible.

Really? And when it's the coat checker that steals the watch, you can't expect the police to come? When the coat checker isn't caught, you can't sue the restaurant? You must not live in the USA....

I guess I meant left the wallet of the thief to say an exchange i.e. Mt. Gox  or off to silk road to purchase $15,000 dollars worth of guns and drugs.

I have been trying for a while now. I haven't gotten the email that page says will be sent to allow one to do that.

I thought Taint was the space between the vajayjay and the brown eye.

It would seem Linode is the weakest link for those hosting bitcoin stuff. Customers will need to implement a system that can thwart Linodes retardedness.

Could this be another attempt to manipulate the market with bad news? The stolen funds would remain in hibernation because they are not needed when the theft is for damaging BTC value via bad news.

To any sane person the bad news is all on linode.

Well, before you can determine that, you have to determine how vigilant their security should have been, and that depends on whether you think Linode was marketed as suitable for high-value, easy theft targets like hot Bitcoin wallets.

Well, we don't know yet. But from just the evidence we have so far, I think it's at least reasonably probable that negligence on Linode's part was involved if you think the appropriate standard is sufficient security to host high-value Bitcoin sites.

Take my $50,000 Rolex in the coat room example. If the coat check attendant goes to the bathroom and doesn't have another employee watch the coat room, is that negligent? Yes if the coat room is supposed to be suitable for storing $50,000 Rolexes. Otherwise, no.

Dude. Please don't embarrass us with "coat check" examples anymore. Even a parking lot would be more suitable of an example, or even a storage container facility. Those have contracts at least and expect you to store things for extended periods of time.

You're saying I can't sue the parking garage of one of their employees breaking into my car and stealing it?

You're saying that I can't sue the shipping container company for leaving their keys outside of my container and letting someone just rob me?

Give me a break.

They're not covering anything up. I think it's quite likely an inside job involving a Linode employee or former employee. Linode hasn't said so, but they haven't denied it. It's possible they don't know.

You can certainly expect the police to come and the employee, if caught, to go to jail. But you aren't likely to recover $50,000 from the restaurant. They're not required to make Fort Knox to check coats.

It's tough to say if Linode should be held liable for the damages, but only because I don't think they're going to give us the full story of what happened.

If this was an outsider accessing their Customer Service administration, then that seems like negligence to me.  Under no circumstances should a "super admin" style of account be accessible from anything but pre-approved IP addresses.  That is negligence to allow such a powerful type of account to be public facing.

If this was an inside job (rogue sys admin), Linode shouldd be liable to the customers.  It is then up to them to decide if they are going to sue the now former employee to recoup the damages on their end.

If it looks like an insider job, and it smells like an insider job, it is.... ... an insider job.

First off, what strikes me as odd (and forgive me not to read through everything that's been written about these issues in the forum for the last hours) is that the attacker targeted just Linode customers that had bitcoind running. I mean, if it was only Bitcoinica that was targeted, an outside attacker would seem more plausible, but eight customers that all ran bitcoind, and those were the only ones affected ? Seems very plausible that it's a superadmin that did this. After all, that makes logical sense, when tracks are hidden that well.

The first thing I would do if I were to investigate this case would be to interview everyone that have superadmin access at Linode, and I mean though confrontive cross examination, and lie detectors tests, everything you can throw at them + getting at all and every server logs. I assume Bitcoinica works with law enforcement on this one ? It's a lot of money gone here.

Anyway, let's take the lessons we can, and thumbs up for all the good operators that decided to use their own money to reimburse the customers.

Wow, Bitcoinica actually lost over 43,000 coins.

Damn.

http://arstechnica.com/business/news/2012/03/bitcoins-worth-228000-stolen-from-customers-of-hacked-webhost.ars?clicked=related_right

So you are not willing to believe that the Dole food chain (salads specifically) were poisoned in order to profit from put option trades?
There are hundreds more of this type of examples for stocks. Of course there are no put options for Bitcoins but the method can still be used to profit or attack the value. Good and bad news has reactive tangible effects on volatile markets, of which Bitcoin is one.

Yes of course it makes more sense in the realworld that someone stole the coins not to sell it for personal gain but to only crash the market due to tinfoil conspiracies.

Guys, CoinExchanger is turning out to be the likely culprit in the hack.



CoinExchanger.com is an admittedly unregistered MLB (money license business) that must be registered by FinCEN within 6 months of opening their doors and sharing their first stored value. They have not done so and are in direct violation of federal law.

The owner of CoinExchanger.com is Leo Camilo, who advertises his address as 440 9th ave, New york, New York,10001 US and personal telephone number 1 (347) 469-1040.

His private email (search google) is atqcapital@gmail.com.

He has publicly stated on multiple occasions that:

• bitcoin is fake money, "monopoly money" and has no value and should not be trusted for this reason.
• his exchange is functional with a large user base, when not a single user has ever reportedly done business with him
• he is holding coins stolen from Zhou Tong's Bitcoinica and says "fuck you Zhou, you're just a stupid 17 year old kid, these coins are mine now" basically.

He also:

• goes under the sock puppet scammer account name "Maria"
• claims to be a millionaire and restaurant owner

He is currently in possession of stolen Bitcoins from the Linode hack and any coins purchased from him will not be accepted by MtGox or anyone in the Bitcoin community.

Except after they have been properly laundered through the Silk Road.

/devil's advocate

The part in big letters there... how do you know that he is in possession of coins from the hack?

We cannot know for sure to be totally honest, he claims this is a transaction to his service;
http://blockchain.info/address/0d9e2cd87cef275505cd1a831a8fdf86cd2ff571

See... some other thread for proof, to many thread to look through.
But it was something like "Hey, we just received another 12k deposit!"

Got it - thanks.

I  am rather confused. Don't like to judge until I have better information about this CoinExchanger, ie, posts and comments.

The point is the difference between using a service in a way that requires the normal level of security and using a service in a way that requires an extraordinary level of security from the provider.

Yes, but don't expect to get back the $5 million if you store a prototype car there.

Yes, but don't expect them to cover the costs if your shipment was diamonds, unless they knew and agreed to extra security appropriate to diamonds.

In your world, every business would have to provide security adequate to cover the most bizarre uses of their service. FedEx would have to have a team of armed guards follow every truck they dispatch just in case a package had millions of dollars worth of diamonds in it and the owner of the shipment made the shipment details public so thieves knew just what to target. But in fact, that's not how such services operate. They have precisely-defined liability limits and they require shippers to declare high-value operations and pay extra if you want them to insure them.

Yes or no, do you believe FedEx is legally obligated to defend every package they ship in a way that's suitable to protect millions of dollars worth of diamonds from an inside job? If yes, how do you think they should pay for that? If no, how can they be negligent if their security was adequate for ordinary shipments?

I wasn't aware Mt. Gox retained the right to pick and choose which Bitcoins they'll accept. Where can I find their policy on this? What happens if I transfer them coins they choose not to accept? How can Bitcoins remain useful as a currency if people start picking and choosing which coins they'll accept? This increases everyone's risk when accepting Bitcoins to intolerable levels.

Yep as I mentioned a few pages back :)
https://bitcointalk.org/index.php?topic=66916.msg777985#msg777985

This might be relevant as well.


Source: https://mtgox.com/terms_of_service

Just to point out, unless they changed their mind in the past few weeks, TradeHill isn't trading anymore, they ceased operations a few weeks ago.

-- Smoov

Ok.. it's clear now...

They follow their own rules and not Bitcoin rules....

what "Bitcoin rules" were you expecting them to follow? your "Bitcoin rules"?

The definition of Anti-money laundering is exactly that-- blocks to prevent the laundering of money. Just because it was BTC stolen and not USD doesn't change anything. Don't shoot the messenger just because I was right.

Ok.. I had some thinking about it during last 2 hours..

I think it's MTGOX's right not to accept stolen money...  the only thing they can NOT - is to expropriate it...

Imagine somebody was killed during this breach... they even can report that they saw somebody trying to deposit stolen money..

But God forbid them from acting like Bitcoin Police....

...?

Yeah, you guys act weirdly. I've seen it!  ::)

lmao, we probably do, but we actually do a lot of stuff behind the scenes that not many people see.

heh, that was just my weird sense of humour at work, not a statement of fact at all :D

haha i figured, still. :)

I'm talking about http://en.wikipedia.org/wiki/Fungibility

Are you sure you did enter your correct email-adress? Because, when entering an address that is not in the system, the page doesn't respond with an error, just reloads the same page again.

isn't there legal precedent for criminal charges taken against individuals who stole world of warcraft items?  even if the courts don't treat bitcoin as currency or a derivative, there seems to be a clear path for legal action against the theft of digital assets of value.

It's the best we have IMO... if we ever get any solid evidence about who's the perp. The main difference is WoW's items being perfectly locatable so the legal dispute was just about the criminality of the act. Now, to get law enforcement to try to find out who was it, that's a different matter.

Which relates directly to what I said 10 pages back :)

Which of course, if it did happen, would mean one (or a combination) of 2 things:
1) Linode got back the stolen bitcoins and gave them to slush et al who lost them
2) Linode went and bought $X00K worth of bitcoins and gave them to slush et al who lost them

There is of course the interesting effect of 2) ... and who would gain from that ...

I think Andrew makes perfect sense, even before the wiki link.

After reading Mt. Gox terms of service over and over again, it is probably easier to just describe it like this:

"We do as we please with your account, but if you play nice we might send your held currencies to a bank account upon termination. We also reserve the right to change our mind at any time."

That's pretty typical of terms of service. That's why it's very important to distinguish what the terms of service say and what a company actually does. Mt. Gox's terms of service claim to allow them to steal someone's Bitcoins, but they certainly don't have a policy of doing that. (Nor could they actually get away with it if they tried.)

Quoting their ToS in response to a question of whether Mt. Gox actually has a policy of rejecting "tainted" Bitcoins is spectacularly unhelpful. The question is -- what would Mt. Gox actually do if someone deposited Bitcoins traceable to the Linode theft into their account. And my hope would be that they might notify authorities or notify the depositor, but they most certainly would process that deposit normally, absent some evidence the depositor was involved in the theft somehow.

Almost anything else destroys the usability of Bitcoins. If I have to worry that my Bitcoins might become unspendable in the future, how can I accept them as payment?

+1

Okay, if this shall be inevitably the case I will leave the bitcoin project sooner or later. I'm guessing sooner. Eventually, with all that risk and technical verification that will need to be involved by everyone, it means there will be no future for BTC and I won't continue to invest in something that has no future.

I think there's a lot I, and others, can do to prevent it from happening. The first thing is to make stakeholders understand that this is a huge threat. The second thing is to come up with better responses that don't involve tainting coins. (Which, from the evidence I've seem so far, seems to be what Mt. Gox is doing. So kudos to Gox.)

Just because you can't prevent something from happening doesn't mean it is inevitable!

Sure, if Bitcoin businesses and individuals started to check for the reputation of coins they receive then there's little you can do about it but I highly doubt this will happen. It is simply too much of a hassle to do this in a sensible way. Amongst other things, you'd have to establish a reputation infrastructure, a dispute resolution process and of course you have to get it supported by the standard client. Also, what if the coin reputation service goes down or is DDoSed? Do you suspend the Bitcoin network?

I honestly don't see that happening - especially with Bitcoin often being used in an automated fashion this becomes much too much of a hassle. Also I really hope that with multisig / two-factor authentication becoming established, we won't see many large thefts in Bitcoinland anymore.

Agreed. Let's hope it is too much of a hassle. But let's hope MtGox, who is by far the biggest and possibly indispensible exchange, sees it that way also.

I don't think there's a technical problem. What problem do you think needs a technical solution? If you mean working on a way to help thieves more easily make their coins untrackable, you're way off track. Dollar bills are quite trackable, every one has a serial number on it, and they don't have this problem. Bitcoins should not need to optimize themselves for thieves and money launderers but should instead optimize themselves for use by honest folk.

Anything adding an extra layer of complexity is a massive NO-GO for bitcoin IMO.

You have to take into account that there are hundreds of ways such a system would be gamed. A complexity arms race is the least thing bitcoin needs.

For example: order of transactions within a block is not deterministic. I can have a clean account with, say, only freshly mined coins, and a tainted account. I give you the clean address and you accept the payment by some automatic means of checking taint. Then I immediately transfer a boatload of highly tainted coins to this address. Both transactions happen in the same block and you cannot reliably tell which happened first. Your account is now highly tainted, you may just have lost a lot of value if untainted coins have a big premium due to this system. Then you have to add even more delay to the already high delay there is to have a proper number of confirmations, and you really cannot have an automated system.

Off the top of my head I can think of dozens of attacks.

I wouldn't work in a system like this. Not while I still have coins.

i disagree that something as neutral as money should be biased towards any specific kind of person.

i do agree that as you say, this is not a technical problem.

+1

As soon as there's such a mechanism, stolen coins will find a way to avoid being detected, there's just no way you can do that 100% reliably. This would only result in a great big mess - people wrongfully accusing others of having their coins stolen (even if it was a regular payment or donation) just to get them into trouble, people fighting over evidence and reputation, online wallet services getting into trouble because some think their acceptance policies are not strict enough, tainting coins of innocent others in the process, people flooding donation addresses with tainted coins,...
Also, what would be the next step? Refuse blocks from "shady" miners who include transactions with tainted fees?

We really don't need that - fighting Bitcoin thefts at that level is just not the way to go. You'd only make it a bit harder for Bitcoin thieves at the cost of making Bitcoin a much more miserable experience for everyone else!

Oh and I'm not trying to talk anybody out of implementing such a system, please go ahead and do it, just don't expect it to become widely adopted. Even people thinking such a system would be a good idea in principle are likely to disagree on the details, fighting and lobbying for their favored policies, etc... In the end, it would have been much more effective to just make two-factor authentication easy to use for everyone.

http://articles.cnn.com/2009-08-14/health/cocaine.traces.money_1_cocaine-dollar-bills-paper-bills?_s=PM:HEALTH

"Coming soon to Bitcoin..."

But we will have - some drugs, pedophile, guns... maybe even murders

what else? :)

Who cares?...  ::)

A coin is a coin is a coin, just like a dollar bill is a dollar bill is a dollar bill, with or without coke on it.

That was the point...

1)  product idea:  "level of taint on my bitcoin" site, with a formula to determine level of taint, how many transactions ago, etc

2)  I agree, Gox or any other exchange shouldn't judge your coins, a coin is a coin and it's a brutal, unforgiving system but that's what it has been created here / can't police the coins.

http://forum.linode.com/viewtopic.php?p=49004#49004 (http://forum.linode.com/viewtopic.php?p=49004#49004)

Apparently they're still dealing with it internally.

I wonder if this could become a sort of marketing tool:

• bad security incident happens to company
• company gets negative press, loads of it
• company acts responsibly and betters itself, improves security
• company shines, gets new customers who think company must now be very secure

it worked for mtgox

I myself didn't even know of linode before. If they act correctly now, I might even consider them next time I look for a VPS provider -> successful marketing.

If you should learn anything from this incident is that you shouldn't keep any big amounts of coins on a vps, linode or not.

+1

If all you need is to accept Bitcoin in an e-commerce, then you do not need to leave your private keys on the server. For example, you can use a deterministic wallet to generate your addresses without the private keys.

If your server needs to send bitcoins to customers (which was the case for bitcoinica and slush's pool), it is probably not reasonable to use a VPS, especially if large amounts are involved.

You'd trust a company that had a hidden backdoor? (yes that description is correct, it did not show up for the logs for slush and was either unknown by the person he contacted originally or the access was hidden by them)
As I said early on, I think they deserve to go bankrupt and be done with.
Not a chance in hell I'd trust them for anything.

Public addresses are derived from the private key, so deterministic wallet is not the solution.  However, you are correct that you don't need the private keys.  You can simple keep a buffer of a few thousand address in your db that match private keys you store in a safe location.

I think electrum has implemented a solution where the addresses can also be derived from a seed.

I don't see how.... the private key is the only input to the formula for generate the public key/address.  Sure, you can throw away the private key after you calculate the address, but if you're hacked they will just take the seed and generate the private keys.

Do you have a link to the solution you mentioned?

took me a while, but found it, what I meant is called a "type 2 deterministic wallet". see this post: https://bitcointalk.org/index.php?topic=19137.0 ("Deterministic Wallets")

Thanks... that would work.  In case it's not obvious to someone else, this may help:



A_public_key = A_private_key*point, so B_public_key = B_private_key*point

B_private_key = A_private_key + B_secret -> B_public_key = (A_private_key + B_secret)*point

Since A_private_key*point is our A_public_key, this gives us B_public_key = A_public_key + B_secret*point

Like you quoted, as long as you have the first public key you can generate all the public keys in the sequence without providing enough information to reveal the private keys.



Thanks again for digging up that information.

yes, I was referring to "type 2" deterministic wallets. This solution is currently implemented in Electrum and Armory.

in addition, Electrum has a working example of address generator in python-php: http://ecdsa.org/remote.php

....i hope they dont reveal cutomer data - to finish that line of thought :)

They don't even bother to file a police report when they get robbed or their servers breached, why would they offer a bounty? ;)

Excellent work.

Looks like it could be a potential slip-up of the thief. Now to make sure the exchanges are aware of the two extra addresses:

nevermind, haven't had much sleep and was confused by gap in dates.

still, it seems possible that the earliest ones could be the most revealing. there really should be a database for tagging addresses (as belonging to different exchanges, pools, services, etc.), or is there one i'm not aware of?

Hey slush--so eight months later can you give us any update on this incident?  I'm curious to know if Linode host ever compensated you, even partly.  Did you get enough donations to cover even a small percentage of your losses?

Linode "compensated" me by providing one year of VPS server "for free". I sent them official snail mail letter asking for some compensation, I contacted them by email, no response.

I received around 30BTC on donations from many people and I'm really glad for that support. Still, I had to covered the rest of stolen 3094 BTC from my pocket...

+1

As soon as there's such a mechanism, stolen coins will find a way to avoid being detected, there's just no way you can do that 100% reliably. This would only result in a great big mess - people wrongfully accusing others of having their coins stolen (even if it was a regular payment or donation) just to get them into trouble, people fighting over evidence and reputation, online wallet services getting into trouble because some think their acceptance policies are not strict enough, tainting coins of innocent others in the process, people flooding donation addresses with tainted coins,...
Also, what would be the next step? Refuse blocks from "shady" miners who include transactions with tainted fees?

We really don't need that - fighting Bitcoin thefts at that level is just not the way to go. You'd only make it a bit harder for Bitcoin thieves at the cost of making Bitcoin a much more miserable experience for everyone else!

Oh and I'm not trying to talk anybody out of implementing such a system, please go ahead and do it, just don't expect it to become widely adopted. Even people thinking such a system would be a good idea in principle are likely to disagree on the details, fighting and lobbying for their favored policies, etc... In the end, it would have been much more effective to just make two-factor authentication easy to use for everyone.

Lol we can only hope

Confirmed FBI operation.

Jeremy Hammond:  hacks for the FBI, sentenced to 10 years in prison in return:

http://www.dailydot.com/news/jeremy-hammond-fbi-foreign-governments-list/

http://lists.randombit.net/pipermail/cryptography/2012-March/002586.html


http://pastebin.com/xy8aQY9W


Anyone still not aware of what is going on here?

Damn.

Wow, wonder what else has happened behind the scenes?

EDIT: Wait, didn't our bitcoinica funds get hacked from a Linode server? 

pastebin.com

lol, is this crap....

You're saying the Bitcoins were stolen by an FBI employee?  Or what are you inferring?

LulzSec was working for the FBI.  This is openly admitted.

http://www.theguardian.com/technology/2012/mar/06/lulzsec-sabu-working-for-us-fbi