bitfloor needs your help!

[DannyHamilton]

. . . The wallet box had all public ports blocked but was able to be connected to from a few of the other boxes.

. . . This is why I prefer no incoming connections allowed on the secure box . . .

How do you solve getting to the secluded bitcoind to command it to sent bitcoins out?

I'm no expert on security matters and am not familiar with what the current best practices are in such a matter, so there are probably better solutions than the one I'm about to explain.  Regardless, as a system architect, it only took me a few seconds to come up with one possible solution to your question:

While nothing should be allowed to connect in to the wallet box, processes on the wallet box can still reach out to other boxes.

A separate daemon can be written to run on the wallet box and reach out regularly (every second?) to a database held on some other box.  Entries in this database can indicate actions that should be taken on the hot wallet (procedure numbers to indicate which function to run, additional columns or tables to hold parameters).  The daemon can be written to validate the database entries against a set of sanity checks (size, frequency, destination, etc). It can then trigger an alert (pager, txt msg, phone call) to system administrators and refuse to process the request from the database when any transaction, or set of transactions, exceeds any sanity check.  System administrators can then determine if the attempted interaction with the hot wallet is legitimate and handle the situation as needed. If the private keys for the hot wallet are known to be kept offline in cold storage, the daemon can even shut down bitcoind and delete the local copy of wallet.dat when it detects any obvious sign of intrusion.

[1713957498]

1713957498

[Vladimir]

A separate daemon can be written to run on the wallet box and reach out regularly (every second?) to a database held on some other box.  Entries in this database can indicate actions that should be taken on the hot wallet (procedure numbers to indicate which function to run, additional columns or tables to hold parameters).  The daemon can be written to validate the database entries against a set of sanity checks (size, frequency, destination, etc). It can then trigger an alert (pager, txt msg, phone call) to system administrators and refuse to process the request from the database when any transaction, or set of transactions, exceeds any sanity check.  System administrators can then determine if the attempted interaction with the hot wallet is legitimate and handle the situation as needed. If the private keys for the hot wallet are known to be kept offline in cold storage, the daemon can even shut down bitcoind and delete the local copy of wallet.dat when it detects any obvious sign of intrusion.

Yep, this is a no brainer.

[marcus_of_augustus]

Sheesh ... 25K unencrypted keys lying around at an exchange ... really??

I'm sure they were doing all the right AML/KYC fiat protection racket crap though ... it does make one wonder if any of these exchanges have actually been set-up intentionally to fail?

Maybe time for another Goxing?

[muyuu]

How do you solve getting to the secluded bitcoind to command it to sent bitcoins out?

IMO, the best solution is to walk a request to the box. You can also have the box connect out to a web server that provides it with transfer requests that then just have to be manually approved at the box. The most common transaction will be to move coins to the cold wallet, so all you need is an amount.

IMO the best solution is this: https://bitcointalk.org/index.php?topic=92496.0 (Armory on web servers)

Plus an offline wallet to store the bulk of the reserves, not in any actual running computer connected to anything. http://bitcoinarmory.com/index.php/using-offline-wallets-in-armory

If etothepi could commit to Bitfloor I'd be all over the IPO with a 100K+ valuation.

[bitbonga]

How do you solve getting to the secluded bitcoind to command it to sent bitcoins out?

IMO, the best solution is to walk a request to the box. You can also have the box connect out to a web server that provides it with transfer requests that then just have to be manually approved at the box. The most common transaction will be to move coins to the cold wallet, so all you need is an amount.

The secluded server will have a payment processor that will access the production database from behind a firewall, verify transactions for fraud and send the payments out.

Polling another box? Connections allowed only from select IPs?

While nothing should be allowed to connect in to the wallet box, processes on the wallet box can still reach out to other boxes.

Thanks for the replies.

I guess most newly incoming bitcoins can go straight to the cold wallet and have the exchange run on a manually updated hot wallet.

It's more the hot wallet I'm trying to understand. It is needed for the exchange to instantly process transactions directed by customers. So there'll always be a kind of command path going from website to wallet, no matter how far away you hide the hot wallet, and we'll have to trust that path we setup ourselves. A good hacker will find that path and command the bitcoind. So there's actually no need to trust our path if we can't trust our website.

Now, of course you can have the hot wallet pull for commands and transactions, but then.. how do you trust the content of those commands and transactions? Because, basically, that is that same public website with input from customers.

If we can't trust the website giving commands into the hot wallet, [edited:]how can we trust that same website to collect and offer the hot wallet valid and intended commands to pull?

[JoelKatz]

I guess most newly incoming bitcoins can go straight to the cold wallet and have the exchange run on a manually updated hot wallet.

That's correct. There are serious risks if you don't have incoming bitcoins go directly to the cold wallet.

For example, recently an exchange was hacked with a bogus LR transfer. The hackers bought up Bitcoins and pushed the price up to some absurd amount. This encouraged people to deposit more Bitcoins in their accounts to sell them at the bogus rate. If deposits had gone to the hot wallet, these deposits could have been stolen as well.

It's more the hot wallet I'm trying to understand. It is needed for the exchange to instantly process transactions directed by customers. So there'll always be a kind of command path going from website to wallet, no matter how far away you hide the hot wallet, and we'll have to trust that path we setup ourselves. A good hacker will find that path and command the bitcoind. So there's actually no need to trust our path if we can't trust our website.

Right, that's why you don't keep that much money in the hot wallet. A starting point is your average one-day volume.

Now, of course you can have the hot wallet pull for commands and transactions, but then.. how do you trust the content of those commands and transactions? Because, basically, that is that same public website with input from customers.

You can't. That's the point of the hot wallet. It holds the coins you can afford to lose if you're hacked.

If we can't trust the website giving commands into the hot wallet, [edited:]how can we trust that same website to collect and offer the hot wallet valid and intended commands to pull?

You can't, that's why you don't keep a lot of money in the hot wallet.

To be clear, you do make the hot wallet as secure as you can. But the risks you mention above are fundamental and they are the reason having a cold wallet is essential. If an account is holding more than about $100,000, the keys to that account should never appear on any one machine that provides remote access from the Internet.

[DeathAndTaxes]

If we can't trust the website giving commands into the hot wallet, [edited:]how can we trust that same website to collect and offer the hot wallet valid and intended commands to pull?

Well no security system is absolutely foolproof.  Even a 100% cold wallet can be compromised by physically attacking the cold wallet and/or operator.  The idea is to make the attack harder.

The hot wallet can validate commands internally.  For example it could limit the tx based on:
a) limit per hour
b) max limit until supervisor reset
c) limit per tx/address
d) limited to only preset withdraw addresses (adhoc address may be allowed but at lower limit)

If the attacker can't steal the keys directly, and must send commands to the hot wallet indirectly the attack becomes more difficult and the amount limited.  It is possible the attacker doesn't know that a withdraw request of 6,000 BTC will trigger a lockout and in doing so prevents him from stealing anything.   Combine that with sending all incoming funds to a cold wallet and only keeping say 10% of total funds in hot wallet and the potential theft becomes even more limited.

Even simply limiting the size of the hot wallet to say 10% of total funds and doing nothing else would have limited the attack to ~$20,000.  That is a bad breech but far easier to overcome than a complete loss of funds.

[justusranvier]

If we can't trust the website giving commands into the hot wallet, [edited:]how can we trust that same website to collect and offer the hot wallet valid and intended commands to pull?

You never fully can trust it, but you can make it more difficult for an attacker by having the hot wallet independently check the incoming commands for deviations from normal patterns which could indicate the website has been compromised.

At the cost of requiring more manual human action you can add more safeguards, like requiring customers to pre-register their withdrawal addresses and transferring a list of valid addresses via sneakernet to the hot wallet every 8 hours. Now an attacker can't break into the website and send the hot wallet a command to withdraw all the bitcoins to some arbitrary address because that address won't be on the authorized list.

[epetroel]

A separate daemon can be written to run on the wallet box and reach out regularly (every second?) to a database held on some other box.  Entries in this database can indicate actions that should be taken on the hot wallet (procedure numbers to indicate which function to run, additional columns or tables to hold parameters).  The daemon can be written to validate the database entries against a set of sanity checks (size, frequency, destination, etc). It can then trigger an alert (pager, txt msg, phone call) to system administrators and refuse to process the request from the database when any transaction, or set of transactions, exceeds any sanity check.  System administrators can then determine if the attempted interaction with the hot wallet is legitimate and handle the situation as needed. If the private keys for the hot wallet are known to be kept offline in cold storage, the daemon can even shut down bitcoind and delete the local copy of wallet.dat when it detects any obvious sign of intrusion.

Yep, this is a no brainer.

I've been thinking about a similar method as part of the code for an exchange I'm working on, and it's almost correct other than if somebody has access to your database and knows your rules, they can insert or alter records in the database table that controls your payment processing service.  The solution here would be to have the requests (database records) be nonced & signed.  Preferably with both a server/application private key and a per-user private key derived from the users password.

[BkkCoins]

If we can't trust the website giving commands into the hot wallet, [edited:]how can we trust that same website to collect and offer the hot wallet valid and intended commands to pull?

You could enforce some simple rules. Such as require payments into the exchange be made by an "owned" address such that withdrawls will only go back to the same address. And then check any btc withdraw against separate account balances such that people cannot withdraw more than they have on account. And then there is always that thing that Paypal and most other money handlers do - delay xfers for a few days giving time for reconciliation and verification. Given the track record of exchanges allowing instant withdraws I'd think users would be ready to put up with some delay.

But all of this is moot if you keep an unencrypted backup copy of keys anywhere even remotely possible to reach - which makes this yet another fiasco.

[DannyHamilton]

. . .How do you trust the content of those commands and transactions? Because, basically, that is that same public website with input from customers.

If we can't trust the website giving commands into the hot wallet, [edited:]how can we trust that same website to collect and offer the hot wallet valid and intended commands to pull?

You can't eliminate your risk 100% and still have a functional usable interface, but you can drastically limit your risk.  If you put sanity checks into the daemon that runs on the box that does the polling, you can limit the size of any single transaction, you can limit the total amount of BTC that are transferred out over any time range, you can require authorization for certain transactions, and as I said earlier you can immediately shut down bitcoind, and the polling daemon as well as delete the wallet.dat in the event of anything that is identified as a theft attempt.  Without access to the code in this daemon, a hacker can't know what limits exist or what triggers will result in the system administrators being alerted to his efforts.  If the hacker unknowingly attempts a few really large transfers he will be stopped in his tracks.  If a hacker is more patient, he might be able to size his transactions to match the typical transaction of the service and keep the total number of transactions in line with typical traffic.  This might allow the hacker to get away with a few coins, but it will slow them way down limiting your exposure and allowing you time to identify the breach and respond.  I suspect that most users of most services would be willing to wait on human authorization and manual transfer of any large transaction if it means keeping their money safe in the first place.

[muyuu]

Bitfloor Hacked, $250,000 Missing (bitcoinmagazine.net)
http://news.ycombinator.com/item?id=4477376

http://www.reddit.com/r/Bitcoin/comments/zchfn/bitfloor_hacked/

http://arstechnica.com/tech-policy/2012/09/hacker-steals-250k-in-bitcoins-from-online-exchange-bitfloor/

BBC as well: http://www.bbc.com/news/technology-19486695

[barbarousrelic]

It's simply not believable that anyone involved enough with Bitcoin to make such a site, who has undoubtedly heard about all the other large-scale hacks, would simply leave an unencrypted wallet file worth a quarter mil lying around waiting to be hacked.

Right now, the owner should be desperately trying to convince us he didn't steal the money himself.

I have but a tiny percentage of the wealth of that wallet, and I am not smart enough to run such an exchange website, but I know enough to encrypt my fucking wallet .

[ErebusBat]

The system was connected to from one of our other boxes which was accessed through a virtual console. The wallet box had all public ports blocked but was able to be connected to from a few of the other boxes.

Thanks for confirming.  This is why I prefer no incoming connections allowed on the secure box.  If you must have occasional ssh, you can have it enabled on boot and then login to disable it.  That way you can reboot first if you must login.

How do you solve getting to the secluded bitcoind to command it to sent bitcoins out?

You don't.  You have a process on the 'wallet server' that checks an external source and base it off of that.

In a 100% ideal security scenario you don't have ANY incoming connections.  That isn't 100% possible because bitcoind needs to get blocks so that has to be at least port 8333 open.  Also other ports were probably open as well for convenience, just firewalled to allow certain machines access.

EDIT:  Or what the other 100 people above me said (teach me to reply without reading the whole thread).

[Cluster2k]

Interesting.  After we saw hacks and/or incompetent management from MtGox, Bitomat, Bitconica (x2), etc, exchanges are still getting hacked.  Probably best to leave matters to the professionals with 24/7 monitoring of both hardware and software involved in the project.

Anyone else running an exchange without 24/7 surveillance and independent auditing?  Best to own up now so users can proactively flee.

[peasant]

It's a shame. I really liked this exchange. Lets see if the owner makes things right.

[DeathAndTaxes]

Interesting.  After we saw hacks and/or incompetent management from MtGox, Bitomat, Bitconica (x2), etc, exchanges are still getting hacked.  Probably best to leave matters to the professionals with 24/7 monitoring of both hardware and software involved in the project.

Anyone else running an exchange without 24/7 surveillance and independent auditing?  Best to own up now so users can proactively flee.

Tangible Cryptography doesn't have 24/7 manned monitoring or independent auditing.  Then again we never accept sales we can't promptly pay and we hold all coins in an offline wallet which greatly reduces out attack surface.

The reality is setting the bar that high is probably useless.  Bitcoin simply isn't that big.  MtGox "maybe" meets your requirement.  All the hacks to date could have been prevented and/or reduced in scope with some more realistic standards.

[sd]

Interesting.  After we saw hacks and/or incompetent management from MtGox, Bitomat, Bitconica (x2), etc, exchanges are still getting hacked.  Probably best to leave matters to the professionals with 24/7 monitoring of both hardware and software involved in the project.

This is exactly right. Banking website have to comply with countless rules and regulations. BitCoin websites are setup by people with no practical experience of building secure sites and those people always miss things.

[bg002h]

the stupid feature of exchanges are instant withdrawals....why not keep everything in cold wallets and process btc withdrawls once a day...manually...

[SgtSpike]

the stupid feature of exchanges are instant withdrawals....why not keep everything in cold wallets and process btc withdrawls once a day...manually...

That's kind of what I was thinking... I'd much rather have secured funds at this point than the ability to instantly withdraw them.