What can really be done about server hacking

[bfever]

I also don't see the point why on earth people are putting a wallet file or running a bitcoind on a hosted server. 

Instead, put the bitcoind with its wallet on a simple PC/server behind a firewall (at home/office, right where you can keep an eye on it) only letting traffic in from the server IP on a particular (non-standard) port where bitcoind listens.
Let the hosted server send its RPC's to the "off-line" bitcoind.
No easy wallet.dat to be copied if the hosted server gets cracked (that is what supposedly happened).

Instead the cracker needs to gather all the info how to contact that offline bitcoind, compile a client, upload it to the server, and only after that the cracker could send some bogus RPC's to "steal" some bitcoins.
Probably some simple countermeasures can be taken against that too: some special sequence for the RPC's or whatever (if you need to send X.Y bitcoins, sequence could be: ask block header X, send X.Y bitcoins, ask block header Y), so that an simple attempt to spend some bitcoins from the main wallet address can be easily detected (the normal server code would never make a false sequence) and that cuts off the offline bitcoind from the Internet until the problem has been investigated (fail2ban style for example).

If you put together a site in just a few days, of course if can't be much more then some copy & paste of standard blocks...

[mb300sd]

Pretty simple stuff:

1) Use your own hardware in a colo-cage.
2) No external password reset.  Period.
3) Remote access only via a dedicated NIC
4) Hardware firewall/VPN which handles IP whitelisting
5) Two factor authentication for all server logins.

TL/DR version:
How about don't let the hacker reset your password and login to your server?

You forgot the three most common exploits:

1. XSS.
2. CSRF.
3. SQL Injection.

Absolutely none of the protections you just just listed will help if the site has shitty/lazy code.

Who writes shitty code? Well for a start MtGox was vulnerable to at least two of those in the past. As have been a large amount of Bitcoin sites (almost all of them).

Xss and csrf dont expose a site to losing an entire wallet, only one customers account.

[DeathAndTaxes]

in my experience no hsms work like this. (none of nciphers' nshield range or thales' hsm products act in this manner,  100% sure on that. I dont think the yubi does either. not sue. that is why I am going to order one, see what it does do, if it will do sha256 signing.)

SHA-256 is a hash function not a signing function.  SHA-256 would be useless for outputting a signed bitcoin tx while keeping the private key(s) inaccessible.

This is exactly how all code signing hsms work, although for us rather than signing code, you sign a bitcoin transaction.

So the devices you mention know how to take a set of internal ECC private keys and an output address provided by the host, determine the value of the keys,  verify the transaction against business rules (velocity, tx volume, time), then generate the public key from the private key, create the Bitcoin transaction and sign it, and output only the signed transaction.

Input:
a payment address
value of various addresses

Internal:
Private keys
Business rule counters

Output:
Signed bitcoin tx.

I am thinking the answer is no?

The Edge and Solo both do on chip signing, so the keys never leave the device, nor are in plaintext (both are fips level 3 certified devices.)

On chip signing is equally useless.  Attacker says here HSM sign this tx for 18K withdrawal.  What security did an on chip signing accomplish.  

using and edge and a solo, the keys can be split amongst people in different parts of the world and never be on the datacentre machines. nor have one person know the whole key. (you dont have to use this functionality)

Which would accomplish absouletely nothing since the hot wallet would by defintion allow the attacker to withdrawal funds.  Securing a cold wallet is trivially easy and doesn't require an HSM.

I hope this clears things up a bit.

Not really.

Maybe you should state exactly (as in low level protocol) what the HSM would do and how you think that would provide enhanced security.  What are the inputs, what are the outputs?

[mistfpga]

in my experience no hsms work like this. (none of nciphers' nshield range or thales' hsm products act in this manner,  100% sure on that. I dont think the yubi does either. not sue. that is why I am going to order one, see what it does do, if it will do sha256 signing.)

SHA-256 is a hash function not a signing function.  SHA-256 would be useless for outputting a signed bitcoin tx while keeping the private key(s) inaccessible.

of course your are correct, i was over simplifying and went too far, to the point where i was wrong.  I was trying to create payshield type hsm functionality and ended up messing it up.
however, the point i was making was valid... see below.

So the devices you mention know how to take a set of internal ECC private keys and an output address provided by the host, determine the value of the keys,  verify the transaction against business rules (velocity, tx volume, time), then generate the public key from the private key, create the Bitcoin transaction and sign it, and output only the signed transaction.

Input:
a payment address
value of various addresses

Internal:
Private keys
Business rule counters

Output:
Signed bitcoin tx.

I am thinking the answer is no?

the answer is very much yes. what you have outlined above was the usecase i was trying to put forward (although without the explicit business rules and simple value based ones instead, the way you have proposed it is much clearer.), from the page on the solo:

http://www.thales-esecurity.com/Products/Hardware%20Security%20Modules/nShield%20Solo.aspx
(also has quotes from the solo feature page)

Thales nShield Solo is a family of embedded, general-purpose HSMs for servers and appliances that safeguard encryption and digital signing keys and that can optionally run custom applications on the module to protect data in use.

The Secure Execution Engine runs application software in a proven, certified hardware environment. It protects data, processes, and intellectual property that would otherwise be at risk.

Elliptic curve cryptography is becoming increasingly popular. All nShield Solo cards can process elliptic curves inside the HSM, which requires the Elliptic Curve (ECC) Activation. nShield 500 offers especially good performance because it features hardware acceleration of elliptic curve operations.

CodeSafe protects data in hostile environments
All HSMs can protect key material against breaches, but most cannot actually protect your valuable data while it is in use. Data breaches have shown that Trojans or rogue administrators still have access to sensitive information on the host system after it has been decrypted by the HSM. The Thales CodeSafe technology enables you to process sensitive information inside the HSM so that it is never exposed on the host system. This enables you to run critical processes in hostile environments, for example:

Where facilities cannot be physically secured
Where you need to protect against rogue individuals with access to the host system
Where host systems may be hacked or become infected by Trojans 
Thales offers off-the-shelf CodeSafe applications as well as CodeSafe Developer Software to create custom applications.

Ensure project success with Thales deployment services
Thales offers professional services to ensure a best practice implementation of Thales HSMs. Organizations can benefit from developer support to integrate Thales HSMs with custom applications or to develop custom applications to be executed on the HSM to process sensitive data.

see, this stuff _will_ make things safer.  you could have the client and blockchain on the device if you wanted (not sure why you would, but you can)

On chip signing is equally useless.  Attacker says here HSM sign this tx for 18K withdrawal.  What security did an on chip signing accomplish. 

By on chip, i meant all the action takes place in the device.

Maybe you should state exactly (as in low level protocol) what the HSM would do and how you think that would provide enhanced security.  What are the inputs, what are the outputs?

I intend on doing this if/when my edge gets here (I have been told I will get one to play with). until then i have other things to be doing (like the bitcoin testing project) - I hope you can see that the solo can cope with what you want it to do.  The Edge will too, but it does not have the secure execution engine.  The edge does have ECC and will run custom code too.   afaik these boxes run posix compliant microkernels, so development is pretty simple.  There are a set of api's that you can use, if you do not want to write a completely custom solution.  See the Thales website for more info.

cheers,

steve

[DeathAndTaxes]

Interesting.

Although for $20K it likely isn't cost effective for most operations.  A "poor mans" version would be a dedicated locked down box.


Web Application server connects to a tx server via serial port.  The tx server is the "poor mans's HSM".  It has no uncessary applications, no ssh, no remote login.  It simply communicates to the outside world via serial port.

It gets inputs from the serial port, runs them against its "rules", builds tx from private keys and outputs signed tx.  Changes to the tx server could be accomplished by simply providing it a new complete distro.

I intend on doing this if/when my edge gets here (I have been told I will get one to play with). until then i have other things to be doing (like the bitcoin testing project) - I hope you can see that the solo can cope with what you want it to do.  The Edge will too, but it does not have the secure execution engine.  The edge does have ECC and will run custom code too.   afaik these boxes run posix compliant microkernels, so development is pretty simple.  There are a set of api's that you can use, if you do not want to write a completely custom solution.  See the Thales website for more info.

Interesting.  I will look into it.  The yubi HSM though would not be useful for securing a hotwallet.  It has no ability to run internal code and merely provides secure key generation and storage.  If the host has access to the yubi then so does the attacker.

On edit: The docs on the edge don't seem to indicate the ability to run internal custom code but maybe I am missing it.

[DeathAndTaxes]

Pretty simple stuff:

1) Use your own hardware in a colo-cage.
2) No external password reset.  Period.
3) Remote access only via a dedicated NIC
4) Hardware firewall/VPN which handles IP whitelisting
5) Two factor authentication for all server logins.

TL/DR version:
How about don't let the hacker reset your password and login to your server?

You forgot the three most common exploits:

1. XSS.
2. CSRF.
3. SQL Injection.

Absolutely none of the protections you just just listed will help if the site has shitty/lazy code.

Who writes shitty code? Well for a start MtGox was vulnerable to at least two of those in the past. As have been a large amount of Bitcoin sites (almost all of them).

Agreed however I would point out the Bitcoin "robbery" was the digital version of a smash and grab.  The list wasn't intended to be comprehensive (and shouldn't be taken as such).  There will always be an escalation.  As simple attacks are defeated attackers will move to more complex attacks.

However codebase security/audits without physical security is worthless.  It doesn't matter if your codebase is hardened against intrusion if the theif can simply change the locks and let himself in the front door and walk out with all the loot. Rock solid physical security can lay the foundation for a more comprehensive and hardened system.  It is the beginning not the end but it has to start there or the rest is futile.

As a db developer I can tell you #3 is pretty simple to secure against.  Don't allow the web application to execute ANY arbitrary SQL.  Period.  With no exceptions (no matter how much the web developers bitch and moan).

[Sukrim]

* Let your page/backend/whatever create a list of tuples like this: [(address1, amount1), (address2, amount2), ...]
* get a current copy of the blockchain from a trusted bitcoin node
* take the blockchain copy + payout list to the offline PC, let it create and sign transactions and save them to a third file.
* feed the transactions back to the bitcoin network

The above can be done as often/frequently as you like. No PC will ever have a chance (other than virus codes embedded in the blockchain?) to be compromised, keys are offline from the beginning etc.

Some other use cases:
If you need fresh keys for deposits, just generate a bunch of them on the offline PC and transfer a text file with the resulting addresses only.
If you need to know how much was deposited to which account, you don't need the private keys for that - just run a standard bitcoind and don't use it's wallet at all, instead query for address balances etc.

Really, is the "convenience" of being able to pull money from an exchange/bitcoinica/... within one hour instead of one day really worth the real danger that they can get hit as hard as they repeadedly were?

[DeathAndTaxes]

Really, is the "convenience" of being able to pull money from an exchange/bitcoinica/... within one hour instead of one day really worth the real danger that they can get hit as hard as they repeadedly were?

That kinda sums it all up.  Just because Bitcoin can be instant may not mean it always should be instant.

[Dalkore]

Pretty simple stuff:

1) Use your own hardware in a colo-cage.
2) No external password reset.  Period.
3) Remote access only via a dedicated NIC
4) Hardware firewall/VPN which handles IP whitelisting
5) Two factor authentication for all server logins.

TL/DR version:
How about don't let the hacker reset your password and login to your server?

How about keeping your Bitcoins offline as much as possible?  It is difficult to hack a machine that is off.  Theft is really your only course of action.   Encrypted hard-drives help keep that safe in the event of theft.  Proper policy is the large step in the right direction.

[mistfpga]

Interesting.

Although for $20K it likely isn't cost effective for most operations.  A "poor mans" version would be a dedicated locked down box.

very true, however seeing as now there is a proven market for Bitcoin Finacial Services, these services should be using these sorts devices to protect the coins of their customers.  It is like the old addage, "best practices are for those who are not capale doing thier own due dilligence". 

The good thing about spending this amount of cash is you have the potential to get insurace from proper finacial brokers and you can publish your security model, in all its gory details so it can be peer reviewed.

Web Application server connects to a tx server via serial port.  The tx server is the "poor mans's HSM".  It has no uncessary applications, no ssh, no remote login.  It simply communicates to the outside world via serial port.

It gets inputs from the serial port, runs them against its "rules", builds tx from private keys and outputs signed tx.  Changes to the tx server could be accomplished by simply providing it a new complete distro.

Can this be done already? using the armoury client? rather than usb from machine to machine, just connect them over serial with a python/perl socket script. still doesnt help too much with physical access.  still useful for protecting a home setup.  I might see if I can get this running later today.

Interesting.  I will look into it.  The yubi HSM though would not be useful for securing a hotwallet.  It has no ability to run internal code and merely provides secure key generation and storage.  If the host has access to the yubi then so does the attacker.

On edit: The docs on the edge don't seem to indicate the ability to run internal custom code but maybe I am missing it.

yeah the yubi looks very awkward to set up (you might be able to do this with 4 yubi keys and their one time password feature, but I cannot be bothered to think about it.)

I would be interested in what you look up.  I might be able to give you remote access to the edge when it gets here (no promises though)

Running custom code is part of the nCore interface. 

http://www.thales-esecurity.com/Resources/Solution%20Sheets/Options%20for%20nShield%20HSMs.aspx

CipherTools Developer Software
Thales HSMs can be integrated with many business applications through standardized APIs (Application Programming Interfaces), including Microsoft CAPI/CNG, PKCS#11, Java JCE and OpenSSL. In some cases, the official standards these interfaces support are too limiting, so Thales HSMs also offer the nCore interface for advanced integrations. CipherTools is for all application developers regardless of whether they’re using nCore or a vendor-neutral API. It contains documentation and example code to enable developers to take full advantage of the advanced functionality offered by nShield HSMs.

If you look at the bottom of that link there is a supported feature list for the nShield series HSMs.  All of them (the nShield range) support custom code.  Although for running in an off site location I would probably only go for a solo (i am pretty sure it is cheaper than the netHSM or nShield Connet).

the edge is not completely weak, iirc it will only run signed code... this is a pretty good defense, not amazing, but it is alright.

cheers,

steve

[DeathAndTaxes]

Can this be done already? using the armoury client? rather than usb from machine to machine, just connect them over serial with a python/perl socket script. still doesnt help too much with physical access.  still useful for protecting a home setup.  I might see if I can get this running later today.

I don't know enough about the armory client to know what kind of API access it has.  It can be done with the Satoshi client though.  Physical access can be hardened through the use of secure chassis (lockable and not w/ those $0.20 "computer locks"), locked co-location cage, and intrusion tamper switch.  Connect the intrusion tamper switch to the motherboard hard reset pins.  Open case and motherboard reboots disabling access to the encrypted wallet. 

Like I said poor mans solution  but a 1U wallet server w/ hardened chassis, IPMI, a linux distro, and serial port should be <$1K even with niceties like redundant drives/PSU, TPM (to prevent decryption of wallet file even if stolen inroute and passphrase leaked/stolen).  I haven't checked transaction throughput on an atom based system but if it is sufficient you could likely build a "HSM in a box" for ~$500.

Running custom code is part of the nCore interface. 

http://www.thales-esecurity.com/Resources/Solution%20Sheets/Options%20for%20nShield%20HSMs.aspx

CipherTools Developer Software
Thales HSMs can be integrated with many business applications through standardized APIs (Application Programming Interfaces), including Microsoft CAPI/CNG, PKCS#11, Java JCE and OpenSSL. In some cases, the official standards these interfaces support are too limiting, so Thales HSMs also offer the nCore interface for advanced integrations. CipherTools is for all application developers regardless of whether they’re using nCore or a vendor-neutral API. It contains documentation and example code to enable developers to take full advantage of the advanced functionality offered by nShield HSMs.

The quote doesn't clearly indicate if that code is running INTERNALLY on the HSM or if "nCore" is marketing for their API to allow external programs to interface w/ the HSM.  The former is awesome but the later doesn't provide much additional security.  Sadly it is tough to get any real information from thales website.  I hate marketing "buzz".  Come on Thales, show me the specs, the API, code examples.  How much internal memory do the HSM have, how much nonvolatile storage, etc.

[spiccioli]

DeathAndTaxes,

there's one thing I don't grasp...

What stops someone having access to the web fronting compromised machine from sending a command through the serial port to the hardened PC requesting an 18K withdrawal?

spiccioli.

[SgtSpike]

DeathAndTaxes,

there's one thing I don't grasp...

What stops someone having access to the web fronting compromised machine from sending a command through the serial port to the hardened PC requesting a 18K withdrawal?

spiccioli.

That's where your customized secure code comes in.  You can refuse withdrawals over a certain limit, or program in some sort of two-factor authentication for withdrawals, or even just a simple password needs to be input before a set of withdrawals is processed.

If you keep the wallet file on the web server though, all it takes is a compromise of the web server and the criminal can copy the private keys and do what he wants.

[spiccioli]

That's where your customized secure code comes in.  You can refuse withdrawals over a certain limit, or program in some sort of two-factor authentication for withdrawals, or even just a simple password needs to be input before a set of withdrawals is processed.

If you keep the wallet file on the web server though, all it takes is a compromise of the web server and the criminal can copy the private keys and do what he wants.

SgtSpike,

ah OK, yes, now it is clear.

Thanks.

spiccioli

[DeathAndTaxes]

DeathAndTaxes,

there's one thing I don't grasp...

What stops someone having access to the web fronting compromised machine from sending a command through the serial port to the hardened PC requesting a 18K withdrawal?

spiccioli.

Business rules.

The serial port wouldn't connect to the bitcoind directly.  Instead its access would be limited to only a process/service/daemon which validates tx requests (not commands) against business rules.  

Just brainstorming here but as an example imagine the tx server had these rules:
1) Max Tx value is 1000 BTC (or whatever limit is useful for 90% of users).
2) TX value > 100 BTC results in 60 minute delay before return the signed tx.
3) TX value > 300 BTC reults in 180 minutes delay before returning the signed tx.
4) Certain conditions which indicate either a probable attack or software malfunction trigger an automatic lockdown.**
5) If total tx value in last 15 minutes exceeds preset limit delay all tx an additional 30 minutes (even those under 100 BTC) and notify admins.
5) TX volume 50% higher than 7 day peak causes a security hold. *
6) TX value 50% higher than 7 day peak cause a security hold. *
7) TX velocity increases by more than preset limit causes a security hold. *


Web front end -> API via serial port -> tx processors -> wallet.

* Security hold:
On a security hold the server remains online and queues up tx requests but stops returning signed txs.  Admin are notified and the server remains halted until it receives a security key (which isn't stored on front end server).  This would be useful to put a "man in the loop" when volume is high.  

Beep beep beep.  Business rule violation:  TX volume in the last 24 hours (487 requests) exceeds 72 hour peak (350 requests).

 It is possible this is just a valid but unusual event.  Admin can check the tx logs, access logs, and if everything checks out provide the security key to resume operation.   Grocery stores use a system like this.  Ever seen a cashier mess up and she needs the manager key?  Same concept.  It limits the potential for fraud based on business rules of unusual behavior.

** Lockdown:
In a lockdown (possible intrusion) the server powers off.  Resuming tx processing requires powering on the server, providing the startup key, and optionally waiting for a preset delay.  All admins are notified of the lockdown and the front end server is halted also.   "Red alert.  Shields up".  

For more security the box could be designed to allow a security hold or lockdown command to be issued from any internet connected computer.  The command would need to be digitally signed to prevent abuse but this would provide a "manual override".  Zhou noticed the admin reset and the "weird" tx within minutes.   Even if the tx server hadn't realized it was fraudulent if it was still in the delayed signing queue Zhou could have forced the server into lockdown and saved six figures.

The rules could start out simple and get more complex as the business evolves.  The goal is to prevent direct access to the keys or funds by limiting unusual activity even when someone has "access".  Does anyone think in a traditional bank if a teller uses their terminal to make a "valid" $100,000,000 cash withdrawal the bank's vault automatically swings opens and bags of money roll out?  Meanwhile security guards and bank managers stand by helpless as thieves fill a dumptruck with cash because "she had the right code".

[JusticeForYou]

D&T seems to know a little something here.

I would have a team write a withdrawal.log file that is read from another server somewhere that reads the withdrawal requests, verifies against the preknown accounts (hashed of course), and executes the outgoing to the network from a completely different location. This Floating 'float' account can be moved at intervals to provide another level of security.

OFC, this doesn't prevent messing with the positions of people in the code but it would help to not lose $300K.

D&T has already pointed out in another thread in a strongly worded opinion things that should have been learned. I would have thought $300K would have taught somethings if at least another degree.

[spiccioli]

D&T,

very clear, thank you.

spiccioli.

[Sukrim]

4) Certain conditions which indicate either a probable attack or software malfunction trigger an automatic lockdown.**

So just by writing a program that requests 1000 withdrawals of 1 Satoshi (which would very likely look like a malfunction or an attack) one could bring down the site potentially for hours? Nice!

I would still go with my initial idea:
Pay-ins are credited as soon as they are confirmed
Pay-outs are processed every x hours en block (saves on Tx-fees, keeps wallet offline etc.etc.)

Instant payouts will always be a security risk and unless you have a site that pays out less than a handful of BTC every day anyways, you really should just use an offline wallet, collect transactions and manually create a sendmany every once in a while on an offline PC.

[DeathAndTaxes]

So just by writing a program that requests 1000 withdrawals of 1 Satoshi (which would very likely look like a malfunction or an attack) one could bring down the site potentially for hours? Nice!

The tx server only accepts inputs from serial port hardwired to the web server.

If the attacker has admin access to the frontend webserver and is executing arbitrary code then the site likely SHOULD be locked down don't you think?

[Sukrim]

I would not execute code on your server, I would (if you are MtGox for example) spam you with withdrawal requests.