Bitcoin Forum

In recent times the Bitcoin community has suffered from major service providers being hacked and coins being stolen. Mitigation techniques include the use of hot/cold wallets and secure coding techniques, but secure coding does not help much if the infrastructure providers themselves are breached.

One possible solution proposed by Gavin is having a "monitor" server and multisig transactions. The monitor would check the transactions submitted to it for consistency and only sign if valid. This is a useful approach but suffers from some limitations:

• It's expensive (you need to pay extra hosting costs)
• It is tempting to reduce management overhead by duplicating configuration, eg, using Linux on both systems instead of Linux and Windows, or having the SSH keys for each in the same place, or using the same VPS for both. That reduces the security benefits dramatically.
• There are likely to be bandwidth/latency constraints between the two hosts that will make verification of the transactions harder

Trusted computing lets us solve some of these problems by allowing the creation of the "monitor server" inside the untrusted, possibly compromised server. The bandwidth between them is similar to your regular memory bandwidth. Only one piece of hardware is needed. The monitor runs an absolutely minimal piece of code, not even including a regular operating system, thus there is little or nothing that can be hacked. The monitor is resistant to physical attack by bad insiders.

I'd like to convince you of the benefits of this approach in the hope that an intrepid entrepreneur may start offering shell accounts on TC servers to the Bitcoin community, along with tutorials and libraries to assist in the development of secure monitors. By reducing the costs of the dual-sided approach I hope to see it become a lot more common, and accessible even to small or amateur operations that would likely not consider a multi-server setup.

You start with a platform that implements the Trusted Computing Groups technical specifications, for example:

• Dell PowerEdge T105 server
• HP dc7800 workstation
• Dell Optiplex 740 workstation
• Dell Optiplex 755 workstation
• Lenovo T400 laptop
• HP 8530p laptop
• HP 8540p laptop
• HP Elite 8100 desktop

The specs I'm talking about are open and you can read them directly, but I recommend a decent book like this one (http://www.amazon.com/Computing-Initiative-Building-Blocks-Trusted/dp/0976483262) to become familiar with the underlying technologies. I only offer a brief overview here.

A TC platform contains a chip called a trusted platform module (TPM), which can be standalone but these days is often integrated into the southbridge. The TPM is a tamper resistant crypto chip. It's important to understand that a TPM is not a crypto accelerator. TPMs are very slow. Instead they provide a repository for cryptographic keys that is difficult to compromise even by a physically present and determined attacker.

Together with some special CPU instructions, it's possible to create isolated software worlds which are extremely difficult to attack, even by somebody who has compromised the host OS. It works like so:

• Code running in the (possibly compromised) regular operating system environment loads sensitive application code into memory, for example, in Bitcoinicas case it might be a part of the trading engine, in slushs case it might be the code that examines some data files to create the payout transactions. It also loads any data files that program might need, typically these will be encrypted.
• You execute the magic instruction. On AMD this is the SKINIT instruction, on Intel chips it's the more complicated SENTER instruction. What happens next is a bit convoluted, but to simplify, the CPU performs a kind of reset in which the host operating system loses control - interrupts are suspended, other cores are shut down and execution resumes at the start of the sensitive code block. Crucially as a part of this process the TPM is initialized with hashes of the sensitive code and any other code in the system that can affect execution, like the BIOS.
• The TPM has the ability to "seal" encryption keys such that they are only released from the chip when the CPU is in particular states. For instance you can seal a key under a hash of the sensitive code. It then isn't possible for the host OS to make the TPM cough up the key. The sensitive code asks the TPM to unseal the key, and it does so because the CPU has already (electrically) informed it that the hardware is in the expected state.
• The sensitive code then performs whatever calculations it needs, for example, it may decrypt the wallet loaded into memory for it by the untrusted code, create and sign some Bitcoin transactions, which it then deposits in a region of memory that was previously reserved for it by the untrusted/possibly compromised code.
• The code wipes the memory it used, then exits the secure state with the SEXIT instruction (or AMD equivalent).
• The untrusted host OS can then take the signed transactions and relay them to the Bitcoin network.

Actually implementing all this is complex. Fortunately some nice people have written Flicker (http://sparrow.ece.cmu.edu/group/flicker.html) which makes the process easy. Flicker is very small, which means it's easy to audit and convince yourself there are no flaws that might allow the host OS to make the isolated world do something unexpected.

The nice thing about this approach is that TC is a very general tool. For instance it has applications in intelligent agents, secure timestamping, oracles and other topics that may be of interest to Bitcoiners. The isolated worlds can create cryptographic proofs of their execution that are verifiable by others - this means you can dramatically increase your trust in a third party, providing an interesting and often cheaper alternative to the "quorum of independent participants" paradigm.

There are some open questions that need to be solved first:

• Services like Linode reduce their costs by virtualizing the hardware. It's possible to virtualize TC hardware, indeed, the open source "tboot" project from Intel suspends the host OS and then literally installs a Xen-based hypervisor underneath it on the fly, allowing an entirely separate copy of Linux to boot which runs in parallel to the first one - all tracked by the TPM. However this increases complexity a lot! First versions of TC services would probably want to find a simpler way to support virtualization.
• How do you make writing secure monitors easy for people who are not necesssarily comfortable with C or C++? Flicker helps a lot but it still assumes fluency in low level programming.
• How do you abstract and simplify the techniques needed to track external state. For instance, a pool server needs to track who is owed what payouts. Even if an attacker can't compromise the wallet, they could submit bogus payment data to the monitor and trick it into signing away its money. The monitor could potentially store and manage the entire database itself, but there are often better ways. The trick is to make them simple. This is a problem shared with the multi-server approach. Common design patterns would help a lot.

I plan to research some of these topics in the coming months.

This sounds promising.  Subscribing.

Finally, TC being used for good instead of evil!

Great ideas, Mike. I think the "use two servers" approach will happen first (you typically want at least two servers anyway just for backup and redundancy), but trusted code running in a TPM could be an extremely powerful and cost-effective building block for much more secure solutions.

RE:

That seems like the hardest problem.

Does the TPM have access to a guaranteed-to-be-unhackable time source? If it does, then a policy like "do not approve more than X bitcoins worth of transactions in timeperiod Y" should work well enough to prevent more than petty theft, assuming you've got some other way of watching the blockchain for unauthorized transactions and shutting down the compromised machine or using an emergency offline backup key to move the funds to a non-compromised wallet.

What if you periodically feed the hot wallet from the outside, only from a secure server,
(offline transactions ?). Can this be a simpler approach to limit loss, or is it just nonsense ?

If I'm understanding this correctly, it's similar to the plans for Open Transactions. They will have several servers running OT and will only sign a multisig transaction if they agree.

Trusted computing would be more open-ended, since you could run whatever code you want. Open Transactions would be cheaper and more secure, but one would need to build everything out of smart contracts. Could be useful for something like Bitcoinica, though.

I thought of a similar concept, the idea in my head being that the trusted box only have an RS232 connection to the world, using a port that is completely nonresponsive until a process of some sort opens it directly (e.g. No shell, no getty or opportunity to ctrl-c the controlling program back to a shell prompt etc.). Such a box could use plain untrusted hardware off the shelf.

With such a setup, control of the box remains airgapped while leaving a connection just big enough to pass the activity being validated and the signatures.

If a trusted time source is needed just hook a common GPS receiver to another serial port, which reports UTC derived from the GPS system.

Um... yes. The bitcoin block-chain.

No.  The TPM has no time data beyond system clock.

Are there any trusted time servers which sign timestamps and provide a public key for verification.  If so the TPM could require timestamps which it then verifies against a stored copy of the public key (which it keeps encrypted so only TPM can access).   Failure to authenticate timestamp, failure to provide timestamp, or failure to meet payment rules would result in TPM denying all transactions until it receives a verifiable timestamp 24 hours in the future.

On edit: as indicated above DUH the blockchain.  Close enough for this work and nearly impossible to spoof at least economically.

Agree that you could just use the block chain itself if you wanted to throttle payouts. More complex conditions would have to rely on a challenge-able trusted clock, ie, one that returns signed timestamps including your nonce.

I think a more interesting exploration is how to have the monitor linked to your database such that it can know the DB has not been tampered with, without maintaining the entire DB itself. Is it possible to do for general data models? If you simply replicate the entire database the monitor has no guarantee it's seeing the database as the primary server sees it. That means the monitor has to examine not only the snapshot state of the DB but actually, every single mutation. It's not easy. Probably some research on this topic can be found in the literature.

Alternatively, having the untrusted side provide a regular key/value store and have the secure side simply return encrypted/signed key/value pairs to update could also work. So then you don't have the full overhead of a database in the secure space (disk management, cleaning, compacting, etc) but you can still do arbitrary data storage there.

Other problems that need to be considered:

• Backups. You need to be able to export the secure data, but that must not become a weak point for attack. The secure code could maybe only export data encrypted under a public key that is hard-coded into it. You'd have to keep the private part safe somewhere unhackable (like a piece of paper). The problem then becomes how do you restore the backup .... you'd need a different private key in the software and keep the public part safe somewhere unhackable.
• Upgrades. Sealing data to the software means a compromised host OS can't access the wallet by changing the code that runs, but it also means YOU cannot change the code that runs because then you'd lose access to the data. You could handle it by seeing this as a specific case of backup/restore, but then you are frequently loading the backup keys onto some internet connected device. A possible alternative is to sign new versions of the code offline, like you'd take the new binary on a USB stick to a device, sign it with a key intended only for that purpose, and then the previous version could export its data sealed under a key available only when the CPU is running the new code (which is known to be valid because it's signed). That way the sensitive data is never readable by the developer, even. It just moves from signed version to signed version. You can use the TPM monotonic counters to prevent downgrade attacks in the case of a secure monitor that had a vulnerability.

I was thinking of a kind of replay attack:

+ Control the TPM's view of the world (e.g. make it seem like it is Jan 1, 2010)
+ Get the TPM to sign a small transaction, shut it down.
+ Increment time, get it to sign another transaction
+ Repeat.

Replace "time" with "blockchain" and you've got the same problem: can the TPM know that it's view of the external world is correct?  If it sends a nonce (to prevent replay attacks) to some external service that adds a timestamp and signs it with a public key known to the TPM code... then we're back to using two different servers.

The TPM provides counters that are guaranteed to only increase, and which can be incorporated into sealed state, exactly to avoid rollback/replay attacks.

So what you can do is have the block chain be presented to the secure code. It looks at the timestamps and performs whatever throttling it wants. Then it returns in the sealed state the current chain head hash, and the current value of the monotonic counter which it then increases by one. The next time it runs, it checks that the counter matches. If it doesn't you are being presented with old data.

One problem is that TPMs are quite slow and limited, so techniques to use their limited NVRAM write cycles most effectively can be obscure. This is why we'd need libraries and tools to make all this easy. It's harder than the two server approach at first, but once set up it should let anyone who has access to the relevant hardware have a higher degree of security for less effort and cost.

There's a paper on how best to achieve state continuity here:

http://www.ece.cmu.edu/~jmmccune/papers/PLDMM2011.pdf

I knew you were going somewhere with it I just couldn't see it.  Yes it would be nice if TPM had secure timer but IIRC (from a non-Bitcoin project) it doesn't.

I wonder if building a secure hardware timer is that cost prohibitive.  The TPM could simply get verifiable timestamps from a second module.

Interesting read but this seems like a killer for what we are looking at:

If incremented on each block it is only good for ~1.9 years.  The paper shows a novel approach to only incrementing the counter on boots but I don't see how they helps us for our issue.

Good point. You can select arbitrary tradeoffs. Increment the counter every 10 blocks instead and now it lasts nearly 20 years, which is probably longer than the expected lifetime of the rest of the hardware. It means you can potentially replay blocks within the last couple of hours, but depending on your throttling scheme it may not be an issue.

I'd hope that if one day people were using TPMs regularly better/stronger chips would come onto the market as well. Until then there are various convoluted workarounds to get more out of the limited devices.

Yeah that is a good point.  Even incrementing once every 3 blocks gives you 6 years.  TPM module is usually replaceable (simply plugs into MB via 10 pin header) and only costs $20 or so for max security buying a $20 part every year or so isn't a bad tradeoff.  My MB has a header for it.  I thought about buying one just to play around.

On edit: Link removed to protect the uninformed from themselves due to potential system damage (see post below).

I'd caution against just buying a TPM. A full TC setup requires a TPM, but also support from the CPU, motherboard, BIOS, etc. Intels implementation in particular is notorous for bricking motherboards if you don't use the absolute latest versions of the BIOS. And by "brick" I'm not using it in the script-kiddie sense that you so often see these days. I mean it actually turns the motherboard into a worthless chunk of metal that has to be thrown out. There are also systems that theoretically support TC but contain serious hardware bugs which render the setup worthless. It's really your best bet to purchase a full system that's been tested and known to work.

I've been wanting to play with these technologies for ages, but never had the time. That said, I'm in contact with one of the foremost researchers (one of the guys who did Flicker). He is interested in the intersection of TC and Bitcoin and if somebody serious steps up, he'd be willing to provide accounts on TC systems that have TrustVisor installed. TrustVisor is pretty much the cutting edge of minimally-sized TCBs and it makes it easy for code to switch in and out of secure space - you just separate your (c/c++) program and then you can make regular function calls into and out of the monitor (they call it a "piece of application logic" or PAL). TrustVisor handles parameter marshalling and other things for you. It has good performance and it virtualizes the TPM in clever ways so you can get more performance out of it.

If anyone has the necessary skills (C++/assembly should not scare you), time and interest, let me know and I'll put you in touch. I'm happy to answer any questions about this sort of technology as well.

BTW it isn't only for servers. You can use it to make end-user wallets that can't be compromised by malware on your laptop. That's actually a more straightforward problem than the server case because the secure code can talk to a human via the screen to confirm the action - there's no need for throttling or risk analysis on the transactions.

Good warning although I would imagine that since the MB has a header, and BIOS includes options for enabling TPM they must be anticipating some users would take the option.  Still thanks for the warning.  If I brick it then I only have myself to blame.

Still larger point was that a replacement TPM isn't that difficulty to obtain so even using them as a limited lifespan product would be viable.  It all depends on how much money you are protecting.  For personal wallet an increment every 10 blocks is likely fine.  For a Bitcoinica wallet increment on each block and buy a replacement $20 part every year to be safe.

You could have code on the TPM that checks that the said bitcoin block is next in the chain, form the last block hash that it keeps in secure memory.
Since blocks have a timestamp, this could be used as an 'ever increasing' date system.  Giving a general guarantee of the date.

A more advanced chip could make sure that the transactions it signs are from blocks that it has ‘seen.’

I think that in the future the block chain can be used as a secure time source.

I am curious why a TPM and all kinds of high-tech kernel wizardry with elaborate cutting edge hardware would be more desirable than a low-tech dedicated computer doing this same thing over RS232 where the airgap is visible, easily explained, and easy to comprehend?  Not to mention the hardware very common and cheap and the validation rules can be written in any well-known language, high level or low, for any platform or OS.  RS232 is oldschool and boring, but that's the point - it'll never be overly complicated, and compatible hardware is everywhere at every price point and in every form factor.

RS232 essentially guarantees that the access footprint is slow, with no remote access to the OS, and supports no functionality that hasn't been explicitly implemented.  It keeps the computer pretty much airgapped, wired just enough to do its intended job.  On the other hand, if you need a special kind of obscure hardware that has an obscure chip in it and requires obscure OS support, and someone writes open source support for it, hardly anybody's going to bother with it because the bar of them caring enough to acquire the special hardware is that much higher.

For an average setup (even one Bitcoinica's size), Zhou need only two computers at his house.  All he needs is a "hot" client that runs on some internet-facing computer and passes the transactions across RS232 to the "secure" client that hosts the keys and signs the transactions.  The hot client relays the signed transactions back to the network.  If the secure side needs to know about blocks and the block chain to make its decision (or unconfirmed transactions), then the RS232 protocol simply shall contain a "here's an incoming block/transaction" message.  If his secure computer decides it wants human interaction before signing a transaction (example: unusual big transaction), then it can blow bells and whistles out the local speaker, or ask the "hot" computer to send him an SMS.

If you look at the thread from slush, he says he used Linode because connectivity from Prague is poor so he does not want to run the pool there. VM providers are very popular these days, I guess because many people don't want to run servers from home. Having a machine with no remote access as the monitor basically requires it be physically close to you, either at home (limited bandwidth, running servers may violate consumer ISP ToS), or at a local colo (prices/connectivity may not be competitive).

The hardware isn't really that obscure, there are over 200 million TPMs out there. The combination of TPM+all the other stuff is a bit more exotic but still available from several well known manufacturers.

Once the initial setup work is done, it can be made available to everyone relatively easily. People can SSH in to their account, run their site and have the sensitive parts run in the isolated space where even the people with physical access to the box would find it hard to compromise. But you can still get the benefits of having other people manage power, cooling, connectivity, etc.

That said, until the work is actually done, I agree that the two server solution is more straightforward.

I would point out there is a 3rd option.  Secure colocation.  

Maybe it didn't make sense in slush case but for Bitcoinica there is simply no excuse.  They were holding $250K with $50 in security.  It would be like locking up a briefcase holding $250K in cash with a bycicle lock.  No jeweler would have done that, no gold miner would do that.  The "safe" should be priced accordingly to the assets it is protecting.

So not to hit zou while he is down but hopefully that is a wakeup call.  $250K in digital assets should be protected like any other assets worth $250K.

Renting private cage space secured with private key and ID verified access is sub $1000 in many parts of the world.  Most provide secure shipping and remote installation of servers and have bonded employees.  Data security during installation can be ensured with things like truecrypt.  Features like KVM over IP, remote switched PDUs, and private firewalls allow you to build a physical fortress around your hardware which holds your information[/b] that ensures there is no "backdoors".  The cage would never even need to be opened except to replace hardware.  The amount that was stolen could have paid for 20 years of such remote co-location. 

Information Security starts with physical security.  Using a virtual machine is fine to house your personal wallet to $100 in assets.  $250K in assets should have enterprise grade PHYSICAL SECURITY.  That doesn't mean things like 2 factor authentication and TPM aren't important.  They are because future thieves will only become more resourceful but one needs to start with ironclad physical security.

+1

It's very easy to start Bitcoin businesses ... securing them as they explode and grow larger is difficult sometimes.
I'm sure Gavin is much less devastated about the few BTC stolen from the faucet than Zhou. What's fit for a freemium non-profit website like the Faucet isn't what's fit for a medium-sized business, isn't what's fit for Mt. Gox, Bitcoinica or any semi-serious Bitcoin exchange for that matter.

You want to implement something like this, but without the secure switch and additional physical servers?
http://www.nsa.gov/ia/media_center/video/orlando2010/flash.shtml

Apologies for resurrecting this old thread, but I wanted to mention a new development. The people that brought you Flicker and TrustVisor, both mentioned by Mike, have a new project out. xmhf (http://xmhf.org) is a hypervisor framework built around Trusted Computing. Its main advantage is that it works on both Intel and AMD, but you still need a newer, relatively high-end machine. TrustVisor has been ported to xmhf, so now it works on both architectures whereas previously it was just AMD.

I agree with the comments above that TC may not be quite right for bitcoin. For one thing these secure program compartments can't do any I/O directly. They have to rely on the insecure code to relay data, although crypto keeps the data secure. So if we wanted the user to approve a transaction, you'd have to send the data to a secure device for approval. In which case you might as well use multisig or even just keep all the keys there.

You could try to implement a self-contained policy like rate limiting, although as discussed above you need a secure time source and state rollback protection. I'm worried that using the blockchain as a time standard might be vulnerable to timing games by the untrusted code, although there might be mitigations. A couple of other potential sources of secure time: the network time protocol, which is how a lot of computers keep their time in sync, has a crypto layer. Unfortunately it doesn't seem suitable for public use, although I found out the US NIST will supposedly supply you with an authenticated time if you go through a complicated application process, http://www.nist.gov/pml/div688/grp40/auth-ntp.cfm (http://www.nist.gov/pml/div688/grp40/auth-ntp.cfm).

Thinking way outside the box, you could open an SSL connection to a web page that's updated frequently, and use the Date: header from the http response. You'd hard code the CA root cert and all the relaying could be done by the untrusted code and still be secure. I've tried this with https://google.com and the time is pretty accurate. TrustVisor includes a version of openssl so this would seem to be feasible.

But even if you got rate-limiting working, the untrusted code could substitute its addresses for the target addresses, or maybe just skim a percentage off each transaction, hoping to evade detection. A lot of things have to go right for the created transaction to match the user's intentions. Assuming a malware takeover and still trying to protect the user is aiming too high IMO. Maybe we can limit the damage though.

Superb, Jonathan got TrustVisor open sourced. For a while I think they were considering keeping it proprietary. I talked about TrustVisor with him before, it seems they went even beyond that. I will explore it later.

Thanks for the pointer, Hal. I agree with everything you wrote: I think the design of a supervisor server suitable for use on exchanges/trading platforms/other businesses with hot wallets is very much an open question. Once you have figured out such a design though, moving it into a PAL seems like a fairly straightforward transformation. Issues like how to do rate limiting, to stop a hacked system replacing addresses all need to be resolved for a multi-signer solution to work anyway.

Yes, finding a source of secure time is annoying. Doing an HTTPS request to google.com works - I should know because I've written software that does that before :) Of course google.com is really not supposed to be a global time server, but serving 404s with a date header is fortunately very cheap. IIRC every Google server is synced to GPS time using an internal NTP pool so it should always be accurate.

Throttling/limiting the damage is a valuable goal in and of itself, as it gives you much more time to notice something has gone wrong and throw the off switch. Even if you don't have any automated way to detect badness, the ground truth is often user complaints. If the losses are small enough you can make things right from your profits.

received as a PM, thought I'd post in the thread:


I definitely think so...

There needs to be a base use case where the box is a wallet and does nothing more than ask a user to confirm a proposed transaction on its own display, and signs the proposed transaction if the user agreed to it.  Basic messages to the box would include "getnewaddress", "please sign this transaction", and "here is a new transaction/block I heard on the network".  Basic messages from the box would include "here is a new receiving address", "here is your signed transaction", and "I refuse to sign this transaction".

Then the protocol and/or app could be added with user extensions (different for everyone using it for more than a wallet) to implement the security policy of the operator's choice.  The app could be extended to limit total withdrawals within a 24h period without console intervention, or the protocol could be extended to allow the wallet to learn PGP public keys and then require PGP signatures for user withdrawals, or whatever.

The advantage of RS232 is that it's dumb, slow, universally compatible, and well-understood.  The "black box" can be implemented on another PC, a credit card machine, a Raspberry Pi, or even a microcontroller.

TC doesn't require physical access to the server. Your RS232 solution does. 

Apart from being more costly than using a dedicated hosting service, this simply isn't practicable for a lot of bitcoin entrepreneurs, eg. those with a nomadic lifestyle.  Where is Zhou Tong going set up this machine? In his student dorm?

You don't need a serial cable. A regular network cable is plenty secure enough as long as you tightly control the traffic allowed to it from other machines. In this case it sounds like there was an SSH daemon listening and that is how the attacker was able to log in. So don't do that - require physical access to change the configuration of the server.

The reason for investigating TPMs and related technologies is that if they are quite general and easily integrated by server vendors, so it's cheap and totally reasonable to provision yourself a trusted computing environment in a remote datacenter then do everything via SSH - whilst still having similar security properties that a piece of dedicated hardware would give you. Yes, it's advanced and exotic today, but we're building the payment system of tomorrow, right? So it's worth thinking about these things.

In any event, this currency exchange was still running on Linode months after that provider was found to be unable to withstand a motivated assault. I think these kinds of hacks can be avoided by people doing basic due diligence if they're about to (reality check!) entrust large sums of money to a random guy off the internet!