Best protocols for cold wallet, hot wallet. We are building an exchange.

[throwaway084575]

Hi good folks,

We are currently building an exchange but up until now have just been focusing on the logic of the trading engine.  I have been tasked with the goal of creating a bullet proof system of cold and hot storage for the exchange. I'm not a programer but have been assigned to just collate different technologies and options to that then the development team and management can go through and discuss each option.

The ratio of getting the convenience / Timeliness / security correct is the hard but the goal is security more than .

I wanted to get some ideas from everyone on the bast ways to go around this. I got massively stung by Gox and have made it my mission never to let a goxing happen to our customers.
But the hard thing is knowing all the tech out there. Because it is all growing so fast now.

I know that http://bitcore.io have some great tools.
so does https://api.trustedcoin.com/#/
So does blockchain.info
So does Armory


Anyway
What is the best way to do this?
We have a very talented development team so nothing is off the table.


So here is one basic idea

--------
HOT WALLET
Using the blockchain.ifo API to generate addresses for each user to deposit funds.

Pay out addresses get locked in on our system and can only be changed through re uploading ID for us to cross check or 2 factor Auth.
A maximum payout per 24h.

COLD WALLET (addresses will be publicly available for audited for proof of solvency)
The Cold wallet is stored on a computer that has never been online and is used to sign transactions using a USB stick. (Armory wallet) to then send funds to the hot wallet.
The cold wallet is backed up twice a day on multiple hard drives.

I would love this to need 2 out of 3 keys so that both directors need to sign transactions every day to minimise trust.  The 3rd is with our Lawyer.
--

Cons and pros

Cons:

• Utilising a 3rd party to handle the hot wallet opens us up to trusting their security. Blockchain.info has proven them selves to be very competent and trustworthy but mistakes on their end can and could happen
• Blockchain could go offline for technical reasons leaving our customers angry at us because they can not withdrawal funds until blockchain.info is back online.
• Cold wallet USB signing could leave us open to a virus that embeds itself in a USB stick to then infect our cold storage offline computer. (need a USB condom LOL)
• Having a hot wallet leaves us open to that getting stolen

Pros

• A hot wallet lets our customers have instant withdrawal to a certain amount
• Cold storage can transfer a % of funds to the hot wallet on a daily basis + any extra that where over flow from the previousness day withdrawal requests.
• If a director is sick, goes on holiday or dies then a 3rd key is available.

[1714041657]

1714041657

[1714041657]

1714041657

[doof]

I think you are out of your depth from the questions you're asking.  Cold wallets should never be stored on a computer, so backing up twice daily isn't really a solution, and means you are creating a hot wallet.

If you have a good dev team, don't use a third party like block chain info.  They can get DOSSed, disappear.. what ever.

Creating addresses in code is easy.  Signing takes a bit of time but doable.

[doof]

I got massively stung by Gox and have made it my mission never to let a goxing happen to our customers.

By using a 3rd party, you can "gox" your customers if they go bust.

[ThePurplePlanet]

Use bitgo http://www.youtube.com/watch?v=DuinfYuE8Ng One signature to you one to bitgo and one to the customer and two out of 3 spend the transaction usually you and the customer

[TierNolan]

COLD WALLET (addresses will be publicly available for audited for proof of solvency)
The Cold wallet is stored on a computer that has never been online and is used to sign transactions using a USB stick. (Armory wallet) to then send funds to the hot wallet.
The cold wallet is backed up twice a day on multiple hard drives.

I would love this to need 2 out of 3 keys so that both directors need to sign transactions every day to minimise trust.  The 3rd is with our Lawyer.

I don't think Armory supports it, but there is no reason you can't have a 2 of 3 multi-sig watching only cold wallet.

They do support 2 of 3 backups for the root keys though.

You could give each directory a share of the root key and print it to a piece of paper.  Any 2 of them could recreate the root key.

Even more fancy would be to give each directory 3 shares each.  You could then require 5 of 9 shares to release the key.

This allows each director to store the shares is more than 1 location, which increases physical security.  If a share is stolen, then there is no loss.

If you do create a cold wallet, you should test the process with a small amount of money first. 

In addition, don't store all the funds in a single output.

If there is a bug in the program that causes unspendable outputs 1% of the time, then spreading the funds over 100 coins limits the risk.

If you do 20-30 small transfers, then the probability of the code failing is pretty low, but it still could have a hidden bug that is rare.

This is where using Armory has an advantage over your own code.  It has been tested, and if there are bugs they are (very) low probability ones.

[throwaway084575]

I think you are out of your depth from the questions you're asking.  Cold wallets should never be stored on a computer, so backing up twice daily isn't really a solution, and means you are creating a hot wallet.

If you have a good dev team, don't use a third party like block chain info.  They can get DOSSed, disappear.. what ever.

Creating addresses in code is easy.  Signing takes a bit of time but doable.

I'm not sure, This computer would never have seen the internet so I would say it is as cold as paper.

I am simply collating different methods from you guys that maybe the dev's have not thought of yet.

The team lead also stated that 3rd parties are not the way to go but I just wanted to throw ideas out there. Anything and everything.

[Rannasha]

- Don't use blockchain.info (or ideally any other third party) to process transactions. Blockchain.info has had periods of downtime sometimes lasting for several hours or more. Use your own bitcoin client instead.
- Keep the hot wallet on a separate machine, not on the webserver. This way you can close the wallet-machine almost completely from all traffic, only allowing very specific requests from the webserver machine.
- If you set up your hot<->cold transfer system properly, there will be no need for periodic backups. Set up the hot wallet in such a way that it only sends out outputs of a fixed size (say, 10 BTC) to the cold wallet address. If you then always send complete batches of 10 BTC back to refill the hot wallet, then there will be no need for change-addresses to be generated and your single cold wallet address is sufficient, which means you only need to backup the cold wallet at the start. Alternatively, configure your cold wallet software to send change back to the input address.

[arosca]

I ran into some of these questions with the system I am building.

One tentative solution my coworker and I have come up with is to keep the wallet on a separate server than the one that is user-facing (i.e. the website). I will call the web server WS and the separate machine that is interfacing with the Bitcoin network the transaction server, or TS.

In our proposed architecture the WS does not connect directly to the TS. This way, if a hacker gets into your WS, they do not have access to your wallet, or even know where your wallet is running. The TS does not even need to be accessible from the internet (does not need a public IP).

Instead, all communication between the WS and the TS happens via a message queue. The message broker can run on the same machine as the WS or on a third server. Anytime the WS requires information from the wallet (such as funds received at an address) it pushes a message on the queue and waits for a reply.

The TS listens to the message queue and processes any requests. Sensitive requests, such as sending funds to an external address, can go through a separate authorization process. For example, if users want to withdraw funds from their account, a request is pushed to the queue, but the TS does not immediately process it. Instead, it waits for a human administrator to approve the request. The human can either contact the user (via phone, email, etc.) to authenticate the request, or decide based on other criteria if the transfer should be authorized. In the future we may automate this (for example by introducing a two-factor authentication process) but for now we want to keep it simple.

Apart from securing your wallet you will have other issues to deal with. There are not many Bitcoin clients designed to work in a server environment and to deal with concerns such as disaster recovery, load balancing, etc. The only one I’ve come across is https://bitsofproof.com, but I don’t know how good it is.

Some of the issues to think about:

• How to reliably and securely back up your wallet
• What happens to your system if you are servicing a large number of users simultaneously
• How to keep track of the balance in each user’s account

The latter is of particular concern for me. See this thread for more info:
https://bitcointalk.org/index.php?topic=586013.0

[arosca]

To be clear, this is by no means a complete solution. I am just giving you some of our thoughts. You should still adopt a hot/cold wallet strategy and keep most of the funds completely offline.

[TierNolan]

Some of the issues to think about:

• How to reliably and securely back up your wallet

If you use a deterministic wallet, then everything is generated from 1 root seed number.

You can print that out and put it in a safe.

It can even be split into pieces.  You have 9 pieces of paper, and you need 5 of them to generate the root seed.  You can pick any values you want, 90 of 100 is possible.

Even if you lose everything else, you can regenerate all the keys from that one seed.

This is the key advantage of deterministic wallets.

What is cool is that you can create "watching-only" wallets, which are able to generate all the public keys (but not the private ones).

This means that you can detect a deposit into the cold wallet without having the private keys at all.

You can create transactions with this watching-only wallet and have them ready for signature.

The computer with the private keys is supposed to be completely isolated.[/list]

[arosca]

If you use a deterministic wallet, then everything is generated from 1 root seed number.

Indeed, but that only applies to keys. Account info is unrecoverable obviously (hence my other comment about user balances).

Also, it's nice that you can recover the key to an address if you know the wallet seed, but if you lose the wallet how do you keep track of which addresses you control in the first place? Is it possible to build a list just from the wallet seed?

[TierNolan]

Also, it's nice that you can recover the key to an address if you know the wallet seed, but if you lose the wallet how do you keep track of which addresses you control in the first place? Is it possible to build a list just from the wallet seed?

Yes.

It generates a series of key pairs.

If you had money in address number 20,000, then you would need to compute addresses 1 to 19,999 before you can work out 20,000.

It is reasonably fast.  It takes a few ms to generate an address.  Dedicated code can do it faster.

If you thought the first 100k addresses were used, you might compute the first million when doing a recovery, so you don't miss any when scanning the blockchain.

The new deterministic wallets have a suggested "gap" length of 20.  This is for use by the general public.  Merchants could have higher gap lengths.

It computes the first 20 addresses.  If it sees address 17 in the block chain, then it computes addresses up to address 37 and so on.

This means that you can have a "gap" of up to 20 unused addresses and it would still find all your addresses.

The client might warn the user if they asked for the 20th address in a row of spaces that it is the last one and it must be used before any more can be computed.

When importing, it might use a gap of 40, in order to be safe.

[arosca]

That's very cool, thanks for all the detail!

[TierNolan]

That's very cool, thanks for all the detail!

It seems Armory does do 2 of 3 multisig wallets, but it is at the beta stage and it just re-uses the same keys over and over.

[gokudev]

I like to have a discussion on how one can maintain a cold storage (offline system) that has no way to reach the hot wallet. How does one transfer transaction between the two systems? One way would be to have a api server that would allow unsigned transactions to be sent to the cold storage, get them signed and then send them back to hot wallet for broadcasting. A human can also authorize transactions like withdrawal requests.
But using the api method, we are then left with a connected system, a cold storage is offline and no automatic withdrawal exists, withdrawal must be done manually. What protocols are available for a manual withdrawl from coldstorage?

How can a tx be moved from a cold storage to an internet connected machine for submission ? I am able to build a api server that could technically have some api exposed to transfer signed tx from cold storage to hot wallet for submission but then coldstorage wont really be offline. Without using usb keys in an exchange environment, I like to know of other possible methods to transfer tx from cold storage.

[TierNolan]

But using the api method, we are then left with a connected system, a cold storage is offline and no automatic withdrawal exists, withdrawal must be done manually. What protocols are available for a manual withdrawl from coldstorage?

Cold storage withdrawal is supposed to be manual.  Armory uses a USB stick to move info over and back.

You use Armory running in "watching-only" wallet mode to create an unsigned transaction.

This is copied to the USB stick as a file.

You boot up your offline computer (not connected to the internet).

The offline computer reads the file from the USB stick.

It displays information about the transactions, i.e. destination address and amount.

You press OK, and it signs it and saves it back to the USB stick.

You take the USB back to the internet connected main computer.

Armory running on the main computer broadcasts the transaction.

This assumes that the USB doesn't auto-run and your initial download/setup was correct.

[gokudev]

But using the api method, we are then left with a connected system, a cold storage is offline and no automatic withdrawal exists, withdrawal must be done manually. What protocols are available for a manual withdrawl from coldstorage?

Cold storage withdrawal is supposed to be manual.  Armory uses a USB stick to move info over and back.

You use Armory running in "watching-only" wallet mode to create an unsigned transaction.

This is copied to the USB stick as a file.

You boot up your offline computer (not connected to the internet).

The offline computer reads the file from the USB stick.

It displays information about the transactions, i.e. destination address and amount.

You press OK, and it signs it and saves it back to the USB stick.

You take the USB back to the internet connected main computer.

Armory running on the main computer broadcasts the transaction.

This assumes that the USB doesn't auto-run and your initial download/setup was correct.

while using usb stick to move info over and back might be suitable for a individual miner it is not feasible in an exchange environment where several withdrawals are possible. So what is an ideal design for a exchange environment?

[TierNolan]

while using usb stick to move info over and back might be suitable for a individual miner it is not feasible in an exchange environment where several withdrawals are possible. So what is an ideal design for a exchange environment?

The cold wallet is supposed to be reserve funds.

Cold Wallet
 - funds can be deposited automatically
 - funds can be monitored via watching wallets
 - withdrawal is difficult

Hot Wallet
 - all wallet operations are automatic

The idea is that you get a notification if the Hot Wallet is running low on funds.  Funds can then be transferred to the Hot Wallet in a single transaction from the Cold Wallet.

When a client withdraws money, it is from the Hot Wallet.

In an exchange, there is like a "float" that varies daily.  If the total funds stored on the exchange went up an down by 10%, then you only need to store 10% in the hot wallet.

You could have an intermediate level (warm wallet?) that have less security than the cold wallet but more than the hot wallet.  For example, transfers might be automatic, but need to be signed by 3 of 3 separate servers.

[gokudev]

while using usb stick to move info over and back might be suitable for a individual miner it is not feasible in an exchange environment where several withdrawals are possible. So what is an ideal design for a exchange environment?

The cold wallet is supposed to be reserve funds.

Cold Wallet
 - funds can be deposited automatically
 - funds can be monitored via watching wallets
 - withdrawal is difficult

Hot Wallet
 - all wallet operations are automatic

The idea is that you get a notification if the Hot Wallet is running low on funds.  Funds can then be transferred to the Hot Wallet in a single transaction from the Cold Wallet.

When a client withdraws money, it is from the Hot Wallet.

In an exchange, there is like a "float" that varies daily.  If the total funds stored on the exchange went up an down by 10%, then you only need to store 10% in the hot wallet.

You could have an intermediate level (warm wallet?) that have less security than the cold wallet but more than the hot wallet.  For example, transfers might be automatic, but need to be signed by 3 of 3 separate servers.

so your implying,  the withdrawal from cold wallet will be manual and deposits would go in warm wallet?  transfer will be automatic from warm wallet to hot wallet using a server api (custom) ?

What other options are available for manual withdrawal from coldstorage besides usb stick?
 

[throwaway084575]

Thanks for all the great responses.

I love the idea of a warm wallet!
I am collating all the ideas and schematics for the devs to talk over.

Just looking at armory and it would be great if you could import a CSV with transactions that you need to happen and then then sign those on the cold wallet and finally broadcast them from hot.

So Armory would effectively be both the cold and warm wallet but we daily export a CSV of transactions that are waiting to happen and then send them out from cold storage.

Its not ideal because I was thinking it would be best to use Armory to just recharge the HOT wallet and so if the hot wallet runs out of funds then every new transaction goes into a Queue which are then executed once the hot wallet is full again.

What I like about this is that its more automated but if we are compromised I want a way to detect dodgy withdrawals in the queue because we don't want to fall into the alleged Karpeles tunnel syndrome, where funds are tunnelled away under the noses. But thats a whole other problem.

The main problem at the moment is cold warm hot wallet architecture.

Armory really is an amazing piece of work for cold storage, the issue is hot and warm really. I think it might be that every 24 hours we manually top up the hot wallet to a percentage of the cold.

If there is an overflow of withdrawals then we could manually process them. (This could annoy customers but if they realise that its for security reasons then people will be a bit more accepting, although people are fast to yell scam all over the internet if they don't get there bitcoins within seconds :/ so theirs that.  )