Camp BX Hacker / Security Audit: Results

[jimrandomh]

It's a start, but security for a financial institution takes a whole lot more than an automated test. You need to think about things like managing an offline wallet, physical security for that wallet and for your servers, and background checks for employees. You need a non-automated inspection by an expert, who will actually take the time to look at your source code.

If you run a Bitcoin exchange and it takes off, you aren't just up against script kiddies.

[Keyur @ Camp BX]

Error,
     Thank you very much for a thorough and unbiased review of the McAfee result.  We really appreciate this from you!

Wanted to add that we have a patch and upgrade schedule in place for our environment.  Admins prioritize patches based on criticality, and test / deploy them accordingly.  For majority of software we are on most recent codebase.

Thank you again,
     Keyur

[joepie91]

From campbx.com...

"Tested according to U.S. Government requirements"

I seriously doubt anyone will be impressed by that, it's more like a seal of certainty that lulzsec will breeze through the security measures in five minutes.

Fancy logos and certifications aside, any site can be hacked, what is more important is how hack attempts are dealt with from the user point of view (are losses covered?).

BTW:

Site running PHP/MySQL - Pass

PHP/MySQL do not have any specific vulnerabilities that are not also present in comparable other languages/platforms. They are not any worse of a language/platform than any other.
That most vulnerable sites are written using PHP/MySQL, does not mean that all sites using PHP/MySQL are vulnerable. Correlation, causation, etc.

[Keyur @ Camp BX]

It's a start, but security for a financial institution takes a whole lot more than an automated test. You need to think about things like managing an offline wallet, physical security for that wallet and for your servers, and background checks for employees. You need a non-automated inspection by an expert, who will actually take the time to look at your source code.

If you run a Bitcoin exchange and it takes off, you aren't just up against script kiddies.

Jim,
     Agree with you 100% - Coming from a corporate background we consider what you mentioned essential for security.

Our servers are housed in a physically secured data-center designed to survive F3 category tornadoes (if I am not mistaken), and have connectivity with three telco backbones.  There are two Caterpillar diesel generators for extended power outages.

We have identified primary and secondary owners for Wallet, and only these two people have access to it.  
Same goes for the database.

We also background check our employees as part of the security policy, and have a matching MSA with contracting firms.


Thank you,
      Keyur

[datguywhowanders]

It's a start, but security for a financial institution takes a whole lot more than an automated test. You need to think about things like managing an offline wallet, physical security for that wallet and for your servers, and background checks for employees. You need a non-automated inspection by an expert, who will actually take the time to look at your source code.

If you run a Bitcoin exchange and it takes off, you aren't just up against script kiddies.

Jim,
     Agree with you 100% - Coming from a corporate background we consider what you mentioned essential for security.

Our servers are housed in a physically secured data-center designed to survive F3 category tornadoes (if I am not mistaken), and have connectivity with three telco backbones.  There are two Caterpillar diesel generators for extended power outages.

We have identified primary and secondary owners for Wallet, and only these two people have access to it.  
Same goes for the database.

We also background check our employees as part of the security policy, and have a matching MSA with contracting firms.


Thank you,
      Keyur

Keyur,

I have to say that I feel you have done an excellent job in responding to customer questions and issues. Far more so than any of the other exchanges, although I will say in all fairness that I think TradeHill does a decent job as well.

It's good to see that your team is taking security very seriously, and I enjoy knowing that CampBX is taking a multi-tiered approach towards security. I'm obviously biased being in the United States, but it's reassuring to see you playing by all the rules and regulations. I think you guys are positioned to do very well.

I already have some of my BTC on your site, and I will probably have more in the future.

Keep up the good work!

[evoorhees]

Thank you OP. It is great to see free-market certification solutions meeting the needs of customers, instead of ridiculous government laws "mandating" security.

Cheers to you, I've opened an account.

[billyjoeallen]

money went in, no problem. Now I'm just waiting to get my orders filled :-)

SELL SELL SELL! (I'm buying)

[sanchaz]

Are you planning to accommodate EU customers as well?

[dacoinminster]

Camp BX got some press coverage:
http://www.zippycart.com/ecommerce-news/2796-ecommerce-solution-camp-bx-makes-bitcoin-legitimate.html
http://venturebeat.com/2011/07/07/camp-bx-bitcoin/

Also, they just announced a new affiliate program. The deal is the same as TradeHill.com, 10% off your trading fees if you sign up through an affiliate link such as this one (mine): https://CampBX.com/register.php?r=mdslj19rhcD

If you registered before the affiliate program started, email them, and they will get you the discount. (At least, they did it for me).

Their website claims they will disable your affiliate link if people complain that you are spamming it everywhere, so don't do that.

[Stephen Gornick]

This is an old thread but there was a question asked of great importance and I don't see that it was answered:

It's a start, but security for a financial institution takes a whole lot more than an automated test. You need to think about things like managing an offline wallet, physical security for that wallet and for your servers, and background checks for employees.

Jim,
     Agree with you 100% - Coming from a corporate background we consider what you mentioned essential for security.

Our servers are housed in a physically secured data-center designed to survive F3 category tornadoes (if I am not mistaken), and have connectivity with three telco backbones.  There are two Caterpillar diesel generators for extended power outages.

We have identified primary and secondary owners for Wallet, and only these two people have access to it.

The question specifically asks about managing an offline wallet.  The response is ambiguous and uses "wallet" singular and "it" when referring to "wallet", so that is nowhere near to being an assertion that that customer's bitcoin funds are held in cold storage.

There was a recent post pointing to the site's FAQ, but that FAQ doesn't address the use of a cold wallet either.

CampBX has been operating securely without incident for over a year now.  I am a data-center guy and not very good at marketing on this forum, but I invite you to check out our security best practices here: https://campbx.com/faq.php#security-compliance

I wish this specific question and others had been asked of a competing U.S.-based bitcoin exchange as thousands of bitcoins would still be with their rightful owners as once they would have discovered that no cold storage was being used by that exchange things would have been different.

So, I'm submitting these questions, looking first specifically for the answer to:

 - Does Camp BX use cold storage (an offline wallet that cannot be accessed should the exchange's service become compromised)

If so, then there are other questions:

 - Is there a target as to how much of customer's funds are kept in cold storage?  (e.g., percent of total, or perhaps relative to recent withdrawal requirements)?

 - Do new deposits go to cold storage?  (if the hot wallet is compromised, new deposits made (e.g., automated payouts by mining pools) would still be secure)

 - Does the offline wallet where the cold storage resides remain protected due to an "air gap" (no access to it electronically, not connected to the network)?


And I have other questions that I'ld like to now the answers to:

 - Does CampBX maintain full reserve?  (i.e., Camp BX controls bank accounts with all customer USD funds and controls wallets with 100% of BTC funds.  None of these amounts loaned out.)

 - Does CampBX maintain offsite backups of its accounts and transactions?  If for some reason the exchange's primary account database were lost due to a security breach, what information (and how recent) is still available from backup or archives?

 - If there is a security breach and CampBX cannot meet withdrawal requests of its customers, what is the withdrawal preference that Camp BX would follow?  Various preferences are:
 - -  A.) All deposited funds are of equal standing with bitcoins being valued at their market rate at the time of the loss,
 - -  B.) Withdrawals of USD funds, if not impacted by the breach, are made available to those customers who held a USD balance. in full.
 - -  Do customer deposits have preference over any other creditor claims?  (i.e., a contract stating so such that they don't become unsecured creditors ending up in the same pool as the landlord for office space and hosting bill.)
 - -  or is there some other approach?

[thezerg]

These are good questions. 

Really, what is needed is a white paper describing a belt-and-suspenders approach to securing customer deposits.  CampBX has a great opportunity as the last US exchange standing and could really capitalize on that if they provided a comprehensive security document.  I really don't think such a document would help hackers much... if it did that would be an indicator that the security had issues (security through obscurity is not true security).

Additional questions:

Are these wallets encrypted?

Are there ANY unencrypted backups of the wallet, hard copies of the private key, etc?  If so how are they protected?

How are USD deposits secured?

Are USD/BTC deposits held individually and separately or is there a similar issue to bitfloor where USD on deposit could be used to pay off other losses (operating, or hacking).

(I've been a campBX user for several months now)

Thanks!

[Mt.Gox_Alex]

GOX are you watching? Learning?

Learning? Everyday we are learning something new... Watching? Yes very carefully... Now it is good that others start to finally work on their security... As far as we are concerned we are a year ahead of others on this matter and never stop on improving/checking things when it comes security.

[Stephen Gornick]

- Does CampBX maintain full reserve?  (i.e., Camp BX controls bank accounts with all customer USD funds and controls wallets with 100% of BTC funds.  None of these amounts loaned out.)

I see this question (and perhaps others) is addressed in another thread -- one which I wish I had seen earlier, as that thread is the right place for this line of questioning. (if responding, please feel free to respond there)

- No fractional reserve: We hold 100% of user funds in reserve at all times
- All banking done on-shore in the USA
- We do not do business with companies that don't have a registered office in USA.  (Paxum, Liberty Reserve)

[The_Duke]

As far as we are concerned we are a year ahead of others on this matter

If that is the case, then how about giving the MtGox answer to the questions raised by Stephen Gornick in his post above?

[MagicalTux]

If that is the case, then how about giving the MtGox answer to the questions raised by Stephen Gornick in his post above?

If you want (replaced CampBX with MtGox and replies in bold for readability):

 - Does [MtGox] use cold storage (an offline wallet that cannot be accessed should the exchange's service become compromised)

Yes.

 - Is there a target as to how much of customer's funds are kept in cold storage?  (e.g., percent of total, or perhaps relative to recent withdrawal requirements)?

On average 98% of customer bitcoins are held in cold storage, with possible variations on large bitcoin moves (large deposits or customers asking for large withdrawals).

 - Do new deposits go to cold storage?  (if the hot wallet is compromised, new deposits made (e.g., automated payouts by mining pools) would still be secure)

No, this wouldn't be practical in terms of number of bitcoin addresses to keep in cold storage. This could change thanks to BIP 0032 which we are working on implementing. It should be noted however that we are using a hardware security module for the hot wallet

 - Does the offline wallet where the cold storage resides remain protected due to an "air gap" (no access to it electronically, not connected to the network)?

Offline wallets are generated from an offline system and kept in paper format in three separate locations, using a technology based on raid. It will likely be changed to use Shamir's Secret-Sharing method in the future, and all existing offline wallets will be converted to this.

And I have other questions that I'ld like to now the answers to:

 - Does [MtGox] maintain full reserve?  (i.e., [MtGox] controls bank accounts with all customer USD funds and controls wallets with 100% of BTC funds.  None of these amounts loaned out.)

As described in our Terms of Service, customer funds are kept in full, and none are loaned.

 - Does [MtGox] maintain offsite backups of its accounts and transactions?  If for some reason the exchange's primary account database were lost due to a security breach, what information (and how recent) is still available from backup or archives?

We have realtime onsite backups on a separate system, and offsite backups at regular intervals. We are working on modifying the system to have a multi-site cluster working (working with people from Percona to reach the best system on this) - which would allow us to have a node of the cluster used to make backups way more often

 - If there is a security breach and [MtGox] cannot meet withdrawal requests of its customers, what is the withdrawal preference that [MtGox] would follow?  Various preferences are:
 - -  A.) All deposited funds are of equal standing with bitcoins being valued at their market rate at the time of the loss,
 - -  B.) Withdrawals of USD funds, if not impacted by the breach, are made available to those customers who held a USD balance. in full.
 - -  Do customer deposits have preference over any other creditor claims?  (i.e., a contract stating so such that they don't become unsecured creditors ending up in the same pool as the landlord for office space and hosting bill.)
 - -  or is there some other approach?

Fiat balances and Bitcoin balances would be accounted separately based on current rules, especially because of the difficulty to give a value to a given balance in Bitcoin (value at current rate or based on depth). This may change as we are discussing with a large insurance company in Japan to get all funds deposited on MtGox insured. This will however be only possible once the Japanese FSA provides its position on Bitcoin - which we expect to happen in the next months.

[The_Duke]

Thanks Tux, that's some clear answers

I think the "year ahead" only goes for experience, not really "technology wise", but it's still something that counts.

[thezerg]

 - If there is a security breach and [MtGox] cannot meet withdrawal requests of its customers, what is the withdrawal preference that [MtGox] would follow?  Various preferences are:
 - -  A.) All deposited funds are of equal standing with bitcoins being valued at their market rate at the time of the loss,
 - -  B.) Withdrawals of USD funds, if not impacted by the breach, are made available to those customers who held a USD balance. in full.
 - -  Do customer deposits have preference over any other creditor claims?  (i.e., a contract stating so such that they don't become unsecured creditors ending up in the same pool as the landlord for office space and hosting bill.)
 - -  or is there some other approach?

Fiat balances and Bitcoin balances would be accounted separately based on current rules, especially because of the difficulty to give a value to a given balance in Bitcoin (value at current rate or based on depth). This may change as we are discussing with a large insurance company in Japan to get all funds deposited on MtGox insured. This will however be only possible once the Japanese FSA provides its position on Bitcoin - which we expect to happen in the next months.

Your service seems good but of course I am concerned with the legal reachability of a company in Japan (and the extra hassle of overseas fiat transfer).  However, its all about balancing this risk against that, so I recently got verified with MtGox.  I think "B" should be the fiat option.  Essentially, I do not want my fiat to be held as security for people who choose to leave excessive bitcoin on the exchange.  This risk makes it hard to hold funds on the exchange.

How are the paper wallets physically secured?  Hopefully they are not in an envelope in your sock drawer...

[Keyur @ Camp BX]

Hi Stephen,
     Excellent questions, and a lot of them!  I will post an update to answer these shortly.

Thank you,
      Keyur

[ripper234]

Hi Stephen,
     Excellent questions, and a lot of them!  I will post an update to answer these shortly.

Shortly < 1 month

[Keyur @ Camp BX]

Hi Stephen,
      We have been tweaking out security and monitoring procedures since launch to stay ahead of any potential issues.  I reviewed our change tracker sheet and there have been over 120 changes in 2012 (this includes everything from cosmetic edits to server hardware upgrades and patches), which should give you an idea of the work behind the scenes to stay current.  Here is the specific information you requested:

 - Does Camp BX use cold storage (an offline wallet that cannot be accessed should the exchange's service become compromised)
Absolutely.  We have hot wallet / cold wallet split system.


If so, then there are other questions:

 - Is there a target as to how much of customer's funds are kept in cold storage?  (e.g., percent of total, or perhaps relative to recent withdrawal requirements)?
Our wallet has higher churn rate, so percentage kept in hot wallet needs to be much higher than MTG's 3% number.  We set it based on current activity levels + volatility headroom.

 - Do new deposits go to cold storage?  (if the hot wallet is compromised, new deposits made (e.g., automated payouts by mining pools) would still be secure)
No - new deposits go to the hot wallet.  We have considered sending new deposits to cold wallet, but implementing and operating that code will require us to touch cold storage much more often.  This may defeat the purpose of cold storage.

 - Does the offline wallet where the cold storage resides remain protected due to an "air gap" (no access to it electronically, not connected to the network)?
Yes - air gap is a must otherwise it would be a "luke-warm wallet"! 

And I have other questions that I'ld like to now the answers to:

 - Does CampBX maintain full reserve?  (i.e., Camp BX controls bank accounts with all customer USD funds and controls wallets with 100% of BTC funds.  None of these amounts loaned out.)
Yes - for both USD and BTC. We do not lend or spend any of the funds.

 - Does CampBX maintain offsite backups of its accounts and transactions?  If for some reason the exchange's primary account database were lost due to a security breach, what information (and how recent) is still available from backup or archives?
Yes - this has been part of our DR plan since launch day.  In case of a server crash (much more likely than security breach, IMO) we can recover up to 1-hour recent data.

 - If there is a security breach and CampBX cannot meet withdrawal requests of its customers, what is the withdrawal preference that Camp BX would follow?  Various preferences are:
 - -  A.) All deposited funds are of equal standing with bitcoins being valued at their market rate at the time of the loss,
 - -  B.) Withdrawals of USD funds, if not impacted by the breach, are made available to those customers who held a USD balance. in full.
 - -  Do customer deposits have preference over any other creditor claims?  (i.e., a contract stating so such that they don't become unsecured creditors ending up in the same pool as the landlord for office space and hosting bill.)
 - -  or is there some other approach?
Answer to this question really comes down to the situation at hand.  If it was an overnight heist like what happened to MtGox or BitFloor, we may have to go for (B).  If this was a trickle-heist like MyBitcoin the answer may be more complicated.  The good news is that we do not have any creditors, so in case of a breach all funds will go back to customers.  Verified customers will get preference over unverified customers.

Hope this helps,
     Keyur