Bitcoin Forum

We're not an exchange, but given that we deal with people's Bitcoins we do have an obligation to state this: We have always maintained 1) hourly database backups to a second data center, 2) daily offline backups, 3) a hot wallet stored in a third datacenter, on a dedicated server, and 4) offline wallet storage of all funds other than petty cash. Furthermore, everything we run other than our blog is on offshore dedicated servers at datacenters with casino-grade physical security measures, NOT on VPS. A hacker who accessed one of our dedis would find our hot wallet basically empty and our user passwords hashed. At most we'd lose a hundred bucks or so.

We don't have anything near the volume of Bitcoinica. We've got about 1000 users. When we launched, and started paying for the servers involved in this elaborate setup, we had no users. There's no doubt the added security has come at a cost that dug into our bottom line. But what's the alternative? Hosting on a VPS somewhere and waiting for disaster? You don't screw around with cutting costs on security; a wise guy once told me it's better to be "insurance poor" than temporarily rich and waiting for the other shoe to drop. One of the dumbest things I've done in recent memory was send some of our first positive revenues into a Bitcoinica account. I would never have imagined the security there would be more lax than ours, but it's my fault for not doing more research. I accept that.

There is no formulaic way yet of definitively securing a site that deals in BTC, but having lots of backups and dedicated servers seems like kind of a no-brainer place to start. It's a shame all the "security experts" floating around the Bitcoinica scene couldn't afford an extra $200 a month for a dedicated / offline backup solution. We don't make the kind of money they do, and we trusted them with our profit; if I'd known they were so poverty-stricken I would have paid for it myself goddamn it.

This is reassuring.

Thanks, Stoch! We'll soon find out which agents have their client's best interest at heart, and which agents...

~Bruno~

I would like to see these services doing security audits by professionals and getting some hacking insurance.

Hi Bruno,

For Crypto X Change we have spared no expense on security, hardware or backups.

Automated Backups are taken Hourly, Daily & Monthly and are taken Off Site.

We perform manual backups also for DB's, Wallets and Data. This Data is also Taken Off Site.

We have a top level Cisco Hardware firewall in place.

Dedicated Servers only, with RAID 10 SAS 15K RPM Arrays for maximum performance and reliability

Data Center is the most secure in Australia by far ( Location & Name Will NOT be released for security reasons )

Access to our systems are absolutely limited to 2 people within our company via very secure means, code updates by our programmers are installed by one of the 2 people who have sole access to our systems.

We have services separated to each dedicated server, enabling maximum security

Our site is PenTested often, any and all updates are also checked prior to being installed on the production servers.

Development servers are also locked down to only 2 people also.

Our company has staff that have been in the IT industry for nearly 20 years with real world experience and working and servicing some of the largest government networks in Australia, we take security very seriously.

Our Exchange is a continuous work in progress, improving all the time, building new features. There are still features un released from when we opened the exchange, we have had our problems, slow trade engine, slow API BUT never any security breaches at all, or any issues with human error and data.

Before opening, we asked the community to do their best to hack our exchange, and there were Zero breaches.

We hope this explains our security, if anyone has any further questions please do ask! If we are slow to reply here we appologise, we have a lot of development going on and new services coming out shortly.

Regards

Crypto X Change Team

Basic principles

Due to the nature of our business, security is very important and we have a very unique approach to the issue, defined by the following principles:

Transparency
Contrary to popular belief, the most secure system is one where everyone can see every aspect of it and still can not break in. Imagine a safe where even with the blueprints you can not find a way to get it open. This principle is one reflected in the history of information security and cryptography. Open-source and free software systems have a track record of rapidly fixing security issues with systems such as OpenBSD having a reputation for heavy security. Cryptographic algorithms such as ECDSA (on which bitcoin is heavily based) are open for inspection by all, and it is this which gives them strength. For more on the principle of transparency and why full disclosure is the ideal we refer you to this excellent essay by Bruce Schneier: Full Disclosure of Security Vulnerabilities a 'Damned Good Idea'.

Proactivity
Our second principle is that of proactivity, taking an active rather than responsive approach to potential security threats. Rather than responding after an incident we take active measures to audit our system and our business practices to find any potential threats before they occur. This includes even theoretical and obscure threats, as we assume any attacker targeting us will explore all possible avenues of attack. In practice, this principle means regular audits on at least a daily basis using vulnerability scanners from multiple different vendors in addition to analysis of system log files for anomalious entries. We also conduct reviews of changes to the filesystem on any of our servers and flag up automatically any changes outside of those specifically defined as normal and authorised.

Awareness
We keep track of multiple sources of information about the current state of our systems as well as external reports of security issues with the software we use. Our support system automatically creates tickets for any potential security issues discovered by automated scans and audits as well as for any advisories on public mailing lists. We treat these support tickets with high priority. On top of this, we track multiple metrics for performance, potential attack attempts and other measures.

Paranoia
Encrypt everything, use default deny on all firewalls, lock down ACLs and filesystem access, restrict syscalls on daemon processes and trust nobody. We assume that 24/7, somebody is actively trying to break into our system and is desperate to do so. This assumption assures that in the scenario where the bad guys are not so active our system stands up.

Realism
Realistically we are humans and not perfect security-hardening machines, therefore we assume that there's always something we've missed - this is why we use outside vendors to do scans and audits, but in keeping with the principle of paranoia we also assume outside vendors are full of bad guys or employees that can't be trusted. We also do audits again after system configuration changes to ensure that nothing is missed.

Specific practices

Multiple vendor security scanning
We currently use the following software and services to audit our systems: Nessus, Nikto, snort IDS, checksecurity, TrustGuard PCI scanning, HackerTarget, nmap, tripwire. These tools are run on a daily basis where possible and any flaws found are dealt with immediately after audit. In addition to manual checks, our servers have a daily cronjob that provides up to date information and metrics relating to performance and security. To prevent a possible attacker from compromising a system and covering their tracks, these reports are emailed to our staff and backed up in an encrypted form onto amazon S3. Hashes are stored of logfiles using different algorithms (MD5 and SHA512) and regular checks are made that no alterations have been made.

Configuration backups
On a regular basis we make backups of all servers at the disk level (raw block device dumps) and store them in order to allow for easily reverting in the event of a compromised system. Should any of our systems end up compromised, our policy would be to estimate the date of first breakin and then restore the backup taken at least 1 month previous to that in a controlled environment (no external network access except for SSH from our staff). We would then install the latest security patches and close whatever hole enabled the initial compromise before running all auditing tools at our disposal in addition to a manual check of all operating system binaries for potential hidden rootkits.

Untainted logs
Every quote, every transaction stage, every completed transaction - all of this information is logged to multiple locations with cryptographic hashes used to verify it is untainted. Our systems are not able to do anything other than write to the transaction log and we keep the entire log archived. Physical backups to removeable media are also made on a regular basis and all copies of the transaction log are encrypted with the key known only to our 2 founding partners. Should every other security somehow fail, we will always be capable of rebuilding our entire service using this information - this is our key asset, and it is also your key asset - customer transactions.

Source code management and end user software
Like any online service, our software is what powers everything. For every piece of code, we ask whether it would be safe to release that code to the general public or if doing so would open up security holes, and if releasing it would open up security holes then we rewrite it. As a service provider whose business is selling a service rather than the software powering it, we do not require our customers to run any specific software other than a modern web browser (we do not even require javascript - our site will generally work without it, although we do recommend it is enabled for ease of use). As we do not require our customers to run any software we restrict our attention to the server side and to the small class of potential attacks against javascript and HTML (XSS and injection attacks). Our customers are free to examine and audit any javascript or HTML sent by our webservers and ensure there are no security flaws, and if any flaws are found we encourage customers to inform support.

Default deny
Any of our systems has a firewall installed with the default being to deny any and all connections. This host firewall is in addition to any upstream firewall provided by our hosting provider (to control for the scenario in which our hosting provider is compromised). The same "default deny" philosophy applies to filesystem privileges and ACLs. Whenever a system component can be configured to deny everything by default, we do so and whitelist specific authorised uses.

System log anomaly monitoring
During regular operation there are various log entries written by a linux server. These log entries fall into 3 basic categories as regards security. Legitimate use, "background noise" and anomalies. We define "background noise" as the fruitless attempts by botnets and individuals with port scanners to try and break into any IP host they come across, usually in the form of dictionary attacks. Background noise is mostly an annoyance but we do not take the risk and so IP blocks from which these kinds of attempts originate are automatically blocked by our firewall. Anomalies are targeted attempts or requests sent to our servers which are neither part of normal usage or background noise. We assume all anomalies to be an attempt to break in and block IPs sending such requests before analysing the requests to look at what exploits (if any) were attempted. We assume that such attempts are successful until conducting audits to verify they were not.

Regular key changing
On a regular basis, all keys and passwords are changed and new ones are generated randomly with a high-quality entropy source. Specifically, the entropy source used is static noise from a soundcard blended with prices of random stocks on public markets on random dates before being passed into a hash function and fed to /dev/random on a linux server.

No production configuration in version control
We use the subversion version control system to develop code, but we make use of the .dist design pattern, wherein configuration files are named (for example) "importantstuff.cfg.dist" rather than "importantstuff.cfg". We then set subversion to ignore plain .cfg files and perform regular audits to ensure they never make their way into version control. When a code change requires merging an old configuration file, this is done by hand. By following this pattern we ensure that should our SVN repository be compromised it will not lead to disclosure of important server configuration information.

Treating all errors as potential security flaws
Our web application code logs HTTP 500 errors caused by exceptions or other problems in the underlying code and these logs are emailed to support in addition to generating support tickets that are treated with a high priority. This also allows us to respond rapidly in case of problems affecting customer transactions and in theory resolve issues before the customer even has time to contact support seperately.

Question to Phinneas: How do we determine that the owner of the said company is telling the truth? Perhaps we should specific that certain evidence is required in orer to prove one has adequate backups?

More important than a self-authored "adequate backup" claim, would be a statement from each business indicating:

1. Whether the business guarantees that it will "make good" any customer balances after a catastrophic failure
2. How the business will do so
3. Who is the person or entity responsible for seeing that this guarantee is carried out

More important than a self-authored "adequate backup" claim, would be a statement from each business indicating:

1. Whether the business guarantees that it will "make good" any customer balances after a catastrophic failure
2. How the business will do so
3. Who is the person or entity responsible for seeing that this guarantee is carried out

It might make people feel good to have businesses make such a statement, but ultimately such reassurances are pretty meaningless.  The community has shown itself to have significant aversion to any kind of externally supervised recovery process when Bitcoin enterprises fail and it's also been unwilling to use legal processes to recover funds.  All the promises in the world are meaningless when there is neither a means nor an inclination to enforce them.

Social pressure isn't the answer to everything, but it can solve a lot of problems that "externally supervised enforcement" will never solve. And even the act of making a public statement will cause all but the most sociopathic to be more aware of the consequences of their actions, and perhaps to take more care.

And this ladies and gentlemen is how a market regulated by strictly market consumers (i.e. a free market) regulates itself. Isn't it beautiful?  8)

It is beautiful, yes, but we're not even getting the best part.

Many of the people who would be this market's honest players are frightened off by the fear that Bitcoin might be (or might be declared) illegal. Dishonest players, of course, don't care about this. So the distribution of market participants is inevitably skewed somewhat.

Before we launched I spent almost a year on casinomeister talking to players, reading complaints against other casinos and trying to figure out how to build a site that would be safe, responsive to players, and would make sure that even under catastrophic circumstances we would always have enough backups and funds to cover it and never land on the rogue list.

Using the numbers that Andre recently provided, on top of my most recent backup shows that WBX should currently be holding
1,769.0417 BTC and 25,779.49 AUD.  If we assume a price of $5 AUD/BTC then that's a total of 1,769.0417 + 5,155.898 = 6,924.9397 BTC.

So. While we won't post details of our security procedures in a public forum, we would be willing to share some information with the OP, in confidence, based on which he can make a well-informed recommendation as to whether what I've said here is true. This obviously sets a precedent that gives Phineas a fair amount of power, potentially. But I do think his intentions are honest. I've been approached by certain scammers on this board out of the blue, saying they wanted access to our systems to "audit" us. Good luck. But if we can prove to Phineas that we are what we say we are, then hopefully that will set people at ease, and it would set my mind at ease if more Bitcoin companies were willing to be forthcoming with those kinds of details.

As part of running Casascius Physical Bitcoins, I am occasionally in the position of holding others' funds, typically for overseas or large orders.

Bitcoins aren't kept online.  Payments to my website actually go to an offline wallet.  The web server knows only a list of pre-generated addresses and dispenses one with each order.  The offline wallet was generated deterministically and therefore could be recovered with just the seed.

In addition to regular database backups, I make sure my e-mail contains everything I would need to recover in the event of data loss, the e-mail being completely independent from the web server and database of course.  I receive e-mails with order details, and whenever I send unfunded coins, I send a complete list of addresses to the recipient at the time I fill the package.  While there's nothing high-tech about using e-mail, it's effective as a secondary measure.

Not to be a prat, but it took about two seconds to penetrate your STO.  Location and name are quite easily found.

Just want to say thanks for the distinction. I wish this information had been available a couple months ago.

I will name some of the features and safety precautions that ensure stability of our system:

* Operating system from prominent North American Enterprise Linux vendor
* Secured by National Security Agency guidance for hardening OS
* Tape storage backup
* Bitcoin cold storage on separate server and location
* Backups of database and wallet every hour 24/7
* Industry standard router and network switches
* Industry standard servers

Best regards,
Nejc Kodrič
Bitstamp.net

I hope this thread is the beginning of this (https://bitcointalk.org/index.php?topic=82876.msg928375#msg928375).

Hey Phinnaeus Gage, have you thought about contacting the http://bitcoincounsel.com/ guys and ask them if maybe they'd be willing to add a page for a security standard testimony list or something like that which could be updates once more businesses come forward in this thread?

Bitstamp is now on the list.

This is a waste of time.  You're asking owners for a statement saying their stuff is adequately backed up and that's how they get on the list?  No verification or auditing, just that they think their procedures are adequate.  Similar to how people thought it was adequate to go with cheap hosting on vps providers to hold thousands of btc only to have it stolen right out from under them?

This is a waste of time.  You're asking owners for a statement saying their stuff is adequately backed up and that's how they get on the list?  No verification or auditing, just that they think their procedures are adequate.  Similar to how people thought it was adequate to go with cheap hosting on vps providers to hold thousands of btc only to have it stolen right out from under them?

What this really comes down to is that for the list to be of any value, there needs to be a list of Bitcoin businesses who have an independent audit.

... as well as paid by that bitcoin business to carry out that audit.

Well, no. As they say, "he who pays the piper calls the tune". And look how poorly the credit rating agencies did in 2007.

Much better that the auditors are paid by the customers and depositors, than by the businesses.

+1

Best by someone known and trustworthy in bitcoin community as well as paid by that bitcoin business to carry out that audit.

Yeah it's too bad they are so expensive though. I doubt we will see much of that until there is some major profit to be made and many businesses involved.

Question to Phinneas: How do we determine that the owner of the said company is telling the truth? Perhaps we should specific that certain evidence is required in orer to prove one has adequate backups?

ssaCEO of StrikeSapphire (https://strikesapphire.com) is the first (https://bitcointalk.org/index.php?topic=81045.msg925308#msg925308) Bitcoin related company to publically state that their site, one that deals with people's bitcoin, has an adequate backup system in place, protecting their user's funds.