btcd: a bitcoind alternative written in Go

[behindtext]

Is it normal for inbound peers to consistently quit after approximately 20 seconds?

Code:

08:22:04 2013-12-17 [INF] BMGR: Lost peer 95.154.245.77:62066 (inbound)
08:22:28 2013-12-17 [INF] BMGR: New valid peer 23.226.234.12:59544 (inbound)
08:22:45 2013-12-17 [INF] BMGR: Lost peer 23.226.234.12:59544 (inbound)
08:23:12 2013-12-17 [INF] BMGR: Processed 1 block in the last 6m33.27s (320 transactions, height 275387, 2013-12-17 08:22:09 +0000 UTC)
08:24:18 2013-12-17 [INF] BMGR: New valid peer 31.220.26.209:37172 (inbound)
08:24:28 2013-12-17 [INF] BMGR: Lost peer 31.220.26.209:37172 (inbound)
08:25:42 2013-12-17 [INF] BMGR: New valid peer 85.17.207.181:8333 (outbound)
08:26:07 2013-12-17 [INF] BMGR: New valid peer 23.226.234.12:49392 (inbound)
08:26:18 2013-12-17 [INF] BMGR: Lost peer 23.226.234.12:49392 (inbound)
08:26:34 2013-12-17 [INF] BMGR: Processed 1 block in the last 3m22.93s (142 transactions, height 275388, 2013-12-17 08:23:19 +0000 UTC)
08:27:12 2013-12-17 [INF] BMGR: New valid peer 178.18.90.41:56321 (inbound)
08:27:12 2013-12-17 [INF] BMGR: Lost peer 178.18.90.41:56321 (inbound)
08:27:20 2013-12-17 [INF] BMGR: Processed 1 block in the last 45.05s (38 transactions, height 275389, 2013-12-17 08:29:10 +0000 UTC)
08:29:25 2013-12-17 [WRN] PEER: Peer 68.228.71.101:8333 (outbound) no answer for 5 minutes, disconnecting
08:29:25 2013-12-17 [INF] BMGR: Lost peer 68.228.71.101:8333 (outbound)

that is indeed interesting. i haven't seen anything like that to date.

did you file a GH issue?

[justusranvier]

that is indeed interesting. i haven't seen anything like that to date.

did you file a GH issue?

Not yet - I wanted to make sure it wasn't a known issue.

If you check the log file here, the same behaviour shows up:

https://github.com/conformal/btcd/issues/62

[behindtext]

oh, and for an update:

• headers-first has been committed to master - takes 1.75 hrs to get to last checkpoint at block 267300 vs 5 hrs previously
• TLS and auth are available between all 3 daemons that comprise the software: btcd, btcwallet and btcgui
• it is possible to use tor both between the public internet and btcd, and between btcd and btcwallet
• everything is working on mainnet just fine, but it is only recommended for early adopters
• mempool tx notifications will go into master shortly
• optimized secp256k1 handling is roughly 50% complete and will be showing up in master in the next couple weeks

there is surely more but those are the big ones

[jcv]

What strategy are you using to make sure that you're implementing the distributed algorithm consistently with the reference software?

Hi cczarek,

I assume you mean the verification of the blockchain (but if you mean something else, just correct me and I'll try to answer that).

We did a number of things to ensure that we match the behavior of bitcoind/-qt.  For starters, there is a regression test with a number of test blocks and reorganizations of the chain that we made sure btcd could pass (we talk a little more about that in one of our blog posts https://blog.conformal.com/btcd-not-your-moms-bitcoin-daemon/#more-421).  We've spent a lot of time writing test code (which doesn't exactly show that we match bitcoind, but it does help show that the code matches what we think it does which is usually half the battle).

Largely, this has come down to testing.  We've collectively run btcd for a huge number of hours on a huge number of blocks to ensure it behaves the way bitcoind does.  We also pay close attention to comments people have on github about to help catch bugs.

Hopefully that gives you some more confidence.  And you can always look at the code if so inclined, that is part of why it is open source.

[coinrevo]

For future reference I am saving Mike's critique on slashdot below, which raises some interesting questions

A full node is a really, really large amount of work. I feel that lots of people don't realise this, get enthusiastic and think, "I love Bitcoin! I love Go! I'll write Bitcoin in Go" where for Go you can substitute basically any language that's fun or popular. Then they write the easy bits (like wire marshalling) and eventually the project dies around the time that it's time to implement the wallet or Bloom filtering or robust test suites. Possibly Conformal is different, we'll have to wait and see, but the feature set they advertised in their blog is very much what has been seen many times before. In particular there's no handling of the block chain, re-orgs, no wallet and they haven't got any infrastructure to test edge cases.

One reason implementing Bitcoin properly is not fun is an entire class of bugs that doesn't exist in normal software - chain splitting bugs - which can be summed up as "Your software behaves how you thought it's supposed to work rather than how the original bitcoind actually does work". Bitcoin is highly unusual in that it implements group consensus - lots of nodes have to perform extremely complicated calculations and arrive at exactly the same result in lockstep, to a far far higher degree of accuracy than other network protocols. This means that you have to replicate the same set of bugs bitcoind has. Failure to do so can lead to opening up security holes via consensus failure which can in turn lead to double spending (and thus your users lose money!).

Being compatible with the way bitcoind is written (bugs and all) may require you to break whatever abstractions you have introduced to make the code cleaner or more elegant or whatever reason you have for reimplementing Bitcoin. Here's a trivial example - signatures in Bitcoin have an additional byte that basically selects between one of a few different modes. It's actually one of three modes plus a flag. So a natural way to implement this is as an enum representing the three modes plus a boolean for the flag. But that won't work. There is a transaction in the block chain which has a sighash flag that doesn't fit any of the pre-defined values (it's zero) and because Satoshi's code uses bit testing it still works. But if you turn the flag into an enum, when you re-serialise the mode flags you'll re-serialise it wrong and arrive at an incorrect result. So you have to pass these flags around as integers and select via bit testing as well.

Bitcoin is full of these kinds of weird edge cases. Eventually you come to realise that reimplementing it is dangerous and probably whatever benefits you thought it had, it probably doesn't. Some people believe there should be independent reimplementations anyway and I can understand and respect that, but doing it safely is an absolutely massive piece of work. You have to really, really, really believe in diversity to do it - the features of language-of-the-day aren't good enough to justify the effort.

[coinrevo]

I think this work is very valuable for several reasons, most importantly to have a clearer distinction between the software (reference client) and the protocol. I would second this comment of an anonymous poster on slashdot. One can acknowledge all the great work done and still be paranoid about backdoors. Ultimately adopters might know nothing about who implements bitcoin, but if there are many checks & balances in the process that is much more secure than one development tree.

Your post is extremely interesting, but the mandatory conclusion I make from it is the exact opposite of yours. If the original code is so full of idiosyncracies and gotchas then it's an extreme liability to everyone who values Bitcoin, and quite likely contains backdoors or deliberate weaknesses that are hidden by the obscurity.

There can be no more important task for the Bitcoin community I think than to specify all elements of the static protocol and dynamic behavior of all parts, and reimplement them in other languages, especially safe languages.

Go is certainly a good candidate for this large body of work, safe, clean, and fast.

I would like to understand more about the points raised in these comments. In how far does a re-implementation have to reproduce weird behaviour (not quite the same as bugs)? In how far are these documented at all?

Does btcd have a different concurrency model? How much of the security depends on the go-lang libs themselves?

[coinrevo]

Here is Dave's talk on the 2013 conference http://www.youtube.com/watch?v=d1IYvJs5GGw

[behindtext]

For future reference I am saving Mike's critique on slashdot below, which raises some interesting questions

to keep this in context, note that this comment was made by mike when all we had was a wire protocol and the rest of the project was incomplete. the 'story' that went up on slashdot was misleading in its title and suggested we were nearly done with the project.  pretty much everything except a chunk of the JSON RPC commands is done at this point and we're at the 10 month mark from starting the project.

i'll quickly address each point.

A full node is a really, really large amount of work. I feel that lots of people don't realise this, get enthusiastic and think, "I love Bitcoin! I love Go! I'll write Bitcoin in Go" where for Go you can substitute basically any language that's fun or popular. Then they write the easy bits (like wire marshalling) and eventually the project dies around the time that it's time to implement the wallet or Bloom filtering or robust test suites. Possibly Conformal is different, we'll have to wait and see, but the feature set they advertised in their blog is very much what has been seen many times before. In particular there's no handling of the block chain, re-orgs, no wallet and they haven't got any infrastructure to test edge cases.

all our devs used to code in POSIX-like C and that is really slow. writing this in Go allowed us to produce code much more quickly and get high test coverage for the code without needing a system external to the language.  several comments here are no longer relevant because everything mentioned, except bloom filters, is implemented.

One reason implementing Bitcoin properly is not fun is an entire class of bugs that doesn't exist in normal software - chain splitting bugs - which can be summed up as "Your software behaves how you thought it's supposed to work rather than how the original bitcoind actually does work". Bitcoin is highly unusual in that it implements group consensus - lots of nodes have to perform extremely complicated calculations and arrive at exactly the same result in lockstep, to a far far higher degree of accuracy than other network protocols. This means that you have to replicate the same set of bugs bitcoind has. Failure to do so can lead to opening up security holes via consensus failure which can in turn lead to double spending (and thus your users lose money!).

Being compatible with the way bitcoind is written (bugs and all) may require you to break whatever abstractions you have introduced to make the code cleaner or more elegant or whatever reason you have for reimplementing Bitcoin. Here's a trivial example - signatures in Bitcoin have an additional byte that basically selects between one of a few different modes. It's actually one of three modes plus a flag. So a natural way to implement this is as an enum representing the three modes plus a boolean for the flag. But that won't work. There is a transaction in the block chain which has a sighash flag that doesn't fit any of the pre-defined values (it's zero) and because Satoshi's code uses bit testing it still works. But if you turn the flag into an enum, when you re-serialise the mode flags you'll re-serialise it wrong and arrive at an incorrect result. So you have to pass these flags around as integers and select via bit testing as well.

Bitcoin is full of these kinds of weird edge cases. Eventually you come to realise that reimplementing it is dangerous and probably whatever benefits you thought it had, it probably doesn't. Some people believe there should be independent reimplementations anyway and I can understand and respect that, but doing it safely is an absolutely massive piece of work. You have to really, really, really believe in diversity to do it - the features of language-of-the-day aren't good enough to justify the effort.

this is a bit of a strawman argument that suggests it is not worthwhile to develop a proper alternative full node implementation because it *might* fork the blockchain. davec, our lead dev, has taken great care to include all of the bitcoind behaviors for chain selection exactly. check the blog entry https://blog.conformal.com/btcchain-the-bitcoin-chain-package-from-bctd/ for more info here.

it really wasn't that massive a piece of work. sure, it was a lot of work, but it took less than 12 months to catch up with 5 years of development plus whatever time the SN crew took to do their initial dev work. doing this in Go made it a lot quicker than C or C++, but we did have to work through several nasty bugs in Go. it took less than 7 manyears labor to hack out over the past 10 months and we'll be at feature parity within the next 2 months, at most.

last friday davec finished the initial secp256k1 optimizations and the entire chain downloads from scratch in roughly 3 hours (1.75 hrs to last checkpoint, 1.25 hrs from checkpoint to last block) on an i7 + ssd linux machine.

[hazek]

..this is a bit of a strawman argument..

I think it's fantastic what you are doing and that while it's great that you defend against criticism, you should really pay no attention to that thinks he knows it all because he works for that company guy.

[grau]

it took less than 7 manyears labor to hack out over the past 10 months and we'll be at feature parity within the next 2 months, at most.

Congratulations. I know how that work feels.

It took me about half a man year to reach feature parity in Java though - passing the block tests used by satoshi and bitcoinj - thereafter I worked mostly on higher level projects using the code in production.

Regarding 100% security in distributed consensus I think in the meanwhile that it is a moving target we will not reach until we split out and freeze that part of bitcoind.

I hope you succeed, as there is no push in the core team to isolate concerns.

[justusranvier]

Regarding 100% security in distributed consensus I think in the meanwhile that it is a moving target we will not reach until we split out and freeze that part of bitcoind.

I hope you succeed, as there is no push in the core team to isolate concerns.

Why could there not be a crowdfunding effort to pay someone to push improvements upstream from the reimplementations?

Do BoP and btcd both have more comprehensive unit tests than the Satoshi client? Surely everyone would benefit from getting that same level of coverage in the reference implementation.

At some point all there needs to be inter-implementation coordination of features and specs, kind of like the W3C for HTML.

[grau]

Regarding 100% security in distributed consensus I think in the meanwhile that it is a moving target we will not reach until we split out and freeze that part of bitcoind.

I hope you succeed, as there is no push in the core team to isolate concerns.

Why could there not be a crowdfunding effort to pay someone to push improvements upstream from the reimplementations?

Do BoP and btcd both have more comprehensive unit tests than the Satoshi client? Surely everyone would benefit from getting that same level of coverage in the reference implementation.

At some point all there needs to be inter-implementation coordination of features and specs, kind of like the W3C for HTML.

Unit tests of satoshi that were portable (means defined in JSON input and expected output) are all supported by BOP. It has some additional unit tests but is not as impressive as btcd sounds. I have lots of tests on much higher level, testing behavior of production systems that build on BOP, these however do not help the community as not open source.

I did expect that the Bitcoin foundation would fill the role you describe, but seen nothing noteworthy. They instead focus on US lobbyism and building a profitable international franchise. 

[justusranvier]

I did expect that the Bitcoin foundation would fill the role you describe, but seen nothing noteworthy. They instead focus on US lobbyism and building a profitable international franchise.

I'm not really impressed with them either. Apparently some other organization is going to have to take on that role instead.

[coinrevo]

At some point all there needs to be inter-implementation coordination of features and specs, kind of like the W3C for HTML.

I think this is exactly the discussion that is needed. We have the BIPs, which I understand something Amir pushed for and is now part of a formal procedure: https://github.com/bitcoin/bips , They are like RFC or W3C specs, but can't cover everything. It's a good first step, but there are several weakness how these are structured.

I think there also has to be stronger firewalls and more transparency in terms of the relationship between the users of the network, and how development interacts with the users or developers who are not core, but build on top of the network.

It seems with so many people having made so much money of Bitcoin, there would/should be some capital re-investment into the economy. Instead we are seeing big VC money for online wallets, and more and more corporate elitists getting into this. Over time this will have its effects. The interactions with the legal system are much more complex than HTML.

[behindtext]

it took less than 7 manyears labor to hack out over the past 10 months and we'll be at feature parity within the next 2 months, at most.

Congratulations. I know how that work feels.

It took me about half a man year to reach feature parity in Java though - passing the block tests used by satoshi and bitcoinj - thereafter I worked mostly on higher level projects using the code in production.

Regarding 100% security in distributed consensus I think in the meanwhile that it is a moving target we will not reach until we split out and freeze that part of bitcoind.

I hope you succeed, as there is no push in the core team to isolate concerns.

thanks grau

btcwallet + btcgui is coming along and is totally usable but has some bugs. the wallet format is deterministic (iirc) so if you backup your wallet every so often, especially after importing any new privkeys, it's totally safe.

if it weren't for all the test coverage on the various component packages, we would have been able to build a lot quicker.

totally agree with you that it's not feasible to reach distributed consensus until the entire protocol is properly documented, which it is most definitely not atm. having to dig through the tightly coupled bitcoind code is not exactly the proper way to document a protocol. it makes you wonder if it is thinly documented on purpose...

[behindtext]

I think there also has to be stronger firewalls and more transparency in terms of the relationship between the users of the network, and how development interacts with the users or developers who are not core, but build on top of the network.

It seems with so many people having made so much money of Bitcoin, there would/should be some capital re-investment into the economy. Instead we are seeing big VC money for online wallets, and more and more corporate elitists getting into this. Over time this will have its effects. The interactions with the legal system are much more complex than HTML.

the developer network is definitely in need of more transparency imo.

i probably could not agree with you more about the reinvestment angle and VC involvement. every time VCs get involved it is only a matter of time before there is (1) a race to the bottom and (2) centralization driven by heavy marketing. i figured i'd flip the script and preemptively contribute to the community before trying to build businesses on top of it.

[davec]

In how far does a re-implementation have to reproduce weird behaviour (not quite the same as bugs)?

In order to avoid chain forks, multiple implementations have to reproduce the same behavior exactly.  Even the slightest difference can cause a chain fork.  There are certainly some cases that can be considered weird behavior, but there are also some issues I would consider outright bugs.

For example, the transaction script language has a well-defined set of allowed opcodes.  However, it doesn't check the scripts for invalid opcodes until it actually executes them.  This means that you can craft valid scripts which have invalid and nearly arbitrary data in dead execution branches.  This is an outright bug in my mind.  There are also a couple of other things in the transaction scripts which are indisputable bugs.

In how far are these documented at all?

The exact rules are not really documented anywhere from what I could find.  The wiki entry for the wire protocol is fairly accurate, but the block acceptance rules are largely undocumented.  There is a wiki entry for the block acceptance rules, but it's not very accurate and doesn't provide sufficient depth.  However, I don't believe a publicly editable wiki is the right place for something as important as the protocol and block acceptance rules anyways.

This is an area I believe really needs to be improved.  Unfortunately, the general consensus I have seen amongst the core bitcoind devs seems to be "the code is the spec".  I would be happy if my take on what the general consensus appears to be is wrong.  I fully understand that writing documentation and specs is not glamorous nor fun work, but I think it's paramount for the longtime survival and growth of Bitcoin as it directly plays into multiple implementations.  I can't imagine the internet would be anywhere near what it is today if there weren't legitimate specs for things like TCP/IP and instead everyone that implemented a stack had to just go read and properly interpret some mostly undocumented code.  Unfortunately, even if a spec is written, unless the majority implementations (realistically this is only bitcoind currently) choose to follow it, it is meaningless.

On the plus side, there is a block tester tool that does a pretty good job of testing the block acceptance rules programatically.  While this doesn't eliminate the need for a spec by any means, it is definitely an excellent tool that helps support multiple implementations and would be a fantastic supplement to a real spec.

Does btcd have a different concurrency model?

Yes.  Since it is written in Go, btcd uses CSP (Communicating Sequential Processes) heavily for concurrency.  The Go mantra is "Do not communicate by sharing memory; instead, share memory by communicating", which is the model btcd strives to follow.  For example, in btcd, the server, block manager, and peers are all separate entities running in different goroutines that use channels to communicate with one another.  This provides a clear concurrency model with a clean and concise data flow.

How much of the security depends on the go-lang libs themselves?

I'm not really sure what area of security you are referring to here.  If you're referring to security issues such as exploitability of the software, I would argue that libraries written in Go eliminate an entire class of the most common causes of vulnerabilities such as stack corruption, heap corruption, uninitialized variable use, use of freed variables, and invalid frees.  This is because Go is garbage collected, all variables are initialized to their respective zero values by default, all array accesses are bounds checked by the run-time, and pointer arithmetic is disallowed.  That is not to say they're fully immune to exploits of course, but eliminating the vast majority of the most common ones by default is quite nice.

On the other hand, if you're referring more to the security issues such as coin theft, that has less to do with the implementation language and more to do with things like private key management, trusting third parties with your wallet, etc.

It took me about half a man year to reach feature parity in Java though - passing the block tests used by satoshi and bitcoinj - thereafter I worked mostly on higher level projects using the code in production.

Thanks grau.  Reaching feature parity in that time frame is an impressive one-man effort.  Well done!  That sounds very close to what it would have taken us as well to just implement everything without the architectural changes, drive for 100% test coverage in the core packages, and heavy focus on producing fully-documented reusable development components for the community as a part of the process.  I'm really glad you posted your stats as I think this goes to show that while it's definitely complicated, it's not something that is so complicated that nobody else should bother as some comments have suggested.

[justusranvier]

Unfortunately, even if a spec is written, unless the majority implementations (realistically this is only bitcoind currently), it is meaningless.

I am confident there is sufficient willingness in the community to fund the development time it would take to push the required cleanups upstream. The only problem appears to be that the organization which you'd naturally expect to be taking that effort on does not consider it to be a priority.

[LightRider]

I appreciate what everyone at conformal is doing. Thanks for taking on this important work and producing a viable alternative.

[genjix]

that async principle is a solid one to create for a bitcoin impl.

been checking in on btcd every so often, and the design/arch decisions are solid.

the code is the spec is a bit of a crap base for the future of finance and something we'll leave behind. the more of us there are, the quicker bitcoin will move forward. decentralise everything and distribute the power. we need a solid infrastructure and the bitcoind is really poor.

For example, the transaction script language has a well-defined set of allowed opcodes.  However, it doesn't check the scripts for invalid opcodes until it actually executes them.  This means that you can craft valid scripts which have invalid and nearly arbitrary data in dead execution branches.  This is an outright bug in my mind.  There are also a couple of other things in the transaction scripts which are indisputable bugs.

yes omg this is terrible. output scripts dont even need to parse. but there's plenty more like the copy paste serialization code, deeply nested locking mechanisms, cscript inheriting an std:: container (forbidden by standard), undefined behaviour (using integer loop around / invalid casts), non-modular code, ... more i can't recall right now