Bitcoin Forum

Hello.

I couldn't find any other thread for this issue so here goes.

The following pull request was denied a while ago. https://github.com/bitcoin/bitcoin/pull/63
I would like to take it up for discussion since I know I am not alone to feel that it is a central thing to have the protocol version != client version.

Surely the other client makers out there can agree that this is a necessary thing?

The issue as I understand it is that Satoshi's implementation in C++ is considered the reference implementation, and the protocol is defined by it, hence the coupling between the protocol version and client version. Besides, despite bitcoin now being a multi-million economy, the protocol is still not fully documented, and no production ready alternative implementations exist for that reason. Bitcoin is still considered beta software, and the protocol is supposedly still evolving. Additionally, Satoshi's code is so complex that nobody really knows if the current protocol covers all the code paths in his implementation.

I also think that a lot of the core developers are opposed to alternative implementations as that will dilute their power over the bitcoin network.  

I personally believe that we should have a simple and clearly defined protocol as soon as possible, and encourage a proliferation of alternative clients.

Really?  I've been encouraging alternative implementations, who is the power-hungry core developer?
Ok.  Start here:  https://github.com/bitcoin/netspec
Or here: https://en.bitcoin.it/wiki/Category:Technical

I guess this is kinda on-topic: http://dwellonit.taterunino.net/2011/05/19/reference-implementations/

Well, that document and the Wiki differs somewhat. https://en.bitcoin.it/wiki/Protocol_specification.
Also, that text document says that the version "When a node receives an incoming connection, it will immediately advertise its version." which I think is no longer valid. What is the point of having the documentation if it is not up to date?

Also, the bootstrapping documentation is nowhere as far as I know. There are some pieces here and there but nothing that solid.

What I would like to see is something like a feed that describes all protocol changes and what motivated them. If I am to build an alternative client I want the core developers to know that it is a PITA to follow protocol changes and keep them to a minimum. Do not misunderstand me, I am all for evolvement and such, but I would like a real discussion (mailinglist anyone? bitcoin-protocol-dev?) before making anything permanent forever.

I think that using a separate protocol version will help facilitate the "keep the protocol stable" thought and also dot introduce protocol versions where the actual protocol are the same. For example, did you change anything between 32100 and 32200? As far as I know nothing P2P was changed (a bit bootstrapping though) - a protocol version increase here is accordingly to be totally uncalled for and only confuses developers.

 

I am grateful for all the work that you are doing on bitcoin, and I didn't mean to say that you are power-hungry developer.

However, it is frustrating to see how the issue of separating the protocol version from the client version was rejected in this Github issue without any clear explanation.

Absolutely correct that the protocol must be hammered out ASAP.

Satoshi did the crucial work of getting the reference implementation out. This fooling around with minor client changes is a waste of valuable time. More important are protocol adjustments which will strengthen the network and consistent documentation. Many different clients will follow once the documentation exists.

Obvious adjustments: perhaps consider SSL-encrypted connections (we had best anticipate the blocking of bitcoin connections by states/ISPs). IRC bootstrapping looks way too much like botnet activity and may easily be stopped by ISPs. Those kinds of steps would make things much easier and prevent the obvious "bumps in the road" that Gavin keeps mentioning.

Also, there is no manual for bitcoind. Documentation is an obvious requirement.

https://en.bitcoin.it/wiki/Protocol_specification

Also, it's important that the client has no problem running on any arbitrary port. I tried to change the default port to have a mainnet and testnet client on the same box but the non-standard port client didn't get any connections. Maybe it's me that did something wrong, jsut saying.

Aren't a bunch of node IPs already harcoded in the client ? (haven't got a clue about this one, but could be a good temporary fallback)

IIRC jgarzik was arguing that the protocol version could be differentiated using a set of flags instead and that the protocol version field should be frozen, can't find that pull request anymore :/

EDIT : That was in OPs post... Protocol extension is different than protocol versioning, I think gavin should stand up :)

I opened (and closed) the pull request, even if I found it quite stupid to create a pull request for a semantic change...

Anyway I wanted to freeze the version number in the version message, and only change it when breaking changes are introduced. This way we could create clients that see a version number and say "ok, I speak that too", and then go ahead and communicate with each other. As it is now each new client version potentially brings breaking changes, and we have no way of being sure that the protocols are compatible.

I closed the pull request, because it was just plain stupid to have one for 2 added comments, and I thought it best to wait for the topic to come up again.

I would love to have more comments from the guys in charge - there seems to be a good number of persons who would love to see this happen and frankly I cannot see that this would be something troublesome on your part.

Would be nice if there was one downloadable document containing the complete Bitcoin specification. Would be nice if the specification were set up in a way that it wouldn't have to change anymore. Would be nice that if it had to change, all the community would have a vote in it. Whole lotta woulds, I know...

http://tinyurl.com/6xfhypu

It has already been linked, that document is wrong in many areas and appears to be updated by people who detect errors instead of the developers when something is actually changed. There is also a git:ed document that differs from the wiki on other parts. Both are outdated.

Would you be interested in collaboratively creating a PDF in LaTex that would capture both the message format as well as the protocol behavior (something like the Pseudo-code presented in http://amzn.to/mgpDhH )? I had the idea a while back, but I'm under exams for another week.

Actually I would. I promise you that either I will work on such a document of a competing currency or help you with one that nails down the current Bitcoin specs. However, either it's going to be a spec that is unlikely to ever change, or I'd want some guarantees that it couldn't be randomly adapted by a small group of people, other then including me |-).

I wouldn't call this encouraging:


Perhaps you've changed your mind in the mean time?

My bad :)

Reality check: Gavin wrote to me only days after the BitCoinJ release to tell me how happy he was to see an alternative implementation. Satoshi expressed very similar sentiments. Nobody is against alternative implementations.

What some people, especially Satoshi, have said is that there's an unusual amount of risk involved with reimplementing the full system and using that reimplementation to mine. Bitcoin is very complex and if you aren't skilled and very thorough you are likely to diverge from its behavior in small, hard to detect ways. This can fork the chain and split the economy. It's one of the few things that could instantly kill Bitcoin beyond legal harassment of its users.

Anyone who isn't completely fluent in C++ and the cryptography Satoshi used should think twice before deciding to write a new implementation. It's a huge amount of difficult work. BitCoinJ targets the simpler, less risky SPV/client-mode profile for this reason: bugs in it only will impact the user of that app and not the entire network because it can't mine. Even so, it's a large effort.

My general opinion is that if the thought of reimplementing Bitcoin from scratch doesn't make you take a deep breath, then you don't understand the system well enough to do it.

As to the version number, this is a pretty trivial thing. Does anyone serious about re-implementations actually care about it? The protocol is backwards compatible to the first version, so the number only identifies which set of features the other side supports. There are service flags for the case where you want to express optional features independent of protocol version.

Better protocol documentation would be great and I encourage people to write some (I've done my part with the articles on alt chains and contracts). All the core devs are busy right now with other things so, you'll have to dive in yourself.

Good, it may just be that the core developers are busy (as you pointed out) now when Bitcoin has taken public interest.

Isn't this why we need client diversity? Bad blocks will not be accepted by other clients, having 1/3 of client A, 1/3 of client B and 1/3 of client C will guarantee that no one client will fork the chain. As it stands now the whole chain may have been hashed faulty.

I assume you mean that the C++ knowledge is needed in order to understand the current client, right? If so, I hardly think that you need complete fluency in C++ as the client (the parts I've looked on) doesn't really use many of the advantages of C++. Sure, it has boost but that is about it. For the record, I have only reviewed the network and IRC code - might be some C++ voodoo elsewhere I assume. Anyhow, I think you are overestimating the difficulty of the task.

Of course it is a deep breath! It's huge - I think this can be concluded from all the stalled alternative clients (A bunch of libbitcoins, bitcoin-alt, pybitcoin). Implementing basic client support would be the first step - hashing and other stuff would be step two obviously.

For me it may be a more ideological standpoint that if the developers do not want to merge such a trivial change that will greatly enhance the community by the looks of the discussions - then they do not appear to want 3:rd party developers. The protocol is the heart of a P2P - can you blame people for wanting it somewhat fixed?

If I were to create a client the last week, I would probably have announced it as "protocol version 32100". This week it is "32200" but as far as I can tell no changes. In my eyes, it just seems stupid.

The wiki has some nice discussions on the Talk page. There is a number of concerns how the fields are used and such, but it is hard to get a consensus on how it "is supposed to be". I would love to have a "bitcoin-protocol" mailing list that such issues could be discussed - but AFAIK the development team does not seem too excited about mailing lists.

That's FUD and bullshit. Seriously, a wrongly implemented client? "fork the blockchain"? "split the economy"?


Alternative clients are a good thing, so is protocol documentation :)

If the mainline client is really complex to understand and hard to read then it would be a very good thing to have a high-level language client which source is easy to read.

Mining is not really the point here, if you can check transactions and hash a block header then obviously you have everything you need to mine. So the distinction between clients than can mine and those who cannot is irrelevant.

No, unless all three clients are precisely synchronized in every last detail to Satoshis code it guarantees spurious chain forks. The worst possible configuration is a 55/45 type split between two clients, but A/B/C is also not great - assuming the clients have subtle bugs. If none of the implementations are buggy, no problems.


Sure, but I mean you need to be aware of differences like uint32 vs int32 and so on.


Real world protocols have uglyness that isn't worth fixing. My browser identifies itself as Mozilla, Gecko, Safari and Chrome simultaneously. It speaks HTTP/1.1 with extensions that never resulted in a version number bump and HTML "5" that has an ever shifting feature set.

Usually, there are more important problems to solve. I agree it'd have been better to have separate version numbers from the start, but it wasn't done.


You might want to read the code of conduct thread in the project development forum. Anyway, consider the worst case scenario. Two clients, A and B. A is Satoshis code and B is from somebody else. They have roughly equal usage.

During the implementation of B, by mistake the third condition in CheckBlock wasn't copied:


Things work just fine for months, then one of two things happens. A miner whose clock is set wrong finds a block using client B and broadcasts it. Or, somebody notices the bug and decides to exploit it by deliberately mining a block with a far future timestamp.

Once this bad block is broadcast, the chain splits in two because client A refuses to accept it and the miners keep working on the existing block. But client B accepts it and begins building on top of it. The bogus chain B is now ahead of chain A ensuring the bad clients keep building on it.

This situation is pretty disastrous. Things will seem to work at first, because the same transactions will be accepted by both chains, but it has halved the network speed/security. Worse, double spending is now easily possible because the network can be turned against itself.

If you notice the split has happened, or were expecting it, you can broadcast a tx that sends the contents of your wallet to a new address you control. The tx has an nLockTime set to the current height of chain B and a low sequence number. It is immediately considered final and included by clients on chain B because their chain is one block ahead of chain A, but clients on chain A will wait a block before including it. After a block including your tx is broadcast on chain B, you broadcast a replacement tx changing the destination address to another new address you control. Clients on B ignore it because it's a double spend, but clients on chain A accept it because that tx wasn't finalized yet.

Congratulations, you just doubled your money - temporarily. You can now find merchants who are on chain B and spend your wealth with them to get whatever you want, safe in the knowledge that when the chains re-merge those spends will all be undone. Whilst you're doing this, the Bitcoin devs are running around like crazy trying to find/contact miners on chain B to get them to stop mining so A can overtake it. But, there's no central registration point for miners or even an announcements mailing list, so this will take a while. There will also need to be a crash upgrade of miners using client B to avoid the same bug being exploited over and over again.

Once the split occurs, there is less than 24 hours available to re-merge the chains before newly minted coins mature, become spendable and causing even more rolled back transactions.

Such an event would cause chaos and seriously undermine confidence in the whole Bitcoin system. Worse, if you have one such bug, it's a reasonable inference you weren't thorough enough, so attackers will immediately start looking for more.

This is why it's absolutely critical that re-implementations match the behavior of Satoshis code in every last detail, including features that aren't being used today. There are a lot of details in Bitcoin, so that's not easy to do, and the costs of failure could be very high if you client becomes popular.

Your example only works in the very specific context of two clients with roughly 50% hashing power each.

The attack/bug you describe is absolutely impossible with more than two equally distributed implementations.

The moment you introduce a C client that has 33% of the network power, a bug in client B will have no other consequences than to trigger a couple of chain reorgs for B miners before someone notices something is wrong in the B client implementation. End of story.

That's why I think and maintain that the point you spread is pure FUD.

...and if the client version were allowed somewhere an honest but buggy alternate implementation could have detected that it had diverged from the satoshi code and disabled itself.

Alternate umplementers are unlikely to want to break things, but it's made hard for them to be helpful without this sort of extra information being allowed. Seriously, how hard would it be to redesignate the (now unused) subversionnum field as clienr version? It could be done now while there are no alternates and allowed to propagate for a few official versions, but the devs are denying it.  Oh well.

Busy... very busy...

Several mini-proposals related to protocol version were just posted to the bitcoin-development list (https://sourceforge.net/mailarchive/forum.php?thread_name=BANLkTi%3D4UeK2D0cCHYGp1tHUTGA2iyWHcA%40mail.gmail.com&forum_name=bitcoin-development).

Those are all systemic changes that can be sufficiently provisioned by voting with your feet. I think there's potential for your ideas but they are firmly biased toward top down management. I can't hide the fact that it sets off all sorts of triggers.

What would help is libxcwallet, libxcpolicy, libxcinterface. xc stands for any coin. We will want to interoperate with bitcoin competitors as well.

I like the idea of including the protocol version.

I'm not quite sure how we managed to get onto the topic of different clients implementing the protocol differently, but I personally can't see how adding an extra field (particularly while most people use the most recent version of the original bitcoin client) could cause any harm.  Assuming the protocol remains backwards compatible and miners accept blocks with the new field.

If I am missing something though, feel free to try to explain it in somewhat different words.

What would the goal be for multiple implementations of the core logic?  Different languages or something?