How to report and warn on protocol incompatibilities between clients?

[Sergio_Demian_Lerner]

Since I have audited the security of the main client for a while and I'm getting bored, I turned to look at to the alternate clients (e.g. Bitcoinj Bitsofproof, etc.).

In the past both in Bitcoinj and Bitsofproof I detected incompatibilities between the Satoshi code rules and their implementations, so their clients could be "forked out" of the best chain by specially crafted blocks/transactions.

They were reported to the project maintainers and they have been fixed long ago.

Today I found another incompatibility between the rules of  Bitsofproof and the rules of Satoshi client. I will report today to Grau. Bitsofproof is still in BETA, but looks very promising.

But the point is: Should all the Bitcoin community (apart from the alternate client project maintainer) be notified of the possibility of a network split? If we get to a point where 40% of the network is running client 1 and another 40% is running client 2, then a "bug" in client 2 is also a problem for all users (not only the ones using client 2). Attacks that affect a large part of the network also undermine the credibility of the network as a whole.

So eventually in the future there should be a "higher level" list of problems/vulnerabilities of the Bitcoin network (independent of the client app), that could probably be maintained by the Bitcoin Foundation.

Best regards,
 Sergio.

[TierNolan]

I think a better way is multiple verification by miners.

Create a new message "checkblock".  This responds with a block with any illegal transactions removed (or just a list of their hashes).

This way a miner could pass a block around clients representing > 90% of the userbase and have incompatible transactions removed.  

The block would go around the circle until all clients have returned the block unchanged.

[Mike Hearn]

As far as I know, there are no re-implementations of Bitcoin where the authors suggest or recommend mining with them. So right now it doesn't seem to be a big concern.

[TierNolan]

As far as I know, there are no re-implementations of Bitcoin where the authors suggest or recommend mining with them. So right now it doesn't seem to be a big concern.

If 0.7 and 0.8 supported a checkblock packet, then the recent fork may not have happened.  The difficult transactions would have been dropped.

[kjj]

As long as the maintainers are responding in reasonable times, I'd keep things private.  The full disclosure movement arose because certain vendors were collecting private flaw reports, but then not fixing them.  Going to full public release was a way to force their hand and make them fix things promptly.

Other people see things differently, of course, but if it were me, I'd stick with private communication until that stops working.

[grau]

Today I found another incompatibility between the rules of  Bitsofproof and the rules of Satoshi client. I will report today to Grau. Bitsofproof is still in BETA, but looks very promising.

But the point is: Should all the Bitcoin community (apart from the alternate client project maintainer) be notified of the possibility of a network split?

Thank you for taking a deep dive into the code and reporting your findings. I am not (yet) convinced that the difference you spotted by reading the code does lead to exploitable different behavior, since there are numerous tests shared between Satoshi and bits of proof targeted exactly at these subtleties. Specifically the tests https://github.com/bitsofproof/supernode/blob/master/server/src/test/resources/script_valid.json
and https://github.com/bitsofproof/supernode/blob/master/server/src/test/resources/script_invalid.json would have very likely caught the difference you claim.

The process you followed by notifying the author with the details in the first place is correct. It would be fair to wait for a confirmation of a vulnerability and its fix before you announce that you found something, just to avoid it sound like FUD-ing an implementation. There are bugs in every software and there must be countless differences between implementations. Bugs have to be fixed but differences have to be carefully evaluated if they really offer a practical exploit.

Your work is valuable to all of us, please continue but be vary not only the network security but the reputation of the implementations that in some respect also support security.

[Sergio_Demian_Lerner]

Today I found another incompatibility between the rules of  Bitsofproof and the rules of Satoshi client. I will report today to Grau. Bitsofproof is still in BETA, but looks very promising.

But the point is: Should all the Bitcoin community (apart from the alternate client project maintainer) be notified of the possibility of a network split?

Thank you for taking a deep dive into the code and reporting your findings. I am not (yet) convinced that the difference you spotted by reading the code does lead to exploitable different behavior, since there are numerous tests shared between Satoshi and bits of proof targeted exactly at these subtleties. Specifically the tests https://github.com/bitsofproof/supernode/blob/master/server/src/test/resources/script_valid.json
and https://github.com/bitsofproof/supernode/blob/master/server/src/test/resources/script_invalid.json would have very likely caught the difference you claim.

The process you followed by notifying the author with the details in the first place is correct. It would be fair to wait for a confirmation of a vulnerability and its fix before you announce that you found something, just to avoid it sound like FUD-ing an implementation. There are bugs in every software and there must be countless differences between implementations. Bugs have to be fixed but differences have to be carefully evaluated if they really offer a practical exploit.

Your work is valuable to all of us, please continue but be vary not only the network security but the reputation of the implementations that in some respect also support security.

Yes Grau, you're right. That's why I didn't make public the problem details. But honestly, the problem is there in the Bitsofproof code, I'm quite sure.
The fact that BisofProof is still in Beta (but won't be in Beta forever) is the reason that I'm asking people: how should we handle a problem with one (but not all) of the client implementations.

I think that, for the community good, as fast as a bug that can cause possible network split is discovered, all users must be notified. I mean all (including users that do not use this client). This does not mean that the bug must be exposed. But if people know their clients can bee "knocked out" of the best chain, they can put additional protective measures to periodically check if they were.

Related to the test cases Grau posted, I'm quite sure they do not check the problem I spotted.
Those test cases do not test the result of each opcode, only they test if the script verification fails or not.
That's completely wrong, or at least it's incomplete.
If you want to check if OP_ADD1 works, you should check that before executing the opcode and afterward the stack have some known values. Now the script test cases only check that the result is TRUE or anything else.

Gavin or donor coders should try to build more detailed test cases for each script arithmetic opcode, to avoid screwing up things in future versions and to help other implementers to verify their codes.

Grau: my impression is that your code is good, as is Mike Hearn's code. The problem is not on your code, but in the lack of a "Bitcoin Bible" manual to specify the hidden and often forgotten rules of the protocol.

Would you allow me to post the problem here?

[grau]

Related to the test cases Grau posted, I'm quite sure they do not check the problem I spotted.

Spotting a problem with code review might be successful, but this is more about expected behavior than
code differences.

In this concrete case we have tests that document expected behavior. Those can be executed against
several implementations. They might miss something you spotted, therefore I suggest you formulate
the case you think exposes different behavior in that JSON notation, then let us review, then give me
a few days to fix and then ...

Would you allow me to post the problem here?

... you get the glory and the community a new test case that is a piece of the standard for implementations.

[Sergio_Demian_Lerner]

Dear Grau,
 I failed to exploit the supposed "problem". I cannot create a JSON test that differentiates the two implementations. 

I apologize for the inconvenience.

Nevertheless the subject of the thread, which was not to talk about any particular problem, remains useful: how to handle the notification and collaboration between parties when a flaw is discovered in an implementation that is not the reference one but affects the rest.

Best regards.
 Sergio.

[Sergio_Demian_Lerner]

Eureka! 

[scintill]

Create a new message "checkblock".  This responds with a block with any illegal transactions removed (or just a list of their hashes).

This would be too much traffic, I think.  It's effectively rebroadcasting dozens of tx every time a block is solved, in preparation for making the next one.  It could also lead to arbitrary txn censorship (which has merits, but I assume wasn't intended).

[grau]

Create a new message "checkblock".  This responds with a block with any illegal transactions removed (or just a list of their hashes).

This would be too much traffic, I think.  It's effectively rebroadcasting dozens of tx every time a block is solved, in preparation for making the next one.  It could also lead to arbitrary txn censorship (which has merits, but I assume wasn't intended).

The idea is to use this by the mining pool operator, not in the wild.

The pool operator would add value by validating blocks agains multiple implementations and versions before distributing to work on it. This would be a value add for the pool, since reducing chances that work is wasted in a branch orphaned later by majority.

[TierNolan]

This would be too much traffic, I think.  It's effectively rebroadcasting dozens of tx every time a block is solved, in preparation for making the next one.  It could also lead to arbitrary txn censorship (which has merits, but I assume wasn't intended).

I was thinking that it would be an option that miners would enable, rather than everyone.  You would add the list of slave nodes to bitcoin.conf, "addverificationnode=127.0.0.1:3456"

A mining pool could run multiple versions the software.  One node would be master and would run as normal.  It could begin mining immediately when a new block is created.

However, it would broadcast the new block and any blocks it receives to the slave nodes.  The slave nodes would be on the same machine or at worst the same LAN.  The response could just be a list of illegal transactions.  If any object to a block it received, it doesn't accept the block and if any object to a block it was mining, it would remove the transactions and try again.  99% of the time, the slave nodes would clear the block without change, if it cleared them.

Smaller miners should run the old and new versions of their current software, especially when there has been a recent update.

The process could produce a log.  If a TX was banned, the entire block could be logged (with spam protection to prevent DoS due to disk access by sending loads of bad blocks).  The client could even have an option to automatically send the block to the devs.

The idea is to use this by the mining pool operator, not in the wild.

The pool operator would add value by validating blocks agains multiple implementations and versions before distributing to work on it. This would be a value add for the pool, since reducing chances that work is wasted in a branch orphaned later by majority.

Exactly