[Stratum] Overlay network protocol over Bitcoin

[slush]

slush, how would the transaction mechanism work?

Creating transactions is done in few steps:

1. Choose destination address (or addresses)
2. Choose source addresses (=coin selection)
3. Calculate transaction fee
4. Sign inputs
5. Combine everything into transaction as a stream.

All those steps must be addressed in bitcoinj already, because those are necessary steps for every client. Those steps are implemented also in Electrum client (python).

However things can be slightly easier, because steps 2,3 and 5 can be done on server side (it's usually called super lightweight client, because it depends on server even with coin selection). BitcoinSpinner (java based android client) is doing it already.

[1713267598]

1713267598

[1713267598]

1713267598

[1713267598]

1713267598

[slush]

But one thing that I don't understand is why are people so in a rush to implement client-server solutions, when a p2p lightweight solution is perfectly possible, and we already have BitcoinJ to demonstrate.

Because both technologies solve different problem. Bitcoin p2p is distributed database, however I'm trying to create protocol for *services* on top of this distributed database.

If you want privacy and security, you can every time use blockchain. If you need speed and convenience, use some overlay network, which masks compexity for you and provide only necessary service. It's giving freedom of choice, obviously.

By the way, not all services offered by overlay must be related to blockchain and they cannot be provided in decentralized way. For example you probably want to pick some escrow service and use it's API directly or over trusted channel, not using some anonymous peer in p2p network.

[slush]

For the usual case, you don't care that multiple messages might be equally valid.  The signature is for "this message", not for "any message bearing the same content as this message".

I probably failed I expressed my worries. So I'll try again.

Main benefit of JSON is that it can be serialized/deserialized in almost every language in transparent way, just obj = json_decode(ser_data). However, those json_encode/decode implementations don't care about order of variables in serialized data, because order isn't a part of json specification.

So json_encode(json_decode(data)) can provide different stream than original data, which is necessary for validating json message. Thanks to this issue, you need to fiddle with serialized data directly as with text stream and remove signature-related data from this stream, instead of deserializing message, removing signature-related data, serializing it back and checking signature.

As for the ID, don't leave it out of the signature unless you enjoy replay attacks.  Just force the client to make it globally unique.  There are already well established techniques for doing this.

As a prevention for replay attacks I'm proposing "timestamp" field. It's also the reason why I think that request should be contained inside the signature in some form.

[EhVedadoOAnonimato]

But one thing that I don't understand is why are people so in a rush to implement client-server solutions, when a p2p lightweight solution is perfectly possible, and we already have BitcoinJ to demonstrate.

Because both technologies solve different problem. Bitcoin p2p is distributed database, however I'm trying to create protocol for *services* on top of this distributed database.

If you want privacy and security, you can every time use blockchain. If you need speed and convenience, use some overlay network, which masks compexity for you and provide only necessary service. It's giving freedom of choice, obviously.

Fair enough. I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

By the way, not all services offered by overlay must be related to blockchain and they cannot be provided in decentralized way. For example you probably want to pick some escrow service and use it's API directly or over trusted channel, not using some anonymous peer in p2p network.

True, not everything can be done on p2p. But why those different services need to be done by the same entity anyway? You could plug your MultiBit client on that escrow.


Anyway, I don't want to sound like a naysayer. You're all doing a great work and not asking anything in return. That's noble, my sincere congratulations.
I just like the lightweight p2p solution better, that's all.

[slush]

I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

I have experience with BitcoinJ client on my android phone. Yes, it's better than downloading whole blockchain, that's without a discussion. But it's not *so* good as BitcoinSpinner client, which is lightweight. As and example, in bitcoinj I still need to wait to synchronize block headers, which is especially annoying when I'm using this wallet only time by time for face2face transactions. And lightweight clients cleanly wins in such use cases...

Personally I love freedom of choice, when everybody can choose solution which fits his needs. And I see that infrastructure for lightweight and ultra-lightweight clients is still weak, so I'm trying to fix it. Then integration into existing applications will become extremely easy, because those applications won't need to care about blockchain at all, just use some internet service with standard API which will *completely* mask the complex world of bitcoin.

[kjj]

For the usual case, you don't care that multiple messages might be equally valid.  The signature is for "this message", not for "any message bearing the same content as this message".

I probably failed I expressed my worries. So I'll try again.

Main benefit of JSON is that it can be serialized/deserialized in almost every language in transparent way, just obj = json_decode(ser_data). However, those json_encode/decode implementations don't care about order of variables in serialized data, because order isn't a part of json specification.

So json_encode(json_decode(data)) can provide different stream than original data, which is necessary for validating json message. Thanks to this issue, you need to fiddle with serialized data directly as with text stream and remove signature-related data from this stream, instead of deserializing message, removing signature-related data, serializing it back and checking signature.

No, I get it.  What I'm saying is that you need another function, json_encode_canonical() such that canonical_data = json_encode_canonical(json_decode(arbitrary_data)) and you need to implement it the same way on all platforms, so that when you calculate digital_signature(canonical_data) you are always doing it on the same data.  The json_encode_canonical() function can even ignore fields that don't factor into the signature.

As for the ID, don't leave it out of the signature unless you enjoy replay attacks.  Just force the client to make it globally unique.  There are already well established techniques for doing this.

As a prevention for replay attacks I'm proposing "timestamp" field. It's also the reason why I think that request should be contained inside the signature in some form.

Timestamps have their own problems.  Either the clocks need to be synchronized very well, or you need a window of validity, but that window of validity is also a window of vulnerability.  You'd think that synchronizing clocks would be an easy assumption, but it turns out not to be at all.

[ripper234]

slush, how would the transaction mechanism work?

Creating transactions is done in few steps:

1. Choose destination address (or addresses)
2. Choose source addresses (=coin selection)
3. Calculate transaction fee
4. Sign inputs
5. Combine everything into transaction as a stream.

All those steps must be addressed in bitcoinj already, because those are necessary steps for every client. Those steps are implemented also in Electrum client (python).

However things can be slightly easier, because steps 2,3 and 5 can be done on server side (it's usually called super lightweight client, because it depends on server even with coin selection). BitcoinSpinner (java based android client) is doing it already.

Thanks, I asked this on Stack Exchange and emailed Mike Hearn, so I imagine I'll have some direction soon enough.

[ripper234]

Fair enough. I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

To developers it's not convenient enough. I wanted to use bitcoinj for a java web app, but then discovered I would have to maintain the blockchain in order to use it. This means consuming a lot of traffic for a simple web app that will be idle 99.9% of the time.

Having an external API can make me practically stateless - I just have to store keys, that's all - which simplifies my life considerably.

[ripper234]

Timestamps have their own problems.  Either the clocks need to be synchronized very well, or you need a window of validity, but that window of validity is also a window of vulnerability.  You'd think that synchronizing clocks would be an easy assumption, but it turns out not to be at all.

What types of API call are vulnerable to replay anyway? Transactions can only be made once, and once they're made there's no use trying to replay them. Since a malicious server can't do anything too serious except lie in responses or delay things, a malicious traffic sniffer can't do anything worse in a replay attack.

[kjj]

Timestamps have their own problems.  Either the clocks need to be synchronized very well, or you need a window of validity, but that window of validity is also a window of vulnerability.  You'd think that synchronizing clocks would be an easy assumption, but it turns out not to be at all.

What types of API call are vulnerable to replay anyway? Transactions can only be made once, and once they're made there's no use trying to replay them. Since a malicious server can't do anything too serious except lie in responses or delay things, a malicious traffic sniffer can't do anything worse in a replay attack.

Until someone adds a new RPC call that actually is dangerous.  And why not add some?  The messages are signed and secure.

A system that is 90% safe is a trap waiting to spring on the first guy that forgets about the other 10%.

[Red Emerald]

Fair enough. I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

To developers it's not convenient enough. I wanted to use bitcoinj for a java web app, but then discovered I would have to maintain the blockchain in order to use it. This means consuming a lot of traffic for a simple web app that will be idle 99.9% of the time.

Having an external API can make me practically stateless - I just have to store keys, that's all - which simplifies my life considerably.

This.

I'm not interested in my thin client downloading and saving anything.  I'm working on a phonegap application that is pretty much just one html file, one css file, and one javascript file.  Private keys are stored encrypted in local storage (like GLBSE) or generated deterministically when the client is opened (we'll see how well js can do this /me crosses fingers). There is no room to download even a lightweight blockchain.

As far as signing, we should look at how other people have done it.  OAuth alphabetizes the fields and then does some more. I don't like the idea of having to implement a "canonical_encode" but it looks like it might be necessary.

http://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object

[slush]

http://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object

Thanks for great link, it's exactly the answer what I has been searching for. It's actually very similar to the solution which I discussed with genjix on IRC, but it's nice to see that others found same solution independently. It looks that I'll choose those conventions described on stackexchange. It is slightly complicating all the signing the stuff, because it de-facto needs custom object encoder. However it is necessary to implement only in clients which require signing of proxied calls; common clients (using only blockchain service directly on overlay server) probably won't care about signing at all.

Edit: excellent, I found that "canonical JSON" is widely used term and there are some implementations in various languages already, like jsonical for pyhon (http://pypi.python.org/pypi/jsonical/). I just need to test if those implementations are compatible, but it looks like step in good direction...

[slush]

Btw I closed contest about name for this overlay network. I picked name "Stratum", which is latin word for "layer".

[Red Emerald]

http://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object

Thanks for great link, it's exactly the answer what I has been searching for. It's actually very similar to the solution which I discussed with genjix on IRC, but it's nice to see that others found same solution independently. It looks that I'll choose those conventions described on stackexchange. It is slightly complicating all the signing the stuff, because it de-facto needs custom object encoder. However it is necessary to implement only in clients which require signing of proxied calls; common clients (using only blockchain service directly on overlay server) probably won't care about signing at all.

Edit: excellent, I found that "canonical JSON" is widely used term and there are some implementations in various languages already, like jsonical for pyhon (http://pypi.python.org/pypi/jsonical/). I just need to test if those implementations are compatible, but it looks like step in good direction...

This looks interesting too.  It doesn't seem to match some of the other specs I've seen though.

https://github.com/jchris/canonical-json#readme

[ripper234]

BTW, I posted this thread on reddit and it got downvoted for some reason.
(Some) Bitcoin lovers don't understand that for Bitcoin to succeed it can't remain just a P2P network used by hackers and computer geeks.

[slush]

(Some) Bitcoin lovers don't understand that for Bitcoin to succeed it can't remain just a P2P network used by hackers and computer geeks.

Don't worry, I already see there's some interest for such solution even between developers. And I'm sure that people (=users) will love solutions based on lightweight clients (and developers will be enjoying writing websites using Stratum API).

[ripper234]

(Some) Bitcoin lovers don't understand that for Bitcoin to succeed it can't remain just a P2P network used by hackers and computer geeks.

Don't worry, I already see there's some interest for such solution even between developers. And I'm sure that people (=users) will love solutions based on lightweight clients (and developers will be enjoying writing websites using Stratum API).

BTW, you can checkout my initial skeleton of an java binding here - subject to change, of course.

[slush]

You can checkout my initial skeleton of an java binding here - subject to change, of course.

Excellent! Are you going to implement HTTP transport (push or poll?) or TCP socket transport? I encourage you to implement TCP socket because of lower overhead, although I understand that HTTP can be easier to implement.

Btw yesterday I implemented message signing and succesfully tested HTTP Poll and Push transports, although Google document isn't updated yet (thanks to 2112 suggestions, I added Content-Length, Content-MD5 and X-Content-Sha256 headers, there's also X-Push-Url for turning HTTP Poll transport into HTTP Push).

I'll try to commit all changes soon so you'll be able to test your client against working server, but I still need to cleanup the code. If you have any questions or suggestions, join #stratum channel anytime!

[ripper234]

You can checkout my initial skeleton of an java binding here - subject to change, of course.

Excellent! Are you going to implement HTTP transport (push or poll?) or TCP socket transport? I encourage you to implement TCP socket because of lower overhead, although I understand that HTTP can be easier to implement.

Btw yesterday I implemented message signing and succesfully tested HTTP Poll and Push transports, although Google document isn't updated yet (thanks to 2112 suggestions, I added Content-Length, Content-MD5 and X-Content-Sha256 headers, there's also X-Push-Url for turning HTTP Poll transport into HTTP Push).

I'll try to commit all changes soon so you'll be able to test your client against working server, but I still need to cleanup the code. If you have any questions or suggestions, join #stratum channel anytime!

I'll do HTTP, efficiency really isn't relevant to my use case, and leave the TCP implementation for the future.

I don't really use IRC a lot. Do you plan to have something more than a google doc page (e.g. a homepage, wiki, reddit, ...) ?

I still don't know exactly how to generate the offline TX, and haven't yet gone over your doc - sadly the work week is starting tomorrow for me (Israeli work weeks are Sunday-Thursday), and I won't have time to work on it during the week. Perhaps next weekend.

I'm really looking forward to getting an alpha implementation up & running.

[slush]

Excellent! Are you going to implement HTTP transport (push or poll?) or TCP socket tra
I'll do HTTP, efficiency really isn't relevant to my use case, and leave the TCP implementation for the future.

Great. Keeping in mind that there might be more transports is always good idea.

I don't really use IRC a lot. Do you plan to have something more than a google doc page (e.g. a homepage, wiki, reddit, ...) ?

If you don't use any IRC client, webchat inteface can do the job for you: http://webchat.freenode.net/?randomnick=0&channels=#stratum&prompt=1

I already tried to install MediaWiki to my server, unfortunately I have some missing dependencies on the server and didn't find a time to fix it yet.  Then I move specification from Google docs there.

Homepage at stratum.bitcoin.cz will appear at some time, too. At this moment there's not a lot things to publish anyway, there's even not any server release or even installation howto :-). I'm focusing on getting it work right now...

I still don't know exactly how to generate the offline TX, and haven't yet gone over your doc - sadly

Honestly, I'm not a specialist in bitcoin p2p protocol, so I'm always finding inspiration in some existing code. For creating transactions, Electrum client is good code to start for me. However if you're Java programmer, BitcoinJ should be good choice for you and I see you're already using some code from bitcoinj...