[Stratum] Overlay network protocol over Bitcoin

[ThePiachu]

Very interesting project! Watching this (having read through all the previous post, took me awhile :\ ).

Was also considering doing something similar, guess I'll do my best to conform to this standard. Might be a bit hard though, as Google App Engine with Go has some issues with TCP/IP and RPCs. URL encoding would be easier, but we'll see if I can hammer the code to work.

[1713481885]

1713481885

[1713481885]

1713481885

[1713481885]

1713481885

[1713481885]

1713481885

[slush]

GAE has no problem with json, so no problem with encoding here. However GAE does not support TCP sockets and persistent connections, except "Channel API", which is highly proprietary stuff and there's no reason to include this into Stratum "standard". However HTTP transport is definitely the way how to implement Stratum on GAE. Are you using Python or Java on GAE?

I'm going to implement set of unit tests, so it should be pretty easy to test if alternative server implementation fits all protocol requirements. I'm still working on another project right now, but I'll be back on Stratum in few days.

[ThePiachu]

I'm writing in Go, so if I get everything to work it should be quite fast. But Go apparently has no support for JSON-RPC over HTTP - http://code.google.com/p/go/issues/detail?id=2738 (so I had to create my own package for that), and the RPC package doesn't like GAE - https://groups.google.com/forum/#!msg/google-appengine-go/uQ0cv0m9j0E/H3VCrFgEWvcJ .

So far I managed to get a simple Python request to go through to my app - https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29#Python , and got the app to communicate a bit with bitcoind, but I'm having some problems communicating with a miner, so I have to look into it further.

[slush]

I'm writing in Go, so if I get everything to work it should be quite fast. But Go apparently has no support for JSON-RPC over HTTP - http://code.google.com/p/go/issues/detail?id=2738 (so I had to create my own package for that), and the RPC package doesn't like GAE - https://groups.google.com/forum/#!msg/google-appengine-go/uQ0cv0m9j0E/H3VCrFgEWvcJ .

Well, Stratum is not clear JSON-RPC, it's only based on it (format of the messages is the same, but with some additional things like session management and callback URLs). For your purposes the JSON library & possibility to process HTTP POST requests should be enough to implement Stratum in few lines of code.

Interesting, I didn't notice Go support on GAE. I had to say that I don't follow GAE progress in last year.

[casascius]

I would like to propose one more thing that Stratum should keep track of, and perhaps one LESS thing.

The More thing:  Nodes should inform Stratum that they can exchange P2P information about subject X, where X is an arbitrary string that is chosen by the developer of whatever X is.  In turn, Stratum helps nodes aware of subject X find one another.

The Less thing:  The block chain should not be assumed as something that will be exchanged by default, but rather, should be one of many subjects that can be subscribed to.

Now, why?!  At some point, I believe that the block chain as we know it is going to get called the "Satoshi" block chain, or the "orthodox" block chain, and people are going to start new chains that work as subsets of the Satoshi block chain, so they can function as a whole bitcoin node without having to download the whole damn thing.

Someone who accepts a pruned block chain is ultimately putting their trust in whoever pruned it.  This is a tradeoff that becomes more likely to happen every day the block chain gets bigger.  Someone who accepts a pruned chain also cannot participate in exchanging historical blocks with orthodox clients - they can only exchange such blocks with others who trust in the same chain.  Hence Stratum being an ideal hookup mechanism for these clients - but without Stratum having to bear the burden of being the relay (except where the operator chooses to make his server relay such things).

An example is if I go and compile a digest of the block chain that includes only the unspent outputs as of block 150,000, I sign it, I publish it.  People want it because it's only 150 MB instead of a gig, and let's assume others are vouching for the integrity of it (in other words, consensus on the forums is that the outputs it contains are consistent with the orthodox block chain as of the same block height, and someone publishes a utility to confirm that).  From then onward, all newer blocks are interchangeable with the Satoshi block chain, it's just that orthodox Satoshi clients will not understand how to deal with a peer that has recent blocks but not the whole chain.  I would want Stratum to help me hook up with clients that are expecting a floor of 150,000 as well.

Let's say somebody else goes and creates a combined client that can trade Bitcoins, Namecoins, Litecoins, and whatever other kinds of coins, but all within the same client and protocol.  Ideally, Stratum should help that client find other like-minded clients so they can exchange all of those blocks, all without burdening Stratum with block types its server operators don't want to waste resources on.

[Red Emerald]

I would like to propose one more thing that Stratum should keep track of, and perhaps one LESS thing.

The More thing:  Nodes should inform Stratum that they can exchange P2P information about subject X, where X is an arbitrary string that is chosen by the developer of whatever X is.  In turn, Stratum helps nodes aware of subject X find one another.

The Less thing:  The block chain should not be assumed as something that will be exchanged by default, but rather, should be one of many subjects that can be subscribed to.

This is a really great abstraction.  Building the stratum protocol to transmit arbitrary data is a great idea.

Someone who accepts a pruned block chain is ultimately putting their trust in whoever pruned it.  This is a tradeoff that becomes more likely to happen every day the block chain gets bigger.  Someone who accepts a pruned chain also cannot participate in exchanging historical blocks with orthodox clients - they can only exchange such blocks with others who trust in the same chain.  Hence Stratum being an ideal hookup mechanism for these clients - but without Stratum having to bear the burden of being the relay (except where the operator chooses to make his server relay such things).

I was under the assumption that the pruned blockchains would have verifiable hashes in them so that you wouldn't have to trust the distributor.  Stratum should definitely keep this in mind.

Let's say somebody else goes and creates a combined client that can trade Bitcoins, Namecoins, Litecoins, and whatever other kinds of coins, but all within the same client and protocol.  Ideally, Stratum should help that client find other like-minded clients so they can exchange all of those blocks, all without burdening Stratum with block types its server operators don't want to waste resources on.

Wow. This is a great idea.  I don't want to have a run a stratum server for bitcoin and a stratrum server for namecoin and so on.

[casascius]

I was under the assumption that the pruned blockchains would have verifiable hashes in them so that you wouldn't have to trust the distributor.  Stratum should definitely keep this in mind.

Somehow, I believe downloading a "seed" blockchain will be more preferable in practice than getting pruned blockchains bit-by-bit from peers, for two reasons.

One, I am convinced (though smart others have disputed) that if you accept pruned blockchains bit-by-bit from other nodes, you can be disrupted (DoS'd) simply by getting fed a block that has the wrong things pruned from it (example, someone pruned an unspent transaction that should not have been pruned).  Sure, you'll be able to detect you don't have the complete picture, because the "verifiable hash" you refer to won't check out, but you'll have a difficult time finding which block(s) you were misfed, which you need to ask for again - recovery from which will be at least as burdensome - and definitely more complex - as compared to downloading a seed blockchain as a signed flat file (either via http, torrent, etc.).

Second, even when blocks are pruned, there's plenty of "cruft" that you really don't need to keep if you don't demand complete (orthodox) cryptographic verification of the block chain file.  When you prune nodes off a merkle tree, you still have to leave behind hashes of the branches you cut off to be able to validate the tree.  Those are wasted space - what I mean by this is they take up bandwidth and permanent storage but are useful only once - for validating the integrity of the file - and will never be accessed for any other reason.

Another thing that's a total waste of space is the inputs of all the kept transactions, which are much larger than the outputs.  The inputs take up way more space each (they contain a public key and a digital signature - over 130 bytes each) and outputs are much smaller (like around 30 bytes).  You don't need the inputs for validating new transactions, just the outputs.  But with the current merkle tree setup, you can only prune a whole transaction at a time - all inputs and all outputs must go together or not at all.

Also, another big space waster: transactions with "multiple outputs" (generated by sendmany), which must be kept in their entirety if a single output remains unspent, they cannot be pruned with the current merkle tree spec.  This would not be so bad except for the fact that some mining pools issue huge sendmany's with numerous tiny outputs worth only pennies each - to distribute payout to miners in the coinbase of their blocks.  These transactions under the orthodox specification can never be pruned until every penny output has been spent - which for most of these blocks may very well be never. (example transaction from P2Pool: http://blockexplorer.com/t/42BmXQaM9T)

Now, on the other hand, if every 10k blocks or so, somebody can compile all of the unspent outputs into a signed file, discarding all inputs and all spent outputs, that file is always going to be a tiny fraction of the size the orthodox block chain.  If they also released source code to a validation utility (which can be run only by those possessing the complete orthodox block chain), the integrity of that file can be confirmed (and also signed) by the more concerned members of the community, and the file can then be trusted as safe based on community consensus.

A useful function of Stratum would be to pass around those endorsing signatures so that a client can quickly determine which is the most recent pre-compiled "block chain digest" that enjoys the overwhelming majority of community support.

[Red Emerald]

Hows this going? I re-arranged my server and have been to busy coding to play around with the protocol

[bc]

A useful function of Stratum would be to pass around those endorsing signatures so that a client can quickly determine which is the most recent pre-compiled "block chain digest" that enjoys the overwhelming majority of community support.

Should Stratum keep track/report on nodes who endorse digests later proven to be in some way false? Is there enough peer pressure to promote only honest endorsements?

[genjix]

Hows this going? I re-arranged my server and have been to busy coding to play around with the protocol

I think slush doesn't have much time. Anyway, I'll pick up the slack. I'm working off slush's original proposal, but need to make some design decisions in coordination with ThomasV.

I'll only focus on two use-cases:
- Merchants
- Users

[slush]

I think slush doesn't have much time. Anyway, I'll pick up the slack. I'm working off slush's original proposal, but need to make some design decisions in coordination with ThomasV.

Genjix, can be more specific where and why you needed to decide something? Afaik the protocol is fully covered. I read your proposal (https://bitcoinconsultancy.com/wiki/Intersango/Overnet) and I think there are many things to discussion. Namely - why you choose to not follow json-rpc specification? I'm really missing "id". Also variables "method" and "result" together don't sound clear.

Btw after a while, I'm back in the project. Expect some progress in git repository in following days.

[ThomasV]

I think slush doesn't have much time. Anyway, I'll pick up the slack. I'm working off slush's original proposal, but need to make some design decisions in coordination with ThomasV.

Genjix, can be more specific where and why you needed to decide something? Afaik the protocol is fully covered. I read your proposal (https://bitcoinconsultancy.com/wiki/Intersango/Overnet) and I think there are many things to discussion. Namely - why you choose to not follow json-rpc specification? I'm really missing "id". Also variables "method" and "result" together don't sound clear.

Btw after a while, I'm back in the project. Expect some progress in git repository in following days.

Please do not consider the current syntax as a conscious choice; we just needed to start with something concrete.
Genjix asked me to describe what information is needed by the client, and your document does not include that. This is why I wrote this specification.

Please feel free to suggest improvements concerning the syntax & protocol. For example, we can use an "id" field if you think it is better.
Again, I did not make the conscious choice not to use an "id" field; I was focusing on something else. I was focusing on the specification of the services

[slush]

Thomas, I understand and I'm fine with that. Then it's better to specify services/methods/parameters and forget to wire format at all. I specified wire format, but I didn't specify blockchain methods, because I didn't have enough experience to do it months ago. I assume that you have much more knowledge to do that than me now, so we can work together.

Everything necessary is to define method names (e.g. 'blockchain.address.subscribe'), their parameters (e.g. 'address', 'status'), result types and exceptions (possible exceptions which can be thrown by given method, and exception parameters).

Btw this is pretty hard to do clearly and cover all possibilities (especially all those possible exceptions and corner cases). This is the reason why I prefer easy API without complicated internal logic. But with the list of necessary methods, parameters etc. it's pretty easy to transform it into wire format.

[slush]

Thomas, I composed very simple document to begin with the blockchain specification:
https://docs.google.com/spreadsheet/ccc?key=0Al5UmiyYRxBudHl2VFhJWG5rT09maU9kbzltR3hBdmc . Give me your google docs email, I'll enable edit mode for you.

I combined proposal from Stratum draft document and wikipage on bitcoinconsultancy from you and genjix (marked bold). I have some comments to parameters/responses defined on that wiki page, I'll wait to you and genjix on IRC, it will be much faster than discuss it on forum or wiki...

[dogisland]

Wow, I've seen this thread hanging round for months but this is the first time I've really got it.

This is a great piece of work, I'm going to re-write Strongcoin to sit on top of Stratum, this is excellent.

[Traktion]

What are the odds that this overlay network could be used to permit two mobile devices to transact directly, in the absence of Internet access for one or both parties?  For example, a direct ad-hoc wifi connection or an indirect connection via a 'piratebox' type device?  Asuuming, of course, that the sending device doesn't need Internet access to produce a transaction from it's own, known address funds.

+1

Great thread + software!

I would like to know if you could pass a signed transaction onto another party (say, from the payer to the payee) too. This would be very useful for areas with limited data connectivity, as it would only require the receiver to 'upload' the transaction.

You could also have situations where both parties had no Internet access and such transactions could be treated like cheques, which could be cashed when the payee is next online.

[slush]

I would like to know if you could pass a signed transaction onto another party (say, from the payer to the payee) too. This would be very useful for areas with limited data connectivity, as it would only require the receiver to 'upload' the transaction.

Yes, this is possible with signed transaction.

You could also have situations where both parties had no Internet access and such transactions could be treated like cheques, which could be cashed when the payee is next online.

But you need to be really careful here. The fact that transaction is signed doesn't mean that it is valid. If you cannot fully trust payind side, you shouldn't receive such transaction offline...

[marcus_of_augustus]

But you need to be really careful here. The fact that transaction is signed doesn't mean that it is valid. If you cannot fully trust payind side, you shouldn't receive such transaction offline...

People who write rubber checks are not a new phenomena !

There could even be an off-line market in "Junk signedTx", where such things are traded at discount to face value in the off-chance delinquent accounts are eventually made good by people looking to resurrect their financial reputations.

[markm]

The signed transaction probably should be transfered in siged messages, signed by various personal or corporate identity and/or web of trust signatures, so there is not only a signature showing what bitcioin addresses committed to pay the amount but also which entity certified that they themselves issued that "signed bitcoin-transaction".

Maybe including pubkeys of actual entities the receiving addresses were believed to belong to, with signature from them certifying those receiving addresses are indeed theirs.

Then receiver of the junk "cheque" has legs to stand on come collection time or saleback-of-rep time.

-MarkM-

[apetersson]

is there a method to query addresses in bulk?
like the proposed bloom filter commands in the main client?

and what is the story if a chain reorganisation occurs? how can a transaction become invalid?