Bitcoin Forum
December 04, 2016, 08:37:45 PM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 5 6 7 8 9 [10] 11 »  All
  Print  
Author Topic: [Stratum] Overlay network protocol over Bitcoin  (Read 33230 times)
ThePiachu
Sr. Member
****
Offline Offline

Activity: 442



View Profile WWW
January 21, 2012, 11:11:39 PM
 #181

Very interesting project! Watching this (having read through all the previous post, took me awhile :\ ).

Was also considering doing something similar, guess I'll do my best to conform to this standard. Might be a bit hard though, as Google App Engine with Go has some issues with TCP/IP and RPCs. URL encoding would be easier, but we'll see if I can hammer the code to work.

1HWbVLhxj7bhewhyapMZpyhqWAeAhJd51E
My Bitcoin Calculator:
http://tpbitcalc.appspot.com/
1480883865
Hero Member
*
Offline Offline

Posts: 1480883865

View Profile Personal Message (Offline)

Ignore
1480883865
Reply with quote  #2

1480883865
Report to moderator
1480883865
Hero Member
*
Offline Offline

Posts: 1480883865

View Profile Personal Message (Offline)

Ignore
1480883865
Reply with quote  #2

1480883865
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
January 22, 2012, 12:00:25 AM
 #182

GAE has no problem with json, so no problem with encoding here. However GAE does not support TCP sockets and persistent connections, except "Channel API", which is highly proprietary stuff and there's no reason to include this into Stratum "standard". However HTTP transport is definitely the way how to implement Stratum on GAE. Are you using Python or Java on GAE?

I'm going to implement set of unit tests, so it should be pretty easy to test if alternative server implementation fits all protocol requirements. I'm still working on another project right now, but I'll be back on Stratum in few days.

ThePiachu
Sr. Member
****
Offline Offline

Activity: 442



View Profile WWW
January 22, 2012, 04:06:08 AM
 #183

I'm writing in Go, so if I get everything to work it should be quite fast. But Go apparently has no support for JSON-RPC over HTTP - http://code.google.com/p/go/issues/detail?id=2738 (so I had to create my own package for that), and the RPC package doesn't like GAE - https://groups.google.com/forum/#!msg/google-appengine-go/uQ0cv0m9j0E/H3VCrFgEWvcJ .

So far I managed to get a simple Python request to go through to my app - https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29#Python , and got the app to communicate a bit with bitcoind, but I'm having some problems communicating with a miner, so I have to look into it further.

1HWbVLhxj7bhewhyapMZpyhqWAeAhJd51E
My Bitcoin Calculator:
http://tpbitcalc.appspot.com/
slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
January 22, 2012, 04:36:59 AM
 #184

I'm writing in Go, so if I get everything to work it should be quite fast. But Go apparently has no support for JSON-RPC over HTTP - http://code.google.com/p/go/issues/detail?id=2738 (so I had to create my own package for that), and the RPC package doesn't like GAE - https://groups.google.com/forum/#!msg/google-appengine-go/uQ0cv0m9j0E/H3VCrFgEWvcJ .

Well, Stratum is not clear JSON-RPC, it's only based on it (format of the messages is the same, but with some additional things like session management and callback URLs). For your purposes the JSON library & possibility to process HTTP POST requests should be enough to implement Stratum in few lines of code.

Interesting, I didn't notice Go support on GAE. I had to say that I don't follow GAE progress in last year.

casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1344


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
January 27, 2012, 03:23:51 AM
 #185

I would like to propose one more thing that Stratum should keep track of, and perhaps one LESS thing.

The More thing:  Nodes should inform Stratum that they can exchange P2P information about subject X, where X is an arbitrary string that is chosen by the developer of whatever X is.  In turn, Stratum helps nodes aware of subject X find one another.

The Less thing:  The block chain should not be assumed as something that will be exchanged by default, but rather, should be one of many subjects that can be subscribed to.

Now, why?!  At some point, I believe that the block chain as we know it is going to get called the "Satoshi" block chain, or the "orthodox" block chain, and people are going to start new chains that work as subsets of the Satoshi block chain, so they can function as a whole bitcoin node without having to download the whole damn thing.

Someone who accepts a pruned block chain is ultimately putting their trust in whoever pruned it.  This is a tradeoff that becomes more likely to happen every day the block chain gets bigger.  Someone who accepts a pruned chain also cannot participate in exchanging historical blocks with orthodox clients - they can only exchange such blocks with others who trust in the same chain.  Hence Stratum being an ideal hookup mechanism for these clients - but without Stratum having to bear the burden of being the relay (except where the operator chooses to make his server relay such things).

An example is if I go and compile a digest of the block chain that includes only the unspent outputs as of block 150,000, I sign it, I publish it.  People want it because it's only 150 MB instead of a gig, and let's assume others are vouching for the integrity of it (in other words, consensus on the forums is that the outputs it contains are consistent with the orthodox block chain as of the same block height, and someone publishes a utility to confirm that).  From then onward, all newer blocks are interchangeable with the Satoshi block chain, it's just that orthodox Satoshi clients will not understand how to deal with a peer that has recent blocks but not the whole chain.  I would want Stratum to help me hook up with clients that are expecting a floor of 150,000 as well.

Let's say somebody else goes and creates a combined client that can trade Bitcoins, Namecoins, Litecoins, and whatever other kinds of coins, but all within the same client and protocol.  Ideally, Stratum should help that client find other like-minded clients so they can exchange all of those blocks, all without burdening Stratum with block types its server operators don't want to waste resources on.

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper wallets instead.
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742



View Profile WWW
January 27, 2012, 04:56:19 AM
 #186

I would like to propose one more thing that Stratum should keep track of, and perhaps one LESS thing.

The More thing:  Nodes should inform Stratum that they can exchange P2P information about subject X, where X is an arbitrary string that is chosen by the developer of whatever X is.  In turn, Stratum helps nodes aware of subject X find one another.

The Less thing:  The block chain should not be assumed as something that will be exchanged by default, but rather, should be one of many subjects that can be subscribed to.
This is a really great abstraction.  Building the stratum protocol to transmit arbitrary data is a great idea.

Quote
Someone who accepts a pruned block chain is ultimately putting their trust in whoever pruned it.  This is a tradeoff that becomes more likely to happen every day the block chain gets bigger.  Someone who accepts a pruned chain also cannot participate in exchanging historical blocks with orthodox clients - they can only exchange such blocks with others who trust in the same chain.  Hence Stratum being an ideal hookup mechanism for these clients - but without Stratum having to bear the burden of being the relay (except where the operator chooses to make his server relay such things).
I was under the assumption that the pruned blockchains would have verifiable hashes in them so that you wouldn't have to trust the distributor.  Stratum should definitely keep this in mind.

Quote
Let's say somebody else goes and creates a combined client that can trade Bitcoins, Namecoins, Litecoins, and whatever other kinds of coins, but all within the same client and protocol.  Ideally, Stratum should help that client find other like-minded clients so they can exchange all of those blocks, all without burdening Stratum with block types its server operators don't want to waste resources on.
Wow. This is a great idea.  I don't want to have a run a stratum server for bitcoin and a stratrum server for namecoin and so on.

casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1344


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
January 27, 2012, 05:24:03 AM
 #187

I was under the assumption that the pruned blockchains would have verifiable hashes in them so that you wouldn't have to trust the distributor.  Stratum should definitely keep this in mind.

Somehow, I believe downloading a "seed" blockchain will be more preferable in practice than getting pruned blockchains bit-by-bit from peers, for two reasons.

One, I am convinced (though smart others have disputed) that if you accept pruned blockchains bit-by-bit from other nodes, you can be disrupted (DoS'd) simply by getting fed a block that has the wrong things pruned from it (example, someone pruned an unspent transaction that should not have been pruned).  Sure, you'll be able to detect you don't have the complete picture, because the "verifiable hash" you refer to won't check out, but you'll have a difficult time finding which block(s) you were misfed, which you need to ask for again - recovery from which will be at least as burdensome - and definitely more complex - as compared to downloading a seed blockchain as a signed flat file (either via http, torrent, etc.).

Second, even when blocks are pruned, there's plenty of "cruft" that you really don't need to keep if you don't demand complete (orthodox) cryptographic verification of the block chain file.  When you prune nodes off a merkle tree, you still have to leave behind hashes of the branches you cut off to be able to validate the tree.  Those are wasted space - what I mean by this is they take up bandwidth and permanent storage but are useful only once - for validating the integrity of the file - and will never be accessed for any other reason.

Another thing that's a total waste of space is the inputs of all the kept transactions, which are much larger than the outputs.  The inputs take up way more space each (they contain a public key and a digital signature - over 130 bytes each) and outputs are much smaller (like around 30 bytes).  You don't need the inputs for validating new transactions, just the outputs.  But with the current merkle tree setup, you can only prune a whole transaction at a time - all inputs and all outputs must go together or not at all.

Also, another big space waster: transactions with "multiple outputs" (generated by sendmany), which must be kept in their entirety if a single output remains unspent, they cannot be pruned with the current merkle tree spec.  This would not be so bad except for the fact that some mining pools issue huge sendmany's with numerous tiny outputs worth only pennies each - to distribute payout to miners in the coinbase of their blocks.  These transactions under the orthodox specification can never be pruned until every penny output has been spent - which for most of these blocks may very well be never. (example transaction from P2Pool: http://blockexplorer.com/t/42BmXQaM9T)

Now, on the other hand, if every 10k blocks or so, somebody can compile all of the unspent outputs into a signed file, discarding all inputs and all spent outputs, that file is always going to be a tiny fraction of the size the orthodox block chain.  If they also released source code to a validation utility (which can be run only by those possessing the complete orthodox block chain), the integrity of that file can be confirmed (and also signed) by the more concerned members of the community, and the file can then be trusted as safe based on community consensus.

A useful function of Stratum would be to pass around those endorsing signatures so that a client can quickly determine which is the most recent pre-compiled "block chain digest" that enjoys the overwhelming majority of community support.

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper wallets instead.
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742



View Profile WWW
February 03, 2012, 05:29:26 AM
 #188

Hows this going? I re-arranged my server and have been to busy coding to play around with the protocol

bc
Member
**
Offline Offline

Activity: 72



View Profile
February 27, 2012, 03:19:41 AM
 #189

A useful function of Stratum would be to pass around those endorsing signatures so that a client can quickly determine which is the most recent pre-compiled "block chain digest" that enjoys the overwhelming majority of community support.

Should Stratum keep track/report on nodes who endorse digests later proven to be in some way false? Is there enough peer pressure to promote only honest endorsements?

"Democracy is the original 51% attack." - Erik Voorhees
genjix
Legendary
*
expert
Offline Offline

Activity: 1232


View Profile
March 05, 2012, 04:08:23 PM
 #190

Hows this going? I re-arranged my server and have been to busy coding to play around with the protocol

I think slush doesn't have much time. Anyway, I'll pick up the slack. I'm working off slush's original proposal, but need to make some design decisions in coordination with ThomasV.

I'll only focus on two use-cases:
- Merchants
- Users
slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
March 15, 2012, 11:36:59 AM
 #191

I think slush doesn't have much time. Anyway, I'll pick up the slack. I'm working off slush's original proposal, but need to make some design decisions in coordination with ThomasV.

Genjix, can be more specific where and why you needed to decide something? Afaik the protocol is fully covered. I read your proposal (https://bitcoinconsultancy.com/wiki/Intersango/Overnet) and I think there are many things to discussion. Namely - why you choose to not follow json-rpc specification? I'm really missing "id". Also variables "method" and "result" together don't sound clear.

Btw after a while, I'm back in the project. Expect some progress in git repository in following days.

ThomasV
Legendary
*
Offline Offline

Activity: 1722



View Profile WWW
March 15, 2012, 12:49:22 PM
 #192

I think slush doesn't have much time. Anyway, I'll pick up the slack. I'm working off slush's original proposal, but need to make some design decisions in coordination with ThomasV.

Genjix, can be more specific where and why you needed to decide something? Afaik the protocol is fully covered. I read your proposal (https://bitcoinconsultancy.com/wiki/Intersango/Overnet) and I think there are many things to discussion. Namely - why you choose to not follow json-rpc specification? I'm really missing "id". Also variables "method" and "result" together don't sound clear.

Btw after a while, I'm back in the project. Expect some progress in git repository in following days.

Please do not consider the current syntax as a conscious choice; we just needed to start with something concrete.
Genjix asked me to describe what information is needed by the client, and your document does not include that. This is why I wrote this specification.

Please feel free to suggest improvements concerning the syntax & protocol. For example, we can use an "id" field if you think it is better.
Again, I did not make the conscious choice not to use an "id" field; I was focusing on something else. I was focusing on the specification of the services




Electrum: the convenience of a web wallet, without the risks
slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
March 15, 2012, 01:48:30 PM
 #193

Thomas, I understand and I'm fine with that. Then it's better to specify services/methods/parameters and forget to wire format at all. I specified wire format, but I didn't specify blockchain methods, because I didn't have enough experience to do it months ago. I assume that you have much more knowledge to do that than me now, so we can work together.

Everything necessary is to define method names (e.g. 'blockchain.address.subscribe'), their parameters (e.g. 'address', 'status'), result types and exceptions (possible exceptions which can be thrown by given method, and exception parameters).

Btw this is pretty hard to do clearly and cover all possibilities (especially all those possible exceptions and corner cases). This is the reason why I prefer easy API without complicated internal logic. But with the list of necessary methods, parameters etc. it's pretty easy to transform it into wire format.

slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
March 15, 2012, 02:14:44 PM
 #194

Thomas, I composed very simple document to begin with the blockchain specification:
https://docs.google.com/spreadsheet/ccc?key=0Al5UmiyYRxBudHl2VFhJWG5rT09maU9kbzltR3hBdmc . Give me your google docs email, I'll enable edit mode for you.

I combined proposal from Stratum draft document and wikipage on bitcoinconsultancy from you and genjix (marked bold). I have some comments to parameters/responses defined on that wiki page, I'll wait to you and genjix on IRC, it will be much faster than discuss it on forum or wiki...

dogisland
Sr. Member
****
Offline Offline

Activity: 261



View Profile
April 03, 2012, 07:28:57 PM
 #195

Wow, I've seen this thread hanging round for months but this is the first time I've really got it.

This is a great piece of work, I'm going to re-write Strongcoin to sit on top of Stratum, this is excellent.

Traktion
Full Member
***
Offline Offline

Activity: 168


View Profile
June 04, 2012, 07:09:58 PM
 #196

What are the odds that this overlay network could be used to permit two mobile devices to transact directly, in the absence of Internet access for one or both parties?  For example, a direct ad-hoc wifi connection or an indirect connection via a 'piratebox' type device?  Asuuming, of course, that the sending device doesn't need Internet access to produce a transaction from it's own, known address funds.

+1

Great thread + software!

I would like to know if you could pass a signed transaction onto another party (say, from the payer to the payee) too. This would be very useful for areas with limited data connectivity, as it would only require the receiver to 'upload' the transaction.

You could also have situations where both parties had no Internet access and such transactions could be treated like cheques, which could be cashed when the payee is next online.
slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
June 11, 2012, 03:38:43 PM
 #197

I would like to know if you could pass a signed transaction onto another party (say, from the payer to the payee) too. This would be very useful for areas with limited data connectivity, as it would only require the receiver to 'upload' the transaction.

Yes, this is possible with signed transaction.

Quote
You could also have situations where both parties had no Internet access and such transactions could be treated like cheques, which could be cashed when the payee is next online.

But you need to be really careful here. The fact that transaction is signed doesn't mean that it is valid. If you cannot fully trust payind side, you shouldn't receive such transaction offline...

marcus_of_augustus
Legendary
*
Offline Offline

Activity: 2086



View Profile
June 11, 2012, 11:51:14 PM
 #198

Quote
But you need to be really careful here. The fact that transaction is signed doesn't mean that it is valid. If you cannot fully trust payind side, you shouldn't receive such transaction offline...

People who write rubber checks are not a new phenomena !

There could even be an off-line market in "Junk signedTx", where such things are traded at discount to face value in the off-chance delinquent accounts are eventually made good by people looking to resurrect their financial reputations.

markm
Legendary
*
Offline Offline

Activity: 1778



View Profile WWW
June 12, 2012, 12:25:44 AM
 #199

The signed transaction probably should be transfered in siged messages, signed by various personal or corporate identity and/or web of trust signatures, so there is not only a signature showing what bitcioin addresses committed to pay the amount but also which entity certified that they themselves issued that "signed bitcoin-transaction".

Maybe including pubkeys of actual entities the receiving addresses were believed to belong to, with signature from them certifying those receiving addresses are indeed theirs.

Then receiver of the junk "cheque" has legs to stand on come collection time or saleback-of-rep time.

-MarkM-

Browser-launched Crossfire client now online (select CrossCiv server for Galactic  Milieu)
Free website hosting with PHP, MySQL etc: http://hosting.knotwork.com/
apetersson
Hero Member
*****
Offline Offline

Activity: 666


mycelium.com


View Profile WWW
June 20, 2012, 05:32:11 PM
 #200

is there a method to query addresses in bulk?
like the proposed bloom filter commands in the main client?

and what is the story if a chain reorganisation occurs? how can a transaction become invalid?
Pages: « 1 2 3 4 5 6 7 8 9 [10] 11 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!