Bitcoin Forum
March 29, 2024, 07:18:37 AM *
News: Latest Bitcoin Core release: 26.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 3 4 5 6 7 8 [9] 10 11 »  All
  Print  
Author Topic: [Stratum] Overlay network protocol over Bitcoin  (Read 37817 times)
slush (OP)
Legendary
*
Offline Offline

Activity: 1386
Merit: 1097



View Profile WWW
January 12, 2012, 02:39:08 PM
 #161

I do not know. The initial idea was to propose a new name for the communication protocol, so that it can be distinguished from the software.

Thomas, I proposed the protocol and asked you for the opinion. You told me that you don't understand and/or don't see any point in it and you want reference implementation of the protocol. So I created Stratum server implementation. What's unclear in this?

Quote
Now it seems that slush has decided to rewrite client and server from scratch, and that "Stratum" is being used to designate his software, not the protocol.

It's still Stratum protocol + server reference implementation + I'm writing some "default" services into it. Protocol implementation is already here and you know that I decided to separate Stratum from Electrum project, because I see some potential for lightweight clients generally, not only for Electrum.

But of course I'm open to help you with integrating Stratum into Electrum client, if you want. We already had the discussion about Twisted on the IRC and I accept that you don't like Twisted in the client, so I'll provide you clean-python implementation of Stratum client.

1711696717
Hero Member
*
Offline Offline

Posts: 1711696717

View Profile Personal Message (Offline)

Ignore
1711696717
Reply with quote  #2

1711696717
Report to moderator
1711696717
Hero Member
*
Offline Offline

Posts: 1711696717

View Profile Personal Message (Offline)

Ignore
1711696717
Reply with quote  #2

1711696717
Report to moderator
1711696717
Hero Member
*
Offline Offline

Posts: 1711696717

View Profile Personal Message (Offline)

Ignore
1711696717
Reply with quote  #2

1711696717
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
slush (OP)
Legendary
*
Offline Offline

Activity: 1386
Merit: 1097



View Profile WWW
January 12, 2012, 02:42:59 PM
 #162

I used the rpc calls that you provided in your document where it was possible, but some calls are missing for session management.

What exactly are you missing? I see only three session-related calls in server.py (new_session, update_session, poll_session) which are useless in Stratum protocol.

Edit: To be more verbose:

new_session: You just don't send session cookie to the server and it will create new session for the client automatically. No RPC command needed.

update_session: Replaced with blockchain.address.subscribe()

poll_session: Perform just blank HTTP request to Stratum server, it will put pending messages to the response. No RPC command needed.

ThomasV
Legendary
*
Offline Offline

Activity: 1896
Merit: 1343



View Profile WWW
January 12, 2012, 03:04:01 PM
 #163

new_session: You just don't send session cookie to the server and it will create new session for the client automatically. No RPC command needed.

update_session: Replaced with blockchain.address.subscribe()

poll_session: Perform just blank HTTP request to Stratum server, it will put pending messages to the response. No RPC command needed.


Can you explain a bit more your decisions?
Performing blank HTTP requests is an interesting idea, but I would like to know why you decided that polling should be performed that way. is this in order to minimize traffic?
Also, how would that work with TCP?

Electrum: the convenience of a web wallet, without the risks
slush (OP)
Legendary
*
Offline Offline

Activity: 1386
Merit: 1097



View Profile WWW
January 12, 2012, 03:20:01 PM
 #164

Can you explain a bit more your decisions?

Of course. HTTP is just a transport carrying abstract Stratum payload. There is TCP transport already and there can be a lot more transports in the future. Stratum protocol is designed to be bi-directional, so server can send some payload to client (information about new block, new transaction) at any time. It's not the part of application layer itself to decide HOW transport will deliver such payload to the client. So server just tell "send <this> to the client" and don't care about it anymore.

HTTP Poll transport is just the alternative for the environment where real bi-directional communication (like TCP socket or websocket) is not possible. So every poll request just give a chance to send pending messages from the server to the client in the HTTP response, simulating bi-directional communication, just with some delay (=polling interval).

From the view of this idea, creating RPC command for polling is going against the whole conception, because RPC commands are handled by different layer than HTTP transport mechanism. And there's nothing like 'poll' RPC command for standard bi-directional transports anyway.

Quote
Also, how would that work with TCP?

In the TCP transport, which is bi-directional, polling is not needed at all. Server will send the payload when it's ready, in real time. It's the beauty of pub-sub mechanism.

ripper234
Legendary
*
Offline Offline

Activity: 1358
Merit: 1003


Ron Gross


View Profile WWW
January 13, 2012, 02:32:59 PM
 #165

slush,

I'm finally starting to work on this, and have some questions.
I think I'll fire them one by one on this thread, instead of accumulating them.

The first one is a meta question - how about using Shapado as a Q&A forum? It's "setup in 8 seconds", and better than a forum thread. It might be better than just firing these questions away at the Stack Exchange. The advantage of a dedicate forum is that you (and other Stratum devs) could follow that website and quickly respond to questions, and the answers would be findable later via Google.

Another question is about a sessions while polling:

From the Google Doc:

Quote
Cookie should contains all cookies provided by the server in previous calls, especially the STRATUM_SESSION cookie which is identifying current HTTP session. Blank or missing STRATUM_SESSION will initiate new session, even on the same TCP connection

The APIs that I need right now appear to be only address.get_history and transaction.broadcast (I'll start by implementing the simple & naive polling protocol). For these purposes, I'm not sure why I need to maintain a session. As a matter of fact ... can you give an example when a session is useful?

Please do not pm me, use ron@bitcoin.org.il instead
Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
slush (OP)
Legendary
*
Offline Offline

Activity: 1386
Merit: 1097



View Profile WWW
January 13, 2012, 03:11:18 PM
 #166

The first one is a meta question - how about using Shapado as a Q&A forum? It's "setup in 8 seconds", and better than a forum thread. It might be better than just firing these questions away at the Stack Exchange. The advantage of a dedicate forum is that you (and other Stratum devs) could follow that website and quickly respond to questions, and the answers would be findable later via Google.

Yes, some better tools should be better in the future, but at this moment the forum thread is doing well. There is only few people so the discussion is pretty clean now. Especially I'll try to move that google docs to some wiki which allow me to separate whole document into more parts and include some additional info (as an examples).

Quote
The APIs that I need right now appear to be only address.get_history and transaction.broadcast (I'll start by implementing the simple & naive polling protocol). For these purposes, I'm not sure why I need to maintain a session. As a matter of fact ... can you give an example when a session is useful?

What's your purpose of such calls? You probably don't want to download whole address history on every request (say 5 seconds), right? I'm not sure, because I don't know your application, but you probably want to download address history on the startup and then subscribe your client for changes on given address, to receive only new transactions. In this way you save server resources and bandwidth. It's also possible that server may block your requests when you start overflooding server with "get_history" requests every few seconds, because there's no reason to do that so frequently (rate limiting isn't implemented yet, but it is pretty good feature for the future).

There are mostly two main usecases:
a) The address you're interested in is already existing, so you need to ask for history + subscribe for new transactions.
b) The address is freshly generated by you, so you know that there's no history and you want ONLY to subsribe for new transactions. This is the case of some merchant tool, which created fresh address for receiving coins for the order...

case a)
first http call (one HTTP request can contain more RPC calls, divided by newline character):
blockchain.address.get_history('1SomeAddress') => list of transactions
blockchain.address.subscribe('1SomeAddress') => returns only True on a success, but server will send you data for incoming transactions in following HTTP responses.

following http calls: blank HTTP request (with the session) to the server

case b)
first HTTP call:
blockchain.address.subscribe('1SomeAddress') (no get_history() command is needed!)

following HTTP calls: blank HTTP request (with the session) to the server

I think there's big misunderstanding about the "polling" thing. Polling does not mean that the connection is stateless and you need to ask for everyting on every request. Polling mechanism is only the workaround for the situation when only the client can initiate the communication and is giving a chance to counterparty to send pending data back in the polling response (like information about incoming transaction)...

So yes, you need to "care" about the session even in your use case. But almost every http client library can do cookie management internally, when you perform more calls ("http polling requests") in the row..

ripper234
Legendary
*
Offline Offline

Activity: 1358
Merit: 1003


Ron Gross


View Profile WWW
January 13, 2012, 03:20:47 PM
 #167

Quote
The APIs that I need right now appear to be only address.get_history and transaction.broadcast (I'll start by implementing the simple & naive polling protocol). For these purposes, I'm not sure why I need to maintain a session. As a matter of fact ... can you give an example when a session is useful?

What's your purpose of such calls? You probably don't want to download whole address history on every request (say 5 seconds), right? I'm not sure, because I don't know your application, but you probably want to download address history on the startup and then subscribe your client for changes on given address, to receive only new transactions. In this way you save server resources and bandwidth. It's also possible that server may block your requests when you start overflooding server with "get_history" requests every few seconds, because there's no reason to do that so frequently (rate limiting isn't implemented yet, but it is pretty good feature for the future).

There are mostly two main usecases:
a) The address you're interested in is already existing, so you need to ask for history + subscribe for new transactions.
b) The address is freshly generated by you, so you know that there's no history and you want ONLY to subsribe for new transactions. This is the case of some merchant tool, which created fresh address for receiving coins for the order...

case a)
first http call (one HTTP request can contain more RPC calls, divided by newline character):
blockchain.address.get_history('1SomeAddress') => list of transactions
blockchain.address.subscribe('1SomeAddress') => returns only True on a success, but server will send you data for incoming transactions in following HTTP responses.

following http calls: blank HTTP request (with the session) to the server

case b)
first HTTP call:
blockchain.address.subscribe('1SomeAddress') (no get_history() command is needed!)

following HTTP calls: blank HTTP request (with the session) to the server

I think there's big misunderstanding about the "polling" thing. Polling does not mean that the connection is stateless and you need to ask for everyting on every request. Polling mechanism is only the workaround for the situation when only the client can initiate the communication and is giving a chance to counterparty to send pending data back in the polling response (like information about incoming transaction)...

So yes, you need to "care" about the session even in your use case. But almost every http client library can do cookie management internally, when you perform more calls ("http polling requests") in the row..


For context, this is the application I'm developing. I wanted to implement a polling client at first, that polls every minute or so, just as an initial proof of concept - I'd like to see the protocol works. I intend to move to a more efficient implementation soon after that.

Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

I really would like to get all transactions on these addresses. The addresses should in theory only have 1-2 transactions on each of them, but I'd still like to know all the transactions history.

This is not the final implementation, just a temporary POC.

Please do not pm me, use ron@bitcoin.org.il instead
Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
slush (OP)
Legendary
*
Offline Offline

Activity: 1386
Merit: 1097



View Profile WWW
January 13, 2012, 03:30:41 PM
 #168

Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...

ripper234
Legendary
*
Offline Offline

Activity: 1358
Merit: 1003


Ron Gross


View Profile WWW
January 13, 2012, 04:04:03 PM
 #169

Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...

Another question - I haven't seen any reference to testnet in this thread ... are there testnet servers? Does the protocol support it?

Please do not pm me, use ron@bitcoin.org.il instead
Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
ripper234
Legendary
*
Offline Offline

Activity: 1358
Merit: 1003


Ron Gross


View Profile WWW
January 13, 2012, 04:28:06 PM
 #170

Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...

Another question - I haven't seen any reference to testnet in this thread ... are there testnet servers? Does the protocol support it?

What's the expected time that requests are supposed to take?

I sent a request to http://chicago.stratum.bitcoin.cz:8000, with a timeout of 60 seconds ... and the timeout expired.

content-type: application/stratum
id: 1
method: blockchain.address.get_history
params: mw5h6BDh77GUPc4DCmzjoU7ry2R4exoaYr   (this is a testnet address, so I guess it wouldn't work ... but I should get an error, not a timeout)

Please do not pm me, use ron@bitcoin.org.il instead
Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
ripper234
Legendary
*
Offline Offline

Activity: 1358
Merit: 1003


Ron Gross


View Profile WWW
January 13, 2012, 04:43:23 PM
 #171

This also happens when I try a prodnet address:

POST / HTTP/1.1
Host: chicago.stratum.bitcoin.cz:8000
Content-Type: application/stratum
Connection: keep-alive
Accept: */*
User-Agent: NING/1.0
Content-Length: 84

id=1&method=blockchain.address.get_history&params=14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA

Please do not pm me, use ron@bitcoin.org.il instead
Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
ripper234
Legendary
*
Offline Offline

Activity: 1358
Merit: 1003


Ron Gross


View Profile WWW
January 13, 2012, 04:45:49 PM
 #172

Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...

Any documentation on what is and is not implemented right now?
ETAs?

Please do not pm me, use ron@bitcoin.org.il instead
Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
slush (OP)
Legendary
*
Offline Offline

Activity: 1386
Merit: 1097



View Profile WWW
January 14, 2012, 06:31:41 PM
 #173

This also happens when I try a prodnet address:

POST / HTTP/1.1
Host: chicago.stratum.bitcoin.cz:8000
Content-Type: application/stratum
Connection: keep-alive
Accept: */*
User-Agent: NING/1.0
Content-Length: 84

id=1&method=blockchain.address.get_history&params=14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA

Hi, your payload is wrong, you must use json-encoded data. So

{"id":1, "method":"blockchain.address.get_history", "params": ["14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA"]}

instead of url-encoded data.

slush (OP)
Legendary
*
Offline Offline

Activity: 1386
Merit: 1097



View Profile WWW
January 14, 2012, 06:43:26 PM
 #174

Any documentation on what is and is not implemented right now?

get_history() works now, however the result data may be slightly changed in the future. I'm writing API for standard services, then will be much cleaner what method expect what parameters and what will be the result object.

ripper234
Legendary
*
Offline Offline

Activity: 1358
Merit: 1003


Ron Gross


View Profile WWW
January 15, 2012, 08:47:49 AM
 #175

This also happens when I try a prodnet address:

POST / HTTP/1.1
Host: chicago.stratum.bitcoin.cz:8000
Content-Type: application/stratum
Connection: keep-alive
Accept: */*
User-Agent: NING/1.0
Content-Length: 84

id=1&method=blockchain.address.get_history&params=14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA

Hi, your payload is wrong, you must use json-encoded data. So

{"id":1, "method":"blockchain.address.get_history", "params": ["14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA"]}

instead of url-encoded data.

Hmm, I'll fix and retry ... possibly next weekend.

Please do not pm me, use ron@bitcoin.org.il instead
Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
rapeghost
Sr. Member
****
Offline Offline

Activity: 419
Merit: 250



View Profile
January 16, 2012, 04:17:18 PM
 #176

It's actually hosted by BitVPS.COM not .Net....

slush (OP)
Legendary
*
Offline Offline

Activity: 1386
Merit: 1097



View Profile WWW
January 16, 2012, 05:01:42 PM
 #177

It's actually hosted by BitVPS.COM not .Net....

Fixed, thanks Smiley

ripper234
Legendary
*
Offline Offline

Activity: 1358
Merit: 1003


Ron Gross


View Profile WWW
January 17, 2012, 05:46:54 AM
 #178

Check out my answer here on why The Stratum protocol is secure.

Please do not pm me, use ron@bitcoin.org.il instead
Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
slush (OP)
Legendary
*
Offline Offline

Activity: 1386
Merit: 1097



View Profile WWW
January 19, 2012, 10:53:16 AM
 #179

Check out my answer here on why The Stratum protocol is secure.

Very nice, thank you :-).

I didn't commited into git for few days, because I'm finishing one project in my job, but then I'll work more actively on the Stratum again. I'm missing it!

ripper234
Legendary
*
Offline Offline

Activity: 1358
Merit: 1003


Ron Gross


View Profile WWW
January 19, 2012, 01:10:21 PM
 #180

Check out my answer here on why The Stratum protocol is secure.

Very nice, thank you :-).

I didn't commited into git for few days, because I'm finishing one project in my job, but then I'll work more actively on the Stratum again. I'm missing it!

I'm missing it too Smiley
I wish I could devote more time to this, but I'm so swamped with everything nowadays.

Baby steps.

Please do not pm me, use ron@bitcoin.org.il instead
Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
Pages: « 1 2 3 4 5 6 7 8 [9] 10 11 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!