Bitcoin Forum
December 13, 2017, 04:01:08 AM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Stratum protocol documentation  (Read 69609 times)
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
April 05, 2014, 02:47:55 AM
 #1

I've tried to collate all the relevant stratum documentation in one place since it is scattered over the forum and the original documentation links are hard to find. I'll add whatever relevant extensions were developed in this thread. I will try to add them to this thread to avoid them being lost on the internet.

Original protocol announce thread:
https://bitcointalk.org/index.php?topic=108533.0

The original draft documentation by Slush is available here:
https://docs.google.com/document/d/17zHy1SUlhgtCMbypO8cHgpWH73V5iUQKk_0rWvMqSNs/edit?hl=en_US

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
1513137668
Hero Member
*
Offline Offline

Posts: 1513137668

View Profile Personal Message (Offline)

Ignore
1513137668
Reply with quote  #2

1513137668
Report to moderator
1513137668
Hero Member
*
Offline Offline

Posts: 1513137668

View Profile Personal Message (Offline)

Ignore
1513137668
Reply with quote  #2

1513137668
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1513137668
Hero Member
*
Offline Offline

Posts: 1513137668

View Profile Personal Message (Offline)

Ignore
1513137668
Reply with quote  #2

1513137668
Report to moderator
1513137668
Hero Member
*
Offline Offline

Posts: 1513137668

View Profile Personal Message (Offline)

Ignore
1513137668
Reply with quote  #2

1513137668
Report to moderator
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
April 05, 2014, 02:48:27 AM
 #2

Copy of the protocol documentation:
Code:
Network protocol specification
Stratum platform


draft


________________


Overview
        Stratum is basically a platform for creating lightweight Bitcoin clients (= clients without a blockchain, keeping only private keys). Absence of the blockchain on the client side provide very good user experience, client has very small footprint and still provide high level of security and privacy.
        More technically, Stratum is an overlay network on the top of Bitcoin P2P protocol, creating simplified facade for lightweight clients and hiding unnecessary complexity of decentralized protocol. However there’s much bigger potential in this overlay network than just providing simplified API for accessing blockchain stored on Stratum servers. For utilization of such potential, we definitely need some robust protocol providing enough flexibility for various type of clients and their purposes.


Some advanced ideas for Stratum network, which will need flexible network protocol:
* Integration of BTC/fiat exchanges into clients
* Wallet storages for diskless or extremely low-resource clients (AVR-based hardware wallets)
* Server-side escrows (sending bitcoins to email)
* Integration of bitcoin laundry
* Exchange calculators (for providing “fiat” equivalents of BTC in clients)
* Firstbits support
* Mining support for clients
* Various transport protocols (especially HTTP Push, which allows PHP websites to integrate with Bitcoin easily)


________________
Requirements
1. Protocol should be as simple as possible. Some clients aren’t capable to handle high-level message systems like Google’s Protocol buffers or Apache Thrift.
2. Protocol should be text-based. Some languages cannot handle binary data (javascript) or it’s pretty difficult to implement it correctly (PHP). It’s also easier to debug text-based protocol than binary data.
3. Protocol must support standard RPC mechanism (request-response). Request should contains method identifier and parameters, response should be able to transfer error states/exceptions.
4. Mapping between request and response must be clear. Don’t allow vague relation between request and response. Some message-based systems use only textual conventions to map requests and responses together, like ‘firstbits_resolve <firstbits>’ expects ‘firstbits_response <firstbits> <address>’ or ‘firstbits_error <firstbits> <reason>’. It creates ambiguous data flow, avoid it.
5. Protocol should support publish-subscribe mechanism. Client can subscribe on server for receiving some kind of information. After this request, server will proactively broadcast messages to subscribed clients until client disconnects or cancel it’s subscription.
6. Protocol must be bi-directional. In the opposite of standard client-server model, we sometimes need to allow server to initiate the communication. As an example, server can ask client to reconnect to another node (before switching to maintenance mode) or send some text message to user.


________________


Network layers
        Stratum network layer should be flexible on many levels. This document is proposing complete stack for Stratum networking, however such complex problem can be divided into three independent layers:


1. Transports
2. Application protocol
3. Services
Transports
        Transports are the way how to obtain bi-directional communication between a lightweight client and Stratum servers. All Stratum servers should support some basic range of supported transports like plain socket transport, HTTP Poll (for environments with restrictive network access), HTTP Push (for non-interactive clients like PHP scripts), websocket/socket.io (for javascript clients) etc.
        Transport is responsible for keeping the session, it’s not a responsibility of any upper layer. It means that once any kind of transport is established, upper layers of protocol can send/receive messages like it’s transferred over already established TCP socket.
 
* Socket transport
   * Standard TCP socket, clients initiate connection, then both sides can initiate transport of payload (application protocol) anytime.
* Websocket
   * It’s HTTP-based, socket-like transport, widely supported by javascript clients. It’s probably the best way how to provide instant notification to browser-based applications.
   * After initial handshake, it works like “Socket transport”.
* HTTP Poll
   * Client performs HTTP POST every N seconds, storing payload as request body, processing server payload from response body.
   * HTTP POST without a session identifier creates new session (cookie?), client is responsible for using this cookie in following requests during the session.
   * Server store all messages for delivery to client into server-side buffer, flush them to client in following poll request.
* HTTP Push
   * Transport for clients which cannot keep open connection and they’re unable to ask server frequently enough. Typical example is PHP-based website, which want to subscribe for some blockchain changes (notification about incoming transaction for given address).
   * On the beginning of the HTTP Poll session, client provide callback URL, turning the current HTTP Poll transport into HTTP Push.
   * When there’re some new data on Stratum server, server performs HTTP POST request to callback URL with payload in the request body.
   * Client must perform keep-alive HTTP Poll request for indicating it’s still alive and want to receive HTTP Push messages. It can be done by crontab entry with one hour period, for example.


HTTP Poll
Important HTTP headers in the request:
* Cookie should contains all cookies provided by the server in previous calls, especially the STRATUM_SESSION cookie which is identifying current HTTP session. Blank or missing STRATUM_SESSION will initiate new session, even on the same TCP connection. Using many different cookies in different HTTP requests is possible, which allow serving more independent clients over one TCP connection.
* Content-Type must be always “application/stratum” indicating that payload is following Stratum protocol. This is especially useful for filtering requests in the server configuration (Apache/Nginx), so ports 80 and 443 can still be used for non-Stratum traffic.
* X-Callback-Url Provides the URL (always in the full form, including desired HTTP/HTTPS protocol type) for HTTP Push transport. Occurence of this header in the request turns the session into the HTTP Push protocol.


Important HTTP headers in the response:
* Set-Cookie contains cookies identifying current HTTP session and client must send back all those cookies in following request. By sending the STRATUM_SESSION cookie, server is indicating that the request/response is part of newly created session. When the client receive this cookie in the middle of the communication, he must re-initialize the session (subscribe for all services, …). This may happen when client perform request with the session which is already expired.
* Content-Type is always “application/stratum”, indicating that response was created by Stratum application server.
* Content-MD5 contains the MD5 hash of the body, for checking that content wasn’t corrupted during the transmission.
* Server indicates the version of server implementation.
* X-Session-Timeout contains lifetime of current session (in seconds). After this time, every consequential request with the same session ID is considered as new session and client must perform initial handshake (subscribing for all services, …)
* X-Content-SHA256 has the same purpose as Content-MD5, but provides checksum for clients which don’t handle MD5. SHA256 must be presented in the clients anyway for creating Bitcoin transactions, so the possibility of checking transport checksum with SHA256 is minimizing dependencies.


Important HTTP status codes in the response:
* 200 OK - server processed the request succesfully
* 5xx Server error - client should perform consequential request with some delay to prevent server overloading.


HTTP Push
        This transport works on the top of HTTP Poll transport and it is fully backward compatible. It is enabled by sending X-Callback-Url header in HTTP Poll request, indicating that server may actively send payload to the client using given URL as HTTP POST request.
Client can continue to work with the transport like it’s still the HTTP Poll transport, but broadcasts from the server are delivered instantly over callback URL, without waiting on consequential poll request.
Callback URL can be changed during the session, by providing updated URL in the header. HTTP Push can be also switched back to HTTP Poll, by providing X-Callback-Url with blank string value.
Client must perform occasional HTTP polling even with HTTP Push transport, to indicate he’s alive and interested in receiving callbacks. Don’t forget that those keep alive calls are still valid HTTP Poll requests and server can provide real data in the response.
Keep alive poll requests must be called before session on the server timed out (timeout is provided in X-Session-Timeout response header). It’s recommended to perform keep alive request few seconds before session will actually timed out to protect session before expiration thanks to temporary network failure. Don’t forget that occurence of STRATUM_SESSION cookie in the keep alive response indicates expiration of previous session on the server and client must perform re-initialization all previous subscriptions.


Important HTTP headers in callback:
* Content-Type is always “application/stratum”, indicating that request was created by Stratum application server.
* X-Session-Id contains session ID of HTTP Poll connection. This is useful for handling multiple connections over the same callback URL.
* X-Session-Timeout contains remaining time to session expiration in seconds.
Application protocol
        Protocol itself is the way how to exchange information between client and server, using already established transport. Protocol doesn’t understand the meaning of exchanged information, it only keeps the track on request-response and internal data format. Protocol format is the same for all supported transports.
        Basic structure of proposed protocol is line-based, json-encoded message. Every message is on separate line and there exists only two formats of messages - request and response. Those messages use the format of JSON-RPC 2.0 protocol (http://json-rpc.org/wiki/specification).
        JSON-RPC allows two types of requests. One is part of standard RPC mechanism when every request expects some response. Second type is the notification formatted as a JSON-RPC request, but it doesn’t expect any kind of response. Typical example is broadcasting of new block or transaction from server to clients. The difference between RPC request and notification is that notification always has id=null.
Request
Every RPC request contains three parts:
   * message ID - integer or string
   * remote method - unicode string
   * parameters - list of parameters


message ID must be an unique identifier of request during current transport session. It may be integer or some unique string, like UUID. ID must be unique only from one side (it means, both server and clients can initiate request with id “1”). Client or server can choose string/UUID identifier for example in the case when standard “atomic” counter isn’t available (multi-processing environment like PHP servers).


Examples:


* Retrieve transaction history of given Bitcoin address (client->server):
{“id”: 1, “method”: “blockchain.address.get_history”, “params”: [“1DiiVSnksihdpdP1Pex7jghMAZffZiBY9q”]}


* Broadcast transaction to Bitcoin network (client->server):
{“id”: 2, “method”: “blockchain.transaction.broadcast”, “params”: [“tx_payload”]}


* Subscribe for receiving information about new blocks in blockchain (client->server):
{“id”: 3, “method”: “blockchain.block.subscribe”, “params”: []}


* Unsubscribe from receiving information about new blocks in blockchain (client->server):
{“id”: 4, “method”: “blockchain.block.unsubscribe”, “params”: []}


Notification
Notification is like Request, but it does not expect any response and message ID is always null:
   * message ID - null
   * remote method - unicode string
   * parameters - list of parameters




Examples:
* Broadcast message about new block (server->client). Server don’t expect any response to this message. Client must perform call “blockchain.block.subscribe” to start receiving requests like this:
{“id”: null, “method”: “blockchain.block.new”, “params”: [“block_payload”]}


Response
        Every response contains following parts
   * message ID - same ID as in request, for pairing request-response together
   * result - any json-encoded result object (number, string, list, array, …)
   * error - null or list (error code, error message)


Examples:
* Received transaction history of an address (server->client):
{“id”: 1, result: [“tx1_id”: {}, “tx2_id”: {}], error: null}


* Received resolved firstbits address (server->client):
{“id”: 2, result: [“1DiiVSnksihdpdP1Pex7jghMAZffZiBY9q”], “error”: null}


* Resolving of firstbits address failed (server->client):
{“id”: 2, result: null, “error”: (1, “Firstbits cannot be translated to valid Bitcoin address”)}


Message signatures
#TODO
sign, sign_type, sign_id, sign_time
Replay attacks
#FIXME: Timestamp IS a part of signed data, it’s up to client to decide if cached message is good enough.
To protect messages against replay attacks, RPC commands vulnerable to such attack should contain some unique parameter. Because every signature is constructed from both request and response data, current timestamp in the request should provide enough security.
        Timestamp is a part of signature metadata. Although some types of RPC calls are immutable by design and adding timestamp to signature doesn’t provide additional security against replay attack, it’s up to client to decide if cached response (with timestamp in the history) is good enough for him. Caching of signatures is a reasonable way to improve server load and using signature with older timestamp shouldn’t be an issue for immutable calls.
Exceptions
Every RPC error returns 3-tuple with error code, human-readable error message and detailed traceback of exception (may be just blank string on production servers for security reasons). Error codes < 0 are protocol specific or unhandled exceptions. Error codes > 0 are exceptions generated by services and every service should offer detailed description of possible error states.


-1, Unknown exception, error message should contain more specific description
-2, “Service not found”
-3, “Method not found”
-10, “Fee required”
-20, “Signature required”, when server expects request to be signed
-21, “Signature unavailable”, when server rejects to sign response
-22, “Unknown signature type”, when server doesn’t understand any signature type from “sign_type”
-23, “Bad signature”, signature doesn’t match source data


#TODO
Services
#TODO Documentation for blockchain-related services (for Electrum)
Server side:


discovery.list_services()
discovery.list_vendors(service_name)
discovery.list_methods(service_name, vendor)
discovery.list_params(service_name, vendor, method)


node.peers.subscribe()
node.stop()
node.get_signature_pubkey()


# TODO:
blockchain.block.subscribe
blockchain.block.unsubscribe
blockchain.block.broadcast


blockchain.address.subscribe
blockchain.address.unsubscribe
blockchain.address.get_history
blockchain.address.get_balance


blockchain.transaction.broadcast
blockchain.transaction.get


blockchain.transactions.guess_fee
blockchain.transactions.subscribe
blockchain.transactions.unsubscribe


firstbits.resolve
firstbits.lookup
Client side:


node.get_version
node.reconnect(hostname)


Notifications:
blockchain.block.notify()
blockchain.transaction.notify()

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
April 05, 2014, 02:48:58 AM
 #3

X stratum http
Is the X-Stratum HTTP header documented somewhere?

I don't think so. It should be "getwork extension", but I'm not aware of any process how to "standardize it". Maybe just edit wiki? :-)

Quote
What should be the contents?

stratum+tcp://url:port

Quote
How should it be interpreted by miners? As "stratum is there, if you want it" or "stratum is there, switch to that"

Existing miners (cgminer, poclbm) implement it as "Stratum is there, switch to that". Thanks to this, users (who don't understand difference between getwork/stratum) don't need to change anything in their config, just update miner.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
April 05, 2014, 02:49:40 AM
 #4

Suggest difficulty
Basically the idea is that miner remember the last difficulty given by the previous mining session and it sends mining.suggest_difficulty(difficulty) on the beginning of the next session (it may be sent before mining.subscribe or mining.resume, but it should not be a requirement). The response to the call is irrelevant for the miner, but if the pool accept the suggestion in any way, it will send standard mining.set_difficulty with new computed difficulty.

Obviously the difficulty can differ from pool to pool while switching between backup pools, because every pool can have different SPM target . The main goal is to give at least some approximation for the newly connected pool, so it won't start with difficulty 1 for 10 THash/s monster rig...

After extensive discussion on IRC it was determined that having a variable keyname with the difficulty in it was not sensible use of JSON and better execution of this command would place the difficulty in the more logical position of within the parameters.

eg:
Code:
{"id": 2, "method": "mining.suggest_difficulty", "params": [42]}

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
April 05, 2014, 02:50:14 AM
 #5

Reconnect support
Ok for those not watching the discussion on #stratum on IRC, I believe we have a final protocol for mining resume which should not break any clients or pools.

It was decided that the parameters could include both the user agent and the session ID as the first and second parameters. If this fails cgminer/other clients should resort to sending blank set of parameters as previously.

The session Id, it was decided, would be encoded along with the notification message "mining.notify" as a member of the 3 deep array parameters returned for "result" by the pool.

See the following for an example of an initial connect followed by a reconnect.
Code:
[2013-02-25 10:30:58] SEND: {"id": 0, "method": "mining.subscribe", "params": ["cgminer/2.10.5"]}
 [2013-02-25 10:30:58] RECVD: {"result": [[["mining.notify", "02000000b507a8fd1ea2b7d9cdec867086f6935228aba1540154f83930377ea5a2e37108"], ["mining.set_difficulty", "02000000b507a8fd1ea2b7d9cdec867086f6935228aba1540154f83930377ea5a2e371082"]], "02000000", 4], "id": 0, "error": null}
 [2013-02-25 10:30:58] Pool 0 stratum session id: 02000000b507a8fd1ea2b7d9cdec867086f6935228aba1540154f83930377ea5a2e37108
 [2013-02-25 10:30:58] Pool 0 confirmed mining.subscribe with extranonce1 02000000 extran2size 4

 [2013-02-25 10:33:00] SEND: {"id": 2, "method": "mining.subscribe", "params": ["cgminer/2.10.5", "02000000b507a8fd1ea2b7d9cdec867086f6935228aba1540154f83930377ea5a2e37108"]}
 [2013-02-25 10:33:00] RECVD: {"result": [[["mining.notify", "02000000c33cfaa37a964c2ba76c78b99dc170a1b1fe7a5fe025f72e89afba7fc6f23d0e"], ["mining.set_difficulty", "02000000c33cfaa37a964c2ba76c78b99dc170a1b1fe7a5fe025f72e89afba7fc6f23d0e2"]], "02000000", 4], "id": 2, "error": null}
 [2013-02-25 10:33:00] Pool 0 stratum session id: 02000000c33cfaa37a964c2ba76c78b99dc170a1b1fe7a5fe025f72e89afba7fc6f23d0e
 [2013-02-25 10:33:00] Pool 0 confirmed mining.subscribe with extranonce1 02000000 extran2size 4
Note the session ID has changed between the initial subscription and the resume, BUT the extranonce1 has remained the same. This tells the miner it can (re)submit shares before the disconnection.

Hopefully we will not need to revise this any further.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
April 05, 2014, 02:50:37 AM
 #6

Managing difficulty changes
I have read the discussion about separating target/difficulty and work data. I have to say I think this was a mistake as it opens possibilities for abuse. I think something about these issues should go in the docs. Otherwise someone is going to create a naive implementation with some nasty issues.

Well, after discussion with Kano I slightly changed my mind. As I described few posts ago, with slightly modified meaning of "set difficulty" message, everything should be fixed.

Instead of meaning "stop sending lower difficulty shares than X immediately", set_difficulty should mean "all next jobs are using difficulty X". This still keeps the concept of separated difficulty message from the job itself, it doesn't require sending both messages together, fixes the problem of roundtrips (sending some bad-diff shares by the client because of latency) etc. It also gives the possibility to choose if pool wants to change difficulty in aggressive or in lazy way; by just sending "set difficulty" message without new job with clean_jobs=True, clients will slowly adopt higher difficulty as miners will start using new jobs, but pool can enforce higher difficulty with immediate effect by sending clean_jobs=True, as it is now.

This change in protocol is also backward compatible, because pools using current meaning of set_difficulty don't need to change anything, they just *may* stop sending clean_jobs job, if they want. Because of this, it is win-win solution for me. I described this already, but the idea didn't get any significant attention yet.

Quote
Is it ok to pipeline commands to the client? That is, send the difficulty change and new work (2 RPC calls) in quick succession without waiting for a reply to the first.

Yes, commands (notifications) can be pipelined. set_difficulty() and notify() don't expect any response from the client. Just sending one packet with both commands serialized is possible.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
April 05, 2014, 02:52:38 AM
 #7

Documentation from Slush's site:
http://mining.bitcoin.cz/stratum-mining

Code:
For mining software developers

Stratum protocol is based on JSON-RPC 2.0. In this chapter I expect that you're familiar with this protocol and you understand terms like "request", "response" and "notification". Please read JSON-RPC specification for more details.

For high level image of the Stratum protocol concept, please read Stratum protocol specification on Google docs. This document needs some care, but give you the basic examples how to connect to Stratum server.
Exception handling

Stratum defines simple exception handling. Example of rejected share looks like:

{"id": 10, "result": null, "error": (21, "Job not found", null)}

Where error field is defined as (error_code, human_readable_message, traceback). Traceback may contain additional information for debugging errors.

Proposed error codes for mining service are:

    20 - Other/Unknown
    21 - Job not found (=stale)
    22 - Duplicate share
    23 - Low difficulty share
    24 - Unauthorized worker
    25 - Not subscribed

Real-world example

This chapter contains real log of miner-pool communication which solved testnet3 block 000000002076870fe65a2b6eeed84fa892c0db924f1482243a6247d931dcab32
Miner connects the server

On the beginning of the session, client subscribes current connection for receiving mining jobs:

{"id": 1, "method": "mining.subscribe", "params": []}\n
{"id": 1, "result": [[["mining.set_difficulty", "b4b6693b72a50c7116db18d6497cac52"], ["mining.notify", "ae6812eb4cd7735a302a8a9dd95cf71f"]], "08000002", 4], "error": null}\n

Reminder: The newline character \n is a part of the message and must be added to the end of *every* JSON message. Server may wait to this magic character to start processing the message. This is the most common mistake which people implementing line-based clients do!

The result contains three items:

    Subscriptions details - 2-tuple with name of subscribed notification and subscription ID. Teoretically it may be used for unsubscribing, but obviously miners won't use it.
    Extranonce1 - Hex-encoded, per-connection unique string which will be used for coinbase serialization later. Keep it safe!
    Extranonce2_size - Represents expected length of extranonce2 which will be generated by the miner.

Authorize workers

Now let authorize some workers. You can authorize as many workers as you wish and at any time during the session. In this way, you can handle big basement of independent mining rigs just by one Stratum connection.

{"params": ["slush.miner1", "password"], "id": 2, "method": "mining.authorize"}\n
{"error": null, "id": 2, "result": true}\n

Server start sending notifications with mining jobs

Server sends one job *almost* instantly after the subscription.

Small engineering note: There's a good reason why first job is not included directly in subscription response - miner will need to handle one response type in two different way; firstly as a subscription response and then as a standalone notification. Hook job processing just to JSON-RPC notification sounds a bit better to me.

{"params": ["bf", "4d16b6f85af6e2198f44ae2a6de67f78487ae5611b77c6c0440b921e00000000",
"01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff20020862062f503253482f04b8864e5008",
"072f736c7573682f000000000100f2052a010000001976a914d23fcdf86f7e756a64a7a9688ef9903327048ed988ac00000000", [],
"00000002", "1c2ac4af", "504e86b9", false], "id": null, "method": "mining.notify"}

Now we finally have some interesting stuff here! I'll descibe every field of the notification in the particular order:

    job_id - ID of the job. Use this ID while submitting share generated from this job.
    prevhash - Hash of previous block.
    coinb1 - Initial part of coinbase transaction.
    coinb2 - Final part of coinbase transaction.
    merkle_branch - List of hashes, will be used for calculation of merkle root. This is not a list of all transactions, it only contains prepared hashes of steps of merkle tree algorithm. Please read some materials for understanding how merkle trees calculation works. Unfortunately this example don't have any step hashes included, my bad!
    version - Bitcoin block version.
    nbits - Encoded current network difficulty
    ntime - Current ntime/
    clean_jobs - When true, server indicates that submitting shares from previous jobs don't have a sense and such shares will be rejected. When this flag is set, miner should also drop all previous jobs, so job_ids can be eventually rotated.

How to build coinbase transaction

Now miner received all data required to serialize coinbase transaction: Coinb1, Extranonce1, Extranonce2_size and Coinb2. Firstly we need to generate Extranonce2 (must be unique for given job_id!). Extranonce2_size tell us expected length of binary structure. Just be absolutely sure that your extranonce2 generator always produces extranonce2 with correct length! For example my pool implementation sets extranonce2_size=4, which mean this is valid Extranonce2 (in hex): 00000000.

To produce coinbase, we just concatenate Coinb1 + Extranonce1 + Extranonce2 + Coinb2 together. That's all!

For following calculations we have to produce double-sha256 hash of given coinbase. In following snippets I'm using Python, but I'm sure you'll understand the concept even if you're a rubyist! It is as simple as:

import hashlib
import binascii
coinbase_hash_bin = hashlib.sha256(hashlib.sha256(binascii.unhexlify(coinbase)).digest()).digest()

How to build merkle root

Following Python snippet will generate merkle root for you. Use merkle_branch from broadcast and coinbase_hash_bin from previous snippet as an input:

import binascii

def build_merkle_root(self, merkle_branch, coinbase_hash_bin):
    merkle_root = coinbase_hash_bin
    for h in self.merkle_branch:
        merkle_root = doublesha(merkle_root + binascii.unhexlify(h))
    return binascii.hexlify(merkle_root)

How to build block header?

Now we're almost done! We have to put all together to produce block header for hashing:

version + prevhash + merkle_root + ntime + nbits + '00000000' +
'000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000'

First zeroes are blank nonce, the rest is padding to uint512 and it is always the same.

Note that merkle_root must be in reversed byte order. If you're a miner developer, you already have util methods there for doing it. For some example in Python see Stratum mining proxy source codes.
Server can occasionally ask miner to change share difficulty

Default share difficulty is 1 (big-endian target for difficulty 1 is 0x00000000ffff0000000000000000000000000000000000000000000000000000), but server can ask you anytime during the session to change it:
{ "id": null, "method": "mining.set_difficulty", "params": [2]}

This means that difficulty 2 will be applied to every next job received from the server.
How to submit share?

When miner find the job which meets requested difficulty, it can submit share to the server:

{"params": ["slush.miner1", "bf", "00000001", "504e86ed", "b2957c02"], "id": 4, "method": "mining.submit"}
{"error": null, "id": 4, "result": true}

Values in particular order: worker_name (previously authorized!), job_id, extranonce2, ntime, nonce.


Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
April 05, 2014, 02:53:23 AM
 #8

Alternative documentation from btcguild:

https://www.btcguild.com/new_protocol.php
Code:
Summary

The Stratum protocol reduces client-server communications to levels that are usable with very low bandwidth, and reduces the strain on servers drastically.
The key concept behind Stratum based mining is "push" based work, where the server pushes work to the miner, and the miner is able to utilize that work until the next push, regardless of the hash rate.
This is done by allowing the miner to increase a counter in the coinbase transaction, and build a new merkleroot for the block header, which effectively means the miner generates new work continuously without contacting the server.
Additionally, this new protocol utilizes a single asynchronous socket connection for all communication, rather than opening/closing HTTP connections with each communication.



Building Local Work

The mining pool provides a miner with two pieces of information upon connecting. The first is ExtraNonce1. The second piece of information is ExtraNonce2_size. This is how many bytes should be used for the ExtraNonce2 counter. ExtraNonce2 is a hexadecimal counter, and should be padded to fill up the number of bytes identified as the ExtraNonce2_size.

When the pool pushes work, it provides the coinbase transaction in two pieces. When the miner is building the block header to hash, it builds a merkleroot by creating a coinbase transaction. This is done by combining: Coinbase1 + ExtraNonce1 + ExtraNonce2 (padded) + Coinbase2.

This transaction is then passed through a double SHA256 to get the coinbase tx hash. You then combine that with first merkle branch provided by the work (if any), and double SHA256 the combined string. You then take that result, and do the same with the next merkle branch, repeating this process until all merkle branch hashes have been combined with previous results.

This process creates a unique merkle root for the new block header, which can then be pushed through the miner using the standard 32-bit nonce counter.



Initial Connection Communication

Client: {"id": 1, "method": "mining.subscribe", "params": []}\n
Server:  {"id": 1, "result": [["mining.notify", "ae6812eb4cd7735a302a8a9dd95cf71f"], "08000002", 4], "error": null}\n

Connection uses 'mining.subscribe' to subscribe to work from the server.
Server result response contains:
result[0] = "mining.notify", "ae6812eb4cd7735a302a8a9dd95cf71f" - Unique string used for the subscription
result[1] = ExtraNonce1, used for building the coinbase.
result[2] = Extranonce2_size, the number of bytes that the miner users for its ExtraNonce2 counter



Server Work Communication

Server: {"params": ["bf", "4d16b6f85af6e2198f44ae2a6de67f78487ae5611b77c6c0440b921e00000000",
"01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff20020862062f503253482f04b8864e5008",
"072f736c7573682f000000000100f2052a010000001976a914d23fcdf86f7e756a64a7a9688ef9903327048ed988ac00000000",
["c5bd77249e27c2d3a3602dd35c3364a7983900b64a34644d03b930bfdb19c0e5","049b4e78e2d0b24f7c6a2856aa7b41811ed961ee52ae75527df9e80043fd2f12"],
"00000002", "1c2ac4af", "504e86b9", false], "id": null, "method": "mining.notify"}\n

This is sent at regular intervals by the server. It should be sent almost immediately after an acknowledge subscription.
Work pushes contain:
params[0] = Job ID. This is included when miners submit a results so work can be matched with proper transactions.
params[1] = Hash of previous block. Used to build the header.
params[2] = Coinbase (part 1). The miner inserts ExtraNonce1 and ExtraNonce2 after this section of the coinbase.
params[3] = Coinbase (part 2). The miner appends this after the first part of the coinbase and the two ExtraNonce values.
params[4][] = List of merkle branches. The coinbase transaction is hashed against the merkle branches to build the final merkle root.
params[5] = Bitcoin block version, used in the block header.
params[6] = nBit, the encoded network difficulty. Used in the block header.
params[7] = nTime, the current time. nTime rolling should be supported, but should not increase faster than actual time.
params[8] = Clean Jobs. If true, miners should abort their current work and immediately use the new job. If false, they can still use the current job, but should move to the new one after exhausting the current nonce range.



Worker Authorization

Client: {"params": ["eleuthria_miner1", "password"], "id": 2, "method": "mining.authorize"}\n
Server: {"error": null, "id": 2, "result": true}\n

Client authorizes a worker using the method "mining.authorize". Client request contains:
params[0] = Worker Name
params[1] = Worker Password (can be "" or omitted if server does not require passwords)

Server response is "result": true for successful authorization, "result": false for unsuccessful authorization.



Work Submissions

Client: {"params": ["eleuthria_miner1", "bf", "00000001", "504e86ed", "b2957c02"], "id": 4, "method": "mining.submit"}\n
Server: {"error": null, "id": 4, "result": true}\n

Miners submit shares using the method "mining.submit".
Client submissions contain:
params[0] = Worker Name
params[1] = Job ID
params[2] = ExtraNonce 2
params[3] = nTime
params[4] = nonce
Server response is result: true for accepted, false for rejected.



Difficulty Adjustment

Server: { "id": null, "method": "mining.set_difficulty", "params": [2]}\n

The server can adjust the difficulty required for miner shares with the "mining.set_difficulty" method.
The miner should begin enforcing the new difficulty on the next job received. Some pools may force a new job out when set_difficulty is sent, using clean_jobs to force the miner to begin using the new difficulty immediately.



Cheat Sheet

Methods:
mining.subscribe     : Used to subscribe to work from a server, required before all other communication.
mining.authorize     : Used to authorize a worker, required before any shares can be submitted.
mining.notify        : Used to push new work to the miner.  Previous work should be aborted if Clean Jobs = true!
mining.submit        : Used to submit shares
mining.set_difficulty: Used to signal the miner to stop submitting shares under the new difficulty.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
April 05, 2014, 02:53:35 AM
 #9

Get transactions
Today I implemented and released to my pool new method called mining.get_transactions(job_id). This call simply dumps transactions used for given job. Thanks to this, miners now have everything needed to reconstruct source block template used by the pool and they can check online if pool isn't doing something nasty.

With this extension, Stratum is still the most optimized mining protocol, but also as open as GBT from the view that miners don't need to trust pool operator that he's not preparing some malicious attack with their hashpower. Anybody is welcome to write some tool for inspecting GBT/Stratum jobs; since now, there's no practical difference in "mining safety" between those two solutions.

Benefits of get_transactions call instead of sending transactions in job broadcast directly are:
* Lower stale ratio for everybody, because pool firstly broadcast just the job itself and *then* send full transactions to interested clients. Miner don't need to wait to full transaction list to start mining on new template.
* Lower bandwidth usage, because only users interested in block inspection will download the list of transactions
* Possibly lower bandwidth usage even for users doing template inspection, because they don't need to perform inspection on every job. Random checks will work as well, because pool never know who and when will ask for transaction list, so cheating is very unlikely.

Please note that sending of transaction list may become a paid feature on some pools, because it requires large amount of pool bandwidth, which is not required for mining itself. Tiny  fee for such additional feature may become an easy prevention against pool DoS.

Note discussion following this quote on the original thread regarding abusing this as a pool DoS and why no pools implement it. If they do, consider rate limiting it.

As per the discussion above with suggest_difficulty, a more logical evolution of this command has evolved placing the job_id in the params instead of the keyname.

eg:
Code:
{"id": 0, "method": "mining.get_transactions", "params":["545198de00000000"]}

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
May 28, 2014, 10:49:54 AM
 #10

"client.reconnect" discussion

Quote
server restart notification [with timer] so a pool can attempt a graceful restart rather than suddenly dropping connections.

What's the use case for this? If I want to restart the server I can do it without such command as it doesn't require any action from the miner side.

Maybe I can add "wait" option to existing client.reconnect() instead?


The thinking is for when a pool is doing some kind of maintenance (either to update their code/settings, or reboot the machine).  It's "good manners" for the pool to notify the miner in ADVANCE, that way they can move to a fallback pool/server before suddenly losing connection.  A "retry/wait" time should also be included in the restart method, so the miner knows how long it should wait before trying to reconnect.

Example:
Code:
{"id": 0, "method": "client.reconnect", "params":["stratum-lb-usa48.btcguild.com",3333,0]}
The parameters are [URL, port, wait time] as string, integer, integer

If client.reconnect is sent without parameters, the miner is to assume it's to reconnect to the same port and URL.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2366


Ruu \o/


View Profile WWW
May 28, 2014, 10:50:17 AM
 #11

Topic is locked as this thread will only be a repository for accepted defacto adoption of the protocol.
Discuss here:
https://bitcointalk.org/index.php?topic=557991.0

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!