New Mining Protocol Proposal - "Infinite" Scaling, Low Overhead

[eleuthria]

THIS PROTOCOL PROPOSAL IS BEING WITHDRAWN
Now that slush has released more details on  his proposal, I am withdrawing my proposal.  They accomplish the same thing, and do it in the same way, with the only material differences being data order and encapsulation (JSON vs Plaintext command verbs).  Since slush has released an open source proxy AND poold proof of concept, I am going to adopt his proposal.  Since our methods were so similar, doing so will only require a few hours of rewriting my syntax.

Mining software developers:  Please begin working on native mining support!  This is sorely needed by the network.  It reduces overhead to a fraction of what we have today, and scales infinitely.  No matter how fast you are, the data overhead will not grow using the new protocols.

LINK TO SLUSH'S POST: https://bitcointalk.org/index.php?topic=108533.0




Slush is preparing an announcement for a similar protocol, but has not yet posted complete details.  The two protocols are extremely similar (the main differences are the command names), in that they accomplish the same goal with almost identical methods.  I would highly recommend any mining software developers to begin implementing the protocol I've specified below, simply because the documentation is already available.  Once slush posts his protocol specification, I will withdraw this proposal and support his unless there is a significant difference in the details.  

However, if you start working on implementing my protocol proposal into a mining software package, adapting it to slush's is likely to be a few minutes of work changing command names and the order of data being sent/received.  We don't need two competing protocols, but there's also no reason to hold back starting development since the basic structure is the same no matter which one ends up being adopted.


Over the last few months, I have been working with ArtForz to address the issues with the current getwork based protocol method for pooled mining.  As chips are getting faster and faster, the overhead of the protocol is becoming obvious.  The ability to scale a pool with getwork cannot be solved by share difficulty, which only reduces the number of shares that need to be validated, not the amount of work being generated/sent.  Current getwork methods require multiple connections (since an LP connection is open and only used to wait for new work), and all work is request-based.

This new protocol corrects the issues, and would not require any change to the bitcoin standard, this would only affect pools and mining software that want to support it.

This new protocol uses a single TCP socket connection to stream data back and forth from a miner and the pool.  Work is pushed by the pool, rather than requested by the miner.  The pool providers the miner with the data needed to build it's own blocks, removing the work creation from the pool side, and pushing it to the miner, without requiring a local bitcoind.  It does this by forwarding the miner the coinbase of the pool, and the merkle branch list required to build a merkleroot, and the remaining items needed to put together a block header.

One major change however, is this new protocol also delegates a portion of the "ExtraNonce" portion of the coinbase to the miner to be adjusted locally.  The protocol allows the pool to specify the ExtraNonce portion it wants to delegate to the miner.  An example of 4 bytes would provide miners with 4 billion times more work than a single getwork currently provides.  Extending the ExtraNonce delegation to 8 bytes provides an additional 4 billion times more work than 4 bytes.  Changing this value only increases the overhead of miner submissions by the number of bytes extended to the miner.


The following is a breakdown of all currently defined commands in the new protocol:
       

• Login
            
  • LOGIN_MESSAGE - Initial string sent by server, 'Message of the Day' type message
  • LOGIN,<username>,<password> - Login from client to server.  If no password, end after username (no comma)
  • RECONNECT,<token>,<username>,<password> - Reconnect string in case of disconnect, to avoid loss of work association.
  • INVALIDLOGIN - Sent from server to client when credentials don't match known user.
         
• Server Connection Messages
            
  • SERVERRESTART,<wait> - Message from server to client indicating that the server will be rebooting.  Optional 'wait' to identify time to wait before reconnecting, in seconds.
  • REDIRECT,<address>,<port> - Message from server to client requesting the client connect to a different server.  Can be sent at any time.
         
• Server Settings Messages
            
  • DIFFICULTY,# - Minimum difficulty the pool will accept from connection.  Can be changed dynamically.  # is difficulty in 2^x notation (2^32 = difficulty 1).
  • EXTRANONCEBYTES,# - How many bytes of ExtraNonce the miner can adjust locally.  Not adjusted after initial connection.
  • TOKEN,# - Token used to reconnect a lost session with the server.  32-bit Unsigned Integer.
         
• Share Submissions
            
  • SUBMIT,<WorkID>,<ExtraNonce>,<Nonce> - Client submission of share to server.  WorkID is sent as number.  ExtraNonce and Nonce are sent in hex.
  • SHARE,ACCEPT - Server accepted the share
  • SHARE,REJECT,<reason> - Server rejected the share.  Reason is sent as plaintext.  Standard messages: LOWDIFFICULTY, STALE, BADTIME, DUPLICATE, UNKNOWN
         
• New Work (all of the following come in one message)
            
  • NEWDATA (Sent as plaintext) - Signal of new work from server
  • ABORTWORK (Sent as 0 or 1) - 0 means this is just a work update, the client can finish its current hashing before updating.  1 signals a new block and old work should be aborted.
  • WORKID (sent as #) - ID of the work, used for matching the proper server data to the share submissions.  32-bit unsigned integer.
  • PREVBLOCKHASH (Sent as hex string) - Hash of the previous block on the network
  • NBITS (Sent as hex string) - nBits used for hashing work
  • CURTIME (Sent as hex string) - Current time to begin for work generation
  • COINBASE1 (Sent as hex string) - Raw transaction data for first half of coinbase.  Client appends ExtraNonce to this string
  • COINBASE2 (Sent as hex string) - Raw transaction data for the rest of the coinbase.
  • NBRANCH (Sent as #) - How many branches of the MerkleRoot are being sent.  If 0, no more data is sent and coinbase is the MerkleRoot.
  • BRANCHES (Comma separated hex strings) - Merkle Root branches, starting at the bottom and working to the top.
         

   

   Example communication (S = Server, C = Client)
   

S: LOGIN_MESSAGE,Welcome to BTC Guild!
C: LOGIN,eleuthria_fpga
S: DIFFICULTY,32
S: EXTRANONCEBYTES,4
S: TOKEN,42
S: NEWDATA,0,129,699dca00fc6490967a1c68b36c2a82e26b18fa20e868f01ac902000000000000,1a06dfbe,8e334550,01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff0a040000000104,ffffffff0170995f2c0100000043410456579536d150fbce94ee62b47db2ca43af0a730a0467ba55c79e2a7ec9ce4ad297e35cdbb8e42a4643a60eef7c9abee2f5822f86b1da242d9c2301c431facfd8ac00000000,10,bef55fb7ccc8440a1e7ce9074597705d5102de8d7de817a21cf6a56dcb315df3,c6b9a65168ed2463bcbd05f3a06c3dc20463e3f895c26144e26d644aeb216fcb,580cff4642d62c84901e77d97d964930a3fd808b8a8eecee92e54f25c3248987,4ffcfe8121cbafea9afbd63d796aeb195ef5c37120ec7976f9eb2270280afa96,522d9ae8fed0f7161007d0e0308460112d0af64ac5ab9a13d9457cbce15b8047,1b63da8c29fe5278e4e2f276f34c99cec6d8e0a3510629a7487dc66ad5450a8b,77d77bbd3ad69ad9c0c090212f995b70a5bd83fe15666d0e1d53af83448c5f1e,bce06954e8f33eeb761542e911f63898ce0e5e3128401b8c11f5f5e52b8cb7f1,0d506f82a03bb996181c428bb5910a695f02d49ae573930e215d2626ff265100,6ffe998be56d7ac12415da1c445443a05a5d50a243e6b37c6477131c75af4eb9
C: SUBMIT,129,00000001,abcdef12
S: SHARE,ACCEPT
C: SUBMIT,129,00000001,21fedcba
S: SHARE,ACCEPT
C: SUBMIT,129,00000005,419be9cf
S: SHARE,ACCEPT
C: SUBMIT,129,00000005,419be9cf
S: SHARE,REJECT,DUPLICATE
S: NEWDATA,0,130,699dca00fc6490967a1c68b36c2a82e26b18fa20e868f01ac902000000000000,1a06dfbe,cb334550,01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff0a040000000104,ffffffff01d4329e2c0100000043410456579536d150fbce94ee62b47db2ca43af0a730a0467ba55c79e2a7ec9ce4ad297e35cdbb8e42a4643a60eef7c9abee2f5822f86b1da242d9c2301c431facfd8ac00000000,10,bef55fb7ccc8440a1e7ce9074597705d5102de8d7de817a21cf6a56dcb315df3,c6b9a65168ed2463bcbd05f3a06c3dc20463e3f895c26144e26d644aeb216fcb,d04f869df7c5d8e1268b92ae923f1a6ace4ebb041bf4206c79da4b9c3a714cbc,d9b11e757304d3315c593886e98d7ea3ee62909c8aa31ab04fd351fd5fe4b12f,1b1251b43b32eff12cf2387648836ba4b4be505804027b95c9055f8061f621d4,020f95702e4fefac6b2dab07e9bd261a83f0c2db25cb357fb49d98fd6f9dce22,669c559891e364b647f28912e04a9d35ae23097edb010e90b8d596e0d5cbaa5f,39fe7146f9de3e442f2b717c119a1c8025bfe8a6c9ffb15cbd4ccdc5d970d0af,0cadbdfd2cece0eea1c28a928590296c3374536beaacc8923548bac4c1a07689,9272c10c9f67a9ce52c5e2b9c30d4974e8c7aa2f6263c4b50b8da066ce0c50d0
C: SUBMIT,129,00000005,f3b2a6cc
C: SUBMIT,130,00000001,0a2304b1
S: SHARE,ACCEPT
S: SHARE,ACCEPT
S: NEWDATA,1,131,ca952564b1efb225325f7e1f43331731a7f2a82ea2e5bbe49800000000000000,1a06dfbe,d7314550,01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff0a040000000104,ffffffff0100f2052a0100000043410456579536d150fbce94ee62b47db2ca43af0a730a0467ba55c79e2a7ec9ce4ad297e35cdbb8e42a4643a60eef7c9abee2f5822f86b1da242d9c2301c431facfd8ac00000000,0
C: SUBMIT,130,00000003,ab123456
S: SHARE,REJECT,STALE

I would like to work with any mining software developers who want to support this.  I will be providing a proxy that allows old mining software to connect to a pool already running this new protocol in the next few days.  However, there is already a pool running this new protocol, so I will happily provide information on it to any mining software developer so they can begin.

[eleuthria]

Protocol Overhead
The new protocol requires one TCP connection per worker (which can be multiple cards or even multiple machines running through a proxy).

Recommended settings for a pool are as follows:

• Work updates pushed once per minute.
• Difficulty adjusted to 1 share per 5-10 seconds
• ExtraNonce size of 4 bytes

The following is a bandwidth overhead estimation for ANY speed, using the above recommended settings:
Inbound: Work received per minute, assuming 512 TX blocks: 950 bytes per minute (15.83 bytes/sec)
Inbound: Share responses per minute (12): 144 bytes (2.4 bytes/sec)
Inbound: "Longpoll" work pushes: 365 bytes per 10 minutes (0.6 bytes/sec)
Outbound: Shares submitted per minute (12): 348 bytes (5.8 bytes/sec)


Inbound bandwidth: 18.83 bytes/second
Outbound bandwidth: 5.8 bytes/second
Maximum inbound burst: 950 bytes of data/sec*
Maximum outbound burst: 29 bytes of data/sec*

*Possibly a few bytes more if a share is submitted and returned in the same second as a work update, or if multiple shares submitted in one second.



This will scale until a single miner is able to run 18 Exahash in a 1 minute period.  At which point, ExtraNonce would require increasing.  At 8 bytes ExtraNonce, outbound bandwidth increases to 7.4 bytes/sec.  The protocol was designed with ExtraNonce being variable, so this would just require a pool to change their settings, not a new revision to the protocol.

[eleuthria]

Protocol Benefits for Pools

• Connection Scaling Problems
  
  • 1) Removal of redundant connections.  Longpoll connections are currently redundant, because it requires the server to hold open a single connection that is only used roughly every 10 minutes.  All other communication is done by opening/closing new connections.  This puts significant stress on a server compared to simply using an already established connection to handle communication.
  • 2) Work generation overhead.  Pools currently generate work for miners on demand, or constantly fill a work cache to respond to work requests.  This requires the pool to handle the overhead of calculating merkleroots for each work request.  While this is not a huge demand on a per worker basis, it adds up.  This is also very noticeable in the case of botnets, where they often demand work far faster than needed.  The pool only has to calculate and store the list of merklebranches once, and then forward that data to the miners.  In a properly setup pool with the new protocol, the pool server only calculates merklebranches once per work push, rather than once per miner.
    
  • Additionally, this solves scalability problems by incorporating the ability for pools to send miners to other servers as part of the protocol.  The new protocol has no "random" factor in terms of scalability.  Overhead is purely based on the number of established connections, not the number of graphics cards, or speed of the miners.  By incorporating a REDIRECT message into the protocol, a pool operator can determine the capacity of their server, and set the server to send redirects when capacity is reached.  
    
    Alternatively, a pool can be designed can redirect miners to new servers via Round Robin or Smart Loadbalancing depending on how they design their pool daemon.  It removes the uncertainty of round robin DNS, and also removes the need for any dedicated load balancing servers/hardware.

• Bandwidth and Botnet Problems
  
  • Bandwidth with the new protocol is reduces to a fraction of current getwork based methods.  With the new protocol, the bandwidth requirements scale only with the number of miners, rather than the total speed of miners.  A pool server with 1,000 active miners require peak burst bandwidth at roughly 950,000 bytes.  This can be accomplished on any server with a 10mbit connection.  At 100 mbit, this can be done in less than 1/10th of a second.
  • Botnets have been a plague on pool servers ever since the major bubble last year.  With the new protocol, miner connections are persistent.  This makes it significantly easier to identify botnets, since each PC would require a separate persistent connection.
    

Protocol Benefits for Miners

• Connection Issues
  
  • As identified above, miners will be able to reduce their bandwidth usage to less than 100 bytes/second.  Pull out your old dial-up modems and plug in an entire farm of ASICs, your bandwidth will never increase regardless of how fast your computer mines.
  • Reduced reliance on latency.  With the new system, your miner does not need to queue work so that it is always busy.  Your miner receives enough work to last until the next piece of work is sent, meaning it will never again experience 'Pool not providing work fast enough' or idle miner errors.
    

• Pool Disconnects/Backup Pools
  
  • By using a persistent connection, it is easy for mining software to simply maintain a connection with two pools.  It is apparent when a pool has a problem because the socket will be closed.
  • The protocol standard includes a warning message servers can give to miners to identify that a restart will occur.  This allows miners to immediately move to their backup server until the pool is restored, eliminating the possibility of work being completed but unable to send due to a pool restarting.

[iCEBREAKER]

Dude, awesome!     

Even our private servers have been giving me "not providing work fast enough" messages, so this will really help!

[slush]

I'm writing documentation for quite similar specification right now, I call it "Stratum mining protocol", because I re-used protocol specification "Stratum" designed for lightweight bitcoin client Electrum. My pool software is opensource and there's already getwork-compatible proxy.

https://github.com/slush0/stratum-mining
https://github.com/slush0/stratum-mining-proxy

Edit: As a bonus you can use one TCP connection for more workers :-).

[eleuthria]

Looks like you integrated a lot of the details from the PMs we shared a few months ago ;p

[slush]

Looks like you integrated a lot of the details from the PMs we shared a few months ago ;p

Not really, I discussed these ideas also with m0mchil year ago on Prague conference . The things which I discussed in the PM was "democratic mining mode", which means that miners can pick their own transactions. This is actually not possible in both of our proposals/implementations.

[koin]

This new protocol corrects the issues, and would not require any change to the bitcoin standard, this would only affect pools and mining software that want to support it.

will this do anything to improve the defenses a pool has when being attacked with a ddos?

[eleuthria]

This new protocol corrects the issues, and would not require any change to the bitcoin standard, this would only affect pools and mining software that want to support it.

will this do anything to improve the defenses a pool has when being attacked with a ddos?

No protocol can prevent a DDoS (except small ones).  The big DDoS's that have hit major don't just crash/flood the pool software, it floods the switch the pool connects to with enough packet flow/data to make it unusable.

However, this protocol does make it easier to run a setup like BTC Guild has moved to, where the connectable IPs are all external from the pool server, making it easier to move servers around during an attack.  The vast reduction in bandwidth required, along with significantly lower CPU usage enables things like setting up a dozen small VPS/EC2 instances up as the frontends, which just act as proxies for the actual backends.

[doublec]

Code:

C: SUBMIT,129,00000001,abcdef12
S: SHARE,ACCEPT
C: SUBMIT,129,00000001,21fedcba
S: SHARE,ACCEPT
C: SUBMIT,129,00000005,419be9cf
S: SHARE,ACCEPT

After a "SUBMIT" does the connection not process any more submissions until "SHARE" is returned? Or is it accepting and returning results in parallel? If the latter, how does a client associate a result with a particular submission? If they are processed serially, will this be a bottleneck for clients unless they have multiple connections to the server?

Code:

DIFFICULTY,# - Minimum difficulty the pool will accept from connection.  Can be changed dynamically.  # is difficulty in 2^x notation (2^32 = difficulty 1).

I don't mean to bikeshed, but why another format for difficulty? Already we have floating point format, target format and compresses bits. Can the protocol use one of those already well understood formats?

Code:

SUBMIT,<WorkID>,<ExtraNonce>,<Nonce> - Client submission of share to server.  WorkID is sent as number.  ExtraNonce and Nonce are sent in hex.

What are the restraints on the number for the WorkID? Is it a a 32 bit/64 bit integer? Can it wrap around or be reused? What does the server use it for?

Is there an error response from the server to indicate bad data sent by the client (eg. malformed hex strings, signed numbers vs unsigned,etc) or will it just disconnect?

[eleuthria]

Code:

C: SUBMIT,129,00000001,abcdef12
S: SHARE,ACCEPT
C: SUBMIT,129,00000001,21fedcba
S: SHARE,ACCEPT
C: SUBMIT,129,00000005,419be9cf
S: SHARE,ACCEPT

After a "SUBMIT" does the connection not process any more submissions until "SHARE" is returned? Or is it accepting and returning results in parallel? If the latter, how does a client associate a result with a particular submission? If they are processed serially, will this be a bottleneck for clients unless they have multiple connections to the server?

Code:

DIFFICULTY,# - Minimum difficulty the pool will accept from connection.  Can be changed dynamically.  # is difficulty in 2^x notation (2^32 = difficulty 1).

I don't mean to bikeshed, but why another format for difficulty? Already we have floating point format, target format and compresses bits. Can the protocol use one of those already well understood formats?

Code:

SUBMIT,<WorkID>,<ExtraNonce>,<Nonce> - Client submission of share to server.  WorkID is sent as number.  ExtraNonce and Nonce are sent in hex.

What are the restraints on the number for the WorkID? Is it a a 32 bit/64 bit integer? Can it wrap around or be reused? What does the server use it for?

Is there an error response from the server to indicate bad data sent by the client (eg. malformed hex strings, signed numbers vs unsigned,etc) or will it just disconnect?

Miners can push as many submissions as needed to the pool.  A properly designed pool would process the shares in the order submitted, and respond with ACCEPT/REJECT in the order submitted.  The miner does not have to wait for a response before pushing new submissions (though if they aren't getting responses it would be a sign of a server/connection problem).


Using a 2^X for difficulty was suggested by ArtForz for two primary reasons.  The first is it means the protocol can be used by non-bitcoin altcoins which use shares of difficulty less than 1 (2^32).  The second was that there was no need for precision share difficulty (non-power of 2).  With a target rate of 5-10 seconds/share, doubling the difficulty easily accomplishes this.  By using powers of two, this also requires far less work on the pool side to validate if a share is the proper difficulty.


WorkID can wrap around, or reset with each ABORTWORK type work push.  That would be up to the individual pool.  When the miner submits their share, the pool matches the WorkID so it knows which merklebranch list to use in computing the merkleroot, as well as all of the transactions that were used to build that merkleroot.  I will add to the spec that the WorkID should be limited to a 32 bit integer (unsigned).

[eleuthria]

I've posted a notice above about slush's protocol.  As of right now, I don't believe there is any reason to wait for slush to post his specification before starting to work on adapting mine into mining software.  I have gone through his proxy code, and the only difference is the order of data being sent and the "commands" being sent.  Once he posts his full spec, I will withdraw my proposal assuming there is no "gamebreaker" visible in his proposal.

By starting work on my protocol, you'll be able to get a head start.  The basic structure remains the same:  The mining software will receive two pieces of a pool coinbase with a gap filled by a locally adjusted ExtraNonce, and then hashes that with a list of merklebranch hashes to generate a merkleroot, then combine that with the rest of the block header for hashing.  Other than that, it's just a manner of the order the data is relayed back and forth.