[ANN] Stratum mining protocol - ASIC ready

[eleuthria]

Giving this a little boost:  This is way too important to let it slip from the top topics.

It'll take a while for those of us who've only just seen it to implement it. I'm interested in adding support for it to my pool but it'll take time to write and test.

I know it will take a while for more pools to adopt it, especially when most of the pool operators are utilizing projects like ecoinpool, poolserverj, and pushpool.  Luckily, slush has provided an open source pool example for the new protocol along with his proxy, allowing anyone to step through the protocol from both sides to gain a better understanding of it and implement their own version (either by editing the example or starting fresh).

You can contact slush or myself in #stratum on FreeNode if you're running into any problems or have questions about efficiency/scalability.

doubles, just telling anyone's support is valuable feedback at this time. Any vote counts, because there's still a long way to roll native support of this protocol into major miners. During this period we'll have painfull times while running both pools at the same time .

As slush posted, just identifying interest in implementing the new protocol on your pool will help give a push to mining software developers.  Likewise, as mining software developers express interest in native support, it will hopefully encourage more pools to begin adopting the new protocol as well.

[1714982084]

1714982084

[1714982084]

1714982084

[1714982084]

1714982084

[1714982084]

1714982084

[1714982084]

1714982084

[doublec]

I know it will take a while for more pools to adopt it, especially when most of the pool operators are utilizing projects like ecoinpool, poolserverj, and pushpool.  Luckily, slush has provided an open source pool example for the new protocol along with his proxy, allowing anyone to step through the protocol from both sides to gain a better understanding of it and implement their own version (either by editing the example or starting fresh).

Yes, slush's example will be useful. My pool is custom software which I think will actually make it easier for me since I know it better than trying to add support to other pool software. I think it's great that you and slush sorted out the differences between the proposals and hope the new protocol gets some traction.

[Mobius]

how is the port changed on the proxy since this conflicts with p2pool proxy?

[slush]

mining_proxy.py --help will prints all options.

Specifically to your request, use "mining_proxy.py -gp 8330" (or any other port number). "gp" identifies "getwork port"

[Luke-Jr]

I've written up a forum post for getblocktemplate; replies on that topic are probably off-topic on this thread.

[slush]

I just released new version of mining proxy, version 0.5.0. Older versions still works nicely, this adds some few features:

* Implemented method client.reconnect(host, port), so pool can now easily balance clients between backends or gracefully shutdown a backend without a miner outage.

* Added "--no-midstate" parameter. When used, proxy won't generate "midstate" field in the getwork response. Some old miners and Diablo still requires this field, but modern miners like poclbm/cgminer can generate midstate ourselves and in much faster way. So if you're using these miners and want to save a bit of CPU time, this argument is just for you. It speeds up getwork creation twice.

In the near future I'll publish complete API documentation of such commands like client.reconnect(), client.get_version() or mining.get_hashrate(). Stay tuned :-).

[slush]

Two hours ago my Stratum-powered pool core mined its first block: http://blockchain.info/block/0000000000000322b90111260dcbb877c62c14e672c23a99a86ca444dc5525ee?site=slush

This is a confirmation that opensource stack which I provided builds blocks which are accepted by main chain.

[dooferorg]

I noticed that BTCGuild, a favored pool that I use, is testing the stratum mining proxy and that's what lead me here. I've set this up and it's working well. No stales/dupes at all from a host that would occasionally show a few.

I also use a PHP based proxy for managing pools and workers: https://github.com/cdhowie/Bitcoin-mining-proxy

Of course for testing it I have a pool set to my local machine that is running the stratum proxy.

I may consider trying to integrate these (being a programmer myself) so that different pools can be set up easily and have 'stratum proxying' enabled if they support it. Seems very sensible as I look over the description of the work and protocol that you listed. If I do make a PHP version that works like your one I will of course post it here. Do you think the protocol is pretty much 'set' for now and it's just a matter of testing and getting it rolled out?

[slush]

Yes, I think that protocol itself is stable. I'll just add some additional methods for managing connections, like client.reconnect() or so, but it's not directly related to protocol core and you can implement it "as is".

I'm just not sure how you can implement persistent connections with PHP. Maybe running "official" python proxy and implement your software to talk to it locally would be smarter (and easier) choice.

[eleuthria]

Just to add to the good news, BTC Guild's stratum beta also found it's first block on the live Bitcoin network: http://blockchain.info/block-height/198695

Before beta ends I'll have to go back and add back our tag in the coinbase stating it came from BTC Guild.

[makomk]

I notice that this spec doesn't seem to document what values of ntime are acceptable. That's a bit unfortunate.

Edit: Also, clean_jobs probably ought to be split up. As it stands there's no way of signalling that miners should stop working on old jobs immediately without also causing them to discard any unsubmitted shares they might have, but for some setups (e.g. ones that use merged mining or p2pool) not being able to force miners onto new work without discarding shares kills efficiency.

[slush]

I notice that this spec doesn't seem to document what values of ntime are acceptable. That's a bit unfortunate.

The best value is actual ntime. Actually rolling ntime is unnecessary with Stratum. But I understand that at least until miners will support Stratum natively, it's necessary to accept rolled ntime. My pool currently accepts ntime in interval <job_ntime, current_ntime+600>, which fits all known current configuration (it has been tested with BFL FPGA rig doing around 30GHash/s as a single device).

Edit: Also, clean_jobs probably ought to be split up. As it stands there's no way of signalling that miners should stop working on old jobs immediately without also causing them to discard any unsubmitted shares they might have, but for some setups (e.g. ones that use merged mining or p2pool)

I think it is stated clearly. If miner received "clean_job" flag, it should not submit shares from previous jobs. Nothing serious will happen if you submit stale job, they just probably won't be accepted. It's up to server if/when he decides to send such flag.

not being able to force miners onto new work without discarding shares kills efficiency.

If I understand this correctly, it's inefficiency included in merged mining concept.

[jgarzik]

Custom binary protocol using Google Protocol Buffers would be much preferred.  You write

Protocol buffers by Google is interesting concept which may fit the needs, except that only C++, Python and Java are supported.

Your list of supported languages is woefully incomplete, and therefore, does not support the argument.  PB are supported for: C, C++, Python, Perl, Java, PHP, Ruby, and Haskell at a minimum.  Any language used by the bitcoin community is highly likely to be supported by PB.

Furthermore, PB encoding is strict and well debugged, unlike any newly created protocol (including my own binary protocol from years ago) and this new protocol.

Something for ASIC miners should be bandwidth and CPU efficient, and text/JSON is not that.  This has already been shown to be a problem in bitcoind, where pool server operators replaced the default JSON hex decoding routines with a faster routine, to decrease CPU usage associated with useless text encode/decode.

PB is used in thousands of apps, including most of the Google applications we all use (like search).

[slush]

Jeff, it makes me smile that after all of our discussion you became a pb advocate .

Actually the amount of data is *so* tiny that even json message fits in one tcp packet in most of cases. Plus json has an advantage od human readability.

Protocol is not bounded to miner performance at all, so theres no reason why asic miners should need more compressed format. And the cpu overhead is more the issue on server side than on miners. I still picked json, because I expect that miner developers would be agains including another external dependency in their software.

[Luke-Jr]

Something for ASIC miners should be bandwidth and CPU efficient, and text/JSON is not that.  This has already been shown to be a problem in bitcoind, where pool server operators replaced the default JSON hex decoding routines with a faster routine, to decrease CPU usage associated with useless text encode/decode.

I'm not sure this has been a problem for anything but bitcoind.

[dooferorg]

Yes, I think that protocol itself is stable. I'll just add some additional methods for managing connections, like client.reconnect() or so, but it's not directly related to protocol core and you can implement it "as is".

I'm just not sure how you can implement persistent connections with PHP. Maybe running "official" python proxy and implement your software to talk to it locally would be smarter (and easier) choice.

Yes, I had pondered this too after I wrote my previous post. It might indeed not be worth it to implement it in PHP. I have written PHP processes that sit and run outside of a webserver (daemons, if you will). Within PHP too you can do all the regular socket operations so persistent network connections would be okay, there is just the question of whether it's worth duplicating the effort - probably not

Since I run a proxy for talking with different pools and having it manage the failover to those different pools so my miners can just keep plugging away I was thinking about how best to set up multiple stratum connections, i.e. one for each of the pools that will implement it.

As you mention, it might be best to simply run N instances of the python proxy and just plug that info into the PHP web-based proxy that I use right now. Would certainly mean less effort to implement. K.I.S.S. as they say.

[Hawkix]

After a day of playing and running through the Stratum mining proxy I must say I am really HAPPY with it. I can now manage 5 GPUs (1700 MH/s) total over 8 kbit/sec (yes, 1000 chars per second) poor local slow-ping GPRS limited connection link (tunelled via SSH with compression) without any problems and minimal stales! Something I was not able to accomplish previously with http requests (a lot of them).

Kudos to Slush! I think you should really more advertise this method.

[marcus_of_augustus]

Sounds interesting. Is it any way, except name, related to the Stratum (electrum) client-server protocol used for bitcoin transaction forwarding?

[slush]

The protocol has been designed for Electrum client and it is fully compatible. "Stratum mining" is just the Stratum service used for bitcoin mining instead of providing user's balance...

[GernMiester]

Tried it. Like it. Using it.