[ANN] Stratum mining protocol - ASIC ready

[slush]

Hello all,

I'd like to announce my proposal for new pooled mining protocol.

All details and links can be found here: http://mining.bitcoin.cz/stratum-mining

I'm running experimental pool since yesterday. You can connect to it using mining proxy, which is linked from the document above.

Both pool source and mining proxy is opensource!

I'm waiting to your comments :-)

[eleuthria]

Well, I knew we were both working on something, and our limited conversation a few months back made me think we were attacking it in two very different ways, with my proposal being significantly better.  Now that you've released yours, I see they're almost identical.

I am withdrawing the protocol I've proposed and amending previous posts shortly.  Since our designs were so similar, I'll be updating my custom pool code to utilize the syntax you've defined.  We don't need a VHS vs Betamax competition for protocols which accomplish the same thing in almost the same way.  All pools and miners benefit from this new protocol, and the faster more pools show support to adopt it the better.

Please just adopt the two commands I posted about previously:  A redirect command for a poolserver to send miners elsewhere, and a server restart notification [with timer] so a pool can attempt a graceful restart rather than suddenly dropping connections.

Aside from that, lets get this stuff natively supported by mining software!

[Graet]

Thanks guys, looking closely
Graet

[slush]

We don't need a VHS vs Betamax competition

+1

Please just adopt the two commands I posted about previously:  A redirect command for a poolserver to send miners elsewhere

In the proxy source you can see methods client.reconnect(host) and client.add_peers(list). First one may be used for redirecting miners elsewhere, peer list is more for backup locations, when server crashes unexpectedly.

server restart notification [with timer] so a pool can attempt a graceful restart rather than suddenly dropping connections.

What's the use case for this? If I want to restart the server I can do it without such command as it doesn't require any action from the miner side.

Maybe I can add "wait" option to existing client.reconnect() instead?

[eleuthria]

server restart notification [with timer] so a pool can attempt a graceful restart rather than suddenly dropping connections.

What's the use case for this? If I want to restart the server I can do it without such command as it doesn't require any action from the miner side.

Maybe I can add "wait" option to existing client.reconnect() instead?

The thinking is for when a pool is doing some kind of maintenance (either to update their code/settings, or reboot the machine).  It's "good manners" for the pool to notify the miner in ADVANCE, that way they can move to a fallback pool/server before suddenly losing connection.  A "retry/wait" time should also be included in the restart method, so the miner knows how long it should wait before trying to reconnect.

[Graet]

server restart notification [with timer] so a pool can attempt a graceful restart rather than suddenly dropping connections.

What's the use case for this? If I want to restart the server I can do it without such command as it doesn't require any action from the miner side.

Maybe I can add "wait" option to existing client.reconnect() instead?

The thinking is for when a pool is doing some kind of maintenance (either to update their code/settings, or reboot the machine).  It's "good manners" for the pool to notify the miner in ADVANCE, that way they can move to a fallback pool/server before suddenly losing connection.  A "retry/wait" time should also be included in the restart method, so the miner knows how long it should wait before trying to reconnect.

This is  a very nice idea

[slush]

The thinking is for when a pool is doing some kind of maintenance (either to update their code/settings, or reboot the machine).  It's "good manners" for the pool to notify the miner in ADVANCE, that way they can move to a fallback pool/server before suddenly losing connection.  A "retry/wait" time should also be included in the restart method, so the miner knows how long it should wait before trying to reconnect.

Ok, I can add it into the specs, but most likely I won't implement it in my code, just because all these fancy features are making things complicated, without providing any materialized benefit.

For example the twisted framework in proxy is solving reconnections automatically for me, in exponential back-off (it will reconnect and if it fail, next retry will be after longer time). I think it's quite fine behaviour and forcing it to strictly follow the wait interval provided by the server will just require some hacks.

[Luke-Jr]

Not sure this is really any different/better than the BIP 22/23 standards which have been in open development for months...
as for the comments on it in the Stratum mining protocol doc:

• the fundamental BIP 22 operation is fairly easy to implement, just making use of the more advanced functionality (which StratumMP doesn't even support yet) is more complex
• GBT does not require or depend on HTTP; it is intentionally designed to be JSON-RPC over any transport
• GBT does not require a complete dump of the server's memory pool
• checking shares shouldn't be any different AFAIK?

We don't need a VHS vs Betamax...

Perhaps it would have been a good idea for you and slush to participate in forming the standards instead of privately working on duplicate protocols? :/

[slush]

Luke, nobody is saying that your baby is bad. I just think that BIP22/23 is too complex in parts which are irrelevant for pooled mining and it don't solve issues which are relevant there. So we're just solving another problem. Of course there's no reason to implement Stratum directly to bitcoind, as an example.

Actually I spent 90% of time on decisions not related to payload. As you wrote "it is intentionally designed to be JSON-RPC over any transport". Stratum protocol itself is the same, as you can read on google document specification. But to have viable solution for real-world miners, you have to propose complete stack and step-by-step algorithm, which obvously BIP22/23 isn't at this time.

For this reason I picked the best things lying around and knocked up this proposal & implementation, which can be used here and now.

[Luke-Jr]

Luke, nobody is saying that your baby is bad. I just think that BIP22/23 is too complex in parts which are irrelevant for pooled mining and it don't solve issues which are relevant there. So we're just solving another problem.

BIP 22/23 is only complex if you make use of the extensions that don't exist for StratumMP at all. If you don't, it's in fact just as simple (perhaps simpler).

But to have viable solution for real-world miners, you have to propose complete stack and step-by-step algorithm, which obvously BIP22/23 isn't at this time.

Block building is the same no matter what protocol conveys the data... if you're not happy with the current level of documentation, it makes more sense to improve it rather than just start from scratch.

For this reason I picked the best things lying around and knocked up this proposal & implementation, which can be used here and now.

gmp-proxy has been functional on open source Eloipool-based pools for months.

[Garr255]

Thanks slush!

[slush]

gmp-proxy has been functional on open source Eloipool-based pools for months.

To be honest, I don't understand how gmp-proxy works, so I'm unable to find out what the payload itself is. Cryptic variable names and not-a-single comment in the source code don't help the readability much.

However you're still talking about payload, but it is just small fragment of the problem. I'm not sure how many connected users your pool has, but maybe keeping 10k+ connections from weird cpu miners will change your opinion. With the Stratum protocol, load is completely driven by the server, so I can handle these 10k connections with a single CPU core.

[Luke-Jr]

gmp-proxy has been functional on open source Eloipool-based pools for months.

To be honest, I don't understand how gmp-proxy works, so I'm unable to find out what the payload itself is. Cryptic variable names and not-a-single comment in the source code don't help the readability much.

Sure, gmp-proxy isn't exactly written to be readable - it's just a quick proxy cobbled together. Doesn't change the fact that it works.

However you're still talking about payload, but it is just small fragment of the problem. I'm not sure how many connected users your pool has, but maybe keeping 10k+ connections from weird cpu miners will change your opinion. With the Stratum protocol, load is completely driven by the server, so I can handle these 10k connections with a single CPU core.

You have to keep 10k connections no matter what protocol you use over them, unless you're using UDP (which is a bad idea for other reasons). Long polling itself moves load to be server-driven, though that of course won't help the problem of announcing new blocks all at once no matter how you do it. A pool that doesn't want users to request new templates on their own could simply set the expiration on it to 90 minutes later and only return new ones via long poll responses. Besides, CPU mining is dead and will only get more dead as ASICs go online. There's no reason they can't continue to use getwork until they're gone completely.

[slush]

You're right.

[VeeMiner]

great stuff for the bitcoin community, thank you!

[wknight]

This is excellent. Very nice work!

[eleuthria]

Giving this a little boost:  This is way too important to let it slip from the top topics.

I have almost finished reworking my custom pool to use the stratum protocol over my proposal.  By this weekend there will be at least one more publicly available pool supporting it (1 week trial period at 100% PPS rates).

[slush]

Today I publicly asked developers of cgminer and phoenix if they're willing to support Stratum protocol. I already have confirmation from poclbm and Guiminer that they'll support it (m0mchil will implement it proactively for poclbm and Kiv will accept my patch which bundles proxy into the GUIminer).

[doublec]

Giving this a little boost:  This is way too important to let it slip from the top topics.

It'll take a while for those of us who've only just seen it to implement it. I'm interested in adding support for it to my pool but it'll take time to write and test.

[slush]

doubles, just telling anyone's support is valuable feedback at this time. Any vote counts, because there's still a long way to roll native support of this protocol into major miners. During this period we'll have painfull times while running both pools at the same time .