[ANN] Stratum mining protocol - ASIC ready

[dlasher]

I ran into an interesting failure state today, and the stratum proxy, I believe, played a roll in the failure. I posted over in the cgminer thread about the cgminer portion, but wanted to post here about the proxy portion.

Shouldn't the proxy, in some way, tell the clients when it's primary upstream connection dies?

Yes, it should. Let me ask some questions:

a) Are you using latest Windows binary?
b) If you're using Linux, are you sure you ran "setup.py install" or "setup.py develop" after last upgrade?
c) Do you see any error/traceback in the proxy during connection outage?

Proxy should propagate all connection failures to all connected miners immediately. It is implemented in method on_disconnect (https://github.com/slush0/stratum-mining-proxy/blob/master/mining_proxy.py) and I don't see any reason why it shouldn't work, except ther was another error.

A: No. cgminer 2.8.3 (at the time) linux, proxy 1.1.1 linux.
B: yes, and yes
C: no, proxy just sat.. idle.

primary pool hasn't crashed again since, so haven't had the chance to see again.. but I'll let you know if it happens.

[1715080924]

1715080924

[1715080924]

1715080924

[1715080924]

1715080924

[1715080924]

1715080924

[slush]

Maybe this is a possible future feature of cgminer, being a proxy itself.

Proxy already support "mesh networking" feature; it acts as a UDP multicast responder and it is able to report to which pool it is connected and on which port is proxy listening.

My idea for the future is that every miner on startup send UDP multicast request to the network and get a list of existing proxies on local network. It pick these connected to desired pool and talk to them instead of using direct connection to that pool. Next level is that miner, which don't found alive proxy for his pool, became a proxy itself and start responding for UDP requests for other miners.

I don't know how difficult would be to implement proxying feature directly to cgminer, but probing local network for listening proxies is pretty trivial and it can be the first step: https://github.com/slush0/stratum-mining-proxy/blob/master/example_multicast.py (You can use this script to get a list of alive proxies on the network.)

[slush]

primary pool hasn't crashed again since, so haven't had the chance to see again.. but I'll let you know if it happens.

I think that I found some possible race condition which can trigger your issue. It is more theoretical than practical chance, but definitely worth of fixing. I'll release bugfix tomorrow.

[kano]

So ... after effectively being told I was wrong by slush ...

... and not using stratum since the 2 pools were not in my list of pools that I would consider using ...

I added a stratum pool to my list (of 13 - so now 14) pools and pool duplicates today
I added BTCGuild to help with testing the code.

And the very first difficulty switch I got:

 [2012-10-21 09:11:09] Accepted fbab8535 Diff 1/1 ICA 0 pool 2
 [2012-10-21 09:11:16] Accepted 85774b9a Diff 1/1 GPU 1 pool 2
 [2012-10-21 09:11:42] Stratum from pool 2 requested work restart
 [2012-10-21 09:11:42] Rejected 3b789247 Diff 4/2 ICA 1 pool 2 (olddifficulty)
 [2012-10-21 09:12:03] Accepted 77567e1c Diff 2/2 GPU 1 pool 2
 [2012-10-21 09:12:24] Accepted 5326e93f Diff 3/2 GPU 1 pool 2

Lost me a share ... but looking at it I don't even see why it would since it was higher difficulty than the required target.

Of course if it had been lower difficulty, then I would have lost it due to the argument that slush said was invalid.
But I lost it when it was higher difficulty so I guess that's a problem with BTCGuild?

[eleuthria]

So ... after effectively being told I was wrong by slush ...

... and not using stratum since the 2 pools were not in my list of pools that I would consider using ...

I added a stratum pool to my list (of 13 - so now 14) pools and pool duplicates today
I added BTCGuild to help with testing the code.

And the very first difficulty switch I got:

 [2012-10-21 09:11:09] Accepted fbab8535 Diff 1/1 ICA 0 pool 2
 [2012-10-21 09:11:16] Accepted 85774b9a Diff 1/1 GPU 1 pool 2
 [2012-10-21 09:11:42] Stratum from pool 2 requested work restart
 [2012-10-21 09:11:42] Rejected 3b789247 Diff 4/2 ICA 1 pool 2 (olddifficulty)
 [2012-10-21 09:12:03] Accepted 77567e1c Diff 2/2 GPU 1 pool 2
 [2012-10-21 09:12:24] Accepted 5326e93f Diff 3/2 GPU 1 pool 2

Lost me a share ... but looking at it I don't even see why it would since it was higher difficulty than the required target.

Of course if it had been lower difficulty, then I would have lost it due to the argument that slush said was invalid.
But I lost it when it was higher difficulty so I guess that's a problem with BTCGuild?

BTC Guild forces you to submit new work when updating difficulty currently.  It won't accept work from previous work when you're adjusted because it keeps track of the miner's difficulty separate from the work sent to the miner.  With the current setup, if I accepted old work, you'd get credited at the new difficulty.  This opens up an exploit where a very fast miner can purposely withhold higher diff shares at first, knowing that they are going to force multiple difficulty increases in a short time, then submit them all for higher credit than what they should have been earning.

While it could be done in another way (keeping a list of difficulties for the miner as of each WorkID), it would add a lot of extra code for effectively no gain.  We're talking maybe a few pennies over the course of a year, if that.  Difficulty adjustments are a rare occurrence.  They only happen when you first start mining over the connection.  That share you "lost" from the diff adjustment was 0.0001 dollars (1/100th of a penny).

[-ck]

Hopefully you don't make the mistake of rejecting block solves then just because they happen across difficulty changes

[kano]

So ... after effectively being told I was wrong by slush ...

... and not using stratum since the 2 pools were not in my list of pools that I would consider using ...

I added a stratum pool to my list (of 13 - so now 14) pools and pool duplicates today
I added BTCGuild to help with testing the code.

And the very first difficulty switch I got:

 [2012-10-21 09:11:09] Accepted fbab8535 Diff 1/1 ICA 0 pool 2
 [2012-10-21 09:11:16] Accepted 85774b9a Diff 1/1 GPU 1 pool 2
 [2012-10-21 09:11:42] Stratum from pool 2 requested work restart
 [2012-10-21 09:11:42] Rejected 3b789247 Diff 4/2 ICA 1 pool 2 (olddifficulty)
 [2012-10-21 09:12:03] Accepted 77567e1c Diff 2/2 GPU 1 pool 2
 [2012-10-21 09:12:24] Accepted 5326e93f Diff 3/2 GPU 1 pool 2

Lost me a share ... but looking at it I don't even see why it would since it was higher difficulty than the required target.

Of course if it had been lower difficulty, then I would have lost it due to the argument that slush said was invalid.
But I lost it when it was higher difficulty so I guess that's a problem with BTCGuild?

BTC Guild forces you to submit new work when updating difficulty currently.  It won't accept work from previous work when you're adjusted because it keeps track of the miner's difficulty separate from the work sent to the miner.  With the current setup, if I accepted old work, you'd get credited at the new difficulty.  This opens up an exploit where a very fast miner can purposely withhold higher diff shares at first, knowing that they are going to force multiple difficulty increases in a short time, then submit them all for higher credit than what they should have been earning.

While it could be done in another way (keeping a list of difficulties for the miner as of each WorkID), it would add a lot of extra code for effectively no gain.  We're talking maybe a few pennies over the course of a year, if that.  Difficulty adjustments are a rare occurrence.  They only happen when you first start mining over the connection.  That share you "lost" from the diff adjustment was 0.0001 dollars (1/100th of a penny).

Ignoring all the other arguments I have already made about this issue ... specifically pointing out the expected % of shares that will fall in this category that may be of lower difficulty ... but you've just explained that it is ALL shares of any difficulty (and as ckolivas has pointed out - it will also include valid block solves that you will be rejecting) ...

... this also directly means on a PPS pool, the more diff changes you send, the more money the pool saves.

[eleuthria]

Ignoring all the other arguments I have already made about this issue ... specifically pointing out the expected % of shares that will fall in this category that may be of lower difficulty ... but you've just explained that it is ALL shares of any difficulty (and as ckolivas has pointed out - it will also include valid block solves that you will be rejecting) ...

... this also directly means on a PPS pool, the more diff changes you send, the more money the pool saves.

Except I don't check shares that are rejected as block solves, so it's not saving money, it's a break even (I don't pay the miner for submitting olddifficulty shares, and I don't check to send them upstream).  Even so, the fact is difficulty changes are infrequent.  They will only happen when you first turn on your miner.  And only a few times (if any).

Once I track down the current bug on Stratum that caused the last crash, I'll look into adding a small window where old work can be submitted after a diff adjustment.  This really isn't a high priority though since any miner that isn't constantly hopping between pools will rarely see an olddifficulty reject.

[kano]

Ignoring all the other arguments I have already made about this issue ... specifically pointing out the expected % of shares that will fall in this category that may be of lower difficulty ... but you've just explained that it is ALL shares of any difficulty (and as ckolivas has pointed out - it will also include valid block solves that you will be rejecting) ...

... this also directly means on a PPS pool, the more diff changes you send, the more money the pool saves.

Except I don't check shares that are rejected as block solves, so it's not saving money, it's a break even (I don't pay the miner for submitting olddifficulty shares, and I don't check to send them upstream).  Even so, the fact is difficulty changes are infrequent.  They will only happen when you first turn on your miner.  And only a few times (if any).

Once I track down the current bug on Stratum that caused the last crash, I'll look into adding a small window where old work can be submitted after a diff adjustment.  This really isn't a high priority though since any miner that isn't constantly hopping between pools will rarely see an olddifficulty reject.

But the pool has hundreds (or thousands) of miners.
On PPS this then equates to skimming a tiny amount of the top of everyone.

Yeah each user isn't losing much, but when you multiply that by X hundred or X thousand, it's not 'nothing' on a PPS pool
Thus PPS pools have an incentive to actually change difficulty

Hopping isn't relevant ... and seriously? Who would hop a PPS pool?

Yes ignoring the block solves will probably be worse off for the pool vs's the gain from difficulty changes - but when you get to fixing one make sure you fix them both

Edit: hmm ... is work expiry missing from stratum?

[eleuthria]

Hopping isn't relevant ... and seriously? Who would hop a PPS pool?

My point was unless the miner is constantly leaving the pool, they won't be seeing difficulty adjustments outside of the first few minutes.

[kano]

Hopping isn't relevant ... and seriously? Who would hop a PPS pool?

My point was unless the miner is constantly leaving the pool, they won't be seeing difficulty adjustments outside of the first few minutes.

Hopefully no one has a hash rate near a difficulty change point

[eleuthria]

Hopping isn't relevant ... and seriously? Who would hop a PPS pool?

My point was unless the miner is constantly leaving the pool, they won't be seeing difficulty adjustments outside of the first few minutes.

Hopefully no one has a hash rate near a difficulty change point

If they did it wouldn't matter.  The rate is based on doubling difficulty.  To increase, you have to be above 20 shares per minute (for 5 minutes).  To decrease you have to be under 6 shares per minute (for 5 minutes).  So the theoretical range is 6.1-19.9 shares per minute, though it would require very awkward share spikes to end up outside of 10~19 shares per minute.  Average being 15 SPM.

[slush]

Today I implemented and released to my pool new method called mining.get_transactions(job_id). This call simply dumps transactions used for given job. Thanks to this, miners now have everything needed to reconstruct source block template used by the pool and they can check online if pool isn't doing something nasty.

With this extension, Stratum is still the most optimized mining protocol, but also as open as GBT from the view that miners don't need to trust pool operator that he's not preparing some malicious attack with their hashpower. Anybody is welcome to write some tool for inspecting GBT/Stratum jobs; since now, there's no practical difference in "mining safety" between those two solutions.

Benefits of get_transactions call instead of sending transactions in job broadcast directly are:
* Lower stale ratio for everybody, because pool firstly broadcast just the job itself and *then* send full transactions to interested clients. Miner don't need to wait to full transaction list to start mining on new template.
* Lower bandwidth usage, because only users interested in block inspection will download the list of transactions
* Possibly lower bandwidth usage even for users doing template inspection, because they don't need to perform inspection on every job. Random checks will work as well, because pool never know who and when will ask for transaction list, so cheating is very unlikely.

Please note that sending of transaction list may become a paid feature on some pools, because it requires large amount of pool bandwidth, which is not required for mining itself. Tiny  fee for such additional feature may become an easy prevention against pool DoS.

[slush]

I just added simple tables comparing available Stratum pools and miners to wiki: https://en.bitcoin.it/wiki/Stratum

I've been busy with other Stratum-related things, so I hope that tomorrow my pool will have "dynamic difficulty: yes" in that table!

[DavinciJ15]

From what I have read Stratum wins based on the fact that it's less work and less hardware load for both pools and miners.  This is my opinion based on threads I have read.

Now anyone have a stratum pool back end they are selling?

ckolivas
I am sending you 10.25 BTC for your work on cgminer, I noticed you implemented my request of "sticking to an IP address until it fails" after rejecting it because you thought it was for merge mining.   It was my fault I should have pointed out the benefits such as less DNS lookups.  Good work keep it up.

Davinci

[Keninishna]

From what I have read Stratum wins based on the fact that it's less work and less hardware load for both pools and miners.  This is my opinion based on threads I have read.

Now anyone have a stratum pool back end they are selling?

ckolivas
I am sending you 10.25 BTC for your work on cgminer, I noticed you implemented my request of "sticking to an IP address until it fails" after rejecting it because you thought it was for merge mining.   It was my fault I should have pointed out the benefits such as less DNS lookups.  Good work keep it up.

Davinci

I'd be willing to help an open source implementation of stratum.

[slush]

I'd be willing to help an open source implementation of stratum.

You can improve https://github.com/slush0/stratum or at least you can read how server side works...

[-ck]

From what I have read Stratum wins based on the fact that it's less work and less hardware load for both pools and miners.  This is my opinion based on threads I have read.

Now anyone have a stratum pool back end they are selling?

ckolivas
I am sending you 10.25 BTC for your work on cgminer, I noticed you implemented my request of "sticking to an IP address until it fails" after rejecting it because you thought it was for merge mining.   It was my fault I should have pointed out the benefits such as less DNS lookups.  Good work keep it up.

Davinci

That's a very generous donation and I only implemented unwittingly while implementing stratum, but I gladly accept it 

[Keninishna]

I'd be willing to help an open source implementation of stratum.

You can improve https://github.com/slush0/stratum or at least you can read how server side works...

I've been looking over the python code which works pretty well actually, just need to add DB work.

[DavinciJ15]

I've been looking over the python code which works pretty well actually, just need to add DB work.

I wish I could read python code, it's not structured code, it's not object oriented, it's not even assembly language.  I can't follow control flow, I can't understand variable usage, nothing.  I could learn it but, I don't believe computer languages should be harder to use for all developers at all levels.  This is my opinion and I understand I may be wrong, I believe I'm right.  

So why do you guys use Python?

I'd be willing to help an open source implementation of stratum.

So please virtually any other language except python.
I would pay for it if it's coded in Mono C#.  

That's a very generous donation and I only implemented unwittingly while implementing stratum, but I gladly accept it  

No problem glad to help out, just not sure what you ment did you implement GBT as well when implementing stratum?