Bitcoin Forum
November 25, 2017, 10:04:32 AM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 [12] 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 »
  Print  
Author Topic: [ANN] Stratum mining protocol - ASIC ready  (Read 143701 times)
dlasher
Sr. Member
****
Offline Offline

Activity: 417



View Profile WWW
October 19, 2012, 09:40:14 PM
 #221

I ran into an interesting failure state today, and the stratum proxy, I believe, played a roll in the failure. I posted over in the cgminer thread about the cgminer portion, but wanted to post here about the proxy portion.

Shouldn't the proxy, in some way, tell the clients when it's primary upstream connection dies?

Yes, it should. Let me ask some questions:

a) Are you using latest Windows binary?
b) If you're using Linux, are you sure you ran "setup.py install" or "setup.py develop" after last upgrade?
c) Do you see any error/traceback in the proxy during connection outage?

Proxy should propagate all connection failures to all connected miners immediately. It is implemented in method on_disconnect (https://github.com/slush0/stratum-mining-proxy/blob/master/mining_proxy.py) and I don't see any reason why it shouldn't work, except ther was another error.

A: No. cgminer 2.8.3 (at the time) linux, proxy 1.1.1 linux.
B: yes, and yes
C: no, proxy just sat.. idle.

primary pool hasn't crashed again since, so haven't had the chance to see again.. but I'll let you know if it happens.

1511604272
Hero Member
*
Offline Offline

Posts: 1511604272

View Profile Personal Message (Offline)

Ignore
1511604272
Reply with quote  #2

1511604272
Report to moderator
1511604272
Hero Member
*
Offline Offline

Posts: 1511604272

View Profile Personal Message (Offline)

Ignore
1511604272
Reply with quote  #2

1511604272
Report to moderator
1511604272
Hero Member
*
Offline Offline

Posts: 1511604272

View Profile Personal Message (Offline)

Ignore
1511604272
Reply with quote  #2

1511604272
Report to moderator
Join ICO Now Coinlancer is Disrupting the Freelance marketplace!
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
October 20, 2012, 09:23:49 PM
 #222

Maybe this is a possible future feature of cgminer, being a proxy itself.

Proxy already support "mesh networking" feature; it acts as a UDP multicast responder and it is able to report to which pool it is connected and on which port is proxy listening.

My idea for the future is that every miner on startup send UDP multicast request to the network and get a list of existing proxies on local network. It pick these connected to desired pool and talk to them instead of using direct connection to that pool. Next level is that miner, which don't found alive proxy for his pool, became a proxy itself and start responding for UDP requests for other miners.

I don't know how difficult would be to implement proxying feature directly to cgminer, but probing local network for listening proxies is pretty trivial and it can be the first step: https://github.com/slush0/stratum-mining-proxy/blob/master/example_multicast.py (You can use this script to get a list of alive proxies on the network.)

slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
October 20, 2012, 09:24:56 PM
 #223

primary pool hasn't crashed again since, so haven't had the chance to see again.. but I'll let you know if it happens.

I think that I found some possible race condition which can trigger your issue. It is more theoretical than practical chance, but definitely worth of fixing. I'll release bugfix tomorrow.

kano
Legendary
*
Offline Offline

Activity: 2282


Linux since 1997 RedHat 4


View Profile
October 20, 2012, 10:28:31 PM
 #224

So ... after effectively being told I was wrong by slush ... Tongue

... and not using stratum since the 2 pools were not in my list of pools that I would consider using ...

I added a stratum pool to my list (of 13 - so now 14) pools and pool duplicates today
I added BTCGuild to help with testing the code.

And the very first difficulty switch I got:

 [2012-10-21 09:11:09] Accepted fbab8535 Diff 1/1 ICA 0 pool 2
 [2012-10-21 09:11:16] Accepted 85774b9a Diff 1/1 GPU 1 pool 2
 [2012-10-21 09:11:42] Stratum from pool 2 requested work restart
 [2012-10-21 09:11:42] Rejected 3b789247 Diff 4/2 ICA 1 pool 2 (olddifficulty)
 [2012-10-21 09:12:03] Accepted 77567e1c Diff 2/2 GPU 1 pool 2
 [2012-10-21 09:12:24] Accepted 5326e93f Diff 3/2 GPU 1 pool 2

Lost me a share ... but looking at it I don't even see why it would since it was higher difficulty than the required target.

Of course if it had been lower difficulty, then I would have lost it due to the argument that slush said was invalid.
But I lost it when it was higher difficulty so I guess that's a problem with BTCGuild?

Pool: https://kano.is Here on Bitcointalk: Forum BTC: 1KanoPb8cKYqNrswjaA8cRDk4FAS9eDMLU
FreeNode IRC: irc.freenode.net channel #kano.is Majority developer of the ckpool code
Help keep Bitcoin secure by mining on pools with full block verification on all blocks - and NO empty blocks!
eleuthria
Legendary
*
Offline Offline

Activity: 1750



View Profile
October 20, 2012, 10:46:47 PM
 #225

So ... after effectively being told I was wrong by slush ... Tongue

... and not using stratum since the 2 pools were not in my list of pools that I would consider using ...

I added a stratum pool to my list (of 13 - so now 14) pools and pool duplicates today
I added BTCGuild to help with testing the code.

And the very first difficulty switch I got:

 [2012-10-21 09:11:09] Accepted fbab8535 Diff 1/1 ICA 0 pool 2
 [2012-10-21 09:11:16] Accepted 85774b9a Diff 1/1 GPU 1 pool 2
 [2012-10-21 09:11:42] Stratum from pool 2 requested work restart
 [2012-10-21 09:11:42] Rejected 3b789247 Diff 4/2 ICA 1 pool 2 (olddifficulty)
 [2012-10-21 09:12:03] Accepted 77567e1c Diff 2/2 GPU 1 pool 2
 [2012-10-21 09:12:24] Accepted 5326e93f Diff 3/2 GPU 1 pool 2

Lost me a share ... but looking at it I don't even see why it would since it was higher difficulty than the required target.

Of course if it had been lower difficulty, then I would have lost it due to the argument that slush said was invalid.
But I lost it when it was higher difficulty so I guess that's a problem with BTCGuild?

BTC Guild forces you to submit new work when updating difficulty currently.  It won't accept work from previous work when you're adjusted because it keeps track of the miner's difficulty separate from the work sent to the miner.  With the current setup, if I accepted old work, you'd get credited at the new difficulty.  This opens up an exploit where a very fast miner can purposely withhold higher diff shares at first, knowing that they are going to force multiple difficulty increases in a short time, then submit them all for higher credit than what they should have been earning.

While it could be done in another way (keeping a list of difficulties for the miner as of each WorkID), it would add a lot of extra code for effectively no gain.  We're talking maybe a few pennies over the course of a year, if that.  Difficulty adjustments are a rare occurrence.  They only happen when you first start mining over the connection.  That share you "lost" from the diff adjustment was 0.0001 dollars (1/100th of a penny).

RIP BTC Guild, April 2011 - June 2015
-ck
Moderator
Legendary
*
Offline Offline

Activity: 2352


Ruu \o/


View Profile WWW
October 20, 2012, 10:55:10 PM
 #226

Hopefully you don't make the mistake of rejecting block solves then just because they happen across difficulty changes Wink

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
kano
Legendary
*
Offline Offline

Activity: 2282


Linux since 1997 RedHat 4


View Profile
October 20, 2012, 11:29:24 PM
 #227

So ... after effectively being told I was wrong by slush ... Tongue

... and not using stratum since the 2 pools were not in my list of pools that I would consider using ...

I added a stratum pool to my list (of 13 - so now 14) pools and pool duplicates today
I added BTCGuild to help with testing the code.

And the very first difficulty switch I got:

 [2012-10-21 09:11:09] Accepted fbab8535 Diff 1/1 ICA 0 pool 2
 [2012-10-21 09:11:16] Accepted 85774b9a Diff 1/1 GPU 1 pool 2
 [2012-10-21 09:11:42] Stratum from pool 2 requested work restart
 [2012-10-21 09:11:42] Rejected 3b789247 Diff 4/2 ICA 1 pool 2 (olddifficulty)
 [2012-10-21 09:12:03] Accepted 77567e1c Diff 2/2 GPU 1 pool 2
 [2012-10-21 09:12:24] Accepted 5326e93f Diff 3/2 GPU 1 pool 2

Lost me a share ... but looking at it I don't even see why it would since it was higher difficulty than the required target.

Of course if it had been lower difficulty, then I would have lost it due to the argument that slush said was invalid.
But I lost it when it was higher difficulty so I guess that's a problem with BTCGuild?

BTC Guild forces you to submit new work when updating difficulty currently.  It won't accept work from previous work when you're adjusted because it keeps track of the miner's difficulty separate from the work sent to the miner.  With the current setup, if I accepted old work, you'd get credited at the new difficulty.  This opens up an exploit where a very fast miner can purposely withhold higher diff shares at first, knowing that they are going to force multiple difficulty increases in a short time, then submit them all for higher credit than what they should have been earning.

While it could be done in another way (keeping a list of difficulties for the miner as of each WorkID), it would add a lot of extra code for effectively no gain.  We're talking maybe a few pennies over the course of a year, if that.  Difficulty adjustments are a rare occurrence.  They only happen when you first start mining over the connection.  That share you "lost" from the diff adjustment was 0.0001 dollars (1/100th of a penny).
Ignoring all the other arguments I have already made about this issue ... specifically pointing out the expected % of shares that will fall in this category that may be of lower difficulty ... but you've just explained that it is ALL shares of any difficulty (and as ckolivas has pointed out - it will also include valid block solves that you will be rejecting) ...

... this also directly means on a PPS pool, the more diff changes you send, the more money the pool saves.

Pool: https://kano.is Here on Bitcointalk: Forum BTC: 1KanoPb8cKYqNrswjaA8cRDk4FAS9eDMLU
FreeNode IRC: irc.freenode.net channel #kano.is Majority developer of the ckpool code
Help keep Bitcoin secure by mining on pools with full block verification on all blocks - and NO empty blocks!
eleuthria
Legendary
*
Offline Offline

Activity: 1750



View Profile
October 21, 2012, 01:21:42 AM
 #228

Ignoring all the other arguments I have already made about this issue ... specifically pointing out the expected % of shares that will fall in this category that may be of lower difficulty ... but you've just explained that it is ALL shares of any difficulty (and as ckolivas has pointed out - it will also include valid block solves that you will be rejecting) ...

... this also directly means on a PPS pool, the more diff changes you send, the more money the pool saves.

Except I don't check shares that are rejected as block solves, so it's not saving money, it's a break even (I don't pay the miner for submitting olddifficulty shares, and I don't check to send them upstream).  Even so, the fact is difficulty changes are infrequent.  They will only happen when you first turn on your miner.  And only a few times (if any).

Once I track down the current bug on Stratum that caused the last crash, I'll look into adding a small window where old work can be submitted after a diff adjustment.  This really isn't a high priority though since any miner that isn't constantly hopping between pools will rarely see an olddifficulty reject.

RIP BTC Guild, April 2011 - June 2015
kano
Legendary
*
Offline Offline

Activity: 2282


Linux since 1997 RedHat 4


View Profile
October 21, 2012, 03:13:11 AM
 #229

Ignoring all the other arguments I have already made about this issue ... specifically pointing out the expected % of shares that will fall in this category that may be of lower difficulty ... but you've just explained that it is ALL shares of any difficulty (and as ckolivas has pointed out - it will also include valid block solves that you will be rejecting) ...

... this also directly means on a PPS pool, the more diff changes you send, the more money the pool saves.

Except I don't check shares that are rejected as block solves, so it's not saving money, it's a break even (I don't pay the miner for submitting olddifficulty shares, and I don't check to send them upstream).  Even so, the fact is difficulty changes are infrequent.  They will only happen when you first turn on your miner.  And only a few times (if any).

Once I track down the current bug on Stratum that caused the last crash, I'll look into adding a small window where old work can be submitted after a diff adjustment.  This really isn't a high priority though since any miner that isn't constantly hopping between pools will rarely see an olddifficulty reject.
But the pool has hundreds (or thousands) of miners.
On PPS this then equates to skimming a tiny amount of the top of everyone.

Yeah each user isn't losing much, but when you multiply that by X hundred or X thousand, it's not 'nothing' on a PPS pool
Thus PPS pools have an incentive to actually change difficulty

Hopping isn't relevant ... and seriously? Who would hop a PPS pool?

Yes ignoring the block solves will probably be worse off for the pool vs's the gain from difficulty changes - but when you get to fixing one make sure you fix them both Smiley

Edit: hmm ... is work expiry missing from stratum?

Pool: https://kano.is Here on Bitcointalk: Forum BTC: 1KanoPb8cKYqNrswjaA8cRDk4FAS9eDMLU
FreeNode IRC: irc.freenode.net channel #kano.is Majority developer of the ckpool code
Help keep Bitcoin secure by mining on pools with full block verification on all blocks - and NO empty blocks!
eleuthria
Legendary
*
Offline Offline

Activity: 1750



View Profile
October 21, 2012, 03:24:42 AM
 #230

Hopping isn't relevant ... and seriously? Who would hop a PPS pool?

My point was unless the miner is constantly leaving the pool, they won't be seeing difficulty adjustments outside of the first few minutes.

RIP BTC Guild, April 2011 - June 2015
kano
Legendary
*
Offline Offline

Activity: 2282


Linux since 1997 RedHat 4


View Profile
October 21, 2012, 03:34:43 AM
 #231

Hopping isn't relevant ... and seriously? Who would hop a PPS pool?

My point was unless the miner is constantly leaving the pool, they won't be seeing difficulty adjustments outside of the first few minutes.
Hopefully no one has a hash rate near a difficulty change point Smiley

Pool: https://kano.is Here on Bitcointalk: Forum BTC: 1KanoPb8cKYqNrswjaA8cRDk4FAS9eDMLU
FreeNode IRC: irc.freenode.net channel #kano.is Majority developer of the ckpool code
Help keep Bitcoin secure by mining on pools with full block verification on all blocks - and NO empty blocks!
eleuthria
Legendary
*
Offline Offline

Activity: 1750



View Profile
October 21, 2012, 03:38:09 AM
 #232

Hopping isn't relevant ... and seriously? Who would hop a PPS pool?

My point was unless the miner is constantly leaving the pool, they won't be seeing difficulty adjustments outside of the first few minutes.
Hopefully no one has a hash rate near a difficulty change point Smiley

If they did it wouldn't matter.  The rate is based on doubling difficulty.  To increase, you have to be above 20 shares per minute (for 5 minutes).  To decrease you have to be under 6 shares per minute (for 5 minutes).  So the theoretical range is 6.1-19.9 shares per minute, though it would require very awkward share spikes to end up outside of 10~19 shares per minute.  Average being 15 SPM.

RIP BTC Guild, April 2011 - June 2015
slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
October 21, 2012, 01:11:15 PM
 #233

Today I implemented and released to my pool new method called mining.get_transactions(job_id). This call simply dumps transactions used for given job. Thanks to this, miners now have everything needed to reconstruct source block template used by the pool and they can check online if pool isn't doing something nasty.

With this extension, Stratum is still the most optimized mining protocol, but also as open as GBT from the view that miners don't need to trust pool operator that he's not preparing some malicious attack with their hashpower. Anybody is welcome to write some tool for inspecting GBT/Stratum jobs; since now, there's no practical difference in "mining safety" between those two solutions.

Benefits of get_transactions call instead of sending transactions in job broadcast directly are:
* Lower stale ratio for everybody, because pool firstly broadcast just the job itself and *then* send full transactions to interested clients. Miner don't need to wait to full transaction list to start mining on new template.
* Lower bandwidth usage, because only users interested in block inspection will download the list of transactions
* Possibly lower bandwidth usage even for users doing template inspection, because they don't need to perform inspection on every job. Random checks will work as well, because pool never know who and when will ask for transaction list, so cheating is very unlikely.

Please note that sending of transaction list may become a paid feature on some pools, because it requires large amount of pool bandwidth, which is not required for mining itself. Tiny  fee for such additional feature may become an easy prevention against pool DoS.

slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
October 21, 2012, 11:11:56 PM
 #234

I just added simple tables comparing available Stratum pools and miners to wiki: https://en.bitcoin.it/wiki/Stratum

I've been busy with other Stratum-related things, so I hope that tomorrow my pool will have "dynamic difficulty: yes" in that table!

DavinciJ15
Hero Member
*****
Offline Offline

Activity: 736


Bitcoin - helping to end bankster enslavement.


View Profile WWW
October 25, 2012, 03:05:18 PM
 #235

From what I have read Stratum wins based on the fact that it's less work and less hardware load for both pools and miners.  This is my opinion based on threads I have read.

Now anyone have a stratum pool back end they are selling? Smiley

ckolivas
I am sending you 10.25 BTC for your work on cgminer, I noticed you implemented my request of "sticking to an IP address until it fails" after rejecting it because you thought it was for merge mining. Smiley  It was my fault I should have pointed out the benefits such as less DNS lookups.  Good work keep it up.

Davinci
Keninishna
Hero Member
*****
Offline Offline

Activity: 555



View Profile WWW
October 25, 2012, 03:45:04 PM
 #236

From what I have read Stratum wins based on the fact that it's less work and less hardware load for both pools and miners.  This is my opinion based on threads I have read.

Now anyone have a stratum pool back end they are selling? Smiley

ckolivas
I am sending you 10.25 BTC for your work on cgminer, I noticed you implemented my request of "sticking to an IP address until it fails" after rejecting it because you thought it was for merge mining. Smiley  It was my fault I should have pointed out the benefits such as less DNS lookups.  Good work keep it up.

Davinci

I'd be willing to help an open source implementation of stratum.
slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
October 25, 2012, 08:55:42 PM
 #237

I'd be willing to help an open source implementation of stratum.

You can improve https://github.com/slush0/stratum or at least you can read how server side works...

-ck
Moderator
Legendary
*
Offline Offline

Activity: 2352


Ruu \o/


View Profile WWW
October 25, 2012, 09:26:33 PM
 #238

From what I have read Stratum wins based on the fact that it's less work and less hardware load for both pools and miners.  This is my opinion based on threads I have read.

Now anyone have a stratum pool back end they are selling? Smiley

ckolivas
I am sending you 10.25 BTC for your work on cgminer, I noticed you implemented my request of "sticking to an IP address until it fails" after rejecting it because you thought it was for merge mining. Smiley  It was my fault I should have pointed out the benefits such as less DNS lookups.  Good work keep it up.

Davinci
That's a very generous donation and I only implemented unwittingly while implementing stratum, but I gladly accept it  Wink

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
Keninishna
Hero Member
*****
Offline Offline

Activity: 555



View Profile WWW
October 25, 2012, 11:23:06 PM
 #239

I'd be willing to help an open source implementation of stratum.

You can improve https://github.com/slush0/stratum or at least you can read how server side works...

I've been looking over the python code Smiley which works pretty well actually, just need to add DB work.
DavinciJ15
Hero Member
*****
Offline Offline

Activity: 736


Bitcoin - helping to end bankster enslavement.


View Profile WWW
October 26, 2012, 11:52:07 AM
 #240

I've been looking over the python code Smiley which works pretty well actually, just need to add DB work.

I wish I could read python code, it's not structured code, it's not object oriented, it's not even assembly language.  I can't follow control flow, I can't understand variable usage, nothing.  I could learn it but, I don't believe computer languages should be harder to use for all developers at all levels.  This is my opinion and I understand I may be wrong, I believe I'm right.  Tongue

So why do you guys use Python?

I'd be willing to help an open source implementation of stratum.

So please virtually any other language except python. Smiley
I would pay for it if it's coded in Mono C#.  Smiley

That's a very generous donation and I only implemented unwittingly while implementing stratum, but I gladly accept it  Wink

No problem glad to help out, just not sure what you ment did you implement GBT as well when implementing stratum?
Pages: « 1 2 3 4 5 6 7 8 9 10 11 [12] 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!