Bitcoin Forum
July 14, 2020, 11:56:59 PM *
News: Latest Bitcoin Core release: 0.20.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 ... 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 [106] 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 ... 814 »
  Print  
Author Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool  (Read 2587123 times)
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1007


Gerald Davis


View Profile
April 26, 2012, 01:48:29 PM
 #2101


  • The actual number of "difficulty 1" shares submitted before the block was found.  Note, that since we don't actually know how many "difficulty 1" shares were submitted,

I don't understand that. Sounds contradictious to me?

p2pool doesn't record the # of 1 diff shares anywhere.  twmz is estimating the # of 1 diff shares by looking at avg hashrate during the block.   If there is some inaccuracy in the avg hashrate then it will reflect as inaccurate # of "actual shares".

"actual" is more an estimate.  He is using the word actual to mean actual vs expected.
1594771019
Hero Member
*
Offline Offline

Posts: 1594771019

View Profile Personal Message (Offline)

Ignore
1594771019
Reply with quote  #2

1594771019
Report to moderator
1594771019
Hero Member
*
Offline Offline

Posts: 1594771019

View Profile Personal Message (Offline)

Ignore
1594771019
Reply with quote  #2

1594771019
Report to moderator
1594771019
Hero Member
*
Offline Offline

Posts: 1594771019

View Profile Personal Message (Offline)

Ignore
1594771019
Reply with quote  #2

1594771019
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1007


Gerald Davis


View Profile
April 26, 2012, 04:38:43 PM
Last edit: April 26, 2012, 06:18:50 PM by DeathAndTaxes
 #2102

So I have been taking a close look at p2pool lately (for my own peace of mind if nothing else).

Shouldn't p2pool node report all DOA as rejected back to the miner.

Across my entire farm cgminer shows ~ 1.5% reject rate ("R" in cgminer).  p2pool shows ~2.5% DOA rate.

My assumptions is the logic flow is something like this
1) miner requests work (getwork).
2) p2pool provides work.
3) miner returns low diff shares (I have mine set to a static diff 1).
4) p2pool verifies share
   a) if share is DOA it increments DOA count by one, and sends reject notification to iminer
   b) if share is > share diff and not dead it submits it to all peers for inclusion in share chain and sends accept notification to miner
   c) if share is > block diff it submits it to bitcoin network and sends accept notification to miner

So why doesn't DOA % match reject % in cgminer?
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742
Merit: 500



View Profile WWW
April 26, 2012, 06:02:59 PM
 #2103

So I have been taking a close look at p2pool lately (for my own peace of mind if nothing else).

Shouldn't p2pool node report all DOA as rejected back to the miner.

Across my entire farm cgminer shows ~ 1.5% reject rate ("R" in cgminer).  p2pool shows ~2.5% DOA rate.

My assumptions is the logic flow is something like this
1) miner requests work (getwork).
2) p2pool provides work.
3) miner returns low diff shares (I have mine set to a static diff 1).
4) p2pool verifies share
   a) if share is DOA it increments DOA count by one, and sends reject notification to iminer
   b) if share is < share diff and not dead it submits it to all peers for inclusion in share chain and sends accept notification to miner
   c) if share is < block diff it submits it to bitcoin network and sends accept notification to miner

So why does DOA % match reject % in cgminer?
I think you meant less than, not greater than. I don't know the answer to your question though Sad

DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1007


Gerald Davis


View Profile
April 26, 2012, 06:18:28 PM
 #2104

less than target or greater than difficulty one or the other.
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742
Merit: 500



View Profile WWW
April 26, 2012, 06:23:33 PM
 #2105

less than target or greater than difficulty one or the other.
Ah yes. I see now.

twmz
Hero Member
*****
Offline Offline

Activity: 737
Merit: 500



View Profile
April 27, 2012, 04:17:29 AM
 #2106

Thanks for all that.  If there is an error my guess is it is in the "average hashrate".  It might be better to simply count total shares (and dificulty) received by your node.  

That is exactly how pool hashrate is calculated, as far as I know.  p2pool does the calculation and publishes the hashrate at http://localhost:9332.  I just pull from that value and calculate an average over the duration of a block.

I will take a closer look at how the avg hashrate is calculated.  An error there would mean you are starting with "dirty data".

Agreed.  I have said basically that same thing several times in this thread.  If the hashrate reported by p2pool is too high, our luck will appear artificially lower than it actually is.  Forrest has several times reassured me that the hashrate calculation being done by p2pool is correct, but I have not validated that myself by reviewing code.


A clarification
Code:
"ActualShares":545905
You are calculating this by polling http://localhost:9332/rate periodically? how often?
then getting avg hashrate over the block?
then duration * avg hashrate = # of hashes?
then # of hashes / 2^32 = # of shares?

Yes.  I collect the pool's hashrate from the http://localhost:9332/rate API every 5 minutes.  Then calculate the average hashrate during the time between when blocks are found.  Then calculated # of shares exactly as you indicated (hashrate * duration / 2^32).  The only reason I use diff 1 shares instead of hashes directly is the convenience of using smaller numbers.

Raw hashrate data is here:  http://p2pool.info/stats

Note, hashrate stats are only 1 per hour for the first few months of p2pool's life because that is what forrest had available to backfill my database.  From the time p2pool.info was created forward (sometime in early Feb), hashrate data is in 5 minute increments.  The code that calculates the average hashrate actually calculates a weighted average so as to deal with mixed frequency of hashrate stats within a single block.

Was I helpful?  1TwmzX1wBxNF2qtAJRhdKmi2WyLZ5VHRs
WoT, GPG

Bitrated user: ewal.
forrestv
Hero Member
*****
Offline Offline

Activity: 516
Merit: 514


View Profile
April 27, 2012, 03:54:37 PM
 #2107

So I have been taking a close look at p2pool lately (for my own peace of mind if nothing else).

Shouldn't p2pool node report all DOA as rejected back to the miner.

Across my entire farm cgminer shows ~ 1.5% reject rate ("R" in cgminer).  p2pool shows ~2.5% DOA rate.

My assumptions is the logic flow is something like this
1) miner requests work (getwork).
2) p2pool provides work.
3) miner returns low diff shares (I have mine set to a static diff 1).
4) p2pool verifies share
   a) if share is DOA it increments DOA count by one, and sends reject notification to iminer
   b) if share is > share diff and not dead it submits it to all peers for inclusion in share chain and sends accept notification to miner
   c) if share is > block diff it submits it to bitcoin network and sends accept notification to miner

So why doesn't DOA % match reject % in cgminer?

You do mean the dead on arrival pseudoshare rate, not the dead share rate, right? The dead on arrival rate should match cgminer's, but the one displayed on the console is only averaged over 10 minutes. Did you look at the graphs to get a more accurate rate? (area of local dead/area of local total)

1J1zegkNSbwX4smvTdoHSanUfwvXFeuV23
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1007


Gerald Davis


View Profile
April 27, 2012, 04:16:59 PM
 #2108

You do mean the dead on arrival pseudoshare rate, not the dead share rate, right?

By psueodshare I assume you mean the low diff share cgminer thinks it is mining?
Which DOA stat is for the diff 1 shares and which is for "full sharechain difficult shares?

DOA is reported in 3 ? places
1) In graphs both total and per miner
2) On the /static page
Code:
Local rate: 11.6GH/s (2.0% DOA)
3) In the console window (which I find myself using less and less).

It may have just been a timing issue.  I think this weekend I will restart p2pool w/ no historical stats and restart all miner instances to get a "baseline".

I am also trying to figure out an easy way to convert JSON file into relational db tables so I can do some deeper digging.

My hunch (and I would like to stress completely unfounded) is that there is some error in the p2pool.info stats.

I think this error comes from the double estimations.

The raw data is shares seen by p2pool node.
p2pool abstracts that into global hashrate averaged over 10? minutes
p2pool poools that hashrate every 5 min?
p2pool uses that to construct avg hashrate over the block
p2pool use that and block time to determine # of hashes in block
p2pool compares # of hashes in block vs expected # of hashes in block to get "luck"

I am wondering if there is a better way.  simply look at all the shares since last block.
diff 1 shares for block = sum of difficulty of shares seen by node since last block
right? 
twmz
Hero Member
*****
Offline Offline

Activity: 737
Merit: 500



View Profile
April 27, 2012, 05:27:48 PM
 #2109

I am wondering if there is a better way.  simply look at all the shares since last block.
diff 1 shares for block = sum of difficulty of shares seen by node since last block
right?  

I don't fully understand the internals of the share chain distribution, but I think there are a couple problems that would have to be solved:

  • We have a significant number of blocks where the duration was longer than the length of the share chain.  So in order to add up shares after a block is found, I would have to invent my own persistence of the the share chain that never got cleaned up.  Not trivial or fun (and fun = motivation to do it)
  • The "hashes done by the pool" include hashes contained in stale shares that are not fully distributed through the p2p network.  Because those stale shares mostly could have been valid blocks, they need to be included in the luck calculation.  In fact a lot of our blocks actually were stale shares (as determined by the fact that they often aren't announced in IRC).  The current p2pool hashrate calculation handles this by counting the valid shares and then increasing the count based on the pool-wide stale rate.

Was I helpful?  1TwmzX1wBxNF2qtAJRhdKmi2WyLZ5VHRs
WoT, GPG

Bitrated user: ewal.
twmz
Hero Member
*****
Offline Offline

Activity: 737
Merit: 500



View Profile
April 27, 2012, 05:31:02 PM
 #2110

I am wondering if there is a better way.  simply look at all the shares since last block.
diff 1 shares for block = sum of difficulty of shares seen by node since last block
right?  

I don't fully understand the internals of the share chain distribution, but I think there are a couple problems that would have to be solved:

  • We have a significant number of blocks where the duration was longer than the length of the share chain.  So in order to add up shares after a block is found, I would have to invent my own persistence of the the share chain that never got cleaned up.  Not trivial or fun (and fun = motivation to do it)
  • The "hashes done by the pool" include hashes contained in stale shares that are not fully distributed through the p2p network.  Because those stale shares mostly could have been valid blocks, they need to be included in the luck calculation.  In fact a lot of our blocks actually were stale shares (as determined by the fact that they often aren't announced in IRC).  The current p2pool hashrate calculation handles this by counting the valid shares and then increasing the count based on the pool-wide stale rate.

It would be an interesting experiment (and much easier because you can ignore the first problem by picking blocks that are shorter than 18 or so hours) to spot check.  If you want to calculate how many hashes you think went into a few blocks, I can tell you how many hashes p2pool.info thinks went into it and we can see how much variation there is in the two methods.

Was I helpful?  1TwmzX1wBxNF2qtAJRhdKmi2WyLZ5VHRs
WoT, GPG

Bitrated user: ewal.
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742
Merit: 500



View Profile WWW
April 27, 2012, 05:55:44 PM
 #2111

When do the rest of us get this page? This looks nice. http://p2pool.hopto.org:9332/static/nodes.html

DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1007


Gerald Davis


View Profile
April 27, 2012, 05:56:09 PM
 #2112

I hadn't thought about the fact that blocks can be longer than sharechain. I see a couple work arounds
a) quick & dirty for analysis I could simply cron job to archive the sharechain at quicker interval than shares fall off the end and recombine it to get a continual history

b) slightly better would be looking into the code to make p2poool keep shares around longer and/or make it a command line param

As far as orphaned (but not DOA) shares I need to look at the code on propogation.  If they aren't fully distributed then how does a node calculate hashrate & stale (better called orphan) rate?  

Quote
The current p2pool hashrate calculation handles this by counting the valid shares and then increasing the count based on the pool-wide stale rate.

So following that train of thought how does an arbitrary node determined the stale (orphan) rate for the network?  My node right now says orphan rate is 8.9%.  How does it know?  It must be notified of all orphaned shares (either by receiving shares and incrementing its local orphan count or by some orphan share message).  My understanding is things like hashrate, orphan %, etc all all calculated locally.  We just need to tap into this.

BTW I am not expecting you to do any of this based on my hunch.  This weekend I will dive into the code and see if we can find a method to count shares (well technically sum of share difficulty).  If I can and it diverges from p2pool.info stats then we can look at using that to improve accuracy of the stats.  If it doesn't well then I was the only one who wasted his time. Smiley
cabin
Sr. Member
****
Offline Offline

Activity: 606
Merit: 250


View Profile
April 27, 2012, 10:49:16 PM
 #2113

I hadn't thought about the fact that blocks can be longer than sharechain. I see a couple work arounds
a) quick & dirty for analysis I could simply cron job to archive the sharechain at quicker interval than shares fall off the end and recombine it to get a continual history

b) slightly better would be looking into the code to make p2poool keep shares around longer and/or make it a command line param

As far as orphaned (but not DOA) shares I need to look at the code on propogation.  If they aren't fully distributed then how does a node calculate hashrate & stale (better called orphan) rate?  

Quote
The current p2pool hashrate calculation handles this by counting the valid shares and then increasing the count based on the pool-wide stale rate.

So following that train of thought how does an arbitrary node determined the stale (orphan) rate for the network?  My node right now says orphan rate is 8.9%.  How does it know?  It must be notified of all orphaned shares (either by receiving shares and incrementing its local orphan count or by some orphan share message).  My understanding is things like hashrate, orphan %, etc all all calculated locally.  We just need to tap into this.

BTW I am not expecting you to do any of this based on my hunch.  This weekend I will dive into the code and see if we can find a method to count shares (well technically sum of share difficulty).  If I can and it diverges from p2pool.info stats then we can look at using that to improve accuracy of the stats.  If it doesn't well then I was the only one who wasted his time. Smiley

Pages like this record tons of info about each share, and it is recording atleast some deads:
http://p2pool.hopto.org:9332/static/share.html#0000000000555d58d25a50651e47ec138edd63f4d7dfa868a9b679e160c79524
cabin
Sr. Member
****
Offline Offline

Activity: 606
Merit: 250


View Profile
April 27, 2012, 10:54:46 PM
 #2114

When do the rest of us get this page? This looks nice. http://p2pool.hopto.org:9332/static/nodes.html

You can save the html page to your own web-static/ folder and then you'll have it too Smiley

The ping times will be unique to where you are and the pools are hard coded from ones people wanted to be public. It is possible to grab a list of all your connected peers but maybe not everyone would appreciate that.
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742
Merit: 500



View Profile WWW
April 27, 2012, 11:15:26 PM
 #2115

When do the rest of us get this page? This looks nice. http://p2pool.hopto.org:9332/static/nodes.html

You can save the html page to your own web-static/ folder and then you'll have it too Smiley

The ping times will be unique to where you are and the pools are hard coded from ones people wanted to be public. It is possible to grab a list of all your connected peers but maybe not everyone would appreciate that.
Cool thanks.

Seems like hopto's ping time to my node is consistently less than the ping time to hopto. I guess I have a good network?

Smoovious
Hero Member
*****
Offline Offline

Activity: 504
Merit: 500

Scattering my bits around the net since 1980


View Profile
April 28, 2012, 01:51:15 AM
 #2116

is it possible to show what version of p2pool the node is running, or is that not available?

also... woohoo! lowest ping time on your list! Cheesy

138ms! Cheesy

-- Smoov
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742
Merit: 500



View Profile WWW
April 28, 2012, 06:03:14 AM
 #2117

is it possible to show what version of p2pool the node is running, or is that not available?

also... woohoo! lowest ping time on your list! Cheesy

138ms! Cheesy

-- Smoov

I'm starting to think those are ping times from the client to the p2pool nodes, not the node to the other nodes. Loading the page from the miner would be most accurate.

kano
Legendary
*
Offline Offline

Activity: 3192
Merit: 1286


Linux since 1997 RedHat 4


View Profile
April 28, 2012, 06:36:14 AM
 #2118

Meh - I somehow managed to post this in the BFL thread when I mean to post it here ...
OK fixing that, here it is here where it should be.

... and now I'll ask an evil question coz I'm curious about it ...

I added the code that did network block detection in cgminer.
It reports "BLOCK!" on BOTH Accepted and Rejected shares.
I've wondered in the past if anyone has ever had a Rejected "BLOCK!"
i.e. you found a block, but you were JUST unlucky enough to find it in the time frame where it would be Rejected
( of course people have, but they'd need to keep their logs and check them ... I do keep all my logs and sometimes check them Cheesy )

They should be rare but also easy to calculate how often you should get them ...
The Rejection rate of your shares is a percentage.
That Rejection rate is also the % of blocks you find that should also be a Rejected "BLOCK!"


... now for a code change suggestion ...

So if you get 10% of your shares rejected then 10% of the blocks you find should also be rejected ....................
However ........... of that 10% that get rejected, 98.3% of them would be accepted by the bitcoin block chain but are invalid for the share chain and are thus thrown away .............

(I also wonder if this is a factor in the luck discussion - but I've no idea if it is)

Yeah I think there should be an exception in the code to allow rejected shares, that match this circumstance, and enable support for the exception in the share chain
i.e. if you get a Rejected BLOCK! it should make it all the way to the bitcoind and then it's up to bitcoind to reject it if it was a real block chain LP (not a P2Pool share chain LP)
Of course if this was implemented: it would also mean that everyone mining with cgminer on P2Pool should enable submit stale
(and also P2Pool should tell all miners to submit stale shares)

Pool: https://kano.is - lowest fee PPLNS 3 Days Here on Bitcointalk: Forum
Discord support invite at https://kano.is/ Majority developer of the ckpool code
Help keep Bitcoin secure by mining on pools with full block verification on all blocks - and NO empty blocks!
cabin
Sr. Member
****
Offline Offline

Activity: 606
Merit: 250


View Profile
April 28, 2012, 12:03:29 PM
 #2119

is it possible to show what version of p2pool the node is running, or is that not available?

also... woohoo! lowest ping time on your list! Cheesy

138ms! Cheesy

-- Smoov

I'm starting to think those are ping times from the client to the p2pool nodes, not the node to the other nodes. Loading the page from the miner would be most accurate.

Yes, it is always the ping from your current computer to the node. Everyone will get different numbers and if your node is on your machine or on the same network your ping will be very low. In effect it shows you the node that will be best for you and you alone to connect to.

The version number isn't in the data below so I don't display it.
http://p2pool.hopto.org:9332/local_stats
cabin
Sr. Member
****
Offline Offline

Activity: 606
Merit: 250


View Profile
April 28, 2012, 12:15:48 PM
 #2120

Meh - I somehow managed to post this in the BFL thread when I mean to post it here ...
OK fixing that, here it is here where it should be.

... and now I'll ask an evil question coz I'm curious about it ...

I added the code that did network block detection in cgminer.
It reports "BLOCK!" on BOTH Accepted and Rejected shares.
I've wondered in the past if anyone has ever had a Rejected "BLOCK!"
i.e. you found a block, but you were JUST unlucky enough to find it in the time frame where it would be Rejected
( of course people have, but they'd need to keep their logs and check them ... I do keep all my logs and sometimes check them Cheesy )

They should be rare but also easy to calculate how often you should get them ...
The Rejection rate of your shares is a percentage.
That Rejection rate is also the % of blocks you find that should also be a Rejected "BLOCK!"


... now for a code change suggestion ...

So if you get 10% of your shares rejected then 10% of the blocks you find should also be rejected ....................
However ........... of that 10% that get rejected, 98.3% of them would be accepted by the bitcoin block chain but are invalid for the share chain and are thus thrown away .............

(I also wonder if this is a factor in the luck discussion - but I've no idea if it is)

Yeah I think there should be an exception in the code to allow rejected shares, that match this circumstance, and enable support for the exception in the share chain
i.e. if you get a Rejected BLOCK! it should make it all the way to the bitcoind and then it's up to bitcoind to reject it if it was a real block chain LP (not a P2Pool share chain LP)
Of course if this was implemented: it would also mean that everyone mining with cgminer on P2Pool should enable submit stale
(and also P2Pool should tell all miners to submit stale shares)

If a share is DOA or orphan it will still count as a real block. If you look at the source in main.got_response(header, request) the first thing it does is check if it is a real block. Only after that does it bother to check for DOA. CGminer will submit dead shares if you use --submit_stale, or if you are running a fairly recent version it will do this automatically. If you are using an old version of cgminer, and not using --submit_stale, then you could discard a valid block before it even reaches P2Pool.
Pages: « 1 ... 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 [106] 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 ... 814 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!