Bitcoin Forum
December 06, 2016, 12:35:34 PM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 ... 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 [106] 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 ... 744 »
  Print  
Author Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool  (Read 2031555 times)
organofcorti
Donator
Legendary
*
Offline Offline

Activity: 1946


Poor impulse control.


View Profile WWW
April 26, 2012, 12:58:42 PM
 #2101

Either way, both 'actual' and 'estimated' shares should be geometrically distributed, right?

Bitcoin network and pool analysis 12QxPHEuxDrs7mCyGSx1iVSozTwtquDB3r
follow @oocBlog for new post notifications
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 26, 2012, 01:42:27 PM
 #2102

@D&T:

The '% of expected' round 'lengths' published on p2pool.info are uniformly distributed, and have a very regular mean of about 160% (p-value for the linear model is less than 10^-12). If the '% of expected' works the way I think it does (normalises round length to D) then I'd expect '% of expected' to be geometrically distributed and the average to be 100%.

I'm think I'm not getting something about the way the '%luck' is calculated?

Raw data is here:  http://p2pool.info/blocks

For each block, you can see:

  • The actual number of "difficulty 1" shares submitted before the block was found.  Note, that since we don't actually know how many "difficulty 1" shares were submitted, this is an estimate of how many difficulty 1 shares should have been submitted based on the average hashrate at the time and the duration of the round.  And so if the hashrate published by http://localhost:9332/rate is wrong, then this number of shares will also be wrong.
  • The estimated number of "difficulty 1" shares theoretically needed based on the bitcoin difficulty at the time

% expected for a single block is:  actual shares / expected shares

% luck over a 30 day window is (sum of expected shares for all blocks found within 30 days) / (sum of all actual shares for all same blocks)

Thanks for all that.  If there is an error my guess is it is in the "average hashrate".  It might be better to simply count total shares (and dificulty) received by your node.  

I will take a closer look at how the avg hashrate is calculated.  An error there would mean you are starting with "dirty data".

Note: I am not saying there IS an error just that given the 15% divergence we should take a closer look.  Your computation "downstream" from the avg hashrate computation looks valid.

On edit:
A clarification
Code:
"ActualShares":545905
You are calculating this by polling http://localhost:9332/rate periodically? how often?
then getting avg hashrate over the block?
then duration * avg hashrate = # of hashes?
then # of hashes / 2^32 = # of shares?
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 26, 2012, 01:48:29 PM
 #2103


  • The actual number of "difficulty 1" shares submitted before the block was found.  Note, that since we don't actually know how many "difficulty 1" shares were submitted,

I don't understand that. Sounds contradictious to me?

p2pool doesn't record the # of 1 diff shares anywhere.  twmz is estimating the # of 1 diff shares by looking at avg hashrate during the block.   If there is some inaccuracy in the avg hashrate then it will reflect as inaccurate # of "actual shares".

"actual" is more an estimate.  He is using the word actual to mean actual vs expected.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 26, 2012, 04:38:43 PM
 #2104

So I have been taking a close look at p2pool lately (for my own peace of mind if nothing else).

Shouldn't p2pool node report all DOA as rejected back to the miner.

Across my entire farm cgminer shows ~ 1.5% reject rate ("R" in cgminer).  p2pool shows ~2.5% DOA rate.

My assumptions is the logic flow is something like this
1) miner requests work (getwork).
2) p2pool provides work.
3) miner returns low diff shares (I have mine set to a static diff 1).
4) p2pool verifies share
   a) if share is DOA it increments DOA count by one, and sends reject notification to iminer
   b) if share is > share diff and not dead it submits it to all peers for inclusion in share chain and sends accept notification to miner
   c) if share is > block diff it submits it to bitcoin network and sends accept notification to miner

So why doesn't DOA % match reject % in cgminer?
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742



View Profile WWW
April 26, 2012, 06:02:59 PM
 #2105

So I have been taking a close look at p2pool lately (for my own peace of mind if nothing else).

Shouldn't p2pool node report all DOA as rejected back to the miner.

Across my entire farm cgminer shows ~ 1.5% reject rate ("R" in cgminer).  p2pool shows ~2.5% DOA rate.

My assumptions is the logic flow is something like this
1) miner requests work (getwork).
2) p2pool provides work.
3) miner returns low diff shares (I have mine set to a static diff 1).
4) p2pool verifies share
   a) if share is DOA it increments DOA count by one, and sends reject notification to iminer
   b) if share is < share diff and not dead it submits it to all peers for inclusion in share chain and sends accept notification to miner
   c) if share is < block diff it submits it to bitcoin network and sends accept notification to miner

So why does DOA % match reject % in cgminer?
I think you meant less than, not greater than. I don't know the answer to your question though Sad

DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 26, 2012, 06:18:28 PM
 #2106

less than target or greater than difficulty one or the other.
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742



View Profile WWW
April 26, 2012, 06:23:33 PM
 #2107

less than target or greater than difficulty one or the other.
Ah yes. I see now.

twmz
Hero Member
*****
Offline Offline

Activity: 737



View Profile
April 27, 2012, 04:17:29 AM
 #2108

Thanks for all that.  If there is an error my guess is it is in the "average hashrate".  It might be better to simply count total shares (and dificulty) received by your node.  

That is exactly how pool hashrate is calculated, as far as I know.  p2pool does the calculation and publishes the hashrate at http://localhost:9332.  I just pull from that value and calculate an average over the duration of a block.

I will take a closer look at how the avg hashrate is calculated.  An error there would mean you are starting with "dirty data".

Agreed.  I have said basically that same thing several times in this thread.  If the hashrate reported by p2pool is too high, our luck will appear artificially lower than it actually is.  Forrest has several times reassured me that the hashrate calculation being done by p2pool is correct, but I have not validated that myself by reviewing code.


A clarification
Code:
"ActualShares":545905
You are calculating this by polling http://localhost:9332/rate periodically? how often?
then getting avg hashrate over the block?
then duration * avg hashrate = # of hashes?
then # of hashes / 2^32 = # of shares?

Yes.  I collect the pool's hashrate from the http://localhost:9332/rate API every 5 minutes.  Then calculate the average hashrate during the time between when blocks are found.  Then calculated # of shares exactly as you indicated (hashrate * duration / 2^32).  The only reason I use diff 1 shares instead of hashes directly is the convenience of using smaller numbers.

Raw hashrate data is here:  http://p2pool.info/stats

Note, hashrate stats are only 1 per hour for the first few months of p2pool's life because that is what forrest had available to backfill my database.  From the time p2pool.info was created forward (sometime in early Feb), hashrate data is in 5 minute increments.  The code that calculates the average hashrate actually calculates a weighted average so as to deal with mixed frequency of hashrate stats within a single block.

Was I helpful?  1TwmzX1wBxNF2qtAJRhdKmi2WyLZ5VHRs
WoT, GPG

Bitrated user: ewal.
forrestv
Hero Member
*****
Offline Offline

Activity: 510


View Profile
April 27, 2012, 03:54:37 PM
 #2109

So I have been taking a close look at p2pool lately (for my own peace of mind if nothing else).

Shouldn't p2pool node report all DOA as rejected back to the miner.

Across my entire farm cgminer shows ~ 1.5% reject rate ("R" in cgminer).  p2pool shows ~2.5% DOA rate.

My assumptions is the logic flow is something like this
1) miner requests work (getwork).
2) p2pool provides work.
3) miner returns low diff shares (I have mine set to a static diff 1).
4) p2pool verifies share
   a) if share is DOA it increments DOA count by one, and sends reject notification to iminer
   b) if share is > share diff and not dead it submits it to all peers for inclusion in share chain and sends accept notification to miner
   c) if share is > block diff it submits it to bitcoin network and sends accept notification to miner

So why doesn't DOA % match reject % in cgminer?

You do mean the dead on arrival pseudoshare rate, not the dead share rate, right? The dead on arrival rate should match cgminer's, but the one displayed on the console is only averaged over 10 minutes. Did you look at the graphs to get a more accurate rate? (area of local dead/area of local total)

1J1zegkNSbwX4smvTdoHSanUfwvXFeuV23
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 27, 2012, 04:16:59 PM
 #2110

You do mean the dead on arrival pseudoshare rate, not the dead share rate, right?

By psueodshare I assume you mean the low diff share cgminer thinks it is mining?
Which DOA stat is for the diff 1 shares and which is for "full sharechain difficult shares?

DOA is reported in 3 ? places
1) In graphs both total and per miner
2) On the /static page
Code:
Local rate: 11.6GH/s (2.0% DOA)
3) In the console window (which I find myself using less and less).

It may have just been a timing issue.  I think this weekend I will restart p2pool w/ no historical stats and restart all miner instances to get a "baseline".

I am also trying to figure out an easy way to convert JSON file into relational db tables so I can do some deeper digging.

My hunch (and I would like to stress completely unfounded) is that there is some error in the p2pool.info stats.

I think this error comes from the double estimations.

The raw data is shares seen by p2pool node.
p2pool abstracts that into global hashrate averaged over 10? minutes
p2pool poools that hashrate every 5 min?
p2pool uses that to construct avg hashrate over the block
p2pool use that and block time to determine # of hashes in block
p2pool compares # of hashes in block vs expected # of hashes in block to get "luck"

I am wondering if there is a better way.  simply look at all the shares since last block.
diff 1 shares for block = sum of difficulty of shares seen by node since last block
right? 
twmz
Hero Member
*****
Offline Offline

Activity: 737



View Profile
April 27, 2012, 05:27:48 PM
 #2111

I am wondering if there is a better way.  simply look at all the shares since last block.
diff 1 shares for block = sum of difficulty of shares seen by node since last block
right?  

I don't fully understand the internals of the share chain distribution, but I think there are a couple problems that would have to be solved:

  • We have a significant number of blocks where the duration was longer than the length of the share chain.  So in order to add up shares after a block is found, I would have to invent my own persistence of the the share chain that never got cleaned up.  Not trivial or fun (and fun = motivation to do it)
  • The "hashes done by the pool" include hashes contained in stale shares that are not fully distributed through the p2p network.  Because those stale shares mostly could have been valid blocks, they need to be included in the luck calculation.  In fact a lot of our blocks actually were stale shares (as determined by the fact that they often aren't announced in IRC).  The current p2pool hashrate calculation handles this by counting the valid shares and then increasing the count based on the pool-wide stale rate.

Was I helpful?  1TwmzX1wBxNF2qtAJRhdKmi2WyLZ5VHRs
WoT, GPG

Bitrated user: ewal.
twmz
Hero Member
*****
Offline Offline

Activity: 737



View Profile
April 27, 2012, 05:31:02 PM
 #2112

I am wondering if there is a better way.  simply look at all the shares since last block.
diff 1 shares for block = sum of difficulty of shares seen by node since last block
right?  

I don't fully understand the internals of the share chain distribution, but I think there are a couple problems that would have to be solved:

  • We have a significant number of blocks where the duration was longer than the length of the share chain.  So in order to add up shares after a block is found, I would have to invent my own persistence of the the share chain that never got cleaned up.  Not trivial or fun (and fun = motivation to do it)
  • The "hashes done by the pool" include hashes contained in stale shares that are not fully distributed through the p2p network.  Because those stale shares mostly could have been valid blocks, they need to be included in the luck calculation.  In fact a lot of our blocks actually were stale shares (as determined by the fact that they often aren't announced in IRC).  The current p2pool hashrate calculation handles this by counting the valid shares and then increasing the count based on the pool-wide stale rate.

It would be an interesting experiment (and much easier because you can ignore the first problem by picking blocks that are shorter than 18 or so hours) to spot check.  If you want to calculate how many hashes you think went into a few blocks, I can tell you how many hashes p2pool.info thinks went into it and we can see how much variation there is in the two methods.

Was I helpful?  1TwmzX1wBxNF2qtAJRhdKmi2WyLZ5VHRs
WoT, GPG

Bitrated user: ewal.
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742



View Profile WWW
April 27, 2012, 05:55:44 PM
 #2113

When do the rest of us get this page? This looks nice. http://p2pool.hopto.org:9332/static/nodes.html

DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 27, 2012, 05:56:09 PM
 #2114

I hadn't thought about the fact that blocks can be longer than sharechain. I see a couple work arounds
a) quick & dirty for analysis I could simply cron job to archive the sharechain at quicker interval than shares fall off the end and recombine it to get a continual history

b) slightly better would be looking into the code to make p2poool keep shares around longer and/or make it a command line param

As far as orphaned (but not DOA) shares I need to look at the code on propogation.  If they aren't fully distributed then how does a node calculate hashrate & stale (better called orphan) rate?  

Quote
The current p2pool hashrate calculation handles this by counting the valid shares and then increasing the count based on the pool-wide stale rate.

So following that train of thought how does an arbitrary node determined the stale (orphan) rate for the network?  My node right now says orphan rate is 8.9%.  How does it know?  It must be notified of all orphaned shares (either by receiving shares and incrementing its local orphan count or by some orphan share message).  My understanding is things like hashrate, orphan %, etc all all calculated locally.  We just need to tap into this.

BTW I am not expecting you to do any of this based on my hunch.  This weekend I will dive into the code and see if we can find a method to count shares (well technically sum of share difficulty).  If I can and it diverges from p2pool.info stats then we can look at using that to improve accuracy of the stats.  If it doesn't well then I was the only one who wasted his time. Smiley
cabin
Full Member
***
Offline Offline

Activity: 205


View Profile
April 27, 2012, 10:49:16 PM
 #2115

I hadn't thought about the fact that blocks can be longer than sharechain. I see a couple work arounds
a) quick & dirty for analysis I could simply cron job to archive the sharechain at quicker interval than shares fall off the end and recombine it to get a continual history

b) slightly better would be looking into the code to make p2poool keep shares around longer and/or make it a command line param

As far as orphaned (but not DOA) shares I need to look at the code on propogation.  If they aren't fully distributed then how does a node calculate hashrate & stale (better called orphan) rate?  

Quote
The current p2pool hashrate calculation handles this by counting the valid shares and then increasing the count based on the pool-wide stale rate.

So following that train of thought how does an arbitrary node determined the stale (orphan) rate for the network?  My node right now says orphan rate is 8.9%.  How does it know?  It must be notified of all orphaned shares (either by receiving shares and incrementing its local orphan count or by some orphan share message).  My understanding is things like hashrate, orphan %, etc all all calculated locally.  We just need to tap into this.

BTW I am not expecting you to do any of this based on my hunch.  This weekend I will dive into the code and see if we can find a method to count shares (well technically sum of share difficulty).  If I can and it diverges from p2pool.info stats then we can look at using that to improve accuracy of the stats.  If it doesn't well then I was the only one who wasted his time. Smiley

Pages like this record tons of info about each share, and it is recording atleast some deads:
http://p2pool.hopto.org:9332/static/share.html#0000000000555d58d25a50651e47ec138edd63f4d7dfa868a9b679e160c79524

1Cabinz1RSccAbFx2DikYomSKeMupy7M6V
cabin
Full Member
***
Offline Offline

Activity: 205


View Profile
April 27, 2012, 10:54:46 PM
 #2116

When do the rest of us get this page? This looks nice. http://p2pool.hopto.org:9332/static/nodes.html

You can save the html page to your own web-static/ folder and then you'll have it too Smiley

The ping times will be unique to where you are and the pools are hard coded from ones people wanted to be public. It is possible to grab a list of all your connected peers but maybe not everyone would appreciate that.

1Cabinz1RSccAbFx2DikYomSKeMupy7M6V
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742



View Profile WWW
April 27, 2012, 11:15:26 PM
 #2117

When do the rest of us get this page? This looks nice. http://p2pool.hopto.org:9332/static/nodes.html

You can save the html page to your own web-static/ folder and then you'll have it too Smiley

The ping times will be unique to where you are and the pools are hard coded from ones people wanted to be public. It is possible to grab a list of all your connected peers but maybe not everyone would appreciate that.
Cool thanks.

Seems like hopto's ping time to my node is consistently less than the ping time to hopto. I guess I have a good network?

Smoovious
Hero Member
*****
Offline Offline

Activity: 504

Scattering my bits around the net since 1980


View Profile
April 28, 2012, 01:51:15 AM
 #2118

is it possible to show what version of p2pool the node is running, or is that not available?

also... woohoo! lowest ping time on your list! Cheesy

138ms! Cheesy

-- Smoov
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742



View Profile WWW
April 28, 2012, 06:03:14 AM
 #2119

is it possible to show what version of p2pool the node is running, or is that not available?

also... woohoo! lowest ping time on your list! Cheesy

138ms! Cheesy

-- Smoov

I'm starting to think those are ping times from the client to the p2pool nodes, not the node to the other nodes. Loading the page from the miner would be most accurate.

kano
Legendary
*
Offline Offline

Activity: 1918


Linux since 1997 RedHat 4


View Profile
April 28, 2012, 06:36:14 AM
 #2120

Meh - I somehow managed to post this in the BFL thread when I mean to post it here ...
OK fixing that, here it is here where it should be.

... and now I'll ask an evil question coz I'm curious about it ...

I added the code that did network block detection in cgminer.
It reports "BLOCK!" on BOTH Accepted and Rejected shares.
I've wondered in the past if anyone has ever had a Rejected "BLOCK!"
i.e. you found a block, but you were JUST unlucky enough to find it in the time frame where it would be Rejected
( of course people have, but they'd need to keep their logs and check them ... I do keep all my logs and sometimes check them Cheesy )

They should be rare but also easy to calculate how often you should get them ...
The Rejection rate of your shares is a percentage.
That Rejection rate is also the % of blocks you find that should also be a Rejected "BLOCK!"


... now for a code change suggestion ...

So if you get 10% of your shares rejected then 10% of the blocks you find should also be rejected ....................
However ........... of that 10% that get rejected, 98.3% of them would be accepted by the bitcoin block chain but are invalid for the share chain and are thus thrown away .............

(I also wonder if this is a factor in the luck discussion - but I've no idea if it is)

Yeah I think there should be an exception in the code to allow rejected shares, that match this circumstance, and enable support for the exception in the share chain
i.e. if you get a Rejected BLOCK! it should make it all the way to the bitcoind and then it's up to bitcoind to reject it if it was a real block chain LP (not a P2Pool share chain LP)
Of course if this was implemented: it would also mean that everyone mining with cgminer on P2Pool should enable submit stale
(and also P2Pool should tell all miners to submit stale shares)

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
Pages: « 1 ... 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 [106] 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 ... 744 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!