Bitcoin Forum
December 09, 2016, 05:41:42 PM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 ... 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 [105] 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 ... 744 »
  Print  
Author Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool  (Read 2033438 times)
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 25, 2012, 10:57:09 PM
 #2081

How does the p2pool know the # of dead shares?
Aren't DOA shares detected at the node level, rejected, and then never forwarded to peers?
Do the p2pool.info charts include dead shares?  If so they are slanting the luck measurements.

Note: the questions refer to dead, aka DOA, aka dead on arrival, aka "rejected" (in cgminer) not orphaned shares.
1481305302
Hero Member
*
Offline Offline

Posts: 1481305302

View Profile Personal Message (Offline)

Ignore
1481305302
Reply with quote  #2

1481305302
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
cabin
Full Member
***
Offline Offline

Activity: 205


View Profile
April 25, 2012, 11:30:42 PM
 #2082

How does the p2pool know the # of dead shares?
Aren't DOA shares detected at the node level, rejected, and then never forwarded to peers?
Do the p2pool.info charts include dead shares?  If so they are slanting the luck measurements.

Note: the questions refer to dead, aka DOA, aka dead on arrival, aka "rejected" (in cgminer) not orphaned shares.


The dead shares are not relevant to the bad luck since they can actually solve for a block and result in everyone getting rewarded. P2Pool does look at them and double check if they solve a block before reporting back to cgminer as 'rejected'.

The type of problem we are looking for is exactly the opposite actually.. ie someone who is successfully submitting non-orphan, non-doa shares, but yet not finding any blocks. I can only think of far fetched scenarios under which this might happen but here are two such examples:

1) Someone modifies their p2pool code so they submit shares as normal but never submit a block. Stupid and unlikely, but theoretically possible.
2) Some obscure bug in some combination of bitcoind and p2pool, such that rarely a found block fails to be passed on to bitcoind (and yet shares and getworks continue to flow freely, the bug would have to only affect the case where it is an actual block solve)

As an example of 2), if anyone finds this line in your logs, congratulations.. you are the problem Smiley
"Error while processing potential block"

1Cabinz1RSccAbFx2DikYomSKeMupy7M6V
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 25, 2012, 11:38:00 PM
 #2083

The dead shares are not relevant to the bad luck since they can actually solve for a block and result in everyone getting rewarded. P2Pool does look at them and double check if they solve a block before reporting back to cgminer as 'rejected'.

Of course they are relevent.  Most DOA can't solve a block.  They are stale (due to bitcoind LP) or invalid.  All pools have these (commonly called rejected).  They are never included in luck calculations because they weren't good work to begin with.

Say the average DOA rate is 3%.  If those 3% of shares (unpaid, worthless work) are included in luck calculations that the expected # of blocks will be inflated by 3% making the pool appear to be 3% unluckier.  No conventional pool includes rejected shares in their luck calculations.

As an example of how invalid it is to include DOA shares I could point my miners towards a normal pool and have the found shares also submitted to p2pool.  Obviously the shares wouldn't be valid for p2pool so it would be a 100% DOA rate.  If the DOA rate is/was included in luck calculations then I made the pool appear x% unluckier.

So I will ask again.
1) HOW does p2pool network know the DOA rate?
2) Are they included in p2pool.info luck calculations?

DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 26, 2012, 12:10:03 AM
 #2084

1) Someone modifies their p2pool code so they submit shares as normal but never submit a block. Stupid and unlikely, but theoretically possible.
2) Some obscure bug in some combination of bitcoind and p2pool, such that rarely a found block fails to be passed on to bitcoind (and yet shares and getworks continue to flow freely, the bug would have to only affect the case where it is an actual block solve)

Regarding #2 doesn't the local p2pool node pass the block to p2pool peers also.  So a failure on one node's bitcoind would prevent block propagation.  Now it may result in slower block propagation so we should see that in a higher block orphan rate (which we don't) rather than just lower luck.

If I am mistaken and p2pool node doesn't do that this may be a way to make the network "stronger" so that the failure of one bitcoind can't lower block finding rate.
cabin
Full Member
***
Offline Offline

Activity: 205


View Profile
April 26, 2012, 12:33:29 AM
 #2085

The dead shares are not relevant to the bad luck since they can actually solve for a block and result in everyone getting rewarded. P2Pool does look at them and double check if they solve a block before reporting back to cgminer as 'rejected'.

Of course they are relevent.  Most DOA can't solve a block.  They are stale (due to bitcoind LP) or invalid.  All pools have these (commonly called rejected).  They are never included in luck calculations because they weren't good work to begin with.

Say the average DOA rate is 3%.  If those 3% of shares (unpaid, worthless work) are included in luck calculations that the expected # of blocks will be inflated by 3% making the pool appear to be 3% unluckier.  No conventional pool includes rejected shares in their luck calculations.

As an example of how invalid it is to include DOA shares I could point my miners towards a normal pool and have the found shares also submitted to p2pool.  Obviously the shares wouldn't be valid for p2pool so it would be a 100% DOA rate.  If the DOA rate is/was included in luck calculations then I made the pool appear x% unluckier.

So I will ask again.
1) HOW does p2pool network know the DOA rate?
2) Are they included in p2pool.info luck calculations?



I believe most DOA CAN solve a block in P2Pool's case. They are only DOA with regard to the 10 second shares, but they are probably hashing away on a perfectly valid 10 minute real block. I'm not 100% on this though. And I see your point.. in the extreme case a terribly connected miner, who will never solve a real block, could be hashing away and getting say 50% stales and 50% DOA. He is not getting any shares and so is not hurting the rest of us, but his hashes may be skewing the luck. Again I'm not 100% on this either.

1Cabinz1RSccAbFx2DikYomSKeMupy7M6V
cabin
Full Member
***
Offline Offline

Activity: 205


View Profile
April 26, 2012, 12:38:57 AM
 #2086

1) Someone modifies their p2pool code so they submit shares as normal but never submit a block. Stupid and unlikely, but theoretically possible.
2) Some obscure bug in some combination of bitcoind and p2pool, such that rarely a found block fails to be passed on to bitcoind (and yet shares and getworks continue to flow freely, the bug would have to only affect the case where it is an actual block solve)

Regarding #2 doesn't the local p2pool node pass the block to p2pool peers also.  So a failure on one node's bitcoind wouldn't prevent block propagation.  Now it may result in slower block propagation so we should see that in a higher block orphan rate (which we don't) rather than just lower luck.

If I am mistaken and p2pool node doesn't do that this may be a way to make the network "stronger" so that the failure of one bitcoind can't lower block finding rate.

That is a good question.. the code that would do this would start at around line 577 in main.py.. and in looking it may not in fact recover from this, it seems to just log an error. But I'm going to shutup and let forrestv answer this one.

1Cabinz1RSccAbFx2DikYomSKeMupy7M6V
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 26, 2012, 01:01:42 AM
 #2087

They are only DOA with regard to the 10 second shares, but they are probably hashing away on a perfectly valid 10 minute real block.

DOA simply means the share is invalid by the time the local node gets it.  Don't make any assumptions beyond that.  Conventional pools reject a sizable number of shares and they have no 10 sec LP.  "Some" DOA may still be valid but not all of them are.

Quote
And I see your point.. in the extreme case a terribly connected miner, who will never solve a real block, could be hashing away and getting say 50% stales and 50% DOA.
The "horrible" miner is just an example.  IF (and I am not sure they are) DOA are included in p2pool.info luck calculations they will skew them.
cabin
Full Member
***
Offline Offline

Activity: 205


View Profile
April 26, 2012, 01:23:05 AM
 #2088

They are only DOA with regard to the 10 second shares, but they are probably hashing away on a perfectly valid 10 minute real block.

DOA simply means the share is invalid by the time the local node gets it.  Don't make any assumptions beyond that.  Conventional pools reject a sizable number of shares and they have no 10 sec LP.  "Some" DOA may still be valid but not all of them are.

Quote
And I see your point.. in the extreme case a terribly connected miner, who will never solve a real block, could be hashing away and getting say 50% stales and 50% DOA.
The "horrible" miner is just an example.  IF (and I am not sure they are) DOA are included in p2pool.info luck calculations they will skew them.


My understanding is that > 98% of P2pool's DOA's have the potential to solve a block. The only ones that would not be useful are those DOA's that came right after a real block chain LP, ie once every 10 minutes. This also matches up well with the 1-2% of shares that regular pools reject, and I agree, 100% of these regular pool rejects are useless and have no chance of solving a block.

1Cabinz1RSccAbFx2DikYomSKeMupy7M6V
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 26, 2012, 01:34:18 AM
 #2089

My understanding is that > 98% of P2pool's DOA's have the potential to solve a block. The only ones that would not be useful are those DOA's that came right after a real block chain LP, ie once every 10 minutes. This also matches up well with the 1-2% of shares that regular pools reject, and I agree, 100% of these regular pool rejects are useless and have no chance of solving a block.

Your numbers don't make sense.

You agree conventional pools have 1% to 2% stales due to bitcoin LP and invalid hashes.  Let's say 1.5%.   If 98% of p2pool DOA were NOT due to those reasons then 2% of them are.  Thus 1.5%/2% = 75% it would have an overall DOA rate of 75%.  For 98% of p2pool DOA to be due to 10 sec LP then it would need to have a DOA rate of 75%.  Obviously the % due to 10s LP is much lower.

If p2pool has a 4% DOA rate (??) and 1.5% of that is due to Bitcoin LP and invalid hashes then they make up ~40% of all DOAs.

Still I think I was confused.  p2pool doesn't track global DOA only global "stales" which if forrest can verify I believe is orphans only.


One correction:
Quote
The only ones that would not be useful are those DOA's that came right after a real block chain LP, ie once every 10 minutes.
You are forgetting invalid hashes.  No GPU produces 100% flawless hashes.  Some % (0.1% to 1% depending on overclock, temps, ASIC quality, etc) are just bad.  Nonsense hashes that solve no block.

TL/DR version

Conventional pool rejected shares = invalid hashes + stale hashes (after Bitcoin LP)
p2pool DOA shares = invalid hashes + stale hashes (after Bitcoin LP) + "pre-orphaned" hashes (hash that would be 100% orphaned it submitted)
Only the last one is valid work which can solve a block.
ChanceCoats123
Hero Member
*****
Offline Offline

Activity: 680



View Profile
April 26, 2012, 01:48:53 AM
 #2090

This is a bit off topic, and is probably something major that I should know, but why does p2pool use such high share difficulties?
cabin
Full Member
***
Offline Offline

Activity: 205


View Profile
April 26, 2012, 01:57:47 AM
 #2091


Conventional pool rejected shares = invalid hashes + stale hashes (after Bitcoin LP)
p2pool DOA shares = invalid hashes + stale hashes (after Bitcoin LP) + "pre-orphaned" hashes (hash that would be 100% orphaned if submitted)
Only the last one is valid work which can solve a block.


I agree completely with this summary.. the percents I'm not sure about. I'm pretty sure the leads are worth exploring though. If we are mistakenly including too many dead hashes in the luck calculations then that is no good for the pools image.

1Cabinz1RSccAbFx2DikYomSKeMupy7M6V
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 26, 2012, 02:07:50 AM
 #2092

This is a bit off topic, and is probably something major that I should know, but why does p2pool use such high share difficulties?

No it is perfectly on topic.

p2pool tries to maintain a 10s LP windows (10s between shares).  This is a compromise.  Shorter LP window = higher orphans.  Longer LP window = higher difficulty.

One way to look at it is p2pool is ~350 GH/s.  At diff 1 shares that is ~81 shares per second or <0.01s between shares.  Work would become stale before it even finishes propogating the network.

So the difficulty is simply a compromise.  It takes difficulty of ~800 to keep 350 GH/s producing 1 share per 10 seconds.
twmz
Hero Member
*****
Offline Offline

Activity: 737



View Profile
April 26, 2012, 02:12:48 AM
 #2093


Conventional pool rejected shares = invalid hashes + stale hashes (after Bitcoin LP)
p2pool DOA shares = invalid hashes + stale hashes (after Bitcoin LP) + "pre-orphaned" hashes (hash that would be 100% orphaned if submitted)
Only the last one is valid work which can solve a block.


I agree completely with this summary.. the percents I'm not sure about. I'm pretty sure the leads are worth exploring though. If we are mistakenly including too many dead hashes in the luck calculations then that is no good for the pools image.

p2pool.info relies entirely on the number reported by p2pool, itself, at http://localhost:9332/rate   If that is wrong, then the luck calculation will be wrong.  Fixing it is not something I can do.  The fix, if there is one to be made, needs to come from forrestv.

Was I helpful?  1TwmzX1wBxNF2qtAJRhdKmi2WyLZ5VHRs
WoT, GPG

Bitrated user: ewal.
kano
Legendary
*
Offline Offline

Activity: 1932


Linux since 1997 RedHat 4


View Profile
April 26, 2012, 02:15:31 AM
 #2094

This is a bit off topic, and is probably something major that I should know, but why does p2pool use such high share difficulties?
Because it is transferring around the network to every p2pool every share (not just every block like the normal block chain)

So shares must be rare enough to only happen at a rate that can be handled - 10s is quite fast Smiley

To make it 10s on average per share means that the difficulty must go up to match the network (350GH/s)

350GH/s * 10s / 1 diff = 815 diff (hmm is 815 correct?)
(1 diff = 2^32 Hash)

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
organofcorti
Donator
Legendary
*
Offline Offline

Activity: 1960


Poor impulse control.


View Profile WWW
April 26, 2012, 03:07:23 AM
 #2095

Please stop spreading FUD  Roll Eyes P2pool is open source and there is nothing suspect in the code. Do you know what variability and luck is?

Yes, what I'm suggesting is that we may be having issues with orphaned blocks or some such issue.
Just saying, the odds of luck as bad as p2pool now displays is less than 1%

How are you calculating that?

Binomial distribution - couldn't work out an average # for share difficulty throughout the life of p2pool, so took the values for shares submitted vs expected shares required for our current number of blocks from p2pool.info and assumed avg difficulty of 1.5 million. So N=somehugenumber, p=0.00000066667, k=actual shares found/p.

Not the best, but shouldn't be far off.

Negative binomial?

Bitcoin network and pool analysis 12QxPHEuxDrs7mCyGSx1iVSozTwtquDB3r
follow @oocBlog for new post notifications
ChanceCoats123
Hero Member
*****
Offline Offline

Activity: 680



View Profile
April 26, 2012, 03:52:46 AM
 #2096

Thank you two for the explanation. Makes perfect sense. Smiley
organofcorti
Donator
Legendary
*
Offline Offline

Activity: 1960


Poor impulse control.


View Profile WWW
April 26, 2012, 06:19:04 AM
 #2097

@D&T:

The '% of expected' round 'lengths' published on p2pool.info are uniformly distributed, and have a very regular mean of about 160% (p-value for the linear model is less than 10^-12). If the '% of expected' works the way I think it does (normalises round length to D) then I'd expect '% of expected' to be geometrically distributed and the average to be 100%.

I'm think I'm not getting something about the way the '%luck' is calculated?

Bitcoin network and pool analysis 12QxPHEuxDrs7mCyGSx1iVSozTwtquDB3r
follow @oocBlog for new post notifications
Ente
Legendary
*
Offline Offline

Activity: 1834



View Profile
April 26, 2012, 09:45:00 AM
 #2098


The dead shares are not relevant to the bad luck since they can actually solve for a block and result in everyone getting rewarded. P2Pool does look at them and double check if they solve a block before reporting back to cgminer as 'rejected'.

The type of problem we are looking for is exactly the opposite actually.. ie someone who is successfully submitting non-orphan, non-doa shares, but yet not finding any blocks. I can only think of far fetched scenarios under which this might happen but here are two such examples:

1) Someone modifies their p2pool code so they submit shares as normal but never submit a block. Stupid and unlikely, but theoretically possible.
2) Some obscure bug in some combination of bitcoind and p2pool, such that rarely a found block fails to be passed on to bitcoind (and yet shares and getworks continue to flow freely, the bug would have to only affect the case where it is an actual block solve)

As an example of 2), if anyone finds this line in your logs, congratulations.. you are the problem Smiley
"Error while processing potential block"


About 1):
Every miner working on Bitcoin, no matter if pool, solo or p2pool, has to decide what he wants to deliver before he finds the hash. That is because he has first to decide *what* to hash, then try a billion variances of that data, then, if/when he finds a valid hash, he can not change the data to still have a valid hash.
Exactly that discussion came up some weeks ago about the share-difficulty:

1) decide which share difficulty you work on, i.e. 600 or 2000
2) hash a million nonces, where you are allowed to change a "non-important" part of the data for every try, so you get a different hash result
3) eventually, a hash reaches the (in 1) decided upon) target. that means the first x letters of the hash are zero
4) submit exactly the block you solved (including the exact transactions and random nonce data) to the pool, or bitcoin, or p2pool network
5) profit

what you try:

3) you found a block+nonce combination with a valid hash. lets say you worked on "600 difficulty" data, which you decided in 1). You now luckily and randomly found a hash with much more zeros than needed for "600", in fact it would pass a difficulty of "5000"!
4a) thinking "yay, I want to be paied out for those "5000" now, not just the stinky "600"!
4b) you take that exact block+nonce and edit that "this is a 600 block" to "this is a 5000 block lol"
4c) you hash that block+nonce from 4b)
4d) you get an entirely different hash, which surely won't even pass a "1" target. crash and burn.

I wrote that example about p2pool share difficulty because I find it easier to understand. The point is, you have no chance to change the block+nonce to still have the same or a similar hash. The hash, and the hash only decides upon "jackpot" or "totally worthless".  And, of course, all important data is included in the block. Actually that stuff is called coinbase, and, for example, includes the hash of the block before too, among other stuff.
Now when you work on p2pool shares, it includes both the "p2pool share data" as well as the "bitcoin block data". Any manipulation will change the hash, so if you strip the p2pool part of the data you found a valid hash for, the hash does not fit any more.
(I didnt read about the details yet, the combination of both p2pool- and bitcoindata is quite clever, else we couldn't ever solve a bitcoinblock at all.. merge mining is on the same topic)

The only thing you, as a p2pool miner, can do, is drop the valid bitcoin block hash you found. No profit for the p2pool gang, but obviously no profit for you neither. The only effect this has is it will make the "luck" graph look bad. Which is exactly what we are talking about, of course..

Hope that helps, and correct anything I got wrong!

Ente

twmz
Hero Member
*****
Offline Offline

Activity: 737



View Profile
April 26, 2012, 12:34:35 PM
 #2099

@D&T:

The '% of expected' round 'lengths' published on p2pool.info are uniformly distributed, and have a very regular mean of about 160% (p-value for the linear model is less than 10^-12). If the '% of expected' works the way I think it does (normalises round length to D) then I'd expect '% of expected' to be geometrically distributed and the average to be 100%.

I'm think I'm not getting something about the way the '%luck' is calculated?

Raw data is here:  http://p2pool.info/blocks

For each block, you can see:

  • The actual number of "difficulty 1" shares submitted before the block was found.  Note, that since we don't actually know how many "difficulty 1" shares were submitted, this is an estimate of how many difficulty 1 shares should have been submitted based on the average hashrate at the time and the duration of the round.  And so if the hashrate published by http://localhost:9332/rate is wrong, then this number of shares will also be wrong.
  • The estimated number of "difficulty 1" shares theoretically needed based on the bitcoin difficulty at the time

% expected for a single block is:  actual shares / expected shares

% luck over a 30 day window is (sum of expected shares for all blocks found within 30 days) / (sum of all actual shares for all same blocks)

Was I helpful?  1TwmzX1wBxNF2qtAJRhdKmi2WyLZ5VHRs
WoT, GPG

Bitrated user: ewal.
Ente
Legendary
*
Offline Offline

Activity: 1834



View Profile
April 26, 2012, 12:43:08 PM
 #2100

I would say the "difficulty 1 shares" are close enough as an estimate of hashing power. They are easy enough to find and other pools use them for calculating the payout (and surely the pool's hashpower) too.
So, if we can be sure the "number of diff-1-shares submitted between the last share and this share" is correct, I would use it as the base of all later calculations.


  • The actual number of "difficulty 1" shares submitted before the block was found.  Note, that since we don't actually know how many "difficulty 1" shares were submitted, [..]
I don't understand that. Sounds contradictious to me?

Ente[/list]
Pages: « 1 ... 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 [105] 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 ... 744 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!