[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[btharper]

Yes, that's new. Preparation for ASICs! It tries to regulate the work's difficulty to keep it at one response per second.

Is there any way to configure the period server or client side?

Not currently, no. Do you have a need to?

Is this per miner or per node? Ie I have only 150MH, some 1GH miner is uisng my node. What will be share diff for me and for him? Same?

In this case each miner would receive work with the target adjusted such that on average each would report back once per second. The slower hasher would get a lower difficulty share (though solving a sharechain block would be just as hard for either one) and the faster hasher would get a higher target so they still report back once per second.

I was wondering if this could be changed by the server/node (to reduce how often they're contacted) or by the client/miner (so they didn't have to report back as often/more often). Less often leads to lower badwidth and less time used waiting for the server's response to the submitted work, lower times give a more accurate picture of miner performance at the cost of more overhead.

I don't have any particular need, I can make up use cases (miners over local network who don't care about server and network usage, wanting to reduce overhead, and wanting lower variability in stats) and I figured it would be similar to setting share difficulty manually at the miner using :diff syntax on the username. Wouldn't have known without asking basically. In truth I could see this being abused much more than it's worth by people trying to tweak their statistics and making overall performance worse.

[sharky112065]

why does p2pool always seem to fall on its face when the hashrate is > 380gh?

I used to think the same except I said > 300gh.

I now believe it's all variance. 

No one seems to complain when we get 6 blocks in a day, when we should get about 2.2.  We just complain when we have 24 hr blocks..

M

The problem is that we never seem to get any of those 6 or 5 block days when we are above 350gh. If we would get just one of those 6 or 5 block days, then I will be convinced there is not some kind of bug/design flaw.

I say 350 but really don't know the exact rate it is. Seems over 300 but less than 400 when it happens.

I used to be the same way.  But if you look at the numbers... right now we're at 360g/h, which works out about 11 hours if met right on "schedule".  With this low a hash rate, we will have really long blocks, and hopefully really short blocks.  According to http://p2pool.info/, we're well above the average time for block.  So yes the 24 hour blocks are painful, but we still come out ahead.

I'm hoping a bunch of people have the gumption to point their ASICs to p2pool so we can get the hash rate up to lower the variance.   (I know I intend to point at least one here...)

M

OK, show me 5 or 6 blocks in a day when we are over 350 GH for the entire day and I will believe you. I know what variance is, but variance to the good when we are over 350 GH for 24 hours and I'm sold. It doesn't even need to be 5 or 6 blocks, just positive luck for that day.

[rav3n_pl]

In mean time V9 is passing 40% of pool.
Trouble is, that V5+V7 is about 10%, so if those ppl will not switch to v9 95% hardfork will not happen soon...

Upgrade mates! Upgrade!

[K1773R]

In mean time V9 is passing 40% of pool.
Trouble is, that V5+V7 is about 10%, so if those ppl will not switch to v9 95% hardfork will not happen soon...

Upgrade mates! Upgrade!

v8 update rush has worked after some time, then no fork. ppl disappointed -> longer time to do it now.

[Aseras]

Yes, that's new. Preparation for ASICs! It tries to regulate the work's difficulty to keep it at one response per second.

Is there any way to configure the period server or client side?

Not currently, no. Do you have a need to?

Is this per miner or per node? Ie I have only 150MH, some 1GH miner is uisng my node. What will be share diff for me and for him? Same?

In this case each miner would receive work with the target adjusted such that on average each would report back once per second. The slower hasher would get a lower difficulty share (though solving a sharechain block would be just as hard for either one) and the faster hasher would get a higher target so they still report back once per second.

I was wondering if this could be changed by the server/node (to reduce how often they're contacted) or by the client/miner (so they didn't have to report back as often/more often). Less often leads to lower badwidth and less time used waiting for the server's response to the submitted work, lower times give a more accurate picture of miner performance at the cost of more overhead.

I don't have any particular need, I can make up use cases (miners over local network who don't care about server and network usage, wanting to reduce overhead, and wanting lower variability in stats) and I figured it would be similar to setting share difficulty manually at the miner using :diff syntax on the username. Wouldn't have known without asking basically. In truth I could see this being abused much more than it's worth by people trying to tweak their statistics and making overall performance worse.

It should work like this but it doesn't. It's handing 3+ diff shares to a miner of mine that is 2 150 mhash cards and every share is dead for >24 hours.

I've gone back to 8.2.

There need to be a way to toggle or control it, at least for now for the slower workers.

[rav3n_pl]

In mean time V9 is passing 40% of pool.
Trouble is, that V5+V7 is about 10%, so if those ppl will not switch to v9 95% hardfork will not happen soon...

Upgrade mates! Upgrade!

v8 update rush has worked after some time, then no fork. ppl disappointed -> longer time to do it now.

v7 was fork update, v8 - anti-v7-fork ;]
Now finally v9
I just hope that tx sending will be rolling again.
Maybe do it in this way:
- preload: get all current txns form bitcoind
- found new txn in bitciond -> ask connected peers about that txn
- send full tx data to peer that want it
- got ask from peer -> check that you have it and ask for data if not
- all txns that are not in block need to be cashed to prevent asking/sending same txes over and over
- node forget txes that are put in block 2 blocks back - to prevent asking txes from nodes that are 1 block behind network when new block is propagated
- purge tx cache data from txes that are not in block every few blocks/refresh from bitcoind

[Syke]

OK, show me 5 or 6 blocks in a day when we are over 350 GH for the entire day and I will believe you. I know what variance is, but variance to the good when we are over 350 GH for 24 hours and I'm sold. It doesn't even need to be 5 or 6 blocks, just positive luck for that day.

The odds of having a 6-block day, when the hashrate is high, is quite low and simply due to variance. Not finding one isn't going to prove anything. Our 90-day moving average is over 110% of expected payout. That's impressive.

[Clock Loop]

BUGZZZZZ
I didn't do anything abnormal, using bcoin0.71
Started p2pool like I do normally,  but then I saw that an invalid hash message poped up, and then the bug showed up.

Code:

2012-11-16 13:14:56.292000 RECV forget_tx 019b904c00308d0f7d57af02ddfccc1559a8f16b9d6228a75f4bb147311b0fc152
2012-11-16 13:14:56.292000 > Error handling message: (see RECV line)
2012-11-16 13:14:56.292000 > Traceback (most recent call last):
2012-11-16 13:14:56.292000 >   File "twisted\internet\tcp.pyc", line 209, in _dataReceived
2012-11-16 13:14:56.292000 >     
2012-11-16 13:14:56.292000 >   File "p2pool\p2p.pyc", line 146, in new_dataReceived
2012-11-16 13:14:56.292000 >     
2012-11-16 13:14:56.292000 >   File "p2pool\util\p2protocol.pyc", line 39, in dataReceived
2012-11-16 13:14:56.292000 >     
2012-11-16 13:14:56.292000 >   File "p2pool\util\datachunker.pyc", line 40, in _DataChunker
2012-11-16 13:14:56.292000 >     
2012-11-16 13:14:56.292000 > --- <exception caught here> ---
2012-11-16 13:14:56.292000 >   File "p2pool\util\p2protocol.pyc", line 66, in dataReceiver
2012-11-16 13:14:56.292000 >     
2012-11-16 13:14:56.292000 >   File "p2pool\p2p.pyc", line 91, in packetReceived
2012-11-16 13:14:56.292000 >     
2012-11-16 13:14:56.292000 >   File "p2pool\util\p2protocol.pyc", line 79, in packetReceived
2012-11-16 13:14:56.292000 >     
2012-11-16 13:14:56.292000 >   File "p2pool\p2p.pyc", line 390, in handle_forget_tx
2012-11-16 13:14:56.292000 >     
2012-11-16 13:14:56.292000 > exceptions.KeyError: 37430759326633628792559864835538716825822374976072746282717190753054796583067L

I think this caused the bug:
2012-11-16 13:14:45.342000 invalid hash for 99.162.89.78 'remember_tx' 248345 04cac86c 86dc28a6********(rest of hash omitted because.....the text was too long for forum "The message exceeds the maximum allowed length (64000 characters). "

And this forum has no way to attach text files.

I am not sure if the invalid hash LENGTH is what cased the bug, because it was over 64,000 characters in length+
So perhaps this is just a bug that is caused by a hash that was too long, overloading a buffer or related.
This might just be a bug caused by getting hashes from peers that are wayyyy too long?   
(no idea how that would happen unless someone is malicious, or their pc flipped out/program crashed....)

[Schleicher]

Started p2pool like I do normally,  but then I saw that an invalid hash message poped up, and then the bug showed up.

Same thing happened here some time ago:
https://bitcointalk.org/index.php?topic=18313.msg1328349#msg1328349

[forrestv]

BUGZZZZZ
I didn't do anything abnormal, using bcoin0.71
Started p2pool like I do normally,  but then I saw that an invalid hash message poped up, and then the bug showed up.

Code:

...

I think this caused the bug:
2012-11-16 13:14:45.342000 invalid hash for 99.162.89.78 'remember_tx' 248345 04cac86c 86dc28a6********(rest of hash omitted because.....the text was too long for forum "The message exceeds the maximum allowed length (64000 characters). "

And this forum has no way to attach text files.

I am not sure if the invalid hash LENGTH is what cased the bug, because it was over 64,000 characters in length+
So perhaps this is just a bug that is caused by a hash that was too long, overloading a buffer or related.
This might just be a bug caused by getting hashes from peers that are wayyyy too long?   
(no idea how that would happen unless someone is malicious, or their pc flipped out/program crashed....)

I think it's due to a packet sent over the network getting corrupted somehow. Did P2Pool continue to work afterwards? It should have. If it didn't, this is a real bug ... but otherwise it's just a rare failure that was handled correctly.

[forrestv]

It should work like this but it doesn't. It's handing 3+ diff shares to a miner of mine that is 2 150 mhash cards and every share is dead for >24 hours.

I've gone back to 8.2.

There need to be a way to toggle or control it, at least for now for the slower workers.

A high difficulty shouldn't cause dead shares (it should only reduce how often pseudoshares are sent back to your P2Pool node), so this is either a bug in P2Pool or a bug in your miner. Can you tell me what mining software that miner was using? And send me p2pool's log file?

[Aseras]

A high difficulty shouldn't cause dead shares (it should only reduce how often pseudoshares are sent back to your P2Pool node), so this is either a bug in P2Pool or a bug in your miner. Can you tell me what mining software that miner was using? And send me p2pool's log file?

Running cgminer newest version. Log file is >70mb. Let me zip it and see how much smaller it gets.

[kano]

...
And this forum has no way to attach text files.
...

If you need to post something big - you put it in pastebin (or similar) and include the link.

[sharky112065]

OK, show me 5 or 6 blocks in a day when we are over 350 GH for the entire day and I will believe you. I know what variance is, but variance to the good when we are over 350 GH for 24 hours and I'm sold. It doesn't even need to be 5 or 6 blocks, just positive luck for that day.

The odds of having a 6-block day, when the hashrate is high, is quite low and simply due to variance. Not finding one isn't going to prove anything. Our 90-day moving average is over 110% of expected payout. That's impressive.

Yes our 90 day moving average is impressive and it is being used to sugar coat or slick over the issue we are trying to bring up. Just because our overall luck is good does not mean we don't have a scalability problem. When we get those bumps in hash speed, the person/s that jumped on board usually see that their payouts are low because of the bad luck during the period they hopped on and they leave. That drops us back down in hash rate back to where the supposed scalability issue is no longer in play.

So until we have a run of good luck in the 350+ GH range I and probably some others will still have doubts.

[K1773R]

OK, show me 5 or 6 blocks in a day when we are over 350 GH for the entire day and I will believe you. I know what variance is, but variance to the good when we are over 350 GH for 24 hours and I'm sold. It doesn't even need to be 5 or 6 blocks, just positive luck for that day.

The odds of having a 6-block day, when the hashrate is high, is quite low and simply due to variance. Not finding one isn't going to prove anything. Our 90-day moving average is over 110% of expected payout. That's impressive.

Yes our 90 day moving average is impressive and it is being used to sugar coat or slick over the issue we are trying to bring up. Just because our overall luck is good does not mean we don't have a scalability problem. When we get those bumps in hash speed, the person/s that jumped on board usually see that their payouts are low because of the bad luck during the period they hopped on and they leave. That drops us back down in hash rate back to where the supposed scalability issue is no longer in play.

So until we have a run of good luck in the 350+ GH range I and probably some others will still have doubts.

i dont mind the variance, u still get more than all other pools, just not in a steady way

[kano]

...
i dont mind the variance, u still get more than all other pools, just not in a steady way

It does depend on how the pool hash rate is calculated ...

The pool hash rate is actually somewhere around 5% or greater above the share rate.

So if the share rate is used to determine the pool hash rate, then yes it is lower than the true hash rate by quite a bit and thus the blocks found would appear to be above the expected 100% ... when in reality the share rate doesn't include the VERY high stale rate ... whereas the block rate does since the blocks are not lost, just the shares.

[gyverlb]

...
i dont mind the variance, u still get more than all other pools, just not in a steady way

It does depend on how the pool hash rate is calculated ...

The pool hash rate is actually somewhere around 5% or greater above the share rate.

So if the share rate is used to determine the pool hash rate

No. kano this is beginning to look like pure trolling: please install p2pool and use it or read the code instead of spreading FUD. I know using the forum to look for this kind of information is difficult, but it has actually already being discussed at lengths in this very thread.

[rav3n_pl]

Stale/doa shares depends on miner config and p2pool server load.
Orphan shares are just luck.
Shares are ONLY to calculate payout.
All shares are tested for possible block found.
Share diff is scaled to pool ratio/share about every 10s.
So stale and orphan ratio are NOT responsible for pool luck/block finding ratio.

[kano]

...
i dont mind the variance, u still get more than all other pools, just not in a steady way

It does depend on how the pool hash rate is calculated ...

The pool hash rate is actually somewhere around 5% or greater above the share rate.

So if the share rate is used to determine the pool hash rate

No. kano this is beginning to look like pure trolling: please install p2pool and use it or read the code instead of spreading FUD. I know using the forum to look for this kind of information is difficult, but it has actually already being discussed at lengths in this very thread.

So - how is the pool hash rate calculated?

[gyverlb]

So - how is the pool hash rate calculated?

It includes orphans and dead on arrival shares as they are seen by every node.