Incorporating the p2pool concept into Bitcoin

[Carlton Banks]

The problem is that you need a way to decrease the share difficulty.  P2pool is 2%? of the total hashing power and already there are complaints about how high the share difficulty is.

If p2pool was 100% of the network hash rate, then share difficulty would be 20X times lower than the main network (since it has a 30 second share rate).  That is an improvement, but almost as bad.  If large miners set their difficulty higher, then it would help.

The bottom line is that you need some way to increase the rate at which shares happen.

Why not increase the length of the sharechain? (and maintain 30 second per share, to increase the overall number of shares). I can't think of any serious drawbacks to this, the current design covers 3 days of shares right now.

[TierNolan]

Why not increase the length of the sharechain? (and maintain 30 second per share, to increase the overall number of shares). I can't think of any serious drawbacks to this, the current design covers 3 days of shares right now.

I don't think the length of the sharechain really matters that much.

As long as the pool hits multiple blocks for each share, the payout per share has reasonably low variance.

P2pool is 3 blocks or 3 days, whichever is shorter.  It hits blocks about 2-3 times a day, so the 3 day rule never kicks in.

The problem is the share difficulty.  If it takes days for miners to actually hit shares, then that gives high variance.

Faster shares means lower difficulty per share.  However, there is a limit to how fast the shares can be due to network latency.

[coins101]

To me, seeing some miners forced out as a consequence (due to difficulty or hardware limitations) seems like a price worth paying for the network's survival.

Would Satoshi turn in his anonymous grave - figuratively speaking? Just a reminder:

"We have proposed a system for electronic transactions without relying on trust. We started with the usual framework of coins made from digital signatures, which provides strong control of
ownership, but is incomplete without a way to prevent double-spending. To solve this, we proposed a peer-to-peer network using proof-of-work to record a public history of transactions that quickly becomes computationally impractical for an attacker to change if honest nodes control a majority of CPU power. The network is robust in its unstructured simplicity. Nodes work all at once with little coordination. They do not need to be identified, since messages are not routed to any particular place and only need to be delivered on a best effort basis. Nodes can leave and rejoin the network at will, accepting the proof-of-work chain as proof of what happened while they were gone. They vote with their CPU power....."

* A system that does not rely on trust.
* "the main benefits are lost if a trusted third party is still required to prevent double-spending"
* "Unstructured simplicity"
*etc.

The fact that ASICs have enabled the security of the network is offset, IMHO, by market driven behaviour pushing people to seek greater market efficiencies in order to maximise profit. Nothing wrong with that, except that bitcoin seemed to have been partly founded on the solution to the double-spending, which was proposed to be resolved by a true and unstructured p2p CPU system.

I would have said forcing small miners out qualifies as a Gorden Gekko moment, n'est-ce pas?

[btc4ever]

If I understand correctly, p2pool currently uses a single difficulty for the entire pool.  The difficulty changes to target the 30 second share time, but changes for everyone.  And it is presently so high that non asic devices are effectively useless anymore, and even some older asics.  ( It used to be 10 second target, but was adjusted up to 30 secs to work better with BFL asics. )

What if instead the p2pool difficulty was unique to each node, based on its avg hash power, with a goal of 30 seconds per share?

So then the node would report to its peers the difficulty along with each share.  The peer can easily validate that the shares exceed the difficulty.  When a block is found, the total hashing input will be the sum of (difficulty * number of shares ) from each node.  So each node would be rewarded based on its percentage of the total.

Wouldn't this allow slower devices to participate?

I haven't looked deeply into p2pool, so most likely I'm neglecting something obvious.  In that case, please educate me.

Edit: so far the biggest obstacle I see is that if we had tens or hundreds of thousands of peers contributing to a block, then the block reward transaction could become HUGE. 

Faster shares means lower difficulty per share.  However, there is a limit to how fast the shares can be due to network latency.

[TierNolan]

What if instead the p2pool difficulty was unique to each node, based on its avg hash power, with a goal of 30 seconds per share?

So then the node would report to its peers the difficulty along with each share.  The peer can easily validate that the shares exceed the difficulty.  When a block is found, the total hashing input will be the sum of (difficulty * number of shares ) from each node.  So each node would be rewarded based on its percentage of the total.

That is way to low a difficulty.  If there are 1000 nodes and each node produces shares every 30 seconds, then you get 33 shares per second.

A better target would be something like one share every 60 minutes.  However, even that gives a high share rate if there are enough nodes.

The current system ensures that there is one share every 30 seconds.

Maybe having branching chains would help.  You could have 10 chains, and every so often sync all 10.

[btc4ever]

hmm, good point.

Yes instead of 1 share approx every 30 sec we would get N shares every 30s, where N is the number of network nodes.

Okay, so let's adjust it to every 10 minutes.   Now we get N shares every 10 mins.   If 10000 nodes, that is 16.66 shares/sec.   Yeah, still too high.    Every 60 minutes with 10k nodes gives us 2.77 shares/sec.    still high.

Here's a random thought for scaling to any number of nodes.  What if the shares diminished with propogation distance?    Let's say for example that for each node your shares pass through, 20% is removed.  So it would travel at most 5 hops in any direction.  If the block is found by a node outside your circle, you wouldn't get any credit.  Possibly the number of hops could vary automatically based on the number of shares being received every X seconds.   I've no idea what the best values would be, but that's the general idea.

I'm not sure how branching chains would work.  Can you elaborate?

What if instead the p2pool difficulty was unique to each node, based on its avg hash power, with a goal of 30 seconds per share?

So then the node would report to its peers the difficulty along with each share.  The peer can easily validate that the shares exceed the difficulty.  When a block is found, the total hashing input will be the sum of (difficulty * number of shares ) from each node.  So each node would be rewarded based on its percentage of the total.

That is way to low a difficulty.  If there are 1000 nodes and each node produces shares every 30 seconds, then you get 33 shares per second.

A better target would be something like one share every 60 minutes.  However, even that gives a high share rate if there are enough nodes.

The current system ensures that there is one share every 30 seconds.

Maybe having branching chains would help.  You could have 10 chains, and every so often sync all 10.

[solex]

How much is it going to cost to get P2Pool in the reference client? Who is going to start a bounty?

It may not enlighten the traditional pool miners out there, but it will be a step in the right direction.

When the network is finally attacked thanks to centralized pools, we will have something to fall back on.

There is a bounty for improving P2Pool which was languishing at about 2 BTC (but that has been drawn down a few days ago).
Advancement of Decentralized Mining - Vital to Bitcoin Network Security

I agree it would be nice to see it in the reference client. The importance of decentralized mining outweighs the argument that the reference client should be focused toward the protocol.

[Carlton Banks]

Why not increase the length of the sharechain? (and maintain 30 second per share, to increase the overall number of shares). I can't think of any serious drawbacks to this, the current design covers 3 days of shares right now.

I don't think the length of the sharechain really matters that much.

As long as the pool hits multiple blocks for each share, the payout per share has reasonably low variance.

P2pool is 3 blocks or 3 days, whichever is shorter.  It hits blocks about 2-3 times a day, so the 3 day rule never kicks in.

The problem is the share difficulty.  If it takes days for miners to actually hit shares, then that gives high variance.

Faster shares means lower difficulty per share.  However, there is a limit to how fast the shares can be due to network latency.

You're missing my point. If there were more shares available over a longer period of time, the necessary difficulty per share could be reduced corresponding to the increase in the number of shares. You cite faster shares as the heart of the problem, but making more shares available is the problem, speeding up the interval between them is just one way of increasing the available shares (and reducing the difficulty by a commensurate amount)

[TierNolan]

You're missing my point.

Perhaps.

If there were more shares available over a longer period of time, the necessary difficulty per share could be reduced corresponding to the increase in the number of shares.

Not really. 

If you increase the length of the share chain, then 1 share pays out over a longer time.

If I hit a share, then I get paid out 10 times less every time p2pool hits a block.  However, my share remains active for 10 times longer.

The value of each share is exactly the same as before.  My variance for finding a share is exactly the same as before.

What changes is that I now have to wait 10 times longer to actually get paid for that share.

With 2 blocks per day and the sharechain running for 30 blocks, it would take 15 days for a constant miner to get their expected output.

There could be a psychological effect though.  If a miner remained at a constant output, after the 15 days, they would have lower day to day variance.

You cite faster shares as the heart of the problem, but making more shares available is the problem, speeding up the interval between them is just one way of increasing the available shares (and reducing the difficulty by a commensurate amount)

Difficulty would be completely unaffected under your scheme.  Difficulty is used to prevent spamming of the sharechain.  It gets set so that there is 1 share every approximately 30 second.

[Mike Hearn]

I question how much ASIC miners should be freaking out about variance. If you just sank $20k into an expensive machine, why can't you tolerate getting rewarded once a day or once a week? As long as the size of that reward is fair, it seems like anyone who can afford the hardware should be able to afford the buffer for electricity costs as well. No?

[Dende]

I think many people just want to get the most profit out of their miner.
I am in for the long run but early december I had a chance of getting a miner and all I think about was how can I roi as soon as possible and not about enhancing bitcoin security.
There need to be an incentive for bitcoin decentralization. Cheaper miner with p2pool contract for example if the manufacturers are willing, or plug and hash p2pool miner for people who is not tech-savvy. Manufacturers could play a role in the decentralization of bitcoin, and they should, they are getting a lot of money from the success of bitcoin.

For the short term, I think convincing miners into going to smaller pool is a priority before p2pool matures either through incentives or education of bitcoin security. ghash.io already has 1/3 of the global hashrate and it looks like it is still growing by comparing 24hr chart with 4 days chart https://blockchain.info/pools

[Carlton Banks]

You're missing my point.

Perhaps.

If there were more shares available over a longer period of time, the necessary difficulty per share could be reduced corresponding to the increase in the number of shares.

Not really.  

If you increase the length of the share chain, then 1 share pays out over a longer time.

If I hit a share, then I get paid out 10 times less every time p2pool hits a block.  However, my share remains active for 10 times longer.

The value of each share is exactly the same as before.  My variance for finding a share is exactly the same as before.

What changes is that I now have to wait 10 times longer to actually get paid for that share.

With 2 blocks per day and the sharechain running for 30 blocks, it would take 15 days for a constant miner to get their expected output.

There could be a psychological effect though.  If a miner remained at a constant output, after the 15 days, they would have lower day to day variance.

You cite faster shares as the heart of the problem, but making more shares available is the problem, speeding up the interval between them is just one way of increasing the available shares (and reducing the difficulty by a commensurate amount)

Difficulty would be completely unaffected under your scheme.  Difficulty is used to prevent spamming of the sharechain.  It gets set so that there is 1 share every approximately 30 second.

There is a pool of shares. It is currently comprised of 17500 shares in total. The owner of each share has it ranked according to how far each share exceeded the difficulty.

So every block found is divided into 17500 unequal rewards, and those nodes with more than one share receive the sum of those rewards to their address.

If the number of shares is increased to more than 17500, the difficulty per share would have to drop by a divisor of the factor of increase in that total number in order to target the same interval.


Your suggestion is to attain this increase in the pool of available shares by increasing the shares available within the current 72 hour window that the pool currently operates over, e.g. 35000 shares with a 15 second target difficulty (I don't think this can work with the majority of ASICs out there right now, forrestv changed the target from 10secs/24hours to 30secs/72hours to accommodate them)

My suggestion is to extend the 72 hour window, and maintain the 30 second difficulty target. This would decrease the difficulty target. It would increase the length of time that a new miner would need to build up a full coterie of shares. It would also increase the size of the sharechain on the disks of the mining nodes. My rationale for this is that the current sharechain is ~500 MB on disk, and that the miners are more likely to be lacking hashing power than they are to be lacking disk space.

[TierNolan]

There is a pool of shares. It is currently comprised of 17500 shares in total. The owner of each share has it ranked according to how far each share exceeded the difficulty.

Right, each share has a difficulty associated with it.

It isn't a fixed number of shares that are paid out.  It is 3 block's worth of shares.  Shares are never kept more than 3 days, so if the pool finds very few block, your share can fall off the end of the chain before 3 blocks reward.

So every block found is divided into 17500 unequal rewards, and those nodes with more than one share receive the sum of those rewards to their address.

Right, but not a fixed number of shares.  It is the most recent shares equal to 3 blocks worth of difficulty.

If the number of shares is increased to more than 17500, the difficulty per share would have to drop by a divisor of the factor of increase in that total number in order to target the same interval.

No, again, the difficulty is determined by the rate at which the shares are generated.  Increasing the number of shares included in the system would just increase the length of time the sharechain has to be kept.

It would also mean that miners get paid more often for each share, but paid a smaller amount.

My suggestion is to extend the 72 hour window, and maintain the 30 second difficulty target. This would decrease the difficulty target.

No, it wouldn't.

Once again, difficulty is determined by the average share rate. 

Ignoring the fact that miners can artificially increase their difficulty target, if you want one share every 30 seconds, then each share has to "cost" 30 seconds worth of hashing.

The share difficulty must be (p2pool hashing fraction) * (bitcoin difficulty) * (share target rate) / (600 seconds).  That is the number of hashes performed by p2pool in the target window.

It would increase the length of time that a new miner would need to build up a full coterie of shares.

Right.  They would get a smaller payout but be paid out for more blocks to compensate.

However, the total payout per share would be the same.

It would also increase the size of the sharechain on the disks of the mining nodes. My rationale for this is that the current sharechain is ~500 MB on disk, and that the miners are more likely to be lacking hashing power than they are to be lacking disk space.

Well, it depends on what they are controlling their mining with.  They might not have a disk, but 500MB isn't that large.

[spartacusrex]

Ignoring the fact that miners can artificially increase their difficulty target, if you want one share every 30 seconds, then each share has to "cost" 30 seconds worth of hashing.

Not sure if I'm missing something but is this not the crux point ?

You HAVE to make the miners increase their individual difficulty. By giving them an incentive to do so. By paying them more for doing so..

I would also say, that to make sure the rationale miner increases his difficulty, and his variance, to allow other miners on the chain, they get 1% more payout if they use difficulty x 512, or whatever..

Maybe more than 1% would be required to make sure that miners DEFINITELY went for it but if you could get the miners to try and get 1 block in every N shares, rather than N shares in every N, the longer the chain the lower the difficulty.

If N was 1024, the successful miner would set his difficulty to 1024 times normal difficulty, so as to get 1 share ONLY at the highest difficulty possible, thereby increasing his revenue. (As long as he still got 1 share in every N, otherwise he would choose a lower extra difficulty). As 1 share at 1024 times difficulty is worth more than 1024 shares at 1 times difficulty.

This would make the chain 1024 times easier to mine on ?

If N was higher, the difficulty would again be made lower - but ONLY if the miners increase their own difficulty to match (And get paid accordingly).

What i am unsure about is HOW MUCH MORE to give the miner for increasing his difficulty ?

If you give too little the miners just won't go for it, but if you give too much, then this will give the large pools X% more and they can give this straight to their users, and attract more users, etc etc.. and they will grow even bigger..   

Maybe a market driven mechanism. Not sure how..

[TierNolan]

Not sure if I'm missing something but is this not the crux point ?

I was responding to Carlton Banks.  He felt that increasing the length of the share window would decrease the difficulty.

Maybe more than 1% would be required to make sure that miners DEFINITELY went for it but if you could get the miners to try and get 1 block in every N shares, rather than N shares in every N, the longer the chain the lower the difficulty.

1% bonus would probably be enough.

Any miner who is getting more than 1 share an hour should definitely increase his difficulty.

This would make the chain 1024 times easier to mine on ?

No, because "small" miners would still be on lower difficulty.  A miner with 1000X the power of the other 1000 would get half the shares.

If he set his difficulty to 1000, then he would get 1 share while the rest got 1000 shares between them.  The difficulty would be halved to being things back to one share every 30 seconds.

If N was higher, the difficulty would again be made lower - but ONLY if the miners increase their own difficulty to match (And get paid accordingly).

The suggestion in the document from the donation thread was to simply charge per share.  The charge could be something like 1% of the expected payout per median share.

[Carlton Banks]

Once again, difficulty is determined by the average share rate. 

Ignoring the fact that miners can artificially increase their difficulty target, if you want one share every 30 seconds, then each share has to "cost" 30 seconds worth of hashing.

The share difficulty must be (p2pool hashing fraction) * (bitcoin difficulty) * (share target rate) / (600 seconds).  That is the number of hashes performed by p2pool in the target window.

Okay, you've convinced me. I wasn't accounting for the relationship between the block difficulty and the share difficulty.

It would also increase the size of the sharechain on the disks of the mining nodes. My rationale for this is that the current sharechain is ~500 MB on disk, and that the miners are more likely to be lacking hashing power than they are to be lacking disk space.

Well, it depends on what they are controlling their mining with.  They might not have a disk, but 500MB isn't that large.

This is true, there's a guy on this forum who was using a 32 GB RAM disk for p2pool. I don't know how he will justify the expense to add the RAM he will need in the future, but he maybe didn't buy the RAM for a p2pool node in the first place (and the cost per GB seems to drop regularly too)

[spartacusrex]

No, because "small" miners would still be on lower difficulty.  A miner with 1000X the power of the other 1000 would get half the shares.

If he set his difficulty to 1000, then he would get 1 share while the rest got 1000 shares between them.  The difficulty would be halved to being things back to one share every 30 seconds.

Ahh.. yes I see now.. A miner can only decrease the overall difficulty of the chain as a proportion of his own hash rate vs the rest of the network.. If he has half the hash power, he can decrease the overall difficulty by a half.

He can bring himself down to the level of the smaller miners..

..TierNolan!! 

[spartacusrex]

He can bring himself down to the level of the smaller miners..

Thinking about it, if you did have a 1024 PPLNS system by default, so that the last 1024 shares were paid, and miners could increase their difficulty and get paid etc etc.., would it not be the case that all the miners would lower themselves down to the smallest miner able to get on the chain ? So that, in the ideal case, each miner still only got 1 share per 1024.

This way you would at least have 1024 miners/mining pools visible on the chain ? Rather than the 10 or so currently available on Bitcoin.

Not sure how important decreasing the difficulty of the individual shares actually is.

The weakest miner able to get on the chain would set the hashrate that the others would have to emulate by increasing their difficulty.

Having 1024 mining pools able to get on the chain is better than less than 1024. No ?

[TierNolan]

Thinking about it, if you did have a 1024 PPLNS system by default, so that the last 1024 shares were paid, and miners could increase their difficulty and get paid etc etc.., would it not be the case that all the miners would lower themselves down to the smallest miner able to get on the chain ? So that, in the ideal case, each miner still only got 1 share per 1024.

This way you would at least have 1024 miners/mining pools visible on the chain ? Rather than the 10 or so currently available on Bitcoin.

The total number of shares is equal to 3 blocks worth of shares (in terms of difficulty).

If the average difficulty per share is lower, you get more shares included (up to the 3 day limit).

Not sure how important decreasing the difficulty of the individual shares actually is.

If the difficulty is to high, then smaller miners would only be on the chain sometimes.

Having 1024 mining pools able to get on the chain is better than less than 1024. No ?

One of the proposals is sub-pools.  Rather than mining against the main network, a mining pool could mine against p2pool.

A smaller pool inherently has higher variance, since it is hard to find bitcoin blocks.  Mining against p2pool gives better variance for smaller pools due to the lower share difficulty.

[btc4ever]

Are any pools known to be mining against p2pool at present?

One of the proposals is sub-pools.  Rather than mining against the main network, a mining pool could mine against p2pool.

A smaller pool inherently has higher variance, since it is hard to find bitcoin blocks.  Mining against p2pool gives better variance for smaller pools due to the lower share difficulty.