Can centralised pools point their hashpower at a p2pool node?

[hannesnaude]

I think there are some advantages to be had if some of the centralised pools would mine through a p2pool node, effectively creating a superpool.

Firstly, the resulting sharechain would provide an easy way for those parties with an interest in accepting low-confirmation transactions to know which transactions the pools are actively mining.

Secondly, if multiple pools point their hashpower at the same p2pool node, they will all see reduced earnings variance and get a share of the p2pool donations. This would remove (or at least drastically reduce) the incentive for miners to join the largest pool in order to reduce variance.

It would allow for much more accurate estimation of network hashpower and the hashpower split between pools (assuming pools don't hide their identities), which would also allow us to determine the fraction of the network in favour of a given proposal (think of the p2sh voting) more rapidly and accurately.

[1715601006]

1715601006

[1715601006]

1715601006

[KyrosKrane]

I've been mulling this one over, and honestly, I'm not sure it's such a good idea, at least not with p2pool in its current state.

Let's assume for the sake of discussion that a large pool does shift their entire backend to p2pool overnight. 

First off, it doesn't preclude any pool of a sufficiently large size from conducting a 51% attack, even with p2pool as an intermediary.  (I bring this up because preventing a 51% attack is frequently cited as a reason for switching to p2pool.) The p2pool share chain is consensus based, just like the bitcoin blockchain. Thus, the way a 51% attack works is still just as applicable with p2pool as it is with any other system.  The attacking pool simply has more hashpower than the rest of the p2pool network combined, and their share of the overall network hash rate is still the same as what it was before the shift to p2pool.

Second, it would immediately render p2pool useless for smaller miners.  Let me give an example with simplified numbers (but still close to reality). The current network speed is on the order of 100Ph/s, or 100,000 Th/s.  p2pool itself is registering on the order of 500Th/s, or 0.5% of the network speed. To get one share per day on p2pool, you need about 5Gh/s at present, or approximately 1/100,000 of the overall hash power of p2pool (again, simplified numbers, don't go crazy over my math! ).  Now let's assume that a pool with 20Ph/s shifts to p2pool.  p2pool's total hash rate jumps to a whopping 40 times its previous hash rate, and to get one share per day, you'd need over 200Gh/s.  So if you have less than that on p2pool, you essentially stop getting shares, which means you get less payouts.  It doesn't matter if the p2pool network is now finding 40 blocks a day or more; if you as an individual miner are sitting at zero shares, then you get zero payout.  You'd be better off solo mining and hoping to hit a jackpot -- or joining a competing pool.

That said, if you have sufficient hashpower yourself to continue getting at least one share per day, this would be a great situation.  Your variance would go down hugely, and while you'd earn much less per individual block found, you'd get many more payments per day as the network will find more blocks.

[hannesnaude]

Thanks for the constructive reply.

I've been mulling this one over, and honestly, I'm not sure it's such a good idea, at least not with p2pool in its current state.

Let's assume for the sake of discussion that a large pool does shift their entire backend to p2pool overnight. 

First off, it doesn't preclude any pool of a sufficiently large size from conducting a 51% attack, even with p2pool as an intermediary.  (I bring this up because preventing a 51% attack is frequently cited as a reason for switching to p2pool.) The p2pool share chain is consensus based, just like the bitcoin blockchain. Thus, the way a 51% attack works is still just as applicable with p2pool as it is with any other system.  The attacking pool simply has more hashpower than the rest of the p2pool network combined, and their share of the overall network hash rate is still the same as what it was before the shift to p2pool.

No, I understand that a 51% pool still has 51% irrespective of mining via p2pool or otherwise. I did not claim that this would prevent a 51% attack.

Second, it would immediately render p2pool useless for smaller miners.  Let me give an example with simplified numbers (but still close to reality). The current network speed is on the order of 100Ph/s, or 100,000 Th/s.  p2pool itself is registering on the order of 500Th/s, or 0.5% of the network speed. To get one share per day on p2pool, you need about 5Gh/s at present, or approximately 1/100,000 of the overall hash power of p2pool (again, simplified numbers, don't go crazy over my math! ).  Now let's assume that a pool with 20Ph/s shifts to p2pool.  p2pool's total hash rate jumps to a whopping 40 times its previous hash rate, and to get one share per day, you'd need over 200Gh/s.  So if you have less than that on p2pool, you essentially stop getting shares, which means you get less payouts.  It doesn't matter if the p2pool network is now finding 40 blocks a day or more; if you as an individual miner are sitting at zero shares, then you get zero payout.  You'd be better off solo mining and hoping to hit a jackpot -- or joining a competing pool.

I've always assumed (incorrrectly it would appear) that there are multiple p2pools, to counter the very problem you highlight here. Nevertheless it would be a trivial modification to the p2pool source code to ensure that the pools that mine into THIS p2pool (lets call it poolp2pool :-)) do not share a sharechain with the already existing p2pool miners. Therefore the existing p2pool difficulty remains unaffected. So if you initially had just one major pool switching to a system like this they would just be competing with themselves for shares on the new sharechain and their revenues would be unaffected, they will however, in the process provide the rest of us with an unforgeable proof of which transactions they are currently working on. This data has value and it could even be sold to interested parties (anyone interested in accepting low-confirmation transactions, or trying to estimate the fee they need to include to get their tx into the next block). This is an advantage in itself. However, when a second pool joins this poolp2pool, things get even more interesting, since both pools (and their miners) now see reduced earnings variance. For a miner with too little hashpower to  mine in p2pool directly, this becomes the next best option and miners have less incentive to join the largest pool since even tiny pools mining into poolp2pool can provide low variance income.

[KyrosKrane]

I'm not positive, but I think you just recreated the bitcoin blockchain itself...

The p2pool network is all one shared mining endeavor.  Anyone can set up a p2pool node (hence the p2p part of the name) and join the network; and your mining efforts on any p2pool node still count towards your reward, so long as you use the same payout address on all nodes.  The rewards are split among participating miners based on shares earned. In turn, the shares are tracked on a sharechain that's almost identical in concept to the bitcoin blockchain, except shares are issued much more quickly (once a minute, I think, rather than once every ten minutes on the blockchain).  It's possible that a dead or orphaned p2pool share is still a valid bitcoin block, and even if it doesn't get added to the sharechain, the miners get paid for it. Similarly, a valid and accepted share is frequently not a valid block on the bitcoin network.

The transactions being considered for inclusion in a block by the p2pool network are obtained from a standard bitcoin node, so tracking what transactions the pool is working on would be no different from getting a list of pending transactions from any bitcoin node's memory pool. In both cases, any given transaction is not guaranteed to be in the next block, or in any block for that matter.

I suppose you could create a fork of p2pool with its own independent sharechain; the software's all open source. If one of the major pools shifted their backend to this forked-p2pool, it would be mostly invisible to end users. For pools where the login is a bitcoin address, no change at all would be needed to the mining end, though some changes would be needed to the UI, of course. For pools where a different login is required, I would imagine they'd still have a bitcoin address on file for eventual payouts, and that could easily be reused to pay the miners when blocks are found.

Bear in mind, though, that funds from newly-found blocks are unspendable ("immature") until they have over 100 confirmations, versus the more standard six confirmations for a payment transaction that most pools currently do. That impacts usability for users.

I strongly suggest reading through the p2pool thread (or at least, the first post and the last few pages, as the thread is huge), then reforming your proposal a little more clearly and posting it there for feedback. While I use and support p2pool, I'm hardly an expert on it.

[hannesnaude]

I'm not positive, but I think you just recreated the bitcoin blockchain itself...

Well, yes, except it  wasn't me it was forrestv. :-). I am actually quite familiar with how p2pool woorks, and the few differences between the p2pool sharechain and the blockchain itself that you point it are integral to my proposal.

In turn, the shares are tracked on a sharechain that's almost identical in concept to the bitcoin blockchain, except shares are issued much more quickly (once a minute, I think, rather than once every ten minutes on the blockchain).

Unless I am very much mistaken it is once every 30 seconds. And this increased rate is what allows the benefits that I highlighted in the OP. Since shares are worth less than blocks a pool mining for shares as ppart of a larger conglomerate of pools will see lower earnings variance. The pools are effectively insuring each other against bad days, while remaining completely independent as far as block construction etc. is concerned. Usually a high blockrate has a downside since it results in more orphans and wasted hashpower. However, since the block rate of the underlying network is unchanged, this does not apply in this case. And since this p2pool is intended for use by a few pre-identified centralised pools, we could even raise the share generation rate higher still if the latency of the network links between the pools is low enough.

The transactions being considered for inclusion in a block by the p2pool network are obtained from a standard bitcoin node, so tracking what transactions the pool is working on would be no different from getting a list of pending transactions from any bitcoin node's memory pool. In both cases, any given transaction is not guaranteed to be in the next block, or in any block for that matter.

There is one big difference. If I get the list of pending transactions from a pool node's memory pool, I have no guarantee that the node is being truthful. It may show a tx paying me as being in the mempool, while behind the scenes it is hashing away on a different (double) spend of the same utxo. However, a sharechain provides an unforgeable proof that one or more of the participants in that chain was actually actively mining my transaction. Pools can provide the same proof in the current framework by simply publishing their best submitted share every X seconds. However they then lose the smoothing of earnings benefit.

I suppose you could create a fork of p2pool with its own independent sharechain; the software's all open source. If one of the major pools shifted their backend to this forked-p2pool, it would be mostly invisible to end users. For pools where the login is a bitcoin address, no change at all would be needed to the mining end, though some changes would be needed to the UI, of course. For pools where a different login is required, I would imagine they'd still have a bitcoin address on file for eventual payouts, and that could easily be reused to pay the miners when blocks are found.

I see no reason why any pool would need to change anyythinng on their fron-end. The pool is still mining to their own address and they are still paying out users with mature coins like they currently do. Users would see very little change other than reduced earnings variance. So if Btc Guild (12%), Discus Fish(12%) and Eligius (8%) were to band together like this they would present to their users earnings variance only slightly higher than a non-aggregated pool with 30% of network hashpower, while remaining independent as far as transaction selection goes (i.e. Eligius would mine certain non-standard txs while Discus Fish does not). Even if this aggregation of pools were to grow to 60% of the network (thus providing lower variance than any non-aggregated pool operator could offer) no single pool operator would be in a position to mount a 51% attack. And the sharechain would provide early warning if a pool operator started withholding hashpower (to perform a Finney attack or selfish mining for example).

Bear in mind, though, that funds from newly-found blocks are unspendable ("immature") until they have over 100 confirmations, versus the more standard six confirmations for a payment transaction that most pools currently do. That impacts usability for users.

Aware of this. Today pool operators do this by having their own stash of mature coins which they can pay out to users while waiting for the freshly mined coins which will be coming to the pool address to mature. This will remain unchanged

I strongly suggest reading through the p2pool thread (or at least, the first post and the last few pages, as the thread is huge), then reforming your proposal a little more clearly and posting it there for feedback. While I use and support p2pool, I'm hardly an expert on it.

I'm pretty familiar with the working of p2pool allthough my knowledge may be a little dated. I hope I have clarified the points in my proposal that created confusion.

[odolvlobo]

There might be benefits to others, but there needs to be an incentive for the pools themselves. Reduced variance is a benefit for a small pool, but why would a big pool want to join?

[DeathAndTaxes]

There is one big difference. If I get the list of pending transactions from a pool node's memory pool, I have no guarantee that the node is being truthful. It may show a tx paying me as being in the mempool, while behind the scenes it is hashing away on a different (double) spend of the same utxo. However, a sharechain provides an unforgeable proof that one or more of the participants in that chain was actually actively mining my transaction.

The sharechain doesn't contain the entire merkle tree and thus you can not verify what tx are being worked on by a share in the sharechain.  Each share has to be as small as possible (to improve efficiency).  It doesn't contain anything more than the bitcoin blockheader, the coinbase tx, and the merkle branch which allows cryptographic verification of the coinbase tx (linking coinbase tx to merkle root).  Actually this is a simplification.  It contains subsets of those elements because there are portions which can be independently computed by peers in the p2pool network and thus don't need to be relayed (i.e. the output of the coinbase tx).  p2pool shares don't contain anything more than what is needed to prevent cheating by p2pool peers.  For a p2pool share to contain all the tx hashes of the bitcoin block which would make them very heavy.

It may be possible to allow more efficient verification of "working tx set" but it would require a radical change to the Bitcoin block structure as well as block rules.   If the merkle trees were generally similar or deterministic it would be possible to provide a merkle tree as well as a instruction set on how to select the txs.  Like I said this is more useful for an altcoin as it would be a radical departure from Bitcoin.

I'm pretty familiar with the working of p2pool although my knowledge may be a little dated. I hope I have clarified the points in my proposal that created confusion.

There is some merit outside of double spend proof for pools using p2pool as a backbone however there is also a tragedy of commons.   Say p2pool represented 30% of the network hashrate this would mean any entity regardless of their actual contribution has the reduced variance of someone with 30% of the hashrate.   Yes all participants benefit but they benefit to a lesser degree.  The larger the participant the less they benefit and "worse" their advantage over other participants is reduced.   Established pools are able to charge a fee because they are trusted and they reduce variance.  By joining p2pool they undermine those two factors which are their sole reason for existence.  

To answer your question there is no technical reason why pools couldn't use p2pool (either the existing network or a private variant) as a backbone.  There never has been.  However there is also no "selfish" reason for them to do so.  All the reasons relate to the "common good" which is a hard sell.

[hannesnaude]

The sharechain doesn't contain the entire merkle tree and thus you can not verify what tx are being worked on by a share in the sharechain.  Each share has to be as small as possible (to improve efficiency).  It doesn't contain anything more than the bitcoin blockheader, the coinbase tx, and the merkle branch which allows cryptographic verification of the coinbase tx (linking coinbase tx to merkle root).  Actually this is a simplification.  It contains subsets of those elements because there are portions which can be independently computed by peers in the p2pool network and thus don't need to be relayed (i.e. the output of the coinbase tx).  p2pool shares don't contain anything more than what is needed to prevent cheating by p2pool peers.  For a p2pool share to contain all the tx hashes of the bitcoin block which would make them very heavy.

OK. It is important to keep in mind that even without any changes and without any pool providing any additional data, multiple pools using a p2pool-like backend would provide reduced variance. In order to prove that a given tx is being hashed by a given pool, there are various ways to go. The simplest would be for the pool to simply provide a merkle branch linking the tx to the merkle root. This may be a paid service.* Alternatively the pool may provide the full ordered list of tx hashes in the share. If the pool is also willing to provide the txs corresponding to any unknown hashes, this proves that it is not working on any private transactions that have not been disclosed to the rest of the network. This additional data (the merkle branch linking a given tx, or the full list of txs, need not be sent on the p2p network, but can be available from an API on the pool's server.)

It may be possible to allow more efficient verification of "working tx set" but it would require a radical change to the Bitcoin block structure as well as block rules.   If the merkle trees were generally similar or deterministic it would be possible to provide a merkle tree as well as a instruction set on how to select the txs.  Like I said this is more useful for an altcoin as it would be a radical departure from Bitcoin.

I don't think any more efficiency is actually required. The additional data that is required to wholly or partly verify the "working tx set" need not be transmitted on the p2p network (in fact if the data is to be sold, then it cannot be sent on the p2p network), and is actually quite lightweight for something served via a traditional webservice.

There is some merit outside of double spend proof for pools using p2pool as a backbone however there is also a tragedy of commons.   Say p2pool represented 30% of the network hashrate this would mean any entity regardless of their actual contribution has the reduced variance of someone with 30% of the hashrate.   Yes all participants benefit but they benefit to a lesser degree.  The larger the participant the less they benefit and "worse" their advantage over other participants is reduced.   Established pools are able to charge a fee because they are trusted and they reduce variance.  By joining p2pool they undermine those two factors which are their sole reason for existence.  

This is not quite correct. Larger pools in the superpool will still have lower variance than smaller pools, but it is possible for smaller pools in the superpool to have lower variance than large pools outside of it. A good example would be if BTC Guild and Discus Fish (both at 12% of network hashpower were to join into a superpool) Now BTC Guild has a lower variance, but it has come at the "cost" to them of also giving their competition Discus Fish a lower variance. Since they are equal in size and relatively small in comparison to the rest of the network, this seems like a good deal for both of them. If they were to let Eligius join their superpool, their own variance would get lower, and while Eligius variance would reduce, it would still be higher than theirs.

To answer your question there is no technical reason why pools couldn't use p2pool (either the existing network or a private variant) as a backbone.  There never has been.  However there is also no "selfish" reason for them to do so.  All the reasons relate to the "common good" which is a hard sell.

As I see it, the only pool that currently does not have an incentive to join such a superpool, is GHash.io. The tiny pools have nothing to lose, and the medium sized pools are better off joining a superpool with other medium sized pools (allthough possibly not with tiny pools) than not joining.

*This allows for new models of tx-fees. For example I could send a tx with no fees that would not normally get included in a block, but have a pay-per-share transaction fee contract in place with Discus fish. Discus fish would then work on including my tx in a block and receive a nanopayment from me for every share that they produce with a  valid merkle branch linking my tx to the merkle root. The downside is that this introduces variance in my tx fee, I may get lucky and pay a very low tx fee if one of the first shares that mining provider produces is a block, but I may also overpay, significantly if it takes very long before a block is found. The upside is that as a consumer I can choose which pools I wish to support with my tx fees.

[spin]

I follow some of the thoughts through I get this:
1. I think this opens the door to a small pool to charge a lower fee and give higher stability in payouts than a large pool.
2. If a couple of smaller pools start doing this, there result variance would reduce even further.  This means they would be more and more competitive for the same or lower costs than large pools.
3. Eventually the size of the pools in the p2pool would be such that they are offering better deals than the non p2pool miners.
4. Once the non-p2p miners are reduced competition would eventually make people wonder about pool-p2p-pool mining vs direct p2p-pool mining.

Which kind of proves the point that if you are not concerned with using your coins within 100 blocks you should be mining p2p as this is most cost-efficient. 

The obstacles has been raised that large (or even smallish) pools in the p2p pool would devaluate the experience for individuals in the p2p pool.  Not sure how to solve this? Maybe tiers of p2p pools?

I think the idea has potential to grow the p2p pool though. 

[jgarzik]

Thinking that there must be only One Blessed P2Pool Chain is linear thinking.

[Hawkix]

How about going tree instead of linear - what if there were multiple "tiers" of P2Pools?

The top of the tree would be Bitcoin blockchain. There will be Tier-1 P2Pool with high hashrate - lets say 10PH with shares difficulty 10 TH. Then, there would be multiple Tier-2 P2Pools, each one trying to mine Tier-1 P2Pool block, using its own blockchain with difficulty 10GH. And so on for Tier-3 P2Pools.

Users would connect to most appropriate P2Pool tier so that they get some payout lets say each day. Of course, payouts for Tier-3 will be in form of shares for Tier-2. But this allows steady-low-variance (holy grail, right) income for any type of device.

What do you think?

[hannesnaude]

I see PetaMine are having a vote on switching to p2pool. http://www.reddit.com/r/Bitcoin/comments/28o1cy/petamine_is_holding_a_vote_to_switch_to_p2pool_if/
While they are not a pool, it is interesting nevertheless. Hopefully this happens and makes it very tempting for some of the smaller pools to join, triggering a cascade.

Would it be too much to ask for all major mining operations to be working via p2pool? It would be so cool to know in near-realtime which txs are being worked on.

[sile16]

If they feel they can switch with minimal loss in output they will switch.  Otherwise they won't.