[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[jtoomim]

I heard from one of the Nicehash miners. He abandoned p2pool because the DOA rates he was getting from Nicehash were significantly eating into his revenue, and he can get more by solo mining. I can think of three ways of addressing this issue.

The first one is to submit DOA shares to the network and have them compete with non-DOA shares, and possibly change the preference algorithm on the receiver's side to give DOA shares a significant and fair chance of winning the subsequent orphan race. This approach would be likely to increase the possibility and severity of selfish mining attacks, so I'm a bit reluctant to pursue it. It would be pretty simple in terms of code changes, though. Some more detailed thoughts on this below.

The second one is to add an uncle mechanism or DAG structure to the share chain. This would likely be much fairer, but it would require a p2pool hard fork and a substantial amount of new code. This would be the best option if developer time weren't an issue.

The third one is to try to increase the average share interval by increasing the default or minimum share difficulty. Changes to the minimum share difficulty require a hard fork, but a simple one. Changes to the default share difficulty just require that some nodes (especially the large ones) run different code. If the average share took 4x as long, then both orphan and DOA rates would be 1/4th their current values. Drawbacks of this approach are (1) that small miners would find it more difficult to get shares into the share chain (unless share difficulty were a function of the number of shares in the chain), and (2) that it's potentially not incentive-compatible, as the benefit of making high-difficulty shares goes to everyone who mines afterwards, and not to the person who mines the high-diff share. A person with a lot of hashrate has a greater chance of winning an orphan race if he works on low-diff shares than if he works on high-diff shares, since p2pool essentially just uses (1) the share height and (2) the time of arrival when determining which share to base its work off of, and ignores difficulty.

But maybe that isn't the best way to choose the share to base work off of? Ultimately, the problem we're having with fairness comes from the fact that some shares have greater expected revenue in the share chain than other shares, due to differences in orphan risk caused by differences in node performance and propagation times. If we no longer take time-of-arrival into account (or weight it less heavily), then the incentive to run a large, high-performance p2pool node dissipates. That might sound like a bad thing, since it could result in a higher block orphan rate for p2pool, but it's not really that bad. The issue comes from the fact that Bitcoin block intervals are 600 seconds, but shares are 30 seconds, which means that we are punishing low-performance nodes by 20x more than is fair.

Another problem with the existing algorithm for choosing which share to base work off of is that all shares are treated the same, regardless of their difficulty, which makes low-diff shares have an advantage for selfish mining attacks. Large-diff shares contribute more work to p2pool, so they should have a greater chance of winning an orphan race, ceteris paribus. But you can't just pick the highest-diff share of all shares in an orphan race, since that would cause a race-to-the-top scenario where the optimal difficulty to use for your shares is the highest diff that a competitor uses plus one. That would also allow a selfish miner to reliably win orphan races against siblings: instead of working on a child share of the best known share, you could just orphan it by creating a sibling with +1 difficulty. So it needs to be probabilistic.

The algorithm I'm thinking of for choosing the winner of an orphan race is this:

1. Take all of the shares at the best chain height (using the current 5th-gen-parent work done heuristic)
2. Find the time that the first of those candidate shares arrived, and call that start_time.
3. For each share, calculate the difference between that share's arrival time and start_time, and call that share_delay.
4. Calculate share_nonorphan_chance = 1 - share_delay/600 (for Bitcoin) for each share. This gives a correction factor for how much less likely than the best share this share would have been to not be orphaned as a bitcoin block. That is, a share that took 6 seconds longer than the fastest share would get a 1% higher orphan risk, and would get a share_nonorphan_chance of 0.99.
5. Calculate share_corr_share_diff = share_diff * share_nonorphan_chance.
6. Pick a share randomly, giving a chance of share_corr_share_diff/sum(share_corr_share_diffs) for each share.
7. Mine using that share as the parent.

If my math/logic is correct, this algorithm would give pretty much everyone a fair chance of winning a share orphan race, regardless of their node performance or share difficulty.

[LevinSwe]

Message written in incorrect english with a touch of swe..
I has spend some time to watch bitcoin-q and p2pool's prompt window.. and summary is:

1. Sometimes, core gets a new block, verify it and then p2pool prompt "new work for workers.. (value13.77btc 1200tx..example)" within a second. No empty block mining. The thing is when core gets the block, it's somtimes about 5-20 seconds old already.

2. Sometimes p2pool prompt, "new work for worker.. (value 12.5btc 0tx) and nothing happens in core for a few seconds.. then the block show up, about 5-20 seconds old, gets verified and within a second p2pool prompt "new work for worker.. (value 13,77btc 1200tx)

I'm a machine programmer that heats my house with miners, not an expert. But I can't see how the empty block mining problem can be solved if not the local bitcoin core gets the block before some other remote p2pool node heard of the block and tells the local nod that there is a new block. I don't think the problem is between p2pool and btc core, is more an internet problem that core needs to have the block faster and before p2pool heard of it. Maybe if p2pool can relay blocks to other nodes, but why should that be faster then core get the block from an other core node?...

And the ophran and doa thing, it's the same amount we inside the pool share, if all involved in this gets about 10% ophran shares, it does not mean that we earn 10% less.. otherwise I had get it all wrong with the share thing and now I'm going to open a beer and paint a ceiling

[kano]

...
And the ophran and doa thing, it's the same amount we inside the pool share, if all involved in this gets about 10% ophran shares, it does not mean that we earn 10% less.. otherwise I had get it all wrong with the share thing and now I'm going to open a beer and paint a ceiling

No you don't all get the same 10%
People with well optimised good setups, or much higher hashing power, get lower %.
Each miner with a lower % is taking from those above with higher %
I guess most don't quite understand that bit.

[LevinSwe]

...
No you don't all get the same 10%
People with well optimised good setups, or much higher hashing power, get lower %...

I get your point, and I think that's fair, if you have a better line, setup and hash rate you have a grater chance to find a block, it can't be a huge % diff.., if you get a high % on your node you can use a public node with total higher hash rate instead. Or use a non decentralized pool. Is it worth to spend time on a orphan fix before the empty block problem is solved?, 5 years ago it really didn't was a problem, but today when the tx limit is forced this is really bad for bitcoin. My opinion is it's better to just save the power and stop mining for a few seconds instead of public a empty block. A decc pool is a wonderful thing, but it is what it is.. I have great respect for forrestv

[frodocooper]

I don't think the problem is between p2pool and btc core, is more an internet problem that core needs to have the block faster and before p2pool heard of it. Maybe if p2pool can relay blocks to other nodes, but why should that be faster then core get the block from an other core node?...

There is a way for your Bitcoin full node to get blocks faster: connect it to one of the Bitcoin FIBRE nodes.

Also, Bitcoin Core 0.14.0 and later has some under-the-hood performance improvements that helps to speed up block propagation and validation. If you don't want the segwit part of 0.14.0 and later but want the faster P2P block propagation and validation that 0.14.0+ offers, you can stick a non-segwit version of Bitcoin Core (e.g., 0.13.0) in between your P2Pool node and 0.14.0+ as a filter. This might introduce a bit of additional latency, however, depending on how much juice your underlying hardware has.

...
No you don't all get the same 10%
People with well optimised good setups, or much higher hashing power, get lower %...

I get your point, and I think that's fair, if you have a better line, setup and hash rate you have a grater chance to find a block, it can't be a huge % diff.., if you get a high % on your node you can use a public node with total higher hash rate instead. Or use a non decentralized pool. Is it worth to spend time on a orphan fix before the empty block problem is solved?, 5 years ago it really didn't was a problem, but today when the tx limit is forced this is really bad for bitcoin. My opinion is it's better to just save the power and stop mining for a few seconds instead of public a empty block. A decc pool is a wonderful thing, but it is what it is.. I have great respect for forrestv

I disagree. P2Pool should provide any and every node with a level playing field. That's one of the points of having a decentralized pool. Having certain nodes that outperform others introduces an undesirable layer of centralization, in terms of orphan rates, etc. So yes, it is worth the time and effort to fix this issue. We should want as many miners as possible to mine at a decentralized pool.

That said, I do agree with you on the point that the empty block problem should be solved first. Demand for block space is already far outstripping supply, and P2Pool should not be a bottleneck in that regard. A decentralized pool should also be helping the network confirm as many transactions as possible. Nevertheless, I do understand that this is a tricky problem to solve; decentralization does seem to throw a spanner in the works here. But until a viable solution is found — I'm optimistic that we will find one soon — I am inclined to agree with jtoomim that mining on an empty block at the right height is a more appropriate compromise than mining on a full block at the wrong height or not mining at all.

[jtoomim]

Maybe if p2pool can relay blocks to other nodes, but why should that be faster then core get the block from an other core node?...

This is actually pretty straightforward to do. P2pool has access to the share chain as an index of recently used and recently seen transactions, so it is generally able to encode a block in much less space than bitcoind can. I think most transactions take 2 bytes to encode (8 bits to refer to which ancestor share first mentioned the transaction, and another 8 bits to refer to the index of the transaction within that share), although the mean is much higher than that, especially when there are a lot of novel transactions.

There are other changes to make to p2pool that are a higher priority (e.g. improving fairness), so it's not likely that I'll implement block propagation via p2pool any time soon.

[frodocooper]

Maybe if p2pool can relay blocks to other nodes, but why should that be faster then core get the block from an other core node?...

This is actually pretty straightforward to do. P2pool has access to the share chain as an index of recently used and recently seen transactions, so it is generally able to encode a block in much less space than bitcoind can. I think most transactions take 2 bytes to encode (8 bits to refer to which ancestor share first mentioned the transaction, and another 8 bits to refer to the index of the transaction within that share), although the mean is much higher than that, especially when there are a lot of novel transactions.

There are other changes to make to p2pool that are a higher priority (e.g. improving fairness), so it's not likely that I'll implement block propagation via p2pool any time soon.

If P2Pool were to include block propagation, would blocks received from other P2Pool nodes then be subsequently forwarded to and verified by the recipient's bitcoind before being mined on, or would the recipient P2Pool node simply assume that the received block is valid?

[jtoomim]

If P2Pool were to include block propagation, would blocks received from other P2Pool nodes then be subsequently forwarded to and verified by the recipient's bitcoind before being mined on?

Yes. p2pool has no code for maintaining a mempool of valid transactions or deciding which transactions to include in a block, so the block would need to be processed by bitcoind and a new block template received from bitcoind before p2pool could start new work. That should take a little less than a second, so it's not a big deal.

[m1n3r]

Hi! I spent alot of time on the build of Litecoin 0.14-dev from sources and p2pool starting. After this I've cloned jtoomim's p2pool fork [1mb_hardforked] and tried to start it with --testnet parametres, but p2pool can't to connect to the network and displays it: '... taking a while. Common reasons for this include all of bitcoind's connection slots being used ... '
Here is the start line:

Code:

user@p2pool:~/Github/p2pool$ python run_p2pool.py --give-author 1 --net litecoin --testnet --bitcoind-rpc-port 19332 --bitcoind-p2p-port 19333 usr fcknpasswd
2017-06-01 01:00:31.225798 p2pool (version 15.0-5-g6f55d05)
2017-06-01 01:00:31.225947
2017-06-01 01:00:31.226030 Testing bitcoind P2P connection to '127.0.0.1:19333'...
2017-06-01 01:00:36.226441  ...taking a while. Common reasons for this include all of bitcoind's connection slots being used...

UPD!
Today I've tried with a last master cloned from https://github.com/forrestv/p2pool.git and got exactly the same error. Damn, really nobody faced this issue?

Code:

user@p2pool:~/Github/p2pool$ python run_p2pool.py --give-author 1 --net litecoin --testnet --bitcoind-rpc-port 19332 --bitcoind-p2p-port 19333 usr fcknpasswd
2017-06-01 11:16:39.083345 p2pool (version 16.0-4-gde1be30)
2017-06-01 11:16:39.083470
2017-06-01 11:16:39.083554 Testing bitcoind P2P connection to '127.0.0.1:19333'...
2017-06-01 11:16:44.083965     ...taking a while. Common reasons for this include all of bitcoind's connection slots being used...

forrestv and/or jtoomim, guys could you explain me what i do wrong and how to fix it? Thanks in advance!

[frodocooper]

Block! Mainnet's first in 19 days!

Hi! I spent alot of time on the build of Litecoin 0.14-dev from sources and p2pool starting. After this I've cloned jtoomim's p2pool fork [1mb_hardforked] and tried to start it with --testnet parametres, but p2pool can't to connect to the network and displays it: '... taking a while. Common reasons for this include all of bitcoind's connection slots being used ... '
Here is the start line:

Code:

user@p2pool:~/Github/p2pool$ python run_p2pool.py --give-author 1 --net litecoin --testnet --bitcoind-rpc-port 19332 --bitcoind-p2p-port 19333 usr fcknpasswd
2017-06-01 01:00:31.225798 p2pool (version 15.0-5-g6f55d05)
2017-06-01 01:00:31.225947
2017-06-01 01:00:31.226030 Testing bitcoind P2P connection to '127.0.0.1:19333'...
2017-06-01 01:00:36.226441  ...taking a while. Common reasons for this include all of bitcoind's connection slots being used...

UPD!
Today I've tried with a last master cloned from https://github.com/forrestv/p2pool.git and got exactly the same error. Damn, really nobody faced this issue?

Code:

user@p2pool:~/Github/p2pool$ python run_p2pool.py --give-author 1 --net litecoin --testnet --bitcoind-rpc-port 19332 --bitcoind-p2p-port 19333 usr fcknpasswd
2017-06-01 11:16:39.083345 p2pool (version 16.0-4-gde1be30)
2017-06-01 11:16:39.083470
2017-06-01 11:16:39.083554 Testing bitcoind P2P connection to '127.0.0.1:19333'...
2017-06-01 11:16:44.083965     ...taking a while. Common reasons for this include all of bitcoind's connection slots being used...

forrestv and/or jtoomim, guys could you explain me what i do wrong and how to fix it? Thanks in advance!

The P2Pool branches that you tried to use are currently not compatible with Litecoin's segwit. You will not be able to mine using those P2Pool branches.

There is, however, a fork of P2Pool that is compatible with Litecoin's segwit. You can find it here: https://github.com/ilsawa/p2pool-ltc

I made a p2pool for ltc with support for SegWit.
If you want, you can try installing from the repository https://github.com/ilsawa/p2pool-ltc

[m1n3r]

frodocooper, thank you very much for your reply! This was the most useful answer in this topic.

[jezus]

Founded UASF( BIP148) segwit is planning.

[frodocooper]

frodocooper, thank you very much for your reply! This was the most useful answer in this topic.

You're welcome, m1n3r. Glad to be of help.

Founded UASF( BIP148) segwit is planning.

Speaking of BIP148, are there any contingency plans in place for P2Pool handling the fork? As far as I'm aware, both mainnet and jtoomimnet are currently not compatible with segwit. So when BIP148 activates — and in the off-chance that it replaces the legacy non-segwit chain, or if circumstances post-BIP148 manage to trigger the 95% segwit threshold on the legacy chain — we P2Pool miners would be hung out to dry.

It's better to be well-prepared as early as possible than to be caught off-guard as the last minute. Just saying.

[tubexc]

frodocooper, thank you very much for your reply! This was the most useful answer in this topic.

You're welcome, m1n3r. Glad to be of help.

Founded UASF( BIP148) segwit is planning.

Speaking of BIP148, are there any contingency plans in place for P2Pool handling the fork? As far as I'm aware, both mainnet and jtoomimnet are currently not compatible with segwit. So when BIP148 activates — and in the off-chance that it replaces the legacy non-segwit chain, or if circumstances post-BIP148 manage to trigger the 95% segwit threshold on the legacy chain — we P2Pool miners would be hung out to dry.

It's better to be well-prepared as early as possible than to be caught off-guard as the last minute. Just saying.

We are already hung out to dry with this ambiguous situation   

[jtoomim]

Speaking of BIP148, are there any contingency plans in place for P2Pool handling the fork? As far as I'm aware, both mainnet and jtoomimnet are currently not compatible with segwit. So when BIP148 activates — and in the off-chance that it replaces the legacy non-segwit chain, or if circumstances post-BIP148 manage to trigger the 95% segwit threshold on the legacy chain — we P2Pool miners would be hung out to dry.

BIP148 states that all miners must signal for SegWit after August 1st. Currently, p2pool is capable of signaling for SegWit, so p2pool is compatible with this phase. If a majority of miners are following BIP148, then SegWit (BIP141) will enter into the LOCKED_IN state for two weeks, followed by the ACTIVE period two weeks after that. If 100% of miners follow BIP148, then the signaling phase will last 2 weeks. (If 10% follow, then 20 weeks, etc.) This plus the LOCKED_IN period means that we will have at least 4 weeks after August 1st before p2pool becomes incompatible with the Bitcoin chain.

It should take a week or less to merge and test SegWit support into jtoomimnet and/or mainnet's code. We'll be fine.

[kano]

Speaking of BIP148, are there any contingency plans in place for P2Pool handling the fork? As far as I'm aware, both mainnet and jtoomimnet are currently not compatible with segwit. So when BIP148 activates — and in the off-chance that it replaces the legacy non-segwit chain, or if circumstances post-BIP148 manage to trigger the 95% segwit threshold on the legacy chain — we P2Pool miners would be hung out to dry.

BIP148 states that all miners must signal for SegWit after August 1st. Currently, p2pool is capable of signaling for SegWit, so p2pool is compatible with this phase. If a majority of miners are following BIP148, then SegWit (BIP141) will enter into the LOCKED_IN state for two weeks, followed by the ACTIVE period two weeks after that. If 100% of miners follow BIP148, then the signaling phase will last 2 weeks. (If 10% follow, then 20 weeks, etc.) This plus the LOCKED_IN period means that we will have at least 4 weeks after August 1st before p2pool becomes incompatible with the Bitcoin some minority alt-coin based on Bitcoin chain.

It should take a week or less to merge and test SegWit support into jtoomimnet and/or mainnet's code. We'll be fine.

Corrected a typo there ...

[veqtrus]

Speaking of BIP148, are there any contingency plans in place for P2Pool handling the fork? As far as I'm aware, both mainnet and jtoomimnet are currently not compatible with segwit. So when BIP148 activates — and in the off-chance that it replaces the legacy non-segwit chain, or if circumstances post-BIP148 manage to trigger the 95% segwit threshold on the legacy chain — we P2Pool miners would be hung out to dry.

BIP148 states that all miners must signal for SegWit after August 1st. Currently, p2pool is capable of signaling for SegWit, so p2pool is compatible with this phase. If a majority of miners are following BIP148, then SegWit (BIP141) will enter into the LOCKED_IN state for two weeks, followed by the ACTIVE period two weeks after that. If 100% of miners follow BIP148, then the signaling phase will last 2 weeks. (If 10% follow, then 20 weeks, etc.) This plus the LOCKED_IN period means that we will have at least 4 weeks after August 1st before p2pool becomes incompatible with the Bitcoin some minority alt-coin based on Bitcoin chain.

It should take a week or less to merge and test SegWit support into jtoomimnet and/or mainnet's code. We'll be fine.

Corrected a typo there ...

If the majority of miners follow BIP148 there will be no chain split since it is a soft fork.

[-ck]

The semantic discussion by whalingoutbox was too offtopic for this thread so I've moved it to its own thread:
https://bitcointalk.org/index.php?topic=1954341.0

[whalingoutbox]

The semantic discussion by whalingoutbox was too offtopic for this thread so I've moved it to its own thread:
https://bitcointalk.org/index.php?topic=1954341.0

The issue of semantics was non-existent. I was asserting I wasn't asking basic questions. I was simply making the implied argument that the posted was a sophist who used lower parts of his/her brain. Basic questions? Really?

So, I can't argue against them not being alleged basic questions?

The argument was that the blocksize from 2009 inspired more processing power in computer in equipment for Bitcoin in contrast to the white paper, which the whitepaper was arguing for decentralization (more than likely with a shared system, like P2Pool). This Satoshi Nakamoto event has had an effect on P2Pool ever since: And I want to see if anyone will admit or deny that and why.

[-ck]

The semantic discussion by whalingoutbox was too offtopic for this thread so I've moved it to its own thread:
https://bitcointalk.org/index.php?topic=1954341.0

There wasn't an issue of semantics. The argument was that the blocksize from 2009 inspired more processing power in computer in equipment for Bitcoin in contrast to the white paper, which the whitepaper was arguing for decentralization (more than likely with a shared system, like P2Pool). This Satoshi Nakamoto event has had an effect on P2Pool ever since: And I want to see if anyone will admit or deny that and why.

Not an issue of semantics. Where were the semantics?

I have no problem with you discussing what you want to discuss it's just not relevant to this p2pool support thread. Hence I've moved it to its own thread. The posts are all still there intact and others are free to respond if they feel inclined but any more posts to this thread along that nature will be removed.

Please don't play the free speech card, this is a private forum and you have to obey the rules of the forum primarily. I'm not even censoring your posts, I'm just moving them to their own topic. But if you keep posting unwanted posts in this thread you will be silenced.