[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[M8BWNNRFMNdak68c]

so can you tell me if i got this right: if i now use this version: https://github.com/jtoomim/p2pool/commits/1mb_segwit and mine on my own node, the following 3 conditions are fulfilled:

1) i join the network with 1.4 PHash visible here: http://p2pool.org/stats/
2) it is not possible to create orpahn blocks,as every node will signal segwit support
3) it will be possible to mine blocks in around three weeks, when there are normal tx as well as segwit tx in the bitcoin network. ( and get fees from both )

[1714001389]

1714001389

[1714001389]

1714001389

[1714001389]

1714001389

[frodocooper]

so can you tell me if i got this right: if i now use this version: https://github.com/jtoomim/p2pool/commits/1mb_segwit and mine on my own node, the following 3 conditions are fulfilled:

1) i join the network with 1.4 PHash visible here: http://p2pool.org/stats/
2) it is not possible to create orpahn blocks,as every node will signal segwit support
3) it will be possible to mine blocks in around three weeks, when there are normal tx as well as segwit tx in the bitcoin network. ( and get fees from both )

1. Yes.

2. Only if the Bitcoin full node that your P2Pool node is connected to is itself signaling for BIP141 (i.e., setting segwit's BIP9 versionbit of bit 1), and if you haven't explicitly configured your P2Pool node to signal differently. The P2Pool code, by default, will use whatever signaling that is included in the block template that is provided by the Bitcoin full node that it is connected to. If you are running Bitcoin Core 0.13.1 or later, Bitcoin Knots 0.13.1 or later, btc1's segwit2x client, or the UASF client, you would be signaling for BIP141.

3. In theory, yes. However, jtoomim's 1mb_segwit code has not yet been fully tested to see if it can actually mine segwit blocks.

Sorry this took so long. I haven't fully tested this to make sure it works on testnet and can make segwit blocks, but if people can help review it and maybe help test it, that would be nice.

https://github.com/jtoomim/p2pool/commits/1mb_segwit

Segwit compatibility stuff starts with the July 19th commits.

...

[OurLink]

Quick question about Pseudo Share settings for you miner when using P2Pool.

Everything I have read says to set your Pseudo Share using ADDRESS+SETTING. In the calculation for SETTING I see where you use the (miner speed * 0.0000016).

My question is what does the 0.0000016 represent?  I can't find any information regarding this calculation anywhere...

[frodocooper]

Quick question about Pseudo Share settings for you miner when using P2Pool.

Everything I have read says to set your Pseudo Share using ADDRESS+SETTING. In the calculation for SETTING I see where you use the (miner speed * 0.0000016).

My question is what does the 0.0000016 represent?  I can't find any information regarding this calculation anywhere...

I've never seen that formula before, nor have I read anywhere that it's a must to manually set a pseudoshare difficulty level (unless you're renting from NiceHash, which requires a minimum pseudoshare difficulty level of 65,536.

There isn't much, if any, of a performance advantage in manually setting a pseudoshare difficulty level. It's usually best to simply leave it to P2Pool to set the pseudoshare difficulty level for you.

[jtoomim]

My question is what does the 0.0000016 represent?

It represents an arbitrary number that some dude thought worked well for making the statistics graphs nice and smooth. It does not affect revenue at all unless you set the difficulty low enough that it overloads your poolserver's CPU or network connection with processing an excessive number of pseudoshares. I think it's better to not touch the pseudoshare difficulty at all, as p2pool's default values are generally reasonable.

[jtoomim]

sawa, those hash > target errors are weird, and I have no good idea what could be causing them. It seems as though the PoW function that p2pool is using to verify the hashes is not the same as the PoW function that the miners are using when solving the shares. I don't know why that would be the case.

The error at the beginning is something that I've never seen before, and is likely related to starting up a new chain. It does not seem to be closely connected to any code that I touched. There's a chance that it could be causing the hash > target errors, but I'm not sure how exactly.

Can you verify that your set-up procedure works fine with https://github.com/p2pool/p2pool master?

Can you try running 1mb_segwit using an existing share chain on an altcoin?

[sawa]

Can you verify that your set-up procedure works fine with https://github.com/p2pool/p2pool master?

Yes, that's the way it worked http://crypto.office-on-the.net:12125/static/

Can you try running 1mb_segwit using an existing share chain on an altcoin?

That's exactly what I did. This can be seen from the previous log:

Code:

2017-08-06 13:45:55.994236 Loading shares...
2017-08-06 13:45:57.582201     1000
2017-08-06 13:45:58.598017     2000
2017-08-06 13:45:59.447829     3000
...

It seems to me that somewhere in the code 1mb_segwit does not take into account that POW_FUNK can differ from POW_FUNC = data.hash256

[Cryptonomist]

Hello

I've been using p2pool for a couple of months now using forrestv's version. Everything worked perfectly. Recently I tried to switch to jtoomim's 1mb segWit fork. However I haven't been able to make it work. I run the p2pool node on the same system as my bitcoin node. I use bitcoin-core version 0.14.2. I've run this system with forrestv's version for months, without any issues.

To get jtoomim's fork I deleted the p2pool directory from the forrestv version, and subsequently I used the command "git clone https://github.com/jtoomim/p2pool.git p2pool_jtoomim_1mb_segwit". Next I used in the folder p2pool_jtoomim_1mb_segwit the command "make".

To start p2pool I use the following command: "screen -d -m -S btcp2pool_jtoomim_1mb_segwit ~/p2pool_jtoomim_1mb_segwit/run_p2pool.py".

I get the following output:

2017-08-09 13:02:23.239586 p2pool (version 15.0-5-g6f55d05)
2017-08-09 13:02:23.239691
2017-08-09 13:02:23.239779 Testing bitcoind RPC connection to 'http://127.0.0.1:8332/' with username 'bitcoinrpc'...
2017-08-09 13:02:23.704666     ...success!
2017-08-09 13:02:23.704780     Current block hash: 101631b2ef7bd157c712553100f7e3ca23fd87561339640
2017-08-09 13:02:23.704829     Current block height: 479797
2017-08-09 13:02:23.704865
2017-08-09 13:02:23.704918 Testing bitcoind P2P connection to '127.0.0.1:8333'...
2017-08-09 13:02:23.707403     ...success!
2017-08-09 13:02:23.707479
2017-08-09 13:02:23.707539 Determining payout address...
2017-08-09 13:02:23.707674     Loaded cached address: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2017-08-09 13:02:23.711848     ...success! Payout address: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2017-08-09 13:02:23.711919
2017-08-09 13:02:23.712004 Loading shares...
2017-08-09 13:02:23.712177     ...done loading 0 shares (0 verified)!
2017-08-09 13:02:23.712248
2017-08-09 13:02:23.712294 Initializing work...
2017-08-09 13:02:24.336359     ...success!
2017-08-09 13:02:24.336479
2017-08-09 13:02:24.336545 Joining p2pool network using port 9333...
Unhandled Error
Traceback (most recent call last):
Failure: twisted.internet.error.DNSLookupError: DNS lookup failed: address 'vps.forre.st' not found: [Errno -2] Name or service not known.

2017-08-09 13:02:24.618120     ...success!
2017-08-09 13:02:24.618252
2017-08-09 13:02:24.618807 Listening for workers on '' port 9332...
2017-08-09 13:02:24.705814     ...success!
2017-08-09 13:02:24.705986
2017-08-09 13:02:24.706078 Started successfully!
2017-08-09 13:02:24.706151 Go to http://127.0.0.1:9332/ to view graphs and statistics!
2017-08-09 13:02:24.706233 Donating 1.0% of work towards P2Pool's development. Thank you!
2017-08-09 13:02:24.706301 You can increase this amount with --give-author argument! (or decrease it, if you must)
2017-08-09 13:02:24.706358

2017-08-09 13:02:27.706779 P2Pool: 0 shares in chain (0 verified/0 total) Peers: 0 (0 incoming)
2017-08-09 13:02:27.706959  Local: 0H/s in last 0.0 seconds Local dead on arrival: Expected time to share:
2017-08-09 13:02:42.783897 P2Pool: 0 shares in chain (0 verified/0 total) Peers: 0 (0 incoming)
2017-08-09 13:02:42.784038  Local: 0H/s in last 0.0 seconds Local dead on arrival: Expected time to share:
2017-08-09 13:02:57.785560 P2Pool: 0 shares in chain (0 verified/0 total) Peers: 0 (0 incoming)

2017-08-09 12:58:01.356016 Incoming connection to peer 71.0.29.168:40692 established. p2pool version: 1600 '16.0-27-g18367de'
2017-08-09 12:58:01.678196 Lost peer 71.0.29.168:40692 - Connection to the other side was lost in a non-clean fashion.
in handle_share_hashes:
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion.

The time descrepancy in the last output block is because I copied it from the output of a different attempt, but I get the same error in all the attempts I do.

There are a couple of things I noticed:
1) No shares are loaded.
2) My node isn't able to connect to other nodes in the network. Sometimes I get one connection, but the connection is immediately terminated. From the output I think I can deduce that the node it connects to is p2pool version: 1600 '16.0-27-g18367de', which isn't a jtoomim fork but a forrestv fork. Is this correct? This is strange in my opinion. The fact that my node doesn't find any other nodes is also strange.
3) I get the error "Failure: twisted.internet.error.DNSLookupError: DNS lookup failed: address 'vps.forre.st' not found: [Errno -2] Name or service not known". I've read on a forum that this is caused by vps.forre.st being offline. The sollution that was suggested is to add "-n p2pool.org" when starting p2pool: so the command I use is "screen -d -m -S btcp2pool_jtoomim_1mb_segwit ~/p2pool_jtoomim_1mb_segwit/run_p2pool.py -n p2pool.org". However this hasn't changed anything. I keep on getting the same output and errors.

I've also tested the forrestv version again and this version works again perfectly. I don't get any error messages.

Can anyone tell me what I'm doing wrong? It's probably something stupid, but I don't see my error.

Thank you in advance.

[Duce]

Go back and read here this might help. https://bitcointalk.org/index.php?topic=18313.msg19006724#msg19006724

[OurLink]

My question is what does the 0.0000016 represent?

It represents an arbitrary number that some dude thought worked well for making the statistics graphs nice and smooth. It does not affect revenue at all unless you set the difficulty low enough that it overloads your poolserver's CPU or network connection with processing an excessive number of pseudoshares. I think it's better to not touch the pseudoshare difficulty at all, as p2pool's default values are generally reasonable.

I think I found the answer in the code...

Code:

if desired_pseudoshare_target is None:
            target = 2**256-1
            local_hash_rate = self._estimate_local_hash_rate()

2 to the power of 256 - 1 in my Excel yields 1.16e+77 and yes it does look like some arbitrary number...

[jtoomim]

2 to the power of 256 - 1 in my Excel yields 1.16e+77 and yes it does look like some arbitrary number...

Excel doesn't have bignum support. Python does.

Code:

>>> 2**256-1
115792089237316195423570985008687907853269984665640564039457584007913129639935L
>>> hex(_)
'0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffL'

[ruey220]

is the windows version of the new fork going to be released anytime soon?

If I am running the old 16.0 version of p2pool, am I still going to be part of any blocks that are discovered by the new fork? If so why then is the calculation of the rewards different on each separated network?

It would appear beneficial to the cause of p2pool miners if we combine the hashing power between both networks just so we can actually get closer to finding a block, since the sub 2ph hashing power is not enough to get a block anytime soon with today's difficulty in the bitcoin network. I have some hashing power going to other pools for now until we can address this issue, hopepfully sooner than later because I like the benefits of using p2pool over other mining pools which you basically make dust with if your hashing power is much less compared to what else is out there.

[kano]

2 to the power of 256 - 1 in my Excel yields 1.16e+77 and yes it does look like some arbitrary number...

Excel doesn't have bignum support. Python does.

Code:

>>> 2**256-1
115792089237316195423570985008687907853269984665640564039457584007913129639935L
>>> hex(_)
'0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffL'

Which has nothing to do with the question or answer

The aim on a normal pool is to get around 18 shares per minute, to ensure variance isn't too high, and of course you aren't submitting too many shares.

Back in the dark ages when we mined with GPUs or FPGAs it would take around 20 or more seconds to find a 1 diff share, but as mining got faster, finding 1 diff shares got faster, of course.
At some point most pools realised that you had to mine higher difficulty shares, otherwise you were sending way too many share, too quickly, to the pool.

So if we say 18 shares per minute, then 3.33.. seconds per share, then you can work out how to convert a hash rate to 18 shares per minute as so:

A 1 diff share averages 2^32 hashes per share (and 2^32 is roughly 4.3 billion)
Thus a 4.3GHs miner gets approximately 1 share per second at 1 diff
Thus a 1.3GHs miner gets approximately 1 share per 3.33.. seconds at 1 diff

So to get one share per 3.33.. seconds with an XHs miner, you need to divide it's hash rate by 2^32*0.3 (i.e. 1.3GHs) = multiply it by 7.76*10^-10  to get the difficulty it should mine at.
e.g. a 13THs miner = 13*10^12 * 7.76*10^-10 = ~10,089 Diff

So ... I guess that number is some vague relation to 7.76*10^-10 ... you'd have to check the code, I hate python so I'm not gonna work that part out

[jtoomim]

is the windows version of the new fork going to be released anytime soon?

You mean binaries? I am a bit too busy to build binaries for non-release beta versions. You can still run from source on Windows, though.

If I am running the old 16.0 version of p2pool, am I still going to be part of any blocks that are discovered by the new fork? If so why then is the calculation of the rewards different on each separated network?

No.

It would appear beneficial to the cause of p2pool miners if we combine the hashing power between both networks just so we can actually get closer to finding a block

Agreed. Before we can merge the two pools, we need to get the 1mb_segwit branch tested on testnet and the bugs that prevent it from working with altcoins fixed. Can you help with that?

[jtoomim]

sawa, can you try commenting out lines 350 through 357 in p2pool/work.py on your emerald test server and see if that helps? It's possible that target = min(target, ...) line is doing something strange on alts. The code you should comment out to test is this:

Code:

            else:
                # If we don't yet have an estimated node hashrate, then we still need to not undershoot the difficulty.
                # Otherwise, we might get 1 PH/s of hashrate on difficulty settings appropriate for 1 GH/s.
                # 1/100th the difficulty of a full share should be a reasonable upper bound. That way, if
                # one node has the whole p2pool hashrate, it will still only need to process one pseudoshare
                # every 0.3 seconds.
                target = min(target, 1000 * bitcoin_data.average_attempts_to_target((bitcoin_data.target_to_average_attempts(
                    self.node.bitcoind_work.value['bits'].target)*self.node.net.SPREAD)*self.node.net.PARENT.DUST_THRESHOLD/block_subsidy))

It's probably something else, though. I'll probably not have time to look deeply into this until the weekend.

[jtoomim]

We just added another 0.4 PH/s to jtoomimnet. That makes about 1.8 PH/s total.

[frodocooper]

On a side note, my 1mb_segwit node seems to be using approximately twice the amount of memory as my old lowmem node did.

Memory consumption on my 1mb_segwit node was recorded as 1.38 GB approximately one and a half hours after the node was started up for the first time, roughly five days ago. The latest record (just over one and a half hours ago) marks memory consumption as 2.18 GB. Memory consumption seems to grow at an average rate of roughly 200 MB per day.

Memory consumption on my old lowmem node never exceeded 1.5 GB, even after more than seven days of continuous running.

This is on an Amazon EC2 c4.large instance, which has 3.75 GB of total available memory. Bitcoin Core 0.14.2 is configured to use 700 MB of memory for its UTXO cache (it used to be 1 GB when I was running lowmem) and the default 300 MB for its mempool. This leaves less than 2.75 GB for my 1mb_segwit node to use, and it is already fast reaching that mark.

The blockmaxsize and blockmaxweight parameters have been set to 1000000 and 4000000 respectively.

Also, whereas efficiency on my old lowmem node would hover around the 90% mark, efficiency on my 1mb_segwit node hovers around the 80% mark, with a noticeably higher mean DOA rate (10%-20% on lowmem and as high as 30% on 1mb_segwit).

GetBlockTemplate latency is also noticeably higher on 1mb_segwit. Mean GBTL would settle at just north of 0.4 s on lowmem, while mean GBTL hovers at around 0.55 s on 1mb_segwit.

[jtoomim]

I pushed a small change to 1mb_segwit that adds compatibility for pre-fork segwit2x mining with e.g. btc1. The 1mb_hardforked and lowmem branches do not need this change, as it's only the 1mb_segwit branch (and veqtrus's segwit PR) that have this issue.

The issue is that p2pool will refuse to mine if a softfork is detected which is not in the p2pool/network/(coinname).py:SOFTFORKS-REQUIRED list. It may be best to remove that check entirely, as it seems to be explicitly anti-forwards-compatible, which sounds like it may be a bad idea. Still thinking it over.

frodocooper, I'm now running a 1mb_segwit node and a lowmem node in parallel on the same machine. I'll keep an eye on memory consumption. If I can replicate it, I may look into the issue further and try to solve it if I can do so quickly.

My medium-term goal for fixing the p2pool CPU/RAM performance issues is to remove all transaction references (except the merkle root hash and maybe the coinbase transaction) from the consensus protocol and p2p layer. This will reduce RAM, CPU, and network usage by around 90% or more, but it will mean that p2pool's fast block propagation algorithm will no longer work. Given that p2pool's fast block propagation is no longer state-of-the-art, and has about 10x higher transmission time than the state of the art (Bitcoin FIBRE), and is slighly slower than the near-universal Compact Blocks and xthin protocols, and is about 100x more CPU intensive due to being written in Python, I do not think that the current p2pool fast block propagation tech is worth keeping in the codebase. If I get bored, I might try to replace the fast block propagation code with something else that runs outside of the p2pool consensus layer, maybe using bloom filters and/or IBLTs, but I think the current version is causing more harm than good.

If I remove p2pool's fast block prop code, I might add a check at startup (for the bitcoin and bitcoincash networks only) to make sure that bitcoind has some sort of fast block prop set up, whether it's xthin, CB, Falcon, or FIBRE, and make p2pool not mine at all unless one is present or unless a command-line override flag is set. This would be to prevent lazy or incompetent p2pool sysadmins from driving up p2pool's orphan rate.

[frodocooper]

frodocooper, I'm now running a 1mb_segwit node and a lowmem node in parallel on the same machine. I'll keep an eye on memory consumption. If I can replicate it, I may look into the issue further and try to solve it if I can do so quickly.

Thanks, jtoomim.

Update: I restarted my node two hours ago. Memory consumption 15 minutes after startup was recorded as 1.84 GB. Memory consumption then stabilized and remains at 1.85 GB as of 20 minutes ago (i.e., just over two hours after the restart).

If I remove p2pool's fast block prop code, I might add a check at startup (for the bitcoin and bitcoincash networks only) to make sure that bitcoind has some sort of fast block prop set up, whether it's xthin, CB, Falcon, or FIBRE, and make p2pool not mine at all unless one is present or unless a command-line override flag is set. This would be to prevent lazy or incompetent p2pool sysadmins from driving up p2pool's orphan rate.

Would it be possible to also include checks to make sure that each P2Pool node's bitcoind has its blockmaxsize and blockmaxweight parameters set to the network's maximum limits (i.e., 1 MB and 4 MB respectively for Bitcoin and whatever limits Bitcoin Cash is using), or at least as close to the limits as possible? There may yet be a significant number of P2Pool nodes that continue to run with Bitcoin Core's default blockmaxsize and blockmaxweight configurations of 750 kB and 3 MB respectively.

[jtoomim]

That is possible, but I would deem that worthy of a warning at most. I suspect that that misconfiguration is not common enough to merit the programming time.