[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[Xantus]

me too, usind p2pool_win32_V16. on Windows 10. But Old version, Segwit2x is an problem for me up to now.

You misunderstood, I have been running from source on windows and made my own exe using py2exe.

i using this version : http://u.forre.st/u/wanckfqm/p2pool_win32_16.0.zip found on http://p2pool.in. is an Windows Executabe file (simpel .exe)

[1713537950]

1713537950

[Duce]

So which version will work with segwit, the one that is supported by Bitcoin core? Is that 1Meg now lowmem version? I would like to stay with P2P for Bitcoin and not fork to an alt coin. After reading the post in the link provided this again makes sense to me https://bitcointalk.org/index.php?topic=2090805.0.

It seems that there is a divide on which direction P2P will take. We have the old fork still running that will not be compatible when segwit is implemented around August 23rd, true? Then the the 1Meg or lowmem version that would be the correct version for segwit, true? Then we have segwit 2 which will be an alt coin.

[veqtrus]

It seems that there is a divide on which direction P2P will take. We have the old fork still running that will not be compatible when segwit is implemented around August 23rd, true? Then the the 1Meg or lowmem version that would be the correct version for segwit, true? Then we have segwit 2 which will be an alt coin.

"1Meg"+segwit requires 2x which will be an altcoin. "1Meg"+segwit is based on my PR against the main p2pool repo so when it will be merged that will be the correct version for bitcoin.

[frodocooper]

Who receives percentage for running the p2pool code (forrestv) is not called voluntary.

It is, when each and every P2Pool node operator is free to edit (and even disable) this percentage with the --give-author parameter.

Show you some respect for a member with many posts helping newbies with his problems in this forum.

I don't see how disparaging P2Pool's volunteer developers helps anyone.

It seems that there is a divide on which direction P2P will take. We have the old fork still running that will not be compatible when segwit is implemented around August 23rd, true?

Not entirely. Provided that whichever fork of P2Pool you are on does not hard-fork into a new segwit-only sharechain, and that your P2Pool node is connected to a segwit-supporting Bitcoin full node (e.g., Bitcoin Core 0.13.1 or later), your P2Pool node should still continue to work without issues even after segwit activates on the Bitcoin network, as the segwit soft-fork does not require miners to produce segwit-style blocks.

[jtoomim]

Duce, there is disagreement about what will be Bitcoin and what will be the altcoin. I think SegWit2x will be the branch that survives and will be called Bitcoin. Others (including the owner of this forum, who often disallows dissenting opinions) think that No2x will be the branch that survives. Simply asking for the branch that is not the altcoin is ambiguous.

If you want to mine Segwit with Bitcoin Core, that is fine with me, as long as you are aware of the risks of choosing the chain with a hashrate minority. We have about 2 months until the fork happens. When I get a chance (this weekend? earlier?), I'll modify the commit from veqtrus that makes p2pool require that bitcoind have the same softforks flags set as p2pool (i.e. segwit and/or segwit2x) and change it so that it only displays a warning message or give it a command-line override or something. I'll do what I can to make sure that the share chain splits cleanly with or before the splitting of the Bitcoin blockchain.

For now, the lowmem branch will work fine for mining with Bitcoin core. lowmem does not support segwit, so you will need to upgrade before Segwit is activated next week, but after I submit the changes that allow non-segwit2x clients to run on 1mb_segwit.

[Blue Bear]

A lot of people stopped mining here a long time ago because there is no non-linux OS support.

I have been running p2pool on windows only and had no problem.

I am a newbie here and am running on widows as well Frodocooper was very helpful to me when I started

[kano]

Duce, there is disagreement about what will be Bitcoin and what will be the altcoin. I think SegWit2x will be the branch that survives and will be called Bitcoin. Others (including the owner of this forum, who often disallows dissenting opinions) think that No2x will be the branch that survives. Simply asking for the branch that is not the altcoin is ambiguous.
...

The answer is actually pretty straight forward, but doesn't always allow for a decision before-hand.

Bitcoin is a Peer2Peer system.
If someone wants a centrally controlled system, by Core, Ver, or some other minority, then they are the altcoin.
What this forum advocates has nothing to do with that either.

Bitcoin is what the majority of hash rate is mining.
If an ambiguous fork comes along - like BCH - where you don't know in advance who is switching to it, then that will become quite clear, soon after the fork, what is still Bitcoin - again, the majority.

The problem with the upcoming 2x is actually BIP91.
Some believed BIP91 was the /NYA/ agreement, but instead it was simply a lock in for segwit, and nothing more.
Yeah it's not hard to fool people

While most blocks show /NYA/ in their coinbase transaction, that doesn't affect the upcoming 2x change.

Since I'm a small pool, I'm sitting in the rafters waiting to see what happens, I can't affect Bitcoin in any significant way

[jtoomim]

I generally agree with what Kano wrote above.

The problem with the upcoming 2x is actually BIP91.
Some believed BIP91 was the /NYA/ agreement, but instead it was simply a lock in for segwit, and nothing more.
While most blocks show /NYA/ in their coinbase transaction, that doesn't affect the upcoming 2x change.

The percentage of blocks signing /NYA/ is currently floating around 89% to 92%. This is why I think that Segwit2x is the future of Bitcoin. The percentage supporting BIP91 was similar when BIP91 was active.

According to the NYA, bit 4 was intended to be a signal for Segwit2x, and when btc1 is queried for the required softforks via GBT, btc1 has included segwit2x since bit 4 was triggered. Bit 4 is also supposed to be BIP91, so bit 4 is ambiguous. /NYA/ is unambiguous, but non-binding.

It's worth mentioning that Segwit2x can only exist if it has a majority of the hashrate. If your definition of Bitcoin is the post-fork chain with the most hashrate, then Segwit2x will either be Bitcoin or it won't exist. It's only if your definition of Bitcoin is Bitcoin Core that Segwit2x can be an altcoin.

Whatever. It shouldn't be too hard to get p2pool to support all three Bitcoins. It looks like I'll have a little bit of time later tonight, so I'll see what I can get done.

[kano]

...
It's worth mentioning that Segwit2x can only exist if it has a majority of the hashrate. If your definition of Bitcoin is the post-fork chain with the most hashrate, then Segwit2x will either be Bitcoin or it won't exist. It's only if your definition of Bitcoin is Bitcoin Core that Segwit2x can be an altcoin.
...

Yes Segwit2x is an example of knowing the major fork in advance, it won't activate until it is the major fork, so should be no issue for miners on p2pool ... unless they want to back a centrally controlled minor fork ...

[jtoomim]

I've pushed two changes to https://github.com/jtoomim/p2pool/commits/1mb_segwit:

1. Allow mining with clients that do not support all locked-in or active forks (e.g. Bitcoin Core which does not support segwit2x) if p2pool is run with the --allow-obsolete-bitcoind command-line option.
2. Switch back to using time.time() in order to hopefully fix Xantus's "exceptions.OverflowError: mktime argument out of range" error on Windows

[chasaamm]

I've pushed two changes to https://github.com/jtoomim/p2pool/commits/1mb_segwit:

1. Allow mining with clients that do not support all locked-in or active forks (e.g. Bitcoin Core which does not support segwit2x) if p2pool is run with the --allow-obsolete-bitcoind command-line option.
2. Switch back to using time.time() in order to hopefully fix Xantus's "exceptions.OverflowError: mktime argument out of range" error on Windows

If a miner don't uses Bitcoin Core client to set up his p2pool node, witch client should he use and where to download it?

[bitsolutions]

According to the NYA, bit 4 was intended to be a signal for Segwit2x, and when btc1 is queried for the required softforks via GBT, btc1 has included segwit2x since bit 4 was triggered. Bit 4 is also supposed to be BIP91, so bit 4 is ambiguous. /NYA/ is unambiguous, but non-binding.

In the btc1 code the HF codepath was activated by bit 1, bit 4 was used only for BIP91.

It's worth mentioning that Segwit2x can only exist if it has a majority of the hashrate. If your definition of Bitcoin is the post-fork chain with the most hashrate, then Segwit2x will either be Bitcoin or it won't exist. It's only if your definition of Bitcoin is Bitcoin Core that Segwit2x can be an altcoin.

Segwit2x has wipeout protection for full nodes(but not SPV wallets) so technically it can be a minority fork.

[belcher]

I'm sure this has been discussed before but this thread is very large and hard to search, and the other discussion I found didn't answer.



What are the downsides to increasing the PPLNS windows/length of the sharechain? The only ones I can think of are a bigger coinbase transaction and more dust outputs (because more individual miners would get rewarded).

Everyone's been thinking about Lightning Network lately, I wonder if there's a way to have p2pool mine directly to a LN payment channel and have LN payments trustlessly made (somehow) to pay all the individual miners. This would avoid the problem of a large coinbase tx and many dust outputs. The backout transaction in case of a unilateral close could be paying to all the individual p2pool addresses similar to the coinbase today.

I'm thinking of ways to reduce variance of p2pool.

You could also reduce variance by reducing the share interval, but its already 30 seconds and reducing it further would increase stales more I suppose.

Although now right p2pool's variance is probably more caused by it's low hashrate compared to other pools, so it's a bit of a chicken-and-egg situation.

Is there somewhere to check whether larger miners are setting their share difficulty too high? That it negatively affects small miner's variance.

[jtoomim]

What are the downsides to increasing the PPLNS windows/length of the sharechain?

For p2pool specifically, or for pools in general? I'm assuming the former.

P2pool needs to store all of the shares in the sharechain plus a few. If the sharechain is 8460 shares long, this means that p2pool needs to store at least 8460 shares in its sharechain. In practice, p2pool stores about 2x as many shares as that, so the sharechain length is typically around 16,000 shares in length. Each share references each transaction in the block that that share would have made, using somewhere between 2 bytes (in my lowmem branch for transactions that have been included in previous shares) and 70-ish bytes (for new transactions, including Python's object overhead). With roughly 2000 transactions per share, that turns into about 10 kB (lowmem) to 50 kB (p2pool master) per share, or 160 MB (lowmem) to 800 MB (master) of memory consumption. Multiply those numbers by something like 4 if you're using pypy.

There are also some minor CPU usage costs on sharechain size when processing shares (e.g. calculating payouts for the coinbase), but they're mostly alleviated by using a skiplist in the work-done-by-user calculations.

I would like to strip all transaction data out of the share data structure in the share chain in order to cut this memory footprint issue and reduce the CPU requirements for processing shares, but until that is done, increasing the share chain length is a bad idea.

The coinbase transaction would not change much in size for typical p2pool use because it almost always contains at least one share per active p2pool user anyway. A single ASIC miner is generally able to mine more than one share per 3 days (the current PPLNS length target).

The coinbase transaction size is not currently a significant problem for p2pool, as it only takes up a few kB once every few weeks. If p2pool grew bigger and if blocks got more frequent, it could make sense to pseudorandomly determine which users get paid out from each block, or to keep a separate balance state for each user that is independent from the payouts, and make payouts only when that balance state exceeds some reasonable threshold.

I'm thinking of ways to reduce variance of p2pool. You could also reduce variance by reducing the share interval, but its already 30 seconds and reducing it further would increase stales more I suppose.

It is also possible to change the share difficulty on a per-user basis, so that large miners make fewer shares per PH than small miners. However, there's currently a design flaw in p2pool's head selection algorithm (the thing that determines which share gets orphaned when there is competition) that disincentivizes the voluntary creation of high-difficulty shares, so that would need to be fixed first. See https://bitcointalk.org/index.php?topic=18313.msg19248232#msg19248232 for a discussion of that issue plus a few other related fairness issues.

But the main source of variance is block variance from p2pool's low hashrate. The two main reasons for p2pool's low hashrate are the fairness problems and the (related) performance problems. We lost 10 PH/s from whortonda a couple months back because he was getting high DOA rates and earning about 3% less than most others on the pool.

Is there somewhere to check whether larger miners are setting their share difficulty too high? That it negatively affects small miner's variance.

You can go onto a p2pool node and use the share chain explorer to look at different shares, the difficulty used for that share, the mining address that created it, and the minimum difficulty that p2pool would have allowed for that share if the miner had chosen a low-diff share.

[Xantus]

I've pushed two changes to https://github.com/jtoomim/p2pool/commits/1mb_segwit:

1. Allow mining with clients that do not support all locked-in or active forks (e.g. Bitcoin Core which does not support segwit2x) if p2pool is run with the --allow-obsolete-bitcoind command-line option.
2. Switch back to using time.time() in order to hopefully fix Xantus's "exceptions.OverflowError: mktime argument out of range" error on Windows

Thanks jtoomim,
i downloaded an started now  (before go to work). up to now it looks fine.
lets see this evening ...

update : 9:45pm :
still runs, shares are downloaded i think :

Code:

P2Pool: 17707 shares in chain (10554 verified/17712 total) Peers: 12 (6 incoming)

-> is that ok that only a part is verified ?

oh, and the Local hash rate lock a little bit strange to me, seams because the difficulty for mining engines is very high.
Isn´t that an dynaminc think vor every node his selv harshrate ?
but im on the right p2pool part an back online again ;-)   - thanks so far !
->http://131.117.146.223:9332/static/

[jtoomim]

Xantus, I see a very high DOA rate on your node, roughly 50%. I suspect your CPU is not fast enough to keep up with the task. You may want to try switching to pypy instead of using Python 2.7. Pypy's CPU usage is about 3x lower than Python 2.7's, although its memory consumption is about 3x higher. I haven't installed Pypy on Windows before, but I expect you'll need to reinstall twisted, zope.interface, and whatever other things you had to install with Python 2.7.

Sorry for not mentioning this earlier; I should have predicted this issue and sent you down the pypy path from the start.

Once I get caught up with getting my p2pool code to support all forks and altcoins, I'll work on cutting down CPU, RAM, and bandwidth requirements for running a node. For the time being, though, we'll need to meet p2pool's high resource requirements with pypy and mid-range hardware.

Other than that, it looks like your node is working.

[Xantus]

Realy CPU usage looks ok to me :
following datas are Peak (i cat this Hardkopies on peaks)
http://www.c-xantus.de/syslast1.jpg
http://www.c-xantus.de/syslast2.jpg
http://www.c-xantus.de/syslast3.jpg

more strange vor me is this dificult think :
http://www.c-xantus.de/s7.jpg

but i will test pypy at this weekend, to test it.

[jtoomim]

You have 2 physical cores, 4 logical cores on your CPU. P2pool is single-threaded, so it can use at most 25% of your CPU (in terms of logical cores). Your CPU is pegged at a little over 25% for the Pythonw (p2pool) process. Your CPU is unable to keep up. Try pypy.

Keep in mind that utilization is a misleading metric. What really matters is latency -- that is, how long, in milliseconds, it takes p2pool to process an incoming share and hand out new work to workers. On my nodes, that's normally around 50 ms to 150 ms on pypy, and 250 ms to 1000 ms on Python. With an average of 30 seconds per share, latency of 1000 ms corresponds to a 3.3% DOA rate. If your CPU is pegged at 100% for one core on the p2pool process, that means that work is queueing up, and before you can process an incoming share, you have to process all the other data that has been flung at your p2pool node before it can process the recently-broadcast share. This can turn 1000 ms latency into 3000 ms latency.

You can think of it like a leaky bucket: if you have a bucket with a hole in the bottom and a faucet pouring into it, you can measure how long it takes for any given drop of water leaving the faucet to make it through the hole in the bottom of the bucket. If the hole is bigger than the faucet, then there will never be much water in the bucket, and the amount of time it takes for a drop to pass through will be small. But if the faucet is bigger than the hole, the bucket will fill up to the top and start to overflow the sides, and it will take very long for drops to make it through the hole, and some drops will never make it at all. Your bucket is overflowing. Some data that gets sent to your node never gets processed, and the data that does gets processed after a substantial delay.

The high difficulty, on the other hand, is normal for p2pool, and not responsible for your high DOA rates. Actually, since you're CPU bound, you would get slightly (but not significantly) better performance with an even higher difficulty, since that would mean your miner would send fewer shares to your node and require less processing on your node. But again, that's not significant.

Go switch to pypy and it should work fine. Or yell at me and get me to make the code faster.

[Xantus]

i dont yell to any one . Because i am thanksfull you talking to my an spend me these time.
i will switch to pypy

by the way, i dont think about these Multicore think  

Edit 18.08.17 - 6PM :
p2pool on pypy is not able to me  . i dont get twistet installed to pypy.
pip is working but i get many error until "pip install twisted"
-> so, if somone does have time to help me please send me an PM (dont want to spam in here)

[jtoomim]

Post your error so that others can learn from it. You probably aren't the last one who will want to get pypy running on Windows.

You might be using the version of pip bound to python2.7 instead of the version of pip bound to pypy. You may have to install setuptools for pypy (I'm not sure it comes installed by default, unlike Python), or install twisted via a mechanism other than pip or easy_install.