jtoomim_performance is an obsolete branch from 3 years ago and is not supported.

To run p2pool from source, you need to install python 2.7, twisted, zope.interface, and a couple other things. There are some instructions on the github page for what you need to install:

https://github.com/jtoomim/p2pool/tree/1mb_segwit


I've never actually run it from source on Windows, so beyond those instructions I can't help you. Maybe someone else has done it?

veqtrus, what I personally prefer or what you personally prefer is irrelevant. SegWit2x has 92.76% support among miners at the moment, well above the needed threshold of 80% for activation. The only reason SegWit is being enabled is because it is part of SegWit2x. P2pool needs to support the majority blockchain.

Supporting minority blockchains like Bitcoin Core or Bitcoin Cash is desirable if it can be implemented in a way that does not conflict with support for the majority chain. Until that code is written, p2pool should consider minority chains to be unsupported.

You don't like 2x. I don't like SegWit. It doesn't matter. The majority has spoken.

https://coin.dance/blocks/proposals

If you wish to maintain your own no-2x fork of p2pool with a separate share chain, that would be appropriate.

Duce, that's not a bug in p2pool. It's because Bitcoin Core 14.2 does not support Segwit2x, which was enacted by 80-90% of miners about 2 weeks ago.  All versions of Bitcoin Core are about to become incompatible with the majority of the Bitcoin hashrate, as they do not support Segwit2x. If you continue to use Bitcoin Core, it is expected that you will be on the minority fork of a chainsplit in 2-3 months, when the 2x part of SegWit2x is enabled. I recommend that you either switch to btc1 or lobby Bitcoin Core to change their policy regarding SegWit2x.

P2pool does not currently have any functionality to handle chainsplits gracefully. If you are aware of the SegWit2x issue and wish to continue to mine with Bitcoin Core even though doing so will support a chainsplit, let me know and I will consider adding code to p2pool to make the p2pool sharechain split in parallel to the Bitcoin chain.

I think I've solved the performance issue with 1mb_segwit. It looks like the PR that veqtrus put on my github to correct what he perceived as a couple of omissions on my part for the old (2014?) jtoomim_performance code had an error. Specifically, it called VariableDict.transitioned.happened() on the known_txs variable whenever a transaction was added or deleted from known_txs variable. The purpose of the jtoomim_performance code was to only call transitioned.happened() when a full assignment was done on known_txs var, since transitioned.happened() calls subsequent code that reviews the full contents of known_txs -- O(n) versus 'mempool' size-- instead of just the added or removed items -- O(1) versus 'mempool' size. Even worse, the place that veqtrus added the call to transitioned.happened() causes this O(n) operation to happen once for every transaction that's added and once for every transaction that's removed, instead of once per batch of transactions added and removed as per the p2pool master code, so in many circumstances that PR would perform around 10x worse than the original p2pool master code.

This bug should not affect veqtrus's base segwit PR on the p2pool/p2pool github, but it likely will affect the branch veqtrus has been working on in his own github repo that merges some performance fixes (plus this performance regression masquerading as a code refactor) into his segwit branch.

I reverted that code from 1mb_segwit and pushed the changes to github, and am now running it on ml.toom.im:9332, and it seems to be performing as well as the lowmem branch. I will likely switch all of my nodes over to 1mb_segwit before the eclipse.

Xantus:

The old p2pool code (https://github.com/p2pool/p2pool master) does not support SegWit. There are two branches of code that do support SegWit. One is veqtrus's SegWit PR, and the other is my 1mb_segwit code, which is derived from veqtrus's code.

The main difference between veqtrus's version and my version is that my version is based on the jtoomimnet hardfork, which I made a few months ago to solve a problem with p2pool being unable to create 999 kB blocks. The jtoomimnet code handles higher transaction volumes than the mainnet p2pool code, but it also contains several performance optimizations, so overall resource requirements are about the same for the two networks. The jtoomimnet code also contains some changes to the protocol that reduce orphan rates, which makes jtoomimnet more fair to large and small miners. The creation of jtoomimnet required changing consensus rules of p2pool, so it had to be a hardfork. I wrote the code in such a way to leave the original p2pool functioning normally, so only those who wanted to use my upgrades needed to switch their code. Thus, we currently have two networks.

I want to do some more testing and optimization on the 1mb_segwit branch first, but I intend to switch jtoomimnet over to 1mb_segwit before SegWit activates.

We lost power at our datacenter this morning. When our p2pool servers automaticallhy restarted after we regained power, they ran the branch of p2pool I happened to have checked out on my nodes, which was 1mb_segwit. This was not intentional, as I meant to run lowmem. We mined a bunch of v33 shares, and you will start to see a message saying:


if you're on jtoomimnet. Please disregard this message for now. We will be switching to 1mb_segwit eventually in order to support SegWit/SegWit2x, but not quite yet, as there are still some performance bugs at least in the 1mb_segwit branch.

has switched over to jtoomimnet.




Edit: Nope, we just got a short-term nicehash renter or something.

I'm not seeing higher memory consumption on 1mb_segwit, but I am seeing what appears to be excessive CPU usage and longer GBT times plus very high DOA rates (~5%). I just restarted the server with profiling enabled and will look into it further.

veqtrus used some dynamic share format code that would either create a share with witness data or without depending on the runtime value of the VERSION attribute instead of putting the segwit code in a different class, and that might be messing up pypy's JIT compilation somehow. Or it could be something else. We'll find out soon enough.

Edit: Profiling shows that the performance issue seems to be in p2p.py:update_remote_view_of_my_known_txs(). This sounds like the performance issue that I addressed a couple years ago in the jtoomim_performance branch. My guess is that veqtrus's code updated the known_txs var somewhere using assignment (which is O(n^2) versus transactions per second) instead of inserts and deletions (which are O(n)).

That is possible, but I would deem that worthy of a warning at most. I suspect that that misconfiguration is not common enough to merit the programming time.

I pushed a small change to 1mb_segwit that adds compatibility for pre-fork segwit2x mining with e.g. btc1. The 1mb_hardforked and lowmem branches do not need this change, as it's only the 1mb_segwit branch (and veqtrus's segwit PR) that have this issue.

The issue is that p2pool will refuse to mine if a softfork is detected which is not in the p2pool/network/(coinname).py:SOFTFORKS-REQUIRED list. It may be best to remove that check entirely, as it seems to be explicitly anti-forwards-compatible, which sounds like it may be a bad idea. Still thinking it over.

frodocooper, I'm now running a 1mb_segwit node and a lowmem node in parallel on the same machine. I'll keep an eye on memory consumption. If I can replicate it, I may look into the issue further and try to solve it if I can do so quickly.

My medium-term goal for fixing the p2pool CPU/RAM performance issues is to remove all transaction references (except the merkle root hash and maybe the coinbase transaction) from the consensus protocol and p2p layer. This will reduce RAM, CPU, and network usage by around 90% or more, but it will mean that p2pool's fast block propagation algorithm will no longer work. Given that p2pool's fast block propagation is no longer state-of-the-art, and has about 10x higher transmission time than the state of the art (Bitcoin FIBRE), and is slighly slower than the near-universal Compact Blocks and xthin protocols, and is about 100x more CPU intensive due to being written in Python, I do not think that the current p2pool fast block propagation tech is worth keeping in the codebase. If I get bored, I might try to replace the fast block propagation code with something else that runs outside of the p2pool consensus layer, maybe using bloom filters and/or IBLTs, but I think the current version is causing more harm than good.

If I remove p2pool's fast block prop code, I might add a check at startup (for the bitcoin and bitcoincash networks only) to make sure that bitcoind has some sort of fast block prop set up, whether it's xthin, CB, Falcon, or FIBRE, and make p2pool not mine at all unless one is present or unless a command-line override flag is set. This would be to prevent lazy or incompetent p2pool sysadmins from driving up p2pool's orphan rate.

We just added another 0.4 PH/s to jtoomimnet. That makes about 1.8 PH/s total.

sawa, can you try commenting out lines 350 through 357 in p2pool/work.py on your emerald test server and see if that helps? It's possible that target = min(target, ...) line is doing something strange on alts. The code you should comment out to test is this:


It's probably something else, though. I'll probably not have time to look deeply into this until the weekend.

You mean binaries? I am a bit too busy to build binaries for non-release beta versions. You can still run from source on Windows, though.

No.

Agreed. Before we can merge the two pools, we need to get the 1mb_segwit branch tested on testnet and the bugs that prevent it from working with altcoins fixed. Can you help with that?

Excel doesn't have bignum support. Python does.

sawa, those hash > target errors are weird, and I have no good idea what could be causing them. It seems as though the PoW function that p2pool is using to verify the hashes is not the same as the PoW function that the miners are using when solving the shares. I don't know why that would be the case.

The error at the beginning is something that I've never seen before, and is likely related to starting up a new chain. It does not seem to be closely connected to any code that I touched. There's a chance that it could be causing the hash > target errors, but I'm not sure how exactly.

Can you verify that your set-up procedure works fine with https://github.com/p2pool/p2pool master?

Can you try running 1mb_segwit using an existing share chain on an altcoin?

It represents an arbitrary number that some dude thought worked well for making the statistics graphs nice and smooth. It does not affect revenue at all unless you set the difficulty low enough that it overloads your poolserver's CPU or network connection with processing an excessive number of pseudoshares. I think it's better to not touch the pseudoshare difficulty at all, as p2pool's default values are generally reasonable.

You're using the wrong branch. The 1mb_segwit branch has the altcoin and empty sharechain fixes. The 1mb_hardforked branch does not.

Note: 1mb_segwit will work on the same sharechain as 1mb_hardforked and 1mb_lowmem. 1mb_segwit will vote for v33 shares, but until v33 happens it should be compatible with the other jtoomimnet branches. The v33 upgrade will be the same as the hardforks that forrestv has done in the past for upgrading p2pool to be compatible with bitcoin softforks (e.g. OP_CLTV).

sawa, I added some changes in the 1mb_segwit branch that I hope will address the altcoin issues. Can you try with that? The 1mb_segwit branch's commits are a superset of the lowmem branch.

The p2pool code will refuse to mine if your node is not connected to any peers.

For some reason, it's not listed on the node dashboards in the "Blocks found in the last day" section, but it's a valid block. If you plug in the block/share's hash into share.html, you can see it. Example:

http://ml.toom.im:9332/static/share.html#00000000000000000068e942f0cc1a00684c039c2c3a729bed92729e2e72f14c