I don't like that this is necessary. Flameruk, can you tell me what operating system you were using, what time zone you had your BIOS set to, and what time zone your OS was set to? I think I need to change the code that I use in p2pool's code when determining UTC time, and I'd like to be able to reproduce your issue.

You should probably also add blockmaxsize=999000 to bitcoin.conf.

I haven't tested on Windows before. That looks like it might be a Windows-specific error. I'll see if I can reproduce it.

https://github.com/jtoomim/p2pool/tree/1mb_hardforked has new code. This should fix the sync issues that people have been having. It also includes a few bootstrap nodes to make it easier for people to connect to the jtoomimnet p2pool.

To set up and run my fork, you will need to do the regular steps for installing p2pool, except that instead of getting the regular github.com/p2pool/p2pool repository, you will do:


If anyone continues to have trouble connecting to jtoomimnet, please let me know.

Thanks for the info. It does seem that there may be something wrong with that commit. It's a pretty important commit, though. Without it, in my early testing, I was getting about 60% of my shares orphaned due to transmission failures. I think that orphan rate mostly applies to the first share after a new block, so the orphan rate is likely to be lower in a production environment. However, it will still cost you money if you try to run completely without it.

The purpose of that commit was to fix a problem in which transaction hashes that were part of a stratum job could not be dereferenced into full transactions for share propagation. To fix it, that commit does a few things:

1. It keeps a separate record of transactions that were most recently received by bitcoind in a getblocktemplate vs. transactions that were most recently handed out to miners.
2. It uses a Python namespace hack to keep the transactions in memory in a separate variable in case #1 above doesn't work
3. If share transmission doesn't work due to missing transactions, it queues additional attempts with a delay.

It looks like the problem was with the share retransmission logic (#3). I don't think the retransmission logic is necessary any more, so I made a branch that excludes just that part and see if it fixes the problems that people have been having. I'm testing it now. Looks good so far; I'll post again once testing is finished.

Email your p2pool/data/bitcoin/log file to (edit: email redacted because of spam).

Also, if anyone is having trouble connecting and doesn't mind giving me ssh access, I would like to be able to investigate these issues further. I've heard several reports from other people of having trouble downloading shares, but I've not yet been able to reproduce the issue myself.

nreal is incorrect. Shares are currently valid for approximately 3 days. P2pool targets a difficulty for one share every 30 seconds, with 8640 shares valid at a time. 30 sec * 8640 / 3600 = 72 hours.

https://github.com/p2pool/p2pool/blob/master/p2pool/networks/bitcoin.py#L10

Previously, p2pool targeted one share every 10 seconds, with 8640 shares valid at a time, for a 24 hour window, but share orphan rates were too high, so forrestv slowed the shares down by a factor of 3.

It's worth mentioning that 30 seconds is what the default share difficulty will target. Individual miners have some leeway to override this share difficulty target (as long as it doesn't go below some pool-wide minimum). I'm not happy about that, as it seems like it could be used for selfish mining attacks or to flush out a competitor's shares from the share chain if they stop mining suddenly.

I could add some code based on https://github.com/donSchoe/p2pool-n/pull/6 to determine a per-miner share difficulty based on their hashrate, so that fewer shares would be issued by large miners and more shares by small miners (thereby minimizing share variance for those who are most affected by it). I'm a little concerned that it might increase the incentive for large miners to modify the codebase in order to do selfish mining, though, so I may wait on that until after I've improved share processing/propagation times and gotten rid of most of the orphan share fairness problem.

frodocooper, please note that my fork includes several improvements, not just a change to the maximum number of transactions per share, and consequently actually has a significantly *lower* share orphan rate than the p2pool mainnet. I don't have any good data on whether jtoomimnet is more fair or less fair than mainnet as of yet, but preliminary data suggests that it's about the same or slightly better.

As for the voting, I'm not sure there was any point. If anyone wants to mine on a share chain that allows less than 1 MB of new transactions per share, then there needs to be two separate share chains, because there is at least one miner (me) who chooses not to mine on anything less than full blocks. Rather than have one chain on which everyone compromises, why not just have two chains to satisfy everyone's goals?

Also, there are plenty of methods by which share network traffic can be reduced or taken out of the latency-critical path. I would prefer to just make shares propagate faster regardless of size rather than organize a vote on how much to cap our block sizes.

Edit: Sorry if I came off as irritable earlier. I'm just tired of politics getting in the way of fixing the code. I think that forking, as a permissionless process, is just socially easier to deal with than trying to get consensus, and in the case of p2pool, it has very few disadvantages. This way, I can just write code, and if you think it's better, you can use it. It's really simple that way. I don't have to worry about writing the code and having it never be used, since I know that I will use it even if nobody else does. And users don't have to worry about voting on features they don't understand; they can see the new code already running before they decide whether or not they like it and want to make a switch.

My attitude on the share difficulty thing is that a hybrid approach would be nice. It would be nice to maybe calculate (a) the difficulty needed to have shares last three days, (b) the difficulty needed to have shares last for an expectation of 3 blocks, and then take e.g. the geometric mean of (a) and (b).

Ultimately, we need to minimize total variance to the end-user. As there are two components (share variance and block variance), we need a formula for the difficulty that balances those two needs.

I've also been playing with some ideas that could dramatically reduce the orphan rates and CPU/memory loads of p2pool. If I implement some of those and they work out, then we might be able to reduce the share time and increase the share chain length, which would allow lower share variance and also lower block variance at the same time. But that's pie in the sky still.

Maybe. I find myself attracted to the idea of having commit access to the actively used repository, so part of me wants to just keep it as a code fork so that I can more easily do further development when the mood hits me. But if people would prefer to have it be in the p2pool/p2pool repository (with consequently fewer updates), then I guess I'd go ahead and submit the PR.

People are welcome to submit PRs to my repo, of course.

So much for doing a controlled small-scale performance test. I guess it's trial by fire. Tally-ho!

So far, the jtoomimnet fork has generated four blocks. All four blocks were larger than 998 kB. Two separate mining addresses have found two blocks each. I think both of them are Nicehash miners, but I only know one of the miners. (whortonda)

I'm back at my Washington mine, but I've got a bunch of mine-related stuff to take care of first before I can work on the DNS seeds and the peer-banning issue that some people are having with joining the fork.

Some preliminary observations on fairness: the large miners (at least whortonda) appear to have a substantial advantage in terms of orphan rate (0.83% for him vs 3.6% to 5.3% for my nodes). However, as nicehash renters, they have a huge disadvantage in terms of DOA rates (4.5% for him vs 0.0% to 0.9% for my nodes). So overall, we are pretty much at parity. That's pretty much the ideal result, but not for the right reasons. Oh well, good enough for now.

The most relevant question for the fork isn't whether jtoomimnet is fair, but whether jtoomimnet is more fair or less fair than mainnet is. Unfortunately, I don't have any good data on that question right now. When I get time, I might try to look through some of our logs and make some comparisons. The baseline orphan rates are much higher on mainnet, and the noise from that makes it much harder to compare orphan rates than on jtoomimnet, so I'm not sure I'll be easily able to get enough precision to definitively answer the question.

I have received full logs from a few people. I am also seeing 7 peers on some of my nodes, which means that at least a few people are able to connect. (I have 3 nodes, plus whortonda's, but that only makes 3 peers, so there are at least 4 others that are running that I can't name offhand.)

I'm afraid I'm going to be pretty busy for the next three days (driving back from CA to WA, then installing 100 new miners), so I might not get to this until the weekend. The peer banning problem prevents new nodes from connecting and mining, but it doesn't seem to affect nodes after they get fully connected. So if you're able to connect and stay connected, it should be safe to mine.

By the way, those of you who are (a) seeing or (b) not seeing this problem, did you manually clear out your share chain (the data/bitcoin/shares* files) or start with a clean installation?

Looking at my logs, I see a bunch of messages like these:
I'm looking into the cause. I'll probably add some debugging messages to the code running on my nodes to give more information on why my nodes are banning yours. I'm going to need to go offline in a bit, so I might not get it ready tonight. Can you PM me your email address or send an email to "j AT toom PERIOD im" so I can let you know when I have new code running and when you should try to reconnect?

sawa, same for you: send me an email with your full logs? The "Missing ... for broadcast" messages are some new messages that I added while figuring out the transaction broadcast bug mentioned here. This message still gets printed out sometimes when a node just starts up, but probably is not related to the problem that's preventing your node from staying connected. I'm not seeing any "Bad peer banned" messages when grepping my logs for your IP address, but maybe your node is banning me?

Actually, http://ml.toom.im:9332 is a better choice. The :9334 node has more different users on it right now so it's running a bit more slowly at handing out new work.

The current hashrate on the fork is around 7 PH/s. Of that, around 1.2 PH/s is from my facility, and will be on 24/7 as baseline hashrate. A lot of the rest is currently rented hashrate, and will be coming and going depending on the Nicehash rental rates.

When I created the new share chain, I mined for several days in a fashion that would have paid out everybody on the mainnet sharechain had a block been found. I think it's fair for the inverse when people migrate to my share chain. Edit: It's also fair because after you switch, the mainnet chain will continue mining based on your shares for a while, ramping down at the same rate that jtoomimfork ramps up.

Note that the old mainnet p2pool's hashrate has fallen significantly, and will experience far fewer blocks than my fork will. My fork is still in the late testing stages, though. No, it hasn't. It looks like hashrate was added to the new fork, not moved from the old fork. My mistake.

I will release code soon-ish that will automatically find a few peers (using DNS seeds) to make it easier to join my fork. However, I want to do some performance testing with an artificially linked node graph/chain, so it will probably be a couple days before that is out. If anyone is planning on running a stable, long-term node on my fork, please let me know your IP and I'll include it as a seed.

I do not support this. The :9334 node is just ONE of the three nodes I run. ml.toom.im:9332 and ml.toom.im:9336 are also on the fork. We are only using 9334 for that node because we have one global IP and multiple nodes on our LAN.

Nodes on the old share chain and the new share chain will refuse to connect to each other. I made sure of this in the code and tested it several times.

You can identify whether a node is running my code by looking at the protocol version that the node reports (≥3200 for my fork, 1600 or 1700 for the forrestv fork), or by looking at the share versions on the chain (16/17 for forrestv, ≥32 for jtoomim).

Or on your LAN's DNS server.

My forked sharechain has switched to v32 shares. When I did this, I forgot to upgrade one of my nodes, which resulted in that node downloading shares from my upgraded nodes, rejecting them due to an unknown share version, and then downloading them. Eventually, it disconnects from the peers due to the perceived misbehavior, but since I had the connections hardcoded with an -n command line option, it reconnected again immediately. It seems p2pool could use a temporary ban feature for misbehaving peers as well as a blacklist for share hashes that are known to be invalid according to the node's ruleset. That should be done before p2pool gets too big in order to prevent DoS attacks.

I just bumped the protocol version up to 3200 and now require peers to have that protocol version. I think we're now ready to start testing with other people's nodes. I'd like to set up a chain of nodes connected via the internet at large (each with one incoming, one outgoing peer) so that I can get an idea how long full shares will take to propagate over many hops. Any volunteers? A chain of 5 or so nodes would be sufficient. See the bottom for instructions.

After this test has been finished, I can add some new DNS seed peers to the code to make it easier for people to join the fork. (Currently, you have to use the -n command line option to manually specify someone to connect to whom you know to be using my code.)

By the way, some rough performance numbers I've observed on my LAN using pypy and fast CPUs: it appears to take around 200 ms per hop to transmit and parse a 1MB share, even when the share is being sent to localhost (i.e. infinite bandwidth). It then takes about 80-120 ms per worker (i.e. per mining address, not per mining rig) to generate and assign new work. After new work has been assigned, it seems to take Antminer S9s around 300 ms before they stop working on the old work and switch to new work (or maybe that's just additional latency on the node side from serialized processing of network input causing delays in the processing of returned work). All told, this seems to add around 600 ms of latency. Over the last week of operation, the forked pool is showing total orphan and DOA rates of 0.59% and 1.2%, respectively, or about 1.8% total, which is pretty much equal to what we'd expect based on the observed latency (0.6 s / 30 s = 2%). This indicates that the changes I made to address the "Block-stale share punished!" and "Tried to broadcast a share without all of its transactions" issues appear to be resolved in this code. The old fork probably would get a DOA+orphan rate above 10%, and certainly no less than 5%, even on a LAN. Most of that high orphan rate was purely random and shouldn't adversely affect revenue, but it could still create unnecessary orphan races which benefit larger miners like me.

It will be interesting to see how high DOA+orphan rates end up being once this fork has more mining nodes. I'm guessing we'll probably be a little under 10%, but maybe 5% is possible. We'll see.

Due to the share chain holding about 2x as many transactions per share, the memory consumption is about 2x higher. Expect about 1.5 GB with CPython and close to 4 GB with pypy. I'm sure this can be improved if it's a major problem for people, but it's not the top of my priority list. (Do we even need all the transactions for old shares? Wouldn't the merkle root + coinbase be enough for shares that are too old to have reused transactions?)

---

If you want to contribute to the networked chain for share propagation testing, please clone https://github.com/jtoomim/p2pool/commits/1mb_hardforked, then configure your node to point to the previous poster's node (e.g. python run_p2pool.py -n previous_node), then list your node IP, what it's connected to, what type and speed of CPU you're using, how fat your pipe is, and whether you're using pypy or CPython (the regular python). Example, and to start us off:

Node IP: ml.toom.im (default ports)
Connected to: ml.toom.im:9334 and :9336 (or :9335 and :9337 for p2p ports)
To connect to me: run_p2pool.py -n ml.toom.im:9333
CPU: Core i7 4790k 4.0 GHz
Pipe: 100m/100m fiber
Using: pypy 2.4.0

Thank you for pointing that out. While the attack you implemented could be defended against by checking all transactions to see if they're in bitcoind's mempool, and if they're not, passing the transaction to bitcoind for validation (thereby incurring the tx validation cost in the critical path only for rare cases when the transaction is heard through a share before bitcoind has received it), and then checking that the coinbase reward is no more than the sum of the transaction fees and the block subsidy, but you're right that this proposed change increases the attack surface area too much for now.

At some time in the future, transaction fees will exceed the block subsidy, and we will likely need to have p2pool shares be weighted by fees at that time to keep the incentives aligned properly. However, it seems to make only a 1% difference in the reward assigned per user right now, this change is probably premature at the moment.

And the install guide should be translated into Chinese.

Currently, p2pool weights each share by its difficulty alone. The expected revenue for the pool from each share equals the difficulty times the block reward (with fees). This means that shares with low fees get overcompensated, and shares with high fees get undercompensated, which means that the optimal strategy for most p2pool miners is to make small shares/blocks and hope that the rest of the pool irrationally mines big blocks.

This misaligned incentive can be fixed by weighting shares by their difficulty times the block reward they would have generated. Pretty simple. As it affects payment calculations, which need to be precise down to the satoshi, t's a completely backwards-incompatible change, but that's something I want for this fork anyway. Yay!

I'm trying to figure out the implementation now. These revenue calculations are done with a SkipList implementation so that p2pool doesn't have to add up every single share's weight every time it assigns new work, and it's taking me a while to understand the code well enough to change it without breaking everything. But I think it will be working soon. Once I've got the calculations correct, I'll deploy them along with a share version change to v64 (just to give the legacy chain room for plenty of upgrades if it continues to be used), and then I think the code will be ready for everyone else to use.

After that, I might take a break from coding on p2pool, or I might continue for another week or two. If I continue, I have some ideas for optimizations that could improve p2pool's performance and orphan rates. They fall into two categories: CPU performance, and network performance.

For CPU performance, almost all of p2pool's CPU time is spent packing and unpacking shares and transactions between Python objects and serialized bytestreams. I can probably accelerate this process either by rewriting the Python code into faster Python code, simplifying the Python data structures for transactions, and/or adding Cython support. If I go the Cython route, I can make it so that p2pool can run without Cython by stripping the static typing information from the code, but if you have Cython and a C compiler installed (or if you downloaded a binary package), you will use C compiled versions of a couple of the most performance-critical modules. This should give us about the same CPU performance benefit of pypy (or better) without any of pypy's 2GB memory overhead. Improving CPU performance could improve share propagation speed and new work generation, thereby reducing orphan/DOA rates and making revenue allocation more fair. Also, it will make it cheaper to run a p2pool node, and help p2pool scale.

For network performance, it seems that the current code has a lot of room for improvement. I'd like to add a new network message that sends a list of transaction hashes to a p2pool peer and asks them to download the corresponding transactions from their bitcoind if possible. This would allow p2pool peers to get most of their transactions via localhost tcp instead of using the p2pool network, which should save a lot of bandwidth. I'd think it would reduce traffic by more than half, since p2pool is a lot more aggressive about forgetting transactions than bitcoind is. Another thing that can be done is to add a shorthash implementation like the one in xthin and compact blocks for share transmission. Once these two optimizations are in place, it should be practical for nodes to send the template of the share they're working on mining *before* it gets mined, and send only the nonce values and miner address once a share is found, at least to one's immediate peers. If I get that done, then I could also teach nodes to encode a message to each peer using the diff between the share being sent and the recipient's current work-in-progress, which would allow 1 MB shares with 95% new transactions to potentially be sent with only a dozen kilobytes or so of one-way traffic in the latency-critical path. After that, maybe I could play with bloom filters or IBLTs and and and...

Anyway, first things first. Gonna try to get the properly weighted share rewards done in the next couple of days, and then things should be ready for others to join in on the fork fun. (By the way, my fork is over 1 PH/s of non-rented miners now.)

It looked to me like it was rentals. The current burst of hashrate started on Mar 29th right after I mined the first block in like 3 weeks. I know there was at least one entity who set up an automatic nicehash p2pool mining scheme with automatic reinvestment of p2pool rewards, and the timing suggests to me that this was something similar.

Personally, I'm hoping that my new fork can attract permanent contributors. I think I've been able to fix some of the many bugs and issues with p2pool, and there are lots more optimizations that I've thought of but haven't had time to implement yet. I think we can get p2pool to scale up quite a bit and still give its users better revenue than they would get with most traditional pools. I may have to write some share propagation code specifically with the GFW in mind, though (e.g. UDP, weak shares, bloom filters to replace remember_tx and forget_tx, maybe IBLTs, and share-headers-first mining). Even without those things, revenue will probably be better in China than the main Chinese pools (which all have large fees). But we'll see.

My company will also be adding quite a bit of hashrate over the next few months. I'm expecting to add 390 TH/s today alone, for example (30 S9s are on their way). We'll probably add a petahash or so by July. All of that hashrate will be going onto the new fork, of course.

Edit: Yes, it's Nicehash. The p2pool user 1Hinenj9woDM8wLDffov4PUwFqqtRg5FRu sends regular payments of 2 to 10 btc to 3FNmBkdxorsjf7PFBf4d5JEYBybMicbDin about two times a week, adding funds from 19XLjEswgycrtbX1TcCJ8grw8zZYvbNaG6 as necessary. The 3FN address belongs to Nicehash, and makes regular payouts to Nicehash customers (examples 1 and 2). The 19XL address first sent money to Nicehash (2 BTC via 3FN) on November 6th, 2016. 19XL appears to be a user of Poloniex, as he frequently gets some money from Poloniex's 2nd-biggest wallet 17A16QmavnUfCW11DAApiJxp7ARnxN5pGX (about 0.2 btc every 2 days).

The p2pool user 1NVoAeLBm3QLRCcezvwUzKYHDLU96dji8x is directly a customer of nicehash, and sells about 18 kSol/s of Equihash hashpower there (about 0.1 btc/day). We can't see via Nicehash's public UI what he buys, but 1NV has sent regular payments of 1.2 to 1.5 btc to 32DdAy7oDoovdxt6egJ2Zoip1sQf25pZs3, which is another Nicehash deposit address.

1Hinen and 1NV have made a ton of money from the recent luck, and have about 30 btc and 6 btc respectively saved up, but they're spending it at a rate of about 20 btc per week. If I'm right and these deposits are being made by a bot, then he can keep it up for about 1.5 weeks with 0% mining luck. If luck remains close to 100% (it has been over 200% recently), then he should be able to hold it up for quite a while, although I presume that Nicehash's fees will run down his balance eventually. That could be a month or two away, though.