The most recent version of 1mb_segwit needs about half as much RAM but a little more CPU than p2pool master.

If you're using pypy, you should probably have about 3-4 GB of available RAM and a 2.2 GHz Intel Core-based CPU or better. AMD Ryzen CPUs should work fine. Older AMD CPUs (e.g. A10 or A8) will need a lot more GHz, but if you have a 3.5 GHz older AMD CPU, that will probably also work fine. You might be able to get away with slower CPUs, but I can't promise anything.

If you're using python2.7, then you only need about 1.5 GB of available RAM but you might need a 3.0 GHz Intel Core i3 CPU.

I strongly recommend using pypy. That's much easier to do on Linux. Orphan rates will be lower and revenue will be higher if you use a fast CPU and if you use pypy.

Thread and core counts are basically irrelevant. GHz and IPC are king. This means a cheap desktop Core i3 CPU will be quite a bit better than a high-end server 16-core Xeon. The Xeon should still work fine, though.

Unfortunately, p2pool requires that the full node's blockchain be unpruned. It needs to access the genesis block for some reason; I haven't looked into the details on why. I recommend making the ~/.bitcoin/blocks/ directory a symlink to a cheap HDD and keeping the ~/.bitcoin/chainstate/ directory on an SSD. The chainstate is the UTXO database, and it needs to be read from and written to frequently and randomly when validating blocks. The blocks directory is just the raw block storage, and is accessed infrequently and sequentially, which makes it perfect for HDDs.

The "Block stale detected" message is normal (though eliminated in jtoomimnet) and does not indicate a problem.

The "submitted share more than once" message is more unusual, but by itself does not itself indicate that things aren't working, nor does it give much of a hint how things would be failing if they were failing. My only guess for why you might be getting that is if the difficulty set by your node is too low. It's also possible that your mining hardware is just buggy or your network is buggy/congested/laggy or your node's CPU is getting saturated.

It currently has the same 0.5% bonus that mainnet has. I might want to increase that in the future, but for now it's the same.

With mainnet's current hashrate of 760 Th/s, it is expected to find one block every 95 days or so. Mainnet last found a block 85 days ago on August 15, 2017, so this doesn't even qualify as bad luck. With this hashrate, "bad luck" would mean not finding a block for about a year.

I don't see how that's relevant. The absence of my code in the main repo was never about 2x; it was about compatibility issueswith altcoins.

We will have about 3x as much hashrate in December.

Last block found on mainnet was 8/15/2017. mainnet currently has 0.8 PH/s, and is expected to find one block every 91 days on average.

Last block found on jtoomimnet was 9/18/2017. jtoomimnet currently has 2.6 PH/s, and is expected to find one block every 28 days on average. jtoomimnet will be adding at least 4 PH/s over the next 45 days.

List of all blocks found (both mainnet and jtoomimnet):
https://blockchain.info/address/1Kz5QaUPDtKrj5SqW5tFkn7WZh8LmQaQi4

Sorry for the slow response, Cryptonomist.

That's one part of the relevant code. The actual tracker for the share chain is the OkayTracker class which inherits from forest.tracker, as instantiated as node.tracker. I think OkayTracker's code is more relevant and interesting.

By the way, it's forest.py with one 'r' (as in a bunch of trees), not forrest.py (as in the author's name).

I would suggest printing out the difference between the time first seen and time.time() at the end of data.py:OkayTracker.attempt_verify(). That seems like useful information for everyone. If you put it under the --bench switch and submit it as a PR to 1mb_segwit, I'd likely merge it. Don't worry about it if you're not good with git/github, as it's not a big deal either way.

Yeah, that code is kinda spaghettified. It might help to insert a "raise" somewhere and then run it so you can get a printout of the stack trace at that point.

Quick from-memory version: the stuff in data.py (the BaseShare class and its child classes) gets called during share object instantiation and deserialization. When p2p.py receives a serialized share over the wire, it deserializes it and turns it into an object, then asks the node object in node.py what to do with it. node.py then passes it along to the node.tracker object and asks the tracker if it fits in the share chain; if it does, then node.tracker adds it to node.tracker.verified, and the next time node.tracker.think() is run (which is probably immediately afterward), node.tracker may choose to use that new share for constructing work to be sent to miners. This causes work.py:get_work() to generate a new stratum job (using data.py:*Share.generate_transaction() to make a coinbase transaction stub and block header) which gets passed via bitcoin/worker_interface.py and bitcoin/stratum.py to the mining hardware.

Sorta. It's a limit of 100 kB of new transactions. Less than 100 kB of new transactions can be added per share. The serialized size of the share is much lower than this, since the transactions are referred to by hash instead of as the full transaction; the serialized size of the candidate block that the share represents is much larger than this, and includes the old (reused) transactions as well as the new ones.

If the transaction that puts it over 100 kB is 50 kB in size, and has 51 kB of new transactions preceding it, then only 51 kB of transactions get added. If some of the old transactions from previous shares have been removed from the block template and replaced with other transactions, then those old transactions don't get included in the new share and your share (and candidate block) size goes down.

In practice, the candidate block sizes grow slower than 100 kB per share. I haven't checked very thoroughly how much slower, but in the one instance that I followed carefully it took around 25 shares to get to 1 MB instead of 10 shares.

The hash of a share *is equal to* the hash of the corresponding bitcoin block header. The share structure includes a hash of all p2pool-specific metadata embedded into the coinbase transaction (search for gentx in data.py). The share has two different serializations: the long serialization (which is exactly equal to the block serialization, and which only includes the hash of the share-specific metadata), and the short serialization (which includes the block header plus the share-specific metadata such as the list of hashes of new transactions, the 2-byte or 3-byte reference links for the old transactions, the share difficulty, timestamps, etc.). Any synced p2pool node can recreate the long serialization from the short serialization, data in the last 200 shares of the share chain, and the hash:full_tx map in the node's known_txs_var.

The transactions aren't "added". If a transaction has been included in one of the last 200 shares in the share chain, then a share can reference that share using the number of steps back in the share chain (1 byte) and the index of the transaction within that share (1 or 2 bytes). These transactions -- "old" transactions -- do not count toward the 100 kB limit. If a transaction has not been included before, then the share will reference this transaction using its full 32-byte hash, and counts its full size (e.g. 448 bytes) against the 100 kB limit. Both types of references are committed into the metadata hash in the gentx, so both are immutable and determined at the time the stratum job is sent to the mining hardware.

https://github.com/jtoomim/p2pool/blob/9692d6e8f9980b057ae67e8970353be3411fe0fe/p2pool/data.py#L156

My code currently has a soft limit of 1000 kB (instead of 100 kB or 50 kB) on new transactions per share, but unlike the p2ool master branch, this is not enforced at the consensus layer, so anyone can modify their code to exceed this limit without consequences from should_punish_reason().

Actually, you'd simply ignore all transactions in uncles except the coinbase transaction, which gets programmatically modified to reduce the amounts.

I'd really rather not, for CPU and RAM performance reasons in addition to orphan rate reasons. It takes a medium-slow CPU on Python2.7 up to 4 seconds to process a share and issue new work with 1 MB blocks. With 4 MB blocks, you would generally be unable to run p2pool on medium-slow CPUs with a 15 second share interval, as it would take 16 seconds to process each share. Share variance is currently insignificant compared to block variance on p2pool. Increasing the share interval to 60 seconds on average is more likely to be a good idea than decreasing it.

While there are a few major changes to p2pool's architecture that would drastically reduce p2pool's performance load, those changes are quite large and may never be done. It would certainly involve more code than just adding uncles.

There would be uncles instead. Uncles have a smaller reward for the person who mined them than normal shares, so there's still a potential fairness penalty to having high uncle rates. The game theory of uncles is complicated, and if you don't set the rules right, it can be in a miner's best interest to not mention other people's shares as uncles and orphan them instead. Giving the uncled shares a hit in revenue is necessary to making sure there's an incentive to include them.

A probably better solution is to just follow the share with the lowest hash, regardless of its difficulty. This inherently prefers high-diff shares, but is fair in that low-diff shares have a chance to win against a high-diff share if the low-diff share is particularly lucky in its hash. Basically, it is effectively equivalent to the algorithm I proposed in the post that you linked to, but where the source of randomness is the hashes themselves.

Correct. The most recent block found on jtoomimnet was found on September 18th. Jtoomimnet currently has around 2.6 PH/s and an expected time per block of around 22 days. Mainnet has around 0.7 PH/s and an expected time per block of around 90 days.

That's what happens when your CPU gets overloaded and stalled. Let's say you have enough incoming traffic in your TCP buffers to keep p2pool's CPU busy for 1.1 minutes. P2pool first sees that it has a queued need to get a new block template, so p2pool sends a getblocktemplate request to bitcoind. Bitcoind may respond immediately, but the response goes to the end of that 1.1 minute queue. P2pool then works through its queue of stuff, then gets to the bitcoind request, and process it, but notices that it took 1.1 minutes and so complains about it.

Same thing for web UI requests. When you point your browser to localhost:9332, that request goes into p2pool's queue of jobs to work on, and p2pool doesn't get around to it until e.g. 1.1 minutes has passed.

The 15% CPU is the problem. 15% is above the 12.5% threshold for fully utilizing a single core on your machine, which means that the work in p2pool's to-do queue is snowballing.

The --bench output is somewhat helpful, but it seems that the bulk of the load is happening in a function that I didn't put benchmarking code into. In your log files, I frequently see line pairs like this one (note the timestamps):

The "Lost contact" message means that your CPU was stalled for 1.6 minutes. 1.6 minutes before the "Lost contact" message at 12:52:59 would be 12:51:20 or something like that. That means that your CPU was busy between 21:52:37 and 21:52:58. However, we don't know what was happening at that time, since the code that was running was something that I didn't add benchmarking code to.

Can you add this to your p2pool startup command line and send me the resulting profile1.log file? Try to run it for about an hour. If you run it for too much less than that, and the profile data will be dominated by the start-up share loading time.


If anyone is curious, you can analyze those cProfile log files with a python script that looks like this:

Xantus, maybe what's happening is that there just aren't any exceptions in Windows Firewall for this. Native Windows apps usually trigger a request to open ports in the firewall when needed, but I maybe WSL pypy p2pool isn't doing that. Try disabling Windows Firewall for an hour to see if that helps. If it does, then you can manually add a firewall rule to allow incoming tcp connections on port 9332 and 9333 (or whatever you're using).

Edit: Pages like this suggest that you do indeed need to open up ports on the Windows Firewall if you want to allow WSL processes to serve as servers.

That sounds like you might be out of RAM and have swapping to disk. Can you check? Task manager -> Performance -> Memory. In Use, Available, Committed, Paged Pool.

Interesting, it seems that time.time() on Windows only has 15 or 16 ms resolution, so all of the short function calls are reported as taking either 0 ms, 15 ms, or 16 ms, even though they're actually probably taking about 4 ms each.

I also see a few lines like this one:
453 ms for handle_remember_tx when no transactions are being added seems strange and significant. I'll have to reread the code and see if I can figure out what could be taking so long.

There's also this:
That makes about 1.7 seconds of latency just in those two functions for your node switching work. (IIRC, handle_shares() calls get_work, so the 750 ms is already included in the 1734 ms.) There may be a few other functions that I didn't add the benchmarking code to that also play a role, but that latency alone should be responsible for about 5.8% of your DOA+orphan rates.

The 1188 ms (unpacking) is something I think I know how to fix via a CPU/RAM tradeoff. That will probably be my next task. However, that will only reduce the CPU usage in processing new getblocktemplate responses, which I think is not in the critical code path for share propagation, so I don't expect it to help with DOA+orphan rates much. It's a pretty easy and obvious optimization, though. The node I do my testing on (4.4 GHz Core i7, pypy) only shows about 40 ms for unpacking instead of 1188 ms, so I wasn't sure it would be worthwhile. But if it's stalling your CPU for over a second, that sounds like it could be interfering with things.

Not sure what's the issue with pypy on your machine. Does your CPU show activity when you run it? How much?

jtoomimnet found another block two days ago.

https://blockchain.info/block/00000000000000000096cef4bfd159307030923b0111acd39bfb6561b277a375

1014 kB, 3998 kWU, 13.858 BTC.

We also added a few hundred TH/s yesterday, and are now up to 2.2 PH/s.

That assertion is inconsistent with most of the benchmarks I have seen, in which WSL is often faster than native Linux and rarely more than 5% slower.
https://www.phoronix.com/scan.php?page=article&item=windows-10-lxcore&num=2

However, for some reason, code compilation appears to be the exception to that rule. For some strange reason, compiling code on WSL is extremely slow:
https://www.phoronix.com/scan.php?page=article&item=windows-10-lxcore&num=4

Since running p2pool is not code compilation, performance on WSL should be quite good. P2pool is mostly just a CPU-bound task with very little disk IO and a small amount of network IO, so the poor results from code compiling should not manifest here.

A 2.0 GHz A10 is not what i would call a medium-speed CPU, unfortunately. Sure, it might have 4 cores, but Python is effectively single-threaded, so that doesn't help at all. Pre-Ryzen AMD chips have poor single-threaded performance, so a 2 GHz A10 is equivalent in performance to a roughly 1.3 GHz Core i3 (if such a thing existed). I wish I could get p2pool to run nicely on such a CPU, but I'm afraid that's out of reach at the moment.

You've got plenty of RAM, so I think you can get it to work decently if you go through the steps I described to get Pypy running on Windows.

Also, if you could run p2pool with the --bench option, that would make the log file much more useful to me.

P2pool has a share chain that is very much like the Bitcoin blockchain, with a few differences. The tip of a chain (a block/share with no children) is known as a head. The root of a chain is known as the genesis block in Bitcoin (the block with no parent), but in p2pool it's known as a tail. P2pool's tails *do* have parents, unlike bitcoin, and consequently have a height greater than 1; however, they're called tails in p2pool because their parents have been pruned off from the node's copy of the share chain, so as far as your node is concerned, they have no living parents.

The best share is always a head. It's usually the head with the most work behind it, but not always. The rules for determining the head are a little complicated, and are different in different versions of p2pool.

Not necessarily, but usually. Just because a share is recent doesn't mean that it is placed at the tip of the chain.

Yes, precisely.

I'm not sure. I think unverified heads or tails are shares that have not yet been connected to the share chain, such as for a share that is was just downloaded during initial sync but whose ancestors have not yet been downloaded. However, I haven't verified that hypothesis, as I have not yet needed to work on that part of the code.

Completely downloaded and mostly deserialized.

Yes, but it's a sendShares or share_reply message.

Correct. In the current p2pool master, the timestamp is required to be between 1 second after the previous share's timestamp and 59 seconds after the previous share's timestamp. Since the timestamps are used for minimum share difficulty adjustments, this can result in a situation where the share difficulty is very slow to adjust downwards after a rapid decrease in the network hashrate. In the jtoomimnet fork, I got rid of this timestamp clipping, and replaced it with a rule that rejects shares that are timestamped to come from the future in order to prevent share difficulty manipulation.

Keep in mind that the timestamp is the time at which the mining job was sent to the hardware, not the time at which the hardware returned the nonce value.

Nope, it's not close. It will usually be off by around 5-50 seconds. But what you can do is use the "Peer first received from" line to browse to the info page for that share on the peer's address (you may have to guess the port number), and repeat, until eventually you get to a peer that says "Peer first received from: Null" for that share, and then you can use the "Time first seen" shown by that ndoe (or just the node that you got the share from, if you're lazy).

When I've done this measurement in the past, it's usually around 1 second per hop for nodes running CPython with 100 ms network latency, and around half that for nodes running pypy.

If you run my code (e.g. 1mb_segwit), you can install the rfoo package for python, then run run_p2pool.py with the --rconsole command line option, and then run the rconsole command in a separate terminal. This will give you a python interpreter that effectively runs inside the p2pool process, and allows you access to all of the variables and functions of p2pool while it's running. The share objects can be found in the gnode.tracker.verified object.
1mb_segwit also now has a --bench command-line option, which I think you would find interesting.

You could also maybe parse the log in data/bitcoin/log. That's just the stdout output of p2pool for (usually) the last few days.

I was able to get pypy running on Windows 10 by using the Linux Subsystem for Windows:

https://bitcointalk.org/index.php?topic=18313.msg21025074#msg21025074

However, the reason I made these CPU improvement changes was in the hope that it would allow people to run CPython on medium-speed CPUs without crazy orphan rates. I'm very much interested in hearing whether it works.

I pushed some more code to https://github.com/jtoomim/p2pool/tree/1mb_segwit that adds more CPU and (maybe) RAM performance improvements for p2pool. I think it should improve CPU usage and latency by about 30% or more. These improvements should reduce DOA and orphan rates on the network a bit. It looks like running p2pool on CPython with medium-slow CPUs should now be viable without huge DOA/orphan costs, although I still recommend using pypy whenever possible. The new code makes fewer memory allocations when serializing objects for network transmission, which might reduce total memory consumption, or it might not. We'll see in a few days.

I also added a performance profiling/benchmark mode. If p2pool is too slow for you, I would find it helpful if you ran python run_p2pool.py --bench and then sent me a snippet of the output, especially if you can get the output near when a share is received over the network.

More like 100%. The current network hashrate for DASH is around 35 TH/s, which is equivalent to about 2250 D3s. I expect Bitmain made at least 2k in their first batch.

All Dash/X11 pools will be affected equally.

This scenario would probably be one in which the capital cost of the miner itself is nearly zero. If the miner capital cost is high, then the miner needs to be run nearly 24/7 in order to make a profit. Currently, capital costs (or amortized depreciation, if you prefer) is the greatest cost in Bitcoin mining, and electricity costs are the second-greatest cost. I expect capital costs of miners will always be significant, but will probably eventually be slightly less than electricity costs. If you're only running your miners when the sun shines (8-12 hours a day), then those capital costs become much harder to recoup.

Utility-scale solar is probably going to always be cheaper than rooftop solar. As of Dec 2016, the cost (LCOE) of rooftop solar was about 3x higher than utility-scale solar ($0.05/kWh vs $0.15/kWh). If it ever makes sense to mine at home with rooftop solar, then it will probably make even more sense to mine in a warehouse in the middle of a desert where it's sunny 360 days out of the year. That said, solar has a long way to go before it can catch up to hydro, which is about $0.015 to $0.05/kWh with 24/7 uptime.

Solar power is great, but it is not very well suited to mining. It's better suited to supplying the grid and helping with peak load, since solar power peaks when demand peaks, rather than trying to supply a 24/7 load like mining. Wind is usually cheaper, and might make a better partner, but still not perfect. Ultimately, hydro, nuclear, or (ugh) fossil fuels are what make the most economic sense.

https://www.reddit.com/r/Bitcoin/comments/6y8305/if_pool_miners_run_segwit2x_they_get_a_better/dmmzl99/