I'm seeing this too. Furthermore, I'm seeing stuff like this:

2016-06-28 14:40:39.001988 > in download_shares:
2016-06-28 14:40:39.002117 > Traceback (most recent call last):
2016-06-28 14:40:39.002181 > Failure: p2pool.p2p.ShareReplyError: too long
2016-06-28 14:40:39.002370 Requesting parent share ea0f2aee from 10.0.1.3:38393
2016-06-28 14:40:39.311156 > in download_shares:
2016-06-28 14:40:39.311285 > Traceback (most recent call last):
2016-06-28 14:40:39.311351 > Failure: p2pool.p2p.ShareReplyError: too long
2016-06-28 14:40:39.311523 Requesting parent share ea0f2aee from 10.0.1.3:37971
2016-06-28 14:40:39.494657 > in download_shares:
2016-06-28 14:40:39.494775 > Traceback (most recent call last):
2016-06-28 14:40:39.494827 > Failure: p2pool.p2p.ShareReplyError: too long

I have switched my nodes over to p2pool v16. The current hashrate is approximately 97% v16 over the last hour, and about 76% v16 over the last day. This means that anyone still on v15 will soon have their shares ignored by the rest of the network.

Bitcoin Classic 1.1.0 is out. It supports CSV and in my limited testing works fine with p2pool v16.

Is anyone else having trouble with peer acquisition? vps.forre.st seems to be down.

BitcoinXT release F has been tagged. It has BIP9, BIP68, BIP112, and BIP113 support, plus Xthin blocks. As such, it should work fine even after the CSV fork is activated, and should be compatible with p2pool version 16.

https://www.reddit.com/r/bitcoinxt/comments/4p7pl3/bitcoin_xt_release_f_has_been_tagged/

Keep in mind that this is based on the 0.11 branch, so performance may be sub-par.

Unfortunately, yes.

On that note, I've been meaning to ask other p2poolers what I should do about that, if anything. My nodes currently comprise around 46% of the p2pool hashrate. This means I could perform selfish mining attacks if I wanted to, and if I boosted my hashrate a bit, I could 51% attack the p2pool network and get 100% of the shares. Mostly, this happened because p2pool shrank from 2 PH/s to 0.8 PH/s, not because we grew much. One way of dealing with this issue is that I could take some of my hashrate off of p2pool (and maybe solo mine instead), but that would make p2pool blocks even rarer. (I think my nodes have found the last 4 p2pool blocks.) I could also maybe spread some of my hashrate among other nodes, although this would reduce our revenue. Or we could get more people to mine on p2pool, somehow.

No, we're talking about expected reward given to you by someone else's share. This is a p2pool conversation, not a traditional pool conversation. You have a probability of other people rewarding you in the shares they find after you find your own share. If there are more peer shares that are supposed to reward you, then the size of the reward that each share would give you (should that share also be a block) will be less. The point I have been trying to make is that the expected reward you get for mining a 1-diff share is not dependent on the number of peer shares that your reward is distributed over.

My simulation for the naive PPLNS-on-DAG system gave about 1% typical variation in payouts when the number of shares per level of the DAG varied uniformly between 1 and 10. Note that the variation is not predictable: you can't guess how many shares per level will be mined in the future unless you're actively manipulating it (and large enough to do so). For the PPLKD system, your reward will be split over a constant amount of other people's work, so the variation due to changes in M should be basically 0%.

As far as I can tell from what you've written, you're objecting that the amount of payout for you per peer's share decreases with the number of shares that go into the last N levels of the DAG. This is true, and is not a problem. What I'm not sure about is whether you're claiming that the sum of the expected payouts for you over all of the peers' shares in the last N levels will be lower if that number of shares decreases. If that's not your claim, then the only other claim that I think you might be making is that the sum of expected payouts is perturbed not by the static level of M/N, but by the rate of change of M/N. Is your objection one of those two? If so, which?

The size of each block reward is a constant, but the expected number of block rewards per N levels is not. The number of block rewards per N levels is proportional to M.

Unfortunately, Bitcoin Classic has not been updated to support CSV and BIP9 yet. Consequently, it will be inadvisable to mine with it after the CSV fork activates. If you try, you will probably have any blocks that you mine get orphaned. This will make other p2pool users unhappy, as you will basically be freeloading off the others in the pool. So please don't.

You can still vote for the Classic BIP109 2 MB hardfork without running Classic. You can do so quite easily in p2pool by editing the source code to bitwise OR the block version with 0x30000000. I will be doing this myself soon. Something like this should work:

    version=max(self.current_work.value['version'] | 0x30000000, 0x30000000),

If the BIP109 vote exceeds about 10-15% of the network hashrate, or if a large miner (e.g. Bitfury or Bitmain) asks me to, I will take a week and merge in CSV, BIP9, and whatever else needs merging into Classic, if Gavin and Zander haven't beaten me to it. In the mean time, I will keep voting for BIP109 but using Core, similar to what f2pool does.

You should upgrade. Do a git pull.

I have not yet upgraded my nodes. Once I do, due to the size of my mining operation, almost all of the network hashrate will be on v16, and v15 shares will start to be rejected. It would be a good idea to upgrade before I do.

If you can specify how you think the system is flawed, I'm all ears. However, simply stating that you think it's flawed without going into any detail or even demonstrating that you understand the system is not very helpful.

It's actually quite simple to fairly order shares within a level, as long as you don't care whether the ordering is chonological. For example, you can sort the shares by their hash -- lowest hash (closest to mining a block) comes first. Fair, simple, easy, and unnecessary.

It's also trivial to make a DAG with a constant M/N ratio. Instead of allowing shares to specify as many uncles as they want, you can force each share to specify exactly K parents, and create a rule where the parent with the lowest hash is the canonical parent, and only the canonical parent's parents are counted as grandparents. Fair, simple, easy, and unnecessary.

The best solution is just to make sure that the portion of the share chain that's used for calculating rewards always represents the same amount of work. This puts the fewest restrictions on the DAG structure, and ensures that the ratio of the numbers that actually matters -- the amount of work used to calculate the size of the payout, and the amount of work done by others that would have made the payout -- remains a constant 1.0. As in the PPLKD scheme I described earlier.

Shares are paid in proportion to their difficulty. You take the difficulty for a share, divide by the summed difficulty of all shares among the last M (or N for vanilla p2pool), and multiply by the block reward for that block (including fees), and that gives you the reward a share would receive for that block if it were found.

In periods where M is higher due to random variation, the probability of a share getting rewarded by another share finding a block increases, but the payout size decreases. These effects cancel out.

For example, if M is 4 (and all shares are the same difficulty), then if I mine a share, it will have 4 chances of probability 1/D to earn 1/4 of a block. If M is 256, then my share will have 256 chances of probability 1/D to earn 1/256th of a block. In either case, the expected revenue is the same.

The problem comes not when M is higher or lower, but when M is rapidly changing.

A share will be payable for a considerable amount of time, and M might change during that time. For example, if N is 4, my share will have its potential reward calculated 4 different times as it travels through different levels of the DAG. Let's say that we start with 1 share per level in all 4 levels of the DAG when my share is mined (my own share being the top level). After this, 1 more level is mined with 1 share per level, and then a level with 5 shares is mined, followed by another two levels with 1 share. The first chance I have to get paid comes from my own share, where the DAG looks like this:

Round 0:
1*
1
1
1

I get one 1/D chance of getting a 1/4-block reward here. (The asterisk denotes my share.) Next round:

Round 1:
1
1*
1
1

1/D chance of getting a 1/4-block reward.

The next round is a bit tricky. Five shares are mined at once, but since they aren't yet aware of their siblings, each one sees the top level of the DAG as having only one share:

Rounds 2a-2e:
1
1
1*
1

Five 1/D chances of getting a 1/4-block reward. Finally:

Round 3:
1
5
1
1*

One 1/D chance of getting a 1/8-block reward.

Total expected reward = 2*(1/4D) + 1*(5/4D) + 1*(1/8D) = 15/8D blocks. The fair reward would be 1/D blocks.

The above was a rather contrived and extreme example. I wrote a script to simulate the rewards for a N=2016 PPLNS system with between 1 and 10 shares per level, and the typical revenue per share varies by less than 1%, with the maximum I've seen being 2.6%.

Again, you can't hop based on this, because it depends on events that happen after you mine a share, which can't be predicted. It's a selfish mining vulnerability, but not a hopping vulnerability.

And also again, this variation is completely eliminated by the PPLKD scheme I described earlier.

Here's some more precise numbers. I only own Machine 1, so that's the only one I've tried to overclock so far. Still testing its limits.

Machine 1:
1. 13.7 TH/s
2. 14.2 TH/s
3. 668 MHz
4. 10°C - 20°C
5. 55°C max Temp(PCB)
6. 85°C max Temp(Chip)

Machine 2 ("m5"):
1. 13.7 TH/s
2. 13.7 TH/s
3. 650 MHz
4. 10°C - 20°C
5. 55°C max Temp(PCB)
6. 86°C max Temp(Chip)

Machine 3 ("m6"):
1. 8.0 TH/s
2. 11-12 TH/s
3. 575 MHz
4. 10°C - 20°C
5. 55°C max Temp(PCB)
6. 86°C max Temp(Chip)

Machine 4 ("m7"):
1. 13.8 TH/s
2. 13.8 TH/s
3. 650 MHz
4. 10°C - 20°C
5. 56°C max Temp(PCB)
6. 87°C max Temp(Chip)

Machine 5 ("m8"):
1. 7.5 TH/s
2. 11.5-12.5 TH/s
3. 575 MHz
4. 10°C - 20°C
5. 53°C max Temp(PCB)
6. 86°C max Temp(Chip)

By "on the next attempt" do you mean on the next share, or on the next level of the DAG, or something else? If you mean on the next share, then you need to multiply by M/N to correct for the expected number of shares per DAG level. According to PPLNS, your share only gets kicked off after a certain number of DAG levels, so you get multiple "attempts" per level.

If you're referring to the situation where you mine a share, then the number of shares per level changes abruptly, then yes, there will be anomalies in expected revenue. I addressed that issue in the part of my post that you didn't read. It doesn't lead to hopping, though, because it depends on events that happen after a share is found, whereas hopping requires the miner to make decisions based on events that happen before a share is mined.

It does happen with PPLNS, because the difficulty of the last N shares is not constant. As I mentioned in my earlier post. And it's easy to fix. As I mentioned in my earlier post.

To clarify, expected reward for a share is independent of the work done since the last block was found. If there are more shares per DAG level/height, that means that your share will stay in the last N height for a longer amount of time, so it will have a greater probability of being rewarded, but the reward will be smaller.

Let M equal the number of shares in the last N levels of the DAG. The average number of shares per level is M/N. Assume all shares have equal difficulty for now, and assume steady-state conditions, where M stays roughly constant for a single share's transit through the last N levels. Should a block be found, each share's reward will be 1/M. The probability that a block will be found depends on the amount of work done among those M shares. Since all shares have equal difficulty, that means that the probability of a block being found will be M/D, where D is the ratio of the network difficulty and the difficulty per share. Under steady state conditions, the expected reward for a share is equal to the probability that a block will be found in the next N levels times the reward should a block be found, and will equal (M/D) * (1/M) = 1/D. This means that the expected reward for a share is independent of M as long as the equal-difficulty and steady-state assumptions hold.

And what if they don't hold? Both can bring up a potential problem: a malicious or selfish entity can change their mining behavior in order to flush out or prolong the lifetime of a cohort of shares. If Alice has a burst of good luck, and mines 10 shares out of 20 when she normally should only mine 2 shares out of 20, then she can change her subsequent mining behavior in order to keep that burst of good luck in the last N levels for as long as possible. She can do this either by working on extra-high-difficulty shares or by intentionally mining siblings instead of children on the DAG. On the flipside, when Bob has a spell of bad luck, he can erase it faster by working on low-difficulty shares and trying to always mine DAG children. Note that this vulnerability already exists in p2pool, as individual nodes can specify different share difficulties. The DAG does not create the issue; it's simply an issue with PPLNS with variable difficulty. The issue also does not promote hopping, as hopping only makes sense when the expected reward for a share depends on the past mining results, whereas this issue makes revenue dependent upon the future behavior of other miners.

In any case, it's simple enough to fix. Instead of doing PPLNS where the window is based on the number of shares (or the height in the share DAG), we can do PPLKW (pay-per last K work) where the window is based on an amount of work performed. We set some work threshold (e.g. enough to find 3 blocks on average), then go back through the DAG one level at a time and count all the shares in each level until the total work counted exceeds our work threshold. Thus, each share will always have the same proportion of the total work regardless of future behavior of other miners. The only way to get an enemy's shares out of the DAG faster is to add hashrate and increase the rate at which blocks that reward them are found.

The shares are partially ordered. Each share has a height associated with it determined by the share's parent. In a share chain, each share that makes it into the final chain has a different height. In a share DAG, multiple shares can have the same height. With both, for PPNLS you reward all shares where share_height >= chain_tip_height - N.

Yes, the analysis of incentives is more complex. I haven't seen any ways in which DAGs could incentivize hopping, as it's still PPLNS and expected reward is independent of the work done since the last block was found. However, there are some potential issues regarding selfish mining on the share DAG with some DAG reward algorithms. I think they aren't too difficult to solve, but it's true that care is required.

Unlikely, as the intake air temperatures have been between 8°C and 20°C while we saw problems. I can't be sure, but it almost seemed like we were having more trouble with the unit at night (8-10°C intake) than during the day.

This sounds more plausible to me. In particular, I suspect an issue during the startup routine for the ASICs. The hashrate stays pretty stable over time as long as I don't restart cgminer, but restarting cgminer even with the same settings can cause the rig to gain or lose a few TH/s. I suspect that some of the strings might be having an issue during startup, causing about half of the ASICs on a hashboard to not hash.

Good news: The Antminer S9 works fine on p2pool with no modifications. I've been running mine on p2pool for the last 8 hours, and it's producing within 1% of the hashrate that I saw when I tested it on a traditional pool.

Toomim Bros received five Batch 1 S9s today. Three of them are hashing away happily at 650 MHz with no issues. They report 13.69, 13.85, and 13.82 TH/s.

The other two ("m6" and "m8") gave low hashrates when run at 650 MHz, around 8 TH/s each. Dropping the frequency to 625 MHz increased the hashrate to 11.5 TH/s for m6 and 11.7 TH/s for m8. Optimal hashrate seems to be at 600 MHz and 12.6 TH/s for m6 and 575 MHz and 12.44 TH/s for m8. Looking at the individual hashboards, it appears that m6 has one weak hashboard (#1) and m8 has two weak hashboards (#1 and #3; note that there is no #2, so the third hashboard is #4).

Power consumption at the wall is 1.37 kW (0.099 J/GH) for the S9 with 13.82 TH/s and a total of 5.06 kW (0.096 J/GH) for the other four totaling 52.58 TH/s. Power supplies used are a mixture of the 90% and 93% efficiency DPS1200FB types, with two hashboards per PSU on all but the 13.82 TH/s unit, which runs exclusively on one PSU without issues. The 1.37 kW is on a single 90% efficiency PSU, whereas the 5.06 kW is half 93% models, which should account for about 0.015 J/GH of the 0.03 J/GH gap in measured efficiency.

I will edit this post soon to add screenshots.