Well I take full responsibility for this and it is going to cost me big time
I've spent the last day doing a root cause analysis regarding the best share being above network difficulty issue as mentioned here before and after the last high diff share and have concluded this is entirely my fault. This was always going to be a risk with running this pool as the development point for ckpool.
I'll summarise what happened here.
The first high diff share was on code that was halfway merged from old and new code giving support for segwit in the pool code but without actually activating it, while still on a bitcoind that was not signalling support for segwit. There was one failed submission of a block that happened concurrently with another block find on the network and there was nothing unusual about failing to submit a block at that time since it would have been a stale block. However it was the lack of debug from bitcoind that failed to tell me that there was actually a problem with the block submission rather than it just being stale.
In my enthusiasm to further update the code to support segwit, I completed the update to the pool code to request segwit block templates (when it activates) and updated the bitcoind to signal said support, along with adding more debug output from bitcoind for failed block submissions. This was a ridiculous error of judgement on my part to persist with the code.
The second high diff share mentioned here it was clear from bitcoind that there was actually a block submission failure rather than a stale block as there was no concurrent block that happened on the network and the extra debug code I added from bitcoind told me as much this time.
I've since rolled back the pool to older safe code while leaving bitcoind to signal support for segwit.
Now I wouldn't expect any other pool operator to simply ignore these failed block solves so I plan to take full responsibility and adequately compensate those 2 users who may have otherwise found blocks. Note that with the corruption, these
actually would not have been blocks but I'm willing to treat them as such.
As I haven't altered any of the pool or user stats in any way, you will find your best ever share will still show your best share so it is time for you to check your userstats and see if you have one of the following bestever shares:
382969404792 or 821809472417
Please contact me either via PM or email to sign a verification that you are the owner of the associated address and I will organise compensation. As I have logs of who these belong to, you cannot just email me and say it was you without confirming your address.
The worst thing about this is not only would it be almost all the pool's profit for the last 2 years, the code itself works fine on testnet leaving me soul searching for how to prevent it happening on mainnet, and too terrified to deploy it again any time soon. As segwit isn't going to be activated any time soon, there is no great urgency to get the code out there, and it does not mean I have to stop signalling support for segwit. I'm tempted to just shut the pool down if segwit activates instead. This may be the straw that breaks the camel's back. I've been increasingly unhappy with the bitcoin mining world at large for a while now and I may just choose to abandon it entirely.
I've never been so embarrassed online and I'm drinking hard liquor and it's not even midday here. Please send me boxes of tissues for I will be weeping for years to come over this incident
