Yesterday we dealt with a diff bomb!
It was a self-inflicted one due to a small problem with the mining algo, but that combined with a few missing notary nodes to almost stop the blocks.
Diff peaked at over 10 billion.
The good news is this was a nice fire fighting drill to see how well komodo responds to such an event. No matter how much you prepare, the unexpected will always arrive unexpectedly. So it is good to be able to respond.
And respond we did. I was already a bit nervous about the rapidly rising diff and had made a corrective release, but we needed a much more forceful way to reduce the diff. I also solved how to avoid such a diff explosion (and implosion) in the future and the good news for non-notary miners is that at least 1 block in 64 will be available for normal mining. Additionally any miner that is not online to mine their blocks will lose it to the non-notary miners. Though the notaries can also do non-notary mining.
I am not sure what percentage of notaries will be offline at any given moment, but now there is a specific financial penalty of the lost KMD to provide a direct feedback to incentivize being online 100% of the time.
By having at least 1 block in 65 go to external miners, the diff will be tied to the actual hashrate. It might sound easy to create a dual-diff mechanism that allows notary nodes to mine at easy diff, while still allowing non-notary nodes to mine at normal diff, with a seamless transition between the two.
Its actually quite tricky to avoid all the edge cases and I have had to evolve the algo to arrive at the current solution. Initially it was a literal round-robin where for a given block only one notary was able to mine it at low diff. This was not bad, but it affected the block production whenever a notary node is not online. Also, the lowdiff makes it trivial to find a block and without a delay, we would end up with a really fast block.
There is another issue that needs to be avoided and that is a mining war between notary nodes. While this is not the end of the world, the financial model is affected as the mined KMD is expected to be a significant part of the notary's revenues and we are wanting to minimize the BTC subsidy needed. If real mining resources are needed on top of everything else, each notary's financial equation changes, not to mention the reallocation of the precious KMD.
Ideally, we can rely on the notary nodes to not compete as they are a team, but considering the massive controversy over what region some border countries belong to, I would feel better if there was a technical enforcement of the round robin.
Now this timer is what created the diff explosion. I had it set to 45 seconds, figuring that it was close to 60 seconds and at the time it was before the ratification of the 64 notaries, so there was a lot of non-notary blocks, thus minimizing the faster block issue. Granted if all notaries changed their timer to 30 seconds or less for whatever reason, this would have been an issue.
Then the ratification of the 64 notaries got into the blockchain and activated.
At that point the vast majority of blocks came in at 45 seconds to 50 seconds, which triggers a diff increase to slow down the blocks. Of course, the notary exemption doesnt care what the actual diff is, so we ended up in a never ending increase of the diff. Which by itself is not such and issue as if a notary is going to mine a block at low diff, what does it matter? It matters when the designated notary node is not there and it would take 20 million CPUs to mine the next block!
I had enhanced the set of eligible notary nodes for low diff to be the chosen node plus any notary that didnt generate any of the previous 63 blocks. Which worked quite well until we got to where all active notaries had generated a block recently and the current chosen notary was missing.
After I released a few hotfixes, we ended up not able to tell which was the right mainchain as the blocks were so slow and you cant resolve a fork on the block it forks. Anyway, after a notary wide fire drill to update to the hotfix version, we got the blocks coming in regularly again and the diff is down over 1000x and is currently ~3 million. Another day and it should get down to a reasonable mineable level which is required for proper operation of the system.
The current post-82000 height solution to this is as follows:
A) At any given height, any notary that hasnt generated a block in the prior 65 blocks is eligible for low diff.
B) After a block is mined, a minimum of 59 seconds from start of mining is ensured before submitting to the network, with another 0 to 2 seconds randomly added.
I believe these two simple rules achieves all the desired properties, including avoiding a mining war between notary nodes and preventing artificial speeding up of blocks.
There is little to gain from reducing the 59 second delay as after a block is mined, a notary needs to wait for all the others to mine a block and also the one external. So all reducing the time would do is create one fast block. And if many notaries somehow all decide to do this, we have the diff explosion to disincentivize such fiddling. What point to generate blocks a bit faster if it just creates a diff explosion event. Also, it will be easily seen on the blockchain that a specific notary has adjusted the setting, so it is not anything that would go unnoticed.
Blocks will come in like clockwork, literally, as most all the blocks are based on the clock 59 seconds + random(2 seconds) for average of 60 seconds. Currently this is not active, but already it can be seen that blocks are quite regular:
http://www.zpool.ca/explorer/1720This method is also very resilient from a missing notary condition as it does not matter which notary, just any notary that doesnt have a recent (within 65 blocks) mined block. which means if any such notary is available, a new block arrives. After the initial race as to who wins the intial blocks, we end up in a randomized round robin as at any given block, all notaries have already mined a recent block and have to wait for the single eligible node to mine its block.
And here is where the external (non-notary) mined block comes into play to calibrate the diff to reality. Whenever all notaries are waiting, which would be (1 + inactive notaries) blocks per 65, the block is up for grabs for external miners and ineligible notaries.
It is a good sign that by simplifying things it achieves more. If anybody sees any issues, please post! There might well be a way to improve it more, but at least the current solution guarantees blocks to online notaries, gets us clockwork blocks, calibrates diff to reality and penalizes offline notaries.
My assessment is that Komodo during pre-mainnet has responded to a blockchain event as well or better than most mainnet coins and kudos to all the notary operators!
On the coding side, I am down to a couple of final issues dealing with the fiat chains and once that is done the current mainchain can become the official mainchain. I estimate 95%+ chance that the current "testnet" chain will become the actual mainnet, which means all test coins mined are the real KMD
