Latest improvement of CalculateNextWorkRequired and smoothing the difficulty?

[TheWolf666]

I am using Basically the same CalculateNextWorkRequired in pow.cpp than Bitcoin in another project.
I only changed the new difficulty target evaluation to move by x2 instead of x4 with Bitcoin.

When my coin was young, difficulty was around 5-10 and we were mining with CPU, that was the cool time.
This algorithm was fine because the steps up and down x2 were not changing the mining requirement that much.

For example we were Diff 5, then it moved to 10, then back to 5, and the blockchain were never stalling because the difficulty was too high.

Now we are at difficulty between 10 Millions and 100 Millions. When the difficulty changes from 10 Millions to 20 Millions, you need to drastically change your hashing power, from 20 TH/s to 50TH/s otherwise, the blockchain stall for hours, or even forever until someone find a block if you are not paying attention.

Are there new algorithms that are more adapted, for example that are changing dynamically the difficulty requirement without finding a new block as soon as the blockchain is stalling more than x minutes?

The current algorithm is not really smart, and require constant re-adjustment for the miners.

Any smart suggestion as a replacement of CalculateNextWorkRequired?

[NotATether]

For reference, this is the part of CalculateNextWorkRequired() that you're trying to replace:

Code:

    // Limit adjustment step
    // this section is left unchanged I am just pasting it here for reference
    int64_t nActualTimespan = pindexLast->GetBlockTime() - nFirstBlockTime;
    if (nActualTimespan < params.nPowTargetTimespan/4)
        nActualTimespan = params.nPowTargetTimespan/4;
    if (nActualTimespan > params.nPowTargetTimespan*4)
        nActualTimespan = params.nPowTargetTimespan*4;

    // Retarget
    // this is the actual section you must change
    const arith_uint256 bnPowLimit = UintToArith256(params.powLimit);
    arith_uint256 bnNew;
    bnNew.SetCompact(pindexLast->nBits);
    bnNew *= nActualTimespan;
    bnNew /= params.nPowTargetTimespan;

You probably already know this but nPowTargetTimespan is the epoch interval (two weeks if you used Bitcoin's chainparams) and n_actualTimeSpan takes the difference between the first and last block times in the epoch and powLimit is the biggest target possible for the smallest difficulty.

Before this snippet there's an If condition that checks if retargeting has been disabled, and I don't think you should change that. After this snippet is a limit check that caps the target to the maximum number powLimit.

If your chain is having a drought of blocks mined then there are two specific cases to worry about: one when there is only one block mined during the epoch (pindexLast->GetBlockTime() == nFirstBlockTime and nActualTimeSpan = 0), and when there are no blocks mined during the epoch (all three variables above are undefined or at least have some dummy default value).

Then the target is multiplied by actualTimeSpan/bPowTargetTimespan (that's 2 weeks in bitcoin, unless you changed it in your chainparams).

I suggest you make a special case when there are zero mined blocks in the interval, to automatically decays the difficulty by some percentage factor. Then if if the chain becomes dormant by nobody mining blocks to it in multiple epoches, the difficulty will get progressively lower until it is restored to easier levels. Maybe even put it in your chainparams if you want to play around with it on your network. In other words, change it to something like this:

Code:

    // Retarget
    const arith_uint256 bnPowLimit = UintToArith256(params.powLimit);
    arith_uint256 bnNew;
    bnNew.SetCompact(pindexLast->nBits);

    int64_t nLastBlockTime = pIndexLast->GetBlockTime();
    if (nLastBlockTime == nFirstBlockTime) {
       // These parameter don't exist, I made them up and
       // you have to define it in params.h and chainparams.cpp
       bnNew *= params.droughtAdjustmentNumerator;
       bnNew /= params.droughtAdjustmentDenominator;
    }
    else {
        bnNew *= nActualTimespan;
        bnNew /= params.nPowTargetTimespan;
    }
    // ...

Because the adjustment has to be done with integers and not floating-point numbers, two additional parameters are needed to simulate a percentage adjustment.

[TheWolf666]

@NotATether thank you for this very interesting answer, I will do some tests.

One question

Code:

       bnNew *= params.droughtAdjustmentNumerator;
       bnNew /= params.droughtAdjustmentDenominator;

I am not sure about these 2 params. Let say that in that case I want to lower the difficulty by 1.1 (or mult by 0.9) what those 2 values would be?

[NotATether]

@NotATether thank you for this very interesting answer, I will do some tests.

One question

Code:

       bnNew *= params.droughtAdjustmentNumerator;
       bnNew /= params.droughtAdjustmentDenominator;

I am not sure about these 2 params. Let say that in that case I want to lower the difficulty by 1.1 (or mult by 0.9) what those 2 values would be?

Just take the difficulty percentage or decimal you want to multiply and represent it as a fraction. In this case, to adjust it by 0.9, the numerator is set to 9 and the denominator to 10.

[TheWolf666]

@NotATether thank you for this very interesting answer, I will do some tests.

One question

Code:

       bnNew *= params.droughtAdjustmentNumerator;
       bnNew /= params.droughtAdjustmentDenominator;

I am not sure about these 2 params. Let say that in that case I want to lower the difficulty by 1.1 (or mult by 0.9) what those 2 values would be?

Just take the difficulty percentage or decimal you want to multiply and represent it as a fraction. In this case, to adjust it by 0.9, the numerator is set to 9 and the denominator to 10.

This adjustment would happen every time a block is submitted, so it might go down really quickly. So the difficulty will go down quickly until a block is found.
Then if a block is found, the difficulty will goes up again x4, which will trigger the adjustment again.
This will result in a permanent oscillation, what do you think?
 
Another idea would be to check the time when the new block is found and adjust every block up or down.
Or just reject any block that is found < 10 min and start to accept blocks >=10 min with this adjustment to make sure that blocks are found very quickly once the 10 min are reached?

What is happening for example when the diff goes down is that blocks are easy again to find, so they are found extremely quickly, which retrigger the diff increase, making difficult to lower the diff and easy to goes up.

[NotATether]

This adjustment would happen every time a block is submitted, so it might go down really quickly. So the difficulty will go down quickly until a block is found.
Then if a block is found, the difficulty will goes up again x4, which will trigger the adjustment again.
This will result in a permanent oscillation, what do you think?

That's true if you use large adjustments like 10% reduction, however this code is intended to be used with extremely small adjustments like 0.5% or 1% because the point of this change is to gradually decay the difficulty the more inactive it gets, such that if you leave it inactive for a couple years without mining any blocks to it, the difficulty will go down considerably. With 1% I estimate a more than 25% reduction in difficulty for not mining blocks to it for a year, more because the reduction is compounded not just added to itself.

Another idea would be to check the time when the new block is found and adjust every block up or down.
Or just reject any block that is found < 10 min and start to accept blocks >=10 min with this adjustment to make sure that blocks are found very quickly once the 10 min are reached?

The first part is basically what Core already does now but only for the last block in every epoch. Rejecting blocks doesn't really solve this problem because then mining turns from "who can find a POW first" to "who can submit a block to the network first after 10 minutes" as the difficulty has become extremely low, and opens up the door to DoS attacks where one group submits all the blocks and selectively drops transactions from addresses they don't like.

[TheWolf666]

The first part is basically what Core already does now but only for the last block in every epoch. Rejecting blocks doesn't really solve this problem because then mining turns from "who can find a POW first" to "who can submit a block to the network first after 10 minutes" as the difficulty has become extremely low, and opens up the door to DoS attacks where one group submits all the blocks and selectively drops transactions from addresses they don't like.

And what about waiting 10 min, then share with who got a correct block in the interval?

For example store each block found this way:

blockfound[second]
rewardresult = (reward/number of seconds) // sliced reward for each second

blockfound is a list of address that found a block at current diff and deserve a share of the reward

Code:

foreach second {
 
 numberofblocksfound= size(blockfound[second]); 

// if nobody found a block in this second give their reward to the next one
 if (numberofblocksfound = empty { rewardresult[second+1]=reward[result[second];}
 else {
    foreach blockfound {
       givereward=   rewardresult / numberofblocksfound;
       givereward(blockfound,givereward); // create the reward transaction for this address, giving its chunk of the reward.
   }
 }
}

[NotATether]

And what about waiting 10 min, then share with who got a correct block in the interval?

For example store each block found this way:

blockfound[second]
rewardresult = (reward/number of seconds) // sliced reward for each second

blockfound is a list of address that found a block at current diff and deserve a share of the reward

I don't see anything wrong with this. You're free to implement whichever you like as its your coin but bear in mind that since the difficulty is still really small from using a large decaying factor, there will possibly be several valid results and as a result the block reward each node will get will be very small, since this change basically puts mining pool code in the consensus.

[TheWolf666]

And what about waiting 10 min, then share with who got a correct block in the interval?

For example store each block found this way:

blockfound[second]
rewardresult = (reward/number of seconds) // sliced reward for each second

blockfound is a list of address that found a block at current diff and deserve a share of the reward

I don't see anything wrong with this. You're free to implement whichever you like as its your coin but bear in mind that since the difficulty is still really small from using a large decaying factor, there will possibly be several valid results and as a result the block reward each node will get will be very small, since this change basically puts mining pool code in the consensus.

You are right, stratum would have to be modified in the pools, which is not something easy to ask.