Suddenly raised regtest difficulty

[stwenhao]

Let's assume, that we want to bring back difficulty adjustments into regtest, just to test, if mining works correctly. This is how it can be done:

Code:

$ git diff HEAD~1
diff --git a/src/kernel/chainparams.cpp b/src/kernel/chainparams.cpp
index 0f128d4c56..24ac639698 100644
--- a/src/kernel/chainparams.cpp
+++ b/src/kernel/chainparams.cpp
@@ -530,7 +530,7 @@ public:
         m_chain_type = ChainType::REGTEST;
         consensus.signet_blocks = false;
         consensus.signet_challenge.clear();
-        consensus.nSubsidyHalvingInterval = 150;
+        consensus.nSubsidyHalvingInterval = 210000;
         consensus.BIP34Height = 1; // Always active unless overridden
         consensus.BIP34Hash = uint256();
         consensus.BIP65Height = 1;  // Always active unless overridden
@@ -543,7 +543,7 @@ public:
         consensus.nPowTargetSpacing = 10 * 60;
         consensus.fPowAllowMinDifficultyBlocks = true;
         consensus.enforce_BIP94 = true;
-        consensus.fPowNoRetargeting = true;
+        consensus.fPowNoRetargeting = false;
         consensus.nRuleChangeActivationThreshold = 108; // 75% for testchains
         consensus.nMinerConfirmationWindow = 144; // Faster than normal for regtest (144 instead of 2016)

And then, let's try to generate regtest blocks with just a CPU, in some infinite bash loop, to make it simple:

Code:

while [ 1 ]
do
    ./bitcoin-cli -regtest generatetoaddress 1 bcrt1pfeesnyr2tx 100000000
done

Then, the initial regtest difficulty is set to "207fffff", which we can see, when we explore blocks, up to block number 143:

Code:

$ ./bitcoin-cli -regtest getblockhash 143
561f4c44c9967c922030b7f0f40d7af912ab373dc5b7b8fce64bf1092c2451c8
$ ./bitcoin-cli -regtest getblock 561f4c44c9967c922030b7f0f40d7af912ab373dc5b7b8fce64bf1092c2451c8
{
  "hash": "561f4c44c9967c922030b7f0f40d7af912ab373dc5b7b8fce64bf1092c2451c8",
  "confirmations": 631,
  "height": 143,
  "version": 536870912,
  "versionHex": "20000000",
  "merkleroot": "01b6fd85017ab60caad4c82c0ce7e2c0939330a23e2d20dc6c5c2a5a9c36865f",
  "time": 1756365704,
  "mediantime": 1756365703,
  "nonce": 0,
  "bits": "207fffff",
  "difficulty": 4.656542373906925e-10,
  "chainwork": "0000000000000000000000000000000000000000000000000000000000000120",
  "nTx": 1,
  "previousblockhash": "27f1b07a0c88220ec90617e95dbf92990fcb1c59e5a777efcea599a18113ec75",
  "nextblockhash": "0000b2dd4a19a3f446fc8f3ad8b2ba5472f2fbc962c8ce8c34e234afad91113e",
  "strippedsize": 196,
  "size": 232,
  "weight": 820,
  "tx": [
    "01b6fd85017ab60caad4c82c0ce7e2c0939330a23e2d20dc6c5c2a5a9c36865f"
  ]
}

However, when we check the next block hash, it suddenly has much more leading zeroes, than we would expect, if the difficulty would raise by a factor of four:

Code:

$ ./bitcoin-cli -regtest getblock 0000b2dd4a19a3f446fc8f3ad8b2ba5472f2fbc962c8ce8c34e234afad91113e
{
  "hash": "0000b2dd4a19a3f446fc8f3ad8b2ba5472f2fbc962c8ce8c34e234afad91113e",
  "confirmations": 662,
  "height": 144,
  "version": 805306368,
  "versionHex": "30000000",
  "merkleroot": "3e5394b4f8d0e50c2ff4d268ddabbb942584b501e37701238196f783a24d00a7",
  "time": 1756365704,
  "mediantime": 1756365703,
  "nonce": 186410,
  "bits": "1f00be2e",
  "difficulty": 2.053947215238338e-05,
  "chainwork": "00000000000000000000000000000000000000000000000000000000000159b9",
  "nTx": 1,
  "previousblockhash": "561f4c44c9967c922030b7f0f40d7af912ab373dc5b7b8fce64bf1092c2451c8",
  "nextblockhash": "0000946b847eda6f6ba7dfaf9e4d34538f3983997bcefba73e5f7501de365250",
  "strippedsize": 196,
  "size": 232,
  "weight": 820,
  "tx": [
    "3e5394b4f8d0e50c2ff4d268ddabbb942584b501e37701238196f783a24d00a7"
  ]
}

And now, it raised from "207fffff" to "1f00be2e", which is not just "four times more difficult" (the maximum adjustment), but rather something like "at least 44108 times more difficult":

Code:

bits2diff(207fffff)=7fffff00 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bits2diff(1f00be2e)=0000be2e 00000000 00000000 00000000 00000000 00000000 00000000 00000000
7fffff0000000000000000000000000000000000000000000000000000000000/0000be2e00000000000000000000000000000000000000000000000000000000=ac4c
hex2dec(ac4c)=44108

Do you know, why it is the case? Next adjustments are not as sharp, for example:

Code:

$ ./bitcoin-cli -regtest getblockhash 287
000076f923e02cd924e1e5a5d563e05a6d4dcd34ad0efc08ecd42a13203dfaeb
$ ./bitcoin-cli -regtest getblock 000076f923e02cd924e1e5a5d563e05a6d4dcd34ad0efc08ecd42a13203dfaeb
{
  "hash": "000076f923e02cd924e1e5a5d563e05a6d4dcd34ad0efc08ecd42a13203dfaeb",
  "confirmations": 572,
  "height": 287,
  "version": 805306368,
  "versionHex": "30000000",
  "merkleroot": "88fbb4039268064fa8de221df70271eb823518c84b62dbf1b7d02f857628c423",
  "time": 1756365728,
  "mediantime": 1756365727,
  "nonce": 23879,
  "bits": "1f00be2e",
  "difficulty": 2.053947215238338e-05,
  "chainwork": "0000000000000000000000000000000000000000000000000000000000c1d730",
  "nTx": 1,
  "previousblockhash": "0000795f0a85fe1d65d5b6200416d6e97f6cc578661ab0d796159bc4014d8e79",
  "nextblockhash": "00002b023539977a6b783706829ed05eb5be422c9c318317939262f954d8d016",
  "strippedsize": 196,
  "size": 232,
  "weight": 820,
  "tx": [
    "88fbb4039268064fa8de221df70271eb823518c84b62dbf1b7d02f857628c423"
  ]
}
$ ./bitcoin-cli -regtest getblock 00002b023539977a6b783706829ed05eb5be422c9c318317939262f954d8d016
{
  "hash": "00002b023539977a6b783706829ed05eb5be422c9c318317939262f954d8d016",
  "confirmations": 573,
  "height": 288,
  "version": 805306368,
  "versionHex": "30000000",
  "merkleroot": "693cf9bcbd2100f057cf21e2b9b504dc97746d4261f379c5b4aa20f49c21eaca",
  "time": 1756365728,
  "mediantime": 1756365727,
  "nonce": 207096,
  "bits": "1e2f8b80",
  "difficulty": 8.215788860953354e-05,
  "chainwork": "0000000000000000000000000000000000000000000000000000000000c73996",
  "nTx": 1,
  "previousblockhash": "000076f923e02cd924e1e5a5d563e05a6d4dcd34ad0efc08ecd42a13203dfaeb",
  "nextblockhash": "000028869e8660bb0a0000a27f53a001fce85ed7b0a7baa80a57ce333b09e3f7",
  "strippedsize": 196,
  "size": 232,
  "weight": 820,
  "tx": [
    "693cf9bcbd2100f057cf21e2b9b504dc97746d4261f379c5b4aa20f49c21eaca"
  ]
}

And then, if we count, how many times it raised:

Code:

bits2diff(1f00be2e)=0000be2e 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bits2diff(1e2f8b80)=00002f8b 80000000 00000000 00000000 00000000 00000000 00000000 00000000
0000be2e00000000000000000000000000000000000000000000000000000000/00002f8b80000000000000000000000000000000000000000000000000000000=4

Only four times, as expected. So, why it was not the case during the first adjustment?

[nc50lc]

However, when we check the next block hash, it suddenly has much more leading zeroes, than we would expect, if the difficulty would raise by a factor of four:

Could be because RegTest has "fPowAllowMinDifficultyBlocks = true" parameter, which can cause it to pass the permitted difficulty transition limits.
github.com/bitcoin/bitcoin/blob/7cc9a087069bfcdb79a08ce77eb3a60adf9483af/src/pow.cpp#L91

The exact reason how it came up with that value however, I can't tell.

[stwenhao]

Could be because RegTest has "fPowAllowMinDifficultyBlocks = true" parameter, which can cause it to pass the permitted difficulty transition limits.

No, it only allows mining blocks with minimal difficulty after 20 minutes, just like it is possible in testnet3 and testnet4.

However, I am still curious, why the real network difficulty is changed in that way. And it also bounces back and forth, when blocks have 15 minutes of delay between each other, without reaching the minimal value, as it should.

[WhyFhy]

Since you mined 144 blocks in zero seconds, the retarget math didn’t just give you 4x it gave you 144x600÷elapsedtime which came out ~44k. Then the ±4x kicked back in on the next window.

It's averaged but the first blocks have nothing to baseline against. You also made it more aggressive going 2016 to 144.

tldr you gamed your own system and got punished.

[stwenhao]

You also made it more aggressive going 2016 to 144.

This is the default value for regtest. I guess I should replace it with 2016.

tldr you gamed your own system and got punished

The difficulty can be trivially lowered, just by using more than 20 minutes of delay between regtest blocks, no matter how huge the network difficulty will be. Another interesting thing is if the difficulty will be bumped for example from 0x207fffff to 0x1f00be2e, then guess what: even if you use hours between blocks, then it won't decrease back to 0x207fffff. Do you know why? And why it bounces back and forth, even if the time between blocks is set to 15 minutes, instead of constantly decreasing?

[WhyFhy]

The difficulty can be trivially lowered, just by using more than 20 minutes of delay between regtest blocks, no matter how huge the network difficulty will be. Another interesting thing is if the difficulty will be bumped for example from 0x207fffff to 0x1f00be2e, then guess what: even if you use hours between blocks, then it won't decrease back to 0x207fffff. Do you know why? And why it bounces back and forth, even if the time between blocks is set to 15 minutes, instead of constantly decreasing?

It retargets at 288 if it theoretically takes 1-2 months on your same settings after you hit block 288. (Stick with the punishment and see how it corrects)
It should agressivley retarget possibly even lower than your target.

It looks like your algo's doing what you told it to. Your deterministic variables are elapsed time. ± is overshoot or undershoot. You just way undershot it , and I think you got the book thrown at you.

[BattleDog]

Let's assume, that we want to bring back difficulty adjustments into regtest, just to test, if mining works correctly. This is how it can be done:

Code:

$ git diff HEAD~1
diff --git a/src/kernel/chainparams.cpp b/src/kernel/chainparams.cpp
index 0f128d4c56..24ac639698 100644
--- a/src/kernel/chainparams.cpp
+++ b/src/kernel/chainparams.cpp
@@ -530,7 +530,7 @@ public:
         m_chain_type = ChainType::REGTEST;
         consensus.signet_blocks = false;
         consensus.signet_challenge.clear();
-        consensus.nSubsidyHalvingInterval = 150;
+        consensus.nSubsidyHalvingInterval = 210000;
         consensus.BIP34Height = 1; // Always active unless overridden
         consensus.BIP34Hash = uint256();
         consensus.BIP65Height = 1;  // Always active unless overridden
@@ -543,7 +543,7 @@ public:
         consensus.nPowTargetSpacing = 10 * 60;
         consensus.fPowAllowMinDifficultyBlocks = true;
         consensus.enforce_BIP94 = true;
-        consensus.fPowNoRetargeting = true;
+        consensus.fPowNoRetargeting = false;
         consensus.nRuleChangeActivationThreshold = 108; // 75% for testchains
         consensus.nMinerConfirmationWindow = 144; // Faster than normal for regtest (144 instead of 2016)

And then, let's try to generate regtest blocks with just a CPU, in some infinite bash loop, to make it simple:

Code:

while [ 1 ]
do
    ./bitcoin-cli -regtest generatetoaddress 1 bcrt1pfeesnyr2tx 100000000
done

Then, the initial regtest difficulty is set to "207fffff", which we can see, when we explore blocks, up to block number 143:

Code:

$ ./bitcoin-cli -regtest getblockhash 143
561f4c44c9967c922030b7f0f40d7af912ab373dc5b7b8fce64bf1092c2451c8
$ ./bitcoin-cli -regtest getblock 561f4c44c9967c922030b7f0f40d7af912ab373dc5b7b8fce64bf1092c2451c8
{
  "hash": "561f4c44c9967c922030b7f0f40d7af912ab373dc5b7b8fce64bf1092c2451c8",
  "confirmations": 631,
  "height": 143,
  "version": 536870912,
  "versionHex": "20000000",
  "merkleroot": "01b6fd85017ab60caad4c82c0ce7e2c0939330a23e2d20dc6c5c2a5a9c36865f",
  "time": 1756365704,
  "mediantime": 1756365703,
  "nonce": 0,
  "bits": "207fffff",
  "difficulty": 4.656542373906925e-10,
  "chainwork": "0000000000000000000000000000000000000000000000000000000000000120",
  "nTx": 1,
  "previousblockhash": "27f1b07a0c88220ec90617e95dbf92990fcb1c59e5a777efcea599a18113ec75",
  "nextblockhash": "0000b2dd4a19a3f446fc8f3ad8b2ba5472f2fbc962c8ce8c34e234afad91113e",
  "strippedsize": 196,
  "size": 232,
  "weight": 820,
  "tx": [
    "01b6fd85017ab60caad4c82c0ce7e2c0939330a23e2d20dc6c5c2a5a9c36865f"
  ]
}

However, when we check the next block hash, it suddenly has much more leading zeroes, than we would expect, if the difficulty would raise by a factor of four:

Code:

$ ./bitcoin-cli -regtest getblock 0000b2dd4a19a3f446fc8f3ad8b2ba5472f2fbc962c8ce8c34e234afad91113e
{
  "hash": "0000b2dd4a19a3f446fc8f3ad8b2ba5472f2fbc962c8ce8c34e234afad91113e",
  "confirmations": 662,
  "height": 144,
  "version": 805306368,
  "versionHex": "30000000",
  "merkleroot": "3e5394b4f8d0e50c2ff4d268ddabbb942584b501e37701238196f783a24d00a7",
  "time": 1756365704,
  "mediantime": 1756365703,
  "nonce": 186410,
  "bits": "1f00be2e",
  "difficulty": 2.053947215238338e-05,
  "chainwork": "00000000000000000000000000000000000000000000000000000000000159b9",
  "nTx": 1,
  "previousblockhash": "561f4c44c9967c922030b7f0f40d7af912ab373dc5b7b8fce64bf1092c2451c8",
  "nextblockhash": "0000946b847eda6f6ba7dfaf9e4d34538f3983997bcefba73e5f7501de365250",
  "strippedsize": 196,
  "size": 232,
  "weight": 820,
  "tx": [
    "3e5394b4f8d0e50c2ff4d268ddabbb942584b501e37701238196f783a24d00a7"
  ]
}

And now, it raised from "207fffff" to "1f00be2e", which is not just "four times more difficult" (the maximum adjustment), but rather something like "at least 44108 times more difficult":

Code:

bits2diff(207fffff)=7fffff00 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bits2diff(1f00be2e)=0000be2e 00000000 00000000 00000000 00000000 00000000 00000000 00000000
7fffff0000000000000000000000000000000000000000000000000000000000/0000be2e00000000000000000000000000000000000000000000000000000000=ac4c
hex2dec(ac4c)=44108

Do you know, why it is the case? Next adjustments are not as sharp, for example:

Code:

$ ./bitcoin-cli -regtest getblockhash 287
000076f923e02cd924e1e5a5d563e05a6d4dcd34ad0efc08ecd42a13203dfaeb
$ ./bitcoin-cli -regtest getblock 000076f923e02cd924e1e5a5d563e05a6d4dcd34ad0efc08ecd42a13203dfaeb
{
  "hash": "000076f923e02cd924e1e5a5d563e05a6d4dcd34ad0efc08ecd42a13203dfaeb",
  "confirmations": 572,
  "height": 287,
  "version": 805306368,
  "versionHex": "30000000",
  "merkleroot": "88fbb4039268064fa8de221df70271eb823518c84b62dbf1b7d02f857628c423",
  "time": 1756365728,
  "mediantime": 1756365727,
  "nonce": 23879,
  "bits": "1f00be2e",
  "difficulty": 2.053947215238338e-05,
  "chainwork": "0000000000000000000000000000000000000000000000000000000000c1d730",
  "nTx": 1,
  "previousblockhash": "0000795f0a85fe1d65d5b6200416d6e97f6cc578661ab0d796159bc4014d8e79",
  "nextblockhash": "00002b023539977a6b783706829ed05eb5be422c9c318317939262f954d8d016",
  "strippedsize": 196,
  "size": 232,
  "weight": 820,
  "tx": [
    "88fbb4039268064fa8de221df70271eb823518c84b62dbf1b7d02f857628c423"
  ]
}
$ ./bitcoin-cli -regtest getblock 00002b023539977a6b783706829ed05eb5be422c9c318317939262f954d8d016
{
  "hash": "00002b023539977a6b783706829ed05eb5be422c9c318317939262f954d8d016",
  "confirmations": 573,
  "height": 288,
  "version": 805306368,
  "versionHex": "30000000",
  "merkleroot": "693cf9bcbd2100f057cf21e2b9b504dc97746d4261f379c5b4aa20f49c21eaca",
  "time": 1756365728,
  "mediantime": 1756365727,
  "nonce": 207096,
  "bits": "1e2f8b80",
  "difficulty": 8.215788860953354e-05,
  "chainwork": "0000000000000000000000000000000000000000000000000000000000c73996",
  "nTx": 1,
  "previousblockhash": "000076f923e02cd924e1e5a5d563e05a6d4dcd34ad0efc08ecd42a13203dfaeb",
  "nextblockhash": "000028869e8660bb0a0000a27f53a001fce85ed7b0a7baa80a57ce333b09e3f7",
  "strippedsize": 196,
  "size": 232,
  "weight": 820,
  "tx": [
    "693cf9bcbd2100f057cf21e2b9b504dc97746d4261f379c5b4aa20f49c21eaca"
  ]
}

And then, if we count, how many times it raised:

Code:

bits2diff(1f00be2e)=0000be2e 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bits2diff(1e2f8b80)=00002f8b 80000000 00000000 00000000 00000000 00000000 00000000 00000000
0000be2e00000000000000000000000000000000000000000000000000000000/00002f8b80000000000000000000000000000000000000000000000000000000=4

Only four times, as expected. So, why it was not the case during the first adjustment?

Good experiment. What you're seeing on the first retarget is expected once you look at what the code actually uses for the window and the timestamps.

The first jump is way more than 4x because the retarget formula is

Code:

new_target = old_target * actualTimespan / targetTimespan

, with the 4x clamp applied to actualTimespan (min = target/4, max = target*4).

actualTimespan is computed from MedianTimePast (MTP): MTP(last) - MTP(first_in_window).

In your loop you're blasting blocks as fast as the CPU can make them, so many consecutive blocks share nearly the same timestamp (or advance by 1). That makes the first window's actualTimespan tiny.

Because you re-enabled retargeting on regtest starting from the powLimit (bits = 0x207fffff) and mined the entire first window at that max target, the computed new_target gets pushed far below 1/4 of the old target before the compact encoding, and after Compact rounding the exponent drops from 0x20 to 0x1f (your 0x1f00be2e). That exponent drop alone is a ~256x step; combined with the tiny timespan you end up with the ~44108x you measured.


After that first "snap," subsequent windows have less extreme MTP deltas (and you're no longer right at the powLimit boundary), so the clamp bites and you see the normal ×4 behavior.

How to see/prove it quickly:

Inspect MTPs across your first window:

Code:

bitcoin-cli -regtest getblockheader <height> | jq .mediantime

Compare heights 0, 143, 144. You'll find MTP(last) - MTP(first) is very small on that first window.
Convert bits to targets to see the exponent drop (20h→1Fh) exactly matches your ~44108 ratio.


Some ways to make regtest adjustments behave "boring" are:

Step time forward while mining:
Use setmocktime (or just sleep) so each block advances ~nPowTargetSpacing. Example:

Code:

t=$(date +%s); for i in $(seq 1 200); do
  bitcoin-cli -regtest setmocktime $((t + i*10))
  bitcoin-cli -regtest generatetoaddress 1 $addr >/dev/null
done

Disable the special-min-difficulty path while testing retargeting:
Set fPowAllowMinDifficultyBlocks = false on regtest if you want pure mainnet-like difficulty math.

Keep params consistent, ensure DifficultyAdjustmentInterval == nPowTargetTimespan / nPowTargetSpacing. Mismatched values give non-intuitive jumps when windows roll.

The 4x bound is on time, not on the compact-encoded bits. If your first window's MTP barely moves and you start at powLimit, the first compact normalization can cross an exponent boundary and look like a mega-jump. After the first window, things stabilize.