Fixing Testnet4: proposal

[BayAreaCoins]

I don't consider a fork a new version.

Testnet5 will have a new start to the blocks.

This will remain Testnet4.  Also, it doesn't need 99.99% support to push the consensus rule, and it doesn't take very many miners to convince.

Exchanges have heavy-handed branding abilities.  A clean fork will not be 5 on my website.  No need to further confuse people fresh into Bitcoin.

(We shall see.  I did not see the email.)


As stated earlier Blackhat, we don't need the mailing list... let me see what you got and I'll talk to some people about getting it mined into consensus.  If anyone has a problem with it, they can mine against it...  Tis how this shit works, but everyone wants this fixed. (Even people mining empty blocks)

[stwenhao]

I don't consider a fork a new version.

You don't, but all of the existing software will do. And that's the problem with hard-forks: you have to convince everyone, or otherwise, you have a forked chain. Which is why soft-forks and no-forks are better.

Which means, that if you make your own changes, without merging them into Bitcoin Core, and without making them in a backward-compatible way, then all users of all old versions will land on a different chain.

And then, at block 150k, you will see your chain with ASIC-only blocks, but mempool.space will still show a different chain with CPU blocks. And all users with old Bitcoin Core versions will also land on the old chain.

I did not see the email.

It is now accepted: https://groups.google.com/g/bitcoindev/c/Jsv1VYpewuU/m/1CFgHWfSCQAJ

[BlackHatCoiner]

Made some changes: https://github.com/bitcoin/bitcoin/compare/master...batmanbytes:bitcoin:testnet4-fix

- The hardfork activates at block 151200 instead of 150000. The reason is that block 151200 is the start of epoch 75, and this is less prone to error and confusion.
- At block 151200, difficulty is reset to 1 million, to compensate for the artificially high difficulty caused by the CPU miners. This would result in a small mining race for epochs 75 to around 78, where miners with hashpower would find blocks in a matter of seconds. The difficulty will then 4x in the 76th epoch, then again 4x, and it goes up until it naturally finds the 10 minute block interval.

The number 1 million was selected arbitrarily to get the best of both worlds: (1) it won't result in a block storm (as it would if difficulty was reset to 1) and (2) it won't result in very long block intervals. If there weren't any difficulty reset, and difficulty would just be calculated as normal, it'd take around an hour for a block to be found, and this would mean epoch 75 alone would take around three months instead of two weeks.

[BlackHatCoiner]

I have pulled a request to Bitcoin Core: https://github.com/bitcoin/bitcoin/pull/34420

[stwenhao]

You need to do a rebase, instead of merging bitcoin:master to your branch, but I guess it is the only failing test case for now: https://github.com/bitcoin/bitcoin/actions/runs/21402441839/job/61617138146?pr=34420

Code:

Run # Run tests on commits after the last merge commit and before the PR head commit
  # Run tests on commits after the last merge commit and before the PR head commit
  git rebase --exec "git merge --no-commit origin/${GITHUB_BASE_REF} && python3 ./.github/ci-test-each-commit-exec.py && git reset --hard" 0ffb20dee17858e0f13b23fce3dd0ec06cbeb9a2
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    CI_FAILFAST_TEST_LEAVE_DANGLING: 1
    CIRRUS_CACHE_HOST: http://127.0.0.1:12321/
    REPO_USE_CIRRUS_RUNNERS: bitcoin/bitcoin
    MAX_COUNT: 6
    FETCH_DEPTH: 5
    TEST_BASE: 0ffb20dee17858e0f13b23fce3dd0ec06cbeb9a2
Rebasing (2/4)
Executing: git merge --no-commit origin/master && python3 ./.github/ci-test-each-commit-exec.py && git reset --hard
fatal: refusing to merge unrelated histories
warning: execution failed: git merge --no-commit origin/master && python3 ./.github/ci-test-each-commit-exec.py && git reset --hard
You can fix the problem, and then run

  git rebase --continue


Error: Process completed with exit code 1.

Which means, that instead of:

Code:

d426b26a   Merge branch 'bitcoin:master' into testnet4-fix
886f755c   test: add unit tests for testnet4 min-difficulty fork
721cc8b9   consensus: disable min difficulty blocks on testnet4 after height 151200

You need something like that:

Code:

........   test: add unit tests for testnet4 min-difficulty fork
........   consensus: disable min difficulty blocks on testnet4 after height 151200
27aeeff6   Merge #34328: rpc: make uptime monotonic across NTP jumps
........   (other Bitcoin Core changes)

So, there should be no "merge Core into your code", but rather "Core changes first", and then "your changes on top of them". Also, if you do that correctly, then Pull Request will show only your changes, without any merges from Core.

Edit: https://github.com/batmanbytes/bitcoin/tree/testnet4-fix

This branch is 2 commits ahead of and 134 commits behind bitcoin/bitcoin:master.

You are almost there. Usually, in these cases, I use:

Code:

git rebase -i HEAD~2

Then, you should see your two commits. You can save them somewhere, in a text editor, or a similar place, and temporarily discard them.

After that, you can pull everything from the master. Then you will be up to date.

And if you will have everything exactly as in the bitcoin:master, then you can do:

Code:

git rebase -i HEAD~0

In this case, you will see no commits. You can bring back your changes, which you saved before, and confirm it. Then, after such rebase, if everything will look correctly, you can force-push things to your own branch, and see, if it can pass all test cases.

[BlackHatCoiner]

I think it is now up-to-date with bitcoin-master. Can you confirm? Git is not a tool that I often use.

[stwenhao]

Can you confirm?

https://github.com/batmanbytes/bitcoin/tree/testnet4-fix

This branch is 2 commits ahead of bitcoin/bitcoin:master.

Yes, now it is correct: you are up to date with master, and on top of it, there are your changes.

Git is not a tool that I often use.

It is not that hard: usually, there are some commands for frequently used things, and they are repeated over and over again.

Rebasing the code is a quite common practice: first you clone the master, and make your changes on top of it. Then, the original repository is updated, but your version is still based on the point when you cloned it. And then, saving your commits, and putting them on top of the history tree is a regular practice, which is repeated as long, as things are not merged, because this is how the end result will look like: there will be other changes merged first, and your changes on the top.

This is needed mainly for one reason: to have a clear history, without having the unneeded information, that 20 different people fetched things from master 20 times, which created 400 "merged master to local" commits, which didn't change a single thing in the code, after being merged.

[LoyceV]

Testnet4 now has only a few (ASIC) blocks per day that confirm a few transactions, the rest are all empty. Who can use this to test anything?

Shower thought: what if Testnet coins become "anyone can spend" after a certain amount of time? The time can be based on the old Bitcoin transaction priority: larger UTXOs reach it faster than small UTXOs. Let's say 1 Testnet4 coin last a week, 168 coins last an hour, and 0,0027397 coins last a year. That should be enough to always have "anyone can spend" coins available, as there won't be enough blockspace to move all coins before they become accessible by anyone. This will make hodling large quantities futile, and should be combined with removing the 20 minute rule so empty blocks will not stop this. It will probably lead to large blocks and higher fees for a while, so I can't really predict how it turns out.

[BlackHatCoiner]

I can predict that someone will write a bot that immediately spends coins the second they become spendable. It is not difficult or complicated to do at all. You just need the UTXO set and a timer.

I would rather have the 20-min rule extended to something like 120-min instead, than do this. More complicated code and all coins will inevitably end up to the bot owner. I would rather have no min-difficulty rule at all, of course.

[LoyceV]

I can predict that someone will write a bot that immediately spends coins the second they become spendable. It is not difficult or complicated to do at all. You just need the UTXO set and a timer.

My theory was that this won't be possible for all coins: there are now 6.3 million coins in circulation. If all of them would end up in 1 coin addresses, that means 6.3 million coins need to change UTXO each week. I think that just fits the total blockspace, a few million coins more would make it impossible to move all. Which means there will always be coins available to send at higher fee than the rest.

But I agree it's far from perfect.

I would rather have the 20-min rule extended to something like 120-min instead, than do this.

Make it 180 minutes so it can only be abused if there are no real blocks for an hour

[BlackHatCoiner]

My theory was that this won't be possible for all coins: there are now 6.3 million coins in circulation. If all of them would end up in 1 coin addresses, that means 6.3 million coins need to change UTXO each week. I think that just fits the total blockspace, a few million coins more would make it impossible to move all.

Where this idea falls short is that the bot attacker can also be the CPU miner who can just double-spend the expired-UTXO spender's coins. And it'd also be far from mimicking mainnet. You can't test scripts like Hashed Time Lock Contracts.

fjahr reviewed my changes recently. There's disagreement on the fork height and the difficulty adjustment at the fork height.

[LoyceV]

There's disagreement on the fork height and the difficulty adjustment at the fork height.

Increasing the 20 minutes to a lot more than the maximum amount the "time" can be off looks much easier Even making it 240 minutes would lead to more blocks-with-transactions per day (in case ASIC miners quit) than the current situation.

[BlackHatCoiner]

Increasing the 20 minutes to a lot more than the maximum amount the "time" can be off looks much easier

The time can be off by up to 2 hours of the median time of the latest 11 blocks, IIRC. The 20 minutes rule has nothing to do with this. It allows you to send a block with difficulty=1 if the last block has a timestamp 20 min + 1 sec before the current time. Of course, you can't add an infinite number of blocks by just changing their timestamp, because nodes will reject blocks that defy the 2-hour former rule.

The best argument for your 180-min suggestion is that it does make things less worse, and it is only like a few lines change. It is a hardfork though, as well.

[BayAreaCoins]

Great job Blackhat!!! I'll pass this review around.  We shall see...

If you would like to speak with the main miners, like Fjahr mentioned, just let me know and I'll be happy to put you in touch.

Another tricky question seems on how this could be introduced with the least disruption. Since this has missed v31 and roughly the current activation height is around the v32 release, ideally I think activation should be pulled forward and if enough Testnet4 miners could be convinced to run the patch it would obviously be much easier to merge this PR into Core to have it in v32. If you are doing the outreach to miners part @batmanbytes I think that would be motivating to get more review here short term.

As previously stated, I don't think this will be a big deal... if this is our biggest hurdle... hell ya.


It doesn't need to be increased or decreased; it needs to be removed.

Anything else will result in gamesmanship.

What is happening is these blocks aren't being solved at 1 dif... they are being presolved and slammed to submit at 20 min + 1 seconds.

Making Testnet more like Bitcoin will ramp up the mining by individuals and blocktimes won't be an issue.  

Testnet4 has a multiple-party pool infrastructure for people to even rent miners. (Testnet3 was "you are on your own"), but there is no point in even trying vs these folks who are mining in a different style that only fits this one rule.

Any funky rule will result in funky mining (or whatever, lol bouncing coins etc, which would take Blockchain space from real Testers)... One of decentralized Testnet main purposes is mining testing, so why introduce funky stuff... let people point next gen rigs at it and such... it's a warzone, let it war.

The only person who has complained to me about their testing is PortlandHodl and I have a feeling he has plenty of mining power at his disposal if the rules are right. (He works at Mara... these aren't smucks)

[BlackHatCoiner]

A few changes I made:

- I increased fork height to block 201,600 (the epoch 100 boundary), since we need more time for further discussion.
- I removed the difficulty adjustment reset to difficulty 1,000,000. One great argument for having no difficulty resets is that it is less complicated in the code, and I believe this means it has higher chances of getting merged.

I have also created a thread on delvingbitcoin: https://delvingbitcoin.org/t/disabling-min-difficulty-blocks-on-testnet4/2350

[ertil]

I increased fork height to block 201,600 (the epoch 100 boundary), since we need more time for further discussion

You don't need to bump it further. Because then, you will be doing that forever. Just deploy something, see, how it goes, and then use your test network, with your rules, if it works.

Because if you assume, that first Core developers will give you some kind of approval, and only then your network will be deployable, then you are wrong. Previously, testnet4 was first deployed, then supported by mempool.space and others, and then included into Bitcoin Core, when it already produced around 40k blocks.

Which means, that in this case, you can do it in a similar way, as fjahr did: first you demonstrate, that your changes are supported, and it works better than previously, and then Bitcoin Core will include it, if it will be better, than currently deployed testnet4.

But if you keep waiting, and discussing, then it can be delayed forever. And if there will be some silence (because many people doesn't care), then you will just keep bumping the block number forever, without any reason. If it is a hard-fork, then you don't have to wait for anyone's approval anyway, you can just agree with some miners and users, and deploy it sooner, if they are ready.

Block 151,200 still means around 12 difficulty adjustments to the future. Which means half a year. I think many miners will be ready sooner than that.

[BayAreaCoins]

I increased fork height to block 201,600 (the epoch 100 boundary), since we need more time for further discussion

You don't need to bump it further. Because then, you will be doing that forever. Just deploy something, see, how it goes, and then use your test network, with your rules, if it works.

Because if you assume, that first Core developers will give you some kind of approval, and only then your network will be deployable, then you are wrong. Previously, testnet4 was first deployed, then supported by mempool.space and others, and then included into Bitcoin Core, when it already produced around 40k blocks.

Which means, that in this case, you can do it in a similar way, as fjahr did: first you demonstrate, that your changes are supported, and it works better than previously, and then Bitcoin Core will include it, if it will be better, than currently deployed testnet4.

But if you keep waiting, and discussing, then it can be delayed forever. And if there will be some silence (because many people doesn't care), then you will just keep bumping the block number forever, without any reason. If it is a hard-fork, then you don't have to wait for anyone's approval anyway, you can just agree with some miners and users, and deploy it sooner, if they are ready.

Block 151,200 still means around 12 difficulty adjustments to the future. Which means half a year. I think many miners will be ready sooner than that.

I agree with this as well. (out of merit )

Bitcoin Core already muffed the date in January and didn't even bother to say why or provide any update... because you know, testnet, who cares.