Fixing Testnet4: proposal

[BlackHatCoiner]

Road C: complicate your code changes: if more than 20 minutes have passed, miners mine one low-difficulty block. But they don't broadcast it, as that would be ignored by all other miners who run your Fork.

It is a smart idea, but it is messy.

What happens if a CPU miner broadcasts a min-difficulty block at the same time the miner tries to add his own? Which one is the miner keeping and why? Bitcoin Core does not know that "this" block is mine. It only sees valid blocks.

[LoyceV]

What happens if a CPU miner broadcasts a min-difficulty block at the same time the miner tries to add his own? Which one is the miner keeping and why?

Miner just keeps the one he created, and mines the next block on top of that. It may very well be too messy for all the required code changes. I just came up with something that could be a softfork, I'm not the one to create the code changes

[NotATether]

I've done the math on nerdminers, and indeed, it's a space heater
But it's not going to help if some ASIC pushed the difficulty up with 10,000 times more hashrate than the BitAxe.

I guess since Antminers and stuff basically have a toggle for testnet mode (right??), asking them nicely to stop is not going to work.

Someone is going to have to run a grassroots operation in a small data center then 

But I think the bigger problem is, they will have to keep up with the latest hardware, otherwise powerful miners turning on/off will mess with the block generation time again.

[stwenhao]

If I understand stwenhao correctly, this should make it a softfork again.

It would make it a no-fork, and fjahr's version can already do that. And if you don't need anything else, then you can just run his version. If the hashrate majority will do so, then it will bring back the normal network difficulty, if this is all you want.

What happens if a CPU miner broadcasts a min-difficulty block at the same time the miner tries to add his own?

Then, each miner will work on its own chain, until the next block will solve it. In general, ASIC blocks solve that kind of things here. ASIC miners could reorg all chains of CPU-only blocks, if they would want to. But instead, they replace it with their own blocks, with minimal difficulty, because it gives them more coins, so there is an incentive to do that.

[hmbdofficial]

Then, each miner will work on its own chain, until the next block will solve it. In general, ASIC blocks solve that kind of things here. ASIC miners could reorg all chains of CPU-only blocks, if they would want to. But instead, they replace it with their own blocks, with minimal difficulty, because it gives them more coins, so there is an incentive to do that.

What prevents ASIC miners from constantly reorging the CPU-only chains?
 You mentioned they have an incentive to replace them with their own blocks instead.. but what stops them from doing aggressive reorgs if it benefits them?

[stwenhao]

What prevents ASIC miners from constantly reorging the CPU-only chains?

Technically? Nothing. They could always build new blocks, on top of the latest ASIC block. If the hashrate majority would do that, then there would be no CPU blocks at all.

but what stops them from doing aggressive reorgs if it benefits them?

It is more profitable to replace some CPU blocks, with your own min-difficulty blocks. Then, by having more coins, they could be sold for BTCs. Because this is the end goal of many altcoins: as long as testnets are used for just testing things, then rejecting thousands of tBTCs from block rewards can be easily done, when they are worthless.

But if 1000 tBTC is something, that you can convert to some real satoshis, then guess what: miners prefer having 6x bigger difficulty, and using the existing system, that works, and gives them many coins in a Speedy Gonzales blockchain, where propagation of min-difficulty blocks is all that matters, than actually fixing the problem, and lowering the network difficulty for everyone.

To change that situation, they would need an incentive. For example: "if this bug will be fixed, then hundreds of thousands of your tBTCs will be worth more BTCs, than they currently are". But because Core Developers believe, that testnet coins should be worthless, and if testnet4 failed at that, then they are simply going to use something else, then nobody cares about actually fixing the situation.

And now, here we are: if someone will create some code, which will fix it, and deploy it properly, then we will have a better testnet4. But if not, then it will stay as it is, and then, after some months or years, people will simply switch to the next version, created from scratch.

Also, as some people probably remember, there were plans to launch testnet5 with 10.5 million premined coins. But when Proof of Work signet faucet succeeded, people simply switched to that, instead of trying to fix testnet4.

Because in open-source projects, things are pushed forward, when there are any developers, interested in doing it. Which can also explain a lot of silence in this topic: the only reason why testnet4 was created, is because fjahr launched it properly, and then it was just included into Bitcoin Core. But if nobody will care about fixing things, then they will be left as they are, just like it is the case with many other abandoned altcoins.

[BayAreaCoins]

I saw the GitHub update from a hard fork to a soft fork this morning.

32a5e57 Comment corrections (changed "hard fork" to "soft fork", because it is not a hard fork without difficulty adjustment to 1 million)

Very nice.  Glad to see that.  It does make life easier if it is doable.

Kind of shocked it isn't a hard fork issue.

83d39a7 Revert back to fork height 151,200

Good call on this as well.  Shit or get off the pot!

[BlackHatCoiner]

It is more profitable to replace some CPU blocks, with your own min-difficulty blocks.

Maybe this is the best solution? fjahr's version works like this: getblocktemplate builds a block on top of the latest real-diff block, ignoring the chain of min-diff blocks that were built on top of it. But this does not incentivize the ASIC miners to run his client. Whether the CPU blocks are reorged or not, does not affect the mining pool in any direct way. The mining pool still earns the same tBTC; it is only indirectly affected by the artificially high difficulty from the CPU miners. But what if the client replaces the last x CPU blocks with the miner's instead of ignoring them?

The only problem is that the problem with empty blocks continues unresolved. It just changes who gets the coinbase reward of those blocks. Transactions still are taking a lot longer than 10 minutes to confirm.

[BayAreaCoins]

It is more profitable to replace some CPU blocks, with your own min-difficulty blocks.

Maybe this is the best solution? fjahr's version works like this: getblocktemplate builds a block on top of the latest real-diff block, ignoring the chain of min-diff blocks that were built on top of it. But this does not incentivize the ASIC miners to run his client. Whether the CPU blocks are reorged or not, does not affect the mining pool in any direct way. The mining pool still earns the same tBTC; it is only indirectly affected by the artificially high difficulty from the CPU miners. But what if the client replaces the last x CPU blocks with the miner's instead of ignoring them?

The only problem is that the problem with empty blocks continues unresolved. It just changes who gets the coinbase reward of those blocks. Transactions still are taking a lot longer than 10 minutes to confirm.

If it were more profitable, "they" would be doing it.

Funny, stwenhao was so anti Testnet profit is now talking in profitable terms.  . (the premine blabber is an idea from Greg Maxwell from Testnet2 to Testnet3 (2010-2011ish) that was rejected; it still isn't a good idea IMO for a number of reasons.)

If we are talking in those terms, I'd say I'm probably one of the best situated users in the world for contributions.  Which I absolutely plan to do and I do contribute as we can... our "testing" isn't cheap either.


I strongly believe dropping the difficulty rule and letting folks hash it out is still absolutely the right direct for the best Bitcoin Testnet can do for Bitcoin.

The thought that only one side can throw stones is veryyyyyy unwise.  If one party does one thing, the unknown party does an unknown thing and then we end up with people crying about stuff being "broken".  I firmly believe atm that it's best to just not have abusable rules rather than us focus on abusing each other because the rules are fucked.

Simplicity.


Also, what changed this morning that caused this to go from a hard fork to a soft fork?

[LoyceV]

See tmp.loyce.club/testnet4_debug_BlackHatCoiner_edition.log for the latest 10,000 log entries (updated every minute).

After 47 hours without a new block, I've abandoned BlackHatCoiner's chain.

If it stays stuck, would it be an option to just restart from block 0 (without the 20 minute rule)? That way it would be able to get started until an ASIC takes over.

What would it take to do this? Call it testnet5, or testnetBHC for all I care. I assume it needs a hard-coded new genesis block and a hard-coded node to connect to. That's all, right? At least that way your chain could get started and someone could start mining blocks without a difficulty that's far too high for testing something.
I've never created my own Fork, and I wouldn't know where to start, but it's tempting

Funny, stwenhao was so anti Testnet profit is now talking in profitable terms.  .

You can be against something and still accept the reality that it's happening.

[BlackHatCoiner]

Also, what changed this morning that caused this to go from a hard fork to a soft fork?

I mistakenly thought that you could mine a real-difficulty block with timestamp more than 20 minutes into the future. stwenhao corrected me, because any block after 20 minutes must have min-difficulty, it is not optional.

We can still make it a softfork, but we'll have to mess with timestamp. I still find it the most user-friendly thing to do. The softfork invalidates any block with timestamp more than 20 minutes into the future.

Assuming the softfork activates in September 1st, and majority of hashrate mines in the fork, the chain would look like in the following table. The epoch 75 would take ~2.5 months to complete, the next one ~1.2 months, and then the timestamp would quickly reach February. So, starting from September, it'd take about 6 months for the average block time to get back to 10 minutes, with timestamp date being up-to-date.

Era	Block	Real date	Timestamp date	Avg block time	Change
75	151,200	1st Sept	1st Sept	60 min	N/A
76	153,216	24th Nov	29th Sept	30 min	-50%
77	155,232	5th Jan	27th Oct	15 min	-50%
78	157,248	26th Jan	24th Nov	7.5 min	-50%
79	159,264	6th Feb	22nd Dec	3.75 min	-50%
80	161,280	11th Feb	19th Jan	1.875 min	-50%
81	163,296	14th Feb	14th Feb	~1 min	~-45%
82	165,312	16th Feb	16th Feb	~4 min	+400%
83	167,328	22nd Feb	22nd Feb	~10 min	+250%

(Difficulty keeps dropping by 50% until timestamp date reaches real date, then it abruptly goes back up.)

[LoyceV]

it'd take about 6 months for the average block time to get back to 10 minutes

This is a huge drawback of a softfork that still keeps the 20 minute rule. Let's say this all works out, and then some Bitcoin miner throws his massive hashrate at testnet for a few hours: he mines 4000 blocks, and testnet is fucked up for another 6 months before it's back to normal timestamps.

[BlackHatCoiner]

This is a huge drawback of a softfork that still keeps the 20 minute rule.

It doesn't. The moment block 151,200 is mined you can no longer send a block with timestamp more than 20 minutes than the previous one, and therefore, all blocks must be mined with real-difficulty to be valid.

Let's say this all works out, and then some Bitcoin miner throws his massive hashrate at testnet for a few hours: he mines 4000 blocks, and testnet is fucked up for another 6 months before it's back to normal timestamps.

This is why I'm presuming the majority of the hashrate will mine my fork. We can know what percentage will mine my fork beforehand, with block signaling. If a miner throws a massive hashrate with the old client, then his chain will be invalid in the forked chain, and unless the majority of the hashrate goes back to the old chain, my chain will outpace the old.

I have noticed that it is mainly Foundry and Mara pool that own the majority of the hashrate. Very often does a new miner throw hashrate at testnet.

[LoyceV]

This is why I'm presuming the majority of the hashrate will mine my fork.

You're thinking of other testnet miners, I'm thinking of a Bitcoin miner who does this. Without checking the difficulty, I guess there's about a million times more hashrate out there than what's currently used for testnet.

[BlackHatCoiner]

You're thinking of other testnet miners, I'm thinking of a Bitcoin miner who does this.

In the case that a Bitcoin miner suddenly switches to mining testnet blocks with the old client, then a temporary split will occur, where he's mining on top of some CPU block, with current timestamp, and Mara & Foundry continue on the softforked chain, rejecting his blocks, and reorging him eventually.

Testnet is not fucked up for another 6 months. Mara & Foundry continue on their journey to February, with no interruption. They just ignore the other chain. Miners do need to be informed about the softfork, though, otherwise they will have their blocks constantly reorged by the majority.

[LoyceV]

In the case that a Bitcoin miner suddenly switches to mining testnet blocks with the old client

I'm not talking about the old client, I'm talking about mining with your client. If someone quickly increases the difficulty and then stops, your client will have to use the wrong timestamp again because each new block has to be found within 20 minutes.

[BlackHatCoiner]

If someone quickly increases the difficulty and then stops, your client will have to use the wrong timestamp again because each new block has to be found within 20 minutes.

I now see what you mean.

The softfork is just one rule: invalidate any block that has a timestamp by 20 minutes more than its previous one. That's it. In pseudocode, when a miner would call to get a block template to mine, it would look like this:

Code:

if (current_timestamp <= prevblock.timestamp + 1200):
    newblock.timestamp = current_timestamp
else:
    newblock.timestamp = prevblock.timestamp + 1200

What you are asking is what happens if, after the journey to February, we are at 10 minutes block interval, a new miner joins, drives the difficulty up by, let's say 3x, and leaves? In that case, blocks take 30 minutes to mine, and difficulty is just reduced by 50% at next epoch, to 15 minutes per block. No big deal really, and I remind you that this has never happened before.

[stwenhao]

You can be against something and still accept the reality that it's happening.

Exactly. If testnet coins would be worthless, then some easier solutions could be deployed. But because they are not, then it simply wouldn't work, because people wouldn't stop mining thousands of tBTCs through min-difficulty blocks, just to make it more user-friendly for everyone else.

Otherwise, fjahr's version would reach more support, and the network difficulty would drop. But it didn't.

What would it take to do this? Call it testnet5, or testnetBHC for all I care. I assume it needs a hard-coded new genesis block and a hard-coded node to connect to. That's all, right? At least that way your chain could get started and someone could start mining blocks without a difficulty that's far too high for testing something.

It depends, what exactly do you want. If you want to just reorg all min-difficulty blocks, then you don't need a new Genesis Block for that. To reach it, you can just reject blocks, which would trigger 20-minute rule, and run the modified client. It will then halt at the first min-difficulty block, and you can use ASICs, to continue that chain.

But of course, reaching support for that version could be more difficult, than if you apply it from some future block number.

We can still make it a softfork, but we'll have to mess with timestamp.

Messing with the timestamp can be done here and now, you don't need any soft-fork for that, it is a no-fork. The only change you need is in the mining code, the validation code can be left as it is.

Which means, that you can change the code for mining, and then, if enough miners will run your version, then the difficulty would start dropping. And then, if all CPU blocks will be always reorged, applying the rule, that "there should be no blocks with minimal difficulty" can be done automatically, if you will have hashrate majority on your side.

Also, fjahr's version already does that "mess with timestamp" part, so you can start from that, and try to improve it. If more miners would use it, then it could fix the situation, but it just didn't reach enough support, because why miners should run a version, which will fix the situation, but which will give them less coins?

[BlackHatCoiner]

Messing with the timestamp can be done here and now, you don't need any soft-fork for that, it is a no-fork. The only change you need is in the mining code, the validation code can be left as it is.

You do need a softfork. Just applying my pseudocode in getblocktemplate is not enough, because not only do miners need to apply that when mining, but they also need to consider any min-diff block they receive as invalid, and this can only be accomplished if the node enforces the rule that a block with more than 20 minutes ahead in terms of timestamp is invalid.
 
If I do it here and now, without informing any miner to prepare their node for some future block, then we will have some miners mining on top of real-diff blocks and some others mining on top of min-diff blocks. This is ugly, and results in reorgs. With block signaling, fork-nodes will only activate the rule if at least 75% of the hashrate votes for it in epoch 74. This will be a smooth transition, for both fork-nodes and old nodes.

[BayAreaCoins]

Also, what changed this morning that caused this to go from a hard fork to a soft fork?

I mistakenly thought that you could mine a real-difficulty block with timestamp more than 20 minutes into the future. stwenhao corrected me, because any block after 20 minutes must have min-difficulty, it is not optional.

We can still make it a softfork, but we'll have to mess with timestamp. I still find it the most user-friendly thing to do. The softfork invalidates any block with timestamp more than 20 minutes into the future.

Assuming the softfork activates in September 1st, and majority of hashrate mines in the fork, the chain would look like in the following table. The epoch 75 would take ~2.5 months to complete, the next one ~1.2 months, and then the timestamp would quickly reach February. So, starting from September, it'd take about 6 months for the average block time to get back to 10 minutes, with timestamp date being up-to-date.

Era	Block	Real date	Timestamp date	Avg block time	Change
75	151,200	1st Sept	1st Sept	60 min	N/A
76	153,216	24th Nov	29th Sept	30 min	-50%
77	155,232	5th Jan	27th Oct	15 min	-50%
78	157,248	26th Jan	24th Nov	7.5 min	-50%
79	159,264	6th Feb	22nd Dec	3.75 min	-50%
80	161,280	11th Feb	19th Jan	1.875 min	-50%
81	163,296	14th Feb	14th Feb	~1 min	~-45%
82	165,312	16th Feb	16th Feb	~4 min	+400%
83	167,328	22nd Feb	22nd Feb	~10 min	+250%

(Difficulty keeps dropping by 50% until timestamp date reaches real date, then it abruptly goes back up.)

This is further away from Bitcoin though isn't it?  Blocks longer than 20 minutes frequently happen irl... probably a good thing in Testnet too.

Just go with the hard fork... I see little to no reason to be scared of it and it fixes what needs to be fixed.

The network will roll over just fine with support.

(I could be confused and likely am, hopefully.)


I also don't understand why slowing the block times down is even a conversation(?). Who cares where the chain theoretically should be, they are wrong in reality.... it is what it is, just "simplify" it down to Bitcoin like with no weird altcoin traits.  


A hard fork of Testnet4 would keep the same name Testnet4 on my website I do believe... for anyone asking.  

Saving the Testnet5 brand for a fresh launch.

I still find it the most user-friendly thing to do.

Honestly, I wouldn't worry about user-friendliness in Testnet... I know this sounds strange, but really, Testnet should kind of suck due to the nature of potential Tests and fuckery going on.  Having to jump through hoops and learn which fork is supported by which service, or blah blah blah, is an important part of the learning experience.

This is part of what 100% will hold the value down.  Although I feel like you people give my shitty little altcoin website far to much credit for any value around Testnet.

Testnet has always had value and part of the value we bring is bringing people together on top of that invisible value.  Nothing code wise has change.  Just the way certain humans interact with the code and how that interaction causes a wave for everyone else to ride.  Good ole Testnet.  Kinda fun.