Fixing Testnet4: proposal

[BlackHatCoiner]

For those who don't know, testnet4 is pretty much broken. CPU miners are taking advantage of the 20-min rule and there's very intense competition on who gets to broadcast the block faster. So intense, that some miners send empty blocks to gain that little advantage over bandwidth and verification time. A discussion started in the Bitcoin Development Mailing list, but it hasn't got anywhere: https://groups.google.com/g/bitcoindev/c/iVLHJ1HWhoU.

My proposal to fix testnet4 is to simply remove the 20-min limit after a certain epoch, months into the future, so that everyone has time to upgrade. Also, difficulty should also be decreased significantly after the epoch of that block (by around 5/6th) because around 5 in 6 blocks are CPU blocks, and by disabling CPU blocks, we should expect the block interval to increase significantly.

One good justification for this proposal is that the purpose of testnet is to mimic mainnet as closely as possible. It'd be good for devs to be able to mine testnet "for free", but this mindset has lead to these unintended consequences where you can't have the cake and eat it too. Having the illusion of "free mining" without the usability of a test network is far worse than having a usable test network where coins are just not free to mine.

What do you think?

[LoyceV]

My proposal to fix testnet4 is to simply remove the 20-min limit after a certain epoch

What if an ASIC miner largely increases difficulty and then quits? That's why the 20 minute rule was created in the first place, so mining can continue without a lot of mining power.
How about turning it into a 180 minutes rule?

[BlackHatCoiner]

What if an ASIC miner largely increases difficulty and then quits?

A miner cannot increase difficulty by more than 4x in a short time, because of consensus rule. If he spent a lot of hashrate for a short time and then quit, then the worst he could accomplish is make the block interval 40 minutes long on average, in the next epoch, considering that no other big miner wants to point extra hashrate (which is almost never the case). 40 minutes is still much better than currently.

How about turning it into a 180 minutes rule?

I don't see how it alleviates the problem. If a 180 min rule replaced the 20 min rule, then we should expect empty blocks every once in 180 minutes. We would just expect more ASIC blocks in each epoch, but it'd still be flawed in principle. You would still have loads of reorgs. Why not getting rid of all them in the first place?

[LoyceV]

A miner cannot increase difficulty by more than 4x in a short time, because of consensus rule.

Sure he can: every 2016 blocks will make the difficulty go up 4x. If he mines 10,000 blocks in a short time, the difficulty goes up 1000x.

40 minutes is still much better than currently.

Agreed.

If a 180 min rule replaced the 20 min rule, then we should expect empty blocks every once in 180 minutes.

There's also the 2 hour rule, so the 180 minute rule can only be (ab)used if there's no ASIC block for an hour, or after 3 hours without a block.

[stwenhao]

Disabling the 20-min rule could also be done as a softfork

Of course. But then, you would need to fake a block header. Because after 20 minutes, it would say: "this block has 0x1d00ffff difficulty". And your soft-forked version would need to reject it, and enforce the real difficulty instead.

Or: any blocks with timestamps bigger than 20 minutes, than the previous block, would need to be invalidated forever.

You wouldn't need a new chain nor a new network.

It would cause a fork, unless the hashrate majority would support it.

Also, difficulty should also be decreased significantly after the epoch of that block

There is no need to change anything. If you would have one ASIC block per hour, instead of per 10 minutes, then it would naturally decrease the difficulty over time.

Also, there were more blocks mined, than there should be, so a little delay wouldn't hurt.

It'd be good for devs to be able to mine testnet "for free", but this mindset has lead to these unintended consequences where you can't have the cake and eat it too.

Signet is mineable, see addresses like tb1pruektj90gg8nysa7yuk07w7ucwlywrf4p02lq3sz49f05xd00djscyt2fw, tb1pffsyra2t3nut94yvdae9evz3feg7tel843pfcv76vt5cwewavtesl3gsph, or tb1pf2v25yk7m8mv203pvjusmk2a8r6tu8p59nhvwux86ck3s3pp0nkqt30dvt. And it is possible to adjust mined amounts, so that for example CPU miners could get million times less coins, for mining million times weaker blocks.

Because in general, it is not a problem, that CPU miners can get a block. It is a problem, that they are getting 50 tBTC, and not for example 5k testnet satoshis.

What do you think?

I think I will keep mining signet, because even if it is centralized, then at least it is not traded that easily. And if it will start being traded, then a single ASIC could easily destroy the whole market, when needed. Which is also why testnet3 or testnet4 is actively traded, while in signet, it is harder, because developers can fully control mined blocks (and could easily blacklist for example some address, used by some exchange, while processing other transactions normally).

Also, the competition in mining signet is growing. Previously, I could mine 100 sBTC easily. Now, it takes me around 4 days. Which means, that other people will probably also join as well, and maybe even some signet pools will appear. Who knows, now getting 2 sBTC, 1 sBTC, or 0.125 sBTC is possible by solo mining challenges from the faucet. Maybe later it will be too hard, if there will be more miners around.

[BlackHatCoiner]

Sure he can: every 2016 blocks will make the difficulty go up 4x. If he mines 10,000 blocks in a short time, the difficulty goes up 1000x.

I don't understand how he can 1000x the difficulty in a short time. Just for the record, right now difficulty is 1.3 billion. Multiply that by 1/6 (the "real" difficulty) and you have 216 million. This is 1.55 PH/s. In order to 4x that you need to throw 4,65 PH/s on average for the entire epoch. That's an insane amount of money for this negligible damage.

And by the way, you can make the current testnet4 unusable by 51% attacking it with empty blocks for a tiny fraction of the same cost. You can even make it barely unusable for free, as we've seen!

I think I will keep mining signet, because even if it is centralized, then at least it is not traded that easily.

I personally don't care what others do with their testnet coins, as long as I have the freedom to test the network.

[NotATether]

I personally think that there needs to be a global hashrate limit for the testnet4 network or whatever future network they create.

Because as it has been said, devs and basically other people should be able to mine coins for almost-free. (Perfectly free mining is never possible in practice because of household electricity costs).

It should be adjusted to be low enough that CPU cores can mine at full speed, but GPUs/ASICs/FPGAs are throttled like in Monero. Because most developers are not using these devices to mine coins for testing purposes.

There should not be a need for racks and racks of testnet miners in order to find a block.

This would also make the 20-minute rule unnecessary and thus make sudden difficulty spikes by a single miner impossible, unless they control most of the CPUs. It would instead be "reject blocks mined with a time stamp <20 minutes ago, and make the network difficulty a constant. Then change the algorithm slightly so as to break existing hardware devices (maybe SHA256d(SHA256d(x))?)"

Unlike in mainnet, there is no monetary value that needs to be protected with extra hashrate.

[LoyceV]

I personally think that there needs to be a global hashrate limit for the testnet4 network or whatever future network they create.

That would push testnet even further from mainnet.

[NotATether]

That would push testnet even further from mainnet.

I know, but I can't think of any better solutions at the moment.

It would need to resemble mainnet's 10-minute average block generation at minimum (which current testnet fails to do) while also ensuring that some guy with a specialized hardware can't come in and disrupt the mining times for everyone else.

[LoyceV]

I know, but I can't think of any better solutions at the moment.

Me neither. It's crazy how testnet difficulty is over a billion. In an ideal world, there would be a few low-hashrate miners to keep it going, anyone could mine temporarily to get some coins, and there would be no ASICs running continuously on testnet, but reality is the way it is.

[BlackHatCoiner]

I personally think that there needs to be a global hashrate limit for the testnet4 network or whatever future network they create.

That would push testnet even further from mainnet.

This is why my proposal is so simple. If you read the mailing list, you'll see many complicated proposals that have no consensus. We will probably never agree on a single one, so let's just accept the fate of testnet, which is that mining is difficult and non-free, and at least let the game theory determine the usability of the network. It's also the only proposal that respects the "mainnet clone" principle.

I know, but I can't think of any better solutions at the moment.

Me neither. It's crazy how testnet difficulty is over a billion. In an ideal world, there would be a few low-hashrate miners to keep it going, anyone could mine temporarily to get some coins, and there would be no ASICs running continuously on testnet, but reality is the way it is.

The real difficulty, as previously mentioned, is 216 million. That's 1.5 PH/s. It seems a big number, but that's the total hashrate. From bitaxes and small rigs trying to test bitcoin, to even MARA pool with hundreds of terahashes. It still blows my mind, but on the other hand, how much is testing the bitcoin network worth?

[LoyceV]

That's 1.5 PH/s. ~ how much is testing the bitcoin network worth?

According to Bitcoin Mining Calculator, that hashrate is worth $55 per day. I know the question was rhetorical, but it's nice to add a price to it.

[BayAreaCoins]

I agree, keep it simple and drop the 20-minute rule as planned on the mailing list.  Get as close to Mainnet as possible with a heavy testing and sharing "vibe" behind it.

I'm a little surprised to see no action taken after the mailing list discussion with dates and plenty of time.  I was discussing this with a fellow Testnet ecosystem business owner the other day.

I do feel like most of us are getting back on the same page.

It still blows my mind

Likewise.

how much is testing the bitcoin network worth?

A huge chunk of it is major corporations as well...

You should hear the stories I hear about companies and how they use their Testnet with applicants... lol, trust me, they want those coins back. haha

I'd say it's probably worth something.

Bitcoin has a ~$1,700,000,000 market cap.
Bitcoin Testnet 3 has a $1,759,590 market cap. (using 21 million*last price@Altquick)
Bitcoin Testnet 4 has a $142,991.48 market cap. (using 4586000 mined coins thus far*last price@Altquick)

[NotATether]

how much is testing the bitcoin network worth?

A huge chunk of it is major corporations as well...

You should hear the stories I hear about companies and how they use their Testnet with applicants... lol, trust me, they want those coins back. haha

I'd say it's probably worth something.

Bitcoin has a ~$1,700,000,000 market cap.
Bitcoin Testnet 3 has a $1,759,590 market cap. (using 21 million*last price@Altquick)
Bitcoin Testnet 4 has a $142,991.48 market cap. (using 4586000 mined coins thus far*last price@Altquick)

Still, that basically sounds like "corporate testnet", where coins have some degree of value because corporations require it (I guess from sites like altquick).  It's not good for the indie dev though. I wouldn't be too happy if I had to pay $20-$30 for some coins to test some open-source project involving bitcoin.

[BayAreaCoins]

how much is testing the bitcoin network worth?

A huge chunk of it is major corporations as well...

You should hear the stories I hear about companies and how they use their Testnet with applicants... lol, trust me, they want those coins back. haha

I'd say it's probably worth something.

Bitcoin has a ~$1,700,000,000 market cap.
Bitcoin Testnet 3 has a $1,759,590 market cap. (using 21 million*last price@Altquick)
Bitcoin Testnet 4 has a $142,991.48 market cap. (using 4586000 mined coins thus far*last price@Altquick)

Still, that basically sounds like "corporate testnet", where coins have some degree of value because corporations require it (I guess from sites like altquick).  It's not good for the indie dev though. I wouldn't be too happy if I had to pay $20-$30 for some coins to test some open-source project involving bitcoin.

There is no corporate testnet or dirty hippie dev testnet... It's all just Testnet.  We are talking *huge* companies with plenty of talent & computing power; they don't need to use Altquick.  Dirty hippies mostly use AltQuick.  We are the indie/alt crowd!!!!!

I got bad news for you $20-$30 isn't going to buy the miner you need, so perhaps putting $30 into testing your project directly and then having the option to pull out $30 out (or more or less) when you're done is actually more economical anyways + you don't have to bother with shipping or equipment BS.

Free coins still exist for people who can work with 0.01 TBTC in v3 and v4, but the reality is that people frequently want whole coins (Cintrea required users to have 10 whole v4 coins to test their stuff).  No faucet that I'm aware of has ever been able to accommodate whole coins.  Who are we to deny them that?  They buy 100 Testnet, dink around with them, and then sell them to the next person, who is doing whatever with them.

Right now $30 buys ~1000 v4 Testnet coins, which is 20 blocks.  Foundry USA (the largest Bitcoin pool in the world atm) has mined 17 blocks v4 Testnet blocks in the past 7 days... so if you got equipment collecting dust and a week to burn for $30 and you mine as well as Foundry, go for it.  (Or throw $30 in for 1000 coins in 10 minutes, test for a week and pray you get $50 when you return them... Degen till the end fellas!)


If you create a faucet with those 1000 coins and give 0.001 TBTC per hit, your website could get 1,000,000 hits, and if you can figure something out with that... you might need 1000 more coins.

There is a long list of fun shit that can be done, and plenty I haven't seen or thought of yet.


The propaganda about the 20-minute rule and ASICS is silly Testnet folklore... Testnet v3 came out in 2012 and ASICS came out in 2013. Testnet has had "funky" mining like ASIC BOOST before... which is now mainstream.  It's the Area 51 of Bitcoin; it's supposed to be weird. Don't put a saddle on a mustang.

ASIC's aren't clicking off... don't worry, if anything, when or if this stupid rule is patched... Testnet will see a heavy flow of ASIC in since the mining is "fair".  Which will help people test their new equipment, mining and blah blah blah.  There is a nice little handful of pools nowadays for people just to rent or point their miners at, much simpler for people than any other time in Testnet history.

Solopool.com just hit their first Bitcoin block today... they are one of the now handful of mining pools supporting Testnet.



I hit a Testnet4 block or two with them today https://mempool.space/testnet4/tx/34990bdf09aacdfb661bd4cf814f82c4b944bf296aaa56ea5201fe6699e19ebb

[LoyceV]

Sure he can: every 2016 blocks will make the difficulty go up 4x. If he mines 10,000 blocks in a short time, the difficulty goes up 1000x.

I don't understand how he can 1000x the difficulty in a short time. Just for the record, right now difficulty is 1.3 billion. Multiply that by 1/6 (the "real" difficulty) and you have 216 million. This is 1.55 PH/s. In order to 4x that you need to throw 4,65 PH/s on average for the entire epoch. That's an insane amount of money for this negligible damage.

Let's inverse the scenario: instead of pushing difficulty up even higher, the few ASIC miners on testnet could just quit. That leaves it at a very high difficulty, and testnet will effectively come to a halt until ASIC miners continue. A simple small developer couldn't keep it running on his own at this difficulty. As far as I know, that's why the 20 minute rule was introduced in the first place, so I'm curious to see how long it will last without it.

[BlackHatCoiner]

Let's inverse the scenario: instead of pushing difficulty up even higher, the few ASIC miners on testnet could just quit. That leaves it at a very high difficulty, and testnet will effectively come to a halt until ASIC miners continue. A simple small developer couldn't keep it running on his own at this difficulty. As far as I know, that's why the 20 minute rule was introduced in the first place, so I'm curious to see how long it will last without it.

My python script found 27609 ASIC blocks mined in the last 100000 blocks.

Code:

-----------------------------
Total blocks scanned: 100000
Hashes with >13 leading zeroes: 27609
-----------------------------

================ ASIC frequency per epoch ================
Epoch 00 | heights 14885–16900 | ASIC    5/2016 = 0.2480% | CPU 2011
Epoch 01 | heights 16901–18916 | ASIC   15/2016 = 0.7440% | CPU 2001
Epoch 02 | heights 18917–20932 | ASIC   73/2016 = 3.6210% | CPU 1943
Epoch 03 | heights 20933–22948 | ASIC  283/2016 = 14.0377% | CPU 1733
Epoch 04 | heights 22949–24964 | ASIC 1048/2016 = 51.9841% | CPU  968
Epoch 05 | heights 24965–26980 | ASIC 1890/2016 = 93.7500% | CPU  126
Epoch 06 | heights 26981–28996 | ASIC 1335/2016 = 66.2202% | CPU  681
Epoch 07 | heights 28997–31012 | ASIC 1407/2016 = 69.7917% | CPU  609
Epoch 08 | heights 31013–33028 | ASIC 1436/2016 = 71.2302% | CPU  580
Epoch 09 | heights 33029–35044 | ASIC 1420/2016 = 70.4365% | CPU  596
Epoch 10 | heights 35045–37060 | ASIC 1303/2016 = 64.6329% | CPU  713
Epoch 11 | heights 37061–39076 | ASIC 1318/2016 = 65.3770% | CPU  698
Epoch 12 | heights 39077–41092 | ASIC 1212/2016 = 60.1190% | CPU  804
Epoch 13 | heights 41093–43108 | ASIC  864/2016 = 42.8571% | CPU 1152
Epoch 14 | heights 43109–45124 | ASIC  640/2016 = 31.7460% | CPU 1376
Epoch 15 | heights 45125–47140 | ASIC  420/2016 = 20.8333% | CPU 1596
Epoch 16 | heights 47141–49156 | ASIC  304/2016 = 15.0794% | CPU 1712
Epoch 17 | heights 49157–51172 | ASIC  340/2016 = 16.8651% | CPU 1676
Epoch 18 | heights 51173–53188 | ASIC  304/2016 = 15.0794% | CPU 1712
Epoch 19 | heights 53189–55204 | ASIC   72/2016 = 3.5714% | CPU 1944
Epoch 20 | heights 55205–57220 | ASIC  213/2016 = 10.5655% | CPU 1803
Epoch 21 | heights 57221–59236 | ASIC  269/2016 = 13.3433% | CPU 1747

# metrics start having meaning after block 60k
Epoch 22 | heights 59237–61252 | ASIC  416/2016 = 20.6349% | CPU 1600
Epoch 23 | heights 61253–63268 | ASIC  438/2016 = 21.7262% | CPU 1578
Epoch 24 | heights 63269–65284 | ASIC  370/2016 = 18.3532% | CPU 1646
Epoch 25 | heights 65285–67300 | ASIC  306/2016 = 15.1786% | CPU 1710
Epoch 26 | heights 67301–69316 | ASIC  341/2016 = 16.9147% | CPU 1675
Epoch 27 | heights 69317–71332 | ASIC  222/2016 = 11.0119% | CPU 1794
Epoch 28 | heights 71333–73348 | ASIC  200/2016 = 9.9206% | CPU 1816
Epoch 29 | heights 73349–75364 | ASIC  138/2016 = 6.8452% | CPU 1878
Epoch 30 | heights 75365–77380 | ASIC  237/2016 = 11.7560% | CPU 1779
Epoch 31 | heights 77381–79396 | ASIC  295/2016 = 14.6329% | CPU 1721
Epoch 32 | heights 79397–81412 | ASIC  296/2016 = 14.6825% | CPU 1720
Epoch 33 | heights 81413–83428 | ASIC  213/2016 = 10.5655% | CPU 1803
Epoch 34 | heights 83429–85444 | ASIC  214/2016 = 10.6151% | CPU 1802
Epoch 35 | heights 85445–87460 | ASIC  242/2016 = 12.0040% | CPU 1774
Epoch 36 | heights 87461–89476 | ASIC  264/2016 = 13.0952% | CPU 1752
Epoch 37 | heights 89477–91492 | ASIC  269/2016 = 13.3433% | CPU 1747
Epoch 38 | heights 91493–93508 | ASIC  311/2016 = 15.4266% | CPU 1705
Epoch 39 | heights 93509–95524 | ASIC  326/2016 = 16.1706% | CPU 1690
Epoch 40 | heights 95525–97540 | ASIC  309/2016 = 15.3274% | CPU 1707
Epoch 41 | heights 97541–99556 | ASIC  275/2016 = 13.6409% | CPU 1741
Epoch 42 | heights 99557–101572 | ASIC  238/2016 = 11.8056% | CPU 1778
Epoch 43 | heights 101573–103588 | ASIC  208/2016 = 10.3175% | CPU 1808
Epoch 44 | heights 103589–105604 | ASIC  211/2016 = 10.4663% | CPU 1805
Epoch 45 | heights 105605–107620 | ASIC  214/2016 = 10.6151% | CPU 1802
Epoch 46 | heights 107621–109636 | ASIC  128/2016 = 6.3492% | CPU 1888
Epoch 47 | heights 109637–111652 | ASIC  158/2016 = 7.8373% | CPU 1858
Epoch 48 | heights 111653–113668 | ASIC  148/2016 = 7.3413% | CPU 1868
Epoch 49 | heights 113669–114884 | ASIC  157/1216 = 12.9112% | CPU 1059

Your concern is reasonable, but reality shows that there's just always some miner with terahashes spent on testing bitcoin mining. Sure, there are periods with less than 10% of blocks being ASIC, just as times where it's higher than that, but I don't find it particularly concerning if some epochs just have longer time intervals.

(Also, consider that CPU mining wasn't being taken advantage like this until block 50 thousand something. Before that time, ASIC miners like Portland were mining very frequently at low difficulties using real hashrate. So the metrics start having meaning at around block ~60k.)

Edit: Yeah, I forgot that at low difficulties, block hashes have lower leading zeroes than 13. So simply ignore the early days of testnet4 and focus on after block 60k.

[stwenhao]

As far as I know, that's why the 20 minute rule was introduced in the first place, so I'm curious to see how long it will last without it.

Testnet4 will halt after 2016 or less blocks, if no ASIC will be there. On the other hand, testnet3 can keep running without any ASICs forever.

Which means, that the only reason, why the network is still running, is that some ASIC miners are keeping testnet4 alive. If all of them would raise the difficulty to insane levels, and then leave, then testnet4 would be stuck, because CPU miners alone wouldn't pass the nearest difficulty adjustment for quite long time.

Also, as long as ASIC miners are confirming chains of blocks with minimal difficulty, then they are basically setting the attack in stone. If each ASIC miner would reorg all CPU-mined blocks, from the top of the chain, then the difficulty would adjust downwards naturally.

My python script found 27609 ASIC blocks mined in the last 100000 blocks.

Yes, it means 27% for ASICs, and 73% for CPUs. So, maybe it is more like 1/4 for ASICs, and 3/4 for CPUs, instead of theoretical 1/6 for ASICs, and 5/6 for CPUs, but still: there are a lot of blocks with CPU difficulty.

But well, since the competition in testnet4 turned everything into "Speedy Gonzales chain", where the winner is not someone with the heaviest chain, but someone with the fastest block propagation, I switched to signet for a while. Maybe I will come back, if testnet will be fixed. But with signet challenge set to OP_TRUE, it is basically the same, as fixed testnet would be.

So simply ignore the early days of testnet4 and focus on after block 60k.

I would rather say "30k", instead of "60k". Somewhere between 30k and 60k, many developers got thousands of testnet4 coins, just by using CPU power, and not much more than that. Also, there were times, where you had to build things from the source code, to even join the network, so there was almost no competition. If I remember correctly, Garlo Nicon mined on his CPU something around 100k tBTC4 at that time, maybe others reached even more.

[BayAreaCoins]

Could someone just ask on the mailing list on the "Unbreaking testnet" email if the proposed date is still going to happen or if it is being pushed back(?)

[BlackHatCoiner]

Which means, that the only reason, why the network is still running, is that some ASIC miners are keeping testnet4 alive.

That's actually a good argument in favor of a proof-of-work-only testnet. What LoyceV described could have already happened if the miner disappeared right before the difficulty adjustment.

Yes, it means 27% for ASICs, and 73% for CPUs. So, maybe it is more like 1/4 for ASICs, and 3/4 for CPUs, instead of theoretical 1/6 for ASICs, and 5/6 for CPUs, but still: there are a lot of blocks with CPU difficulty.

In the early days there were more ASIC blocks, my script simply ignored them because their difficulty was so low that the block hashes begin with less than 13 zeroes. No, I think the 1/6 rule normally applies. The average ASIC/CPU block ratio since epoch 35 in my metrics (across many epochs) is 11.8% ASIC / 88.2% CPU blocks. We should expect 100/6 = 16.6% ASIC blocks, but consider that this is only true under a steady-difficulty environment where hashrate remains constant.

This is the real difficulty graph since block 82626:



I can't think of the math right now, but I suspect that an increase in ASIC block frequency beyond 16.6% is much harder to achieve than a relative decrease in frequency toward 0%. That explains why, even with average hashrate increase, the ASIC block frequency is less than 16.6%.

Edit:

Could someone just ask on the mailing list on the "Unbreaking testnet" email if the proposed date is still going to happen or if it is being pushed back(?)

Yes, I may send the message.