Testnet4 node constantly having its index corrupted

[BlackHatCoiner]

I just want to report this here, before writing on the mailing list.

I have experienced this numerous times already. The testnet4 node cannot start because between the time it shut down, and the time it started again, the chain had been reorged, and it gets Error: Cannot read last block filter index. The only way to restart is to reindex using bitcoind --reindex. I have not found another way around this.

Code:

2026-01-24T15:55:45Z Verification: No coin database inconsistencies in last 6 blocks (6 transactions)
2026-01-24T15:55:45Z Block index and chainstate loaded
2026-01-24T15:55:45Z Opening LevelDB in /home/user/.bitcoin/testnet4/indexes/txindex
2026-01-24T15:55:45Z Opened LevelDB successfully
2026-01-24T15:55:45Z Using obfuscation key for /home/user/.bitcoin/testnet4/indexes/txindex: 0000000000000000
2026-01-24T15:55:45Z Opening LevelDB in /home/user/.bitcoin/testnet4/indexes/blockfilter/basic/db
2026-01-24T15:55:45Z Opened LevelDB successfully
2026-01-24T15:55:45Z Using obfuscation key for /home/user/.bitcoin/testnet4/indexes/blockfilter/basic/db: 0000000000000000
2026-01-24T15:55:45Z [error] ReadFilterHeader: previous block header belongs to unexpected block 00000000b5f4bf2c9ff149e3c9b0c6d6226ee144167525c3fd62e67bc6ff62d8; expected 000000000bdee9c661ab9a8164334f4983f100afc6ad90c62d651845e62cf366
2026-01-24T15:55:45Z [error] Cannot read last block filter header; index may be corrupted
2026-01-24T15:55:45Z Shutdown: In progress...
2026-01-24T15:55:45Z scheduler thread exit
2026-01-24T15:55:45Z Flushed fee estimates to fee_estimates.dat.
2026-01-24T15:55:45Z Shutdown: done

I cannot be the only person testing bitcoin who's experienced this. This is another good argument in favor of removing the 20-min difficulty rule.

[LoyceV]

I have experienced this numerous times already.

Does this happen every time, or only once every few restarts? I just stopped and started mine (running Bitcoin Core 28.0) without problems. See tmp.loyce.club/testnet4_debug.log for my last 10,000 debug.log entries.

[BlackHatCoiner]

I have experienced this numerous times already.

Does this happen every time, or only once every few restarts? I just stopped and started mine (running Bitcoin Core 28.0) without problems. See tmp.loyce.club/testnet4_debug.log for my last 10,000 debug.log entries.

You need to start it after a long period of inactivity, so that CPU miner Bob has reorged another CPU miner Charlie (the chain of which you were following) and an ASIC miner mined on top of Bob instead of Charlie.

Try stopping it, and starting it tomorrow at the current time. It has to be definitely this, because the error clearly says that another block header was expected, which signifies the chain was reorged, and now my chain, and therefore my index is questioned.

[LoyceV]

Try stopping it, and starting it tomorrow at the current time.

I'll do this.

[nc50lc]

I just stopped and started mine (running Bitcoin Core 28.0) without problems. See tmp.loyce.club/testnet4_debug.log for my last 10,000 debug.log entries.

BHC might be using compact block filters with -blockfilterindex=<type> since there's a ".bitcoin/testnet4/indexes/blockfilter/basic/db" indicated in his log.
I haven't experience this either and like yours, I don't use compact block filters on my testnet4 node.

I've applied -blockfilterindex=1 to my testnet4 node, closed it and I'll report tomorrow if it'll result with the same error.

Code:

2026-01-25T05:52:40Z Command-line arg: blockfilterindex="1"

[LoyceV]

My testnet4 node started this morning without problems. See logs.

Did mempool.space get an upgrade? They now show stale blocks (and that's much more visible for testnet4 than anything else):

[BlackHatCoiner]

BHC might be using compact block filters with -blockfilterindex=<type> since there's a ".bitcoin/testnet4/indexes/blockfilter/basic/db" indicated in his log.

I do, but I feel like I've seen this error before introducing block filters.

My testnet4 node started this morning without problems. See logs.

This is normal. You just received the chain which got accepted and not reorged. Try doing this a couple of times.

Did mempool.space get an upgrade?

Yep. And it even misses some in testnet4. Fork-observer is the most reliable of all, and it misses some branches as well!  

[LoyceV]

Try doing this a couple of times.

I entered this on my console:

Code:

~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &
sleep 16h && ~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &
sleep 32h && ~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &
sleep 48h && ~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &
sleep 64h && ~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &
sleep 80h && ~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &
sleep 96h && ~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &
sleep 114h && ~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &
sleep 128h && ~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &
sleep 144h && ~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &
sleep 160h && ~/bitcoin-28.0/bin/bitcoin-cli -testnet4 stop && sleep 8h && ~/bitcoin-28.0/bin/bitcoind -testnet4 &

Feel free to keep an eye on my logs the coming week If I forget it, I hope it keeps running

Update a day later: weird, all my waiting "sleep" commands were gone. Let's try again, I don't know what killed it.

[retaur]

Some Blockchain software faces challenges removing stale blocks. I think it mostly got better with bitcoin core specifically but never fully improved so it may be worth renaming the block with the highest height before it starts a rescan/reindex (don't run it with that flag, it should do what it needs to do).

[nc50lc]

BHC might be using compact block filters

I do, but I feel like I've seen this error before introducing block filters.

I've added it anyways in case it's related to the issue since it's mentioned in the other error line.
Anything else in your bitcoin.conf file that may help with reproducing the errors?

BTW, mine opened normally as well after approximately 24hours.
Current tip:

Code:

2026-01-26T05:26:46Z UpdateTip: new best=000000000f37cf4f7dd342d58fdbad0e92e9062d5c76bbdbd2c40be60a5fa2c0 height=120246 version=0x3fff0000 log2_work=75.203553 tx=14143752 date='2026-01-26T07:25:45Z' progress=1.000000 cache=1.2MiB(7395txo)

[BlackHatCoiner]

Hey Loyce, since your server is still running without any error messages, could you add blockfilterindex=1 in your bitcoin.conf and restart it? It is weird how you've never got the same error as I did. I've had the same issue countless times on my virtual machines since then, but I always had blockfilterindex=1.

I think the problem is that when the node restarts, the chainstate gets reconciled with the new best chain, but the block filter index still holds a reference to a block header that's no longer on the active chain, but I am absolutely not sure.

Edit: you might need to restart with bitcoind --reindex

[satscraper]

on my virtual machines

That may be potential culprit.

VM may be suspended ^{commonly happens if you click X}  before full shutdown at the moment when bitcoind was in the middle of writing data to disk which may result in inconsistent index state.

Try run node on host machine.

[BlackHatCoiner]

VM may be suspended ^{commonly happens if you click X}  before full shutdown at the moment when bitcoind was in the middle of writing data to disk which may result in inconsistent index state.

I am shutting down the node with bitcoin-cli stop before shutting down the VM, so this can't be the problem.

[satscraper]

VM may be suspended ^{commonly happens if you click X}  before full shutdown at the moment when bitcoind was in the middle of writing data to disk which may result in inconsistent index state.

I am shutting down the node with bitcoin-cli stop before shutting down the VM, so this can't be the problem.

Good, but what about snapshots? They can cause the same issue regardless of whether they’re triggered manually or automatically, so if I were running node on VM, I’d avoid using them altogether. Anyway to identify the potential culprit I’d try running the node on the host machine first and then move it back to the VM to compare its behavior.

[LoyceV]

Hey Loyce, since your server is still running without any error messages, could you add blockfilterindex=1 in your bitcoin.conf and restart it?
I think the problem is that when the node restarts, the chainstate gets reconciled with the new best chain

Before I test this: have you had this problem again in recent days? Fork Observer shows the Crazy Fork Days on Testnet4 are over. So I expect I'd need many attempts before I manage to shut it down right before a Fork.

[BlackHatCoiner]

Before I test this: have you had this problem again in recent days? Fork Observer shows the Crazy Fork Days on Testnet4 are over. So I expect I'd need many attempts before I manage to shut it down right before a Fork.

I have not had this problem recently. Yes, you'll need many attempts until there is CPU mining competition and reorgs reappear.

[LoyceV]

I have not had this problem recently.

That makes it very difficult to reproduce. I'll test it if blocks start Forking again.

[stwenhao]

And it even misses some in testnet4.

Of course. Because blocks are propagated in the first-seen model: if you hear block A, and later you receive block B, at the same height, then it is accepted, but not propagated further.

Also note, that anyone can make a CPU stale block, at any point in time. And then, by default, the client will get only your block header of the stale block, and not even ask about its content. And of course, it won't pass it to the rest of the network.

Which means, that even if you will have perfect communication with a lot of nodes, you will still miss some stale blocks, just because the default rules are designed to not share them, unless necessary, or explicitly asked for.

By the way: you can start a few regtest nodes, and start producing stale blocks. Everything on 127.0.0.1, no communication problems. And then, you will see, how stale blocks are not reaching everyone, if you don't have N(N-1)/2 total connections for N nodes.

Edit:

The testnet4 node cannot start because between the time it shut down, and the time it started again, the chain had been reorged, and it gets Error: Cannot read last block filter index. The only way to restart is to reindex using bitcoind --reindex.

Try to reproduce that in regtest. If you succeed, then you can submit it as an issue into Bitcoin Core.

If not, then it may be related to using an unofficial version: if you run the official client, and your modified version, by using the same data directory, then changes from one version may be incompatible with the other, and it may lead to some crashes, or weird bugs.

[FanTuner]

This looks less like “your node is cursed” and more like a real block filter index bug. There’s already a Bitcoin Core issue from January 2026 describing block filter index corruption after a reorg/shutdown sequence, with the same Cannot read last block filter header; index may be corrupted error. So no, you’re definitely not the only one.
Issue: https://github.com/bitcoin/bitcoin/issues/34261          Testnet is supposed to test the software, not your patience, but sometimes it does both.