I can not parse files after blk00975.dat (Bitcoin Core)

[jaruvido]

Bitcoin Core download the blockchain to blkXXXXX.dat files.
also there is a parser of blkXXXXX.dat files, which created in compliance with the documentation https://bitcoin.org/en/developer-reference

At the beginning, all files are parsed without errors. Blocks and transactions are correct.
But since the file blk00975.dat in blocks there are some additional bytes, and incorrect transaction headers.

Unfortunately, I can not find any regular features. And the documentation does not say anything about any changes to the protocol.
I tried to completely reinstall Bitcoin Core, installed previous versions, and reloaded the *.dat files .
But all the same, since blk0097X.dat files are not valid on some blocks. And errors can appear in other places after reload.

Have you encountered such mistakes? Is there a way to fix them?
Or is there an alternative way, how else can I download a full blockchain?

[1715372758]

1715372758

[1715372758]

1715372758

[nc50lc]

The block cluster must be corrupted due to some reasons.
Try to delete blk0097X.dat from the default data directory then re-sync your node.
For Windows, open file explorer/my computer and browse to: C:\Users\"UserName"\Appdata\Roaming\Bitcoin

[jaruvido]

Try to delete blk0097X.dat from the default data directory then re-sync your node.
For Windows, open file explorer/my computer and browse to: C:\Users\"UserName"\Appdata\Roaming\Bitcoin

Yes. so I did several times.
Errors in blocks or transactions (extra bytes or incorrect headers) can appear in different places in the downloaded files - from 97X to the last dat file.

[nc50lc]

I'm sorry, I mistook your post as a "stopped synchronizing" post.I did a quick research and I think it has something to do with SegWit (August 24, 2017).
Also, I have checked the blocks included in blk00971.dat & I've noticed that the timestamps range from last week of August to Mid September 2017.
And blk00975.dat's blocks are from September 2017, one month after SegWit (BIP144) activation.

SegWit input/output scripts structure is a lot different than the regular transaction's.

That "parser" doesn't support and cannot read SegWit Txs.
Disclaimer: I could be wrong.

[starmyc]

I'm sorry, I mistook your post as a "stopped synchronizing" post.I did a quick research and I think it has something to do with SegWit (August 24, 2017).
Also, I have checked the blocks included in blk00971.dat & I've noticed that the timestamps range from last week of August to Mid September 2017.
And blk00975.dat's blocks are from September 2017, one month after SegWit (BIP144) activation.

SegWit input/output scripts structure is a lot different than the regular transaction's.

That "parser" doesn't support and cannot read SegWit Txs.
Disclaimer: I could be wrong.

Nope, you're alright. Most opensource parsers doesn't handle segwit and are not able to parse blocks included in recent blk files. I had to build my own parser to be able to parse files after these ones too.

[jackg]

Nope, you're alright. Most opensource parsers doesn't handle segwit and are not able to parse blocks included in recent blk files. I had to build my own parser to be able to parse files after these ones too.

If I were you, I'd write a script to run everything from the bitcoin daemon or the bitcoin-qt cli (or whatever client you're running) and then you don't have to keep updating it.

Also, if you're going to run it in a separate script (away from bitcoin core) that is going to read the data, put the blockchain in a different folder.

[coder0x15]

Bitcoin Core download the blockchain to blkXXXXX.dat files.
also there is a parser of blkXXXXX.dat files, which created in compliance with the documentation https://bitcoin.org/en/developer-reference

At the beginning, all files are parsed without errors. Blocks and transactions are correct.
But since the file blk00975.dat in blocks there are some additional bytes, and incorrect transaction headers.

Unfortunately, I can not find any regular features. And the documentation does not say anything about any changes to the protocol.
I tried to completely reinstall Bitcoin Core, installed previous versions, and reloaded the *.dat files .
But all the same, since blk0097X.dat files are not valid on some blocks. And errors can appear in other places after reload.

Have you encountered such mistakes? Is there a way to fix them?
Or is there an alternative way, how else can I download a full blockchain?

Try this one > https://github.com/normanvolt/blockchain-parser

[jaruvido]

Also, I have checked the blocks included in blk00971.dat & I've noticed that the timestamps range from last week of August to Mid September 2017.
And blk00975.dat's blocks are from September 2017, one month after SegWit (BIP144) activation.

SegWit input/output scripts structure is a lot different than the regular transaction's.

That "parser" doesn't support and cannot read SegWit Txs.
Disclaimer: I could be wrong.

Thanks !

[giftseller8x]

I fall into this issue before but still cannot solve it.
I change the way to parse block data.
Run bitcoin-qt or bitcoind and parse the raw block from rpc server. In the config file bitcoin.conf you must put this line:
rpcserialversion=0 => this will keep the raw block output in the same format with previous version.

[jaruvido]

I fall into this issue before but still cannot solve it.
I change the way to parse block data.

I got this explanation from https://github.com/bitcoin/bitcoin/blob/master/src/primitives/transaction.h


/ **
  * Basic transaction serialization format:
  * - int32_t nVersion
  * - std :: vector <CTxIn> vin
  * - std :: vector <CTxOut> vout
  * - uint32_t nLockTime
  *
  * Extended transaction serialization format:
  * - int32_t nVersion
  * - unsigned char dummy = 0x00
  * - unsigned char flags (! = 0)
  * - std :: vector <CTxIn> vin
  * - std :: vector <CTxOut> vout
  * - if (flags & 1):
  * - CTxWitness wit;
  * - uint32_t nLockTime
  * /

and from here https://bitcoin.stackexchange.com/questions/55232/parsing-a-segwit-transaction-what-is-the-size-of-witness-data/55234

for each input {
     stackItems = read integer
     if stackItems> 0 {
         for each stackItem {
             bytes = read integer
             // witness data is bytes long
         }
     }
}

[jaruvido]

and now I have a new question)

----
for example in the blk000976.dat file there is a block

https://blockchain.info/block-height/481824
https://blockchain.info/block-index/1619616/0000000000000000001c8018d9cb3b742ef25114f27563e3fc4a1902167f9893

in which there is a transaction # 13  https://blockchain.info/tx/8f907925d2ebe48765103e6845c06f1f2bb77c6adc1cc002865865eb5cfd5c1c

with SegWit script

Can you describe the algorithm (pseudo code or the order of hash operations) how to get addresses for In- and Out transactions from its scripts?

thank you.