Bitcoin Forum
September 15, 2019, 03:15:42 PM *
News: If you like a topic and you see an orange "bump" link, click it. More info.
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: I can not parse files after blk00975.dat (Bitcoin Core)  (Read 182 times)
jaruvido
Newbie
*
Offline Offline

Activity: 12
Merit: 0


View Profile
June 12, 2018, 08:14:49 AM
 #1

Bitcoin Core download the blockchain to blkXXXXX.dat files.
also there is a parser of blkXXXXX.dat files, which created in compliance with the documentation https://bitcoin.org/en/developer-reference

At the beginning, all files are parsed without errors. Blocks and transactions are correct.
But since the file blk00975.dat in blocks there are some additional bytes, and incorrect transaction headers.

Unfortunately, I can not find any regular features. And the documentation does not say anything about any changes to the protocol.
I tried to completely reinstall Bitcoin Core, installed previous versions, and reloaded the *.dat files .
But all the same, since blk0097X.dat files are not valid on some blocks. And errors can appear in other places after reload.

Have you encountered such mistakes? Is there a way to fix them?
Or is there an alternative way, how else can I download a full blockchain?
1568560542
Hero Member
*
Offline Offline

Posts: 1568560542

View Profile Personal Message (Offline)

Ignore
1568560542
Reply with quote  #2

1568560542
Report to moderator
1568560542
Hero Member
*
Offline Offline

Posts: 1568560542

View Profile Personal Message (Offline)

Ignore
1568560542
Reply with quote  #2

1568560542
Report to moderator
1568560542
Hero Member
*
Offline Offline

Posts: 1568560542

View Profile Personal Message (Offline)

Ignore
1568560542
Reply with quote  #2

1568560542
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1568560542
Hero Member
*
Offline Offline

Posts: 1568560542

View Profile Personal Message (Offline)

Ignore
1568560542
Reply with quote  #2

1568560542
Report to moderator
1568560542
Hero Member
*
Offline Offline

Posts: 1568560542

View Profile Personal Message (Offline)

Ignore
1568560542
Reply with quote  #2

1568560542
Report to moderator
nc50lc
Hero Member
*****
Offline Offline

Activity: 714
Merit: 527


Self-proclaimed Genius ㊙️


View Profile WWW
June 12, 2018, 08:19:05 AM
 #2

The block cluster must be corrupted due to some reasons.
Try to delete blk0097X.dat from the default data directory then re-sync your node.
For Windows, open file explorer/my computer and browse to: C:\Users\"UserName"\Appdata\Roaming\Bitcoin

          ███████
      ██   ▀███▀   ██
  ▄█▄  ▄▄█████████▄▄  ▄█▄
 ▄███▄██▀▀       ▀▀██▄███▄
 ██▀███    ▄▄█▄█    ███▀██
    ██      █▀▀██    ██
██  ██     █████▀    ██  ██
    ██   ▄▄█▄▄▄██    ██
 ██▄███    █▀█▀▀    ███▄██
 ▀███▀██▄▄       ▄▄██▀███▀
  ▀██  ▀▀█████████▀▀  ██▀
      ██   ▄███▄   ██
          ███████
bspin   ███████████████████████
███████████████████████████
███████████████████████████
███████▀█▀    ▀▀▀██▀███████
███████            ▄███████
███████ ▄██▄▄▄▄▄▀ ▄████████
███ ▀  ▀  ██▀▀   ▄▀  ▀  ███
███▄▄█▀  █▀     █▄▄█▀  ████
█████   █▀       ██   █████
████▄▄▄▄█▄▄▄▄▄▄▄▄█▄▄▄▄█████
███████████████████████████
███████████████████████████
  ███████████████████████
       ▄▄█████████▄▄
    ▄█████████████████▄
  ▄█████████████████████▄
 ▄█████████  █  █████████▄
▄████████        █████████▄
███████████  ██  ██████████
███████████       █████████
███████████  ███  █████████
▀████████        ▄████████▀
 ▀█████████  █  █████████▀
  ▀█████████████████████▀
    ▀█████████████████▀
       ▀▀█████████▀▀
..PLAY NOW..
jaruvido
Newbie
*
Offline Offline

Activity: 12
Merit: 0


View Profile
June 12, 2018, 09:51:31 AM
 #3

Try to delete blk0097X.dat from the default data directory then re-sync your node.
For Windows, open file explorer/my computer and browse to: C:\Users\"UserName"\Appdata\Roaming\Bitcoin

Yes. so I did several times.
Errors in blocks or transactions (extra bytes or incorrect headers) can appear in different places in the downloaded files - from 97X to the last dat file.
nc50lc
Hero Member
*****
Offline Offline

Activity: 714
Merit: 527


Self-proclaimed Genius ㊙️


View Profile WWW
June 12, 2018, 11:30:24 AM
Last edit: June 12, 2018, 11:48:41 AM by nc50lc
Merited by achow101 (1)
 #4

I'm sorry, I mistook your post as a "stopped synchronizing" post.
I did a quick research and I think it has something to do with SegWit (August 24, 2017).
Also, I have checked the blocks included in blk00971.dat & I've noticed that the timestamps range from last week of August to Mid September 2017.
And blk00975.dat's blocks are from September 2017, one month after SegWit (BIP144) activation.

SegWit input/output scripts structure is a lot different than the regular transaction's.

That "parser" doesn't support and cannot read SegWit Txs.
Disclaimer: I could be wrong.

          ███████
      ██   ▀███▀   ██
  ▄█▄  ▄▄█████████▄▄  ▄█▄
 ▄███▄██▀▀       ▀▀██▄███▄
 ██▀███    ▄▄█▄█    ███▀██
    ██      █▀▀██    ██
██  ██     █████▀    ██  ██
    ██   ▄▄█▄▄▄██    ██
 ██▄███    █▀█▀▀    ███▄██
 ▀███▀██▄▄       ▄▄██▀███▀
  ▀██  ▀▀█████████▀▀  ██▀
      ██   ▄███▄   ██
          ███████
bspin   ███████████████████████
███████████████████████████
███████████████████████████
███████▀█▀    ▀▀▀██▀███████
███████            ▄███████
███████ ▄██▄▄▄▄▄▀ ▄████████
███ ▀  ▀  ██▀▀   ▄▀  ▀  ███
███▄▄█▀  █▀     █▄▄█▀  ████
█████   █▀       ██   █████
████▄▄▄▄█▄▄▄▄▄▄▄▄█▄▄▄▄█████
███████████████████████████
███████████████████████████
  ███████████████████████
       ▄▄█████████▄▄
    ▄█████████████████▄
  ▄█████████████████████▄
 ▄█████████  █  █████████▄
▄████████        █████████▄
███████████  ██  ██████████
███████████       █████████
███████████  ███  █████████
▀████████        ▄████████▀
 ▀█████████  █  █████████▀
  ▀█████████████████████▀
    ▀█████████████████▀
       ▀▀█████████▀▀
..PLAY NOW..
starmyc
Member
**
Offline Offline

Activity: 195
Merit: 88

Some random software engineer


View Profile
June 12, 2018, 12:26:36 PM
Merited by achow101 (1)
 #5

I'm sorry, I mistook your post as a "stopped synchronizing" post.
I did a quick research and I think it has something to do with SegWit (August 24, 2017).
Also, I have checked the blocks included in blk00971.dat & I've noticed that the timestamps range from last week of August to Mid September 2017.
And blk00975.dat's blocks are from September 2017, one month after SegWit (BIP144) activation.

SegWit input/output scripts structure is a lot different than the regular transaction's.

That "parser" doesn't support and cannot read SegWit Txs.
Disclaimer: I could be wrong.

Nope, you're alright. Most opensource parsers doesn't handle segwit and are not able to parse blocks included in recent blk files. I had to build my own parser to be able to parse files after these ones too. Smiley

Hi, I'm just some random software engineer.
You can check my projects: Bitcoin & altcoin balances dumps: https://balances.syndevio.com/
jackg
Copper Member
Legendary
*
Offline Offline

Activity: 1498
Merit: 1299


https://bit.ly/2FR9nyn - free python tutorials


View Profile
June 12, 2018, 07:40:26 PM
 #6

Nope, you're alright. Most opensource parsers doesn't handle segwit and are not able to parse blocks included in recent blk files. I had to build my own parser to be able to parse files after these ones too. Smiley

If I were you, I'd write a script to run everything from the bitcoin daemon or the bitcoin-qt cli (or whatever client you're running) and then you don't have to keep updating it.

Also, if you're going to run it in a separate script (away from bitcoin core) that is going to read the data, put the blockchain in a different folder.

coder0x15
Jr. Member
*
Offline Offline

Activity: 34
Merit: 2


View Profile
June 13, 2018, 10:29:50 AM
Merited by 2112 (1)
 #7

Bitcoin Core download the blockchain to blkXXXXX.dat files.
also there is a parser of blkXXXXX.dat files, which created in compliance with the documentation https://bitcoin.org/en/developer-reference

At the beginning, all files are parsed without errors. Blocks and transactions are correct.
But since the file blk00975.dat in blocks there are some additional bytes, and incorrect transaction headers.

Unfortunately, I can not find any regular features. And the documentation does not say anything about any changes to the protocol.
I tried to completely reinstall Bitcoin Core, installed previous versions, and reloaded the *.dat files .
But all the same, since blk0097X.dat files are not valid on some blocks. And errors can appear in other places after reload.

Have you encountered such mistakes? Is there a way to fix them?
Or is there an alternative way, how else can I download a full blockchain?

Try this one > https://github.com/normanvolt/blockchain-parser
jaruvido
Newbie
*
Offline Offline

Activity: 12
Merit: 0


View Profile
June 13, 2018, 10:30:33 AM
 #8

Also, I have checked the blocks included in blk00971.dat & I've noticed that the timestamps range from last week of August to Mid September 2017.
And blk00975.dat's blocks are from September 2017, one month after SegWit (BIP144) activation.

SegWit input/output scripts structure is a lot different than the regular transaction's.

That "parser" doesn't support and cannot read SegWit Txs.
Disclaimer: I could be wrong.



Thanks !
giftseller8x
Newbie
*
Offline Offline

Activity: 21
Merit: 1


View Profile
June 13, 2018, 01:27:53 PM
 #9

I fall into this issue before but still cannot solve it.
I change the way to parse block data.
Run bitcoin-qt or bitcoind and parse the raw block from rpc server. In the config file bitcoin.conf you must put this line:
rpcserialversion=0 => this will keep the raw block output in the same format with previous version.
jaruvido
Newbie
*
Offline Offline

Activity: 12
Merit: 0


View Profile
June 13, 2018, 01:56:37 PM
 #10

I fall into this issue before but still cannot solve it.
I change the way to parse block data.

I got this explanation from https://github.com/bitcoin/bitcoin/blob/master/src/primitives/transaction.h


/ **
  * Basic transaction serialization format:
  * - int32_t nVersion
  * - std :: vector <CTxIn> vin
  * - std :: vector <CTxOut> vout
  * - uint32_t nLockTime
  *
  * Extended transaction serialization format:
  * - int32_t nVersion
  * - unsigned char dummy = 0x00
  * - unsigned char flags (! = 0)
  * - std :: vector <CTxIn> vin
  * - std :: vector <CTxOut> vout
  * - if (flags & 1):
  * - CTxWitness wit;
  * - uint32_t nLockTime
  * /

and from here https://bitcoin.stackexchange.com/questions/55232/parsing-a-segwit-transaction-what-is-the-size-of-witness-data/55234

for each input {
     stackItems = read integer
     if stackItems> 0 {
         for each stackItem {
             bytes = read integer
             // witness data is bytes long
         }
     }
}
jaruvido
Newbie
*
Offline Offline

Activity: 12
Merit: 0


View Profile
June 13, 2018, 02:07:09 PM
Last edit: June 13, 2018, 03:05:53 PM by jaruvido
 #11

and now I have a new question)

----
for example in the blk000976.dat file there is a block

https://blockchain.info/block-height/481824
https://blockchain.info/block-index/1619616/0000000000000000001c8018d9cb3b742ef25114f27563e3fc4a1902167f9893

in which there is a transaction # 13  https://blockchain.info/tx/8f907925d2ebe48765103e6845c06f1f2bb77c6adc1cc002865865eb5cfd5c1c

with SegWit script

Can you describe the algorithm (pseudo code or the order of hash operations) how to get addresses for In- and Out transactions from its scripts?

thank you.
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!