Bitcoin Forum
September 25, 2018, 12:14:33 AM *
News: ♦♦ New info! Bitcoin Core users absolutely must upgrade to previously-announced 0.16.3 [Torrent]. All Bitcoin users should temporarily trust confirmations slightly less. More info.
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: why every blk00000.dat is different in every wallet?  (Read 429 times)
bigphoenixman
Newbie
*
Offline Offline

Activity: 8
Merit: 0


View Profile
November 20, 2017, 11:22:52 AM
 #1

I am a new one studying bitcoin source. I think that maybe writing a blockchain parser is first step. I realize that every blk00000.dat is different.  I got 2 different blk00000.dat.  I compared the files as below.
http://chuantu.biz/t6/152/1511169018x-1566660940.png
You can see the Magic Number, Size, Version, and the PreHash are different.
I had read https://github.com/bitcoin/bitcoin/issues/6613, but I am still confused.
If there is a key to XOR to make difference, why the blocks before do not XOR? And obviously, Magic Number, Size, Version do not XOR, which field will be XOR?

Thanks a lot!!!!
 

1537834473
Hero Member
*
Offline Offline

Posts: 1537834473

View Profile Personal Message (Offline)

Ignore
1537834473
Reply with quote  #2

1537834473
Report to moderator
1537834473
Hero Member
*
Offline Offline

Posts: 1537834473

View Profile Personal Message (Offline)

Ignore
1537834473
Reply with quote  #2

1537834473
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
achow101
Moderator
Legendary
*
expert
Offline Offline

Activity: 1526
Merit: 1656


3F1Y9yquzvY6RWvKbw2n2zeo9V5mvBhADU


View Profile WWW
November 20, 2017, 03:15:05 PM
 #2

Bitcoin Core downloads and stores in blocks in the order that they are received, not the order that they are in the blockchain. Bitcoin Core downloads multiple blocks simultaneously, so it is likely that blocks will be received and written to disk in a different order from the actual blockchain and in a different order from node to node.

aestrum
Newbie
*
Offline Offline

Activity: 35
Merit: 0


View Profile
November 20, 2017, 06:37:27 PM
 #3

Content is XORed with random key, so that antivirus software does not trigger.
achow101
Moderator
Legendary
*
expert
Offline Offline

Activity: 1526
Merit: 1656


3F1Y9yquzvY6RWvKbw2n2zeo9V5mvBhADU


View Profile WWW
November 20, 2017, 06:42:17 PM
 #4

Content is XORed with random key, so that antivirus software does not trigger.
The block files themselves are not XORed with the random key; the databases are.

aestrum
Newbie
*
Offline Offline

Activity: 35
Merit: 0


View Profile
November 20, 2017, 06:49:30 PM
 #5

Yes, right, I confused it with chainstate db, which is XORed.
jnano
Member
**
Offline Offline

Activity: 224
Merit: 13


View Profile
November 20, 2017, 07:15:44 PM
 #6

Wouldn't it make sense to normalize the block storage format?
That would make it practical to do various operations between computers, like fill missing data or compare/verify.

Why XOR the chainstate data or (index?) database files?
achow101
Moderator
Legendary
*
expert
Offline Offline

Activity: 1526
Merit: 1656


3F1Y9yquzvY6RWvKbw2n2zeo9V5mvBhADU


View Profile WWW
November 20, 2017, 07:58:00 PM
 #7

Wouldn't it make sense to normalize the block storage format?
That would make it practical to do various operations between computers, like fill missing data or compare/verify.

The block storage and file formats are normalized. What you can't do is ensure that every single node has exactly the same data in exactly the same order. This is because during normal operation, orphan blocks will cause nodes to receive different blocks in different orders. Some nodes might see a block which another node does not see at all. Or they might see two different blocks and write them in the same position. Because of orphans, it is not possible to make all nodes have exactly the same data in exactly the same order.

During initial sync, having to write the blocks to disk in blockchain order rather than order received will result in a slower sync.

Why XOR the chainstate data or (index?) database files?
To prevent antivitus software from messing with the database files because sometimes they contain virus signatures because virus signatures are embedded into the blockchain. Having the database files corrupted will require the entire database to be rebuilt which will take a long time.

jnano
Member
**
Offline Offline

Activity: 224
Merit: 13


View Profile
November 20, 2017, 08:18:34 PM
 #8

Well, maybe it could have an optional flag to do store blocks in final blockchain order, or to trigger a one-time normalization pass on existing data.
And if orphan blocks are needed, keep them separate from the main chain.

I thought the AV-triggering data would be in the block files. Or what did you mean by databases?
Is the chainstate data XORed as well? I thought that's just the global UTXO balance, with no arbitrary data.
bigphoenixman
Newbie
*
Offline Offline

Activity: 8
Merit: 0


View Profile
November 21, 2017, 12:29:10 AM
 #9

Bitcoin Core downloads and stores in blocks in the order that they are received, not the order that they are in the blockchain. Bitcoin Core downloads multiple blocks simultaneously, so it is likely that blocks will be received and written to disk in a different order from the actual blockchain and in a different order from node to node.

Thank you Smiley
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!