Bitcoin Forum
December 12, 2018, 10:38:22 PM *
News: Latest Bitcoin Core release: 0.17.0 [Torrent].
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: why every blk00000.dat is different in every wallet?  (Read 430 times)
bigphoenixman
Newbie
*
Offline Offline

Activity: 8
Merit: 0


View Profile
November 20, 2017, 11:22:52 AM
 #1

I am a new one studying bitcoin source. I think that maybe writing a blockchain parser is first step. I realize that every blk00000.dat is different.  I got 2 different blk00000.dat.  I compared the files as below.
http://chuantu.biz/t6/152/1511169018x-1566660940.png
You can see the Magic Number, Size, Version, and the PreHash are different.
I had read https://github.com/bitcoin/bitcoin/issues/6613, but I am still confused.
If there is a key to XOR to make difference, why the blocks before do not XOR? And obviously, Magic Number, Size, Version do not XOR, which field will be XOR?

Thanks a lot!!!!
 

1544654302
Hero Member
*
Offline Offline

Posts: 1544654302

View Profile Personal Message (Offline)

Ignore
1544654302
Reply with quote  #2

1544654302
Report to moderator
1544654302
Hero Member
*
Offline Offline

Posts: 1544654302

View Profile Personal Message (Offline)

Ignore
1544654302
Reply with quote  #2

1544654302
Report to moderator
PLAY NOW
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1544654302
Hero Member
*
Offline Offline

Posts: 1544654302

View Profile Personal Message (Offline)

Ignore
1544654302
Reply with quote  #2

1544654302
Report to moderator
achow101
Moderator
Legendary
*
expert
Offline Offline

Activity: 1610
Merit: 1799


bc1qshxkrpe4arppq89fpzm6c0tpdvx5cfkve2c8kl


View Profile WWW
November 20, 2017, 03:15:05 PM
 #2

Bitcoin Core downloads and stores in blocks in the order that they are received, not the order that they are in the blockchain. Bitcoin Core downloads multiple blocks simultaneously, so it is likely that blocks will be received and written to disk in a different order from the actual blockchain and in a different order from node to node.

aestrum
Newbie
*
Offline Offline

Activity: 35
Merit: 0


View Profile
November 20, 2017, 06:37:27 PM
 #3

Content is XORed with random key, so that antivirus software does not trigger.
achow101
Moderator
Legendary
*
expert
Offline Offline

Activity: 1610
Merit: 1799


bc1qshxkrpe4arppq89fpzm6c0tpdvx5cfkve2c8kl


View Profile WWW
November 20, 2017, 06:42:17 PM
 #4

Content is XORed with random key, so that antivirus software does not trigger.
The block files themselves are not XORed with the random key; the databases are.

aestrum
Newbie
*
Offline Offline

Activity: 35
Merit: 0


View Profile
November 20, 2017, 06:49:30 PM
 #5

Yes, right, I confused it with chainstate db, which is XORed.
jnano
Member
**
Offline Offline

Activity: 241
Merit: 18


View Profile
November 20, 2017, 07:15:44 PM
 #6

Wouldn't it make sense to normalize the block storage format?
That would make it practical to do various operations between computers, like fill missing data or compare/verify.

Why XOR the chainstate data or (index?) database files?
achow101
Moderator
Legendary
*
expert
Offline Offline

Activity: 1610
Merit: 1799


bc1qshxkrpe4arppq89fpzm6c0tpdvx5cfkve2c8kl


View Profile WWW
November 20, 2017, 07:58:00 PM
 #7

Wouldn't it make sense to normalize the block storage format?
That would make it practical to do various operations between computers, like fill missing data or compare/verify.

The block storage and file formats are normalized. What you can't do is ensure that every single node has exactly the same data in exactly the same order. This is because during normal operation, orphan blocks will cause nodes to receive different blocks in different orders. Some nodes might see a block which another node does not see at all. Or they might see two different blocks and write them in the same position. Because of orphans, it is not possible to make all nodes have exactly the same data in exactly the same order.

During initial sync, having to write the blocks to disk in blockchain order rather than order received will result in a slower sync.

Why XOR the chainstate data or (index?) database files?
To prevent antivitus software from messing with the database files because sometimes they contain virus signatures because virus signatures are embedded into the blockchain. Having the database files corrupted will require the entire database to be rebuilt which will take a long time.

jnano
Member
**
Offline Offline

Activity: 241
Merit: 18


View Profile
November 20, 2017, 08:18:34 PM
 #8

Well, maybe it could have an optional flag to do store blocks in final blockchain order, or to trigger a one-time normalization pass on existing data.
And if orphan blocks are needed, keep them separate from the main chain.

I thought the AV-triggering data would be in the block files. Or what did you mean by databases?
Is the chainstate data XORed as well? I thought that's just the global UTXO balance, with no arbitrary data.
bigphoenixman
Newbie
*
Offline Offline

Activity: 8
Merit: 0


View Profile
November 21, 2017, 12:29:10 AM
 #9

Bitcoin Core downloads and stores in blocks in the order that they are received, not the order that they are in the blockchain. Bitcoin Core downloads multiple blocks simultaneously, so it is likely that blocks will be received and written to disk in a different order from the actual blockchain and in a different order from node to node.

Thank you Smiley
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!