Bitcoin Forum
July 23, 2018, 06:16:56 PM *
News: Latest stable version of Bitcoin Core: 0.16.1  [Torrent]. (New!)
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: natural order of blocks in blk-files?  (Read 923 times)
amaclin
Legendary
*
Offline Offline

Activity: 1260
Merit: 1000


View Profile
October 29, 2016, 11:29:15 AM
 #1

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

1532369816
Hero Member
*
Offline Offline

Posts: 1532369816

View Profile Personal Message (Offline)

Ignore
1532369816
Reply with quote  #2

1532369816
Report to moderator
1532369816
Hero Member
*
Offline Offline

Posts: 1532369816

View Profile Personal Message (Offline)

Ignore
1532369816
Reply with quote  #2

1532369816
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1532369816
Hero Member
*
Offline Offline

Posts: 1532369816

View Profile Personal Message (Offline)

Ignore
1532369816
Reply with quote  #2

1532369816
Report to moderator
1532369816
Hero Member
*
Offline Offline

Posts: 1532369816

View Profile Personal Message (Offline)

Ignore
1532369816
Reply with quote  #2

1532369816
Report to moderator
2112
Legendary
*
Offline Offline

Activity: 2114
Merit: 1027



View Profile
October 29, 2016, 05:19:13 PM
 #2

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?
Did you try using "-connect" to force talking to just one peer?

I haven't tried 0.13.1, but with some older versions that resulted in "reproducible ordering", I don't recall if it was "natural order".

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
gmaxwell
Moderator
Legendary
*
qt
Offline Offline

Activity: 2450
Merit: 1353



View Profile
October 31, 2016, 12:16:31 AM
 #3

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?
No. Not reliably, but you can use the script in contrib/linearize to get a linearized copy.

You shouldn't generally assume that you can parse Bitcoin's internal datastructures, they're not a supported interface and may change from version to version.

Bitcoin will not be compromised
amaclin
Legendary
*
Offline Offline

Activity: 1260
Merit: 1000


View Profile
October 31, 2016, 07:57:17 AM
 #4

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?
No. Not reliably, but you can use the script in contrib/linearize to get a linearized copy.
linearizing 80+gb is too boring for everyday usage  Grin
pepethefrog
Member
**
Offline Offline

Activity: 120
Merit: 10


Pepe is NOT a hate symbol


View Profile
November 03, 2016, 08:07:58 AM
 #5

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?
No. Not reliably, but you can use the script in contrib/linearize to get a linearized copy.

You shouldn't generally assume that you can parse Bitcoin's internal datastructures, they're not a supported interface and may change from version to version.

That is good to know.
Pepe has seen some altcoins that have added a 4 byte index to the block headers to number the blocks according to their "height".
This way you can atleast know (during the linear parsing of the chain) that you stay "on track".

Bipcoin: bip1W2nq2vhM4f6kaHSsVD5J1LdRb1M3mCqftwq6erpEeKzsj8Kjrxy5xUs9VAtF233nNzcMQN2ZQfJ fvi2WensZ5tGJv2ysY8
Pepe is NOT a hate symbol.
amaclin
Legendary
*
Offline Offline

Activity: 1260
Merit: 1000


View Profile
November 03, 2016, 08:30:55 AM
 #6

That is good to know.
Pepe has seen some altcoins that have added a 4 byte index to the block headers to number the blocks according to their "height".
This way you can atleast know (during the linear parsing of the chain) that you stay "on track".
This is not a problem. Blocks are linked together with their hashes. No need to check the height field (and this field doesn't exists in early blocks)
Checking "staying on track" is very easy. Some pseudocode:

Code:
void process ( const Block& b )
{
  static Hash256 prev;
  assert ( block.getPrev ( ) == prev );
  prev = block.getKey ( );
// ...your code here...
}

The problem is "what to do with the block loaded from blk-file if it is not possible to link it right now to its parent, because the parent is below in blk-file"
pepethefrog
Member
**
Offline Offline

Activity: 120
Merit: 10


Pepe is NOT a hate symbol


View Profile
November 03, 2016, 09:23:04 AM
 #7

That is good to know.
Pepe has seen some altcoins that have added a 4 byte index to the block headers to number the blocks according to their "height".
This way you can atleast know (during the linear parsing of the chain) that you stay "on track".
This is not a problem. Blocks are linked together with their hashes. No need to check the height field (and this field doesn't exists in early blocks)
Checking "staying on track" is very easy. Some pseudocode:

Code:
void process ( const Block& b )
{
  static Hash256 prev;
  assert ( block.getPrev ( ) == prev );
  prev = block.getKey ( );
// ...your code here...
}

The problem is "what to do with the block loaded from blk-file if it is not possible to link it right now to its parent, because the parent is below in blk-file"


Yes ofcourse, but since the blocks are already in the blk-files, we can assume that the full bitcoin node has already checked and validated their hashes.
Pepe thought that this thread is about whether you can assume that the blocks inside the blk-files are stored in linear order or not?
Now, if they are not, it will require you to either know the index structure to be able to quickly look up the position of the next block,
or, to have a numbered index in the block header of the block to quickly check the linear progression.
Looking up a number is way faster than calculating and comparing hashes.

Just Pepe's 2 cents

Bipcoin: bip1W2nq2vhM4f6kaHSsVD5J1LdRb1M3mCqftwq6erpEeKzsj8Kjrxy5xUs9VAtF233nNzcMQN2ZQfJ fvi2WensZ5tGJv2ysY8
Pepe is NOT a hate symbol.
amaclin
Legendary
*
Offline Offline

Activity: 1260
Merit: 1000


View Profile
November 03, 2016, 10:03:27 AM
 #8

Now, if they are not
They are not.

Quote
, it will require you to either know the index structure to be able to quickly look up the position of the next block,
Too difficult for single-pass parser

For example, the block order is ... 99, 100, 101, 200, 102, 103, ... 199, 201, 202, 203...

The easiest way for blockchain parser is to keep in memory the blocks which can not be added to
a memory-chain memory-tree for processing them later. I wanted to eliminate this routine
by asking bitcoin client save them in right order. OK, it is not possible today. I've already written a
code for linearization-on-the-fly. The topic can be closed.
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!