natural order of blocks in blk-files?

[amaclin]

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

[1715328175]

1715328175

[1715328175]

1715328175

[1715328175]

1715328175

[1715328175]

1715328175

[2112]

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

Did you try using "-connect" to force talking to just one peer?

I haven't tried 0.13.1, but with some older versions that resulted in "reproducible ordering", I don't recall if it was "natural order".

[gmaxwell]

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

No. Not reliably, but you can use the script in contrib/linearize to get a linearized copy.

You shouldn't generally assume that you can parse Bitcoin's internal datastructures, they're not a supported interface and may change from version to version.

[amaclin]

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

No. Not reliably, but you can use the script in contrib/linearize to get a linearized copy.

linearizing 80+gb is too boring for everyday usage 

[pepethefrog]

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

No. Not reliably, but you can use the script in contrib/linearize to get a linearized copy.

You shouldn't generally assume that you can parse Bitcoin's internal datastructures, they're not a supported interface and may change from version to version.

That is good to know.
Pepe has seen some altcoins that have added a 4 byte index to the block headers to number the blocks according to their "height".
This way you can atleast know (during the linear parsing of the chain) that you stay "on track".

[amaclin]

That is good to know.
Pepe has seen some altcoins that have added a 4 byte index to the block headers to number the blocks according to their "height".
This way you can atleast know (during the linear parsing of the chain) that you stay "on track".

This is not a problem. Blocks are linked together with their hashes. No need to check the height field (and this field doesn't exists in early blocks)
Checking "staying on track" is very easy. Some pseudocode:

Code:

void process ( const Block& b )
{
  static Hash256 prev;
  assert ( block.getPrev ( ) == prev );
  prev = block.getKey ( );
// ...your code here...
}

The problem is "what to do with the block loaded from blk-file if it is not possible to link it right now to its parent, because the parent is below in blk-file"

[pepethefrog]

That is good to know.
Pepe has seen some altcoins that have added a 4 byte index to the block headers to number the blocks according to their "height".
This way you can atleast know (during the linear parsing of the chain) that you stay "on track".

This is not a problem. Blocks are linked together with their hashes. No need to check the height field (and this field doesn't exists in early blocks)
Checking "staying on track" is very easy. Some pseudocode:

Code:

void process ( const Block& b )
{
  static Hash256 prev;
  assert ( block.getPrev ( ) == prev );
  prev = block.getKey ( );
// ...your code here...
}

The problem is "what to do with the block loaded from blk-file if it is not possible to link it right now to its parent, because the parent is below in blk-file"

Yes ofcourse, but since the blocks are already in the blk-files, we can assume that the full bitcoin node has already checked and validated their hashes.
Pepe thought that this thread is about whether you can assume that the blocks inside the blk-files are stored in linear order or not?
Now, if they are not, it will require you to either know the index structure to be able to quickly look up the position of the next block,
or, to have a numbered index in the block header of the block to quickly check the linear progression.
Looking up a number is way faster than calculating and comparing hashes.

Just Pepe's 2 cents

[amaclin]

Now, if they are not

They are not.

, it will require you to either know the index structure to be able to quickly look up the position of the next block,

Too difficult for single-pass parser

For example, the block order is ... 99, 100, 101, 200, 102, 103, ... 199, 201, 202, 203...

The easiest way for blockchain parser is to keep in memory the blocks which can not be added to
a memory-chain memory-tree for processing them later. I wanted to eliminate this routine
by asking bitcoin client save them in right order. OK, it is not possible today. I've already written a
code for linearization-on-the-fly. The topic can be closed.