Exact binary map of database blockchain?!

[LvM]

Exact binary map of database blockchain?!

As far I can see its only "JAMES" who tried to document the EXACT structure of the blockchain.

"Bitcoin: 285 bytes that changed the world", 12:22AM, 12th January 2012

http://james.lab6.com/2012/01/12/bitcoin-285-bytes-that-changed-the-world/

Asking the experts:

1. Is JAMES' docu correct?
2. If yes, why is it not included in the "official" https://en.bitcoin.it/wiki/

I looked everywhere, but found nowhere else so extremely important and needed things like that.

[jackjack]

Blockchain = concatenation of blocks
Block structure: https://en.bitcoin.it/wiki/Protocol_specification

[LvM]

Blockchain = concatenation of blocks
Block structure: https://en.bitcoin.it/wiki/Protocol_specification

Yes, ok, "Block" or Blockchain. Just as you like it.

Yes, I know your wiki-link. But sorry, cannot use it at all.
Far away from useable, "normal" docus of databases like that one of James'

[bukaj]

AFAIK there is no such thing as binary map of database blockchain. Every client uses it's own format to store blockchain (levelDB for example).

[LvM]

What I mean is an exact and comprehensive byte-by-byte description
such as provided by Armory for its wallets.

https://bitcoinarmory.com/armory-wallet-files/
https://bitcoinarmory.com/wp-content/uploads/2012/01/ArmoryWalletFile.png

[LvM]

What I mean is an exact and comprehensive byte-by-byte description
such as provided by Armory for its wallets.

https://bitcoinarmory.com/armory-wallet-files/
https://bitcoinarmory.com/wp-content/uploads/2012/01/ArmoryWalletFile.png

Is there really nobody who can or will answer my question ??
I think for a developer its one of the most important infos to know.

[tyrion70]

As far as I can tell, Jamess account is correct. Reason that it's not on the wiki: nobody ever found it interesting enough to put there.. Feel free to create an account and put it there ;-)

As for you other remark.. I guess developers don't need a post like Jamess, they read it in the source if they find they need the information:
https://github.com/bitcoin/bitcoin

[kjj]

The files have a loose format.  The files contain blocks (magic number, size, data), essentially the same format as the block network message.  The files should not contain other data, but all parsers should be careful to discard anything that isn't a valid block.

The data part of a block includes the header, a transaction count, and then transaction data.

All of this is on the wiki.  Start here.  Don't miss the link near the bottom.

[DannyHamilton]

1. Is JAMES' docu correct?

Yes.

2. If yes, why is it not included in the "official" https://en.bitcoin.it/wiki/

Because it isn't "useful".

Those who have use of such information (developers), can get it reliably from source code.  Those who are unable to get what they need from source code, don't really have any useful need for the information.

[LvM]

Yeah! I am really sorry that an experienced banking developer is so silly.

Sorry, I really need these infos somehow on the lowest level.
Functions/APIs already working somehow with a database are much too high for me.
 

[etotheipi]

The blk*.dat files stored in the Bitcoin home directory are quite simple -- just a straight concatenation of blocks of the form:

Code:

[MagicBytes(4) || BlockSizeBytes(4) || BlockHeader(80) || NumTx(VAR_INT) || Tx0 || Tx1 || ... || TxN]

The "BlockSizeBytes" field includes the BlockHeader and NumTx, so you only have to read that and skip that many bytes if you want to jump to the next block.  This is is also how blocks are serialized peer-to-peer. 

[LvM]

The blk*.dat files stored in the Bitcoin home directory are quite simple -- just a straight concatenation of blocks of the form:

Code:

[MagicBytes(4) || BlockSizeBytes(4) || BlockHeader(80) || NumTx(VAR_INT) || Tx0 || Tx1 || ... || TxN]

The "BlockSizeBytes" field includes the BlockHeader and NumTx, so you only have to read that and skip that many bytes if you want to jump to the next block.  This is is also how blocks are serialized peer-to-peer. 

Yes, thank you.
But searching around I already saw your equal posting in the last year:
https://bitcointalk.org/index.php?topic=101514.msg1111214#msg1111214
Its not what I mean, namely something like the perfect description of your own Armory wallet.

Using a third party database developers must exactly know everything about its structure, each of its fields...

We get those infos from each normal Bank.
But in BTC this seems to be top secret.

[etotheipi]

The blk*.dat files stored in the Bitcoin home directory are quite simple -- just a straight concatenation of blocks of the form:

Code:

[MagicBytes(4) || BlockSizeBytes(4) || BlockHeader(80) || NumTx(VAR_INT) || Tx0 || Tx1 || ... || TxN]

The "BlockSizeBytes" field includes the BlockHeader and NumTx, so you only have to read that and skip that many bytes if you want to jump to the next block.  This is is also how blocks are serialized peer-to-peer. 

Yes, thank you.
But searching around I already saw your equal posting in the last year:
https://bitcointalk.org/index.php?topic=101514.msg1111214#msg1111214
Its not what I mean, namely something like the perfect description of your own Armory wallet.

Using a third party database developers must exactly know everything about its structure, each of its fields...

We get those infos from each normal Bank.
But in BTC this seems to be top secret.

The serialization of the headers and the transactions are documented on the protocol specification page.  What I wrote above is how those individual pieces come together in the blk*.dat files which contain the entire blockchain.  I did make an exact binary map of the transactions, though it was made before P2SH.  You can see it here.  There's a few other nice inkscape drawings here. 

The only caveat is that now the blk*.dat files are padded to 16MB chunks, so the end of the files may contain lots of zeros.  But this is easy to detect, because blocks are never split between files.  So you only have to check for (isEndOfFile() OR magicBytes==0x00000000) to determine that there are no more blocks in the file.

[kjj]

The only caveat is that now the blk*.dat files are padded to 16MB chunks, so the end of the files may contain lots of zeros.  But this is easy to detect, because blocks are never split between files.  So you only have to check for (isEndOfFile() OR magicBytes==0x00000000) to determine that there are no more blocks in the file.

Because the block files are externally indexed, I usually tell people not to make assumptions about what is in them.  The -loadblocks code is a very good example of a parser that can tolerate all manners of garbage between blocks, corrupt blocks, etc.

That said, when I was ripping my files for the bootstrap torrent, I found exactly zero cruft in what my node had accumulated over the last few years.

[etotheipi]

The only caveat is that now the blk*.dat files are padded to 16MB chunks, so the end of the files may contain lots of zeros.  But this is easy to detect, because blocks are never split between files.  So you only have to check for (isEndOfFile() OR magicBytes==0x00000000) to determine that there are no more blocks in the file.

Because the block files are externally indexed, I usually tell people not to make assumptions about what is in them.  The -loadblocks code is a very good example of a parser that can tolerate all manners of garbage between blocks, corrupt blocks, etc.

That said, when I was ripping my files for the bootstrap torrent, I found exactly zero cruft in what my node had accumulated over the last few years.

I can vouch that they are extremely consistent.  Armory does a raw scan of them on every load (which will be changed soon, thank god).  Despite being used by thousands of people over the course of the last 12-18 months, there is rarely any problems with it.   Some of the recent Armory problems have actually had to do with memory usage on 32-bit architectures, and not due to actual blk*.dat corruption (switching to the 64-bit version solved it).

[LvM]

The blk*.dat files stored in the Bitcoin home directory are quite simple -- just a straight concatenation of blocks of the form:

Code:

[MagicBytes(4) || BlockSizeBytes(4) || BlockHeader(80) || NumTx(VAR_INT) || Tx0 || Tx1 || ... || TxN]

The "BlockSizeBytes" field includes the BlockHeader and NumTx, so you only have to read that and skip that many bytes if you want to jump to the next block.  This is is also how blocks are serialized peer-to-peer. 

Yes, thank you.
But searching around I already saw your equal posting in the last year:
https://bitcointalk.org/index.php?topic=101514.msg1111214#msg1111214
Its not what I mean, namely something like the perfect description of your own Armory wallet.

Using a third party database developers must exactly know everything about its structure, each of its fields...

We get those infos from each normal Bank.
But in BTC this seems to be top secret.

The serialization of the headers and the transactions are documented on the protocol specification page.  What I wrote above is how those individual pieces come together in the blk*.dat files which contain the entire blockchain.  I did make an exact binary map of the transactions, though it was made before P2SH.  You can see it here.  There's a few other nice inkscape drawings here. 

The only caveat is that now the blk*.dat files are padded to 16MB chunks, so the end of the files may contain lots of zeros.  But this is easy to detect, because blocks are never split between files.  So you only have to check for (isEndOfFile() OR magicBytes==0x00000000) to determine that there are no more blocks in the file.

So far I could not use anything in the BTC wiki.
From the technical point of view we find quite a lot of more or less superficial descriptions of this and that.

But this for the transactions looks much better, thank you!
http://dl.dropboxusercontent.com/u/1139081/BitcoinImg/TxBinaryMap.png

Still missing an equally exact description of the "surrounding" "block".

And btw many thanks for your other good work, Alan.
Armory is definitely the best wallet I know.
Very well designed, very well explained and due to its even me convincingly explained "deterministic" architecture almost foolproof.

All the complaints of users having lost their money due to password problems with things like bitcoin-qt
https://bitcointalk.org/index.php?topic=85495.0;all
https://bitcointalk.org/index.php?topic=191401.0;all
have an end with Armory.

[etotheipi]

So far I could not use anything in the BTC wiki.
From the technical point of view we find quite a lot of more or less superficial descriptions of this and that.

But this for the transactions looks much better, thank you!
http://dl.dropboxusercontent.com/u/1139081/BitcoinImg/TxBinaryMap.png

Still missing an equally exact description of the "surrounding" "block".

And btw many thanks for your other good work, Alan.
Armory is definitely the best wallet I know.
Very well designed, very well explained and due to its even me convincingly explained "deterministic" architecture almost foolproof.

All the complaints of users having lost their money due to password problems with things like bitcoin-qt
https://bitcointalk.org/index.php?topic=85495.0;all
https://bitcointalk.org/index.php?topic=191401.0;all
have an end with Armory.

Thanks.  I did a lot of this because of the learning curve to get into this stuff.  Bitcoin is complicated.   But if you want to dig into it, you'll have to do a little messing around on your own.  I don't want to ruin all the fun for you

The rest of the serialization is trivial once you know the transactions.  The header is exactly 80 bytes, 6 values.  And you can look up how VAR_INTs are read from that protocol page.  The magic bytes are f9beb4d9.  The protocol page isn't pleasant, but it's not useless, either.

[kjj]

[MagicBytes(4) || BlockSizeBytes(4) || BlockHeader(80) || NumTx(VAR_INT) || Tx0 || Tx1 || ... || TxN]

So far I could not use anything in the BTC wiki.
From the technical point of view we find quite a lot of more or less superficial descriptions of this and that.

Still missing an equally exact description of the "surrounding" "block".

I reject your claim that nothing in the wiki is usable.  It really does contain everything that you need to understand a block.  Just takes some getting used to... 

The "surrounding block" is described exactly above.  There is some confusion on exactly what parts are block and which parts are message.  Most people consider the block proper to be "header + NumTx + TX_0...NumTx".  In practice, you'll pretty much never find those stored or communicated without the magic number and the size, except when using the RPC calls that return block bodies.

Keep in mind that bitcoin is nested, and a lot of the complicated bits are in the interactions.  The block itself is very simple, a short header followed by a list of transactions.  Transactions are simple too, a header (really just the version field), a list of inputs, a list of outputs, and a footer (just the locktime field).  Inputs and outputs are where things start getting strange, but even those are pretty simple in representation.

[LvM]

@kjj
If its really all so "simple" - why isn't it likewise simply explained?

Ah!
What do you mean with "bitcoin is nested" ?? What do you mean with "a lot of the complicated bits are in the interactions"
To begin with: what exactly is that "interaction"?
So far I only heard from transfers/transactions.
If you mean the same, what exactly do you mean with "complicated bits" (bits?)?

[LvM]

Bitcoins are money.

The basic logic of money transfers from A to B normally belongs to the easiest things in the world.
Technical things like encryption or the structure of involved databases change nothing in the very, very simple logic of money transfers.

In short:
A minus for the payer is a plus for the receiver.
At least time and reason of transfers are always added normally.
All is collected and balanced in the accounts of payers/receivers.
Easiest bookkeeping.

Some of these things either do not exist in BTC or simply are somehow hidden.
Where e.g. is the field for the transaction date/time?
I assume that there IS one, since Armory restored by the blockchain date and time of my transactions.

Lost after a restore were my manual "Comments" of transactions only.
BTC does indeed not have a field for the reason of transfers.
Other things like the strange "changings" not to mention....
Sorry, but even after reading quite a lot I am far away to understand that.