Where can I find bitcoin-block internal structure description?

[mtn2022]

Hi all.

Where can I find bitcoin-block internal structure description?
I mean how can i parse .dat files with blocks and get transactions like:

Code:

[
 {
    "block-header" : {
    }   
    "body: {
         "transaction-data" : {
            "tx-count" : 22,
            "body" : [
                "transaction" : {...},
                "transaction" : {...}
            ]
         }
    }         
 },
]

Thank you all.

[1713465159]

1713465159

[1713465159]

1713465159

[1713465159]

1713465159

[n0nce]

You are referring to the structure of a blk*.dat file?

It is explained here: https://learnmeabitcoin.com/technical/blkdat
You can even hover over part of the genesis block to get annotations on what bytes are what, as shown below.


I also queried a few selected blocks for you (first 0x90 bytes each).

Code:

bitcoin@localhost:~/.bitcoin/blocks> hexdump -C blk00000.dat | head
00000000  f9 be b4 d9 1d 01 00 00  01 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 3b a3 ed fd  |............;...|
00000030  7a 7b 12 b2 7a c7 2c 3e  67 76 8f 61 7f c8 1b c3  |z{..z.,>gv.a....|
00000040  88 8a 51 32 3a 9f b8 aa  4b 1e 5e 4a 29 ab 5f 49  |..Q2:...K.^J)._I|
00000050  ff ff 00 1d 1d ac 2b 7c  01 01 00 00 00 01 00 00  |......+|........|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ff ff  |................|
00000080  ff ff 4d 04 ff ff 00 1d  01 04 45 54 68 65 20 54  |..M.......EThe T|
00000090  69 6d 65 73 20 30 33 2f  4a 61 6e 2f 32 30 30 39  |imes 03/Jan/2009|
bitcoin@localhost:~/.bitcoin/blocks> hexdump -C blk01337.dat | head
00000000  f9 be b4 d9 e8 e3 11 00  00 00 00 20 ea 58 90 18  |........... .X..|
00000010  60 a5 72 4d 96 f4 86 48  cb 7c c4 1d 35 6f 15 4a  |`.rM...H.|..5o.J|
00000020  fa 9e 26 00 00 00 00 00  00 00 00 00 8a 12 56 61  |..&...........Va|
00000030  81 67 04 09 5e b6 b9 ca  e6 6b 55 11 e8 15 b3 03  |.g..^....kU.....|
00000040  44 d9 ec f2 28 09 37 aa  6a e3 b6 86 8d ff 68 5b  |D...(.7.j.....h[|
00000050  7b 4f 2f 17 36 4f 8f 4c  fd 7e 05 01 00 00 00 00  |{O/.6O.L.~......|
00000060  01 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 ff ff ff ff 55 03  0d 2c 08 41 d6 da 3f e1  |......U..,.A..?.|
00000090  c8 72 c2 41 d6 da 3f e1  58 d3 4b 2f 42 54 43 2e  |.r.A..?.X.K/BTC.|
bitcoin@localhost:~/.bitcoin/blocks> hexdump -C blk03203.dat | head
00000000  f9 be b4 d9 88 e5 16 00  00 a0 1a 21 d2 aa ee 3a  |...........!...:|
00000010  0b 36 2d 02 14 34 b7 e2  f8 28 ac 90 04 15 be ce  |.6-..4...(......|
00000020  2d 25 01 00 00 00 00 00  00 00 00 00 77 ee 79 dd  |-%..........w.y.|
00000030  57 09 1c d6 b3 48 03 c2  c6 84 46 5a f4 c9 cc 2a  |W....H....FZ...*|
00000040  e6 bf 80 db 97 3d bf bf  fb 80 c7 0c d4 b1 29 63  |.....=........)c|
00000050  94 c8 08 17 30 cf 27 9d  fd 9b 0b 01 00 00 00 00  |....0.'.........|
00000060  01 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 ff ff ff ff 4c 03  fc 84 0b 13 62 69 6e 61  |......L.....bina|
00000090  6e 63 65 2f 37 38 39 48  00 0b 03 7e f9 3b fc fa  |nce/789H...~.;..|
bitcoin@localhost:~/.bitcoin/blocks>

As we can see, they are just a concatenation of blocks. Blocks start with the mainnet's magic bytes f9beb4d9, followed by 4 bytes of size of the following block and then the header and body.

Bitcoin Wiki actually defines the whole structure quite well.
More details about the header here: https://en.bitcoin.it/wiki/Block_hashing_algorithm

You can quickly find all the block starts in blk00000.dat like this:

Code:

bitcoin@localhost:~/.bitcoin/blocks> hexdump -C blk00000.dat | grep "f9 be b4" | head
00000000  f9 be b4 d9 1d 01 00 00  01 00 00 00 00 00 00 00  |................|
00000120  ac 00 00 00 00 f9 be b4  d9 d7 00 00 00 01 00 00  |................|
00000200  00 00 00 00 f9 be b4 d9  d7 00 00 00 01 00 00 00  |................|
000002e0  00 00 00 f9 be b4 d9 d7  00 00 00 01 00 00 00 bd  |................|
000003c0  00 00 f9 be b4 d9 d7 00  00 00 01 00 00 00 49 44  |..............ID|
000004a0  00 f9 be b4 d9 d7 00 00  00 01 00 00 00 85 14 4a  |...............J|
00000580  f9 be b4 d9 d7 00 00 00  01 00 00 00 fc 33 f5 96  |.............3..|
00000810  1f f7 05 29 d6 2e 0b a1  ac 00 00 00 00 f9 be b4  |...)............|
000008f0  86 43 f6 56 b4 12 a3 ac  00 00 00 00 f9 be b4 d9  |.C.V............|
000009d0  b3 86 81 64 25 dd ac 00  00 00 00 f9 be b4 d9 d7  |...d%...........|

You can also see where the next block starts by using the 0x125 'offset' from the above command's output and hexdump the block file until that point (plus 8 bytes of the next block just to confirm it really is the next block):

Code:

bitcoin@localhost:~/.bitcoin/blocks> head -c $(python3 -c 'print(0x125+8)') blk00000.dat | hexdump -C
00000000  f9 be b4 d9 1d 01 00 00  01 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 3b a3 ed fd  |............;...|
00000030  7a 7b 12 b2 7a c7 2c 3e  67 76 8f 61 7f c8 1b c3  |z{..z.,>gv.a....|
00000040  88 8a 51 32 3a 9f b8 aa  4b 1e 5e 4a 29 ab 5f 49  |..Q2:...K.^J)._I|
00000050  ff ff 00 1d 1d ac 2b 7c  01 01 00 00 00 01 00 00  |......+|........|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 ff ff  |................|
00000080  ff ff 4d 04 ff ff 00 1d  01 04 45 54 68 65 20 54  |..M.......EThe T|
00000090  69 6d 65 73 20 30 33 2f  4a 61 6e 2f 32 30 30 39  |imes 03/Jan/2009|
000000a0  20 43 68 61 6e 63 65 6c  6c 6f 72 20 6f 6e 20 62  | Chancellor on b|
000000b0  72 69 6e 6b 20 6f 66 20  73 65 63 6f 6e 64 20 62  |rink of second b|
000000c0  61 69 6c 6f 75 74 20 66  6f 72 20 62 61 6e 6b 73  |ailout for banks|
000000d0  ff ff ff ff 01 00 f2 05  2a 01 00 00 00 43 41 04  |........*....CA.|
000000e0  67 8a fd b0 fe 55 48 27  19 67 f1 a6 71 30 b7 10  |g....UH'.g..q0..|
000000f0  5c d6 a8 28 e0 39 09 a6  79 62 e0 ea 1f 61 de b6  |\..(.9..yb...a..|
00000100  49 f6 bc 3f 4c ef 38 c4  f3 55 04 e5 1e c1 12 de  |I..?L.8..U......|
00000110  5c 38 4d f7 ba 0b 8d 57  8a 4c 70 2b 6b f1 1d 5f  |\8M....W.Lp+k.._|
00000120  ac 00 00 00 00 f9 be b4  d9 d7 00 00 00           |.............|
0000012d

This is just to get your toes wet. For a real parser, you don't want to grep for 3 out of the 4 magic bytes like I did, and you don't want to grep line-by-line, as I can easily have this sequence split across newlines through hexdump.
So just read raw bytes with whatever programming language you prefer and get indexes for f9 be b4 d9.
Or do that for the first block in a file and then use the 'size' field to jump to the next block for all subsequent blocks.

Rest of the parsing is just simple block parsing according to https://en.bitcoin.it/wiki/Block.

[mtn2022]

Oh. So much info. Thanks a lot! Looking in...