What are the markers byte in wallet file

[hushan]

So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like

Code:

\07, \09, \0a

etc.

$ db_dump -p wallet.dat
...
 \07keymeta!\03\ff \15\06F0\97a\a4F\e2\15\01\b9;U%\97\d7\8c\0b\0b\e4`\bc\96\b2\ee@\db\8eL
 \0a\00\00\00\ca3\91Z\00\00\00\00\0cm/0'/1'/461'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ff*r{\de5\e4/\a2_\f6+rIv\db(\92r\ee\dd\ce\18\daC\13\df\85\e3l6H
 \0a\00\00\00\c93\91Z\00\00\00\00\0cm/0'/0'/254'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ffD9Y=\8e\0f\a51\95\ecs\98u\9a\da[A\82v\dd=\95\f9\bbf\a5\eby\18\0b9
 \0a\00\00\00\ca3\91Z\00\00\00\00\0cm/0'/1'/594'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ffF\b3\c9\81\d4\1btNv\e9\ad\b3%\bbb\14\93\ff\e2}\9a\b4\dc\05\dc\b01\a5\87\ac^
 \0a\00\00\00\c93\91Z\00\00\00\00\0cm/0'/0'/638'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ff\80A\e8\ea\c8*\92DU\e5W\fe\b3\e9]Vk2\b3\e4\fep\10\e4i\d5\1f\\\83\\3
 \0a\00\00\00\cb3\91Z\00\00\00\00\0cm/0'/1'/921'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07purpose"18V1UjWhRBEq7Kj3cXtLLjh2imd6UPWz9A
 \07receive
 \07purpose"1H1uEWBtsHe4tMzCRqGL6BkZBbu6xfnsmR
 \07receive
 \07purpose"32AvF5Vmnn9zD6VRf1D3vZXFHPuwdgEKqw
 \07receive
 \07purpose"35gnJvBQD424mgH9j2Udu6zoEsSXhzQkky
 \07receive
 \07purpose"3AMa4ZAh6R83sWRKsFvprpXvhzDuZ34yqm
 \07receive
 \07purpose"3M7vPcw9h6KaiRfpNNBRFC6mc918UaW6NF
 \07receive
 \07purpose*bc1qnfqmg4ty3qgft79gk3zgym3w3hqxmznxg9avup
 \07receive
 \07version
 \ac\97\02\00
 \09bestblock
 \ac\97\02\00\00
 \0aminversion
 \9cp\02\00
...

[achow101]

So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like

Code:

\07, \09, \0a

etc.

Those aren't marker bytes. They are length prefixes. The specify the length of the following piece of data.

[hushan]

So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like

Code:

\07, \09, \0a

etc.

Those aren't marker bytes. They are length prefixes. The specify the length of the following piece of data.

Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

[starmyc]

Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

Having took a look sometimes to dbwrapper in Bitcoin Core, I would say it is not absolutely necessary, but it helps for error checking while reading records, especially badly formatted ones (don't forget some parts of wallet/chainstate/whatever db) can be obfuscated/crypted, and having those kind of metadata helps to know if data is valid or not.

[hushan]

Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

Having took a look sometimes to dbwrapper in Bitcoin Core, I would say it is not absolutely necessary, but it helps for error checking while reading records, especially badly formatted ones (don't forget some parts of wallet/chainstate/whatever db) can be obfuscated/crypted, and having those kind of metadata helps to know if data is valid or not.

That makes sense, thanks for answering!

[achow101]

Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

The lengths are there because of Bitcoin Core's serialization framework. It tends to prepend everything with the length and it's kind of annoying to make it not do that. It's easier for it to just do that for everything because some things need to be length prepended and others don't, and it also depends on the use case (e.g. in transactions, length prepending is necessary, but not necessarily for the databse, but the same data ends up in transactions as well as in the database).

Also, Bitcoin Core tends to have multiple things concatenated in each key or value so the length prefix helps to differentiate between the two (e.g. each key in the wallet is some string followed by the actual key like a public key, so the length prefix lets us know where the string ends and the pubkey begins).