Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: hushan on March 22, 2018, 02:23:01 AM



Title: What are the markers byte in wallet file
Post by: hushan on March 22, 2018, 02:23:01 AM
So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like
Code:
\07, \09, \0a
etc.

$ db_dump -p wallet.dat
...
 \07keymeta!\03\ff \15\06F0\97a\a4F\e2\15\01\b9;U%\97\d7\8c\0b\0b\e4`\bc\96\b2\ee@\db\8eL
 \0a\00\00\00\ca3\91Z\00\00\00\00\0cm/0'/1'/461'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ff*r{\de5\e4/\a2_\f6+rIv\db(\92r\ee\dd\ce\18\daC\13\df\85\e3l6H
 \0a\00\00\00\c93\91Z\00\00\00\00\0cm/0'/0'/254'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ffD9Y=\8e\0f\a51\95\ecs\98u\9a\da[A\82v\dd=\95\f9\bbf\a5\eby\18\0b9
 \0a\00\00\00\ca3\91Z\00\00\00\00\0cm/0'/1'/594'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ffF\b3\c9\81\d4\1btNv\e9\ad\b3%\bbb\14\93\ff\e2}\9a\b4\dc\05\dc\b01\a5\87\ac^
 \0a\00\00\00\c93\91Z\00\00\00\00\0cm/0'/0'/638'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ff\80A\e8\ea\c8*\92DU\e5W\fe\b3\e9]Vk2\b3\e4\fep\10\e4i\d5\1f\\\83\\3
 \0a\00\00\00\cb3\91Z\00\00\00\00\0cm/0'/1'/921'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07purpose"18V1UjWhRBEq7Kj3cXtLLjh2imd6UPWz9A
 \07receive
 \07purpose"1H1uEWBtsHe4tMzCRqGL6BkZBbu6xfnsmR
 \07receive
 \07purpose"32AvF5Vmnn9zD6VRf1D3vZXFHPuwdgEKqw
 \07receive
 \07purpose"35gnJvBQD424mgH9j2Udu6zoEsSXhzQkky
 \07receive
 \07purpose"3AMa4ZAh6R83sWRKsFvprpXvhzDuZ34yqm
 \07receive
 \07purpose"3M7vPcw9h6KaiRfpNNBRFC6mc918UaW6NF
 \07receive
 \07purpose*bc1qnfqmg4ty3qgft79gk3zgym3w3hqxmznxg9avup
 \07receive
 \07version
 \ac\97\02\00
 \09bestblock
 \ac\97\02\00\00
 \0aminversion
 \9cp\02\00
...


Title: Re: What are the markers byte in wallet file
Post by: achow101 on March 22, 2018, 04:51:25 AM
So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like
Code:
\07, \09, \0a
etc.
Those aren't marker bytes. They are length prefixes. The specify the length of the following piece of data.


Title: Re: What are the markers byte in wallet file
Post by: hushan on March 22, 2018, 07:44:59 AM
So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like
Code:
\07, \09, \0a
etc.
Those aren't marker bytes. They are length prefixes. The specify the length of the following piece of data.

Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.


Title: Re: What are the markers byte in wallet file
Post by: starmyc on March 22, 2018, 08:04:14 AM
Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

Having took a look sometimes to dbwrapper in Bitcoin Core, I would say it is not absolutely necessary, but it helps for error checking while reading records, especially badly formatted ones (don't forget some parts of wallet/chainstate/whatever db) can be obfuscated/crypted, and having those kind of metadata helps to know if data is valid or not.


Title: Re: What are the markers byte in wallet file
Post by: hushan on March 22, 2018, 09:20:59 AM
Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

Having took a look sometimes to dbwrapper in Bitcoin Core, I would say it is not absolutely necessary, but it helps for error checking while reading records, especially badly formatted ones (don't forget some parts of wallet/chainstate/whatever db) can be obfuscated/crypted, and having those kind of metadata helps to know if data is valid or not.

That makes sense, thanks for answering!


Title: Re: What are the markers byte in wallet file
Post by: achow101 on March 22, 2018, 06:43:56 PM
Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.
The lengths are there because of Bitcoin Core's serialization framework. It tends to prepend everything with the length and it's kind of annoying to make it not do that. It's easier for it to just do that for everything because some things need to be length prepended and others don't, and it also depends on the use case (e.g. in transactions, length prepending is necessary, but not necessarily for the databse, but the same data ends up in transactions as well as in the database).

Also, Bitcoin Core tends to have multiple things concatenated in each key or value so the length prefix helps to differentiate between the two (e.g. each key in the wallet is some string followed by the actual key like a public key, so the length prefix lets us know where the string ends and the pubkey begins).