Bitcoin Forum
February 23, 2019, 04:21:46 PM *
News: Latest Bitcoin Core release: 0.17.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: What are the markers byte in wallet file  (Read 148 times)
hushan
Jr. Member
*
Offline Offline

Activity: 57
Merit: 4


View Profile
March 22, 2018, 02:23:01 AM
 #1

So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like
Code:
\07, \09, \0a
etc.

$ db_dump -p wallet.dat
...
 \07keymeta!\03\ff \15\06F0\97a\a4F\e2\15\01\b9;U%\97\d7\8c\0b\0b\e4`\bc\96\b2\ee@\db\8eL
 \0a\00\00\00\ca3\91Z\00\00\00\00\0cm/0'/1'/461'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ff*r{\de5\e4/\a2_\f6+rIv\db(\92r\ee\dd\ce\18\daC\13\df\85\e3l6H
 \0a\00\00\00\c93\91Z\00\00\00\00\0cm/0'/0'/254'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ffD9Y=\8e\0f\a51\95\ecs\98u\9a\da[A\82v\dd=\95\f9\bbf\a5\eby\18\0b9
 \0a\00\00\00\ca3\91Z\00\00\00\00\0cm/0'/1'/594'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ffF\b3\c9\81\d4\1btNv\e9\ad\b3%\bbb\14\93\ff\e2}\9a\b4\dc\05\dc\b01\a5\87\ac^
 \0a\00\00\00\c93\91Z\00\00\00\00\0cm/0'/0'/638'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ff\80A\e8\ea\c8*\92DU\e5W\fe\b3\e9]Vk2\b3\e4\fep\10\e4i\d5\1f\\\83\\3
 \0a\00\00\00\cb3\91Z\00\00\00\00\0cm/0'/1'/921'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07purpose"18V1UjWhRBEq7Kj3cXtLLjh2imd6UPWz9A
 \07receive
 \07purpose"1H1uEWBtsHe4tMzCRqGL6BkZBbu6xfnsmR
 \07receive
 \07purpose"32AvF5Vmnn9zD6VRf1D3vZXFHPuwdgEKqw
 \07receive
 \07purpose"35gnJvBQD424mgH9j2Udu6zoEsSXhzQkky
 \07receive
 \07purpose"3AMa4ZAh6R83sWRKsFvprpXvhzDuZ34yqm
 \07receive
 \07purpose"3M7vPcw9h6KaiRfpNNBRFC6mc918UaW6NF
 \07receive
 \07purpose*bc1qnfqmg4ty3qgft79gk3zgym3w3hqxmznxg9avup
 \07receive
 \07version
 \ac\97\02\00
 \09bestblock
 \ac\97\02\00\00
 \0aminversion
 \9cp\02\00
...

BTC: 38ykUxZeSv5aqav1keCc4KUxXLPPGkfrm5 LTC: MPpWNvqDCVB1PHJt4A28j9oFfc6Y1KKDno ETH: 0x07C8c68d5253247038947EF9495b054160c8737c
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
achow101
Moderator
Legendary
*
expert
Offline Offline

Activity: 1680
Merit: 2086


bc1qshxkrpe4arppq89fpzm6c0tpdvx5cfkve2c8kl


View Profile WWW
March 22, 2018, 04:51:25 AM
 #2

So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like
Code:
\07, \09, \0a
etc.
Those aren't marker bytes. They are length prefixes. The specify the length of the following piece of data.

hushan
Jr. Member
*
Offline Offline

Activity: 57
Merit: 4


View Profile
March 22, 2018, 07:44:59 AM
 #3

So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like
Code:
\07, \09, \0a
etc.
Those aren't marker bytes. They are length prefixes. The specify the length of the following piece of data.

Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

BTC: 38ykUxZeSv5aqav1keCc4KUxXLPPGkfrm5 LTC: MPpWNvqDCVB1PHJt4A28j9oFfc6Y1KKDno ETH: 0x07C8c68d5253247038947EF9495b054160c8737c
starmyc
Member
**
Offline Offline

Activity: 194
Merit: 86

Software engineer


View Profile WWW
March 22, 2018, 08:04:14 AM
 #4

Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

Having took a look sometimes to dbwrapper in Bitcoin Core, I would say it is not absolutely necessary, but it helps for error checking while reading records, especially badly formatted ones (don't forget some parts of wallet/chainstate/whatever db) can be obfuscated/crypted, and having those kind of metadata helps to know if data is valid or not.

Software engineer.
Tip me if you find my help useful: 3QBRMM2CLUTd7hqL4T9Ja2wy6B1VkbXyZ3
hushan
Jr. Member
*
Offline Offline

Activity: 57
Merit: 4


View Profile
March 22, 2018, 09:20:59 AM
 #5

Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

Having took a look sometimes to dbwrapper in Bitcoin Core, I would say it is not absolutely necessary, but it helps for error checking while reading records, especially badly formatted ones (don't forget some parts of wallet/chainstate/whatever db) can be obfuscated/crypted, and having those kind of metadata helps to know if data is valid or not.

That makes sense, thanks for answering!

BTC: 38ykUxZeSv5aqav1keCc4KUxXLPPGkfrm5 LTC: MPpWNvqDCVB1PHJt4A28j9oFfc6Y1KKDno ETH: 0x07C8c68d5253247038947EF9495b054160c8737c
achow101
Moderator
Legendary
*
expert
Offline Offline

Activity: 1680
Merit: 2086


bc1qshxkrpe4arppq89fpzm6c0tpdvx5cfkve2c8kl


View Profile WWW
March 22, 2018, 06:43:56 PM
Merited by Foxpup (1)
 #6

Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.
The lengths are there because of Bitcoin Core's serialization framework. It tends to prepend everything with the length and it's kind of annoying to make it not do that. It's easier for it to just do that for everything because some things need to be length prepended and others don't, and it also depends on the use case (e.g. in transactions, length prepending is necessary, but not necessarily for the databse, but the same data ends up in transactions as well as in the database).

Also, Bitcoin Core tends to have multiple things concatenated in each key or value so the length prefix helps to differentiate between the two (e.g. each key in the wallet is some string followed by the actual key like a public key, so the length prefix lets us know where the string ends and the pubkey begins).

Pages: [1]
  Print  
 
Jump to:  

Bitcointalk.org is not available or authorized for sale. Do not believe any fake listings.
Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!