Bitcoin Forum
May 26, 2024, 02:10:29 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: How was the average size of blk*.data chosen?  (Read 84 times)
jdom1824 (OP)
Newbie
*
Offline Offline

Activity: 12
Merit: 0


View Profile
May 07, 2024, 12:49:43 AM
 #1

Hello,

I notice that the Bitcoin database is stored sequentially in blk*.dat files, with an average size between 128 to 134MB. I would like to know how this average size is determined and if there are any tests that demonstrate this size is the most efficient.

Regards.
achow101
Moderator
Legendary
*
Offline Offline

Activity: 3402
Merit: 6657


Just writing some code


View Profile WWW
May 07, 2024, 02:18:30 AM
Merited by LoyceV (4), ABCbits (3)
 #2

blk*.dat files have a maximum size of 128 MiB, and new files are simply started if the block that is about to be written to disk would cause the file to exceed this size limit.

The PR that introduced this limit, https://github.com/bitcoin/bitcoin/pull/1677, suggests that this limit was chosen to reduce disk fragmentation of the files. However, I don't think there is anything that determines or indicates that this size is "most efficient", whatever that means. This is often the case with many of the maximums that exist throughout Bitcoin and other protocols - maximums are often "it just feels like that's a big enough number".

jdom1824 (OP)
Newbie
*
Offline Offline

Activity: 12
Merit: 0


View Profile
May 07, 2024, 03:45:06 PM
 #3

blk*.dat files have a maximum size of 128 MiB, and new files are simply started if the block that is about to be written to disk would cause the file to exceed this size limit.

The PR that introduced this limit, https://github.com/bitcoin/bitcoin/pull/1677, suggests that this limit was chosen to reduce disk fragmentation of the files. However, I don't think there is anything that determines or indicates that this size is "most efficient", whatever that means. This is often the case with many of the maximums that exist throughout Bitcoin and other protocols - maximums are often "it just feels like that's a big enough number".

It seems that the limit was set to prevent excessive fragmentation, since the blocks range from 1 to 4 megabytes and we don't want to handle large numbers of files. I apologize, when I talk about efficiency I'm referring to how this size can provide better read/write performance within the node. For example, in the Google File System paper, a chunk size of 64 MB was determined by user behavior because a larger chunk size significantly reduces the amount of interaction required between clients and the master node for read/ write operations.
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!