Should -dbcache be divisible by 1024?

[apogio]

I saw this in another thread and I decided to post a clarification.

Code:

dbcache=5120

5120 seems to be reasonable size as ^firstly it is divisible by 1024 ^{otherwise RAM might be used inefficiently} and ^secondly it is slightly more than 1/4 RAM which left the space for other OS operation.However, if the system isn't heavily used for other tasks it's safe to allocate more memory to dbcache. In such cases, increasing it to 1/2 ^{dbcache = 8192} or even 3/4 ^{dbcache = 12288} of RAM might improve performance. User should test what is the best option for him.

Should -dbcache be set to a value divisible by 1024?

It's a very normal misconception and I thought I should clear it out. Because trying to put a number that's divisible by 1024 is wrong.

There is actually no problem at all, setting the -dbcache to any integer value. You can put -dbcache=8000 or -dbcache=8192, it won't make a difference.

Memory is broken into pages which are normally 4 KiB each. This happens because any program that requests memory won't get a big continuous block, but smaller 4 KiB chunks for memory.

When we say to Bitcoin Core that we need 8000 MiB, it will do the following:

8000 MiB = 8,388,608,000 bytes
4 KiB = 4096 bytes

So there will be 8,388,608,000 / 4096 = 2,048,000 pages of memory allocated for Bitcoin Core.

Let's also explain mathematically why any number that you put as dbcache won't cause a problem

dbcache is an integer and it's measured in MiB (mebibytes).

1 MiB = 1,048,576 bytes = 2^20 bytes.

Every page in memory is 4 KiB = 4096 bytes = 2^12 bytes.

Since 2^20 / 2^12 = 2^8 = 256, it means that 1 MiB = 256 pages of 4 KiB each.

So, go ahead and try it. Put any integer for -dbcache. Let's say you pick n.

Then, the number of pages will be n MiB = n * 1,048,576 bytes. Since 1,048,576 is divisible by 4096, then any n that you choose, the divisibility will be preserved.

To conclude, use any number for -dbcache, without worrying.

About the proper dbcache size, there is a good read here: https://blog.lopp.net/effects-dbcache-size-bitcoin-node-sync-speed/, but in general I agree with satscraper.

[Cricktor]

I agree that it doesn't matter what value of n you specify for dbcache=n, even when I don't exactly know how Bitcoin Core actually allocates memory for such caches.

But 4 KiB pages aren't the only size that most CPU and OS architectures support. There's usually also something called large or huge pages, because having to manage only a fixed small size like 4 KiB isn't efficient in certain cases.

Maybe have a look here to dig deeper, for those who are interested: https://en.wikipedia.org/wiki/Page_(computer_memory)

[apogio]

I agree that it doesn't matter what value of n you specify for dbcache=n, even when I don't exactly know how Bitcoin Core actually allocates memory for such caches.

But 4 KiB pages aren't the only size that most CPU and OS architectures support. There's usually also something called large or huge pages, because having to manage only a fixed small size like 4 KiB isn't efficient in certain cases.

Maybe have a look here to dig deeper, for those who are interested: https://en.wikipedia.org/wiki/Page_(computer_memory)

Absolutely, and it's good that you mention it. In fact, I wanted to write about it myself, but I thought that since typically 4 KiB is the typical situation, I should expand my thesis based on this assumption.

As far as Bitcoin Core is concerned, I 'd love to hear some input from someone who knows, but I know that in general the amount of dbcache that you put in the argument is the total cache that you want Core to use and then the program splits it in smaller batches and uses it for the chainstate, the UTXO etc.

[BADERO]

I agree that it doesn't matter what value of n you specify for dbcache=n, even when I don't exactly know how Bitcoin Core actually allocates memory for such caches.

But 4 KiB pages aren't the only size that most CPU and OS architectures support. There's usually also something called large or huge pages, because having to manage only a fixed small size like 4 KiB isn't efficient in certain cases.

Maybe have a look here to dig deeper, for those who are interested: https://en.wikipedia.org/wiki/Page_(computer_memory)

we know that most systems use 4 KiB as the default memory page size, but typically of 2 MiB or even of 1 GiB in size, they also support large pages (also called huge pages) large pages reduce overhead of managing many small pages and improve CPU cache efficiency(as u said )

 Bitcoin Core does not request explicitly for the use of large pages when allocating memory for dbcache it uses normal memory allocation methods such as malloc and new and it relies on the operating system’s default behavior that uses standard 4 KiB pages bitcoin Core would need to perform special operations in the event that it wanted to use large pages (like mmap with MAP HUGETLB on Linux, or VirtualAlloc with MEM_LARGE_PAGES on Windows) and mainly due to the fact that it does not

voluminous pages require additional system configuration or admins configure

Certain fragmentation or failure issues can occur when managing a number of large pages this can be caused by managing of large pages in a dynamic way

Bitcoin Core needs to be simple,stable and portable across many of the OSes

Bitcoin Core will allocate memory in a safe and efficient manner, regardless of the value you set for dbcache=n as a result each integer n that is for dbcache still will align in a neat way to the page system, without any waste or inefficiency, due to 1 MiB = 1,048,576 bytes = 256 × 4 KiB pages, setting dbcache to a value that is divisible by one thousand twentyfour is not at all necessary each integer is perfectly fine

[satscraper]

In fact, the default value is 450 MiB, which is obviously not divisible by 1024, but still, I don't think Core developers would put a default value that would make RAM be used inefficiently.

In fact 450 MiB is obviously divisible by 1024.

450 MiB = 450 x 1024 KiB.

450 MiB/ 1024 = (450 x 1024) KiB/1024 = 450 KiB

Or 450 MiB/ 1024 = 450 (1024)² Byte/1024 = 450x1024 Byte = 460800 Byte

Core developers has highlighted this in their doc saying that dbcache parameter is specified as MiB(1024) indicating that memory must be measured in binary units rather than in decimal ones.

[apogio]

In fact, the default value is 450 MiB, which is obviously not divisible by 1024, but still, I don't think Core developers would put a default value that would make RAM be used inefficiently.

In fact 450 MiB is obviously divisible by 1024.

450 MiB = 450 x 1024 KiB.

450 MiB/ 1024 = (450 x 1024) KiB/1024 = 450 KiB

Or 450 MiB/ 1024 = 450 (1024)² Byte/1024 = 450x1024 Byte = 460800 Byte

Core developers has highlighted this in their doc saying that dbcache parameter is specified as MiB(1024) indicating that memory must be measured in binary units rather than in decimal ones.

You are right, I will change it in my OP. Because, actually, any number can be put in the dbcache option. Any number is divisible with 1024 since it's measured in MiB.

[ABCbits]

But 4 KiB pages aren't the only size that most CPU and OS architectures support. There's usually also something called large or huge pages, because having to manage only a fixed small size like 4 KiB isn't efficient in certain cases.

I remember those stuff sometimes mentioned on discussion about docker and virtual machine. But AFAIK it's not enabled by default on some OS and linux distro. Although when it's enabled on Linux, some application can take advantage of it without additional programming.

Transparent huge pages are more or less completely handled by the Linux kernel. Depending on your current kernel configuration (see the next section) it can happen that memory in your application is allocated in huge pages without you - as the programmer - even knowing. It may even happen that memory for your application that was not originally allocated in a huge page is later transformed into a huge page!