Selecting an adequate mempool size for a node

[unamis76]

After setting up a few nodes, on different hosts/connections and on virtual and physical machines, there's a question about setting them up that I still haven't solved: what is the correct size for a mempool given the computer's total RAM?

I've tried searching for this and I didn't find any answers or benchmarks for this, neither have I found any kind of user feedback. I've setup nodes on machines from 1 to 6GB of RAM, most of them running only basic OS services and bitcoind, and on all machines bitcoind eventually crashes without any evident reason depicted on the debug file. The client just shuts down and the logs just stop where they were... When I turn it back on sometimes it asks for a reindex, and sometimes it doesn't. I've posted a debug.log of mine on the forums a few months back and the answers I got referred the mempool, but without certainty or any real data.

I'm running some tests on a 1GB machine I currently have serving as a node and I'll report back. meanwhile, it would be nice to make a table, with user input/feedback, regarding what kind of mempool size should bitcoind be launched with.

[achow101]

Bitcoin Core by default has a 300 Mb mempool. AFAIK running Bitcoin Core requires a machine with at least 4 Gb. I would try extrapolating from there, e.g. 600 Mb for a 8 Gb machine etc.

[unamis76]

Bitcoin Core by default has a 300 Mb mempool.

That's correct, but...

AFAIK running Bitcoin Core requires a machine with at least 4 Gb. I would try extrapolating from there, e.g. 600 Mb for a 8 Gb machine etc.

300Mb mempool takes bitcoind to the limit in a 4GB machine. I've seen it quit unexpectedly even in a 6GB machine, both during the "stress test" periods that happened last year. So I think we definitely cannot extrapolate, especially when the machines in question were only running Ubuntu + bitcoind (0.10 and 0.11 at the time).


A question to fellow node owners: how much RAM do you have, what are you running and what bitcoind version?

[achow101]

300Mb mempool takes bitcoind to the limit in a 4GB machine. I've seen it quit unexpectedly even in a 6GB machine, both during the "stress test" periods that happened last year. So I think we definitely cannot extrapolate, especially when the machines in question were only running Ubuntu + bitcoind (0.10 and 0.11 at the time).

That limit was only recently introduced in Bitcoin Core 0.12 so setting that in 0.10 and 0.11 nodes shouldn't work at all since it didn't even exist. I pretty sure it can be extrapolated, you just need to use 0.12 and simulate stress test conditions.

A question to fellow node owners: how much RAM do you have, what are you running and what bitcoind version?

I have been running a build of the latest master branch for a while now (including when the stress tests were happening). I constantly update it. Sometimes I also have a testnet node running and occasionally a segnet (segwit test network) node running as well. I also have Armory running and when I have testnet up usually Armory Testnet is up as well. These are usually all running simultaneously and I don't see any lags or crashes at all. I use Ubuntu 15.10 with 8 Gb of RAM.

[unamis76]

300Mb mempool takes bitcoind to the limit in a 4GB machine. I've seen it quit unexpectedly even in a 6GB machine, both during the "stress test" periods that happened last year. So I think we definitely cannot extrapolate, especially when the machines in question were only running Ubuntu + bitcoind (0.10 and 0.11 at the time).

That limit was only recently introduced in Bitcoin Core 0.12 so setting that in 0.10 and 0.11 nodes shouldn't work at all since it didn't even exist. I pretty sure it can be extrapolated, you just need to use 0.12 and simulate stress test conditions.

A question to fellow node owners: how much RAM do you have, what are you running and what bitcoind version?

I have been running a build of the latest master branch for a while now (including when the stress tests were happening). I constantly update it. Sometimes I also have a testnet node running and occasionally a segnet (segwit test network) node running as well. I also have Armory running and when I have testnet up usually Armory Testnet is up as well. These are usually all running simultaneously and I don't see any lags or crashes at all. I use Ubuntu 15.10 with 8 Gb of RAM.

All right, thanks for the feedback... I'll be testing. Sometimes it took over a week to get crashes, I'll see what can be done now with 0.12 and I'll try to get a 4 or 6GB host for this.

[shorena]

-snip-
how much RAM do you have, what are you running and what bitcoind version?

VPS #1 - 2GB, limited to the default 300MB that came with 0.12, no crashes since the update (March 1st)[1] Edit: I had constant problems with this node before upgrading to 0.12.
VPS #2 - 6GB, limited to 1500 MB has not reached the limit since the last reboot (March 24th)[2]
Home node - 16GB, limited to 1000 MB, barely runs long enough to fill the available memory. I think I left it running for 3 days once since the last update and it did not use more than 900MB.

All given values are for memory used for TX. I dont track overal memory imprint, but whenever I check htop on the VPS's there is memory left "free" (used for caching). In fact I have trouble getting the 6GB VPS to go over 3 GB used. All run 0.12 64 bit. They VPSs are Ubuntu variants, the home node runs Win7.

@knightdk could it be that you see higher memory use becasue you use test builds with additional logging?

[1] http://213.165.91.169/
[2] http://188.68.53.44/

[unamis76]

-snip-
how much RAM do you have, what are you running and what bitcoind version?

VPS #1 - 2GB, limited to the default 300MB that came with 0.12, no crashes since the update (March 1st)[1] Edit: I had constant problems with this node before upgrading to 0.12.
VPS #2 - 6GB, limited to 1500 MB has not reached the limit since the last reboot (March 24th)[2]
Home node - 16GB, limited to 1000 MB, barely runs long enough to fill the available memory. I think I left it running for 3 days once since the last update and it did not use more than 900MB.

All given values are for memory used for TX. I dont track overal memory imprint, but whenever I check htop on the VPS's there is memory left "free" (used for caching). In fact I have trouble getting the 6GB VPS to go over 3 GB used. All run 0.12 64 bit. They VPSs are Ubuntu variants, the home node runs Win7.

@knightdk could it be that you see higher memory use becasue you use test builds with additional logging?

[1] http://213.165.91.169/
[2] http://188.68.53.44/

Thank you for your insights!

How does one see how much of the mempool is filled? Can't find a command for that on the Bitcoin Core documentation.

I find it odd that you see free memory on your nodes and that your 6GB one does not top 3GB... A few minutes after syncing are enough for me to see all my RAM being consumed regardless of the amount and bitcoind version.

[ranochigo]

-snip-
how much RAM do you have, what are you running and what bitcoind version?

VPS #1 - 2GB, limited to the default 300MB that came with 0.12, no crashes since the update (March 1st)[1] Edit: I had constant problems with this node before upgrading to 0.12.
VPS #2 - 6GB, limited to 1500 MB has not reached the limit since the last reboot (March 24th)[2]
Home node - 16GB, limited to 1000 MB, barely runs long enough to fill the available memory. I think I left it running for 3 days once since the last update and it did not use more than 900MB.

All given values are for memory used for TX. I dont track overal memory imprint, but whenever I check htop on the VPS's there is memory left "free" (used for caching). In fact I have trouble getting the 6GB VPS to go over 3 GB used. All run 0.12 64 bit. They VPSs are Ubuntu variants, the home node runs Win7.

@knightdk could it be that you see higher memory use becasue you use test builds with additional logging?

[1] http://213.165.91.169/
[2] http://188.68.53.44/

Thank you for your insights!

How does one see how much of the mempool is filled? Can't find a command for that on the Bitcoin Core documentation.

I find it odd that you see free memory on your nodes and that your 6GB one does not top 3GB... A few minutes after syncing are enough for me to see all my RAM being consumed regardless of the amount and bitcoind version.

For Bitcoin Core, go to Help>Debug Window>Console and type getmempoolinfo. The "size:" is the amount of TXes in your mempool and the "bytes:" is the size of your mempool in bytes.

[achow101]

@knightdk could it be that you see higher memory use becasue you use test builds with additional logging?

Probably not. The test builds do not have additional logging although sometimes they can have bugs which could have memory leaks. I probably see higher memory usage because I also have txindex enabled but that seems to mostly effect my disk usage.

[shorena]

@knightdk could it be that you see higher memory use becasue you use test builds with additional logging?

Probably not. The test builds do not have additional logging although sometimes they can have bugs which could have memory leaks. I probably see higher memory usage because I also have txindex enabled but that seems to mostly effect my disk usage.

I have enabled txindex now as well, but it still does not use a lot of memory. htop reports ~1.1 G, same as free (w/o cache).

-snip-
I find it odd that you see free memory on your nodes and that your 6GB one does not top 3GB... A few minutes after syncing are enough for me to see all my RAM being consumed regardless of the amount and bitcoind version.

Are you sure you are not looking at cached memory values? The entire memory is used all the time anyway, its just not used all at once. Its a very minimalistic setup though. Its just core, the web server and some scripts in the background that update the pictures and collect the data.