Bitcoin Core 0.15 multiwallet limit

[kobik]

In the bitcoin.conf we can add multiple entries of:

Code:

wallet=wallet.dat
wallet=wallet-2.dat
wallet=wallet-3.dat
wallet=wallet-4.dat
...

Is there currently a limit to how many wallets I can load at once?

[1714776172]

1714776172

[1714776172]

1714776172

[1714776172]

1714776172

[1714776172]

1714776172

[1714776172]

1714776172

[achow101]

There's technically no limit, but of course you do have memory limits, so at some point you will run out of memory if you load too many wallets.

[kobik]

There's technically no limit, but of course you do have memory limits, so at some point you will run out of memory if you load too many wallets.

@achow101 Thanks, How are wallets actually loaded into memory? if the wallet.dat file is ~1.5M will the entire file be loaded into process memory (will that take ~1.5M in memory)?

Also, do you know if the new 0.16.0 allow creating new wallets on the fly?
https://bitcoincore.org/bin/bitcoin-core-0.16.0/test.rc3/

[achow101]

@achow101 Thanks, How are wallets actually loaded into memory? if the wallet.dat file is ~1.5M will the entire file be loaded into process memory (will that take ~1.5M in memory)?

All of the wallet data is read off disk and loaded into memory at start. So if your wallet is ~1.5 MB, then it should consume ~1.5 MB of memory.

Also, do you know if the new 0.16.0 allow creating new wallets on the fly?
https://bitcoincore.org/bin/bitcoin-core-0.16.0/test.rc3/

It won't.

[kobik]

I have tried to load 350 wallets by adding the entries to the bicoin.conf

Code:

wallet=wallet-1.dat
wallet=wallet-2.dat
wallet=wallet-3.dat
wallet=wallet-4.dat
...
wallet=wallet-350.dat

And it constantly crashed with fatal error after the 333 wallet in the stage "load wallet.." (although all of them are first verified ok)
At this point the memory usage is about ~750 MB
I have enough available memory.

debug.log

Code:

2018-02-11 14:23:56 init message: Loading wallet...
2018-02-11 14:23:56 nFileVersion = 150100
2018-02-11 14:23:56 Keys: 2002 plaintext, 0 encrypted, 2002 w/ metadata, 2002 total
2018-02-11 14:23:56  wallet                   47ms
2018-02-11 14:23:56 setKeyPool.size() = 2000
2018-02-11 14:23:56 mapWallet.size() = 0
2018-02-11 14:23:56 mapAddressBook.size() = 1
2018-02-11 14:23:56 init message: Loading wallet...
2018-02-11 14:23:56 

************************
EXCEPTION: St13runtime_error       
CDB: Error 12, can't open database wallet-333.dat       
D:\Bitcoin\bitcoin-qt.exe in Runaway exception 

db.log

Code:

Lock table is out of available locker entries

Any way to workaround this?  

[bob123]

And it constantly crashed with fatal error after the 323 wallet in the stage "load wallet.." (although all of them are first verified ok)
At this point the memory usage is about ~750 MB
I have enough available memory.

debug.log

Code:

2018-02-11 14:23:56 init message: Loading wallet...
2018-02-11 14:23:56 nFileVersion = 150100
2018-02-11 14:23:56 Keys: 2002 plaintext, 0 encrypted, 2002 w/ metadata, 2002 total
2018-02-11 14:23:56  wallet                   47ms
2018-02-11 14:23:56 setKeyPool.size() = 2000
2018-02-11 14:23:56 mapWallet.size() = 0
2018-02-11 14:23:56 mapAddressBook.size() = 1
2018-02-11 14:23:56 init message: Loading wallet...
2018-02-11 14:23:56 

************************
EXCEPTION: St13runtime_error       
CDB: Error 12, can't open database wallet-333.dat       
D:\Bitcoin\bitcoin-qt.exe in Runaway exception 

Runaway exception might be caused because the wallet you try to open (wallet-333.dat) is not a HD wallet?
Are you able to open this wallet.dat individually? If not, you may try to start core with this command: -usehd=0

Does the same issue also appear when replacing this wallet file with another (working) one?
Or does it always crash after 323 loaded wallets?

[kobik]

It always crashes in the 333 wallet. (after loading 332 wallets successfully).
All wallets were auto created by Bitcoin Core.
The wallet file is fine. if I put it as the first wallet in the list, no problems. but then the last 333 wallet crashes.


The correct number is 333 (not  323, sorry)

[achow101]

db.log

Code:

Lock table is out of available locker entries

This error is because each wallet.dat file is opened in the same BDB environment. The entire environment has a limited number of database locks, and it seems like 333 wallets hits that number of database locks. Of course, if you are actually receiving transactions, I think you will run into more problems even with 332 wallets since database locks are required in order to write data to the wallets.

Any way to workaround this?  

In the Bitcoin Core data directory, create a file named DB_CONFIG. Put the following lines in that file:

Code:

set_lk_max_locks 400000
set_lk_max_objects 400000

That will increase the number of available locks by tenfold (was 40000, no 400000) which will let you open 10 times more wallets (assuming you have enough memory), in theory. Once you run out of locks again, just keep increasing those two numbers.

[kobik]

In the Bitcoin Core data directory, create a file named DB_CONFIG. Put the following lines in that file:

Code:

set_lk_max_locks 400000
set_lk_max_objects 400000

That will increase the number of available locks by tenfold (was 40000, no 400000) which will let you open 10 times more wallets (assuming you have enough memory), in theory. Once you run out of locks again, just keep increasing those two numbers.

I have created DB_CONFIG with the lines you suggested. I placed the file in the root of the Bitcoin Data folder.
The same happens.  

Code:

Lock table is out of available locker entries

It crashes on the 333 wallet. Did I miss anything? what can be done?

EDIT: I have tested both in 0.15.1 and 0.16.0 (main and testnet). The result is the same. 333 is the number of the beast.

[achow101]

I have created DB_CONFIG with the lines you suggested. I placed the file in the root of the Bitcoin Data folder.
The same happens.  

Code:

Lock table is out of available locker entries

It crashes on the 333 wallet. Did I miss anything? what can be done?

EDIT: I have tested both in 0.15.1 and 0.16.0 (main and testnet). The result is the same. 333 is the number of the beast.

Hmm.

Try adding this line to DB_CONFIG

Code:

set_lg_dir database

[kobik]

Hmm.

Try adding this line to DB_CONFIG

Code:

set_lg_dir database

I have already tried this. but the same results. As if this file is not read at all (?).

[kobik]

Using Process Monitor I verified that the DB_CONFIG is actually read by bitcoind.
If any one willing to try it out, here is the list of wallets you can drop into your bitcoin.conf: to reproduce

https://pastebin.com/8wHD1tL3

EDIT:

From the Berkeley DB  documentation: https://docs.oracle.com/cd/E17276_01/html/api_reference/C/set_lk_max_locks_parameter.html

If the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment.

What to make of this? is it impossible to change this parameter b/c the database environment already exists? or did I miss-read?

[kobik]

Found it!

adding set_lk_max_lockers (not set_lk_max_locks) did the trick:

set_lk_max_lockers 400000

https://docs.oracle.com/cd/E17276_01/html/api_reference/C/set_lk_max_lockers_parameter.html


I tested and was able to auto-create and load 932 wallets, until I hit an out of memory error in the db.log:

Code:

    Logging region out of memory; you may need to increase its size
    DB_TXN->abort: log undo failed for LSN: 227 171878: Not enough space
    PANIC: Not enough space
    PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
    PANIC: fatal region error detected; run recovery

Which

Code:

set_lg_regionmax 1048576

Can resolve.