Title: Bitcoin Core 0.15 multiwallet limit Post by: kobik on February 09, 2018, 10:47:20 AM In the bitcoin.conf we can add multiple entries of:
Code: wallet=wallet.dat Is there currently a limit to how many wallets I can load at once? Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: achow101 on February 09, 2018, 03:54:59 PM There's technically no limit, but of course you do have memory limits, so at some point you will run out of memory if you load too many wallets.
Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: kobik on February 10, 2018, 09:20:00 AM There's technically no limit, but of course you do have memory limits, so at some point you will run out of memory if you load too many wallets. @achow101 Thanks, How are wallets actually loaded into memory? if the wallet.dat file is ~1.5M will the entire file be loaded into process memory (will that take ~1.5M in memory)? Also, do you know if the new 0.16.0 allow creating new wallets on the fly? https://bitcoincore.org/bin/bitcoin-core-0.16.0/test.rc3/ Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: achow101 on February 10, 2018, 05:43:21 PM @achow101 Thanks, How are wallets actually loaded into memory? if the wallet.dat file is ~1.5M will the entire file be loaded into process memory (will that take ~1.5M in memory)? All of the wallet data is read off disk and loaded into memory at start. So if your wallet is ~1.5 MB, then it should consume ~1.5 MB of memory.Also, do you know if the new 0.16.0 allow creating new wallets on the fly? It won't.https://bitcoincore.org/bin/bitcoin-core-0.16.0/test.rc3/ Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: kobik on February 11, 2018, 02:17:11 PM I have tried to load 350 wallets by adding the entries to the bicoin.conf
Code: wallet=wallet-1.dat And it constantly crashed with fatal error after the 333 wallet in the stage "load wallet.." (although all of them are first verified ok) At this point the memory usage is about ~750 MB I have enough available memory. debug.log Code: 2018-02-11 14:23:56 init message: Loading wallet... db.log Code: Lock table is out of available locker entries Any way to workaround this? ??? Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: bob123 on February 11, 2018, 02:50:50 PM And it constantly crashed with fatal error after the 323 wallet in the stage "load wallet.." (although all of them are first verified ok) At this point the memory usage is about ~750 MB I have enough available memory. debug.log Code: 2018-02-11 14:23:56 init message: Loading wallet... Runaway exception might be caused because the wallet you try to open (wallet-333.dat) is not a HD wallet? Are you able to open this wallet.dat individually? If not, you may try to start core with this command: -usehd=0 Does the same issue also appear when replacing this wallet file with another (working) one? Or does it always crash after 323 loaded wallets? Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: kobik on February 11, 2018, 03:05:34 PM It always crashes in the 333 wallet. (after loading 332 wallets successfully).
All wallets were auto created by Bitcoin Core. The wallet file is fine. if I put it as the first wallet in the list, no problems. but then the last 333 wallet crashes. The correct number is 333 (not 323, sorry) Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: achow101 on February 11, 2018, 05:54:29 PM db.log This error is because each wallet.dat file is opened in the same BDB environment. The entire environment has a limited number of database locks, and it seems like 333 wallets hits that number of database locks. Of course, if you are actually receiving transactions, I think you will run into more problems even with 332 wallets since database locks are required in order to write data to the wallets.Code: Lock table is out of available locker entries Any way to workaround this? ??? In the Bitcoin Core data directory, create a file named DB_CONFIG. Put the following lines in that file:Code: set_lk_max_locks 400000 Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: kobik on February 12, 2018, 07:59:51 AM In the Bitcoin Core data directory, create a file named DB_CONFIG. Put the following lines in that file: Code: set_lk_max_locks 400000 I have created DB_CONFIG with the lines you suggested. I placed the file in the root of the Bitcoin Data folder. The same happens. :-\ Code: Lock table is out of available locker entries https://imgur.com/a/29Zsa It crashes on the 333 wallet. Did I miss anything? what can be done? EDIT: I have tested both in 0.15.1 and 0.16.0 (main and testnet). The result is the same. 333 is the number of the beast. Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: achow101 on February 12, 2018, 11:17:14 PM I have created DB_CONFIG with the lines you suggested. I placed the file in the root of the Bitcoin Data folder. The same happens. :-\ Code: Lock table is out of available locker entries https://imgur.com/a/29Zsa It crashes on the 333 wallet. Did I miss anything? what can be done? EDIT: I have tested both in 0.15.1 and 0.16.0 (main and testnet). The result is the same. 333 is the number of the beast. Try adding this line to DB_CONFIG Code: set_lg_dir database Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: kobik on February 13, 2018, 06:24:28 AM Hmm. Try adding this line to DB_CONFIG Code: set_lg_dir database I have already tried this. but the same results. As if this file is not read at all (?). Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: kobik on February 13, 2018, 02:51:47 PM Using Process Monitor I verified that the DB_CONFIG is actually read by bitcoind.
If any one willing to try it out, here is the list of wallets you can drop into your bitcoin.conf: to reproduce https://pastebin.com/8wHD1tL3 (https://pastebin.com/8wHD1tL3) EDIT: From the Berkeley DB documentation: https://docs.oracle.com/cd/E17276_01/html/api_reference/C/set_lk_max_locks_parameter.html Quote If the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment. What to make of this? is it impossible to change this parameter b/c the database environment already exists? or did I miss-read? Title: Re: Bitcoin Core 0.15 multiwallet limit Post by: kobik on February 13, 2018, 03:46:41 PM Found it!
adding set_lk_max_lockers (not set_lk_max_locks) did the trick: Quote set_lk_max_lockers 400000 https://docs.oracle.com/cd/E17276_01/html/api_reference/C/set_lk_max_lockers_parameter.html (https://docs.oracle.com/cd/E17276_01/html/api_reference/C/set_lk_max_lockers_parameter.html) I tested and was able to auto-create and load 932 wallets, until I hit an out of memory error in the db.log: Code: Logging region out of memory; you may need to increase its size Which Code: set_lg_regionmax 1048576 Can resolve. |