kobik (OP)
Jr. Member
Offline
Activity: 39
Merit: 3
|
|
February 09, 2018, 10:47:20 AM |
|
In the bitcoin.conf we can add multiple entries of: wallet=wallet.dat wallet=wallet-2.dat wallet=wallet-3.dat wallet=wallet-4.dat ...
Is there currently a limit to how many wallets I can load at once?
|
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3500
Merit: 6833
Just writing some code
|
|
February 09, 2018, 03:54:59 PM |
|
There's technically no limit, but of course you do have memory limits, so at some point you will run out of memory if you load too many wallets.
|
|
|
|
kobik (OP)
Jr. Member
Offline
Activity: 39
Merit: 3
|
|
February 10, 2018, 09:20:00 AM |
|
There's technically no limit, but of course you do have memory limits, so at some point you will run out of memory if you load too many wallets.
@achow101 Thanks, How are wallets actually loaded into memory? if the wallet.dat file is ~1.5M will the entire file be loaded into process memory (will that take ~1.5M in memory)? Also, do you know if the new 0.16.0 allow creating new wallets on the fly? https://bitcoincore.org/bin/bitcoin-core-0.16.0/test.rc3/
|
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3500
Merit: 6833
Just writing some code
|
|
February 10, 2018, 05:43:21 PM |
|
@achow101 Thanks, How are wallets actually loaded into memory? if the wallet.dat file is ~1.5M will the entire file be loaded into process memory (will that take ~1.5M in memory)?
All of the wallet data is read off disk and loaded into memory at start. So if your wallet is ~1.5 MB, then it should consume ~1.5 MB of memory. It won't.
|
|
|
|
kobik (OP)
Jr. Member
Offline
Activity: 39
Merit: 3
|
|
February 11, 2018, 02:17:11 PM Last edit: February 11, 2018, 03:12:10 PM by kobik |
|
I have tried to load 350 wallets by adding the entries to the bicoin.confwallet=wallet-1.dat wallet=wallet-2.dat wallet=wallet-3.dat wallet=wallet-4.dat ... wallet=wallet-350.dat And it constantly crashed with fatal error after the 333 wallet in the stage "load wallet.." (although all of them are first verified ok) At this point the memory usage is about ~750 MB I have enough available memory. debug.log 2018-02-11 14:23:56 init message: Loading wallet... 2018-02-11 14:23:56 nFileVersion = 150100 2018-02-11 14:23:56 Keys: 2002 plaintext, 0 encrypted, 2002 w/ metadata, 2002 total 2018-02-11 14:23:56 wallet 47ms 2018-02-11 14:23:56 setKeyPool.size() = 2000 2018-02-11 14:23:56 mapWallet.size() = 0 2018-02-11 14:23:56 mapAddressBook.size() = 1 2018-02-11 14:23:56 init message: Loading wallet... 2018-02-11 14:23:56
************************ EXCEPTION: St13runtime_error CDB: Error 12, can't open database wallet-333.dat D:\Bitcoin\bitcoin-qt.exe in Runaway exception db.log Lock table is out of available locker entries Any way to workaround this?
|
|
|
|
bob123
Legendary
Offline
Activity: 1624
Merit: 2481
|
|
February 11, 2018, 02:50:50 PM |
|
And it constantly crashed with fatal error after the 323 wallet in the stage "load wallet.." (although all of them are first verified ok) At this point the memory usage is about ~750 MB I have enough available memory. debug.log 2018-02-11 14:23:56 init message: Loading wallet... 2018-02-11 14:23:56 nFileVersion = 150100 2018-02-11 14:23:56 Keys: 2002 plaintext, 0 encrypted, 2002 w/ metadata, 2002 total 2018-02-11 14:23:56 wallet 47ms 2018-02-11 14:23:56 setKeyPool.size() = 2000 2018-02-11 14:23:56 mapWallet.size() = 0 2018-02-11 14:23:56 mapAddressBook.size() = 1 2018-02-11 14:23:56 init message: Loading wallet... 2018-02-11 14:23:56
************************ EXCEPTION: St13runtime_error CDB: Error 12, can't open database wallet-333.dat D:\Bitcoin\bitcoin-qt.exe in Runaway exception Runaway exception might be caused because the wallet you try to open (wallet-333.dat) is not a HD wallet? Are you able to open this wallet.dat individually? If not, you may try to start core with this command: -usehd=0Does the same issue also appear when replacing this wallet file with another (working) one? Or does it always crash after 323 loaded wallets?
|
|
|
|
kobik (OP)
Jr. Member
Offline
Activity: 39
Merit: 3
|
|
February 11, 2018, 03:05:34 PM Last edit: February 11, 2018, 04:51:38 PM by kobik |
|
It always crashes in the 333 wallet. (after loading 332 wallets successfully). All wallets were auto created by Bitcoin Core. The wallet file is fine. if I put it as the first wallet in the list, no problems. but then the last 333 wallet crashes.
The correct number is 333 (not 323, sorry)
|
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3500
Merit: 6833
Just writing some code
|
|
February 11, 2018, 05:54:29 PM |
|
db.log Lock table is out of available locker entries This error is because each wallet.dat file is opened in the same BDB environment. The entire environment has a limited number of database locks, and it seems like 333 wallets hits that number of database locks. Of course, if you are actually receiving transactions, I think you will run into more problems even with 332 wallets since database locks are required in order to write data to the wallets. Any way to workaround this? In the Bitcoin Core data directory, create a file named DB_CONFIG. Put the following lines in that file: set_lk_max_locks 400000 set_lk_max_objects 400000
That will increase the number of available locks by tenfold (was 40000, no 400000) which will let you open 10 times more wallets (assuming you have enough memory), in theory. Once you run out of locks again, just keep increasing those two numbers.
|
|
|
|
kobik (OP)
Jr. Member
Offline
Activity: 39
Merit: 3
|
|
February 12, 2018, 07:59:51 AM Last edit: February 12, 2018, 01:06:25 PM by kobik |
|
In the Bitcoin Core data directory, create a file named DB_CONFIG. Put the following lines in that file: set_lk_max_locks 400000 set_lk_max_objects 400000
That will increase the number of available locks by tenfold (was 40000, no 400000) which will let you open 10 times more wallets (assuming you have enough memory), in theory. Once you run out of locks again, just keep increasing those two numbers. I have created DB_CONFIG with the lines you suggested. I placed the file in the root of the Bitcoin Data folder. The same happens. Lock table is out of available locker entries It crashes on the 333 wallet. Did I miss anything? what can be done? EDIT: I have tested both in 0.15.1 and 0.16.0 (main and testnet). The result is the same. 333 is the number of the beast.
|
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3500
Merit: 6833
Just writing some code
|
|
February 12, 2018, 11:17:14 PM |
|
I have created DB_CONFIG with the lines you suggested. I placed the file in the root of the Bitcoin Data folder. The same happens. Lock table is out of available locker entries It crashes on the 333 wallet. Did I miss anything? what can be done? EDIT: I have tested both in 0.15.1 and 0.16.0 (main and testnet). The result is the same. 333 is the number of the beast. Hmm. Try adding this line to DB_CONFIG
|
|
|
|
kobik (OP)
Jr. Member
Offline
Activity: 39
Merit: 3
|
|
February 13, 2018, 06:24:28 AM |
|
Hmm. Try adding this line to DB_CONFIG I have already tried this. but the same results. As if this file is not read at all (?).
|
|
|
|
kobik (OP)
Jr. Member
Offline
Activity: 39
Merit: 3
|
|
February 13, 2018, 02:51:47 PM Last edit: February 13, 2018, 03:20:12 PM by kobik |
|
Using Process Monitor I verified that the DB_CONFIG is actually read by bitcoind. If any one willing to try it out, here is the list of wallets you can drop into your bitcoin.conf: to reproduce https://pastebin.com/8wHD1tL3EDIT: From the Berkeley DB documentation: https://docs.oracle.com/cd/E17276_01/html/api_reference/C/set_lk_max_locks_parameter.htmlIf the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment. What to make of this? is it impossible to change this parameter b/c the database environment already exists? or did I miss-read?
|
|
|
|
kobik (OP)
Jr. Member
Offline
Activity: 39
Merit: 3
|
|
February 13, 2018, 03:46:41 PM Last edit: February 13, 2018, 04:21:05 PM by kobik |
|
Found it! adding set_lk_max_lockers (not set_lk_max_locks) did the trick: set_lk_max_lockers 400000 https://docs.oracle.com/cd/E17276_01/html/api_reference/C/set_lk_max_lockers_parameter.htmlI tested and was able to auto-create and load 932 wallets, until I hit an out of memory error in the db.log: Logging region out of memory; you may need to increase its size DB_TXN->abort: log undo failed for LSN: 227 171878: Not enough space PANIC: Not enough space PANIC: DB_RUNRECOVERY: Fatal error, run database recovery PANIC: fatal region error detected; run recovery Which Can resolve.
|
|
|
|
|