Bitcoin Forum
October 20, 2018, 10:09:14 AM *
News: Make sure you are not using versions of Bitcoin Core other than 0.17.0 [Torrent], 0.16.3, 0.15.2, or 0.14.3. More info.
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Bitcoin Core 0.15 multiwallet limit  (Read 117 times)
kobik
Jr. Member
*
Offline Offline

Activity: 39
Merit: 3


View Profile
February 09, 2018, 10:47:20 AM
 #1

In the bitcoin.conf we can add multiple entries of:

Code:
wallet=wallet.dat
wallet=wallet-2.dat
wallet=wallet-3.dat
wallet=wallet-4.dat
...

Is there currently a limit to how many wallets I can load at once?

1540030154
Hero Member
*
Offline Offline

Posts: 1540030154

View Profile Personal Message (Offline)

Ignore
1540030154
Reply with quote  #2

1540030154
Report to moderator
1540030154
Hero Member
*
Offline Offline

Posts: 1540030154

View Profile Personal Message (Offline)

Ignore
1540030154
Reply with quote  #2

1540030154
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1540030154
Hero Member
*
Offline Offline

Posts: 1540030154

View Profile Personal Message (Offline)

Ignore
1540030154
Reply with quote  #2

1540030154
Report to moderator
achow101
Moderator
Legendary
*
Offline Offline

Activity: 1554
Merit: 1704


3F1Y9yquzvY6RWvKbw2n2zeo9V5mvBhADU


View Profile WWW
February 09, 2018, 03:54:59 PM
 #2

There's technically no limit, but of course you do have memory limits, so at some point you will run out of memory if you load too many wallets.

kobik
Jr. Member
*
Offline Offline

Activity: 39
Merit: 3


View Profile
February 10, 2018, 09:20:00 AM
 #3

There's technically no limit, but of course you do have memory limits, so at some point you will run out of memory if you load too many wallets.

@achow101 Thanks, How are wallets actually loaded into memory? if the wallet.dat file is ~1.5M will the entire file be loaded into process memory (will that take ~1.5M in memory)?

Also, do you know if the new 0.16.0 allow creating new wallets on the fly?
https://bitcoincore.org/bin/bitcoin-core-0.16.0/test.rc3/
achow101
Moderator
Legendary
*
Offline Offline

Activity: 1554
Merit: 1704


3F1Y9yquzvY6RWvKbw2n2zeo9V5mvBhADU


View Profile WWW
February 10, 2018, 05:43:21 PM
 #4

@achow101 Thanks, How are wallets actually loaded into memory? if the wallet.dat file is ~1.5M will the entire file be loaded into process memory (will that take ~1.5M in memory)?
All of the wallet data is read off disk and loaded into memory at start. So if your wallet is ~1.5 MB, then it should consume ~1.5 MB of memory.

Also, do you know if the new 0.16.0 allow creating new wallets on the fly?
https://bitcoincore.org/bin/bitcoin-core-0.16.0/test.rc3/
It won't.

kobik
Jr. Member
*
Offline Offline

Activity: 39
Merit: 3


View Profile
February 11, 2018, 02:17:11 PM
 #5

I have tried to load 350 wallets by adding the entries to the bicoin.conf

Code:
wallet=wallet-1.dat
wallet=wallet-2.dat
wallet=wallet-3.dat
wallet=wallet-4.dat
...
wallet=wallet-350.dat

And it constantly crashed with fatal error after the 333 wallet in the stage "load wallet.." (although all of them are first verified ok)
At this point the memory usage is about ~750 MB
I have enough available memory.

debug.log
Code:
2018-02-11 14:23:56 init message: Loading wallet...
2018-02-11 14:23:56 nFileVersion = 150100
2018-02-11 14:23:56 Keys: 2002 plaintext, 0 encrypted, 2002 w/ metadata, 2002 total
2018-02-11 14:23:56  wallet                   47ms
2018-02-11 14:23:56 setKeyPool.size() = 2000
2018-02-11 14:23:56 mapWallet.size() = 0
2018-02-11 14:23:56 mapAddressBook.size() = 1
2018-02-11 14:23:56 init message: Loading wallet...
2018-02-11 14:23:56

************************
EXCEPTION: St13runtime_error      
CDB: Error 12, can't open database wallet-333.dat      
D:\Bitcoin\bitcoin-qt.exe in Runaway exception

db.log
Code:
Lock table is out of available locker entries

Any way to workaround this?  Huh
bob123
Hero Member
*****
Offline Offline

Activity: 714
Merit: 600



View Profile WWW
February 11, 2018, 02:50:50 PM
 #6

And it constantly crashed with fatal error after the 323 wallet in the stage "load wallet.." (although all of them are first verified ok)
At this point the memory usage is about ~750 MB
I have enough available memory.

debug.log
Code:
2018-02-11 14:23:56 init message: Loading wallet...
2018-02-11 14:23:56 nFileVersion = 150100
2018-02-11 14:23:56 Keys: 2002 plaintext, 0 encrypted, 2002 w/ metadata, 2002 total
2018-02-11 14:23:56  wallet                   47ms
2018-02-11 14:23:56 setKeyPool.size() = 2000
2018-02-11 14:23:56 mapWallet.size() = 0
2018-02-11 14:23:56 mapAddressBook.size() = 1
2018-02-11 14:23:56 init message: Loading wallet...
2018-02-11 14:23:56

************************
EXCEPTION: St13runtime_error      
CDB: Error 12, can't open database wallet-333.dat      
D:\Bitcoin\bitcoin-qt.exe in Runaway exception


Runaway exception might be caused because the wallet you try to open (wallet-333.dat) is not a HD wallet?
Are you able to open this wallet.dat individually? If not, you may try to start core with this command: -usehd=0

Does the same issue also appear when replacing this wallet file with another (working) one?
Or does it always crash after 323 loaded wallets?



kobik
Jr. Member
*
Offline Offline

Activity: 39
Merit: 3


View Profile
February 11, 2018, 03:05:34 PM
 #7

It always crashes in the 333 wallet. (after loading 332 wallets successfully).
All wallets were auto created by Bitcoin Core.
The wallet file is fine. if I put it as the first wallet in the list, no problems. but then the last 333 wallet crashes.


The correct number is 333 (not  323, sorry)
achow101
Moderator
Legendary
*
Offline Offline

Activity: 1554
Merit: 1704


3F1Y9yquzvY6RWvKbw2n2zeo9V5mvBhADU


View Profile WWW
February 11, 2018, 05:54:29 PM
 #8

db.log
Code:
Lock table is out of available locker entries
This error is because each wallet.dat file is opened in the same BDB environment. The entire environment has a limited number of database locks, and it seems like 333 wallets hits that number of database locks. Of course, if you are actually receiving transactions, I think you will run into more problems even with 332 wallets since database locks are required in order to write data to the wallets.

Any way to workaround this?  Huh
In the Bitcoin Core data directory, create a file named DB_CONFIG. Put the following lines in that file:
Code:
set_lk_max_locks 400000
set_lk_max_objects 400000
That will increase the number of available locks by tenfold (was 40000, no 400000) which will let you open 10 times more wallets (assuming you have enough memory), in theory. Once you run out of locks again, just keep increasing those two numbers.

kobik
Jr. Member
*
Offline Offline

Activity: 39
Merit: 3


View Profile
February 12, 2018, 07:59:51 AM
 #9

In the Bitcoin Core data directory, create a file named DB_CONFIG. Put the following lines in that file:
Code:
set_lk_max_locks 400000
set_lk_max_objects 400000
That will increase the number of available locks by tenfold (was 40000, no 400000) which will let you open 10 times more wallets (assuming you have enough memory), in theory. Once you run out of locks again, just keep increasing those two numbers.

I have created DB_CONFIG with the lines you suggested. I placed the file in the root of the Bitcoin Data folder.
The same happens.  Undecided

Code:
Lock table is out of available locker entries



It crashes on the 333 wallet. Did I miss anything? what can be done?

EDIT: I have tested both in 0.15.1 and 0.16.0 (main and testnet). The result is the same. 333 is the number of the beast.

achow101
Moderator
Legendary
*
Offline Offline

Activity: 1554
Merit: 1704


3F1Y9yquzvY6RWvKbw2n2zeo9V5mvBhADU


View Profile WWW
February 12, 2018, 11:17:14 PM
 #10


I have created DB_CONFIG with the lines you suggested. I placed the file in the root of the Bitcoin Data folder.
The same happens.  Undecided

Code:
Lock table is out of available locker entries



It crashes on the 333 wallet. Did I miss anything? what can be done?

EDIT: I have tested both in 0.15.1 and 0.16.0 (main and testnet). The result is the same. 333 is the number of the beast.
Hmm.

Try adding this line to DB_CONFIG

Code:
set_lg_dir database

kobik
Jr. Member
*
Offline Offline

Activity: 39
Merit: 3


View Profile
February 13, 2018, 06:24:28 AM
 #11

Hmm.

Try adding this line to DB_CONFIG

Code:
set_lg_dir database

I have already tried this. but the same results. As if this file is not read at all (?).
kobik
Jr. Member
*
Offline Offline

Activity: 39
Merit: 3


View Profile
February 13, 2018, 02:51:47 PM
 #12

Using Process Monitor I verified that the DB_CONFIG is actually read by bitcoind.
If any one willing to try it out, here is the list of wallets you can drop into your bitcoin.conf: to reproduce

https://pastebin.com/8wHD1tL3

EDIT:

From the Berkeley DB  documentation: https://docs.oracle.com/cd/E17276_01/html/api_reference/C/set_lk_max_locks_parameter.html

Quote
If the database environment already exists when this parameter is changed, it is ignored. To change this value after the environment has been created, re-create your environment.

What to make of this? is it impossible to change this parameter b/c the database environment already exists? or did I miss-read?
kobik
Jr. Member
*
Offline Offline

Activity: 39
Merit: 3


View Profile
February 13, 2018, 03:46:41 PM
Merited by achow101 (1)
 #13

Found it!

adding set_lk_max_lockers (not set_lk_max_locks) did the trick:

Quote
set_lk_max_lockers 400000

https://docs.oracle.com/cd/E17276_01/html/api_reference/C/set_lk_max_lockers_parameter.html


I tested and was able to auto-create and load 932 wallets, until I hit an out of memory error in the db.log:

Code:
   Logging region out of memory; you may need to increase its size
    DB_TXN->abort: log undo failed for LSN: 227 171878: Not enough space
    PANIC: Not enough space
    PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
    PANIC: fatal region error detected; run recovery

Which

Code:
set_lg_regionmax 1048576

Can resolve.
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!