[User wants help cracking wallets but is too stupid to ask politely]

[pbies]

Earlier I was importing several thousands of private keys to Bitcoin Core and it took much time, especially rescan of the blockchain. Few hours, >3 surely.

But now, when I am importing 151M private keys (10 GB file with keys) the import process is now taking 2 days and I have 65% for now.

How badly Bitcoin Core is written is unbelivable. There is no other way to scan such an amount of keys, because other apps are for private, limited use and do not offer scanning the blockchain with such a large amount of keys.

Will this be fixed or even rewritten?

[1713583439]

1713583439

[1713583439]

1713583439

[achow101]

Firstly, just because it takes a while for your crazy edge case does not mean that the software poorly written.

Secondly, an extremely large number of keys is not a common use case, thus the wallet is not written nor optimized for this use case.


Will this be fixed? Maybe. It might not be worthwhile to do so. Rewritten? Not any time soon. Rewriting the wallet is a huge undertaking and can't just be done on a whim. It'll be a long and slow process to rewrite it as changes have to be relatively small and it needs to be done incrementally.


If you're wallet is greater than 10 GB, then you will need to have more RAM than that to not constantly go into swap space. Bitcoin Core loads the entire wallet into memory, so if on disk it's 10 GB, then almost all of that is ending up in RAM. If you don't have enough RAM, it's not going to go very well.


If you don't need the transaction history for your 151 million keys, then you can try using the scantxoutset command. It'll go faster because it isn't scanning the entire blockchain history and then copying it to a separate wallet file.

[pbies]

Thanks for prompt response.

I understood what you have written. I am thinking now that there should be many indexes on database to provide better performance in such edge case. This should speed up process of keys importing.
For example: indexing first parts of the keys so the search process would be by few first characters and it would speed up greatly.

Wallet.dat got to 3 GB and then I stopped that, because progress was moving 1% per few hours.
The import file with private keys was 10 GB. Key per line + date as "0".

I didn't observed that whole wallet was loaded into memory, as Bitcoin Core takes much less memory even if the wallet.dat was 2 GB.

I need the transaction and knowledge about history for all keys, so it is not an option.

[achow101]

I understood what you have written. I am thinking now that there should be many indexes on database to provide better performance in such edge case. This should speed up process of keys importing.
For example: indexing first parts of the keys so the search process would be by few first characters and it would speed up greatly.

Indexes don't help in the wallet right now (and probably won't for the foreseeable future). Indexes would probably slow down the import because most of the time spent during an import is in the writing of the imported thing to the wallet file. With more indexes, there will be more data to be written, so that may take even longer. Bitcoin Core actually never reads data from disk except during the initial loading. It does not use the wallet file like an actual database so there aren't constant reads. Thus indexes don't really help with that.

What probably would help is keeping keys in memory in a hash table (std::unordered_map) rather than a RB tree (std::map). The lookups are all happening in memory, and hash tables are O(1). Currently we use a std::map which is a RB tree, so lookups are O(log(n))

I didn't observed that whole wallet was loaded into memory, as Bitcoin Core takes much less memory even if the wallet.dat was 2 GB.

Hmm. It suppose that the internal BTree structure of the underlying BDB file takes up more space than I thought it did.

[gmaxwell]

pbies, please revise your message to not be acutely nasty.

[NotATether]

Earlier I was importing several thousands of private keys to Bitcoin Core and it took much time, especially rescan of the blockchain. Few hours, >3 surely.

But now, when I am importing 151M private keys (10 GB file with keys) the import process is now taking 2 days and I have 65% for now.

Why are you trying to import that many private keys at once? Can you explain if there's a reason you need millions of keys in a single wallet, and we might try to see if there is an alternative you can do?

[RHavar]

Kind of funny, I always thought bitcoin core's wallet did rather well with large amounts of addresses. I tried probably over 5 wallets that supported importing private keys, and most of them shit themselves at 1-2k addresses. While I didn't have any real problem with a few million in bitcoin core. Can't say I ever tried 150 million keys though...


It's hard to tell what you're doing, but .. I guess you're doing some bitcoin service type thing. Core's wallet is really not designed for sort of heavy commercial use, but all the limitations can be worked around (with a bit of effort).

Some things I did, and would strongly recommend for a hardcore service:


a) Store addresses in your own database. Scan blocks (and optionally the mempool) manually, and track user balances/deposits outside of bitcoin core
b) Only import addresses/funds after the address has been used. e.g. if a user requests an address, only add it to bitcoin core after actually been used
c) If you're using bitcoin core's send (well coin selection), try have a strategy for dealing with dust (either locking it, or not importing it or something). I've seen people (accidentally) lose thousands of dollars cause they have been dusted with hundreds of utxos and their wallets coin selection then spent it during high fee times
d) bitcoin core's wallet gets very slow (O(n)) when you have a lot of transactions in your wallet. If you're a busy service, you'll find any `listunspent` (or something that depends on that) is taking 30+ seconds, which can be pretty problematic if you want to instantly send people funds. You can solve this by writing a script that calls `removeprunedfunds` to delete transactions your wallet doesn't need to care about any more. You have to be very very careful about this, because it's extremely easy to corrupt your wallet state if you delete the wrong thing, or delete in the wrong order. Most people I talked to solve it by cycling wallets (like create a new wallet every few months, and import everything still relevant they need from the last one. But I find that a little inelegant.

[NeuroticFish]

Rather than importing all private keys at once before obtaining information (transaction, etc.) you need, why don't you perform the action in smaller batch (e.g. import 1000 keys private, get all information, remove those 1000 private keys, repeat until process is done)?

Or install/use a side software, a block explorer, which will probably index all the data different into a new database and which then he can query as he likes.

[BTCW]

Feel your pain, man. I'm trying to import 2^256 private keys, and it takes bloody ages -- when will THEY fix it?!

(Disclaimer: irony)

[CryptoSh1va]

Earlier I was importing several thousands of private keys to Bitcoin Core and it took much time, especially rescan of the blockchain. Few hours, >3 surely.

But now, when I am importing 151M private keys (10 GB file with keys) the import process is now taking 2 days and I have 65% for now.

Why do you need this, are you trying to check the balances/trx of these addresses?
You can write a script that checks the blockchain data throug API services of several btc projects.
They are mostly output in json format.

[RHavar]

Why do you need this, are you trying to check the balances/trx of these addresses?
You can write a script that checks the blockchain data throug API services of several btc projects.

I've found for serious uses, those APIs generally create more problems than they solve. You've got to deal with (malicious) wrong data, usage restrictions, privacy concerns, up time problems.

They're definitely useful for some ad-hoc shit though. But for an actual commercial service, I think the only API service I can recommend in good-faith (in special circumstances) would be blockcypher's confidence stuff for some 0-conf transactions. They seem to do a good job on that, and the data is not easy to get yourself. [I wouldn't recommend their other APIs though, but I don't want to digress too much]

[David Dagher]

But now, when I am importing 151M private keys (10 GB file with keys) the import process is now taking 2 days and I have 65% for now.

I have generated like 100M WIFs using a python code but I don't know how to import them  
Electrum freezes and I'm doing my best to avoid downloading the whole blockchain  
So, Would you mind me asking what are you using to import such a huge number?
And if It is going to import them into my wallet, will it sync automatically while importing? or it will only import them and sync may take longer time?

I'm totally newbie, so it will be appreciated if you explained to me step by step.

Thank you in advance  

[HCP]

You can import them by repeatedly calling the importprivkey command via "bitcoin-cli" (or the console in Bitcoin Core GUI)... note, that to prevent it doing a very long "rescan" after every import, you need to set the rescan parameter to "false", until the last key you want to import...

Format for the importprivkey command is:

Code:

importprivkey "privkey" ( "label" rescan )


Arguments:
1. privkey    (string, required) The private key (see dumpprivkey)
2. label      (string, optional, default=current label if address exists, otherwise "") An optional label
3. rescan     (boolean, optional, default=true) Rescan the wallet for transactions

So, you'd use something like:

Code:

importprivkey "5Kb8kLf9zgWQnogidDA76MzPL6TsZZY36hWXMssSzNydYXYB9KF" "" false

Should be relatively simple to code a script which would just read a file that contains a list of private keys and execute the bitcoin-cli command for you... and then on the last one, trigger a rescan by using:

Code:

importprivkey "WIF_PRIVATE_KEY_GOES_HERE" "" true