Only 2 keys/sec added with importprivkey ?

[PoorGirl]

How comes its so slow to add private keys using importprivkey ?

On a Intel Dual Core 1.8Ghz machine where bitcoind is stored on a big ramdisk the total speed is approx 2 keys/second.
CPU is idle at 0-1%, means there is massive time wasted for nothing.

Is it the RPC interface that's so slow ? If the database would work hard it would show in CPU usage. But there is none. To calc the public hashes/addresses etc also is not the limiting factor as it would show up in Task Manager. And yes, I set rescan to false ..

I use C# code, basic example here:

https://bitcointalk.org/index.php?topic=166100.msg1733500#msg1733500

However its the same speed to just call bitcoind via cmd.exe and add the required params for importprivkey. So its not a C# problem.

Something is wrong here ..  

Seems currently the only fast way is to add the keys to wallet.dat directly which is quite a painful task to code under Windows ..

[dserrano5]

Set the third parameter of importprivkey to false to avoid the blockchain rescan, then the operation is very quick. Do the rescan only on the last privkey you want to import.

[PoorGirl]

Hi,

I did that already, quote first post "And yes, I set rescan to false .."

Instead to bug hunt my machine, lets collect your stats.

How fast can you import keys with importprevkey via JSON-RPC plus having the full Bitcoin client on a Ramdisk with whatever 3 GB/s or so of I/O speed ? Lets make a competition to import 10.000 private keys .. Proof that I am wrong to say that the current way the Bitcoin client deals with importprevkey is as stupid as I thought BASIC would change the world in school when I was a tick younger.

I start:

2 Keys/sec .. Intel Dual Core 1.8 Ghz, C# Code - beat this !

[Michail1]

Is this really a problem that you would need more than 2 keys per second.

I am curious as to why anyone would need to import 10000 keys, let alone more. 
Seems like something you could just script to have done by morning.

[wumpus]

It's indeed strange that it takes so long, given that you set rescan to false.

Is there any difference in speed between rescan on and off?

If the database works hard that does not necessarily show up in CPU usage. A lot of database operations are I/O bound, so it maybe spends most of the time waiting for disk.

[jackjack]

When I use pywallet to merge wallets, it takes around 2min for 200 addresses so that doesn't shock me
It's the key processing that takes some time

[Mike Hearn]

It's clearly wrong that importing private keys is so slow. That's a rare operation so nobody has spent much time figuring out where the time goes. Profiling it would be a useful contribution.

[jackjack]

Somebody can tweak pywallet in the merge wallet function and put timers to locate the slow line(s)

[gmaxwell]

I just tested and imported 20 keys in under 100ms from the command-line (including the time to fork) with bitcoind git from about a week ago.

I suspect you're providing the rescan flag as the label.

From the command-line the correct invocation is:

Code:

bitcoind importprivkey 5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx "" false

With the "" being the label.

Remuneration for your undue insults in the other thread are accepted at the address in my signature.

[wumpus]

I suspect you're providing the rescan flag as the label.

That was my suspicion as well, but from the linked code: https://bitcointalk.org/index.php?topic=166100.msg1733500#msg1733500 it looks like a label "NewKeyName" is passed. Hence my question whether rescan true/false makes a difference. I'd expect it to be much slower than two keys per second with rescans.

Remuneration for your undue insults in the other thread are accepted at the address in my signature.

[Killdozer]

I'd expect it to be much slower than two keys per second with rescans.

Both that, and also if that was true, the CPU should've been at 100%? If you are using ramdisk, there isn't much more to slow it down...

[PoorGirl]

Using a modded version of the C# Casascius Bitcoin-Address-Utility (https://github.com/casascius/Bitcoin-Address-Utility) I could generate private keys from random passphrases plus their BTC addresses with a speed of 11 keys/s and 50% CPU usage (Intel Dual Core 1.86 Ghz, no HT).

So should be around 20 keys/s with multithreaded code compared to 2.8 Keys/s of bitcoind with its strange 0-1% CPU usage.

[BkkCoins]

I suspect:

1. Scanning the wallet up to see if it's already in the wallet, and /or
2. Performing validity checks to make sure the key is good.

but I haven't checked the code. If you really want to know have a look at the import code and see what it's doing besides adding it to the wallet data.

[deepceleron]

Windows is about 5x slower at doing RPC commands than Linux.

[BkkCoins]

Using a modded version of the C# Casascius Bitcoin-Address-Utility (https://github.com/casascius/Bitcoin-Address-Utility) I could generate private keys from random passphrases plus their BTC addresses with a speed of 11 keys/s and 50% CPU usage (Intel Dual Core 1.86 Ghz, no HT).

So should be around 20 keys/s with multithreaded code compared to 2.8 Keys/s of bitcoind with its strange 0-1% CPU usage.

Umm. Generating is not at all the same as importing. Importing is a record keeping / verification process where as generating is a math process. Generating a key from a passphrase is a pretty fast hash only process. I would fully expect generating to use CPU much more than importing.

[PoorGirl]

The key verification process should be optional as its time expensive. I am sure that my generated keys are valid, so I just want to import them without checks. To create the DEP key and write it to wallet.dat is not expensive.

To increase import speed I ran multiple importprivkey rpc applications in parallel:

10 parallel apps = crash after 5 min
5 parallell apps = crash after 12 hours but finally CPU usage of bitcoind went up to 50% and import speed was also much faster. Similar to my simulation.

So it seems that bitcoind doesn't like parallel rpc commands and gets a hickup somewhere.

So the perfect solution would be:

- make the code path for importprivkey multithreaded whereever possible (OpenMP or automatic with Intel C++ compilers)
- allow parallel incoming RPC commands. If more RPC commands come in then can be processed save them in a buffer and work them down later or respond with an error code 0 for "busy".

I cannot compile bitcoind on Windows. If oneday there is a working VS2012 project I am happy to help.

[jackjack]

Parallel importing means parallel wallet.dat modifying
I think that is the problem

[wumpus]

I cannot compile bitcoind on Windows. If oneday there is a working VS2012 project I am happy to help.

Creating a VS2012 project for Bitcoind should be pretty doable (if you don't need the GUI; Qt+MSVC is somewhat more difficult). I did it once, but it is extremely outdated now so probably useless. That said, I don't have a running windows and no MSVC license, so I can't update it now.

Is there a specific problem that you run against trying to build it in VC2012?

Parallel importing means parallel wallet.dat modifying
I think that is the problem

Indeed. From what I've noticed it is the flush of the wallet database that makes the import slow.  After every key import, for safety, the database is synced to disk, and for some reason this takes longer when there are more keys in it. See also https://github.com/bitcoin/bitcoin/issues/2511 .

This problem will go away when we switch away from BerkelyDB to an append-only wallet format. This is on the roadmap for 0.10/0.11.

[PoorGirl]

Hi John,

btw, thx for your PMs. I had the whole Bitcoin Client on a ramdisk so its not I/O performance which makes it slow.

Expensive seems to be the verification of the imported key plus missing multicore support. Key verification should be optional. Instead to write on each import the client could wait for let say 60 seconds if any more imports are happening or not. If not write to wallet.dat. Keep the new keys in memory, or at least set a buffer limit of 1 MB or so.

For mass import of keys the code logic should be different. Maybe we need a new massimport parameter to handle this situation better.
As a fun test I let it run for some days and had around one million random addresses in the client (wallet.data of ca. 900 MB). Ram usage around 700 MB without any problems. So the basic code is stable to deal with massive addresses, thats good news.

VS2012 with VC or Intel compilers complain about incompatible code. Some sections need to be rewritten.

[wumpus]

For mass import of keys the code logic should be different. Maybe we need a new massimport parameter to handle this situation better.
As a fun test I let it run for some days and had around one million random addresses in the client (wallet.data of ca. 900 MB). Ram usage around 700 MB without any problems. So the basic code is stable to deal with massive addresses, thats good news.

That's good to hear

VS2012 with VC or Intel compilers complain about incompatible code. Some sections need to be rewritten.

Back in the day I've already made some changes to make it compilable with VS2010. However, as no one is regularly compiling it with one of those compilers, likely some new incompatibilities snuck in.
I'll give it another try with VS2012 express when I get around to it.