Bitcoin Forum
March 28, 2024, 08:53:14 PM *
News: Latest Bitcoin Core release: 26.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Only 2 keys/sec added with importprivkey ?  (Read 2560 times)
PoorGirl (OP)
Newbie
*
Offline Offline

Activity: 30
Merit: 0



View Profile
April 03, 2013, 08:52:31 PM
 #1

How comes its so slow to add private keys using importprivkey ?

On a Intel Dual Core 1.8Ghz machine where bitcoind is stored on a big ramdisk the total speed is approx 2 keys/second.
CPU is idle at 0-1%, means there is massive time wasted for nothing.

Is it the RPC interface that's so slow ? If the database would work hard it would show in CPU usage. But there is none. To calc the public hashes/addresses etc also is not the limiting factor as it would show up in Task Manager. And yes, I set rescan to false ..

I use C# code, basic example here:

https://bitcointalk.org/index.php?topic=166100.msg1733500#msg1733500

However its the same speed to just call bitcoind via cmd.exe and add the required params for importprivkey. So its not a C# problem.

Something is wrong here ..  Huh

Seems currently the only fast way is to add the keys to wallet.dat directly which is quite a painful task to code under Windows ..

1711659194
Hero Member
*
Offline Offline

Posts: 1711659194

View Profile Personal Message (Offline)

Ignore
1711659194
Reply with quote  #2

1711659194
Report to moderator
1711659194
Hero Member
*
Offline Offline

Posts: 1711659194

View Profile Personal Message (Offline)

Ignore
1711659194
Reply with quote  #2

1711659194
Report to moderator
1711659194
Hero Member
*
Offline Offline

Posts: 1711659194

View Profile Personal Message (Offline)

Ignore
1711659194
Reply with quote  #2

1711659194
Report to moderator
According to NIST and ECRYPT II, the cryptographic algorithms used in Bitcoin are expected to be strong until at least 2030. (After that, it will not be too difficult to transition to different algorithms.)
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1711659194
Hero Member
*
Offline Offline

Posts: 1711659194

View Profile Personal Message (Offline)

Ignore
1711659194
Reply with quote  #2

1711659194
Report to moderator
1711659194
Hero Member
*
Offline Offline

Posts: 1711659194

View Profile Personal Message (Offline)

Ignore
1711659194
Reply with quote  #2

1711659194
Report to moderator
1711659194
Hero Member
*
Offline Offline

Posts: 1711659194

View Profile Personal Message (Offline)

Ignore
1711659194
Reply with quote  #2

1711659194
Report to moderator
dserrano5
Legendary
*
Offline Offline

Activity: 1974
Merit: 1029



View Profile
April 03, 2013, 09:03:40 PM
 #2

Set the third parameter of importprivkey to false to avoid the blockchain rescan, then the operation is very quick. Do the rescan only on the last privkey you want to import.
PoorGirl (OP)
Newbie
*
Offline Offline

Activity: 30
Merit: 0



View Profile
April 04, 2013, 02:45:42 AM
Last edit: April 04, 2013, 07:25:16 PM by PoorGirl
 #3

Hi,

I did that already, quote first post "And yes, I set rescan to false .."

Instead to bug hunt my machine, lets collect your stats.

How fast can you import keys with importprevkey via JSON-RPC plus having the full Bitcoin client on a Ramdisk with whatever 3 GB/s or so of I/O speed ? Lets make a competition to import 10.000 private keys .. Proof that I am wrong to say that the current way the Bitcoin client deals with importprevkey is as stupid as I thought BASIC would change the world in school when I was a tick younger.

I start:

2 Keys/sec .. Intel Dual Core 1.8 Ghz, C# Code - beat this ! Smiley



Michail1
Legendary
*
Offline Offline

Activity: 1498
Merit: 1164



View Profile WWW
April 05, 2013, 03:41:16 AM
 #4

Is this really a problem that you would need more than 2 keys per second.

I am curious as to why anyone would need to import 10000 keys, let alone more. 
Seems like something you could just script to have done by morning.

wumpus
Hero Member
*****
qt
Offline Offline

Activity: 812
Merit: 1022

No Maps for These Territories


View Profile
April 05, 2013, 09:20:08 AM
Last edit: April 05, 2013, 12:17:03 PM by John Smith
 #5

It's indeed strange that it takes so long, given that you set rescan to false.

Is there any difference in speed between rescan on and off?

If the database works hard that does not necessarily show up in CPU usage. A lot of database operations are I/O bound, so it maybe spends most of the time waiting for disk.

Bitcoin Core developer [PGP] Warning: For most, coin loss is a larger risk than coin theft. A disk can die any time. Regularly back up your wallet through FileBackup Wallet to an external storage or the (encrypted!) cloud. Use a separate offline wallet for storing larger amounts.
jackjack
Legendary
*
Offline Offline

Activity: 1176
Merit: 1233


May Bitcoin be touched by his Noodly Appendage


View Profile
April 05, 2013, 09:32:30 AM
 #6

When I use pywallet to merge wallets, it takes around 2min for 200 addresses so that doesn't shock me
It's the key processing that takes some time

Own address: 19QkqAza7BHFTuoz9N8UQkryP4E9jHo4N3 - Pywallet support: 1AQDfx22pKGgXnUZFL1e4UKos3QqvRzNh5 - Bitcointalk++ script support: 1Pxeccscj1ygseTdSV1qUqQCanp2B2NMM2
Pywallet: instructions. Encrypted wallet support, export/import keys/addresses, backup wallets, export/import CSV data from/into wallet, merge wallets, delete/import addresses and transactions, recover altcoins sent to bitcoin addresses, sign/verify messages and files with Bitcoin addresses, recover deleted wallets, etc.
Mike Hearn
Legendary
*
expert
Offline Offline

Activity: 1526
Merit: 1128


View Profile
April 05, 2013, 09:54:56 AM
 #7

It's clearly wrong that importing private keys is so slow. That's a rare operation so nobody has spent much time figuring out where the time goes. Profiling it would be a useful contribution.
jackjack
Legendary
*
Offline Offline

Activity: 1176
Merit: 1233


May Bitcoin be touched by his Noodly Appendage


View Profile
April 05, 2013, 12:03:51 PM
 #8

Somebody can tweak pywallet in the merge wallet function and put timers to locate the slow line(s)

Own address: 19QkqAza7BHFTuoz9N8UQkryP4E9jHo4N3 - Pywallet support: 1AQDfx22pKGgXnUZFL1e4UKos3QqvRzNh5 - Bitcointalk++ script support: 1Pxeccscj1ygseTdSV1qUqQCanp2B2NMM2
Pywallet: instructions. Encrypted wallet support, export/import keys/addresses, backup wallets, export/import CSV data from/into wallet, merge wallets, delete/import addresses and transactions, recover altcoins sent to bitcoin addresses, sign/verify messages and files with Bitcoin addresses, recover deleted wallets, etc.
gmaxwell
Moderator
Legendary
*
expert
Offline Offline

Activity: 4158
Merit: 8343



View Profile WWW
April 05, 2013, 07:13:01 PM
 #9

I just tested and imported 20 keys in under 100ms from the command-line (including the time to fork) with bitcoind git from about a week ago.

I suspect you're providing the rescan flag as the label.

From the command-line the correct invocation is:
Code:
bitcoind importprivkey 5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx "" false

With the "" being the label.

Remuneration for your undue insults in the other thread are accepted at the address in my signature. Tongue
wumpus
Hero Member
*****
qt
Offline Offline

Activity: 812
Merit: 1022

No Maps for These Territories


View Profile
April 05, 2013, 07:35:09 PM
 #10

I suspect you're providing the rescan flag as the label.
That was my suspicion as well, but from the linked code: https://bitcointalk.org/index.php?topic=166100.msg1733500#msg1733500 it looks like a label "NewKeyName" is passed. Hence my question whether rescan true/false makes a difference. I'd expect it to be much slower than two keys per second with rescans.
Remuneration for your undue insults in the other thread are accepted at the address in my signature. Tongue
Grin

Bitcoin Core developer [PGP] Warning: For most, coin loss is a larger risk than coin theft. A disk can die any time. Regularly back up your wallet through FileBackup Wallet to an external storage or the (encrypted!) cloud. Use a separate offline wallet for storing larger amounts.
Killdozer
Full Member
***
Offline Offline

Activity: 203
Merit: 100



View Profile
April 06, 2013, 08:05:36 PM
 #11

Quote
I'd expect it to be much slower than two keys per second with rescans.

Both that, and also if that was true, the CPU should've been at 100%? If you are using ramdisk, there isn't much more to slow it down...

PoorGirl (OP)
Newbie
*
Offline Offline

Activity: 30
Merit: 0



View Profile
April 10, 2013, 10:25:40 PM
 #12

Using a modded version of the C# Casascius Bitcoin-Address-Utility (https://github.com/casascius/Bitcoin-Address-Utility) I could generate private keys from random passphrases plus their BTC addresses with a speed of 11 keys/s and 50% CPU usage (Intel Dual Core 1.86 Ghz, no HT).

So should be around 20 keys/s with multithreaded code compared to 2.8 Keys/s of bitcoind with its strange 0-1% CPU usage.
BkkCoins
Hero Member
*****
Offline Offline

Activity: 784
Merit: 1009


firstbits:1MinerQ


View Profile WWW
April 11, 2013, 03:51:12 AM
 #13

I suspect:

1. Scanning the wallet up to see if it's already in the wallet, and /or
2. Performing validity checks to make sure the key is good.

but I haven't checked the code. If you really want to know have a look at the import code and see what it's doing besides adding it to the wallet data.

deepceleron
Legendary
*
Offline Offline

Activity: 1512
Merit: 1025



View Profile WWW
April 12, 2013, 02:50:37 AM
 #14

Windows is about 5x slower at doing RPC commands than Linux.
BkkCoins
Hero Member
*****
Offline Offline

Activity: 784
Merit: 1009


firstbits:1MinerQ


View Profile WWW
April 12, 2013, 03:25:03 AM
 #15

Using a modded version of the C# Casascius Bitcoin-Address-Utility (https://github.com/casascius/Bitcoin-Address-Utility) I could generate private keys from random passphrases plus their BTC addresses with a speed of 11 keys/s and 50% CPU usage (Intel Dual Core 1.86 Ghz, no HT).

So should be around 20 keys/s with multithreaded code compared to 2.8 Keys/s of bitcoind with its strange 0-1% CPU usage.
Umm. Generating is not at all the same as importing. Importing is a record keeping / verification process where as generating is a math process. Generating a key from a passphrase is a pretty fast hash only process. I would fully expect generating to use CPU much more than importing.

PoorGirl (OP)
Newbie
*
Offline Offline

Activity: 30
Merit: 0



View Profile
April 13, 2013, 12:36:57 PM
 #16

The key verification process should be optional as its time expensive. I am sure that my generated keys are valid, so I just want to import them without checks. To create the DEP key and write it to wallet.dat is not expensive.

To increase import speed I ran multiple importprivkey rpc applications in parallel:

10 parallel apps = crash after 5 min
5 parallell apps = crash after 12 hours but finally CPU usage of bitcoind went up to 50% and import speed was also much faster. Similar to my simulation.

So it seems that bitcoind doesn't like parallel rpc commands and gets a hickup somewhere.

So the perfect solution would be:

- make the code path for importprivkey multithreaded whereever possible (OpenMP or automatic with Intel C++ compilers)
- allow parallel incoming RPC commands. If more RPC commands come in then can be processed save them in a buffer and work them down later or respond with an error code 0 for "busy".

I cannot compile bitcoind on Windows. If oneday there is a working VS2012 project I am happy to help.

jackjack
Legendary
*
Offline Offline

Activity: 1176
Merit: 1233


May Bitcoin be touched by his Noodly Appendage


View Profile
April 13, 2013, 02:01:58 PM
 #17

Parallel importing means parallel wallet.dat modifying
I think that is the problem

Own address: 19QkqAza7BHFTuoz9N8UQkryP4E9jHo4N3 - Pywallet support: 1AQDfx22pKGgXnUZFL1e4UKos3QqvRzNh5 - Bitcointalk++ script support: 1Pxeccscj1ygseTdSV1qUqQCanp2B2NMM2
Pywallet: instructions. Encrypted wallet support, export/import keys/addresses, backup wallets, export/import CSV data from/into wallet, merge wallets, delete/import addresses and transactions, recover altcoins sent to bitcoin addresses, sign/verify messages and files with Bitcoin addresses, recover deleted wallets, etc.
wumpus
Hero Member
*****
qt
Offline Offline

Activity: 812
Merit: 1022

No Maps for These Territories


View Profile
April 13, 2013, 02:09:13 PM
Last edit: April 14, 2013, 07:04:40 AM by John Smith
 #18

I cannot compile bitcoind on Windows. If oneday there is a working VS2012 project I am happy to help.
Creating a VS2012 project for Bitcoind should be pretty doable (if you don't need the GUI; Qt+MSVC is somewhat more difficult). I did it once, but it is extremely outdated now so probably useless. That said, I don't have a running windows and no MSVC license, so I can't update it now.

Is there a specific problem that you run against trying to build it in VC2012?

Parallel importing means parallel wallet.dat modifying
I think that is the problem
Indeed. From what I've noticed it is the flush of the wallet database that makes the import slow.  After every key import, for safety, the database is synced to disk, and for some reason this takes longer when there are more keys in it. See also https://github.com/bitcoin/bitcoin/issues/2511 .

This problem will go away when we switch away from BerkelyDB to an append-only wallet format. This is on the roadmap for 0.10/0.11.

Bitcoin Core developer [PGP] Warning: For most, coin loss is a larger risk than coin theft. A disk can die any time. Regularly back up your wallet through FileBackup Wallet to an external storage or the (encrypted!) cloud. Use a separate offline wallet for storing larger amounts.
PoorGirl (OP)
Newbie
*
Offline Offline

Activity: 30
Merit: 0



View Profile
April 17, 2013, 04:28:08 PM
 #19

Hi John,

btw, thx for your PMs. I had the whole Bitcoin Client on a ramdisk so its not I/O performance which makes it slow.

Expensive seems to be the verification of the imported key plus missing multicore support. Key verification should be optional. Instead to write on each import the client could wait for let say 60 seconds if any more imports are happening or not. If not write to wallet.dat. Keep the new keys in memory, or at least set a buffer limit of 1 MB or so.

For mass import of keys the code logic should be different. Maybe we need a new massimport parameter to handle this situation better.
As a fun test I let it run for some days and had around one million random addresses in the client (wallet.data of ca. 900 MB). Ram usage around 700 MB without any problems. So the basic code is stable to deal with massive addresses, thats good news.

VS2012 with VC or Intel compilers complain about incompatible code. Some sections need to be rewritten.
wumpus
Hero Member
*****
qt
Offline Offline

Activity: 812
Merit: 1022

No Maps for These Territories


View Profile
April 17, 2013, 05:46:53 PM
 #20

For mass import of keys the code logic should be different. Maybe we need a new massimport parameter to handle this situation better.
As a fun test I let it run for some days and had around one million random addresses in the client (wallet.data of ca. 900 MB). Ram usage around 700 MB without any problems. So the basic code is stable to deal with massive addresses, thats good news.
That's good to hear Smiley

Quote
VS2012 with VC or Intel compilers complain about incompatible code. Some sections need to be rewritten.
Back in the day I've already made some changes to make it compilable with VS2010. However, as no one is regularly compiling it with one of those compilers, likely some new incompatibilities snuck in.
I'll give it another try with VS2012 express when I get around to it.

Bitcoin Core developer [PGP] Warning: For most, coin loss is a larger risk than coin theft. A disk can die any time. Regularly back up your wallet through FileBackup Wallet to an external storage or the (encrypted!) cloud. Use a separate offline wallet for storing larger amounts.
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!