Bitcoin Forum
December 06, 2016, 08:15:18 PM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 [2]  All
  Print  
Author Topic: Any way to get client startup faster?  (Read 3771 times)
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
July 29, 2011, 05:50:26 PM
 #21

Thanks.  I've mentioned repeatedly in other posts before -- I can't grok the C++ code.  Despite my experience with C++, it is completely incomprehendable to me, and I was hoping someone already knew the answer to this.   Yesterday I did try to look at it, I saw a CDB class using a "Db*" and all it's derived classes, with a large web of inheritance.  I never actually found where the data was stored, or how it's accessed, or where various methods are defined (probably all over the inheritance graph).  Perhaps data is stored indexed by pair<hash, something> objects, but I can't tell for sure.  

The even more frustrating part is that even if I develop a solution that's remarkably faster (I'm just about there, I'm reading an organizing headers nearly instantaneously from file), there's no way I can contribute to that code base, because there's no way I could ever get in that code and make a meaningful patch.  I'm not sure what to even do with my code, besides keep it to myself and try to develop my own client...

Sorry my C++ skills aren't up to par.  But that's why we have forums to discuss things.  So in case all you were suggesting was to look at the code... I've done that.  I've tried countless times to read the code and never understand what I find.  Perhaps you can offer suggestions for how to understand it better...

-Eto

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
1481055318
Hero Member
*
Offline Offline

Posts: 1481055318

View Profile Personal Message (Offline)

Ignore
1481055318
Reply with quote  #2

1481055318
Report to moderator
1481055318
Hero Member
*
Offline Offline

Posts: 1481055318

View Profile Personal Message (Offline)

Ignore
1481055318
Reply with quote  #2

1481055318
Report to moderator
1481055318
Hero Member
*
Offline Offline

Posts: 1481055318

View Profile Personal Message (Offline)

Ignore
1481055318
Reply with quote  #2

1481055318
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481055318
Hero Member
*
Offline Offline

Posts: 1481055318

View Profile Personal Message (Offline)

Ignore
1481055318
Reply with quote  #2

1481055318
Report to moderator
1481055318
Hero Member
*
Offline Offline

Posts: 1481055318

View Profile Personal Message (Offline)

Ignore
1481055318
Reply with quote  #2

1481055318
Report to moderator
1481055318
Hero Member
*
Offline Offline

Posts: 1481055318

View Profile Personal Message (Offline)

Ignore
1481055318
Reply with quote  #2

1481055318
Report to moderator
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
August 05, 2011, 05:18:19 PM
 #22

So, as an update:  I finally got some of my tools together and I have implemented the blockchain loading.  Reading all the headers from blk0001.dat, and organzing the chain, with orphan checking, etc, takes approximately 1 second on my system.  I am running a AMD Phenom II X4 840, which is a modest quad-core system, though I suspect only one core is being used since I didn't implement any threading.

So it concerns me that Mike's post says it takes approx 13s to load the headers.  Is there a bug in the client code for executing this process?  Is it just simply inefficient?   According to Mike's post, this is the biggest bottleneck to getting the GUI open, so perhaps it's worth looking at what is going on here.  Perhaps Mike has the capability to profile what part of loading the headers is slow...? 

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
Mike Hearn
Legendary
*
expert
Offline Offline

Activity: 1526


View Profile
August 05, 2011, 06:23:06 PM
 #23

I think you haven't implemented all the checks Bitcoin is doing. It doesn't just verify headers but full blocks in the best chain.
vector76
Member
**
Offline Offline

Activity: 70


View Profile
August 05, 2011, 07:17:55 PM
 #24

I think you haven't implemented all the checks Bitcoin is doing. It doesn't just verify headers but full blocks in the best chain.
On startup?

No way it checks everything in all the blocks.
BitcoinBug
Full Member
***
Offline Offline

Activity: 196


View Profile
August 05, 2011, 08:52:50 PM
 #25

No way it checks everything in all the blocks.

I'm guessing everything but signatures. I think loading time would be explained by this. Never looked at source though...
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
August 06, 2011, 04:33:16 PM
 #26

I doubt it checks everything as I think that would take much longer than 12 seconds.  But I don't really know what else it would be doing... unless it's loading the entire blockchain into memory.  Seems like that could be done after the GUI loads, though...

On client startup speed:  Mike Hearn mentioned that the biggest thing that can be done to improve speeds of... everything... is for everyone to upgrade.  Now, this is a very difficult task with no automatic update/notify code in the client.  Certainly, if it's that critical to the network for smooth operation, we should at least have the client notify the user they are out of date.  But this doesn't work until we put the patch in and get most people to upgrade to that one, so there's a Catch-22.

I just emailed the owner of bitcoinwatch.com, and requested that he put a persistent, high-visibility link on his page that advertises the current client version with a link to bitcoin.org.  He probably gets a high proportion of all BTC users as visitors to his site.  However, I'm not too familiar with the other sites where BTC users go, besides the exchanges.  If you know of such sites, please send them an email asking them to advertise as such, perhaps something along the lines of:

"Current Bitcoin version is 0.3.24.  Please upgrade now!"

It sounds like this is the only chance we have of getting users to upgrade.  I wouldn't have even known about the upgrade myself if I hadn't checked Bitcoinwatch on the day the update was released and saw it on "current news." 

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
vector76
Member
**
Offline Offline

Activity: 70


View Profile
August 06, 2011, 05:45:03 PM
 #27

Ok just now I started it and CTxDB::LoadBlockIndex took 30 seconds to complete.

It seems to take longer if I haven't been doing anything with bitcoin for a while, perhaps because the disk data is not cached by the OS.

It looks like CTxDB::LoadBlockIndex scans through ALL the records within blkindex.dat looking for blockindex records (block headers) which it extracts.

It then sorts them by block height, and calculates the cumulative work for each block and establishes which one is the best chain.

Then it loads the full block data and verifies it using CBlock::CheckBlock().
[edit: it appears that it only loads the most recent 2500 blocks, not all of them.]

CheckBlock() checks size limits, proof of work, coinbase, merkle root, and also checks transactions within the block but it only does simple transaction sanity checks which do not include checking whether the inputs have been spent before or whether the signatures are valid.

It's amazing that all that can be done in so little time.

I guess the next question is why...
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
August 07, 2011, 08:28:29 AM
 #28

Ok just now I started it and CTxDB::LoadBlockIndex took 30 seconds to complete.

It seems to take longer if I haven't been doing anything with bitcoin for a while, perhaps because the disk data is not cached by the OS.

Don't most profiling tools have options to explicitly flush read caches before runs, for just this reason?

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
September 20, 2011, 03:58:33 PM
 #29

Sorry to revive an old thread, but I finally got my technique implemented for reading the blockchain, and I believe it represents a lower-bound on computational time for reading in the blockchain.   I am posting because this is empirical data on what I believe to be the most efficient blockchain scanning technique possible. 

Note: I recognize that a full-RAM implementation is not ideal for the general user, but it's feasible for modern computers (at current tx volume for the next couple years) and this was for the purpose of finding the lower-bound on processing time.

  • Allocate 500MB contiguous space in RAM, do a single file-read of the entire blk0001.dat file into this memory
  • Create maps of blockheader pointers and tx pointers indexed by hash (pointers reference memory locations in RAM).
  • Organize all blockheaders into a chain, calculate the longest one, create a headersByHeight vector of pointers
  • Read in a wallet of a few addresses and do a fresh rescan for relevant transactions
  • Does not do any ECDSA verification in the scan
  • Does not check merkle roots/trees
  • Does not check validity of anything -- assumes if it's in the blockchain, it's valid

I have a quad-core AMD CPU (64-bit), and ran this method on a WinXP virtual machine.  I applied the above process to 135,000 blocks. 
-- Read blockchain into memory:  4s (120MB/s HDD sounds about right)
-- Construct the header and tx maps from scratch in 9s
-- Blockchain organization and finding longest chain from scratch:  0.5s
-- The full wallet scan takes about 15s

So, making the assumption that the longest chain contains only valid data, I can do a full rescan in about 30 seconds.  With 145k blocks, I assume this will be more like 40-45s.  I believe this is a very favorable result.   I recognize there is no fair comparison to the existing client, because I believe the existing client does all the ECDSA signature verifications. 



Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
Alex Zee
Member
**
Offline Offline

Activity: 112



View Profile WWW
September 20, 2011, 04:51:07 PM
 #30

It'd be better if it displayed some kind of startup screen instead of just staying silent

It would be better if it just created UI first and do the work later.

BTC Monitor - systray price ticker
RipTalk.org - new Ripple forum
phorensic
Hero Member
*****
Offline Offline

Activity: 630



View Profile
September 21, 2011, 04:49:12 AM
 #31

It'd be better if it displayed some kind of startup screen instead of just staying silent

It would be better if it just created UI first and do the work later.
Bitcoin-qt does this, and very well.  It will be merged with main client in v0.5.0.  I wish it was sooner, but I don't mind compiling bitcoin-qt every once in a while.
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
October 18, 2011, 12:19:29 AM
 #32

I did some benchmarking of client startup. About 75% of it is validating the block chain (CTxDB::LoadBlockIndex). About 15% of it is loading addresses (CAddrDB::LoadAddresses).

Looking at the block chain load/verify process, about 40% of the time is CBlock::BuildMerkleTree. About 10% is SHA256. The rest is ReadFromDisk, GetBlockHash, InsertBlockIndex, and so on.

Looking at the loading of addresses, it's distributed over many functions. The only standouts are CDB::ReadAtCursor and std::_Rb_tree::_M_insert_unique.

Just not checking that the transaction's hashes are correct and match the block header would halve the startup time.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
2112
Legendary
*
Offline Offline

Activity: 1708



View Profile
October 18, 2011, 12:51:41 AM
 #33

I did some benchmarking of client startup.
Did your profiling tool allocated some time to I/O wait? Seems like you've enumerated a CPU-time profile, not a wall-time profile.

Thanks.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
JoelKatz
Legendary
*
Offline Offline

Activity: 1386


Democracy is vulnerable to a 51% attack.


View Profile WWW
October 18, 2011, 06:32:42 PM
 #34

I did some benchmarking of client startup.
Did your profiling tool allocated some time to I/O wait? Seems like you've enumerated a CPU-time profile, not a wall-time profile.
It's wall time, but I made sure all the files were hot in the cache. At least on my machine, if no data is hot in the cache, disk I/O is about 12% of startup time.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
Mike Hearn
Legendary
*
expert
Offline Offline

Activity: 1526


View Profile
October 19, 2011, 10:06:12 PM
 #35

It rechecks blocks so if a new client has a rule change/bugfix, the chain will be revalidated and blocks that no longer match the rules will be kicked out, triggering a re-org.

A simple optimization would be to only perform this rechecking process when the version has changed.
Pages: « 1 [2]  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!