Any way to get client startup faster?

[pent]

When i start the client it does not show actual client window for a long period of time (from 1 to 5 mins) - Linux and Windows. In that time disk activity is very large. Then it shows the window and start to catch current block count.

I guess there is a lack of some optimization algorithms.

Also client is shutting down slowly too. Window disappeared but process is still running and do something actively enough. I think it will not like normal system shutdown when KILL signals are sent to processes which didn't understand TERM.

[wumpus]

I've noticed this as well. It'd be better if it displayed some kind of startup screen instead of just staying silent, same for shutdown ("updating database... please wait").

I'll probably implement this in my interface, unless it's possible to get the startup time down to a negligible amount some other way. I haven't profiled what causes the delay yet.

[dikidera]

If the devs want Bitcoin to succeed, they need to make it to start right away. I mean, if i need to pay for something _now_ that means now!! not 5 minutes later.

[kokjo]

If the devs want Bitcoin to succeed, they need to make it to start right away. I mean, if i need to pay for something _now_ that means now!! not 5 minutes later.

Go do it yourself!

if you want other poeple to do something for you, you should ask polite!!!!!!

please respect the devs.

[error]

First, update to version 0.3.24.

[bitrick]

I have also seen the client startup slowly (minutes or hours). This occurs with 0.3.24beta which is the latest software available on the front page of bitcoin.org.

When I first encountered this problem, I thought, "Well that sucks. The GUI does not appear for a very, very long time and messages on the forum seem to indicate this is normal while it reads the block chain. WTF? They cannot possibly expect folks to use a client with such a crappy design, do they?".

There is no way for a new user to tell if the client is working or not. It seems the message is not getting through, so let me repeat: There is no way for a new user to tell if the client is working or not. It looks like it just won't run. I was dedicated to getting it work, and I did after a few days of searching the forums and futzing around, but I imagine many users will just give up.

If this is fixed in 0.3.24 release version then please accept my gratitude for fixing this. If not, then please make this issue the top priority. It really is horrible. I suppose I can fix it myself but it will probably be many weeks before I have the time. (BTW I am on Windows 7 64bit).

[JoelKatz]

I've been meaning to profile client startup. I noticed that the latest build seems to take a long time to startup on Linux. I did a quick trace and it appeared to be a massive number of mlock/munlock requests. I haven't had a chance to investigate further yet.

[wumpus]

There is no way for a new user to tell if the client is working or not. It seems the message is not getting through, so let me repeat: There is no way for a new user to tell if the client is working or not.

Yes this is why I proposed the splash screen a few posts up. It'd at least fix the "hey, no GUI appears!" problem.

Another possibility would be to just display the GUI, but have everything disabled until the "database is checked" (what ever it is doing).

Keep up the good detective work JoelKatz

If the devs want Bitcoin to succeed, they need to make it to start right away. I mean, if i need to pay for something _now_ that means now!! not 5 minutes later.

Lol, this "if the devs want bitcoin to succeed do THIS RIGHT NOW" meme is starting to be funny ... it's so much overused, If I had time I'd add it into a kitten image

If you want a dev to do work for you, you should pay him for the time and trouble, not threaten and complain.

[etotheipi]

I agree this needs to be improved, and I agree there should be a splash screen or a disabled GUI while things load.   It would seem that you could store temporary information from the last time the client was closed, to display at startup but gray-out the send-money button.  Then the user can browse the previous transaction data, create new addresses, etc, while the backend goes through and re-verifies the blockchain.  You could even put the progress bar under the "send" button to show the user why it's grayed out.  This should solve a lot of the current speed problems if the multithreading can be done correctly (don't ask me to do it, I've never written MT code).

But, I've been thinking a lot about it how to optimize the file-reading and I'm working a prototype right now.  Unfortunately, it is without much context about what is already there in the client code, but this is fun for me so I'm working on the ultimate-efficiency blockchain file-loading.  I'd like to hear any thoughts on this process...

By my calculation, we have about 400 MB of blockchain information, which for most HDDs should take about 5 seconds to read.  That means, at least while the client is designed to maintain the entire blockchain, we'll never beat 5 seconds load time for the average user.   This may not be too bad right now, but it's not going to get any better.  And of course, it takes a lot longer than that because the blockchain has to be processed/verified as it is read in.  My thought is... what if it doesn't take any extra time?

• (1)  The blockchain data is stored in the file in exactly the same binary format as would be stored in RAM.
• (2)  In memory, allocate 400 MB of storage behind a raw uint8_t pointer
• (3)  Do a direct copy "blk0001file.read(ramChunkPtr, fileSize)"
• (4)  Use BlkHeaderPtr and TxPtr objects  -- the members are pointers to locations in this chunk of memory instead of copies
• (5)  Write the accessor methods to simply dereference the pointers, any time

The overhead of storing these pointers should be small compared to the blockchain itself.  And there's two options for assigning the pointers... either walk through the data and assign them manually (which would be slowed slightly be all the var_ints), or store the variable byte-locations in another file, (known from the last client shutdown).  Hell, you could do the exact same memory chunk idea with this file, but even faster because they are all constant-width fields (and dramatically less data).    Either way, this should be about the best time-efficiency possible, paid for with a slight reduction in space efficiency. 

[bitrick]

I am beginning to think there are two classes of problems: 1. Slow reading of the block chain from disk that results in a significant but reasonable delay on the order of many seconds. 2. Something else which has remained unexplained that causes very long startup delays on the order of minutes or hours (and sometimes it will never start until the machine is restarted.)

With respect to #2, I am not sure what the problem is exactly, but I think these explanations can be explored: no GUI if the port is not open, no GUI until the entire block chain is downloaded, no GUI if a lock file exists, no GUI if you started it more than once before the GUI actually appeared, something specific to Windows 7 64bit, etc.

When I first started the client it looked like the GUI would not appear until the entire block chain was downloaded. I could watch the files being created, but no GUI would come up. I should note that I had issues in my network with the port not being open, but I am unsure how big a factor that was (I eventually fixed it). Whatever the issues, it took hours until I first saw the GUI.

Again, I think the scope of the problem is beyond just slow reading from disk of an up-to-date block chain.

[Mike Hearn]

Startup time on the main client comes from the following sources:

Loading of the binary itself:  <1sec on most computers.

Loading of addr.dat:  ~5secs on older builds, less on newer I think.

Loading the wallet: <0.5 secs unless you have an unusually large wallet. This can take over a minute if you're running something like an exchange or hosted wallet.

Scanning the block chain headers into memory: about 13 seconds.

(the GUI shows at this point)

Peer discovery: <0.5sec on the latest builds which default to DNS over IRC. A few seconds on older builds that use IRC by default.

Finding a peer that is listening: on very old builds (more than a few versions ago) this can take several minutes. On old builds (after connection timeout fixes but before DNS seeds) this can take 30-60 seconds. On newer builds it should be pretty much instant.

Finding a non broken peer that is listening: on older builds that use IRC seeding this can take a very, very long time (hours) because you will spend a lot of time connecting to nodes that are also running <0.3.24 and as such cannot reliably transmit the block chain. Your node will spend time connecting to a peer, asking for the block chain and then being disconnected. Wash, rinse repeat. Eventually you'll connect to a node that is 0.3.24+ and be able to obtain the chain.

Catching up with the block chain: My MacBook Pro can process the chain at about 1 block per second in recent times. So that's 1 second for every 10 minutes of real time, ie if it was a day since you last ran the client that's 144 seconds or a little over two minutes.

As you can see, startup time is heavily dependent on running the very latest version, not just you but for the whole network. By always running the latest version, you not only give yourself better startup time, but everyone else too.

[wumpus]

Thanks for the overview!

I'm primarily interested in the part before the GUI shows, as there is no feedback to the user at that moment.

[etotheipi]

(1)  Does it take 13s to load just the headers?  Or the entire blk0001.dat?  Right now, I can open and scan the 11MB of block headers in C++ in about 1s.  The block data takes considerably longer since there's 400 MB. 

(2)  I noticed that my client in Linux takes a tremendous amount of time to download updated blocks--perhaps 1-4 per minute.  Even when I close the client for only a day, it can take hours for it to get caught up.  But when I boot my windows XP VM that has a different, "savings account" wallet on it, that client gets up to speed almost immediately (probably, 1 per sec as Mike suggested).  I recognize that there is a peer/version discovery issue, but why would it be so dramatically different on the two systems?  This makes it almost unusable on Linux, unless I open the client a full 1-8 hours before I want to send someone money.  It's faster for me to boot my VM, load the BTC client there, and send directly from my savings account.  8 hours later, when my linux client is ready, I reimburse my savings account...

(3) Could the finding of node discovery/version issues warrant looking into using the bit-torrent protocol (or similar) for moving blocks around?  The version should not matter for moving this data around, but the necessity of version matching to make a connection forces this to become a bottleneck.  Also, I thought the getdata() request gets up to 500 blocks from the referenced hash...?  Doesn't that mean, unless you are more than 500 block behind, you only need to find a good peer once, to get the updated blockchain?

[Mike Hearn]

Yeah, you only need one good peer. Older peers have a serious bug that makes them unable to send the chain. I don't think any new protocol is needed, just for people to upgrade. Bitcoin isn't aggressive enough with version update messages. Auto-update would be good but was never implemented.

13 sec is for loading the headers out of blkindex.dat. It involves lots of small bdb reads, as well as connecting the blocks together and verifying them.

I don't know why the Linux version would be much slower than Windows, but wallet size may have something to do with it. There is a lot of unoptimized code in Bitcoin.

[etotheipi]

But, given the difficulty getting people to upgrade their clients, it sounds like this is going to be a extremely persistent problem without doing something different.  We should consider ways to allow clients with different versions to accommodate each other, at least for blockchain downloads.  Rather than disconnecting clients with different versions immediately, maybe give the opportunity to exchange getdata() requests.  And maybe there should also be some kind of announcement message as part of the client, which checks bitcoin.org for newer version and alerts the user they should upgrade. 

The key problem (at least for me), is that if you are not up-to-date on the block headers, you can't do anything else.  The other day I tried sending someone coins, wondering what would happen, and the client said "Payment Sent!" but stayed at "0/Unconfirmed" for 8 hours until it caught up to the top of the chain. 

If you only need one good peer, why does my client take 1 minute per block to download?  It behaves as if it's searching for a node, eventually finds one, gets one block, then gets disconnected and has to start over.  The other day I was only 5 blocks behind, but it was taking like 3 minutes per block, so it took forever to catch up to the blockchain.  Yes, this is linux, client 0.3.24.


P.S. - Perhaps I'm an outlier... does anyone else have these kinds of problems getting blocks from the network?

[wumpus]

Rather than disconnecting clients with different versions immediately, maybe give the opportunity to exchange getdata() requests.

I don't think any forced disconnection happens, ever. At least all the recent clients can communicate with each other, due to protocol versioning.

The bug Mike Hearn is talking about is unrelated to the version difference itself, but it is a throttling bug present in older versions.

P.S. - Perhaps I'm an outlier... does anyone else have these kinds of problems getting blocks from the network?

No problems getting blocks on any of my (Linux) hosts at the moment. A month ago it was problematic sometimes. Do you forward the port 8333?

[etotheipi]

When you say "forward the port 8333," do you mean port forwarding on my router?  I tried that (any traffic to my DMZ IP address at port 8333 will go directly to my computer's port 8333).  It doesn't make much sense to me, but maybe that's what you meant.  I'll try it when I get home.

If you look back to Mike Hearn's post, he suggests that my 0.3.24 client won't be able to get the blockchain except from other 0.3.24 clients... which is how I thought it worked, too (but I haven't gotten to the networking part of the protocol yet, so what do I know?)

And after all this discussion, have we determined if there is a bottleneck that can be addressed?  Assume everyone magically upgrades to 0.3.24 today, and the throttling bug disappears.  Do we still expect speed issues?  Should we be brainstorming better ideas for multi-threading, make GUI more accessible while your system is working in the background? 

And finally, any reason you know why it takes 12 seconds to read the headers from blk0001.dat?  That's how long it takes my python script to do it (with hashing and blockchain organization), so I assumed it could be done dramatically faster in C++ code...

[Mike Hearn]

The loading of wallets, block chains etc should happen after the GUI is brought up. If your Python script takes the same amount of time as the C++ all that shows is that the process is IO bound, which is what you'd expect.

[etotheipi]

Does anyone know how the headers and tx data is stored in RAM?  If everything is referenced by 32-byte hashes, accessing blocks/tx's by hash could be very slow if we're not using a good data structure.  A binary search tree probably isn't even good enough.  A radix/patricia tree would be ideal, since you can access any object in about 40 clock cycles or less. 

If someone is already doing some kind of speed profiling of the code, I'm wondering how long it takes for block/tx access based on hash?  Maybe you could pick out a block hash, and put in a for loop to get the version of that block a million times and time how long it takes.  I thrive on data structures, pointers, and low-level optimizations like this, I'd love to contribute. 

[error]

Does anyone know how the headers and tx data is stored in RAM?  If everything is referenced by 32-byte hashes, accessing blocks/tx's by hash could be very slow if we're not using a good data structure.  A binary search tree probably isn't even good enough.  A radix/patricia tree would be ideal, since you can access any object in about 40 clock cycles or less. 

If someone is already doing some kind of speed profiling of the code, I'm wondering how long it takes for block/tx access based on hash?  Maybe you could pick out a block hash, and put in a for loop to get the version of that block a million times and time how long it takes.  I thrive on data structures, pointers, and low-level optimizations like this, I'd love to contribute. 

https://github.com/bitcoin/bitcoin