Blockchain Tipping Point?

[ImNotHerb]

So I've been syncing my first ever bitcoin wallet, and the process has been going on for the past 3 days - and gets slower and slower the more it downloads. Currently have about 1,400 blocks left to download which at the rate of deceleration I estimate will take at least another 24+ hours.  My poor laptop will probably have a heart attack from all this!

It says the last downloaded block was created only 10 days ago, so I'm at the point where it's only catching up with the last ten days of transactions (apparently the more recent blocks are much MUCH larger than the older ones). At this rate I have to wonder if eventually bitcoin blocks will be growing faster than they can be downloaded and thus make the blockchain impossible to sync since it will take longer to download a block than it does for new blocks to be created. For example, what if it reached the point where it takes 4 days to sync with the last two days of blocks?

Thoughts? 

[cloudnein]

Use  an e-wallet

[Moshi]

If I'm not mistaken, no.

Each block is limited to 1MB in size. It shouldn't take any longer at a certain point, even given the extra overhead from all the housekeeping.

(correct me if I'm wrong; just sharing what I know!)

[ImNotHerb]

If I'm not mistaken, no.

Each block is limited to 1MB in size. It shouldn't take any longer at a certain point, even given the extra overhead from all the housekeeping.

After checking the blk files, I see the blocks are not larger. But for some reason the later blocks are syncing at a much MUCH slower rate than the previous blocks. The blk0001.dat (2,048,205kb) file only took a few hours to download. The blk0002.dat (2,048,141kb) took an entire day. blk0003.dat has been downloading for over 9 hours now and is (so far) 293,000kb and growing very VERY slowly.

So while the more recent blocks might be the same size, for some strange reason it is taking exponentially longer to sync them.

[Stephen Gornick]

If I'm not mistaken, no.

Each block is limited to 1MB in size. It shouldn't take any longer at a certain point, even given the extra overhead from all the housekeeping.

After checking the blk files, I see the blocks are not larger. But for some reason the later blocks are syncing at a much MUCH slower rate than the previous blocks. The blk0001.dat (2,048,205kb) file only took a few hours to download. The blk0002.dat (2,048,141kb) took an entire day. blk0003.dat has been downloading for over 9 hours now and is (so far) 293,000kb and growing very VERY slowly.

So while the more recent blocks might be the same size, for some strange reason it is taking exponentially longer to sync them.

This is definitely a problem and the next release of the Bitcoin.org client (likely v0. will include huge improvements in this area thanks to the use of new storage software (LevelDB) and the storage strategy labeled "ultraprune":

Ultraprune merged in mainline
 - http://bitcointalk.org/index.php?topic=119525.0

Even then, not every piece of low end hardware will be sufficient to run a full node (e.g., the amount of storage to run a full node will always be in the "many gigabytes" range, even after the switch to LevelDB.)

Most of the delay you are experiencing is likely due to the amount of disk I/O occurring during transaction verification.   If you are using an encrypted disk or encrypted filesystem, the Bitcoin.org client will take an incredibly long time to reach "sync" (like the range of time that you are reporting).  Also, if you have a screensaver or other programs that compete for system resources, that too can make the Bitcoin.org client sync process take an extraordinarily long period of time.

Without a beefy system, you might want to consider using an Simplified Payment Verification (SPV) client such as MultiBit, or a "lite" client that like Electrum, or an EWallet, perhaps Blockchain.info/wallet.

[Moshi]

Most of the delay you are experiencing is likely due to the amount of disk I/O occurring during transaction verification.   If you are using an encrypted disk or encrypted filesystem, the Bitcoin.org client will take an incredibly long time to reach "sync" (like the range of time that you are reporting).  Also, if you have a screensaver or other programs that compete for system resources, that too can make the Bitcoin.org client sync process take an extraordinarily long period of time.

That sounds about right; I remember on a few occasions that, while downloading the blockchain, my disk would become completely choked (general unresponsiveness, access light staying on for several seconds). Using an SSD, now, and a few gigabytes in, the light's only flickering, and file access isn't any slower than it normally is.

I wouldn't think, though, that even with the fastest connection (say, 20MB/s) that you would floor the disk (which have througputs generally atleast 50MB/s, although I could see many small writes would be magnified by disk access times). In short: what gives? The client is just writing out the block chain, is it not?

[ImNotHerb]

Even then, not every piece of low end hardware will be sufficient to run a full node (e.g., the amount of storage to run a full node will always be in the "many gigabytes" range, even after the switch to LevelDB.)

Most of the delay you are experiencing is likely due to the amount of disk I/O occurring during transaction verification.   If you are using an encrypted disk or encrypted filesystem, the Bitcoin.org client will take an incredibly long time to reach "sync" (like the range of time that you are reporting).  Also, if you have a screensaver or other programs that compete for system resources, that too can make the Bitcoin.org client sync process take an extraordinarily long period of time.

I'm not too sure what qualifies as "low end" vs "high end", but the laptop is an HP Pavilion G7, 64-bit with 4G RAM and hundreds of gigs of free space. Running Win7 with no disk encryption, screensavers and hibernate turned off, and the wallet client is basically the only program that's been running the past couple of days and nights. Though I suppose the Norton AV program is always doing its background tasks (not sure how much that would slow things down). But it is using the CPU somewhat constantly at 30-50% and the page file is a couple of gigs in size. Network activity is strangely intermittent, with the task manager graph looking like a hibernating whale's EKG (i.e., long periods of no activity interspersed with brief spikes of 20-30% network utilization)

At any rate, I've now installed the Electrum client on my linux machine (*poof* - that was super easy! Thanks for the suggestion). Meanwhile the laptop has a little over 400 blocks to go. I look forward to the switch to LevelDB once it's out of beta. As for the current version, when the sync is complete, does using the wallet program still put a heavy strain on the CPU?

[RyNinDaCleM]

The qt client is more memory intensive than CPU, once synced.

[DeathAndTaxes]

I'm not too sure what qualifies as "low end" vs "high end", but the laptop is an HP Pavilion G7, 64-bit with 4G RAM and hundreds of gigs of free space. Running Win7 with no disk encryption, screensavers and hibernate turned off, and the wallet client is basically the only program that's been running the past couple of days and nights

Once synced the CPU load is negligible (maybe 2% with the occasional burst up to 5%) for an average newish CPU.  It does use up to about 1GB of RAM.  You likely can get away with less but on a server with 16GB of RAM is quickly allocates up to about a gig and then holds there.  The I/O is really only bad in bootstrap mode, once synced the number of blocks is relatively low and thus the I/O load is spread out.   

Laptops are generally ill suited to run full nodes because their hard drives tend to .... "suck".  The bootstrapping process results in a lot of I/O.  Your CPU was idle a lot because it was waiting for that painfully slow laptop HDD to "catch up".  v0.8 is a step in the right direction and more improvements will happen overtime but you have to realize a node is a peer in a global banking network.  To be a full node is going to require some resources and that will only grow over time.  The client will get more efficient, the db better, and hardware will continually improve but on the other hand the daily tx volume will grow and the historical chain will just get longer.  In the short term if you really want to run bitcoind on a laptop you may find an upgrade to either an SSD or 8GB of ram and creating a ram disk will provide is substantial boost.

In 5-10 years I would say 90%+ of users will use some form of eWallet, lite client, or hybrid client.  The "full node" network will be made up of miners, merchants, and Bitcoin enthusiasts.  

[ImNotHerb]

Laptops are generally ill suited to run full nodes because their hard drives tend to .... "suck".

So what I should have done is installed the bt client on my desktop linux and just used a "lite" version for the laptop.

Why do I always rush into things only to do them bass-ackwards?  

If I did want to try running a full node on the linux machine, can I simply transfer the blk00xx.dat and index.dat files and not have to re-download the whole thing before it will sync?

At any rate, I'm relieved to see that there are upcoming implementations to prevent the blockchain from becoming too unwieldy in the near future.  

[Stephen Gornick]

If I did want to try running a full node on the linux machine, can I simply transfer the blk00xx.dat and index.dat files and not have to re-download the whole thing before it will sync?

Yup, though you will need to do a "-detachdb=1" from the command line once after the move.

[DPony13]

I suggest using Electrum or Multibit
Since they are faster and lightweight

[greyhawk]

Use  an e-wallet

Great advice...
















... to get your coins stolen.

[flatfly]

Laptops are generally ill suited to run full nodes because their hard drives tend to .... "suck".  The bootstrapping process results in a lot of I/O.  Your CPU was idle a lot because it was waiting for that painfully slow laptop HDD to "catch up".  v0.8 is a step in the right direction and more improvements will happen overtime but you have to realize a node is a peer in a global banking network.  To be a full node is going to require some resources and that will only grow over time.  

IMHO Bitcoin-Qt should check if it's being installed on a laptop and refuse to proceed if that's the case, or at least warn about the system requirements.

[RyNinDaCleM]

Laptops are generally ill suited to run full nodes because their hard drives tend to .... "suck".  The bootstrapping process results in a lot of I/O.  Your CPU was idle a lot because it was waiting for that painfully slow laptop HDD to "catch up".  v0.8 is a step in the right direction and more improvements will happen overtime but you have to realize a node is a peer in a global banking network.  To be a full node is going to require some resources and that will only grow over time.  

IMHO Bitcoin-Qt should check if it's being installed on a laptop and refuse to proceed if that's the case, or at least warn about the system requirements.

Definitely not refuse, but warn perhaps...
My laptop has dual SSD's, 8 gigs of RAM, and a sandy i7! It's faster and more capable than most desktops, and I like to be able to run a full node.

[grantbdev]

I suggest using Electrum or Multibit
Since they are faster and lightweight

+1

Don't use an e-wallet, just use electrum so you don't have to bother with the blockchain. I had no choice on my laptop because I have a 30GB SSD only and the blockchain was getting scary.

[ImNotHerb]

Laptops are generally ill suited to run full nodes because their hard drives tend to .... "suck".  The bootstrapping process results in a lot of I/O.  Your CPU was idle a lot because it was waiting for that painfully slow laptop HDD to "catch up".  v0.8 is a step in the right direction and more improvements will happen overtime but you have to realize a node is a peer in a global banking network.  To be a full node is going to require some resources and that will only grow over time.  

IMHO Bitcoin-Qt should check if it's being installed on a laptop and refuse to proceed if that's the case, or at least warn about the system requirements.

Definitely not refuse, but warn perhaps...

At this point I agree - and they especially shouldn't entice neophytes with the deceptively simple "Bitcoin wallets are Free" download link that entails more than 24 hours syncing the entire behemoth block chain.

But now I've got Electrum and am just as happy as a little clam. 

[niko]

Those interested in scalability of Bitcoin can a have a look at
https://en.bitcoin.it/wiki/Scalability