Bitcoin Forum

Other => Beginners & Help => Topic started by: deepceleron on November 10, 2011, 09:24:04 AM



Title: HowTo: Jump-start Bitcoin installation download time with blockchain torrent
Post by: deepceleron on November 10, 2011, 09:24:04 AM
New users are often frustrated at the time required for a new Bitcoin installation to update before it becomes useable. This post outlines a faster way to get up and going if you are experiencing longer-than-expected delays.

When you install and run the Bitcoin client software for the first time, Bitcoin must download the complete Bitcoin blockchain from other Bitcoin clients sequentially in pieces starting at block 1. The blockchain is a record of all transactions that have ever taken place, and you must have the complete record before you are able to see your own balance and send payments. The transactions are grouped into cryptographically secure transaction blocks that were created about every 10 minutes since Bitcoin was launched in 2009. As of Dec 2013, there are over 270,000 blocks, over 12GB that need to download over the p2p network, which can take many hours or even days. The download speed is also limited by processing power, as your CPU must verify transaction signatures and the validity of past payments, and your hard drive must keep up with the intensive I/O access building the database where Bitcoin stores transaction information.

Bitcoin-qt 0.5.x-0.8.x screenshot:
http://we.lovebitco.in/img/bitcoin052ss-download-tooltiips.png

In the screenshot above, we can see by hovering over the network icon that this Bitcoin-Qt client has 71 network connections to other users running Bitcoin. The maximum if you are using an Internet router appliance or NAT without port forwarding will be eight connections. We can also see Bitcoin has downloaded all transaction blocks up to block 164944. If a transaction was sent to you now, it would be included in a future block - you would need to wait until that transaction is added to the blockchain and downloaded by your client in order to see an updated balance. There is one payment to this Bitcoin wallet in the screenshot, received around block 150000 - if Bitcoin had only downloaded blocks up to 100000, this transaction wouldn't appear in the client at all.

Your client must be making some connections, and should start downloading blocks within a minute of starting. If this does not happen at all, you need to remedy this problem; typically it is a firewall or antivirus program blocking Bitcoin's connection to the Internet, or something is misconfigured on your network. If you are limited to eight connections, you can improve this by forwarding port 8333 in your Internet router to the computer running Bitcoin. If your Bitcoin is well connected on high-speed Internet and has a fast CPU and SSD, a new installation can be ready in as few as four hours, but may take much longer if conditions are not optimal.

Fortunately, there is a method that is often faster.

A blockchain import feature was added to Bitcoin version 0.7.1+ to "bootstrap" or import an external blockchain file, while still performing a full validation on the Bitcoin blocks being added. Whenever Bitcoin is started, it searches the data directory for a file "bootstrap.dat", and if this is present, it will import all new blockchain blocks contained in the file. This procedure still requires the same CPU and disk resources, but can be faster if network speed through the p2p network is the limiting factor (which is often the case, Bitcoin only attempts connections to one other peer at a time to download blocks).

A torrent is maintained that contains blocks up to the last hard-coded checkpoint block in the client (currently block 250000, 9.1GB of blocks). Only blocks after this checkpoint will need to download off the peer-to-peer network. Download the torrent with this link:
http://sourceforge.net/projects/bitcoin/files/Bitcoin/blockchain/bootstrap.dat.torrent/download

More torrent instructions and links on this thread: https://bitcointalk.org/index.php?topic=145386.0

After completing the torrent download, you will need to copy the file bootstrap.dat to your Bitcoin data directory. To locate the default Bitcoin data directory on your OS, follow instructions on this link (https://en.bitcoin.it/wiki/Data_directory). When you restart Bitcoin, you should see the import progress dialog.

(Edit Apr 2013 - removed all direct download information except bootstrap torrent, Dec 2013, update blockchain and torrent sizes.)


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: mtminer on November 10, 2011, 03:18:19 PM
Thanks should get this stickied and put in the Bitcoin wiki.



Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: sd on November 11, 2011, 07:11:36 PM

Bad idea - Feeding your bitcoin client a blockchain this way makes it skip validating those blocks. They could contain anything.

Do it the long way, it's the only way to be sure. Yes I know waiting for it to finish can be annoying.



Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: Coinabul on November 11, 2011, 07:13:50 PM
New users should instead use a reputable e-wallet while they wait for their client to download the blockchain.

Now to find a reputable e-wallet...


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: DeathAndTaxes on November 11, 2011, 10:04:57 PM

Bad idea - Feeding your bitcoin client a blockchain this way makes it skip validating those blocks. They could contain anything.

Just run the checkblocks command line argument to force a complete validation of block chian.

Code:
bitcoin -checkblocks


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: deepceleron on November 11, 2011, 10:52:57 PM
New users should instead use a reputable e-wallet while they wait for their client to download the blockchain.

Now to find a reputable e-wallet...

You mean one that has been around for a year and a half, has thousands of happy users, and has been featured in the press, then you might think it is reputable? mybitcoin was reputable until they disappeared.


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: pogi23 on November 12, 2011, 01:15:10 AM
So that's why the delay keeps getting slightly longer every time I boot up the client (been thinking it was a hardware issue).  Granted it's only about 56 seconds... I can only imagine how long it'll take when the blockchain is near its end of life.


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: DeathAndTaxes on November 12, 2011, 01:39:35 AM
Are you on current block?  If not then stop shutting it down until you are current.  I don't notice any delay starting the client.


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: ButterKvothe on November 12, 2011, 04:43:36 AM
Thanks!


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: pogi23 on November 12, 2011, 06:53:47 AM
Are you on current block?  If not then stop shutting it down until you are current.  I don't notice any delay starting the client.

Oh I only restart if I absolutely need to (once every 3 weeks or so).  So it seems like the client is downloading  the whole blockchain every time I restart.  Fifty six seconds is no big deal, really.


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: DeathAndTaxes on November 12, 2011, 02:56:16 PM
Are you on current block?  If not then stop shutting it down until you are current.  I don't notice any delay starting the client.

Oh I only restart if I absolutely need to (once every 3 weeks or so).  So it seems like the client is downloading  the whole blockchain every time I restart.  Fifty six seconds is no big deal, really.

The client isn't downloading the entire blockchain when you restart.  If it did it would be more like 50-60 minutes not 50-60 seconds.


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: pogi23 on November 12, 2011, 06:12:40 PM
Are you on current block?  If not then stop shutting it down until you are current.  I don't notice any delay starting the client.

Oh I only restart if I absolutely need to (once every 3 weeks or so).  So it seems like the client is downloading  the whole blockchain every time I restart.  Fifty six seconds is no big deal, really.

The client isn't downloading the entire blockchain when you restart.  If it did it would be more like 50-60 minutes not 50-60 seconds.
Thanks!  Wasn't sure what it was doing.


Title: Howto: Missing transactions? New Bitcoin installation? Do you have all blocks?
Post by: deepceleron on November 18, 2011, 12:16:32 PM
Is this sticky-worthy? I could reply to four noobs a day with links to this post, it seems.


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: ThomasV on November 18, 2011, 12:26:34 PM
New users should instead use a reputable e-wallet while they wait for their client to download the blockchain.
Now to find a reputable e-wallet...

it is also possible to use Electrum: http://ecdsa.org/electrum
with this kind of lightweight client, you do not have to download the blockchain, and you do not have to trust a third party with your private keys


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: theymos on November 20, 2011, 10:33:40 AM
Just run the checkblocks command line argument to force a complete validation of block chian.

Code:
bitcoin -checkblocks

This doesn't do all of the normal checks on the blocks. That switch is designed to detect database corruption, not to check validity of the block chain. Whoever provides the chain still needs to be trusted absolutely.

Newbies should not be encouraged to do this. (And even if they are, they should at least use the official, signed blockchain files (http://sourceforge.net/projects/bitcoin/files/Bitcoin/blockchain/).)


Title: Re: Noobs: How to jump-start Bitcoin installation with a direct-download blockchain
Post by: deepceleron on November 20, 2011, 01:54:20 PM
Newbies should not be encouraged to do this. (And even if they are, they should at least use the official, signed blockchain files (http://sourceforge.net/projects/bitcoin/files/Bitcoin/blockchain/).)

If newbie = someone who has a hard time even locating the Bitcoin data directory, or someone that can't figure out how to log into their router, then yes, patience is best, as opposed to messing with the data files directly - that is why my first paragraph gives users an opt-out if they are not computer-savvy.

I can only personally attest that the blockchain I have posted is exactly as was downloaded by a fresh Bitcoin install, and I have posted file checksums, so that even if my hosting account on secureserver were compromised and the files replaced with a facsimile, one would be able to validate the contents with md5sum (http://www.pc-tools.net/win32/md5sums/) (a signature relies on you already trusting the signer). If someone were to write a hash validator or a blockchain comparator, you would find mine unaltered.


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: EightBit on February 05, 2012, 05:05:53 AM
I agree, the new user sync can take a long time. 

I just installed my first client.  It took about 18 hours to download the complete transaction history.  I am using the current Mac client and am on a usually very fast cable modem.  The client made 8 connections.



Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: LoWang on February 09, 2012, 11:41:24 PM
I suggest that the instructions on a bitcoin homepage (https://en.bitcoin.it/wiki/Getting_started) should be changed a bit from
"it can take from half to a few hours)." to "it can take few hours or half a day") ;-) New users could be really frightened by the slow network synchronization and it also is not always possible for everybody to leave the computer running for that long, because you need to leave office and go home or the other way and from my experience it can cause problems like database corruption (which happened to me (https://bitcointalk.org/index.php?topic=63037.msg735618#msg735618)) if you hibernate or standby your laptop during this process.

Also it may not be a bad idea to have a link to whole block chain downloads somewhere there in "getting started"...


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: JayleeBird on February 10, 2012, 12:01:04 AM
This is the only thing that bugged me when I first started with Bitcoin. It took nearly a week for the blockchain to be downloaded.


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: LoWang on February 10, 2012, 06:01:18 PM
LOL that is crazy... in that case I cannot really imagine too many "normal" non geeky people be interested in using Bitcoins :-\


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: LoWang on February 10, 2012, 07:30:42 PM
4hours? And when was it?


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: deepceleron on February 10, 2012, 07:53:05 PM
4hours? And when was it?
4 hours. Yesterday. That was using my local network and a decent desktop computer with a SSD to process the incoming blocks, so that is about the minimum time required now. If you have an older laptop on non port-forwarded wi-fi, it is of course going to take longer.


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: LoWang on February 10, 2012, 09:20:32 PM
very interesting, but your theory is wrong. You obviously did not read "what happened to me". Check the link in my first post here in this thread. There are other unknown factors in play because neither disk or network or CPU is overloaded and it can take really forever :-\


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: GlooBoy on February 11, 2012, 08:22:31 PM
My first install took about 5-6 hours once I figured that client was 'crashing' during reboots. Was also on a wi-fi at first(not recommended)

If you are not planning on keeping it running until the entire blockchain comes in I would recommend going into options and un-checking the 'Start bitcoin on windows start-up'

She is a fickle bitch when downloading the blockchain at first, and has never started up/shut down, with option checked, well on my slower comps.

Some I know that has trouble with the blockchain, this helped them out. You might need to re-download the entire chain tho.

Some of these older comps have less than a gig of space left on the hard drive, some just don't have the space left.

When starting up for a friend, I grab a copy of my blkindex.dat and blk0001.dat on a thumb drive and copy it to their bitcoin app folder. New installs only.

Just my two-bit cents. And Good Luck.



Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: LoWang on February 11, 2012, 10:29:52 PM
Man thank you for confirming that I am not the only one with these problems. But I see you read probably only the first post, thus did not notice that I have fixed it already ;-)

Anyway I am a total newbie with no BTC, but this just confirms my fear that no normal non geeky people will ever be interested in using this currency when this kind of things happen with an official client :-\


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: deepceleron on May 30, 2012, 07:00:12 PM
I'll give this thread a bump since there are currently several noob posts experiencing Bitcoin and it's slow initial startup - questions answered here. First post updated.


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: 4EverMaAT on July 04, 2012, 07:42:48 AM
I was unaware about adding BitCoin to the firewall (port forwarding?) so that it wouldn't have the 8 connection limit.  Now I am getting 16-20 active connections and it appears to be growing.  If this was done prior to the initial blockchain download (where I maxed out at 8 connections), I might have downloaded the blockchain in a few hours instead of it taking days.  I have dual core windows VPS with 100 Gbps connection; I was wondering WTF was taking so long to download blockchain. 

I'll have to watch now how much quicker transactions take to validate (used to be 1 hour for micro transactions <0.01 BTC)


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: deepceleron on July 17, 2012, 11:11:25 PM
Newer blockchain files added, an uninterrupted blockchain download over local network from updated client. Integrity verified with --checkblocks 0 and --checklevel 6 (no changes). I compressed this one on an x64 machine using 7zip single threaded with dictionary size 256M and word size 273, after 14 hours of pagefile thrashing, I had smashed 2.67GB of blockchain into 1.48GB.


Title: Re: HowTo: Jump-start or repair Bitcoin installation with direct-download blockchain
Post by: deepceleron on November 07, 2012, 05:39:11 AM
When I started this thread, the blockchain was 1GB, now the datadir files are over 4GB!

Files updated to recent block height, similar procedure employed to build this blockchain as the bootstrap torrent uses, resulting in an identical dat structure as other methods, without orphan blocks. I also separated the files so if you have need or use for just one, you can grab it by itself.


Title: Re: HowTo: Jump-start or repair Bitcoin installation with direct-download blockchain
Post by: Frisoso on January 14, 2013, 07:41:31 AM
Any plans to upload a more recent version of the blockchain? Currently on block 206847 with 9617 blocks remaining which go VERY slow...
If it keeps going like this I suspect It'll take at least another day...


Title: Re: HowTo: Jump-start or repair Bitcoin installation with direct-download blockchain
Post by: deepceleron on January 14, 2013, 09:24:04 AM
Yes, I have some half-baked php I've been writing to captcha, rate limit downloads, and only let one file download per user ever, but haven't put it all together and put it up.


Title: Re: HowTo: Jump-start or repair Bitcoin installation with direct-download blockchain
Post by: mmortal03 on January 29, 2013, 11:42:32 PM
New import blocks procedure - Bitcoin version 0.7.1+:
A blockchain import feature was recently added to Bitcoin to "bootstrap" or import an external blockchain file, which allows one to also perform a full validation on the Bitcoin blocks being added. As of now, this procedure still requires CPU and disk resources and can take several hours; however all transaction signatures in the blockchain file are verified by your own client, increasing your security. You do not need to download the blkindex.dat file above, but can use the other dat files to import all blocks up to 206000.

Download just the first or both of the blk000x.dat files, and extract them to your disk (the Windows C: drive in the example below). Then to launch Bitcoin, open "Command Prompt" on Windows, or a Linux terminal/console/TTY. You must change to the program directory of Bitcoin and type in the command below to specify the block import file location.

(this step only required on Windows):
cd "C:\Program Files (x86)\Bitcoin" (remove the ' (x86)' on 32-bit Windows)

bitcoin-qt -loadblock=c:\blk0001.dat -loadblock=c:\blk0002.dat

Other blockchain links:

http://sourceforge.net/projects/bitcoin/files%2FBitcoin%2Fblockchain/ (1.1GB zip from Mar 10 2012, up to block 170000)
http://eu2.bitcoincharts.com/blockchain/ (now a link to 2.3GB bootstrap.dat file)
https://bitcointalk.org/index.php?topic=117982 (bootstrap-only torrent, longer import process)

Making sure that I understand you correctly, with Bitcoin version 0.7.1+, there is now no reason (even for security purposes) to use the bootstrap.dat method as offered by jgarzik at that last link, because now any external blockchain files used are fully validated when following these quoted steps?


Title: Re: HowTo: Jump-start or repair Bitcoin installation with direct-download blockchain
Post by: deepceleron on January 30, 2013, 02:24:18 AM
New import blocks procedure - Bitcoin version 0.7.1+:...

Making sure that I understand you correctly, with Bitcoin version 0.7.1+, there is now no reason (even for security purposes) to use the bootstrap.dat method as offered by jgarzik at that last link, because now any external blockchain files used are fully validated when following these quoted steps?
Rather, there is no reason not to use it other than time required, which is still 6+ hours to import the blocks after the time spent downloading (making total time near that of normal sync on a good connection), and the fact that the method the developers hoped to create - simply drop a bootstrap.dat file in the datadir and it will be automatically imported - will only will import up to 2.0GB of blocks due to a bug (https://github.com/bitcoin/bitcoin/issues/1951).

Here is a summary of methods, using the current 0.7.2 client:

1. From an already-synced copy of Bitcoin (which must have been last closed with the -detach option), copy all blockchain files (BLK000x.DAT) plus the matching index file (BLKINDEX.DAT) to your data directory [fast, no re-verification of block data]
(This is what I offered in the first post, but I've temporarily removed my files due to DOS attempts. The current best drop-in blockchain is this 2.1GB torrent (https://bitcointalk.org/index.php?topic=94881.0)),

2. From an already-synced copy of Bitcoin, copy just the blockchain files (BLK000x.DAT), and use the -loadblock=BLK0001.DAT -loadblock=BLK0002.DAT -loadblock=BLK0003.DAT option to import any or all block data into your fresh Bitcoin, and re-verify the contents. [slow, safe, verifies all data],

3. Get the bootstrap.dat that has been distributed (2.3GB torrent (https://bitcointalk.org/index.php?topic=117982) or  direct download (http://eu2.bitcoincharts.com/blockchain/) ), and drop it in your data directory before you start Bitcoin, to import only 2GB of blocks up to 189205 (due to bug). Basically does the same thing to import data as loadblock command. [slow, safe, only 40% of current size]

3a. Optionally split the bootstrap torrent into two parts, and import both parts with the -loadblock option to use the complete data (instructions here (https://bitcointalk.org/index.php?topic=117982.msg1303629#msg1303629)). [slow, safe, only 50% of current size]


Bitcoin 0.8.0+ fixes the bootstrap.dat import bug and will probably coincide with a new larger bootstrap torrent. It also improves the initial download experience with it's complete refactoring of how data is downloaded and stored.


Title: Re: HowTo: Jump-start Bitcoin installation download time with blockchain torrent
Post by: deepceleron on April 03, 2013, 05:23:31 PM
This is now a noob thread for torrent talk, new versions of Bitcoin have a different database format than previously posted, and importing the bootstrap torrent is safer than using an already-compiled block database from someone else.


Title: Re: Howto: New Bitcoin installation? Slow block downloading? Missing transactions?
Post by: Coiner505 on April 24, 2013, 02:52:57 PM
This is the only thing that bugged me when I first started with Bitcoin. It took nearly a week for the blockchain to be downloaded.
Ditto here. I am on a dual-core MacBook Pro with 8GB of RAM, and this is headed for a week, easily. v0.8.1-beta. It got to 80% within some hours, then started putting along, slower and slower. It's now at 91% of the almost 233k blocks after several days, not 24x7, maybe 16x7. I've also done the 8333 port forward on both TCP and UDP and restarted the client and can't get >8 connections. I've tested the forward: I can telnet to my public IP from the outside on 8333, and it connects.

BitCoin has 1GB virtual memory allocated and is only using 200MB of resident RAM. The disk is not overloaded at all. This is all without any of the tricks posted here, like the bittorrent download. I'm at 91%. BitCoin seems to be doing its own bittorrent, effectively (maybe literally). I should now download a different, 2.5GB torrent compressed (?), to prime BitCoin, when I'm at 91%? And the prime may only work for the first 2 GB of raw data?

My ~/Library/Application Support/Bitcoin is 5.0 GB right now.

This is mostly FYI as a datapoint to developers, as I will probably switch to a different client, unfortunately. Whatever tricks are posted here need to basically be rolled in as the default, coded behavior of the app. Good luck, seriously. I'm a developer and I know the challenges. This is obviously a growing-pain stage.

Bitcoins are really designed so that every client must "know" and verify every single transaction since the beginning of time? That, um, doesn't scale.  :-\

All right, I'll try the torrent. Seems silly.


Title: Re: HowTo: Jump-start Bitcoin installation download time with blockchain torrent
Post by: kimihiro on June 14, 2013, 09:05:00 AM
so if i want to add extra mining rig, do i have to "download" the synchronizing again?


Title: Re: HowTo: Jump-start Bitcoin installation download time with blockchain torrent
Post by: deepceleron on June 14, 2013, 12:58:26 PM
so if i want to add extra mining rig, do i have to "download" the synchronizing again?
If you are pool mining, you don't need Bitcoin with a blockchain installed at all.


Title: Re: HowTo: Jump-start Bitcoin installation download time with blockchain torrent
Post by: mprep on June 14, 2013, 01:01:41 PM
so if i want to add extra mining rig, do i have to "download" the synchronizing again?
If you are pool mining, you don't need Bitcoin with a blockchain installed at all.
That's probably one of the bigger pooled mining pluses.


Title: Re: HowTo: Jump-start Bitcoin installation download time with blockchain torrent
Post by: kimihiro on June 14, 2013, 01:48:07 PM
so if i want to add extra mining rig, do i have to "download" the synchronizing again?
If you are pool mining, you don't need Bitcoin with a blockchain installed at all.

can you explain in "noob" language please..
because this is really new for me

right now, i'm trying to build my 1st rig, and i'm using saphire 7870xt
but, since 2 days ago it keep sync untuill today left like 60 weeks behind
and it worry me a lot, if i put another rig, i have to redo the sync again with the new one

i'm joining slush btw, and using a block chain wallet