[BETA] Bitcoin blockchain torrent

[Richy_T]

We will work around this in the next release of the bitcoin client.

Theoretically it would be sufficient to replace "unsigned int" and "fseek" by "fpos_t" and "fsetpos". But because of "CAutoFile" and "FILE *" mixing the actual fix may be quite complex.

It's been a while since I ran into a similar issue myself but if I recall correctly, it was pretty much a drop-in fix.

[rini17]

I stopped seeding. There were no leechers for few days now, problem discovered is not fixed, and I'm solo mining (50+ connections). 

That may be problem with your client or firewall. I'm seeding only since yesterday evening and now it has > 300M uploaded already, using Deluge 1.3.5 .

[jgarzik]

I stopped seeding. There were no leechers for few days now, problem discovered is not fixed, and I'm solo mining (50+ connections). 

That may be problem with your client or firewall. I'm seeding only since yesterday evening and now it has > 300M uploaded already, using Deluge 1.3.5 .

Under ideal conditions, where you have enough seeders to make each new download saturate the client's download link, that produces a situation where you have several seeders and many periods of idle time, punctuated by short bursts of network activity.

I'm definitely seeing daily downloads here, but we could use some more seeders nonetheless.

Bittorrent also helps more for unexpected bursts of network traffic -- like those accompanying a new bitcoin version release, for example.

[ibno]

Two ideas for improvement:

1) Compression: this will easy shave off 25% of the file size. My 2.6 G blockchain file uncompressed became 1.9 G compressed with zip.

2) Split it up in multiple files. If I stop using bitcoin for a couple of months and then start using it again I might still have a large chunk of the blockchain left on my computer. I really only need the new block. The current mainline has a new way of storing blockfiles (.bincoin/blocks/blk000?.dat instead of .bitcoin/blk000?.dat) each with size 128 MB (instead of the much larger .bitcoin/blk000?.dat files). These smaller, 128 MB, files would be nice to have available via a torrent, making it possible to limit the download to only the latest blocks.

[K1773R]

Two ideas for improvement:

1) Compression: this will easy shave off 25% of the file size. My 2.6 G blockchain file uncompressed became 1.9 G compressed with zip.

2) Split it up in multiple files. If I stop using bitcoin for a couple of months and then start using it again I might still have a large chunk of the blockchain left on my computer. I really only need the new block. The current mainline has a new way of storing blockfiles (.bincoin/blocks/blk000?.dat instead of .bitcoin/blk000?.dat) each with size 128 MB (instead of the much larger .bitcoin/blk000?.dat files). These smaller, 128 MB, files would be nice to have available via a torrent, making it possible to limit the download to only the latest blocks.

im sry but zip compression totally sucks...
read this fora nice list: http://www.maximumcompression.com/index.html

[ibno]

Two ideas for improvement:

1) Compression: this will easy shave off 25% of the file size. My 2.6 G blockchain file uncompressed became 1.9 G compressed with zip.

2) Split it up in multiple files. If I stop using bitcoin for a couple of months and then start using it again I might still have a large chunk of the blockchain left on my computer. I really only need the new block. The current mainline has a new way of storing blockfiles (.bincoin/blocks/blk000?.dat instead of .bitcoin/blk000?.dat) each with size 128 MB (instead of the much larger .bitcoin/blk000?.dat files). These smaller, 128 MB, files would be nice to have available via a torrent, making it possible to limit the download to only the latest blocks.

im sry but zip compression totally sucks...
read this fora nice list: http://www.maximumcompression.com/index.html

There's allot of pseudorandom data in the blockchain that can't be compressed, so I don't know if there will be so much difference in the file size result between different algorithms in this case, maybe we can shave off another 100 MB with a better one. It would be interesting to see a comparison you linked to with the blockchain.

[deepceleron]

(Edit - obsolete, Bitcoin issue fixed)

I have a method for splitting the torrent so it can be successfully imported to completion. It can be split into two files using the GNU split utility with this command:

split --bytes 2147570829 bootstrap.dat

This creates two files:

10/28/2012  10:12 AM     2,147,570,829 xaa
10/28/2012  10:12 AM       344,200,733 xab

sha256sum x??
fec4aa1d5a1d07b832cb22a353a689b6bf7fcb09e4822a19d9051160f195a8ef *xaa
fc07e8d67a39152a5bf1920b3bcd99d50515817ba77a28bd7d780d4cc3aa3938 *xab

This splits the torrent at block 189205, the last block that Bitcoin successfully imports out of the full 2.32GB bootstrap.dat. You can then import all torrent blocks with bitcoind or bitcoin-qt with the command:

bitcoind -loadblock=xaa -loadblock=xab -printtoconsole

(printtoconsole so you can watch the progress of block import for several hours. Use the full path to the files if needed.)

Split should be included in any GNU/Linux distribution plus MacOS. A standalone windows compile of the utility is hosted here: split.exe (72,192 bytes) (or get the GNU coreutils and dependencies full packages at http://gnuwin32.sourceforge.net/packages/coreutils.htm).

I found another bug/undocumented oddity; Bitcoin first processes -loadblock commands, then it looks for a bootstrap.dat file in the datadir. That means the above command will attempt to load bootstrap.dat a second time if it is located in the datadir (but will skip through all the repeated blocks quickly).
BTW: How do I compress the blockchain to 56% of the original size? Have 64 bit 7-zip and lots of RAM:

[Syke]

Blocks are oddly sized

Makes sense.

and their locations within the block files are pretty much arbitrary.

Even for someone starting from block 0?

Yup, it is a consequence of the odd block sizes.  Reading a single block may involve several disk reads.  And getting the file offset to read will also take several reads in the index file.

Oh, they're getting stored in a simple flat file. Well that's easy, download blocks from random peers and store them in temporary files. Then when the next block needed has been downloaded, copy it to the final block file.

[deepceleron]

I have created a method and file for splitting the torrent so it can be successfully imported to completion. It can be split into two files using the GNU split utility with this command:

split --bytes 2147570829 bootstrap.dat

This creates two files:

10/28/2012  10:12 AM     2,147,570,829 xaa
10/28/2012  10:12 AM       344,200,733 xab

sha256sum x??
fec4aa1d5a1d07b832cb22a353a689b6bf7fcb09e4822a19d9051160f195a8ef *xaa
fc07e8d67a39152a5bf1920b3bcd99d50515817ba77a28bd7d780d4cc3aa3938 *xab

This splits the torrent at block 189205...

I made a change to the pynode mkbootstrap.py script that generates the torrent, so it splits block files at the same block as above; the hashes of the split output files are identical to the manual split method.

It's a bit less clever then tracking file sizes and auto-splitting for multiple files <2GB.

patch code:

Code:

--- mkbootstrap.py     2012-11-03 19:31:31.000000000 -0700
+++ mksplitbootstrap.py 2012-11-04 11:33:55.321381032 -0800
@@ -39,8 +39,8 @@
 chaindb = ChainDb.ChainDb(SETTINGS, SETTINGS['db'], log, mempool, netmagic,
                          True)
 
-outf = open('bootstrap.dat', 'wb')
-
+out1 = open('bootstrap.001', 'wb')
+out2 = open('bootstrap.002', 'wb')
 scanned = 0
 failures = 0
 
@@ -62,8 +62,12 @@
        outhdr = netmagic.msg_start
        outhdr += struct.pack("<i", len(ser_block))
 
-       outf.write(outhdr)
-       outf.write(ser_block)
+       if height > 189205:        # start writing second file
+               out2.write(outhdr)
+               out2.write(ser_block)
+       else: 
+               out1.write(outhdr)
+               out1.write(ser_block)
 
        scanned += 1
        if (scanned % 1000) == 0:

Is it time for a torrent with more blocks (in this method?) Block 206000 is a nice recent number and makes for a 2GB + 1.5GB file.

Note that the first file can't be "bootstrap.dat", because Bitcoin actually imports this last after any other manual import statements. Patch Bitcoin to auto-import any "bootstrap.dat", "bootstrap..001", "bootstrap..002", ... in that order if they exist, instead of fixing >2GB import?

I also think the torrent should be immediately-useable blk0001.dat and blk0002.dat files including a blkindex.dat. That way one could either wait the 6+ hours to import and verify blocks (which is little improvement over p2p), or just drop the whole torrent into a Bitcoin data directory. The "split point" should correspond with the actual (earlier) block at which the Bitcoin client starts blk0002.dat when importing these cleaned blocks (188529 blocks). The index could be a spent-transaction-pruned version like luke-jr created here.

[Pieter Wuille]

blkindex.dat is not included in the torrent because that conflicts with Bitcoin's trust model.

If you're going to trust a single someone to do the indexing for you, you may as well trust the network (far less trust required) and run an SPV node (like Multibit). The purpose of the reference client is to be zero trust - not relying on any trusted data. If someone would distributed a faulty (intentional or not) set of block files with included index, and many people - in particular miners - would start using this data without verification, they could end up in a block chain fork if a transactions was included in a legitimate block that conflicts with their index.

Also, 0.8 will switch to an entirely different database layout (there won't be a blkindex.dat anymore), and validate blocks much faster (in particular when they're read from the local disk).

[jgarzik]

Additionally, it should be noted that sipa has created a pull request which should avoid seeking, thus avoiding the 2GB problem:

     https://github.com/bitcoin/bitcoin/pull/1962

[Sukrim]

In checkpoints.cpp there is another checkpoint at 210000 - please update the torrent!

By the way, from time to time people still are using this (I'm a more or less permanent seeder here), so I guess the experiment is working.

[jgarzik]

This torrent will be updated soon, to sync with a new checkpoint in the upcoming 0.8 release.

[deepceleron]

I'll move this over here for more discussion of Torrent 2.0

The blockchain checkpoint in this release has been updated to 216116. I created a bootstrap.dat of this height and successfully imported it in 37 minutes with bitcoin-qt on Win7 64 bit.

02/10/2013  08:44 PM     4,855,459,871 bootstrap.dat
SHA256: bf658c7055b733bfc15ea167f298c5599b89d220b14dbe7c8ef20b18e468c451 *bootstrap.dat

Code:

2013-02-11 06:35:35 SetBestChain: new best=00000000000000f97afdf8ccba49919bb998313ec67e3654798b86a3f6631c1e  height=216115  work=714383275301559958540  tx=11010714  date=2013-01-11 10:59:18
2013-02-11 06:35:35 ProcessBlock: ACCEPTED
2013-02-11 06:35:36 SetBestChain: new best=00000000000001b4f4b433e81ee46494af945cf96014816a4e2370f11b23df4e  height=216116  work=714397232223717792651  tx=11011160  date=2013-01-11 11:11:30
2013-02-11 06:35:36 ProcessBlock: ACCEPTED
2013-02-11 06:35:36 Loaded 216116 blocks from external file in 2218342ms

I will offer this as a torrent and also give direct-download links to those who wish to jump-start and seed such a torrent, since my seeding speed is limited. The question is, should an official torrent going forward be a compressed file? Using the highest compression preset with xz/lzma (installed with most distros, opens with 7-zip or MacOS GUI tools) saves 2GB of downloading, so I say yes:

Also, bootstrap.dat is too big for FAT32, but not the compressed version. Bitcoin 0.9+ could even (feature request) open a compressed bootstrap (lzma is a stream container) directly if standardized.

The semi-official bootstrap.dat torrent is about to be updated, as it is for each bitcoin release.  The 2GB issue has already been raised in that thread, and the FAT32 4GB issue would also be a good one to raise.

Ideally we would like everyone to seed the same torrent -- it's the same data, as guaranteed by bitcoin (as well as torrent) hashes.

"The 2GB issue", failure to import blocks after 2.0GiB on 32 bit builds, was reported by me, so I guess I'm the one that "raised" it here. Now that Bitcoin can actually use a bigger torrent and 0.8.0 has a higher checkpoint, it's finally time for a new torrent!

You should get the same 4.8GB bootstrap.dat SHA256 at that height, if you don't, we've got a big problem!

The only thing that would make a torrent I create less than "semi-official" would be if I make a compressed torrent without consensus. I am slowly convincing myself even more that a compressed torrent is preferable though. Without compression you are downloading the same amount of data as normal p2p. The compressed binary would only be repeatable on the exact xz/lzma/7zip build version and settings, but I think there are probably four people total that have run mkbootstrap anyway, so this is not important. It takes about an hour to smash the blockchain down to 60% the size at the extreme settings I've used over many trials to optimize settings.

Compressed-Pros:
-2000+ MB of uploading and downloading saved for every user,
-2000+ MB less storage used when seeding,
-won't cause problems if torrent HDD is FAT32 (under 4.0GB for now)

Compressed-Cons:
-Not as simple to use, end-user must decompress with third-party utility (although 7-zip is common and opens xz),
-More work creating torrent (doesn't matter to end-users),
-Cannot "update" torrent by simply replacing data file with newer version with additional blocks (likely to be a rare practice anyway, I am the only one seeding the old torrent right now).

Here is example compression, both require about an hour (but decompress in minutes):

xz utils 5.0.4/Win64, 3GB+ RAM required to compress (2,780,285,148 bytes)
xz --compress --keep --format=xz --check=sha256 --verbose --lzma2=dict=256MiB,nice=273,mf=bt4 bootstrap.dat

7-zip GUI win64 - 6GB+ RAM required to compress (2,769,830,975 bytes)
Format: 7z, Compression Level: Ultra, Compression method: LZMA, Dictionary size: 384MB, Word Size 273

[Richy_T]

Compressed-Pros:
-won't cause problems if torrent HDD is FAT32 (under 4.0GB for now)

If this is an issue, you probably would want to start using multi-part compression (which is another pro fwiw).

[notme]

Maybe trackerless will be a #fail, but let's see how it goes.

Yep, your magnet link didn't get me anything.  I had to add some open trackers until I found some peers.

Anyway, I'll be seeding.

Here is a magnet link with a few good trackers added:

Code:

magnet:?xt=urn:btih:0bb0521942f586ed96203c6f4d136324756f8a9a&dn=bootstrap.dat&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.istole.it%3A80

[jgarzik]

Currently beta-testing the following updated torrent:

SHA256 bf658c7055b733bfc15ea167f298c5599b89d220b14dbe7c8ef20b18e468c451

magnet:?xt=urn:btih:6fe493ba606847eac163baf35aae9db319735482&dn=bootstrap.dat&tr=udp://tracker.openbittorrent.com:80&tr=udp://tracker.publicbt.com:80&tr=udp://tracker.ccc.de:80&tr=udp://tracker.istole.it:80

or http://gtf.org/garzik/bitcoin/bootstrap.dat.torrent (please use magnet link, if possible)

When sufficient seeds appear, a new PGP-signed forum post will be made, obsoleting this thread.

[Sukrim]

Oh, for anyone already seeding/downloading the old torrent:
Just remove the seed, open the new torrent and add it at the same location. Your torrent client then verifies the already existing blocks (~first 51.3% of the new torrent) and then loads the rest from other peers.

[deepceleron]

I was able to just drop the 4.7GB bootstrap.dat I already created into the torrent and seed, which I've been doing since 20 minutes after the post - and there are still the same number of seeders.

[Sukrim]

Well, I see 15 seeders and 9 leechers on the current (larger) bootstrap.dat file. Maybe you are not using trackers and/or mainlineDHT or using some restrictive firewall? Also download of the torrent was reasonably fast (= ~380 kB/s on average for the remaining 2.2 GB).

As long as BitcoinQT cannot handle a compressed bootstrap.dat.xz file, I'd rather waste some bandwidth instead of confusing newies even more.