[ANN] Bitcoin blockchain data torrent

[jonathan]

same story on my box. This is the torrent that never sleeps lately.

[kuzetsa]

This is special to our use case:  bootstrap.dat is essentially an append-only file.  Blocks are simply concatenated onto the end.

((...snip...))

The next torrent, a few months from now, will have the same first 2,491,771,562 bytes.

Thus, to bittorrent, the next torrent will simply appear to be a truncated / not fully downloaded bootstrap.dat.  Bittorrent is built to fill in the missing pieces of a file, so that is what it does here

I just copied that over from the beta thread

... wanted to point out that it DID work for me

(I was able to use the older/smaller/earlier bootstrap.dat, and then my bittorrent client was able to smartly "fix" the new torrent... like...
the smaller file is basically the same & I only needed to download the missing end. This procedure results in the same updated bootstrap.dat)

... it should continue to work for the next new/larger/updated torrent just the same.

thanks jgarzik, you rock

[dooglus]

same story on my box. This is the torrent that never sleeps lately.

Am I doing something wrong?  I've been seeding for over a month, but have had very little interest in it:



This is in 'transmission'.

[deepceleron]

Am I doing something wrong?  I've been seeding for over a month, but have had very little interest in it:



This is in 'transmission'.

Me: Uploaded 23.05 GB

When there are 40 seeds (there were up to 60 earlier), you only need to give one downloader 1/40th of 4.7GB, and it will be less if you have a slower connection then the fastest seeds. The three peers I have connecting from Netherlands, Russia, and Iran are barely getting data, they might be mistakenly using their 2400 baud modems to get the torrent.

Make sure you have encryption enabled (many will block if you don't), and the TCP/UDP ports of bittorrent and DHT port forwarded.

[jgarzik]

Am I doing something wrong?  I've been seeding for over a month, but have had very little interest in it:

There tends to be bursts, with idleness following.  There are many seeders, so download-then-idle is quite fast.

It has the most impact in handling unintended bursts, like when bitcoin sees a lot of new press attention.

[dooglus]

There tends to be bursts, with idleness following.  There are many seeders, so download-then-idle is quite fast.

It has the most impact in handling unintended bursts, like when bitcoin sees a lot of new press attention.

My upload speed is slow and my latency is horrible.  I expect most peers take one look at me and decide I'm not worth bothering with. 

[grubles]

seeding on my glorious 5 mb/s uplink.

[dserrano5]

Am I doing something wrong?  I've been seeding for over a month, but have had very little interest in it:



This is in 'transmission'.

I use transmission too and it's not rare for me to see other files uploading at hundreds of Kb/s, but I don't see a lot of interest in bootstrap.dat either. However there's little I can do besides seeding so…

[takagari]

Downloading this torrent file now and uploading to my Gps seedbox.

Enjoy dont rape me too hard

EDIT:
Nope I'm not. my server won't allow public torrents

[dieguito]

I have an up-to-date copy of the blockchain.  Is there an easy way to generate bootstrap.dat from it so I can seed it without having to download the whole thing again?

This script is used to generate bootstrap.dat:
https://github.com/jgarzik/pynode/blob/master/mkbootstrap.py

It requires an up-to-date pynode chain database.

In theory, someone could write a script that processes bitcoind's $DATADIR/blocks directory into bootstrap.dat, but no one has written that yet.

Hi Jeff,

I create some quicks steps to get this from the command line in Ubuntu / Debian:

Code:

$ sudo aptitude install rtorrent
$ rtorrent -n -d ~/.bitcoin/ "magnet:?xt=urn:btih:6fe493ba606847eac163baf35aae9db319735482&dn=bootstrap.dat&tr=udp://tracker.openbittorrent.com:80&tr=udp://tracker.publicbt.com:80&tr=udp://tracker.ccc.de:80&tr=udp://tracker.istole.it:80" 

I'm not sure if the bootstrap.dat needs to go in .bitcoin or in .bitcoin/blocks/.

All the best,
D

PS: A little OT but I see pynode is a listen only client is there something in python like pynode that handles all bitcoin operations?

[tkbx]

This is extremely convenient, when I install bitcoin-qt on another device, it takes about an hour instead of nearly a week.

[romerun]

for some reasons, I get 15 connections after using this bootstrap thing versus 8 if starts from scratch

[plonk420]

it's a pity the torrent isn't 7zipped...

could the next iteration of it be compressed?

edit: i wouldn't mind seeding it on my seedbox, either.

[jgarzik]

it's a pity the torrent isn't 7zipped...

could the next iteration of it be compressed?

Feedback on this is welcomed.  Generally, not all platforms have an easy time getting up and running with 7zip.  Users in the past requested something directly usable with bitcoind.

Opinions differ, and maybe if it saves a gigabyte or two, it would be worth it.

Some users also complain about the doubling of disk space required -- for both compressed and uncompressed copies.

[Sukrim]

I would keep it uncompressed for the time being - it would be great by the way if bitcoin-qt would delete the file after a successful import automatically, I'm sure that some still keep it around unnecessarily. I see the issues with it suddenly "disappearing" but I guess the benefit of not wasting space outweighs it.

I don't care if I seed now ~4.5 GB or ~3 GB, as long as compression doesn't change the file size really significantly (meaning 5 instead of 50 or 500 GB) I rather consider it a burden than helping. There are download sources for compressed blockchain files already on the web anyways.

With 0.8.2 around the corner, can we expect a new check point + torrent file by the way?

[deepceleron]

it's a pity the torrent isn't 7zipped...

could the next iteration of it be compressed?

Feedback on this is welcomed.  Generally, not all platforms have an easy time getting up and running with 7zip.  Users in the past requested something directly usable with bitcoind.

Opinions differ, and maybe if it saves a gigabyte or two, it would be worth it.

Some users also complain about the doubling of disk space required -- for both compressed and uncompressed copies.

It saves more than 3GB currently; 7zip at optimum settings reduces blockchain data to about 57% of the original size.

It would reduce the amount of disk space required; the decompressed copy doesn't need to be kept after an import. The compressed file is seeded from the user's torrent directory, and makes for a good checksummed archive of the blockchain for a user to have as a backup too.

I wrote unzip/install scripts that can be included with the torrent for one-click extraction

http://(deleted)...

....
I am slowly convincing myself even more that a compressed torrent is preferable. Without compression you are downloading the same amount of data as normal p2p. The compressed binary would only be repeatable on the exact xz/lzma/7zip build version and settings, but I think there are probably four people total that have run mkbootstrap anyway, so this is not important. It takes about an hour to smash the blockchain down to 60% the size at the extreme settings I've used over many trials to optimize settings.

Compressed-Pros:
-2000+ 3000+ MB of uploading and downloading saved for every user,
-2000+ MB less storage used when seeding,
-won't cause problems if torrent HDD is FAT32
(another: small split files easier to host/share/download from multiple sources than a single 8GB file)

Compressed-Cons:
-Not as simple to use, end-user must decompress with third-party utility (although 7-zip is common),
-More work creating torrent (doesn't matter to end-users),
-Cannot "update" torrent by simply replacing data file with newer version with additional blocks (likely to be a rare practice anyway, I am the only one seeding the old torrent right now).

Here is example compression, both require about an hour (but decompress in minutes):

xz utils 5.0.4/Win64, 3GB+ RAM required to compress (2,780,285,148 bytes)
xz --compress --keep --format=xz --check=sha256 --verbose --lzma2=dict=256MiB,nice=273,mf=bt4 bootstrap.dat

7-zip GUI win64 - 6GB+ RAM required to compress (2,769,830,975 bytes)
Format: 7z, Compression Level: Ultra, Compression method: LZMA, Dictionary size: 384MB, Word Size 273

A checkpoint of 225430 was added two months ago at the fork. I would guess that this will either be updated to a newer block or another will be added to before 0.8.2.
https://github.com/bitcoin/bitcoin/blob/master/src/checkpoints.cpp

[rc0cl]

Could we have more frequent updates of these compressed files? especially the last 512MB 7zipped block.

[D35TR0Y3R]

When was this last updated?

[TLS freedom]

My setup does not like udp can someone add these trackers so they get the list. Makes no difference seed or peer. I'm looking to seed, help me help you.

http://www.h33t.com:3310/announce

http://exodus.desync.com:6969/announce

[TLS freedom]

Bump, can someone whos seeding or leaching this torrent add these trackers to their client, PLEASE!?

http://exodus.desync.com:6969/announce

http://coppersurfer.tk:6969/announce

http://www.h33t.com:3310/announce