[BETA] Bitcoin blockchain torrent

[waspoza]

Seeding 24/h on my server siting on 100 Mbps link.

[runeks]

I wonder if it could be generated from their existing blockchain (is it basically the same file?) which will presumably be pretty up-to-date.

Yes, this is open data and an open file format.  Seeders may independently generate a byte-for-byte identical bootstrap.dat by running https://github.com/jgarzik/pynode/blob/master/mkbootstrap.py

Each time the blockchain torrent is updated, seeders may run that script to guarantee they have 100% of bootstrap.dat immediately.  A nice, decentralized solution

How do I use that script?

Should some of these variables reference my .bitcoin directory somehow?

The .bitcoin directory is for a different app.

pynode is a full bitcoin client, separate from bitcoind.  The script mkbootstrap.py requires access to the pynode database, after you have downloaded all the blocks.

Sadly you do need to be a bit of a programmer to generate a bootstrap.dat file.

I think I've got pynode synchronized with the network now. The blocks.dat file is 3.5 GB after letting it connect to a local instance of bitcoin-qt and waiting a bit. mkbootstrap.py is running now.

I guess loading the config file should be separated out into a module that can be loaded by mkbootstrap.py. But I'm not sure how to do that elegantly (I'd just create a function that returns the settings dict given the path to the config file).

EDIT: Looks like it succeeded. I'm now seeding the bootstrap.dat file as well.

[jgarzik]

Version 0.7.1, which just entered testing, includes a new feature:  If the file "bootstrap.dat" is found in the bitcoin data directory, it will validate and import all blockchain data found in that file.

So what is the difference vs this procedure? No need to use -loadblock and faster download, given there are many fast seeders?

The difference between 0.7 and 0.7.1 is that 0.7.1 automatically runs "-loadblock=bootstrap.dat" at startup.

The torrent will probably be a faster download...  but if you have an ultrafast network peer, the regular download will be just as fast.  This torrent is just adding option for users; it is not the New Official Recommends Means for getting the blockchain.  As the OP emphasizes, this is an experiment.

[Digigami]

If the results of this experiment are positive, what are the chances we could modify the client so that it downloads the initial blockchain using this method instead of the current system? Is that a long term goal of this?

[jgarzik]

If the results of this experiment are positive, what are the chances we could modify the client so that it downloads the initial blockchain using this method instead of the current system? Is that a long term goal of this?

Not at this time.  It is possible that somebody might create a bitcoin-firstrun.exe application, separate from bitcoin, that torrents the blockchain.

For most users the initial blockchain download will be a one-time event, so there is little interest in directly adding bittorrent code to the bitcoin client.

The import-bootstrap.dat feature was added so that bitcoin does not have to care about the source of the data.  As long as you have the file, it will import it.  Maybe you downloaded the file via torrent... or maybe HTTP.  The point is, from the bitcoin client's perspective, it is agnostic to the download method.

[jgarzik]

Torrent added to PirateBay! I hope you here or my friend - I used his abandoned account - will be OK with this move. Once PB picks some torrent,
it usualy got spread to numerous other torrent sites in matter of few days, so expect (or not?) some more down/uploaders and/or participants here.

In general, spreading the torrent far and wide is fine.

But for trackers or accounts of friends, it is nice to ask [the tracker site / friend] first

[jgarzik]

Torrent added to PirateBay! I hope you here or my friend - I used his abandoned account - will be OK with this move. Once PB picks some torrent,
it usualy got spread to numerous other torrent sites in matter of few days, so expect (or not?) some more down/uploaders and/or participants here.

In general, spreading the torrent far and wide is fine.

But for trackers or accounts of friends, it is nice to ask [the tracker site / friend] first

I haven't added any trackers to torrent. I just uploaded .torrent you linked at starting post, from my HDD, and copy/pasted some info.

Yeah, post the .torrent file far and wide.

[K1773R]

Off-topic, sort of:

There's incredible amount of Anti-P2P activity lately. By the time I finished downloading file, 724 IP addresses known to be used by
organizations and companies openly working against P2P were detected on DHT, and 720 of them tried to connect to my client, LOL!
I was downloading and now seed just this file. Really, those people are very fast - and determined - to find out what's new on P2P.

Level 1 filters those mentioned above:

yea sure, ipv4 adress spaces higher than 255! lmao

EDIT: or is the number a count of ipv4 addresses in this section?
even touhg, ipfilter is a fine thing, but not really accurate at all.

@jgarzik: need more seeds?

[K1773R]

or is the number a count of ipv4 addresses in this section?

Yes, it shows number of different IP addresses for each list. You can check what exactly each of them is filtering (or allowing) here.

well then, sry 4 that are you seeding from home? if so i can understand avoiding the 720 connections

[Gabi]

Sadly you do need to be a bit of a programmer to generate a bootstrap.dat file.

Of course

gabi rage: is it so hard to have a program that you just clic it, tell it where the bitcoin folder with the blockchain is and then you press EXTRACT and it create bootstrap.dat?

[K1773R]

Sadly you do need to be a bit of a programmer to generate a bootstrap.dat file.

Of course

gabi rage: is it so hard to have a program that you just clic it, tell it where the bitcoin folder with the blockchain is and then you press EXTRACT and it create bootstrap.dat?

you forgot that 95% of humanity nowadays is retarded, they will never be able to!

[jgarzik]

Sadly you do need to be a bit of a programmer to generate a bootstrap.dat file.

Of course

gabi rage: is it so hard to have a program that you just clic it, tell it where the bitcoin folder with the blockchain is and then you press EXTRACT and it create bootstrap.dat?

That's what we have here.  "a program" is called a bittorrent client.  Tell it where the bitcoin folder with the blockchain is, and then you press download.

[Gabi]

Oh sorry, my post wasn't clear, i was refering to this:

I wonder if it could be generated from their existing blockchain (is it basically the same file?) which will presumably be pretty up-to-date.

Yes i know, downloading it is very simple and straightforward, what i was speaking about was about creating the file from my blockchain, not about downloading the file via bittorrent.

[Syke]

this data is the precisely the same data you see on the P2P network.

Then how is this different than connecting to high-speed bitcoin peers?

[kjj]

this data is the precisely the same data you see on the P2P network.

Then how is this different than connecting to high-speed bitcoin peers?

Blocks are oddly sized, and their locations within the block files are pretty much arbitrary.  If you don't have enough RAM to cache the whole thing, serving them as blocks one by one will thrash the hell out of your disks.  Also, the way the stock client does the download now, it asks the first node it connects to for the whole chain, block by block.

Bittorrent, on the other hand, doesn't understand the data at all, so it serves it in chunks that are always the same size, which are always aligned with the filesystem blocks.  It also can manage multiple connections, grabbing different chunks from here and there.

The result is that torrents are much more efficient at bulk transfer.  They are faster for the client, and less stressful for the server.

But, for various reasons, we don't like that we have to use an outside distribution method.  The long term goal is to improve the software to make the initial block download faster and smarter.  The bootstrap and torrent are just a temporary thing to lighten the load while work continues.

[VEscudero]

As a workaround it seems ok for me, so I'm seeding this file right now for helping others bootstrap.

[Syke]

Blocks are oddly sized

Makes sense.

and their locations within the block files are pretty much arbitrary.

Even for someone starting from block 0?

Also, the way the stock client does the download now, it asks the first node it connects to for the whole chain, block by block.

That sounds quite un-optimal.

Bittorrent ... can manage multiple connections, grabbing different chunks from here and there.

We could update the Bitcoin client to grab blocks from multiple peers, right? With a few more patches, downloading the chain from peers should approach the speed of a torrent.

[Pieter Wuille]

The way the client currently downloads blocks from peers is hardly more than a hack that somehow frequently manages to work.

We could add even more hacks on top of it to improve it (and it wouldn't take much to improve it a lot), but the right solution in my opinion is doing it right from the ground up: synchronize the headers first, validate those, and then ask your peers to fill in the data for the blocks which you already know in advance are part of the best chain.

The just-merged rewrite of the database layout and validation engine should support such a mode of operation, but someone still has to implement it.

[kjj]

Blocks are oddly sized

Makes sense.

and their locations within the block files are pretty much arbitrary.

Even for someone starting from block 0?

Yup, it is a consequence of the odd block sizes.  Reading a single block may involve several disk reads.  And getting the file offset to read will also take several reads in the index file.

[TheButterZone]

Ooh. Even though I don't use the client, I'm getting this to seed indefinitely (at least when my computer is on and BT is open). Except there's only 1 seed now, no peers. Will it be worth the sectors on my hard drive to keep this going?