Bitcoin Forum

UPDATE Feb 2013: This thread is obsolete.  See the non-beta Bitcoin blockchain data torrent (https://bitcointalk.org/index.php?topic=145386.0) thread for further torrents and updates.


This is a beta test of a blockchain torrent project.  Interested participants are invited to try, and comment.

Version 0.7.1, which just entered testing (https://bitcointalk.org/index.php?topic=117874.0), includes a new feature:  If the file "bootstrap.dat" is found in the bitcoin data directory, it will validate and import all blockchain data found in that file.  The following torrent presents a bootstrap.dat file for that feature.

Here is the PGP-signed torrent information.  Details follow below the signature.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Torrent info hash: 0bb0521942f586ed96203c6f4d136324756f8a9a
Torrent magnet link:
magnet:?xt=urn:btih:0bb0521942f586ed96203c6f4d136324756f8a9a&dn=bootstrap.dat

Filename: bootstrap.dat
Byte size: 2491771562
SHA1: e70ca90775dfdb13fd0014425805a0bdf4a31677
SHA256: a3f258e7af030165360596e4cb0b9beb24b4ce97352c22e65349b89ad5fc5d3e

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIVAwUBUHeHSdodwg8tvwyoAQJ2DA/8CcINxuD7DzLv96fE+R6GVLvRASLcy0ig
8d2YyDbJ9r9dPVODDPGC7I/ooDVJAcQsqdrLYHST2DTCt6c4zZ/7iXzFEahRopsH
PmxYOgkHie7438nqpmH9uC+d5t0pPHUFS6dBSBgdSVPaLfS86fToXrV/bx30rHBi
60FJz5A99aXrrrUny0cGIjafqVv8XjqNoA1lzjsRjeiu3EgTm8Ibcr8ZI6DLp/80
Siv3potlOArTI6sxjc/vKUa6nZILnW8mKdwc/d8LUdRaBPoo71c6Q4YOSQh/OVht
B1rZ8NEX/2SlS3PbhhMELcY/2wvgPGovkIqOgiw6dDIkhsD8iJAD1DqCkZbsq9S1
kKobGmceuQBcyRUCavkafdHJpZzyCAKLnQLV3FvZ3O4QZQVmYGPYn1H8a5UFRDQq
LGKcQmwm4Cet7162woEiNAaR9p7HvTJ4LV2uEvY56m2GfZbToCk2aiycO+v6Fm8K
ZY8cX+cbEOW3AM5rYVa6Rks029LIrFFzIutlo5MJ7uc7oMqciWrcsPtEh59QY+yJ
SENk6cLyWCAHl4jsvUTBvdNGy3fHKSEyQOjG5cmAMXiJTX/iiB/DJf7koTZAj4ub
wez3/HwuenMYjHgjsVXJAXprcEpTjpEcicg4W0MIDw07dPjol0dnRRNpwfJ1HMRc
dlTqLwlXqgs=
=uwRB
-----END PGP SIGNATURE-----


What is bootstrap.dat?

It is a flat, binary file containing bitcoin blockchain data, from the genesis block through height 193,000.  Height 193,000 is the height of the current checkpoint baked into the reference client.

Version 0.7.1 (when released) will automatically validate and import a file in the data directory named "bootstrap.dat".  Version 0.7 or later will also import this file by passing the command line argument "-loadblock=/path/to/bootstrap.dat" to bitcoin-qt or bitcoind.


Who wants bootstrap.dat?

Anyone bringing up a new node using the reference client.  This is one method of accelerating the initial blockchain download process, while helping the bitcoin P2P network by offloading data download traffic from public P2P nodes.

This download is not for those who are already running the bitcoin client.


NOTE: This torrent requires DHT torrent capability

This torrent is a so-called "trackerless torrent", to avoid making any of the open torrent trackers targets of any bitcoin antipathy.  Peers for this torrent are discovered via DHT, and early results seem to indicate that some bittorrent clients take a while to find their initial peers.  We have also discovered clients (rtorrent) that disable DHT by default; you will need to turn it on.

Please report results, good or bad, on using this trackerless torrent.


How often will this torrent be updated?

Assuming this project is deemed useful and worth continuing... the torrent will be updated once every few months, when the checkpoints are updated in the reference client source code (https://github.com/bitcoin/bitcoin/).


Why not update the torrent more often?

A torrent works best when it is a large, static dataset that changes infrequently.  That maximizes the ability to seed the data, enabling even part-timer seeders to contribute meaningfully.  Less frequent changes also minimizes the risk that a malicious torrent will appear, with a long, malicious side chain.  The current policy only updates the torrent after blocks are buried many thousands deep in the chain.


Why should I trust you?

You don't have to:  This data is raw block chain data.  The client will verify this data during import.

Independent third parties may generate their own bootstrap.dat, up to height 193000, and verify that the sha256sum matches that posted above.  The file format is simple and publicly known:

     <4-byte pchMessageStart><32-bit length><CBlock, serialized in network wire format>


Torrent file download?

If the magnet link does not work, download http://gtf.org/garzik/bitcoin/bootstrap.dat.torrent


Comments welcome

Post any comments or experiences in this thread.  I'll update the OP as needed.

Maybe trackerless will be a #fail, but let's see how it goes.

Cool.  It got me thinking and questioning a few things.  I've often thought that a good alternative to downloading & validating the full block chain when bootstrapping a new client is to find a node you trust and simply copy it block chain and transaction data.  I think of this kind of like cloning a trusted node.  You can bypass a all the validation work.  

This is similar to that in the sense that you are downloading a block chain that you trust to be validated.  It happens to match the 193,000 block baked into the client and signed by the core developers (by virtue of their signature on the hash of the software download).  Since this download.dat has block 193,000 whose hash matches the one baked into the client, you can be sure of the fact that you've got a block chain that matches that which was signed by the core developers up to block 193,000.  If all the merkle hashes and previous block pointers check out, you're good to go.

But, I always thought the bulk of the time to sync up the block chain was due to transaction validation and not actually due to the download time.  When it comes to the matters of trust, could you not achieve the equivalent result by starting up the client in a mode that doesn't do full validation for blocks prior to the most recent, baked in block (i.e. 193,000)?  On startup (if it's GUI), the user could be prompted to decide whether to perform a full validation, or to trust the baked in checkpoint.  Maybe downloading through Bittorrent still has advantages (i.e. not burdening the bitcoin network with download traffic), but I wonder how the performance of this would compare with downloading via Bittorrent.

Not quite...  this data is the precisely the same data you see on the P2P network.

The client validates bootstrap.dat data to the same level it validates data downloaded from the P2P network.

If all checkpointing code is disabled, this data remains fully valid and useful.


A lot of the time is due to Berkeley DB slowness (fixed by ultraprune).  Another yet-unfixed cause of slowness is poor bitcoin P2P network peer selection, for block download.


We want to do exactly the same amount of validation as the client does with network blocks...  break no additional link in the trust chain :)

Oh, ic…I guess that's what you meant by "The client will verify this data during import."  ;)

Any idea how long it takes to perform the validation of bootstrap.dat once it's downloaded (on typical hardware)?

Sadly it varies wildly depending on your hard drive configuration.  Import time here is under an hour.

At last, finally somebody made bootstrapping Bitcoin easier.

It was a major pain in the ass for people running full BTC client.

Very nice! This project seems to continue to have the best technology associated with it.

Nice one.  I will dedicated some bandwidth for it.  I could imagine with a couple hundred seeds new users could bootstrap very quickly.  Even if you only updated the torrent once a year it would still provide a significant portion of the blockchain at a high speed.

Won't this torrent only benefit new users if people actually seed it?

The swarm doesn't appear to be particularly large at the moment...

Yes, that's the hope.  Already had reports of 7 MB/s downloads...

Correct.  It is not much use, without participating seeders.  We definitely have several right now... but it does look like users occasionally get partitioned off into "Azureus island" or "rtorrent island" or "everybody else's island."

I wonder why the Azureus island is so lonely right now. All I can see are two seeds and no peers.

That probably is a good idea but I don't think updates that often are necessary.  Think of the torrent as just a jumpstarter.  It doesn't need to get you within days of the current block just rapidly build the "ancient history".  Even having a single base torrent to block 193,000 allows new users to bootstrap ~90% of the blockchain.  

It could be as simple as making a new torrent for each checkpoint. Say in 6 months we are at block 240,000 and there is a checkpoint 220,000 a "torrent2" could be added which contains blocks 193,001 to 220,000.   New users could download both torrents and be within 20,000 blocks of "current".  I think at most quarterly updates is all that is necessary.

On edit: looks like it will take me ~30 min to download the torrent.  If a user can download 80% to 90% of the blockchain in half an hour and then index it in another hour that is a pretty nice jumpstart.

At present, that sort of setup would be challenging for users to use effectively.  It is far easier to have one file to import (automatically, in 0.7.1+) than a collection of files.

Longer term, the idea is to fix any issues with bitcoin P2P peer selection, so that downloading recent blocks from your peers is faster and not burdensome.


That's the plan, with a slight change:  each new torrent will contain all blocks from zero to X.

For bitcoin users, the largest user population served by this is those that have zero blocks, and are jumpstarting a fresh node installation.  Other bitcoin users will likely have blocks fresher than 3-6 months old; catching up via P2P network is fine for them.  There is only a tiny remaining segment of bitcoin users who would then be served by new-torrent-for-checkpoint, those that only turn on their bitcoin clients once every ~6 months.

Note:  For existing torrent seeders, they may simply swap out the .torrent file, perhaps kick their torrent client to manually re-verify a file, and bootstrap.dat in their Uploads directory will simply be extended.  Seeders will automatically already have 90% of each new torrent's bootstrap.dat.

Hmm.  I didn't realize that was possible.  I assumed that changing the torrent would result in no seeds (and all prior seeders needing to download again).  I was thinking many wouldn't and thus it would be a challenge to keep the number of seeders high.  I guess bittorrent is "smarter" than I realized.  

It has been possible to download the blockchain. Torrent is in some ways an improvement in that it improves decentralization but in some ways a step back as it's not really suited to frequent updates. Say, for example, I download and seed v1, eventually v2 will come out. Although I already have all of the data in v2 in my running blockchain, there's no easy way to contribute to seeding v2 without downloading the whole thing again. A way to extract a version of the torrented blockchain from my live blockchain would be good. What might be better would be a custom client that could seed from the live blockchain itself.

D'oh. This is "stuff I know" and have used from time to time. My bad.

This is special to our use case:  bootstrap.dat is essentially an append-only file.  Blocks are simply concatenated onto the end.

Today's torrent at height 193000 is 2,491,771,562 bytes in size.

The next torrent, a few months from now, will have the same first 2,491,771,562 bytes.

Thus, to bittorrent, the next torrent will simply appear to be a truncated / not fully downloaded bootstrap.dat.  Bittorrent is built to fill in the missing pieces of a file, so that is what it does here :)

It will result in seeders needing to re-download a torrent for each update though. They'll also have to download all the intervening data (though they will be seeding from what they already have right away). I wonder if it could be generated from their existing blockchain (is it basically the same file?) which will presumably be pretty up-to-date.

It will result in seeders already having 90% of any new torrent, thus having to fill in the remaining 10%.


Yes, this is open data and an open file format.  Seeders may independently generate a byte-for-byte identical bootstrap.dat by running https://github.com/jgarzik/pynode/blob/master/mkbootstrap.py

Each time the blockchain torrent is updated, seeders may run that script to guarantee they have 100% of bootstrap.dat immediately.  A nice, decentralized solution :)

Yes, it's just that it's an extra step, not automatic. This is a limitation of the torrent protocol and not a criticism of this project, just pointing it out.


Excellent. Though presumably it will be a slightly different length than whatever is torrented (which will be taken care of by the recheck-data option).

Why would a byte for byte copy be a "slightly different length"?

Once the height of the next checkpoint is publicly known, everyone may run that script to independently generate bootstrap.dat with the exact same file size and SHA256 checksum.

How does it know the length of the torrented file? (Note that it is "identical", not a copy) Though from what jgarzick says in the post above there is some kind of checkpointing that it either knows or gets fed into it?

It doesn't need to know the length of the file.  The script doesn't make a dat file of your entire blockchain just through the last checkpoint.  Currently the last checkpoint is block 193,000.   If you run that script on any node it will produce the same file.   Exactly the same file.

Now currently the script has 193,000 hardcoded but I could see future version either getting the checkpoint from the client or making an API call to get the block #.  The script could be included with the client or even better it could be built into the client so you click an [export blockchain > torrent] button and it generates the proper file based on the current checkpoint.

Lots of interesting options.

Torrent works by breaking files up into pieces and hashing each piece.  The parts that you already have will have the same hash as the hashes in the seed, with the exception of the final piece.  Modern torrent clients will fetch that partial piece using the missing byte range, and then verify it with the hash.  And naturally, they will also grab all of the new pieces.

Does anyone know if bittorrent can share streams between multiple versions of the same file?

I mean, lets suppose we publish the torrent for the first x bytes, add y bytes to the file, then publish another torrent for the new version. Will people downloading the new, longer torrent, be able to request blocks from people running clients that have only downloaded the shorter torrent? There does exist a Bittorrent streaming protocol, TS Engine, but as far as I can tell it's purely block based and doesn't efficiently handle the case where every client needs the whole stream, right from the beginning. I know internally bittorrent can identify blocks that is already has using a merkle tree system, but the tree can only have one tip. (1)

It's not a very important optimization for bitcoin, just publishing up to the latest checkpoint is fine for us even if old seeds aren't useful anymore, but I have an application where torrenting a file that is continuously being extended would be useful.

(1) Ironically the data I want to distribute via bittorrent in this fashion is a forest of merkle trees, exactly the sort of data structure that you could use to implement a continuously-appended-to torrent...

I also have a PHP (!) script that parses the block chain and makes a clean sequential bootstrap.dat file.  It is ugly and slow, but I wanted to have an independent verification.  Our two scripts came up with identical files.

And by ugly, I mean embarrassingly ugly, like I'd be ashamed to let anyone see it.  If I have time this weekend, I'll clean it up and post it.

Ah, this is what I was talking about.

Yes. I was talking about generating the file without having to download it. No point having torrents download the missing pieces when you already have them sitting in the blockchain on your hard-drive.

It depends on your definition of "share"... locally or remotely?

A single torrent is simply a hash-of-hashes.  Each stream is a different torrent, with different hashes, even if torrent A is a strict subset of torrent B.

Your client may be modified to share streams which are multiple versions of the same file.  Would probably only need some small mods to existing clients, and would not break the network protocol.  So from that perspective: "yes"

Remote clients will see different hashes, and assume that each stream is separate and independent of each other.  So from that perspective: "no"

Don't think that will work.  Most bittorrent clients check the file size, etc.  It would be super cool if some torrent client would be willing to serve matching chunks out of a file, even if the overall file is wrong.  But none do that I'm aware of.

These are specially cleaned block files.  These are sequential, have no orphans, and no inter-block garbage.  For virtually everyone on the planet, the first N bytes of their actual block files won't match these.  jgarzik has already published his script, and I hope to publish mine soon.  You can use them to recreate the file from your block database without having to download it.

Remotely


Hmm... that's pretty much what I expected. Anyway I thought about it some more, and I think I have a way for my application to even deal with divergent versions of the file, really divergent trees, which bittorrent *definitely* doesn't support. It'd be a very nice feature, so at that point I might as well just bite the bullet and hack bittorrent as required. (or invent Yet Another Peer-to-Peer Network)

Good idea, except the trackerless part, IMHO. Here's a magnet link to the same torrent, with 4 public trackers added:

magnet:?xt=urn:btih:0bb0521942f586ed96203c6f4d136324756f8a9a&dn=bootstrap.dat&tr=udp://tracker.openbittorrent.com:80&tr=udp://tracker.publicbt.com:80&tr=udp://tracker.ccc.de:80&tr=udp://tracker.istole.it:80

Good deal. Unless a torrent is marked as private, the dht will kick in if the trackers ever stop working (though I'm not sure if it's possible to nobble them?)

Great idea, I am seeding with my 2Mb. Just a suggestion: I would love to be notified when new torrent is created so I won't be seeding obsolete one. Could someone create a mailing list for that purpose?

+1
 and many thanks to jgarzick for this useful development

The best place to watch is probably this thread, though I am open to other suggestions.

How do I use that script?

Should some of these variables reference my .bitcoin directory somehow?


I get:

The .bitcoin directory is for a different app.

pynode is a full bitcoin client, separate from bitcoind.  The script mkbootstrap.py requires access to the pynode database, after you have downloaded all the blocks.

Sadly you do need to be a bit of a programmer to generate a bootstrap.dat file.

Seeding 24/h on my server siting on 100 Mbps link.

https://i.imgur.com/q4x9P.png

I think I've got pynode synchronized with the network now. The blocks.dat file is 3.5 GB after letting it connect to a local instance of bitcoin-qt and waiting a bit. mkbootstrap.py is running now.

I guess loading the config file should be separated out into a module that can be loaded by mkbootstrap.py. But I'm not sure how to do that elegantly (I'd just create a function that returns the settings dict given the path to the config file).

EDIT: Looks like it succeeded. I'm now seeding the bootstrap.dat file as well.

The difference between 0.7 and 0.7.1 is that 0.7.1 automatically runs "-loadblock=bootstrap.dat" at startup.

The torrent will probably be a faster download...  but if you have an ultrafast network peer, the regular download will be just as fast.  This torrent is just adding option for users; it is not the New Official Recommends Means for getting the blockchain.  As the OP emphasizes, this is an experiment.

If the results of this experiment are positive, what are the chances we could modify the client so that it downloads the initial blockchain using this method instead of the current system? Is that a long term goal of this?

Not at this time.  It is possible that somebody might create a bitcoin-firstrun.exe application, separate from bitcoin, that torrents the blockchain.

For most users the initial blockchain download will be a one-time event, so there is little interest in directly adding bittorrent code to the bitcoin client.

The import-bootstrap.dat feature was added so that bitcoin does not have to care about the source of the data.  As long as you have the file, it will import it.  Maybe you downloaded the file via torrent... or maybe HTTP.  The point is, from the bitcoin client's perspective, it is agnostic to the download method.

In general, spreading the torrent far and wide is fine.

But for trackers or accounts of friends, it is nice to ask [the tracker site / friend] first :)

Yeah, post the .torrent file far and wide.

yea sure, ipv4 adress spaces higher than 255! lmao :D

EDIT: or is the number a count of ipv4 addresses in this section?
even touhg, ipfilter is a fine thing, but not really accurate at all.

@jgarzik: need more seeds?

well then, sry 4 that ;) are you seeding from home? if so i can understand avoiding the 720 connections :P

Of course

gabi rage: is it so hard to have a program that you just clic it, tell it where the bitcoin folder with the blockchain is and then you press EXTRACT and it create bootstrap.dat?

you forgot that 95% of humanity nowadays is retarded, they will never be able to!

That's what we have here.  "a program" is called a bittorrent client.  Tell it where the bitcoin folder with the blockchain is, and then you press download.

Oh sorry, my post wasn't clear, i was refering to this:


Yes i know, downloading it is very simple and straightforward, what i was speaking about was about creating the file from my blockchain, not about downloading the file via bittorrent.

Then how is this different than connecting to high-speed bitcoin peers?

Blocks are oddly sized, and their locations within the block files are pretty much arbitrary.  If you don't have enough RAM to cache the whole thing, serving them as blocks one by one will thrash the hell out of your disks.  Also, the way the stock client does the download now, it asks the first node it connects to for the whole chain, block by block.

Bittorrent, on the other hand, doesn't understand the data at all, so it serves it in chunks that are always the same size, which are always aligned with the filesystem blocks.  It also can manage multiple connections, grabbing different chunks from here and there.

The result is that torrents are much more efficient at bulk transfer.  They are faster for the client, and less stressful for the server.

But, for various reasons, we don't like that we have to use an outside distribution method.  The long term goal is to improve the software to make the initial block download faster and smarter.  The bootstrap and torrent are just a temporary thing to lighten the load while work continues.

As a workaround it seems ok for me, so I'm seeding this file right now for helping others bootstrap.

Makes sense.


Even for someone starting from block 0?


That sounds quite un-optimal.


We could update the Bitcoin client to grab blocks from multiple peers, right? With a few more patches, downloading the chain from peers should approach the speed of a torrent.

The way the client currently downloads blocks from peers is hardly more than a hack that somehow frequently manages to work.

We could add even more hacks on top of it to improve it (and it wouldn't take much to improve it a lot), but the right solution in my opinion is doing it right from the ground up: synchronize the headers first, validate those, and then ask your peers to fill in the data for the blocks which you already know in advance are part of the best chain.

The just-merged rewrite of the database layout and validation engine should support such a mode of operation, but someone still has to implement it.

Ooh. Even though I don't use the client, I'm getting this to seed indefinitely (at least when my computer is on and BT is open). Except there's only 1 seed now, no peers. Will it be worth the sectors on my hard drive to keep this going?

What bittorrent client are you using?

Seeing 12+ seeds here.

Azureus/Vuze

Let it sit for a while to gather nodes via DHT. Since it's trackerless you won't instantaneously get all the available seeds. My torrent client says it knows of 18 seeds and 3 peers.

As mentioned upthread, Azureus sadly walls itself off onto "Azureus island", and in my testing, does not find peers as quickly as all other clients tested.  Making things worse, Azureus' DHT is separate from the mainline DHT.

TheButterZone:  try installing the "mlDHT" plugin for Azureus/Vuze.  This connects you to the mainline DHT.

Ok, thanks. 11 seeds, 3 peers while downloading, dropped to 0 of 1 seeds, 0 of 0 peers while seeding.

I just downloaded the torrent and imported it into a new datadir with 0.7.1 with this command:

bitcoind.exe -datadir=C:\datadir -loadblock=C:\bootstrap.dat -connect=127.0.0.1 -detachdb -printtoconsole

However, 193000 blocks were not imported, it stopped after 189205. I verified the hash is correct with SHA256:
a3f258e7af030165360596e4cb0b9beb24b4ce97352c22e65349b89ad5fc5d3e *bootstrap.dat


SetBestChain: new best=0000000000000214aa18  height=189201  work=388855085476008295362  date=07/15/12 15:45:16
ProcessBlock: ACCEPTED
SetBestChain: new best=00000000000005478db7  height=189202  work=388862608030743090363  date=07/15/12 16:41:47
ProcessBlock: ACCEPTED
SetBestChain: new best=0000000000000854e370  height=189203  work=388870130585477885364  date=07/15/12 16:53:22
ProcessBlock: ACCEPTED
SetBestChain: new best=00000000000009441aeb  height=189204  work=388877653140212680365  date=07/15/12 16:56:00
ProcessBlock: ACCEPTED
SetBestChain: new best=0000000000000802f60d  height=189205  work=388885175694947475366  date=07/15/12 17:12:54
ProcessBlock: ACCEPTED
Loaded 189205 blocks from external file in 5073035ms
Loading addresses...
ERROR: CAddrman::Read() : open failed
Invalid or missing peers.dat; recreating
Loaded 0 addresses from peers.dat  4ms
RandAddSeed() 196108 bytes
mapBlockIndex.size() = 189206
nBestHeight = 189205
setKeyPool.size() = 100
mapWallet.size() = 0
mapAddressBook.size() = 1
Done loading
ThreadRPCServer started
Error: To use bitcoind, you must set a rpcpassword in the configuration file:
 C:\datadir\bitcoin.conf


Does the torrent really have blocks up to (193000, uint256("0x000000000000059f452a5f7340de6682a977387c17010ff6e6c3bd83ca8b1317"))? The python script says it should. This looks like a bitcoin bug.

Reference: https://bitcointalk.org/index.php?topic=117874.msg1266479#msg1266479

Additional Theory: bitcoind x32 win can't read bootstrap files past 2GB, the generated blk0002.dat is only 49M, blk0001.dat+blk0002.dat=2.00GB which corresponds to import dying after reading 2.00GB of the 2.32 GB bootstrap.

I just started it up with no -loadblock with the torrent in the empty datadir just for shit's 'n giggles, although it is likely caused by the Win32 C libraries not being able to access >2GB in files. I'm running the similar command on x64 Linux, we'll see after 2 more hours or so if that works vs windows.

The easy solution will probably be to do a binary split of the current bootstrap.dat file on a block boundary below block 189205, and import the two new files with -loadblock=bootstrap.001 -loadblock=bootstrap.002. Having Bitcoin look for multiple dat files when auto-importing would be a smaller change than altering the compile environment. Easier still is just get a fully built dat/idx set and stick it in the datadir.

Still didn't work right, block importing stops at 189205. It is surprising that the whole blockchain torrent thing and inclusion of block importing into Bitcoin got this far without anyone noticing...

Is this on FAT32 or NTFS also?

Windows 7 Service Pack 1 64 Bit.

>fsutil fsinfo ntfsinfo C:
NTFS Volume Serial Number :       0xeebxxxxxxxxxxxx
Version :                         3.1
Number Sectors :                  0x000000000770d7ff
Total Clusters :                  0x0000000000ee1aff
Free Clusters  :                  0x00000000005a0838
Total Reserved :                  0x00000000000007c0
Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000009200000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000870d7f
Mft Zone Start :                  0x00000000000c9200
Mft Zone End   :                  0x00000000000c9ec0

Hmm. I'm not aware of any actual Microsoft Win32 implementation having 2GB limit. It was always 4GB-1. Many "compatible" re-implementations of FAT (or SMB) did have the 2GB limit, but Microsoft never had this problem. (Never here means at least Win95 OSR2 or WinNT 3.11 .)

Various 3rd party add-ons, like on-line antivirus, could create this situation.

My bet is on a flaky disk drive, especially if it is MLC SSD. Please run the usual battery of tests: chkdsk /f /r; smartctl and badblocks -n if you can reboot to Linux.

How about try to replicate the problem yourself? Betcha have it too...

There is a definite reason for the number 2GB coming up; why do you think the blockchain files are split before they reach 2GB? Look at these threads concerning MinGW (which is what Bitcoin is built with (https://bitcointalk.org/index.php?topic=5851.msg86700#msg86700)):

http://stackoverflow.com/questions/9026896/get-large-file-size-in-c
http://stackoverflow.com/questions/4003405/32-bit-windows-and-the-2gb-file-size-limit-c-with-fseek-and-ftell

Answer: >2GB file access possible, but less portable coding specifically addressing large file access must be used.

This likely hasn't been noticed because everyone using the bootstrap is connected to the Internet and doesn't look at the logs. It takes so long to import blocks (~1.5 hours) that it is not noticed that Bitcoin prematurely hits the network 400MB early for the rest of the blockchain.

At least one C API uses a 32-bit signed integer ("long") on 32-bit platforms, for file offsets and such.  It is potentially a limitation due to the seeking that LoadExternalBlockFile() performs in the source code.

Thanks for letting me know. I just double-checked on the (private) code I used to maintain: I'm was using a custom build of GCC/MinGW/Cygwin made by/for a global data integrator.
Again, you are right. I most likely wouldn't notice it because of the custom modifications made in 2003-2005 to our Windows build environment. It is almost 2013 and I already forgot what was modified since Microsoft Visual Studio 6.0. It's been almost a decade of patches!

You've prompted me to make a new, truely clean, install of the Windows XP and the associated development tools.

I started seeding ;)

Will this also work with web seeds?

I just got finished doing an import of the bootstrap.dat file on ubuntu 12.04 with 1.7.0 PPA Bitcoin install, and it had the same problem - the import stopped at block 189205. It looks like this problem is platform-independent.

What's a "web seed"? If you are seeding, that's better than a torrent hit-it-and-quit-it. If you are using Azureus/Vuze, you should also install the Mainline DHT plugin so your seeding is more easily discoverable.

webseeds are dedicated or VPS seeds ;)

I use uTorrent and I'm seeding. What would be the point of downloading bootstrap.dat and then remove it after? :P

A web seed is a file you can link directly to. like, domain.com/downloads/bootstrap.dat - You can add that to the torrent and it will also download from that link.

This 2GB bootstrap.dat import problem should be present on all 32-bit systems, as a consequence of using fseek() on a signed 32-bit long internally.

We will work around this in the next release of the bitcoin client.  Until then, 32-bit platform will simply cease importing earlier than expected, but still import a fair amount of the block chain.

Theoretically it would be sufficient to replace "unsigned int" and "fseek" by "fpos_t" and "fsetpos". But because of "CAutoFile" and "FILE *" mixing the actual fix may be quite complex.

The only clean install that I have now is Visual Studio 2012, therefore I can't really test it. But I clearly need a Windows development refresher.

It's been a while since I ran into a similar issue myself but if I recall correctly, it was pretty much a drop-in fix.

That may be problem with your client or firewall. I'm seeding only since yesterday evening and now it has > 300M uploaded already, using Deluge (http://www.makeuseof.com/tag/deluge-awesome-unappreciated-crossplatform-bittorrent-client/) 1.3.5 .

Under ideal conditions, where you have enough seeders to make each new download saturate the client's download link, that produces a situation where you have several seeders and many periods of idle time, punctuated by short bursts of network activity.

I'm definitely seeing daily downloads here, but we could use some more seeders nonetheless.

Bittorrent also helps more for unexpected bursts of network traffic -- like those accompanying a new bitcoin version release, for example.

Two ideas for improvement:

1) Compression: this will easy shave off 25% of the file size. My 2.6 G blockchain file uncompressed became 1.9 G compressed with zip.

2) Split it up in multiple files. If I stop using bitcoin for a couple of months and then start using it again I might still have a large chunk of the blockchain left on my computer. I really only need the new block. The current mainline has a new way of storing blockfiles (.bincoin/blocks/blk000?.dat instead of .bitcoin/blk000?.dat) each with size 128 MB (instead of the much larger .bitcoin/blk000?.dat files). These smaller, 128 MB, files would be nice to have available via a torrent, making it possible to limit the download to only the latest blocks.

im sry but zip compression totally sucks...
read this fora nice list: http://www.maximumcompression.com/index.html

There's allot of pseudorandom data in the blockchain that can't be compressed, so I don't know if there will be so much difference in the file size result between different algorithms in this case, maybe we can shave off another 100 MB with a better one. It would be interesting to see a comparison you linked to with the blockchain.

(Edit - obsolete, Bitcoin issue fixed)

I have a method for splitting the torrent so it can be successfully imported to completion. It can be split into two files using the GNU split utility with this command:

split --bytes 2147570829 bootstrap.dat

This creates two files:

10/28/2012  10:12 AM     2,147,570,829 xaa
10/28/2012  10:12 AM       344,200,733 xab

sha256sum x??
fec4aa1d5a1d07b832cb22a353a689b6bf7fcb09e4822a19d9051160f195a8ef *xaa
fc07e8d67a39152a5bf1920b3bcd99d50515817ba77a28bd7d780d4cc3aa3938 *xab

This splits the torrent at block 189205, the last block that Bitcoin successfully imports out of the full 2.32GB bootstrap.dat. You can then import all torrent blocks with bitcoind or bitcoin-qt with the command:

bitcoind -loadblock=xaa -loadblock=xab -printtoconsole

(printtoconsole so you can watch the progress of block import for several hours. Use the full path to the files if needed.)

Split should be included in any GNU/Linux distribution plus MacOS. A standalone windows compile of the utility is hosted here: split.exe (72,192 bytes) (http://we.lovebitco.in/split.exe) (or get the GNU coreutils and dependencies full packages at http://gnuwin32.sourceforge.net/packages/coreutils.htm).

---

I found another bug/undocumented oddity; Bitcoin first processes -loadblock commands, then it looks for a bootstrap.dat file in the datadir. That means the above command will attempt to load bootstrap.dat a second time if it is located in the datadir (but will skip through all the repeated blocks quickly).

---

BTW: How do I compress the blockchain to 56% of the original size? Have 64 bit 7-zip and lots of RAM:
http://we.lovebitco.in/img/7zip-maxsettings.png

Oh, they're getting stored in a simple flat file. Well that's easy, download blocks from random peers and store them in temporary files. Then when the next block needed has been downloaded, copy it to the final block file.

I made a change to the pynode mkbootstrap.py script that generates the torrent, so it splits block files at the same block as above; the hashes of the split output files are identical to the manual split method.

It's a bit less clever then tracking file sizes and auto-splitting for multiple files <2GB.

patch code:

Is it time for a torrent with more blocks (in this method?) Block 206000 is a nice recent number and makes for a 2GB + 1.5GB file.

Note that the first file can't be "bootstrap.dat", because Bitcoin actually imports this last after any other manual import statements. Patch Bitcoin to auto-import any "bootstrap.dat", "bootstrap..001", "bootstrap..002", ... in that order if they exist, instead of fixing >2GB import?

I also think the torrent should be immediately-useable blk0001.dat and blk0002.dat files including a blkindex.dat. That way one could either wait the 6+ hours to import and verify blocks (which is little improvement over p2p), or just drop the whole torrent into a Bitcoin data directory. The "split point" should correspond with the actual (earlier) block at which the Bitcoin client starts blk0002.dat when importing these cleaned blocks (188529 blocks). The index could be a spent-transaction-pruned version like luke-jr created here (https://bitcointalk.org/index.php?topic=94881.0).

blkindex.dat is not included in the torrent because that conflicts with Bitcoin's trust model.

If you're going to trust a single someone to do the indexing for you, you may as well trust the network (far less trust required) and run an SPV node (like Multibit). The purpose of the reference client is to be zero trust - not relying on any trusted data. If someone would distributed a faulty (intentional or not) set of block files with included index, and many people - in particular miners - would start using this data without verification, they could end up in a block chain fork if a transactions was included in a legitimate block that conflicts with their index.

Also, 0.8 will switch to an entirely different database layout (there won't be a blkindex.dat anymore), and validate blocks much faster (in particular when they're read from the local disk).

Additionally, it should be noted that sipa has created a pull request which should avoid seeking, thus avoiding the 2GB problem:

     https://github.com/bitcoin/bitcoin/pull/1962 (https://github.com/bitcoin/bitcoin/pull/1962)

In checkpoints.cpp there is another checkpoint at 210000 - please update the torrent! :)

By the way, from time to time people still are using this (I'm a more or less permanent seeder here), so I guess the experiment is working.

This torrent will be updated soon, to sync with a new checkpoint in the upcoming 0.8 release.

I'll move this over here for more discussion of Torrent 2.0


"The 2GB issue", failure to import blocks after 2.0GiB on 32 bit builds, was reported by me, so I guess I'm the one that "raised" it here. Now that Bitcoin can actually use a bigger torrent and 0.8.0 has a higher checkpoint (https://github.com/bitcoin/bitcoin/blob/master/src/checkpoints.cpp), it's finally time for a new torrent!

You should get the same 4.8GB bootstrap.dat SHA256 at that height, if you don't, we've got a big problem!

The only thing that would make a torrent I create less than "semi-official" would be if I make a compressed torrent without consensus. I am slowly convincing myself even more that a compressed torrent is preferable though. Without compression you are downloading the same amount of data as normal p2p. The compressed binary would only be repeatable on the exact xz/lzma/7zip build version and settings, but I think there are probably four people total that have run mkbootstrap (https://bitcointalk.org/index.php?topic=130066.msg1400898#msg1400898) anyway, so this is not important. It takes about an hour to smash the blockchain down to 60% the size at the extreme settings I've used over many trials to optimize settings.

Compressed-Pros:
-2000+ MB of uploading and downloading saved for every user,
-2000+ MB less storage used when seeding,
-won't cause problems if torrent HDD is FAT32 (under 4.0GB for now)

Compressed-Cons:
-Not as simple to use, end-user must decompress with third-party utility (although 7-zip is common and opens xz),
-More work creating torrent (doesn't matter to end-users),
-Cannot "update" torrent by simply replacing data file with newer version with additional blocks (likely to be a rare practice anyway, I am the only one seeding the old torrent right now).

Here is example compression, both require about an hour (but decompress in minutes):

xz utils 5.0.4/Win64, 3GB+ RAM required to compress (2,780,285,148 bytes)
xz --compress --keep --format=xz --check=sha256 --verbose --lzma2=dict=256MiB,nice=273,mf=bt4 bootstrap.dat

7-zip GUI win64 - 6GB+ RAM required to compress (2,769,830,975 bytes)
Format: 7z, Compression Level: Ultra, Compression method: LZMA, Dictionary size: 384MB, Word Size 273

If this is an issue, you probably would want to start using multi-part compression (which is another pro fwiw).

Yep, your magnet link didn't get me anything.  I had to add some open trackers until I found some peers.

Anyway, I'll be seeding.

Here is a magnet link with a few good trackers added:

Currently beta-testing the following updated torrent:

SHA256 bf658c7055b733bfc15ea167f298c5599b89d220b14dbe7c8ef20b18e468c451

magnet:?xt=urn:btih:6fe493ba606847eac163baf35aae9db319735482&dn=bootstrap.dat&tr=udp://tracker.openbittorrent.com:80&tr=udp://tracker.publicbt.com:80&tr=udp://tracker.ccc.de:80&tr=udp://tracker.istole.it:80

or http://gtf.org/garzik/bitcoin/bootstrap.dat.torrent (please use magnet link, if possible)

When sufficient seeds appear, a new PGP-signed forum post will be made, obsoleting this thread.

Oh, for anyone already seeding/downloading the old torrent:
Just remove the seed, open the new torrent and add it at the same location. Your torrent client then verifies the already existing blocks (~first 51.3% of the new torrent) and then loads the rest from other peers.

I was able to just drop the 4.7GB bootstrap.dat I already created into the torrent and seed, which I've been doing since 20 minutes after the post - and there are still the same number of seeders.

Well, I see 15 seeders and 9 leechers on the current (larger) bootstrap.dat file. Maybe you are not using trackers and/or mainlineDHT or using some restrictive firewall? Also download of the torrent was reasonably fast (= ~380 kB/s on average for the remaining 2.2 GB).

As long as BitcoinQT cannot handle a compressed bootstrap.dat.xz file, I'd rather waste some bandwidth instead of confusing newies even more.

I added the torrent on my seedbox. The download speed was already very good @ 1.1 MB/s! Everyone interested should be seeing even faster download speeds.
Long live Bitcoin!

This thread is officially obsolete.

Non-beta torrent thread now posted at https://bitcointalk.org/index.php?topic=145386.0 (https://bitcointalk.org/index.php?topic=145386.0)

Thanks for this! It is better than syncing through Bitcoin-qt.  :)
Kindly,
         Muhammed Zakhir

Can anyone recommend a good seedbox provider that accepts bitcoin? I have used several and have not been happy with any of them yet.

Hello! You can find many seedbox providers here (http://seedboxgui.de/seedbox/) , then click on Bitcoin on left side. The lists that appears are seedbox providers who accept Bitcoin.
Kindly,
        Muhammed Zakhir