[BETA] Bitcoin blockchain torrent

[jgarzik]

Ooh. Even though I don't use the client, I'm getting this to seed indefinitely (at least when my computer is on and BT is open). Except there's only 1 seed now, no peers. Will it be worth the sectors on my hard drive to keep this going?

What bittorrent client are you using?

Seeing 12+ seeds here.

[1715597416]

1715597416

[TheButterZone]

Azureus/Vuze

[runeks]

Azureus/Vuze

Let it sit for a while to gather nodes via DHT. Since it's trackerless you won't instantaneously get all the available seeds. My torrent client says it knows of 18 seeds and 3 peers.

[jgarzik]

Azureus/Vuze

Let it sit for a while to gather nodes via DHT. Since it's trackerless you won't instantaneously get all the available seeds. My torrent client says it knows of 18 seeds and 3 peers.

As mentioned upthread, Azureus sadly walls itself off onto "Azureus island", and in my testing, does not find peers as quickly as all other clients tested.  Making things worse, Azureus' DHT is separate from the mainline DHT.

TheButterZone:  try installing the "mlDHT" plugin for Azureus/Vuze.  This connects you to the mainline DHT.

[TheButterZone]

Ok, thanks. 11 seeds, 3 peers while downloading, dropped to 0 of 1 seeds, 0 of 0 peers while seeding.

[deepceleron]

I just downloaded the torrent and imported it into a new datadir with 0.7.1 with this command:

bitcoind.exe -datadir=C:\datadir -loadblock=C:\bootstrap.dat -connect=127.0.0.1 -detachdb -printtoconsole

However, 193000 blocks were not imported, it stopped after 189205. I verified the hash is correct with SHA256:
a3f258e7af030165360596e4cb0b9beb24b4ce97352c22e65349b89ad5fc5d3e *bootstrap.dat


SetBestChain: new best=0000000000000214aa18  height=189201  work=388855085476008295362  date=07/15/12 15:45:16
ProcessBlock: ACCEPTED
SetBestChain: new best=00000000000005478db7  height=189202  work=388862608030743090363  date=07/15/12 16:41:47
ProcessBlock: ACCEPTED
SetBestChain: new best=0000000000000854e370  height=189203  work=388870130585477885364  date=07/15/12 16:53:22
ProcessBlock: ACCEPTED
SetBestChain: new best=00000000000009441aeb  height=189204  work=388877653140212680365  date=07/15/12 16:56:00
ProcessBlock: ACCEPTED
SetBestChain: new best=0000000000000802f60d  height=189205  work=388885175694947475366  date=07/15/12 17:12:54
ProcessBlock: ACCEPTED
Loaded 189205 blocks from external file in 5073035ms
Loading addresses...
ERROR: CAddrman::Read() : open failed
Invalid or missing peers.dat; recreating
Loaded 0 addresses from peers.dat  4ms
RandAddSeed() 196108 bytes
mapBlockIndex.size() = 189206
nBestHeight = 189205
setKeyPool.size() = 100
mapWallet.size() = 0
mapAddressBook.size() = 1
Done loading
ThreadRPCServer started
Error: To use bitcoind, you must set a rpcpassword in the configuration file:
 C:\datadir\bitcoin.conf


Does the torrent really have blocks up to (193000, uint256("0x000000000000059f452a5f7340de6682a977387c17010ff6e6c3bd83ca8b1317"))? The python script says it should. This looks like a bitcoin bug.

Reference: https://bitcointalk.org/index.php?topic=117874.msg1266479#msg1266479

Additional Theory: bitcoind x32 win can't read bootstrap files past 2GB, the generated blk0002.dat is only 49M, blk0001.dat+blk0002.dat=2.00GB which corresponds to import dying after reading 2.00GB of the 2.32 GB bootstrap.

[deepceleron]

I just downloaded the torrent and imported it into a new datadir with 0.7.1 with this command:

bitcoind.exe -datadir=C:\datadir -loadblock=C:\bootstrap.dat -connect=127.0.0.1 -detachdb -printtoconsole

If the file "bootstrap.dat" is found in the bitcoin data directory, it will validate and import all blockchain data found in that file.

Try without -loadblock, after moving bootstrap.dat to datadir (this might be the stupidest suggestion ever, though).

I just started it up with no -loadblock with the torrent in the empty datadir just for shit's 'n giggles, although it is likely caused by the Win32 C libraries not being able to access >2GB in files. I'm running the similar command on x64 Linux, we'll see after 2 more hours or so if that works vs windows.

The easy solution will probably be to do a binary split of the current bootstrap.dat file on a block boundary below block 189205, and import the two new files with -loadblock=bootstrap.001 -loadblock=bootstrap.002. Having Bitcoin look for multiple dat files when auto-importing would be a smaller change than altering the compile environment. Easier still is just get a fully built dat/idx set and stick it in the datadir.

[deepceleron]

I just started it up with no -loadblock with the torrent in the empty datadir just for shit's 'n giggles...

Still didn't work right, block importing stops at 189205. It is surprising that the whole blockchain torrent thing and inclusion of block importing into Bitcoin got this far without anyone noticing...

[jgarzik]

Is this on FAT32 or NTFS also?

[deepceleron]

Is this on FAT32 or NTFS also?

Windows 7 Service Pack 1 64 Bit.

>fsutil fsinfo ntfsinfo C:
NTFS Volume Serial Number :       0xeebxxxxxxxxxxxx
Version :                         3.1
Number Sectors :                  0x000000000770d7ff
Total Clusters :                  0x0000000000ee1aff
Free Clusters  :                  0x00000000005a0838
Total Reserved :                  0x00000000000007c0
Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000009200000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000870d7f
Mft Zone Start :                  0x00000000000c9200
Mft Zone End   :                  0x00000000000c9ec0

[2112]

although it is likely caused by the Win32 C libraries not being able to access >2GB in files.

Hmm. I'm not aware of any actual Microsoft Win32 implementation having 2GB limit. It was always 4GB-1. Many "compatible" re-implementations of FAT (or SMB) did have the 2GB limit, but Microsoft never had this problem. (Never here means at least Win95 OSR2 or WinNT 3.11 .)

Various 3rd party add-ons, like on-line antivirus, could create this situation.

My bet is on a flaky disk drive, especially if it is MLC SSD. Please run the usual battery of tests: chkdsk /f /r; smartctl and badblocks -n if you can reboot to Linux.

[deepceleron]

How about try to replicate the problem yourself? Betcha have it too...

There is a definite reason for the number 2GB coming up; why do you think the blockchain files are split before they reach 2GB? Look at these threads concerning MinGW (which is what Bitcoin is built with):

http://stackoverflow.com/questions/9026896/get-large-file-size-in-c
http://stackoverflow.com/questions/4003405/32-bit-windows-and-the-2gb-file-size-limit-c-with-fseek-and-ftell

Answer: >2GB file access possible, but less portable coding specifically addressing large file access must be used.

This likely hasn't been noticed because everyone using the bootstrap is connected to the Internet and doesn't look at the logs. It takes so long to import blocks (~1.5 hours) that it is not noticed that Bitcoin prematurely hits the network 400MB early for the rest of the blockchain.

[jgarzik]

At least one C API uses a 32-bit signed integer ("long") on 32-bit platforms, for file offsets and such.  It is potentially a limitation due to the seeking that LoadExternalBlockFile() performs in the source code.

[2112]

Answer: >2GB file access possible, but less portable coding specifically addressing large file access must be used.

Thanks for letting me know. I just double-checked on the (private) code I used to maintain: I'm was using a custom build of GCC/MinGW/Cygwin made by/for a global data integrator.

How about try to replicate the problem yourself? Betcha have it too...

Again, you are right. I most likely wouldn't notice it because of the custom modifications made in 2003-2005 to our Windows build environment. It is almost 2013 and I already forgot what was modified since Microsoft Visual Studio 6.0. It's been almost a decade of patches!

You've prompted me to make a new, truely clean, install of the Windows XP and the associated development tools.

[Nicolai Larsen]

I started seeding

Will this also work with web seeds?

[deepceleron]

....I'm running the similar command on x64 Linux, we'll see after 2 more hours or so if that works vs windows.

I just got finished doing an import of the bootstrap.dat file on ubuntu 12.04 with 1.7.0 PPA Bitcoin install, and it had the same problem - the import stopped at block 189205. It looks like this problem is platform-independent.

I started seeding

Will this also work with web seeds?

What's a "web seed"? If you are seeding, that's better than a torrent hit-it-and-quit-it. If you are using Azureus/Vuze, you should also install the Mainline DHT plugin so your seeding is more easily discoverable.

[K1773R]

....I'm running the similar command on x64 Linux, we'll see after 2 more hours or so if that works vs windows.

I just got finished doing an import of the bootstrap.dat file on ubuntu 12.04 with 1.7.0 PPA Bitcoin install, and it had the same problem - the import stopped at block 189205. It looks like this problem is platform-independent.

I started seeding

Will this also work with web seeds?

What's a "web seed"? If you are seeding, that's better than a torrent hit-it-and-quit-it. If you are using Azureus/Vuze, you should also install the Mainline DHT plugin so your seeding is more easily discoverable.

webseeds are dedicated or VPS seeds

[Nicolai Larsen]

....I'm running the similar command on x64 Linux, we'll see after 2 more hours or so if that works vs windows.

I just got finished doing an import of the bootstrap.dat file on ubuntu 12.04 with 1.7.0 PPA Bitcoin install, and it had the same problem - the import stopped at block 189205. It looks like this problem is platform-independent.

I started seeding

Will this also work with web seeds?

What's a "web seed"? If you are seeding, that's better than a torrent hit-it-and-quit-it. If you are using Azureus/Vuze, you should also install the Mainline DHT plugin so your seeding is more easily discoverable.

webseeds are dedicated or VPS seeds

I use uTorrent and I'm seeding. What would be the point of downloading bootstrap.dat and then remove it after?

A web seed is a file you can link directly to. like, domain.com/downloads/bootstrap.dat - You can add that to the torrent and it will also download from that link.

[jgarzik]

This 2GB bootstrap.dat import problem should be present on all 32-bit systems, as a consequence of using fseek() on a signed 32-bit long internally.

We will work around this in the next release of the bitcoin client.  Until then, 32-bit platform will simply cease importing earlier than expected, but still import a fair amount of the block chain.

[2112]

We will work around this in the next release of the bitcoin client.

Theoretically it would be sufficient to replace "unsigned int" and "fseek" by "fpos_t" and "fsetpos". But because of "CAutoFile" and "FILE *" mixing the actual fix may be quite complex.

The only clean install that I have now is Visual Studio 2012, therefore I can't really test it. But I clearly need a Windows development refresher.