Reduced final-state blk0001.dat (with pruned index)

[Luke-Jr]

I built a "final-state" blk0001.dat with the first 188529 blocks (0 to 188528), in order with no orphan blocks. I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.

This accomplishes reducing the overall disk space used by my 19 (!) bitcoin data directories:

• This blk0001.dat can be hard linked between them all - saves about 36 GB
• blkindex.dat is much smaller, since it has all the spent transactions pruned

So here it is for anyone else who wants it (thanks to jrmithdobbs and wizkid057 for 100mbit and gigabit seeds!):

• Installation instructions
• PGP signature
• Torrent download (magnet)

Tip: If you copy your current blk0001.dat into your download directory and rescan, you may be able to skip a large chunk of the download.

I intentionally did not compress blk0001.dat; since you need this file anyway, the remaining data (compressed) only takes up 64 MB - please continue to seed!

Edit: Please note that if you use git master (0.7+), you must not use the included blkindex.dat, or it will break getrawtransaction - use the 0.7 instructions in the INSTALL.txt file!

[1713993548]

1713993548

[Diapolo]

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?

Dia

[koin]

I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.

clarification: bitcoind v0.7.x and above

[Luke-Jr]

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?

Correct.

I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.

clarification: bitcoind v0.7.x and above

Huh?

[Diapolo]

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?

Correct.

Would it be possible to have that behaviour as default for writing the chain-file? I have not a solid Bitcoin background knowledge, but why don't we store blocks in order or remove / don't store orphans (at least when switching to a new blk00x.dat file)?

Dia

[Luke-Jr]

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?

Correct.

Would it be possible to have that behaviour as default for writing the chain-file? I have not a solid Bitcoin background knowledge, but why don't we store blocks in order or remove / don't store orphans (at least when switching to a new blk00x.dat file)?

Because that's how Satoshi did it. sipa's ultraprune may elminate giant blk000*.dat files though.

[bg002h]

Wow....strong work. Can this become a standard part of the satoshi client?

[Pieter Wuille]

Orphan blocks (blocks whose parent is not known) are never stored on disk. They are kept in memory while their parents are requested.

Stale blocks (those connected to genesis, but not in the current best chain) are stored on disk, as blk000?.dat + blkindex.dat represents the whole (known part of the) block tree, not just the best chain. You need the side chains in case of reorganisations.

That said, yes optimizations are possible, and you could theoretically remove old stale blocks once the best chain has progressed enough. But as the blk000?.dat files are monolithical, you can't just remove parts of without rewriting the entire file.

I think this is a minor issue only, as the percentage of stale blocks in a typical blk000?.dat file is quite low. I also don't intend to change this in my experimental ultraprune branch. What it will do is use smaller blk000?.dat files (smaller number of blocks per file), and support removing old block files entirely (even when they still contain unspent outputs).

[Mushoz]

Great work guys! I hope this will eventually make it to the Satoshi client!

Btw, the magnet link doesn't seem to be working. Figured I should let you know

[paraipan]

Great work guys! I hope this will eventually make it to the Satoshi client!

...

+1 

[deepceleron]

I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.

[Maged]

I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.

This thread is obsolete. You'll want to check out this thread:
https://bitcointalk.org/index.php?topic=117982.0

[Luke-Jr]

I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.

This thread is obsolete. You'll want to check out this thread:
https://bitcointalk.org/index.php?topic=117982.0

Not really, they serve different purposes.

[deepceleron]

This thread is obsolete. You'll want to check out this thread:https://bitcointalk.org/index.php?topic=117982.0

As you can see from my post in that thread: https://bitcointalk.org/index.php?topic=117982.msg1303629#msg1303629, this torrent might be more appropriate for the time being, because it doesn't have an extra 350MB of blocks that can't be imported into Bitcoin without tricks. As blk0001.dat/block 0-188528 data will never change, and is also the size/height of the actual Bitcoin datadir file after cleanup, sharing this + another .dat torrent with more blocks 188529- for bootstrapping could mean less torrent maintenance.

I tweaked pynode mkbootstrap.py to make the dat file as described above (if height>188528, start second file), and it also produces an identical blk0001.dat, so creating future block checkpoints in a second file without needing to replace the first is easy; we may have a complete blk0002.dat by the time Bitcoin 0.8.0 comes around.

[Luke-Jr]

Can anyone seed? My blk0001 got corrupt somehow

[deepceleron]

Can anyone seed? My blk0001 got corrupt somehow

Seeding, if you can get to me.

Could you redo a new torrent with pruned index up to block 210000 (checkpoint)? A torrent with only 39% of the blockchain is becoming less useful to get new users started.

[Luke-Jr]

Can anyone seed? My blk0001 got corrupt somehow

Seeding, if you can get to me.

Thanks!

Could you redo a new torrent with pruned index up to block 210000 (checkpoint)? A torrent with only 39% of the blockchain is becoming less useful to get new users started.

These only work because they're right at 2 GB boundaries. blk0002 might be possible if I get the time...

[deepceleron]

Perhaps relevant to your interests:

https://bitcointalk.org/index.php?topic=130066.msg1400898#msg1400898

blk0002.dat is height 210964.

[ProfMac]

Perhaps relevant to your interests:

https://bitcointalk.org/index.php?topic=130066.msg1400898#msg1400898

blk0002.dat is height 210964.

I am new, and reading and assimilating...

I have run bitcoin-qt for a week, and it is now synchronized through block 219926. 

I have three files,
blk0001.dat 2,097,361,271 bytes,
blk0002.dat 2,097,295,438 bytes, and
blk0003.dat 1,249,842,320 bytes.

Are these three files the same for everyone?  Or a crisper question might be, are blk001.dat and blk002.dat the same for everyone who is synchronized?

[deepceleron]

I have run bitcoin-qt for a week, and it is now synchronized through block 219926.  

I have three files,
blk0001.dat 2,097,361,271 bytes,
blk0002.dat 2,097,295,438 bytes, and
blk0003.dat 1,249,842,320 bytes.

Are these three files the same for everyone?  Or a crisper question might be, are blk001.dat and blk002.dat the same for everyone who is synchronized?

Depending on how orphan blocks were received from the network and how data was reorganized on the hard drive, different users will have different checksums of blockchain files, although the contents are essentially the same. The most recent file, blk0003.dat will be changing in size every time a block is received. Data for downloadable blockchain distributions such as in this thread is specially prepared to be an efficient in-order set of blocks and to be verifiable by others.