Bitcoin Forum
December 08, 2016, 04:13:06 AM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Reduced final-state blk0001.dat (with pruned index)  (Read 5845 times)
Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
July 24, 2012, 02:41:23 AM
 #1

I built a "final-state" blk0001.dat with the first 188529 blocks (0 to 188528), in order with no orphan blocks. I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.

This accomplishes reducing the overall disk space used by my 19 (!) bitcoin data directories:
  • This blk0001.dat can be hard linked between them all - saves about 36 GB
  • blkindex.dat is much smaller, since it has all the spent transactions pruned

So here it is for anyone else who wants it (thanks to jrmithdobbs and wizkid057 for 100mbit and gigabit seeds!):
Tip: If you copy your current blk0001.dat into your download directory and rescan, you may be able to skip a large chunk of the download.

I intentionally did not compress blk0001.dat; since you need this file anyway, the remaining data (compressed) only takes up 64 MB - please continue to seed!

Edit: Please note that if you use git master (0.7+), you must not use the included blkindex.dat, or it will break getrawtransaction - use the 0.7 instructions in the INSTALL.txt file!

1481170386
Hero Member
*
Offline Offline

Posts: 1481170386

View Profile Personal Message (Offline)

Ignore
1481170386
Reply with quote  #2

1481170386
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481170386
Hero Member
*
Offline Offline

Posts: 1481170386

View Profile Personal Message (Offline)

Ignore
1481170386
Reply with quote  #2

1481170386
Report to moderator
1481170386
Hero Member
*
Offline Offline

Posts: 1481170386

View Profile Personal Message (Offline)

Ignore
1481170386
Reply with quote  #2

1481170386
Report to moderator
1481170386
Hero Member
*
Offline Offline

Posts: 1481170386

View Profile Personal Message (Offline)

Ignore
1481170386
Reply with quote  #2

1481170386
Report to moderator
Diapolo
Hero Member
*****
Offline Offline

Activity: 769



View Profile WWW
July 24, 2012, 06:54:57 AM
 #2

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?

Dia

Liked my former work for Bitcoin Core? Drop me a donation via:
1PwnvixzVAKnAqp8LCV8iuv7ohzX2pbn5x
bitcoin:1PwnvixzVAKnAqp8LCV8iuv7ohzX2pbn5x?label=Diapolo
koin
Legendary
*
Offline Offline

Activity: 874


View Profile
July 24, 2012, 06:56:13 AM
 #3

I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.

clarification: bitcoind v0.7.x and above
Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
July 24, 2012, 07:44:10 AM
 #4

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?
Correct.

I loaded it into a bitcoind with BlueMatt's transaction-index-pruning code, and produced a 151 MB blkindex.dat to go with it.
clarification: bitcoind v0.7.x and above
Huh?

Diapolo
Hero Member
*****
Offline Offline

Activity: 769



View Profile WWW
July 24, 2012, 07:54:19 AM
 #5

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?
Correct.

Would it be possible to have that behaviour as default for writing the chain-file? I have not a solid Bitcoin background knowledge, but why don't we store blocks in order or remove / don't store orphans (at least when switching to a new blk00x.dat file)?

Dia

Liked my former work for Bitcoin Core? Drop me a donation via:
1PwnvixzVAKnAqp8LCV8iuv7ohzX2pbn5x
bitcoin:1PwnvixzVAKnAqp8LCV8iuv7ohzX2pbn5x?label=Diapolo
Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
July 24, 2012, 07:56:47 AM
 #6

Please enlighten me, what is the technical difference between a normal blk0001.dat and your posted version? You write the blocks are ordered and no orphan blocks are in ... this means a normal node writes blocks unordered and stores orphan blocks in blk0001.dat?
Correct.
Would it be possible to have that behaviour as default for writing the chain-file? I have not a solid Bitcoin background knowledge, but why don't we store blocks in order or remove / don't store orphans (at least when switching to a new blk00x.dat file)?
Because that's how Satoshi did it. sipa's ultraprune may elminate giant blk000*.dat files though.

bg002h
Donator
Legendary
*
Offline Offline

Activity: 1288


Got GLIPH? https://gli.ph/m *p-d-d


View Profile WWW
July 28, 2012, 10:33:46 PM
 #7

Wow....strong work. Can this become a standard part of the satoshi client?

My very own Casascius Bearer Bar: 1GCDzqmX2Cf513E8NeThNHxiYEivU1Chhe
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1036


View Profile WWW
August 02, 2012, 03:57:04 PM
 #8

Orphan blocks (blocks whose parent is not known) are never stored on disk. They are kept in memory while their parents are requested.

Stale blocks (those connected to genesis, but not in the current best chain) are stored on disk, as blk000?.dat + blkindex.dat represents the whole (known part of the) block tree, not just the best chain. You need the side chains in case of reorganisations.

That said, yes optimizations are possible, and you could theoretically remove old stale blocks once the best chain has progressed enough. But as the blk000?.dat files are monolithical, you can't just remove parts of without rewriting the entire file.

I think this is a minor issue only, as the percentage of stale blocks in a typical blk000?.dat file is quite low. I also don't intend to change this in my experimental ultraprune branch. What it will do is use smaller blk000?.dat files (smaller number of blocks per file), and support removing old block files entirely (even when they still contain unspent outputs).

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
Mushoz
Hero Member
*****
Offline Offline

Activity: 686


Bitbuy


View Profile WWW
August 02, 2012, 04:04:11 PM
 #9

Great work guys! I hope this will eventually make it to the Satoshi client!

Btw, the magnet link doesn't seem to be working. Figured I should let you know Wink

www.bitbuy.nl - Koop eenvoudig, snel en goedkoop bitcoins bij Bitbuy!
paraipan
Legendary
*
Offline Offline

Activity: 924


Firstbits: 1pirata


View Profile WWW
August 02, 2012, 04:14:24 PM
 #10

Great work guys! I hope this will eventually make it to the Satoshi client!

...

+1  Smiley

BTCitcoin: An Idea Worth Saving - Q&A with bitcoins on rugatu.com - Check my rep
deepceleron
Legendary
*
Offline Offline

Activity: 1470



View Profile WWW
November 06, 2012, 03:56:52 PM
 #11

I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.

Maged
Legendary
*
Offline Offline

Activity: 1260


View Profile
November 07, 2012, 12:50:00 AM
 #12

I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.
This thread is obsolete. You'll want to check out this thread:
https://bitcointalk.org/index.php?topic=117982.0

Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
November 07, 2012, 12:56:00 AM
 #13

I can confirm that I independently downloaded the blockchain off p2p, processed it with mkbootstrap, and reimported the orphan-cleaned blocks into a new Bitcoin and got an identical SHA256 hash for blk0001.dat. The dat file is trustworthy (and makes mine implicity verified too...).

I will add this torrent to my blockchain download thread, where I also will have a blk0002.dat/full index for blocks up to 206000 for those that want to download a drop-in blockchain up to Oct 2012.

Right now the torrent has three seeds and one peer.
This thread is obsolete. You'll want to check out this thread:
https://bitcointalk.org/index.php?topic=117982.0
Not really, they serve different purposes.

deepceleron
Legendary
*
Offline Offline

Activity: 1470



View Profile WWW
November 07, 2012, 01:33:05 AM
 #14

This thread is obsolete. You'll want to check out this thread:https://bitcointalk.org/index.php?topic=117982.0
As you can see from my post in that thread: https://bitcointalk.org/index.php?topic=117982.msg1303629#msg1303629, this torrent might be more appropriate for the time being, because it doesn't have an extra 350MB of blocks that can't be imported into Bitcoin without tricks. As blk0001.dat/block 0-188528 data will never change, and is also the size/height of the actual Bitcoin datadir file after cleanup, sharing this + another .dat torrent with more blocks 188529- for bootstrapping could mean less torrent maintenance.

I tweaked pynode mkbootstrap.py to make the dat file as described above (if height>188528, start second file), and it also produces an identical blk0001.dat, so creating future block checkpoints in a second file without needing to replace the first is easy; we may have a complete blk0002.dat by the time Bitcoin 0.8.0 comes around.

Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
February 04, 2013, 04:32:06 PM
 #15

Can anyone seed? My blk0001 got corrupt somehow Sad

deepceleron
Legendary
*
Offline Offline

Activity: 1470



View Profile WWW
February 04, 2013, 04:36:07 PM
 #16

Can anyone seed? My blk0001 got corrupt somehow Sad
Seeding, if you can get to me.

Could you redo a new torrent with pruned index up to block 210000 (checkpoint)? A torrent with only 39% of the blockchain is becoming less useful to get new users started.

Luke-Jr
Legendary
*
expert
Offline Offline

Activity: 2086



View Profile
February 04, 2013, 07:18:25 PM
 #17

Can anyone seed? My blk0001 got corrupt somehow Sad
Seeding, if you can get to me.
Thanks!

Could you redo a new torrent with pruned index up to block 210000 (checkpoint)? A torrent with only 39% of the blockchain is becoming less useful to get new users started.
These only work because they're right at 2 GB boundaries. blk0002 might be possible if I get the time...

deepceleron
Legendary
*
Offline Offline

Activity: 1470



View Profile WWW
February 04, 2013, 07:28:18 PM
 #18

Perhaps relevant to your interests:

https://bitcointalk.org/index.php?topic=130066.msg1400898#msg1400898

blk0002.dat is height 210964.

ProfMac
Hero Member
*****
Offline Offline

Activity: 770



View Profile
February 06, 2013, 04:39:01 PM
 #19

Perhaps relevant to your interests:

https://bitcointalk.org/index.php?topic=130066.msg1400898#msg1400898

blk0002.dat is height 210964.

I am new, and reading and assimilating...

I have run bitcoin-qt for a week, and it is now synchronized through block 219926. 

I have three files,
blk0001.dat 2,097,361,271 bytes,
blk0002.dat 2,097,295,438 bytes, and
blk0003.dat 1,249,842,320 bytes.

Are these three files the same for everyone?  Or a crisper question might be, are blk001.dat and blk002.dat the same for everyone who is synchronized?



I try to be respectful and informed.
deepceleron
Legendary
*
Offline Offline

Activity: 1470



View Profile WWW
February 06, 2013, 05:15:29 PM
 #20

I have run bitcoin-qt for a week, and it is now synchronized through block 219926.  

I have three files,
blk0001.dat 2,097,361,271 bytes,
blk0002.dat 2,097,295,438 bytes, and
blk0003.dat 1,249,842,320 bytes.

Are these three files the same for everyone?  Or a crisper question might be, are blk001.dat and blk002.dat the same for everyone who is synchronized?

Depending on how orphan blocks were received from the network and how data was reorganized on the hard drive, different users will have different checksums of blockchain files, although the contents are essentially the same. The most recent file, blk0003.dat will be changing in size every time a block is received. Data for downloadable blockchain distributions such as in this thread is specially prepared to be an efficient in-order set of blocks and to be verifiable by others.

Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!