Bitcoin Forum
November 11, 2024, 06:56:28 AM *
News: Latest Bitcoin Core release: 28.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Bitcoin Blockchain dump/backup  (Read 3107 times)
Med2k (OP)
Newbie
*
Offline Offline

Activity: 7
Merit: 0


View Profile
December 10, 2012, 11:41:43 AM
Last edit: January 06, 2013, 09:41:16 AM by Med2k
 #1

Hello,
I`m new and I spend the last 4 or 5 days to sync the blockchain with the bitcoin-qt client. I can´t find a current blockchain backup on the internet, so I made a new dump and upload it for you.

date: 05-01-2013
current blockchain: ~215270
info:
bootstrap.001 >> 7aac5826b91b4f87a2e9534e0e38e8d64ed21aff8a4eb8ff8dde4e726e67fe1a (sha256sum)
bootstrap.002 >> 796f65be10ef2e5fc27b97b09f312c9f8ddd7d1c3ab0f27f356a5b1dbf5a8963 (sha256sum)
bootstrap.003 >> 7b8775e49419ebc48cd2ac7488b6a41425810009470f3fb994cd55294e1dc924 (sha256sum)


download: https://copy.com/eK8sgbbtdspu

[edit]
update 05-01-2013
BitcoinVideoPoker
Hero Member
*****
Offline Offline

Activity: 804
Merit: 500



View Profile WWW
December 10, 2012, 11:50:27 AM
 #2

Out of curiosity, why hasn't anyone offered the blockchain as a torrent download?

tbcoin
Legendary
*
Offline Offline

Activity: 1022
Merit: 1000



View Profile WWW
December 10, 2012, 11:55:58 AM
 #3

Never a good idea download it from outside the bitcoin network, may have been altered.

Sorry for my bad english Wink
Bitcoin card for deposit and payment + Little POS
Donations:1N65efiNUhH6sEQg7Z6oUC76kJS9Yhevyf
Med2k (OP)
Newbie
*
Offline Offline

Activity: 7
Merit: 0


View Profile
December 11, 2012, 09:27:55 AM
 #4

hm, wouldn`t  the network know it if its altered? I understand it in this way, that the client
check the blockchain when it starts with the network and if there is a collision than he will
fix it?! Or is this wrong?

If not than there is a big problem in the future. If the blockchain reached 10 GB or more,
you can`t never start to download the whole blockchain from the network or it takes
2 month and more Smiley
Maybe you can take down the network to download the blockchain with many clients or
disturb the capacity?

hm, maybe a hash over the blk00x files will help to warrant that the file is ok if the bitcoin
guys post the hashes on there site?
gweedo
Legendary
*
Offline Offline

Activity: 1498
Merit: 1000


View Profile
December 11, 2012, 09:36:32 AM
 #5

hm, wouldn`t  the network know it if its altered? I understand it in this way, that the client
check the blockchain when it starts with the network and if there is a collision than he will
fix it?! Or is this wrong?

If not than there is a big problem in the future. If the blockchain reached 10 GB or more,
you can`t never start to download the whole blockchain from the network or it takes
2 month and more Smiley
Maybe you can take down the network to download the blockchain with many clients or
disturb the capacity?

hm, maybe a hash over the blk00x files will help to warrant that the file is ok if the bitcoin
guys post the hashes on there site?

It only validates it upon downloading the block, if it is already in the database it doesn't validate it. Size of the blockchain has been taken care of in the 0.8.0 Version which you can test right now.
Graet
VIP
Legendary
*
Offline Offline

Activity: 980
Merit: 1001



View Profile WWW
December 11, 2012, 09:47:44 AM
 #6

it is offered as a torrent download by jgarzik -one of the Bitcoin devs- https://bitcointalk.org/index.php?topic=117982.0
I have it waiting on a server with a fast pipe, you need to enable DHT though

| Ozcoin Pooled Mining Pty Ltd https://ozcoin.net Double Geometric Reward System https://lc.ozcoin.net for Litecoin mining DGM| https://crowncloud.net VPS and Dedicated Servers for the BTC community
Med2k (OP)
Newbie
*
Offline Offline

Activity: 7
Merit: 0


View Profile
December 11, 2012, 03:30:00 PM
 #7

Ok I test the bootstrap.dat way once but it takes a long time to verify the data, so I cancel it.
What will you do if your harddisc will crash and you loose your whole blockchain?
Have you got a backup of it?



btw: update of my first post
gweedo
Legendary
*
Offline Offline

Activity: 1498
Merit: 1000


View Profile
December 11, 2012, 05:27:20 PM
 #8

Ok I test the bootstrap.dat way once but it takes a long time to verify the data, so I cancel it.
What will you do if your harddisc will crash and you loose your whole blockchain?
Have you got a backup of it?

It really doesn't take that long, I actually have setup about 3 different bitcoind, 2 main network and 1 test network, and waiting a day, isn't too bad. So why back it up and waste that space.
deepceleron
Legendary
*
Offline Offline

Activity: 1512
Merit: 1036



View Profile WWW
December 11, 2012, 06:39:15 PM
 #9

The blk0001.dat checksum posted is different than one that I obtain on my copy of the blk0001.dat (using the awkward QuickSFV program), and my file is identical to the one generated by pynode off a network download, one that would be created by doing an import of the bootstrap torrent, and the file in luke-jr's torrent. In addition, even if the CRC posted was the same, it is trivial to make a CRC32 collision, as it is not a cryptographically strong algorithm.

Also, the blockchain database files cannot be used independent of the log sequence files which are in the database subdirectory, unless the database is specially prepared beforehand.

I have a thread where I (had) hosted blockchain files, it also has links to two torrents and others who are hosting the bootstrap blockchain.

If you wish to prepare a blockchain download in a way that the integrity can be independently verified, I can help you in this, but it will take more time and software than just copying files out of the datadir.

Med2k (OP)
Newbie
*
Offline Offline

Activity: 7
Merit: 0


View Profile
December 12, 2012, 08:58:56 AM
 #10

hi deepceleron
I have interests to generate a boottrap for the blockchain. Have you got a tutorial?

Thx
Med2k
Grecoin
Member
**
Offline Offline

Activity: 63
Merit: 10



View Profile
December 12, 2012, 02:38:07 PM
 #11

but why rush ..take your time

enjoy ..count the blocks as they download

way better than counting sheeps  Tongue
deepceleron
Legendary
*
Offline Offline

Activity: 1512
Merit: 1036



View Profile WWW
December 15, 2012, 03:33:26 AM
Last edit: March 12, 2013, 05:40:52 PM by deepceleron
 #12

The first thing you need to do is get pynode (python language mini-Bitcoin node) software working. This is easier on Linux, since on Windows you will need to build leveldb to use py-leveldb, which will require lots of C++ stuff installed. I use Kubuntu 12.04 on VirtualBox, since this allows you to have a second "Bitcoin" on another virtual IP address to connect to your main computer node.

https://github.com/jgarzik/pynode
(https://bitcointalk.org/index.php?topic=94645.0)

Here are the steps to get the required python extensions on (k)ubuntu 12:

sudo apt-get install python-setuptools
sudo apt-get install python-leveldb
sudo apt-get install libevent-dev
sudo apt-get install python-all-dev
sudo easy_install greenlet
sudo easy_install gevent


Then download and unzip pynode to a working directory. I have it in the directory /home/USER/pynode/, where USER is the logon username. I also created a subdirectory /home/USER/pynode/db for the blockchain.

You will need to create a pynode-config file. You can cut and paste the file below, but edit the IP address to be another (your local) Bitcoin node, and change the /home/USER lines to your login user name directory:

pynode-config
Code:
# hostname or IP address of network node to connect to
host=192.168.1.101

# port of network node to connect to (default: 8333)
port=8333

# JSON-RPC server user, password.  Uses HTTP Basic authentication.
rpcuser=XXXX
rpcpass=YYYY

# JSON-RPC server incoming TCP port (default: 9332)
rpcport=9332

# database directory
db=/home/USER/pynode/db

# log filename, or '-' or no-value for standard output
log=/home/USER/pynode/db/node.log

# if present, import these blocks into the block database
#loadblock=/tmp/blk0001.dat

# if present, disable all signature checking in new blocks
# (disabled by default)
nosig=1

# if present, force signature checking on all blocks,
# even those normally skipped because they were prior
# to a checkpoint.
# (disabled by default)
#forcesig=1

Then run it with this command:
cd ~/pynode
./node.py pynode-config


It will download all blocks from your other Bitcoin over the network (but go much faster without re-verification).
After it is done downloading (by checking it's log file), you can abort it.


To generate a bootstrap.dat
Find and edit the mkbootstrap.py file, changing the log and db directories to the directories used above. Then run it, to dump a bootstrap.dat up to the checkpoint block listed on the line that says xrange(216116...


To generate importable blocks in multi-part files
Alternately, for Bitcoins less than version 0.8.0, run my modified mkbootstrap.py file that dumps the blockchain and splits the files at exactly the same blocks and size as Bitcoin creates in it's datadir:

Code:
#!/usr/bin/python
#
# mkbootstrap.py
#
# Distributed under the MIT/X11 software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#


import sys
import Log
import MemPool
import ChainDb
import cStringIO
import struct

from bitcoin.coredefs import NETWORKS
from bitcoin.core import CBlock
from bitcoin.scripteval import *

NET_SETTINGS = {
'mainnet' : {
'log' : '/home/USER/pynode/mkbootstrap.log',
'db' : '/home/USER/pynode/db'
},
'testnet3' : {
'log' : '/home/USER/pynode/mkbootstraptest.log',
'db' : '/home/USER/pynode/chaintest'
}
}

MY_NETWORK = 'mainnet'

SETTINGS = NET_SETTINGS[MY_NETWORK]

log = Log.Log(SETTINGS['log'])
mempool = MemPool.MemPool(log)
netmagic = NETWORKS[MY_NETWORK]
chaindb = ChainDb.ChainDb(SETTINGS, SETTINGS['db'], log, mempool, netmagic,
 True)

out1 = open('bootstrap.001', 'wb')
out2 = open('bootstrap.002', 'wb')
out3 = open('bootstrap.003', 'wb')


scanned = 0
failures = 0

for height in xrange(212000+1):     # last block to dump (checkpoint)
heightidx = ChainDb.HeightIdx()
heightstr = str(height)
try:
heightidx.deserialize(chaindb.db.Get('height:'+heightstr))
except KeyError:
log.write("Height " + str(height) + " not found.")
continue

blkhash = heightidx.blocks[0]

block = chaindb.getblock(blkhash)

ser_block = block.serialize()
if (height % 1000 == 0 and height <= 90000) or (height % 100 == 0 and height > 90000):
 print "mkbootstrap: block height " + str(height)
outhdr = netmagic.msg_start
outhdr += struct.pack("<i", len(ser_block))
if height > 210964:        # write 210965- to 3rd file
out3.write(outhdr)
out3.write(ser_block)
elif height > 188528:        # write 188529-210964 to 2nd file
  out2.write(outhdr)
out2.write(ser_block)
else:
out1.write(outhdr)
out1.write(ser_block)

scanned += 1
if (scanned % 1000) == 1:
log.write("Scanned height %d (%d failures)" % (
height, failures))

log.write("Scanned %d blocks (%d failures)" % (scanned, failures))

This will create the "official/final/verifiable/will never change" first two blockchain files (containing blocks up to 210964), blk0001/blk0002.dat in Bitcoin up to v0.7.x will also have the same hash after you import:

>sha256sum bootstrap.*
7aac5826b91b4f87a2e9534e0e38e8d64ed21aff8a4eb8ff8dde4e726e67fe1a *bootstrap.001
796f65be10ef2e5fc27b97b09f312c9f8ddd7d1c3ab0f27f356a5b1dbf5a8963 *bootstrap.002

The script will also create a bootstrap.003 with remaining blocks (up to 212000 in script). We'll ignore this one for now. The original mkbootstrap script dumped a single bootstrap.dat file up to block 193000; if a future checkpoint is set, the final part 003 of will include these blocks.

(Google for your own copy of sha256sum.exe. Put it in the Windows directory and you can run it from anywhere.)

Let's import the blocks into Bitcoin, this will take six hours with an SSD and fast CPU (3GHz Core2), even more time with regular hard drive and less CPU. Here's a Windows command line, import to a blank C:\datadir:

"C:\Program Files (x86)\Bitcoin\daemon\bitcoind.exe" -datadir=C:\datadir -loadblock=C:\bootstrap.001 -loadblock=C:\bootstrap.002 -connect=127.0.0.1 -detachdb -printtoconsole

This will create identical block file hashes as the imported files, and will build the final index file. Bitcoin won't proceed with network download after the last block of the import, instead it will abort, since you don't have bitcoin.conf. Here is the index file's hash on 0.7.1:

1ab115fd8338f0f4d112c8465418619f1219058a2e6f9baeb7621f39b362a177 *blkindex.dat

By using this method, others can create the exact same blockchain files as you from a network download, import them and see that they get the exact same files in their data directory as you created.
Med2k (OP)
Newbie
*
Offline Offline

Activity: 7
Merit: 0


View Profile
December 16, 2012, 10:39:36 AM
Last edit: December 16, 2012, 11:05:31 AM by Med2k
 #13

thank you for the tutorial. I start the script a few minutes ago. We will see how long it takes. Smiley

[edit]
ok i have got a problem with your script.
if i start mkbootstrap.py it will say "no permission" and if i start it with sudo it will say "command not found".

if I check your script there are some paths which are not exists on my linux (kubuntu in a VM)
Quote
NET_SETTINGS = {
   'mainnet' : {
      'log' : '/home/user/pynode/mkbootstrap.log',
      'db' : '/home/user/pynode/chaindb'
   },
   'testnet3' : {
      'log' : '/home/user/pynode/mkbootstraptest.log',
      'db' : '/home/user/pynode/chaintest'
   }
}

I haven´t got chaindb or chaintest or one of these logfiles. I`ve got home/user/pynode/db/leveldb with data. Any idea to fix it?
deepceleron
Legendary
*
Offline Offline

Activity: 1512
Merit: 1036



View Profile WWW
December 16, 2012, 10:46:22 PM
Last edit: February 06, 2013, 05:49:48 PM by deepceleron
 #14

If you create a text file and want to make it into an executable script, you'll usually need to chmod 755 the file to make it executable. You can also just edit the original mkbootstrap.py and save it as my version.

Pynode will have to download the blockchain first - it is it's own Bitcoin client, but it only has a few functions of real Bitcoin. The config script I gave is an example, and has directory locations I indicated you should change, like if your logon user name is "med2k", and you extract pynode to it's own subdirectory in your home directory, then the config file paths would start with "/home/med2k/pynode" because "user" in the script is where your username goes.

There is no need to sudo, nothing needs to run as root.
Med2k (OP)
Newbie
*
Offline Offline

Activity: 7
Merit: 0


View Profile
December 17, 2012, 09:45:53 AM
 #15

ahhh thx chmod was the fault. Unfortunately I´m a child of windows and I start with linux in a VM a few months ago but I only playing a little bit. now the script is running. thx you for your help.
Med2k (OP)
Newbie
*
Offline Offline

Activity: 7
Merit: 0


View Profile
December 18, 2012, 08:54:37 PM
 #16

intermediate result all sha256 hashes are the same.
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!