Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: Indianacoin on January 28, 2016, 10:11:21 AM



Title: How to deal with 80 GB Blockchain?
Post by: Indianacoin on January 28, 2016, 10:11:21 AM
We already know that we have to download Bitcoin Core wallet if we want to contribute the network by running the full node of the blockchain.

But it will be a tedious job download the full chain of size 80 GB again if there happens something wrong with one's wallet and it needs to restart a wallet from the beginning.

So is there a recent bootstrap file available anywhere so that I can keep it as a backup in case I need it whenever I start a new wallet?
Or can anyone help me out on how to create a bootstrap file myself directly from the wallet files?

Technical helps are much appreciated :)


Title: Re: How to deal with 80 GB Blockchain?
Post by: Patatas on January 28, 2016, 10:18:13 AM
Are you really sure that it will start the entire wallet from the beginning in case of any issues ?I think even while running a full node it will resume from the point where it stopped the next time you start it.


Title: Re: How to deal with 80 GB Blockchain?
Post by: shorena on January 28, 2016, 10:45:57 AM
There are a few cases where you need to redownload all data, yes. They are not common though. If you want to be prepared, make regular backups of the entire folder. A bootstrap.dat would not be verified, copied the folder includes all verified data (and your wallet.dat).


Title: Re: How to deal with 80 GB Blockchain?
Post by: Indianacoin on January 28, 2016, 11:15:33 AM
Are you really sure that it will start the entire wallet from the beginning in case of any issues ?I think even while running a full node it will resume from the point where it stopped the next time you start it.

Yes there are many such cases possible. Sometimes your wallet's blkindex.dat might be malfunctioning. In this case you'll have to delete all your files from your 'roaming' folder and re-index your wallet from the beginning.
Also think of a situation where your computer crashes for some other reason and you have to format or change your hard drive.

In all these cases it will be easier to have a bootstrap file ready so that you can just import the chain offline and just re-index a week's chain which wend ahead at that time. It will save your bandwidth too.

If you want to be prepared, make regular backups of the entire folder. A bootstrap.dat would not be verified, copied the folder includes all verified data (and your wallet.dat).

So far I only am backing up my wallet.dat. What do mean by the entire folder? That folder in the roaming section?
But I am talking about a single bootstrap file in general which can be created and can be used by anyone (does not need any verification as such).


Title: Re: How to deal with 80 GB Blockchain?
Post by: boboniera on January 28, 2016, 11:24:20 AM
Blockchain needs only the 1-2 years old transactions. All the others must delete.


Title: Re: How to deal with 80 GB Blockchain?
Post by: Indianacoin on January 28, 2016, 11:45:59 AM
Blockchain needs only the 1-2 years old transactions. All the others must delete.

That is not how a cryptocurrency works ::)

The block chain is the decentralized ledger where all records of transactions ever made from the beginning of genesis block are recorded in a string like fashion.

Nobody can destroy the older chains just like that! Impossible fact.


Title: Re: How to deal with 80 GB Blockchain?
Post by: shorena on January 28, 2016, 01:58:47 PM
Blockchain needs only the 1-2 years old transactions. All the others must delete.

That is not how a cryptocurrency works ::)

The block chain is the decentralized ledger where all records of transactions ever made from the beginning of genesis block are recorded in a string like fashion.

Nobody can destroy the older chains just like that! Impossible fact.

It already does, the old data is not really verified, there are hardcoded checkpoints in place, but the actual blocks are not verified. The data is not deleted true, but its as close it deletion as it gets.

-snip-
So far I only am backing up my wallet.dat. What do mean by the entire folder? That folder in the roaming section?
But I am talking about a single bootstrap file in general which can be created and can be used by anyone (does not need any verification as such).

The entire ~/.bitcoin or %APPDATA/Bitcoin folder, yes. This would not be suitable for others unless they can trust you, but I would think you can trust yourself.
If you prefer a bootstrap.dat you can create one yourself[1] as well, but you must verify the data when you resync from scratch for some reason.

[1] Note 2nd post, not 1st (!) https://bitcointalk.org/index.php?topic=986676.msg10753296#msg10753296


Title: Re: How to deal with 80 GB Blockchain?
Post by: DumbFruit on January 28, 2016, 02:13:19 PM
Blockchain needs only the 1-2 years old transactions. All the others must delete.

That is not how a cryptocurrency works ::)

The block chain is the decentralized ledger where all records of transactions ever made from the beginning of genesis block are recorded in a string like fashion.

Nobody can destroy the older chains just like that! Impossible fact.
Depends on how you define "cryptocurrency". I think the only way it could be defined without excluding something that currently calls itself a cryptocurrency would be a medium of exchange that exists exclusively digitally. (In which case even fiat currency would be a "cryptocurrency" when they abolish cash, like is happening in Sweden.) I would be interested if anyone could name any other distinguishing feature of a "cryptocurrency" (Something applicable to every variation of a cryptocurrency).
In any case, there are several ways that old transactions can be deleted, including demurrage, pruning, and a "rolling blockchain".

To answer the OP, it's fastest just to download the blockchain with the Bitcoin client. After  you have the blockchain, you can keep it stored somewhere (bootstrap.dat) so you don't have to download the whole thing, the client will verify the blockchain on import and then download the rest of the blockchain from where it left off.

You can create bootstrap.dat by concatenating all of the blk*.dat files located in your Bitcoin directory. On Windows it would look like this;
copy /b blk0001.dat+blk0002.dat ...  bootstrap.dat

https://bitcoin.org/bin/block-chain/README.txt
http://bitcoin.stackexchange.com/questions/10381/creating-my-own-bootstrap-dat


Title: Re: How to deal with 80 GB Blockchain?
Post by: Jet Cash on January 28, 2016, 04:32:46 PM
A 128Gb USB stick is so cheap, it seems mad not to back up the Bitcoin directory. As I understand it, you don't have to backup every day. Core will verify the recovered directory, and then build on that from the public blockchain.


Title: Re: How to deal with 80 GB Blockchain?
Post by: shorena on January 28, 2016, 07:28:11 PM
A 128Gb USB stick is so cheap, it seems mad not to back up the Bitcoin directory. As I understand it, you don't have to backup every day. Core will verify the recovered directory, and then build on that from the public blockchain.

Yes, once a week even once a month would easily be enough. With 128 GB you can even hold two backups, at least for a while.


Title: Re: How to deal with 80 GB Blockchain?
Post by: Indianacoin on January 28, 2016, 07:33:25 PM
A 128Gb USB stick is so cheap, it seems mad not to back up the Bitcoin directory. As I understand it, you don't have to backup every day. Core will verify the recovered directory, and then build on that from the public blockchain.

Yes, once a week even once a month would easily be enough. With 128 GB you can even hold two backups, at least for a while.

Thanks for the advice but how exactly do I compile a bootstrap file? Like bootstrap.dat?

I do not want to backup the whole folder anyways.

And is it safe to upload the bootstrap.dat on a cloud?
I think yes because bootstrap does not save sensitive keys of wallet.dat inside it.


Title: Re: How to deal with 80 GB Blockchain?
Post by: shorena on January 28, 2016, 08:53:53 PM
A 128Gb USB stick is so cheap, it seems mad not to back up the Bitcoin directory. As I understand it, you don't have to backup every day. Core will verify the recovered directory, and then build on that from the public blockchain.

Yes, once a week even once a month would easily be enough. With 128 GB you can even hold two backups, at least for a while.

Thanks for the advice but how exactly do I compile a bootstrap file? Like bootstrap.dat?

I do not want to backup the whole folder anyways.

And is it safe to upload the bootstrap.dat on a cloud?
I think yes because bootstrap does not save sensitive keys of wallet.dat inside it.

This[1] should do it. I cant give you any tipps though since I never bothered to create one. I might do this to help migrating to my new server without hammering the CPU/Disk, but not today or tomorrow.

[1] https://github.com/bitcoin/bitcoin/tree/master/contrib/linearize


Title: Re: How to deal with 80 GB Blockchain?
Post by: achow101 on January 28, 2016, 11:18:55 PM
Why download the block chain, can't we just all use online wallets  ?
Someone has to have the blockchain, we still need people to run full nodes so that the entire blockchain is available to those online wallets. And not everyone trusts web wallets. It is more likely that people will use SPV wallets as they are safer than web wallets, but SPV wallet still need people to run full nodes so that they can query information from them.


Title: Re: How to deal with 80 GB Blockchain?
Post by: Jet Cash on January 29, 2016, 05:39:56 AM
I don't trust web wallets. I'm wondering if it will be possible to run 0.12 on a mobile phone to create a pruned node using an SD card. Bandwidth deals are getting insane these days as the supermarkets compete for customers, and most phones have got WiFi and bluetooth now. I guess the problem would be loading the 80Gb blockchain prior to the pruning.


Title: Re: How to deal with 80 GB Blockchain?
Post by: shorena on January 29, 2016, 08:06:45 AM
Why download the block chain, can't we just all use online wallets  ?

If everyone would use online wallets, bitcoin would in consequence be no different than paypal.


Title: Re: How to deal with 80 GB Blockchain?
Post by: btcdevil on January 29, 2016, 08:11:39 AM
Why download the block chain, can't we just all use online wallets  ?

If everyone would use online wallets, bitcoin would in consequence be no different than paypal.

Most of the online wallets are changing address with new transactions and they dont even give signed message option if any problem comes with the wallet address associated with any account or transaction so for this i think hardware wallet is much better and safe. because it will be in full control of the user.


Title: Re: How to deal with 80 GB Blockchain?
Post by: onlinedragon on January 29, 2016, 08:20:23 AM
Why download the block chain, can't we just all use online wallets  ?

If everyone would use online wallets, bitcoin would in consequence be no different than paypal.
Never there will be only online wallet users. When it's 80gb+ for the whole blockchain I really understand people don't want the full version. Light wallet is for most of the people good enough for me XAPO worked perfectly.


Title: Re: How to deal with 80 GB Blockchain?
Post by: Jet Cash on January 29, 2016, 08:44:12 AM
What's the big deal about an 80Gb directory. The file sizes are still fairly small, and the notebook I am using now has a 2Tb hard drive. It wasn't that expensive either. If you are on a chromebook, or a mobile, then I can see it would matter, but for a reasonable PC, I'd rather have full control of my wallet.


Title: Re: How to deal with 80 GB Blockchain?
Post by: Moloch on January 30, 2016, 01:22:48 AM
What's the big deal about an 80Gb directory. The file sizes are still fairly small, and the notebook I am using now has a 2Tb hard drive. It wasn't that expensive either. If you are on a chromebook, or a mobile, then I can see it would matter, but for a reasonable PC, I'd rather have full control of my wallet.

I suppose it depends on the situation.

The web hosting service I use charges $0.10 per Gb. That's $8/month, nearly $100 per year... Plus bandwidth costs...

Running a full node on my website would cost $1-200/year, and the blockchain is certainly not getting smaller


Title: Re: How to deal with 80 GB Blockchain?
Post by: cr1776 on January 30, 2016, 01:33:03 AM
What's the big deal about an 80Gb directory. The file sizes are still fairly small, and the notebook I am using now has a 2Tb hard drive. It wasn't that expensive either. If you are on a chromebook, or a mobile, then I can see it would matter, but for a reasonable PC, I'd rather have full control of my wallet.

I suppose it depends on the situation.

The web hosting service I use charges $0.10 per Gb. That's $8/month, nearly $100 per year... Plus bandwidth costs...

Running a full node on my website would cost $1-200/year, and the blockchain is certainly not getting smaller

Just run it in pruning mode then. If you limit it to 10 GB, then that is only $1/month.

Out of curiosity, what host are you using? (If you don't want to share, no big deal).


Title: Re: How to deal with 80 GB Blockchain?
Post by: BlockSense on January 30, 2016, 07:15:57 AM
Pruning is most definitely the easiest solution to this issue. One could say the blockchain size is a testament to bitcoinss growing usage. I feel some servers (digitalocean?) should have a "deploy bitcoin core" service that will help save on initial startup. Storage and hosting is getting better and cheaper but still has some way to go before cheaper solutions are worthwhile for both company and customers. 


Title: Re: How to deal with 80 GB Blockchain?
Post by: pereira4 on January 30, 2016, 05:29:52 PM
It's a pain in the ass to be honest. Maintaining it is not that hard work, but download it from scratch? That's a nightmare. I just hope that it never corrupts again. It sometimes has corrupted after upgrading the software. Honestly if it happens again I may give up and run pruned mode in 0.12. And to think there's people that want to make it grow twice as fast, what are they smoking? I think we need to wait a bit longer to raise it so the average user has better internet and hardware.


Title: Re: How to deal with 80 GB Blockchain?
Post by: unamis76 on January 30, 2016, 07:42:59 PM
Buy a new drive or more VPS space :)


Title: Re: How to deal with 80 GB Blockchain?
Post by: Hannu on January 31, 2016, 02:36:51 AM
We already know that we have to download Bitcoin Core wallet if we want to contribute the network by running the full node of the blockchain.

But it will be a tedious job download the full chain of size 80 GB again if there happens something wrong with one's wallet and it needs to restart a wallet from the beginning.

So is there a recent bootstrap file available anywhere so that I can keep it as a backup in case I need it whenever I start a new wallet?
Or can anyone help me out on how to create a bootstrap file myself directly from the wallet files?

Technical helps are much appreciated :)

Do not use online wallet? If you dont wanna run fullnode/node on bitcoin network? ::)

Use offline wallet example http://www.coinmotion.com/


Title: Re: How to deal with 80 GB Blockchain?
Post by: bob123 on January 31, 2016, 12:04:43 PM
A backup once per Week/month should be fine.
128 GB USB sticks arent expensive at all anymore


Title: Re: How to deal with 80 GB Blockchain?
Post by: Moloch on January 31, 2016, 07:45:46 PM
Just run it in pruning mode then. If you limit it to 10 GB, then that is only $1/month.

Out of curiosity, what host are you using? (If you don't want to share, no big deal).

AWS (https://aws.amazon.com)

They charge separately for things like HD space and bandwidth