Blockchain Download Time

[Charles Sweeney]

March 17th 2018.  Just for reference.

Just downloaded the entire blockchain.

Number of blocks: 513,962

Download time: 2 days 3 hours

Size on disk: 172GB

Used an Amazon Web Services EC2 t2.small server instance, running Ubuntu. 2GB memory.  Was struggling a bit, hence the download time.  CPU running at 100% constantly.  Memory was fine but I think more would be needed if running a faster processor.  Not sure of the CPU specs on the t2.small instance but it's one of their lowest performing (and cheapest) instances.

[1714023995]

1714023995

[1714023995]

1714023995

[1714023995]

1714023995

[1714023995]

1714023995

[Za1n]

That sounds about right, I just downloaded the blockchain on a new install I setup in January and remembering it took about 30 hours to complete.

Anticipating continued blockchain growth, the dedicated PC I build uses a 1 TB SSD, so hopefully I still can get a couple years service out of it before needing to upgrade again.

[Charles Sweeney]

Anticipating continued blockchain growth, the dedicated PC I build uses a 1 TB SSD, so hopefully I still can get a couple years service out of it before needing to upgrade again.

Will be interesting to see how big it gets, with the increased popularity of bitcoin.

Wondering if the data files can simply be copied to another disk?  Would save downloading and processing everything again when it comes time to upgrade your system.

[aleksej996]

Wondering if the data files can simply be copied to another disk?

Yes. It might not be a good security practice, as your new server won't validate the blocks again, but if you trust the data, sure.
You will need to copy everything though, not just the blocks directory, but chainstate as well, otherwise it will start block validation process from the beginning.

[Oulay]

Will be interesting to see how big it gets, with the increased popularity of bitcoin.

The size of the blockchain has nothing to do with Bitcoin adoption. Blockchain's size increases with every block added to the blockchain, which usually happens every ten minutes. And each block of 1 MB size increases the size of the blockchain by 1 MB.

[DannyHamilton]

Wondering if the data files can simply be copied to another disk?

It can.

Would save downloading and processing everything again when it comes time to upgrade your system.

Correct.

Additionally, if you don't want to store (and copy when you upgrade your system) the entire blockchain, you can turn on pruning mode.  That will purge old blocks that have already been processed, and drastically reduce the amount of storage space you'll need.

[wilwxk]

But the current size of blockchain is a problem if continue growing like now, there will no reason to store 200 gb of blockchain while you can only use a wallet which connects to other nodes, and this will be problematic for the decentralization, with the solutions like segwot and lighting network, maybe this problem can be solved.

[bob123]

But the current size of blockchain is a problem if continue growing like now, there will no reason to store 200 gb of blockchain while you can only use a wallet which connects to other nodes, and this will be problematic for the decentralization

The answer to your 'problem' has been posted 38 minutes before you have decided to post:

Additionally, if you don't want to store (and copy when you upgrade your system) the entire blockchain, you can turn on pruning mode.  That will purge old blocks that have already been processed, and drastically reduce the amount of storage space you'll need.

with the solutions like segwot and lighting network, maybe this problem can be solved

Neither segwit nor the lightning network result in a smaller blockchain to store.
You can read more about segwit here: https://cointelegraph.com/explained/segwit-explained and more about the LN here: https://en.bitcoin.it/wiki/Lightning_Network

[wilwxk]

Neither segwit nor the lightning network result in a smaller blockchain to store.
You can read more about segwit here: https://cointelegraph.com/explained/segwit-explained and more about the LN here: https://en.bitcoin.it/wiki/Lightning_Network

No, they cannot "compress" the current data, but with the adoption of the segwit they can process more transactions using the same amount of data, and the lightning network can reduce the use of registry in blockchain since we can make all outside the network.

[bob123]

No, they cannot "compress" the current data, but with the adoption of the segwit they can process more transactions using the same amount of data

Including more transaction into one block still doesn't reduce the size of the blockchain.
Regardless of segwit/non-segwit blocks are (mostly) full.

and the lightning network can reduce the use of registry in blockchain since we can make all outside the network.

Blocks still will be mined at the same rate.
And if the mempool would be empty (because somehow everyone is using LN without closing channels, whatsoever..) transactions with 1 sat/B would be confirmed within 1 block,
leading to people using on-chain transactions to pay merchants etc.. again.
Why should someone fund a channel and use the LN if fees would be 1 sat/B ? -> This will lead to full blocks again.

So, no. The lightning network does not decrease the size of the blockchain.

[Jet Cash]

Wondering if the data files can simply be copied to another disk?  Would save downloading and processing everything again when it comes time to upgrade your system.

I've got 3 copies of the blockchain ( plus the associated files and wallet data), and I run 2 nodes.

I copied the directory onto an SSD, and gave that to a new installation of Core under Ubuntu. It asked me for the directory address, and when I replied, it went ahead and veried the copied chain, and synchronised it. It even gave me the correct info in the wallet.

I'm impressed with core, it seems to be extremely stable, it even runs under Windows without any problems.

[Charles Sweeney]

The size of the blockchain has nothing to do with Bitcoin adoption. Blockchain's size increases with every block added to the blockchain, which usually happens every ten minutes. And each block being of 1 MB in size would increase the size of the blockchain by 1 MB.

Right.  I'm very new to this.  I wasn't thinking about the size of the chain so much as the size of the blocks.  What if there's tens of thousands, or hundreds of thousands of new transactions per block (as there will be when everyone starts using bitcoin on a regular basis!).  All those transactions in one block of 1MB?

[Charles Sweeney]

Thanks all for the very helpful replies.  I noticed the prune option but I wasn't sure about it (being very new to this).  My understanding was that you needed the full blockchain for maximum integrity.  What if everyone pruned their blockchain, would that mean there wouldn't be any full copies of the blockchain?!  Wouldn't that be a problem?

Yes, Jet Cash, I remember seeing an option to state the data directory.  That would be very handy to have the .bitcoin directory on a SSD.  Initially I was thinking you would just need the blocks and the chainstate data but it makes sense, and is a lot easier, simply to copy the entire directory.

[DannyHamilton]

What if there's tens of thousands, or hundreds of thousands of new transactions per block

This is not possible unless there is a fork and we change the block weight or block size.

Right now, if too many transactions are sent, then the excess transactions that don't fit into the next block must wait until a later block. Those that want fast confirmation of their transaction therefore increase their transaction fee to get into the next block.  The transactions with lower fees are then left waiting a VERY long time to confirm.  Eventually, people that aren't willing to wait abandon the Bitcoin blockchain and use some other method for making their transactions.  Those other methods may or may not Bitcoin off-chain transactions (such as Lightning Network or Account based services like Coinbase)

All those transactions in one block of 1MB?

No.  They just wait for later blocks.

My understanding was that you needed the full blockchain for maximum integrity.

You need to have received and processed the entire blockchain if you want to be 100% certain that you have not been fed any false information about the current state of the UTXO. However, you do not need to have stored/kept that entire history.  You only need to keep the UTXO.

What if everyone pruned their blockchain, would that mean there wouldn't be any full copies of the blockchain?!  Wouldn't that be a problem?

Yes. That would be a problem.  Fortunately, there are many of us that WANT and are willing to store the entire blockchain for various reasons. If you are able to store the entire blockchain and you want the entire blockchain (or you just want to be able to help other new users by supplying the entire blockchain), then you are welcome to do so and we appreciate your willingness to do so.  If you are unable to , or you do not want to, then we can manage without you.

[ranochigo]

My understanding was that you needed the full blockchain for maximum integrity.

You don't. The block files are pretty useless after you've verified them and have gotten the chainstate and other essential information. If your computer can verify its legitmacy, there is no problem throwing them away. You could have problems when importing addresses, etc. None of the potential problems actually affect the security; could just cause some inconvenience.

What if everyone pruned their blockchain, would that mean there wouldn't be any full copies of the blockchain?!  Wouldn't that be a problem?

Yes. If everyone pruned their blockchain, then it would be a huge problem. Since the wallet prune itself as you get more blocks, no one would ever see the entire history if that were to happen. And yes, it would compromise the security greatly in that case. IMO, pruning is just for people who wants to run Bitcoin Core and to support the network, not really that much as to helping the network (debatable but thats my opinion).

[Cryptagio]

Will be interesting to see how big it gets, with the increased popularity of bitcoin.

The size of the blockchain has nothing to do with Bitcoin adoption. Blockchain's size increases with every block added to the blockchain, which usually happens every ten minutes. And each block being of 1 MB in size would increase the size of the blockchain by 1 MB.

Actually it has. You do not have to run a node to be able to transact. But you will be highly reliant on nodes that your wallet uses for broadcasting your transactions to the network. As a medium and large size business you would like to be able to control when you broadcast your txs, be sure that wallets do not falsify your balances, there are no MIM attacks. So having a node, thus, downloading blockchain is quite necessary.

[DannyHamilton]

You do not have to run a node to be able to transact.

Correct.

But you will be highly reliant on nodes that your wallet uses for broadcasting your transactions to the network.

FALSE.

Running a node does not make you any more or less reliant on other nodes for broadcasting your transactions.

As a medium and large size business you would like to be able to control when you broadcast your txs,

Correct.

be sure that wallets do not falsify your balances,

What balances?  What are you talking about?  Why wouldn't you know what your own balance is?

That certainly sounds like a load of nonsense.

So having a node, thus, downloading blockchain is quite necessary.

This isn't a question about downloading the blockchain.  This is a question about storing the blockchain. Did you even read this thread before posting your nonsense?

[Charles Sweeney]

Thanks all.  Good info!  I would like to keep a full node to help the network.  Just tinkering for now!

[Charles Sweeney]

Those that want fast confirmation of their transaction therefore increase their transaction fee to get into the next block.

Do the mining programs automatically select the transactions with the highest fees?  Been looking at BFGMiner but didn't notice a configuration for this or any mention of it in the README.

The transactions with lower fees are then left waiting a VERY long time to confirm.  Eventually, people that aren't willing to wait abandon the Bitcoin blockchain and use some other method for making their transactions.  Those other methods may or may not Bitcoin off-chain transactions (such as Lightning Network or Account based services like Coinbase)

Just been reading some very worrying articles about very high bitcoin transaction fees:

http://uk.businessinsider.com/bitcoin-payment-mining-fees-hit-new-high-2017-12
m/2017/12/19/big-transactions-fees-are-a-problem-for-bitcoin.html]https://www.c[Suspicious link removed]m/2017/12/19/big-transactions-fees-are-a-problem-for-bitcoin.html

...meaning no-one is going to use it to buy coffee or pizza.  One of the articles mentions Lightning Network with instant transactions and low fees, which to me looks very interesting.

[DannyHamilton]

Do the mining programs automatically select the transactions with the highest fees?  Been looking at BFGMiner but didn't notice a configuration for this or any mention of it in the README.

If you are mining in a mining pool, then the pool is selecting the transactions and building the blocks.  Then they just hand off the block header (or, more specifically, the partially hashed header called the midstate) to your software for you to complete the hashing on. In that case, the pool is probably running custom software to build the block headers.

If you are solo mining, then your mining software is probably getting its block headers from Bitcoin Core using a getwork API call.  In that case, Bitcoin Core has the transaction selecting algorithm coded into it. I haven't done any mining, so I don't know how configurable that algorithm is.  You do have a few command line options when starting up Bitcoin Core:  

• -blockmaxweight=<n>        Set maximum BIP141 block weight (default: 3000000)
• -blockmaxsize=<n>            Set maximum block size in bytes (default: 750000)
• -blockprioritysize=<n>        Set maximum size of high-priority/low-fee transactions in bytes (default: 0)
• -blockmintxfee=<amt>       Set lowest fee rate (in BTC/kB) for transactions to be included in block creation. (default: 0.00001)

The transactions with lower fees are then left waiting a VERY long time to confirm.  Eventually, people that aren't willing to wait abandon the Bitcoin blockchain and use some other method for making their transactions.  Those other methods may or may not Bitcoin off-chain transactions (such as Lightning Network or Account based services like Coinbase)

Just been reading some very worrying articles about very high bitcoin transaction fees:

http://uk.businessinsider.com/bitcoin-payment-mining-fees-hit-new-high-2017-12
m/2017/12/19/big-transactions-fees-are-a-problem-for-bitcoin.html]https://www.c[Suspicious link removed]m/2017/12/19/big-transactions-fees-are-a-problem-for-bitcoin.html

...meaning no-one is going to use it to buy coffee or pizza. 

Correct.  Assuming there aren't any significant changes to the protocol to allow more on-chain transactions, eventually Bitcoin will either die or it will develop high enough transaction fees that nobody will be using it for small value on-chain transactions.

One of the articles mentions Lightning Network with instant transactions and low fees, which to me looks very interesting.

Lightning Network is one of a few off-chain solutions to allow a large number of off-chain Bitcoin based transactions.  Another off-chain solution is Bitcoin based accounts with services such as Coinbase or BitPay.  Each has their own benefits and issues, and each will probably find use in various circumstances.