Block chain size/storage and slow downloads for new users

[niko]

Are there any stats someone could provide as to how much bandwidth and storage per month are we talking about to run full node like Blockchain?

My "blocks" folder is currently at 11GB. Since block size is limited, the growth can only be linear in time (even if we occasionally increase the block size limit), but storage per dollar increases exponentially over time. Therefore, I don't see it as a problem in the long run. Also, see http://blockchain.info/charts/blocks-size

Bandwidth is currently a tougher question. I typically see bitcoin-qt using tens of kB/s, sometimes running at 200-300 kB/s for minutes or hours (this is mostly my upload to new users downloading old blocks). Unfortunately, I don't have any solid data to offer, just these observations.

[aficionadog]

Are there any stats someone could provide as to how much bandwidth and storage per month are we talking about to run full node like Blockchain?

My "blocks" folder is currently at 11GB. Since block size is limited, the growth can only be linear in time (even if we occasionally increase the block size limit), but storage per dollar increases exponentially over time. Therefore, I don't see it as a problem in the long run. Also, see http://blockchain.info/charts/blocks-size

Bandwidth is currently a tougher question. I typically see bitcoin-qt using tens of kB/s, sometimes running at 200-300 kB/s for minutes or hours (this is mostly my upload to new users downloading old blocks). Unfortunately, I don't have any solid data to offer, just these observations.

Thank you niko for the info. Those numbers look lite in all things considered.

[Winslow Strong]

Thanks, its nice to hear feedback like this, and get some more understanding of the more technical aspects.

[giszmo]

Ok, so lets assume one of Mike Hearn's microtransaction channel per person with a one year timeout was used. This way only 2 transactions would need to hit the blockchain per person per year. All the rest could be instant, micro and off the chain anonymous transactions without third party risk. The dream of every bitcoin user but:
8billion * 2kB = 16TB of data per year. With 50k blocks per year 16TB/50k = 300MB per block. Lets get there in 10 years, keep incentives to move transactions off the blockchain and nobody will complain. Even Satoshi Dice would work better with microtransaction channels anyway.

[ArticMine]

At some point Bitcoin-Qt will change such that it's able to delete old blocks. The details are still being worked out, but most likely you'll be able to say "Use up to 10 GB of disk space" and it will never use more than that. Nodes will broadcast how much of the chain they have and are able to serve. New nodes that are starting from scratch will have to search out other nodes that still have the full chain and sync from them, but any node that just wasn't online for a while and needs to grab the latest parts of the chain will be able to use most of the others. By controlling disk space usage, you can also indirectly control bandwidth usage (you can't upload data you don't have).

What happens, if in a hundred years from now, almost everyone is using the "Use up to 10 GB of disk space" feature? All the old nodes can probably keep up, but new nodes starting from scratch won't be able to find a node with the full chain, or have difficulty doing so.

Or is that a problem we shouldn't worry about for the next 50 years? I can see full nodes "centralizing" then, but it's possible that there will be entities or countries that will maintain a full node independent of other full nodes.

Right now, everyone running QT or the reference client is a full node.

I can see a far distant future where no normal or regular individual has a full node (except for the geeks / enthusiasts / those who can afford), but there will still be at least maybe 1 full node per country with the bigger countries having several, maybe one per university or one per "department" or "agency" or one per "private corporation".

I mean, large companies already maintain servers for all sorts of purposes. What's another one just dedicated to just bitcoind with 500++ gigs of space (or whatever is the then current capacity of hard drives or equivalent)?

I disagree because we are forgetting about the past in a very big way. While the size of the blockchain is growing at an exponential rate the amount of resources required to store and transmit data is falling at an even greater exponential rate so the ability of running a full node will remain within the financial abilities of regular individuals that choose to for years to come. I actually expect the number of full nodes to increase over time. They will for the most part lie with those who choose freedom, Free Software and GNU / Linux over those who choose subservience, DRM, propriety software, and locked devices such as those running IOS and recently some running Microsoft Windows.

Let us instead of trying to look 100 years into the future take a look at 100 years into the past, and ask the following question: How much would it have cost to send 1MB of data over the telegraph network in 1913? At say 0.50 USD per 10 words http://eh.net/encyclopedia/article/nonnenmacher.industry.telegraphic.us, and let say we can encode 4 bytes per word. This gives us 250000 words or 12500 USD in 1913 dollars. This was way outside of the budget of the average person, but something a millionaire (billionaire in today's USD) such as J. P. Morgan http://en.wikipedia.org/wiki/J._P._Morgan could have afforded. Today of course sending 1MB of data is so close to free that it is very affordable to even some very poor people.

There is a real danger here for Bitcoin it that many for fear of loosing the decentralized nature of Bitcoin to a growing blockchain, will cripple Bitcoin with arbitrary limits, while ignoring the real threat to the decentralized nature of Bitcoin. This threat is that in the future regular people will not be able to run a full Bitcoin node because their computing devices are locked down and infected with DRM. Cost will not be the limitation in the future DRM will. Even today I can purchase a computer perfectively capable of running a full Bitcoin node and pay for it with Bitcoin at FreeGeek http://www.freegeekvancouver.org/ for way less than the price of an iPad or Windows 8 RT "computer".  

[shuttleclock]

Thanks for the complete explanation. This is really a good reading, so at least I can tell some newbs about the reason why they should use Multibit for a start and not worry about it..

A quick question: Under what reason should we use Multibit instead of Electrum?

[Mike Hearn]

You can use either. Electrum relies on special/custom servers and offloads most of the work to them. MultiBit uses the regular P2P network. There are pros and cons to both models. I prefer the pure SPV/P2P model of course (which is why I implemented it!) but if you have some particular reason why you want to use Electrum then go for it.

[niko]

Just wanted to bring back this thread for a quick question:

C:\Users\MYUSER\AppData\Roaming\Bitcoin is 19.2GB. http://blockchain.info/charts/blocks-size says it should be under 10GB. How do I figure out what's unnecessary, and delete the extra files? Thanks.

This question often comes up. I thought the reason was the old blk000x.dat files remaining from the pre-0.7.8 (not sure) version, and which can be safely erased - but it would be nice if a developer here could confirm and explain.

Edit: Here is a relevant quote from deepceleron:

You can remove blk0001.dat, blk0002.dat, blk0003.dat and blkindex.dat from the root data directory after a reindex is complete and you are caught up with the blockchain (and you don't plan on going back to an older version). Only blkindex should actually be using disk space, as the old BLK000x data are moved upon upgrade, and the blk000x.dat files you see there are hardlinks (shortcuts) on any filesystem that supports hardlinking.
...
The new database for v0.8.0+ is stored in two subdirectories, "blocks" (block data, with block id database in "blocks/index"), and "chainstate" (unspent output database). Do not mess with files in these subdirectories.

[Shimple]

Hmmm. Thanks for the update. This is very interesting!

[chandrakant]

Thanks for the explanation. Just downloaded it 

[vindimy]

Ok, so lets assume one of Mike Hearn's microtransaction channel per person with a one year timeout was used. This way only 2 transactions would need to hit the blockchain per person per year. All the rest could be instant, micro and off the chain anonymous transactions without third party risk. The dream of every bitcoin user but:
8billion * 2kB = 16TB of data per year. With 50k blocks per year 16TB/50k = 300MB per block. Lets get there in 10 years, keep incentives to move transactions off the blockchain and nobody will complain. Even Satoshi Dice would work better with microtransaction channels anyway.

Let's not forget that a lot of applications have already started doing their transactions off the blockchain. The blockchain is only updated when coins are deposited and withdrawn.

[PrintMule]

Ok, so lets assume one of Mike Hearn's microtransaction channel per person with a one year timeout was used. This way only 2 transactions would need to hit the blockchain per person per year. All the rest could be instant, micro and off the chain anonymous transactions without third party risk. The dream of every bitcoin user but:
8billion * 2kB = 16TB of data per year. With 50k blocks per year 16TB/50k = 300MB per block. Lets get there in 10 years, keep incentives to move transactions off the blockchain and nobody will complain. Even Satoshi Dice would work better with microtransaction channels anyway.

Let's not forget that a lot of applications have already started doing their transactions off the blockchain. The blockchain is only updated when coins are deposited and withdrawn.

What do you mean? Transaction = withdraw and a deposit. Did I miss something?

[giszmo]

Ok, so lets assume one of Mike Hearn's microtransaction channel per person with a one year timeout was used. This way only 2 transactions would need to hit the blockchain per person per year. All the rest could be instant, micro and off the chain anonymous transactions without third party risk. The dream of every bitcoin user but:
8billion * 2kB = 16TB of data per year. With 50k blocks per year 16TB/50k = 300MB per block. Lets get there in 10 years, keep incentives to move transactions off the blockchain and nobody will complain. Even Satoshi Dice would work better with microtransaction channels anyway.

Let's not forget that a lot of applications have already started doing their transactions off the blockchain. The blockchain is only updated when coins are deposited and withdrawn.

What do you mean? Transaction = withdraw and a deposit. Did I miss something?

Beginners often get it wrong that the deposit address at MtGox is not their private stash of bitcoin at MtGox but merely their identifier to deposit but this address might send to arbitrary addresses without their balance being affected.
Equally if you deposit via a MtGox code you deposit bitcoin on your balance that does not touch the block chain.
Standard bitcoind does book keeping. If you tell bitcoind to book money from address a to address b, it does not send a transaction to the blockchain if a and b are both in the same wallet.
Redeem codes are a more complicated way to achieve this MtGox-internal booking from one person to the other. It's more complicated as the sender does not a priory know which MtGox user will get the value later on.

These are the current off-chain transactions. They have the problem of the user having to trust MtGox in my example but have the advantage that transactions that don't hit the chain are essentially free.

With micro transaction channels, you can take the third party risc out of the equation as "MtGox" would not need to actually have control over the users' money to make that happen. "A" could tie his update to grant MtGox more money to MtGox proofing that the money gets sent on to the right recipient "B".

[vindimy]

Ok, so lets assume one of Mike Hearn's microtransaction channel per person with a one year timeout was used. This way only 2 transactions would need to hit the blockchain per person per year. All the rest could be instant, micro and off the chain anonymous transactions without third party risk. The dream of every bitcoin user but:
8billion * 2kB = 16TB of data per year. With 50k blocks per year 16TB/50k = 300MB per block. Lets get there in 10 years, keep incentives to move transactions off the blockchain and nobody will complain. Even Satoshi Dice would work better with microtransaction channels anyway.

Let's not forget that a lot of applications have already started doing their transactions off the blockchain. The blockchain is only updated when coins are deposited and withdrawn.

What do you mean? Transaction = withdraw and a deposit. Did I miss something?

To summarize post above this one, when you deposit/withdraw money from the service (such as Mtgox), you do a blockchain transaction, but all money movements inside the service (in case of Mtgox it would be trades) are off blockchain, and therefore are very fast and free.

[tvbcof]

To summarize post above this one, when you deposit/withdraw money from the service (such as Mtgox), you do a blockchain transaction, but all money movements inside the service (in case of Mtgox it would be trades) are off blockchain, and therefore are very fast and free.

I've always felt comfortable with this solution since I can keep balances at various like organizations which are affordable to lose.  A bulk of my BTC holding are in very deep storage of course.  Even I cannot easily get a hold of them much less anyone else.

If I feel uncomfortable with one organization I can easily move some funds to one which I trust more or who's policies I like better.  I've done this from time to time.

Going beyond this, larger organizations with a significant userbase can coordinate with each other such that they settle on the blockchain on a periodic basis (say, once per day.)  In this way, the transaction fee can be shared between many thousands of users.  The fee could become high enough to make it well worth solving a block at even a modest TPS rate.

Such a solution keeps the block chain light enough that it can be realistically maintained by private individuals scattered throughout the world.  That is the big win to an 'off-chain with Bitcoin backing' evolution of the ecosystem.  I see it as the best compromise to address the scaling issues as we move forward.

[alkaz]

gah, i cant download it
it uncompatible with windows 8 

[Mike Hearn]

What is incompatible with Windows 8?

[viboracecata]

Not very clear of the technique details, but this function is attractive, thanks man.

[Leehoya]

Just use a online wallet, blockchain, coinbase etc.

[jgarzik]

Jeff runs a blockchain BitTorrent for that exact purpose.

Link:

[ANN] Bitcoin blockchain data torrent
https://bitcointalk.org/index.php?topic=145386.0