Bitcoin Forum

I don't understand why the blockchain has grown exponentially in the latest months, with that huge size of the database on hard drive, it's getting difficult to convince new users to start to use it: They have to wait a whole day to download the block chain.

Is there any light client which download only latest part of blockchain and verify the rest part using only a checksum?

You might find this useful: https://bitcointalk.org/index.php?topic=94102.0

(the other choice is to use an alternative client)

I don't understand the technical details, but look at the blockchain size chart:
https://blockchain.info/charts/blocks-size (https://blockchain.info/charts/blocks-size)

if all the transactions are recorded in blockchain, it will surely grow out of the capacity of any hard disk in one year, after another year, it will grow out of the capacity of any storage server farm I know. This has to be solved as soon as possible

Any client should only interested in those transactions related to his own account, for other transactions he can use block explorer to search, maybe someone from a bank can explain how bank handle huge amount of transaction records?

Indeed the growing size is of concern (and there have been numerous threads about this not that long after Satoshi Dice was launched).

From these other threads I think that the momentum to implement "pruning" of the blockchain (and other scalability solutions) is increasing at an even more rapid rate. :)

(am certain that this will be sorted out before it becomes such a problem - but as stated before there are other clients that don't require the blockchain)

The blockchain size is directly proportional to the number of transactions in Bitcoin's history. The blockchain is larger now because there are more transactions. This is in large part due to SatoshiDice, but also due to general growth.

There are indeed lightweight clients (they don't work in the way you suggested though) and this is an active development subject. There are also eWallets such as blockchain.info's MyWallet which do not require trust.

I believe from client side there will be a solution, but how about blockchain itself?

After some study I can see that with today's 400 transaction/10minute rate, the block size is already 250K, if BTC gained popularity quickly, then there will be thousands of transactions per minute

It's basically a solved problem. There's pruning and there are ways to do transfers outside of the blockchain. Also, not everyone needs the blockchain so it can afford to be a bit large.

But has pruning been implemented?

This is a problem that is dismissed far too quickly by the same old arguments. There is no way that people are going to use bitcoin in the intended way if they need to download 4gb (and then another couple 100 every week) just in order to make payments.

It is counterproductive to simply dismiss this problem as solved, because it isn't actually solved yet

No. But it's being worked on.

It was never intended that every Bitcoin user will download the entire blockchain.

I see it the other way around - the same old arguments about the size of the blockchain are dismissed because they bring nothing new to the discussion and ignore existing work on the subject.

Yes, the solutions still need to be implemented. But complaining about it isn't helping.

I'd say the problem is not solved until there's a reference implementation of what's proposed in this thread (http://"https://bitcointalk.org/index.php?topic=88208.0"). Even with pruning the blockchain is going to be too large before long, if Bitcoin is going to be ever more widely adopted.

Also, after the block size limit is lifted (this must be done at some point) network bandwidth is going to be the bottleneck that limits who can run a full-fledged client.

Thanks for clarifying Meni.

I understand that complaining leads nowhere--my point was just that there is still work to be done here!

No and it would not be enough by itself soon anyway due to bandwidth and CPU usage also rising.

You can search "swarm client" on this forum to find a possible solution to BTCs algorithmic problem.

Until that is developed I suggest using electrum or some online wallet for new users.

Now I understand why banks have to hire so many people to take care of the account related activities ;D

I don't think Bitcoin will ever again be so that it's feasible for everyone in the world to run a full node. I don't know if it ever was, light clients are simply much faster and more convenient and even more so recently. The scaling work is basically needed so at least some people can run the full client. Not everyone needs to run it.

There is no such thing as an ewallet that does not require trust. Even if they propose client-side encryption of the keys, the user is not immune to code poisoning by the server: https://bitcointalk.org/index.php?topic=86278.20 (https://bitcointalk.org/index.php?topic=86278.20)

Supernodes and their ewallets are part of the solution for the growing size of the blockchain. I recommend paytunia to new users.

Hello, my opinion on the subject is :(

The blockchain growth may not be exponential but it grows quickly. (http://blockchain.info/charts/blocks-size?timespan=all&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address=) From May'12 it grows about +1 GB every 3 months! Plus here are my thoughts that it may possible for an attacker to spend $ 10,000 and do the same in 14 days. (https://bitcointalk.org/index.php?topic=94619.0#msg1046793)

On the other hand, I see this can be helped.
Raising fees. I think that fees are extremely low and should be considered at more realistic level, say 0.05 USD/tx. Also miners would welcome this 1000 times :D.
Light wallets, incentivised servers. I see there is a number of alternative clients but the reference client is simply a full node.
Pruning. Spent transactions will be possible to have pruned one day. This would be a relief from Satoshi Dice, however, I can still imagine millions of spam addresses with unspent micro-amounts.

I think if not addressed then it will be embarrassing for new users to download some 4 GB of blochchain next year.

Generally predictions that begin with "I don't understand this BUT" are worthless.

IF you don't understand the technical details how can you make predictions as to the scope of the problem or potential solutions.   

The reality is that only unspent tx need to remain in the blockchain and over time that will represent a smaller and smaller % of total transactions.  Storage requirements are a non-issue long term.  Casual users are best served by lite clients, a development that Satoshi covers in the white paper written over three years ago.

The protocol can't enforce fees.  The reference rules to prevent DOS attacks impose a required fee on LOW PRIORITY TX only.   Miners are free to choose what transactions they include in a block.  If you run a mining node you are free to exclude ANY transaction you wan't (for any reason).   Raising the mandatory fee on LOW PRIORITY TXs would be foolish and not have the desired effect you think it would as most transaction are HIGH PRIORITY and thus aren't affected by tx fee rules.  


Anti-spam rules make this prohibitively expensive.  The attacker would end up paying 10x to 100x the cost incurred by full nodes (and light nodes would incur no cost).  Trying to outspend your enemy when there are more of them and it costs you $100 for every $1 it costs you is kinda dumb.  Sort of like designing weapons which shoot money at the enemy and trying to kill them by burying them in giant piles of cash.  While technically possible there are far more effective methods of attack.


Why?  There is no need for users today to run a full node.  If they wish to run a full node then the 4GB "cost" is something they accept.  A long term project would be one that OpenCL accelerates the blockchain verification.   In theory bootstrapping a new node could be done significantly faster.

I hope I am not wrong here but something about fees may be encoded in the software, like not to transmit low priority transactions with fees under 0.0001 BTC. Anyway, I understand that the fees level to include low priority transactions in blocks is some general agreement among users. Indeed, I'm suggesting to raise this level, because I understand, that both Satoshi Dice and any possible spam fall under low priority tx, and it would help to keep them at more acceptable level.

Yes, at first sight the $10,000 buys a 1 GB of blockchain. But this will be transmitted, stored, re-transmitted, downloaded, etc, eventually forever. I think more of some moral harm to the project or loss of momentum, e.g. if such spammers grow the blockchain to 10 GB this year then many full nodes may just decide to stop. But that's just a matter of opinion. I still think there can be guys who may find the network endangering them, and go out and outspend it.

Indeed, non-technical type of users are not supposed to be running the reference client (something in the sense "Bitcoin-Qt - cool; 2 GB of data - wtf"). Somehow I find this unfortunate for the project, but again it's just a matter of opinion.

I doubt 1gb will be that expensive in the future to store, storage capacity is still growing faster than blockchain size.

Hard drives double about every two years.  If the block chain is doubling every year then it will become a problem.

Transaction fees are currently very low to help adoption, but I think it's inevitable that they will have to be much higher in the future.

Transaction fees are already ~2.5$/trans to the miners. That's already too high in my view.

And yes I am including the block reward as that IS essentially a fee all BTC holders pay to miners.

You're worried that spammers will spend $100,000 to grow the blockchain by 10 GB? That costs a miner $1 worth of disk space. That's a very ineffective attack! That's not going to stop any miners.

As you said yourself it's not a transaction fee, it's a holding fee. It's not the marginal price of making a transaction, you don't have to pay $2500 if you want to broadcast 1000 transactions.

Anyone who holds any bitcoins has signed up for the inflation schedule.

It's NOT a prediction, just some thing is happening now. During the past year, it growed from couple of megabytes into couple of gigabytes, I have not factored that growth rate into my prediction yet:p

It's not a prediction, its not something happening, it's nonsense hysteria.  The blockchain size has a fairly modest linear maximum, which can not change without a hard fork of the protocol.  Can this stupid thread die now please?

Perhaps not a hysteria, but pointing out the unusual fact that users need to download 2 GB of data to use the software nowadays. Also there is a space for a bigger growth rate within the existing protocol.
From what I've heard so far, this big data is here to stay, and the software will have to deal with it some day in future - full nodes acting as servers to lightweight wallets, pruning, etc.

USER DON'T NEED TO DOWNLOAD 2GB.

If they want to be a full node they will however no matter what is done the initial startup time for a new FULL node will be high.

If Bitcoin becomes 20x as popular the load on full nodes will rise.   Yes higher fees (no not stupidly increasing the low priority mandatory fees I am talking about competition among users raising the effective fees), blockchain pruning, openCL acceleration, more efficient indexing, multi-peer parallel downloads, will make the blockchain/nodes more efficient but just the overall increase in the size of Bitcoin "universe" means the load on full nodes will be HIGHER not LOWER in the future. 

So you have two scenarios:
a) Bitcoin growth stagnates at which point it doesn't really matter how large or inefficient the blockchain is.  The project has fizzled out anyways
b) Bitcoin growth continues and no matter what is done the load on full nodes will RISE.  Full nodes will be more efficient so the network being 20x as large won't require 20x the resources but the load will be higher than today.

NOT EVERYONE NEEDS TO RUN A FULL NODE.  A couple thousand full nodes is more than sufficient to ensure the network is secure, stable, and decentralized.  I don't see major businesses, exchanges, miners, mining pools, service providers, ewallet providers, and bitcoin enthusiasts seeing 2GB or 20GB being an impossible hurdle.  I mean people torrent hundreds of GBs a month.  Will 99.999999999% of users opt for the cost and complexity of a full node?  No and absolutely nothing you can propose will change that.  The good news is they don't have to; if someday there are 20 million bitcoin users and only in in 10,000 runs a full node well that would be awesome.

That block size chart started from August 2011. BTC is created in 2009, and up to 2011 the block size did not grow over a couple of megabytes, so it was not a linear growth at all

True, but most newbies will go to bitcoin.org and just get whatever's there, which is currently a full-node client. Their first impressions will be accordingly. Either the reference client needs an SPV mode, or bitcoin.org needs to list more alternative clients and explain the difference.

Maybe so, but they keep a few GB torrents running on a configured area of their hard drive and store the rest on some external hard drive which can be unplugged. The BTC node, on the contrary, keeps it all on the system partition and can't just be split and stored on some external switchable hard drive. Granted 2GB or, to a lesser extent, 20GB aren't too much of a problem, but 200GB would definitely be...

Pruning spent coins sounds great though, can't wait to see it implemented.
+1. As long as the reference client can only be a full node, users are pretty much encouraged to actually run a full node...

http://bitcoin.org/clients.html

Yup there are lite clients available.

The reference client should never be lite.  I mean it is an oxymoron.  It is the reference implementation of how a node interacts with its peers using the Bitcoin protocol.  It by its very definition must be full and complete.  Users are not required to run a full node.  Satoshi never intended for most users to run full nodes.  It just happened that for the first couple years the amount of resources required to run a full node were trivial and lite nodes simply didn't exist. 

I do think decoupling the "satoshi client" from bitcoin.org eventually is useful.  The Bitcoin.org website should be client agnostic.  The client list page is a step in the right direction. 

when the block reward halves, the amount earned from transaction fees will become more relevant and spamming will become more expensive. with more users the blocks will become fuller and fuller and transaction fees will increase to the point were a hard fork is needed.
before that, there is an upper limit to blockchain growth that isnt very threatening. and considering how low transactions fees are right now, i predict we got some time before scalability becomes a problem even without any technological improvements.

unless bitcoin gets a big jump in users. which, in my opinion, is likely to also increase the speed of development since more people mean more potential developers, investors, sponsors etc.

Good thing I did not say linear growth then, I said there is a linear maximum. I have a laptop with enough storage for something like 20 years of growth at the maximum rate.  There are certainly things to be concerned about, like the loss of decentralization that comes from more people lazying out of the initial startup time... but there is no danger from "exponential growth" in terms of size.

Relax, I'm not saying it is stalling my hard drive, but as some one pointed out earlier, increased transaction number will also cause bandwidth and processor power usage surge sooner or later

I do not think this is a technical problem, since this is the first time there is a change has to be made into the system, and there is no existing directive for way of working

The decision making in a GIT world, as I observed for some years, is hierarchical where the core programmer stay in the top. A core programmer maybe very good at programming but have less experience in financial world, so he might not be seeing what is coming. I think someone from the bank/electronic exchange sector back office personal should provide some insight about how they handle the huge amount of transactions today

Hello,
I don't know what solutions are coming for this real issue. But i find there IS a problem with the exponential growth of the blockchain.

In my opinion, the "There is no such problem" position is ignoring the fact that if nothing is changed, then the blockchain will probably be centralizated in a future. So, why didn't satoshi just said "Ok, for this bitcoin thing we will need a server that will store the blockchain"??. Indeed he didn't wanted to have the database centralizated in one place, he wanted the database to be distributed in the whole world probably to prevent bitcoin to be shut down by goverments.

Anyway, my point is that is better to have the blockchain distributed than centralized. So I was wondering... what if the blockchain was fragmented and the client just sends all the fragments to (like) 10.000 computers in the world so the blockchain could be reintegrated on demand.

To give a metaphor, if we were talking about michael jackson's discography  people are saying that you can download all of it, or stream it from a supernode. Why don't we just keep "some" songs, and be sure that all the 'songs' are available?

This option would be ADDED to the other two that are right now available, so you could choose to:
a) be a sopernode
b) connect to a supernode (and not to have the blockchain)
c) have some parts of the blockchain downloaded and 'lend' some of your hard drive space to the community.

What do you think?

I like this idea

I use a firefox plugin that warns me if javascript has changed.

As someone who regularly download hundreds of gigs of movies I can say that 2gigs is not very much.

It seemed to me that parsing the block chain was mostly IO bound, I think it is not very efficient in how it handles things in RAM. Neither my bandwidth or CPU was pegged, but my IO was. I think the blockchain could be parsed much faster if the parser was redesigned.

Parsing the block chain is trivial, it is updating the transaction database while keeping it consistent that makes it slow.

That said, yes there are serious improvements possible, and we're working on it.

Isn't the block chain size growth hard coded to be linear?

10 mins per block, 1MB block size... scarily this is 52GB per year.

However, it is linear not exponential.

A 2TB drive, that's 40 years of block chain, so it's actually not that bad.

The block size ceiling is supposed to be a temporary measure and to be lifted before it becomes a limiting factor on bitcoin's usability.

It exists precisely to prevent the early bloat before Bitcoin can scale to or justify that this thread is concerned with.