Bitcoin Forum
April 23, 2014, 11:10:14 PM *
News: Due to the OpenSSL heartbleed bug, changing your forum password is recommended.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2  All
  Print  
Author Topic: Will the block chain size ever become too large?  (Read 12817 times)
dduane
Jr. Member
*
Offline Offline

Activity: 44



View Profile WWW

Ignore
February 02, 2011, 02:10:47 PM
 #1

As I understand Bitcoin P2P network, there is a block chain that holds a record of every transaction that ever happened.

Does every Bitcoin instance on a peer node download this entire block chain?  If so, at some point won't it become untenable to
download the entire chain as it gets larger and larger ?

I hope that I'm missing something here, but I'm concerned that as the volume of transactions increases we'll have to make some changes to how many of the blocks we keep track of in the nodes.
1398294614
Hero Member
*
Offline Offline

Posts: 1398294614

View Profile Personal Message (Offline)

Ignore
1398294614
Reply with quote  #2

1398294614
Report to moderator
Unbeatable Service & Product Support
Grab Your Miners at GAWMiners.com
Order Before April 25th to receive
Double your Hashing Power for 1 week!

Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1398294614
Hero Member
*
Offline Offline

Posts: 1398294614

View Profile Personal Message (Offline)

Ignore
1398294614
Reply with quote  #2

1398294614
Report to moderator
1398294614
Hero Member
*
Offline Offline

Posts: 1398294614

View Profile Personal Message (Offline)

Ignore
1398294614
Reply with quote  #2

1398294614
Report to moderator
1398294614
Hero Member
*
Offline Offline

Posts: 1398294614

View Profile Personal Message (Offline)

Ignore
1398294614
Reply with quote  #2

1398294614
Report to moderator
ribuck
Donator
Hero Member
*
Offline Offline

Activity: 1218


View Profile

Ignore
February 02, 2011, 02:28:47 PM
 #2

Does every Bitcoin instance on a peer node download this entire block chain? 
At this time, yes, every bitcoin app downloads its own copy of the full block chain.

This is not absolutely necessary. Several levels of optimization are available; it's just a "simple matter of programming".

In the future, big nodes (e.g. those operated by exchangers) will download the whole block chain, but the "average joe" will run a "bitcoin lite" that just downloads the data needed to support his own use of bitcoin.
Zerbie
Member
**
Offline Offline

Activity: 73


View Profile

Ignore
February 05, 2011, 04:24:45 PM
 #3

I've been thinking about this problem as well.  If bitcoin is to go mainstream, there needs to be a way to deal with a huge number of transactions.  Here is an example of the math...

In a population of 300 million, estimate everyone does one transaction per day on average.  The current block generation is about one every 10 minutes, or 6 an hour, or 144 per day.  If you divide 300 million by 144, you get 2.08 million transactions per block.  If the minimum size per each transaction is around 100 bytes (I'm just guessing here), then each block is going to be around 200 million bytes.

So, I see the following issues:

1. Huge bandwidth problems.  (144*200 million bytes is 28 Gigs per day.)
2. Storage issues. (28 Gigs per day is 10 Tera bytes per year)
3. Computational issues. (The current block size is around 1-2k.  Computing a hash for 200 MB is going to be much different.)

Here are a few other things I've thought about:

* What happens in the future when you issue a request to send someone funds and do not attach a fee for processing?  If no-one takes your request to process the funds, will the funds stay in "limbo"?
* I was thinking a good way to deal with the huge number of transactions is to mix bitcoin transactions with something like the ripple project (http://ripple-project.org/), or perhaps semi-centralized bitcoin clearing entities.
* Given the huge number of potential transactions per day, perhaps the block size needs to be limited to 2-4k.  This would force the above idea and keep the block chain manageable.
Gavin Andresen
Hero Member
*****
qt
Offline Offline

Activity: 1330


Chief Scientist


View Profile WWW

Ignore
February 05, 2011, 04:31:40 PM
 #4

We've talked about these kinds of scaling issues in the past, and we've got a ways to go until they become an issue.

I still strongly believe that we shouldn't worry about them until they actually start to become a problem.

Long before we get to 300 million transactions per day there will be big companies spending lots of engineering dollars on their own highly optimized versions of bitcoin.

I bet there will be alternative, secure-and-trusted, very-high-speed network connections between major bitcoin transaction processors.  Maybe it will just be bitcoin transactions flying across the existing Visa/MasterCard/etc networks (I have no idea what their transaction clearing/processing networks look like or how they work).

Instead of worrying about "what will we do when we are wildly successful" we should be worrying about "what should we do RIGHT NOW so we become wildly successful."

Will I see you in Amsterdam?
  http://bitcoin2014.com/
Zerbie
Member
**
Offline Offline

Activity: 73


View Profile

Ignore
February 05, 2011, 05:06:50 PM
 #5

The reason I'm thinking about it now, is because I would like to see bitcoin stay as decentralized as possible.  If it is left till later, then there are two undesirable outcomes:

1. Big corporations step in and take the role of bitcoin managers.
2. With the centralization of the currency, it is then easy for government agencies to step in and regulate.

This is one of the reasons I would like to see the block size limited to 2-4k.  With something so small, the chain can exist on a 500GB HD for 2000 years and reasonably downloadable for 46 years (assuming 10GB download in a day).  This keeps it within reach of the small operator who wants to compete with the big banks.  If we want to see the currency traded freely for many years into the future, we need to look at (and solve) the problems now.
Hal
VIP
Sr. Member
*
expert
Offline Offline

Activity: 314



View Profile

Ignore
February 05, 2011, 07:32:12 PM
 #6

See this analysis by [mike] looking at similar transaction loads and what that would mean for mining servers:

http://bitcointalk.org/index.php?topic=2454.msg34469#msg34469

Mike astutely identifies the bottleneck as the CPU power to verify transaction signatures. I've been doing some profiling and over 90% of the CPU time of the (non-generating) client is spent in ECDSA_verify.

There are plans for lightweight clients that would not download blocks, just headers. Then they would be sent details of transactions involving them. Only the big iron would keep the block chain.

Hal Finney
MoonShadow
Hero Member
*****
Online Online

Activity: 1372



View Profile

Ignore
February 06, 2011, 03:36:10 AM
 #7

The reason I'm thinking about it now, is because I would like to see bitcoin stay as decentralized as possible.  If it is left till later, then there are two undesirable outcomes:

1. Big corporations step in and take the role of bitcoin managers.
2. With the centralization of the currency, it is then easy for government agencies to step in and regulate.

This is one of the reasons I would like to see the block size limited to 2-4k.  With something so small, the chain can exist on a 500GB HD for 2000 years and reasonably downloadable for 46 years (assuming 10GB download in a day).  This keeps it within reach of the small operator who wants to compete with the big banks.  If we want to see the currency traded freely for many years into the future, we need to look at (and solve) the problems now.

There will be some centralization, but what you really fear is a de facto monopoly.  This is, for all practical purposes, impossible.  The size of the blocks is irrelevant, since the only part of the block that is permanent is the 80 byte header.  The rest can be purged eventually, after the transactions have been referenced.

"The powers of financial capitalism had another far-reaching aim, nothing less than to create a world system of financial control in private hands able to dominate the political system of each country and the economy of the world as a whole. This system was to be controlled in a feudalist fashion by the central banks of the world acting in concert, by secret agreements arrived at in frequent meetings and conferences. The apex of the systems was to be the Bank for International Settlements in Basel, Switzerland, a private bank owned and controlled by the world's central banks which were themselves private corporations. Each central bank...sought to dominate its government by its ability to control Treasury loans, to manipulate foreign exchanges, to influence the level of economic activity in the country, and to influence cooperative politicians by subsequent economic rewards in the business world."

- Carroll Quigley, CFR member, mentor to Bill Clinton, from 'Tragedy And Hope'
ShadowOfHarbringer
Hero Member
*****
Offline Offline

Activity: 1246


Bringing Har® to you since 1952


View Profile

Ignore
February 06, 2011, 05:44:29 AM
 #8

Mike astutely identifies the bottleneck as the CPU power to verify transaction signatures. I've been doing some profiling and over 90% of the CPU time of the (non-generating) client is spent in ECDSA_verify.

Of course this is easily solvable using GPUs. So not a bottleneck anymore i guess.

carp
Member
**
Offline Offline

Activity: 81


View Profile

Ignore
February 06, 2011, 06:29:04 AM
 #9

Also, what is "too big".

Just a couple of years ago I built a .75 TB array to backup other machines and run a few VMs from.

Now the 250 GB disks in that were shoved into a couple of desktops, and that ENTIRE array was copied (with plenty of extra space) onto 1 disk.... and then, onto a new 5 and some change (available space) array. "Big" today, can be peanuts in a few years.

Though 300 million tx a day will still be a lot, but, by the time it happens, it will be less than it is now just by virtue of everything else scaling up, and well... though I dunno.... it still does seem like a lot per block. Seems like pruning will be important, with that much churn, I would bet there will be accounts that can have their entire existence purged...once it goes to zero, does it matter that it EVER had anything in it? Which mean, eventually blocks which have had all their tx purged, end up being pruned all the way down to its headers and top level hash derived from all the transactions that were purged.

it will be interesting to see how effective that is at keeping the size of the chain down.
Zerbie
Member
**
Offline Offline

Activity: 73


View Profile

Ignore
February 06, 2011, 08:51:03 AM
 #10

There will be some centralization, but what you really fear is a de facto monopoly.  This is, for all practical purposes, impossible.  The size of the blocks is irrelevant, since the only part of the block that is permanent is the 80 byte header.  The rest can be purged eventually, after the transactions have been referenced.

Perhaps I've not followed the block idea correctly. Where are the transaction stored?  If I send someone a bitcoin, it's history has to be recorded someplace to verify that it was not created out of thin air. (Ha... I just realized that they are created out of thin air...)

Zerbie
Mike Hearn
Hero Member
*****
expert
Offline Offline

Activity: 1232


View Profile

Ignore
February 06, 2011, 10:36:48 AM
 #11

Even if we assume BitCoin grows to VISA scale it'd be possible to run a full network node using only a small number of computers. I cannot forsee a time when BitCoin is out of reach for (well paid) hobbyists. See my previous posts on this topic for why.

12LMm82ZgAzf7yNDpPydEYxEr4Ap7XtSSK
Mike Hearn
Hero Member
*****
expert
Offline Offline

Activity: 1232


View Profile

Ignore
February 06, 2011, 10:42:51 AM
 #12

Zerbie, the transactions are stored in the block chain, but once all the outputs of a transaction are used up that tx can be safely deleted (or stored for historical interest). So disk space can be reclaimed if you are tight. Given that 1T drives are pretty cheap today though, it's unlikely anyone will ever actually need to do this. Storage capacity doesn't seem to be a bottleneck anytime soon.

Short story - don't worry about corporate takeover at large scale because of technical limitations. The places where corporates will dominate in future will be mining conglomerates, currency exchanges, fractional reserve banking and most of the places where they exist today. The companies that will have to find new roles in a hypothetical post BitCoin world will be MasterCard and VISA.

12LMm82ZgAzf7yNDpPydEYxEr4Ap7XtSSK
MoonShadow
Hero Member
*****
Online Online

Activity: 1372



View Profile

Ignore
February 06, 2011, 03:12:43 PM
 #13

There will be some centralization, but what you really fear is a de facto monopoly.  This is, for all practical purposes, impossible.  The size of the blocks is irrelevant, since the only part of the block that is permanent is the 80 byte header.  The rest can be purged eventually, after the transactions have been referenced.

Perhaps I've not followed the block idea correctly. Where are the transaction stored?  If I send someone a bitcoin, it's history has to be recorded someplace to verify that it was not created out of thin air. (Ha... I just realized that they are created out of thin air...)

Zerbie

Transactions are stored within a block in a merkle hash tree, so the block can be pruned of long spent transactions and still be verified as being authentic by it's merckle root hash within the block header.  It only takes a dozen or so kept history to be certain that a transaction is real.

"The powers of financial capitalism had another far-reaching aim, nothing less than to create a world system of financial control in private hands able to dominate the political system of each country and the economy of the world as a whole. This system was to be controlled in a feudalist fashion by the central banks of the world acting in concert, by secret agreements arrived at in frequent meetings and conferences. The apex of the systems was to be the Bank for International Settlements in Basel, Switzerland, a private bank owned and controlled by the world's central banks which were themselves private corporations. Each central bank...sought to dominate its government by its ability to control Treasury loans, to manipulate foreign exchanges, to influence the level of economic activity in the country, and to influence cooperative politicians by subsequent economic rewards in the business world."

- Carroll Quigley, CFR member, mentor to Bill Clinton, from 'Tragedy And Hope'
ribuck
Donator
Hero Member
*
Offline Offline

Activity: 1218


View Profile

Ignore
February 06, 2011, 09:40:31 PM
 #14

I cannot forsee a time when BitCoin is out of reach for (well paid) hobbyists.
If the time comes when it has becme impractical for a (non-well-paid) individual to process the block chain, this can only be because Bitcoin has become wildly successful, and that will have opened up millions of new niches for the Bitcoin hobbyist. Really cool things that haven't even been though of yet.
Mike Hearn
Hero Member
*****
expert
Offline Offline

Activity: 1232


View Profile

Ignore
February 07, 2011, 12:10:44 PM
 #15

Quote
Of course this is easily solvable using GPUs. So not a bottleneck anymore i guess.

Hashing != ECDSA verification. I don't know of any algorithms that will make ECDSA much faster using GPUs. The best bet for speeding this up (assuming we care) is to exploit the special properties of the secp256k1 curve:

http://portal.acm.org/citation.cfm?id=1514739

http://www.hyperelliptic.org/tanja/KoblitzC.html

12LMm82ZgAzf7yNDpPydEYxEr4Ap7XtSSK
abstraction
Member
**
Offline Offline

Activity: 107


View Profile

Ignore
March 23, 2011, 07:30:52 AM
 #16

When transaction blocks start getting really big, this solution may ease the burden on the network.
http://programming.witcoin.com/p/515/If-Sun-can-do-it-wit-Java-why-cant-we-do-it-wit-Bitcoin

Theoretically, new transaction blocks could be encoded against the existing block chain to yield small keys that contain unpacking instructions. Clients only need to unpack it and check the block against a hash of itself.

I'll be able to present the idea more clearly as I learn the right words to use. My best analogy is that we would be creating a new binary language that lets entities communicate large and complex ideas in an efficient manner, like the word "love".
Mike Hearn
Hero Member
*****
expert
Offline Offline

Activity: 1232


View Profile

Ignore
March 23, 2011, 08:18:57 AM
 #17

The block chain does not compress well because it is mostly keys and hashes (ie random data).

12LMm82ZgAzf7yNDpPydEYxEr4Ap7XtSSK
abstraction
Member
**
Offline Offline

Activity: 107


View Profile

Ignore
March 23, 2011, 08:59:45 AM
 #18

The block chain does not compress well because it is mostly keys and hashes (ie random data).

I'm not saying compress the block chain itself. Let me see if relating it to Boggle helps. At least somebody can see where my analogy breaks down. If you are an auditory learner, say what I'm writing out loud to yourself so you can "hear" me speaking it to you.

1) boggle board = block chain (seemingly random data)
2) word = data of interest (a pattern found within seemingly random data)
3) tracing the word on the board = an algorithm to encode or decode data (manipulation of random data)
4) a sentence using the words found = a complete data file (information of interest)
5) a word defining the sentence = a compressed file (an abstract idea)

Items 2-5 do not have to hold the entire information in one big data set. They can be utilized recursively to take a very complex idea and reduce it down to a small word.

Try describing something. What is "love"? What words did you use to describe it? What do those words mean? To get down to the real meaning of "love", you have to recursively define each word using other words. At the end of the recursion, you have a collection of random data that only has meaning because we both have access to the same data and a common knowledge to describe it. We understand this intuitively as a language to transfer an idea from one person to another. So what is the natural Bitcoin block chain language? Where are the linguists?
theymos
Administrator
Hero Member
*
expert
Offline Offline

Activity: 1540


View Profile
March 23, 2011, 05:40:09 PM
 #19

I don't see how that's not compressing random data, which is impossible.

gigabytecoin
Sr. Member
****
Offline Offline

Activity: 280


View Profile

Ignore
March 24, 2011, 04:08:43 AM
 #20

I've been thinking about this problem as well.  If bitcoin is to go mainstream, there needs to be a way to deal with a huge number of transactions.  Here is an example of the math...

In a population of 300 million, estimate everyone does one transaction per day on average.  The current block generation is about one every 10 minutes, or 6 an hour, or 144 per day.  If you divide 300 million by 144, you get 2.08 million transactions per block.  If the minimum size per each transaction is around 100 bytes (I'm just guessing here), then each block is going to be around 200 million bytes.

So, I see the following issues:

1. Huge bandwidth problems.  (144*200 million bytes is 28 Gigs per day.)
2. Storage issues. (28 Gigs per day is 10 Tera bytes per year)
3. Computational issues. (The current block size is around 1-2k.  Computing a hash for 200 MB is going to be much different.)

Here are a few other things I've thought about:

* What happens in the future when you issue a request to send someone funds and do not attach a fee for processing?  If no-one takes your request to process the funds, will the funds stay in "limbo"?
* I was thinking a good way to deal with the huge number of transactions is to mix bitcoin transactions with something like the ripple project (http://ripple-project.org/), or perhaps semi-centralized bitcoin clearing entities.
* Given the huge number of potential transactions per day, perhaps the block size needs to be limited to 2-4k.  This would force the above idea and keep the block chain manageable.

With my home built computer and a 100mbps internet connection, I can (and do) easily download 28 gigs in a matter of minutes.

I also own a few terabytes of hard disk space. It's cheap, and prices are falling faster than the US dollar.

By the time we ever reach 200 million transactions every 10 minutes, the smallest hard drives will probably be a few terabytes in size - and we'll all be rocking 1Gbps dedicated connections!

I don't think the block chain size will ever really be an issure, even for the average joe.
Pages: [1] 2  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!