Bitcoin client operating with a finite amount of disk space

[matt.collier]

Just finished reading some of the topics relating to concerns about the present and eventual size of the block chain.

Presently, the .bitcoin directory which contains the block chain is in excess of 600MB.  In my opinion, this is a non-trivial amount of disk space.  I among others have developed environments designed to assist with client security that involve running the client from a USB memory stick or other devices with resource limitations.

I read in one post, that it not necessary for every client to download/save the entire block chain in order to send/receive bitcoin.  Is a bitcoin client with this feature somewhere on the horizon?

Thanks for considering this issue and all your excellent work!

Matt

[1713534732]

1713534732

[1713534732]

1713534732

[1713534732]

1713534732

[1713534732]

1713534732

[kjj]

The exact shapes of lightweight clients and protocols are not yet known, but lots of people are thinking about / working on them.

[ctoon6]

the blockchain right now is not 600mb, its more like 400, excluding the index files. and that can be compressed to at least 80% of the original size. and. 16 gigs should be good for a linux install and 2 more years worth of blockchain worst case.

[matt.collier]

the blockchain right now is not 600mb, its more like 400, excluding the index files. and that can be compressed to at least 80% of the original size. and. 16 gigs should be good for a linux install and 2 more years worth of blockchain worst case.

Can you say more about compressing the data?  How could this be accomplished in a transparent way?

I understand that there is a maximum number of transactions that can be included in a single block.  Do we know how much disk space a maxed-out block like this will consume?

You state that 16 gigs should should be good for 2 years worth of block chain data.  Is this a wild-ass guess or based on some reasonable assumptions?  If the estimate is based on some assumptions, would you please share the data you used in your calculations?

[Sukrim]

All non-miner clients need are block headers and access to a trusted node that has the transactions cached. The headers are small enough to not be a real issue.

Also you can remove all 0 balance accounts from the local chain again, which also results in very substantial space winnings.

Please write a patch at least for the latter one, it should already be possible to do right now in the current bitcoin client.

[payb.tc]

but the block chain can even be shrunk with actual ZIP compression (or similar). would there be a way of making it compressed like this while being able to be accessed at the same time?

then when people install bitcoin for the first time, it'd be good if the client could download the blockchain in compressed form

[ctoon6]

Can you say more about compressing the data?  How could this be accomplished in a transparent way?

I understand that there is a maximum number of transactions that can be included in a single block.  Do we know how much disk space a maxed-out block like this will consume?

You state that 16 gigs should should be good for 2 years worth of block chain data.  Is this a wild-ass guess or based on some reasonable assumptions?  If the estimate is based on some assumptions, would you please share the data you used in your calculations?

so far bitcoin has been out for around 30 months and has only just reached about 400mb, 500 if you count indexes. thats 2.5 years.

the past 100,000 blocks have an average size of 4244b
the past 50000 7900b
25000 13558b
10000 22871b
5000 24994b
1000 23705b
500 23170b

you can see that it doubles very roughly every 40000-60000 blocks. but this figure could very easily not work depending on bitcoin growth or death. so the average a year from now would be 40000b, so lets just assume from now on the size per block is 40000b,
40000*6*24*365
now double 40000
80000*6*24*365
add
get 6,307,200,000
i don't know if these figures are bits or bytes, but ill assume bytes.
5.87 gigabytes, assuming worst case scenarios. this means i used numbers that would exist at the end of the, at the beginning. so mathematically the numbers can not be higher than this.

again these numbers are probably wrong because of human behavior, my self and others, but it also seems to look like Moore's law a bit, exept the numbers are doubling sooner than every 18 months.

i got the data from block explorer btw

[NetTecture]

All non-miner clients need are block headers and access to a trusted node that has the transactions cached.

Ah, no. All a non miner client needs is a prototcol talking to atrusted server.

No need to store anything except addresses and private keys.

You can send signed transactions to the server and get balacnes and new transfers from the server.

Stick it THIN - so you dont need to sync anything. Laptop open, check, finished.

[kjj]

The block chain is not very compressible, since most of it is hashes.  I got 21% space savings with gzip and 22% savings with bzip2.  (436690683 vs. 345345519 vs. 341013976 bytes)

The real magic comes in when you realize that you can prune old transactions.  Since any transaction in the chain can be the input for at most one new transaction, you can delete any transaction that was spent more than X blocks ago, with no ill effects.  Someone wrote a tool for that, and if I recall, he reported that something like 70% of the chain can be pruned already.

All non-miner clients need are block headers and access to a trusted node that has the transactions cached.

Ah, no. All a non miner client needs is a prototcol talking to atrusted server.

No need to store anything except addresses and private keys.

You can send signed transactions to the server and get balacnes and new transfers from the server.

Stick it THIN - so you dont need to sync anything. Laptop open, check, finished.

The trusted server part is currently difficult, so I expect a medium-weight client to pop up and be useful sooner than a fully stripped lightweight client.

That is, unless one of the 3 or more people/groups working on hardware wallets makes some big progress before a serious smartphone developer gets the itch to code up a medium client.

[timmey]

so far bitcoin has been out for around 30 months and has only just reached about 400mb, 500 if you count indexes. thats 2.5 years.

yes, but bitcoin is also still waiting for it's major break through and doesn't really have many users and shops yet.

[payb.tc]

so far bitcoin has been out for around 30 months and has only just reached about 400mb, 500 if you count indexes. thats 2.5 years.

yes, but bitcoin is also still waiting for it's major break through and doesn't really have many users and shops yet.

if the first 2.5 years made 400mb, i bet the *next* 2.5 years would easily make an extra 4000mb (if not compressed or pruned).

[Bitsky]

the blockchain right now is not 600mb, its more like 400, excluding the index files. and that can be compressed to at least 80% of the original size. and. 16 gigs should be good for a linux install and 2 more years worth of blockchain worst case.

so far bitcoin has been out for around 30 months and has only just reached about 400mb, 500 if you count indexes. thats 2.5 years.

My current bitcoin directory has 705MB right now.
A backup from the 18th has 612MB.
Another from the 14th has 592MB.

So, from 14th->18th, disk space increased by 20MB; that's 4MB/day.
From 18th->27th, another 93MB were saved, which means 9.3MB/day.

Of course it's easy to say "quit whining, diskspace is cheap", but when Bitcoin wants to enter the mobile/smartphone market, diskspace and traffic does matter.
Also, if you are an optimist and hope that Bitcoin will catch on and grow quickly, then the number of transactions will increase which in turn needs even more diskspace.
So, the faster Bitcoin grows, the bigger the storage requirements. This means the initial download/verify time will also increase and most likely anger new users.
Last but not least, when reaching a certain amount of transactions per second, the everyday John Doe simply won't have the bandwith to deal with the blockchain growth.

[netrin]

I would like to see a ewallet service that uses a finite set of keys for each user wallet and lets the user download a copy.

And/or a service that handles the block chain and protocol allowing my client to deal only with my transactions and keys such as Webcoin and BitcoinJS are attempting:

[vector76]

I was thinking about how to remove transactions when all the outputs have been used, but it seems to me that at least the transaction hash must be kept, because without it, it's impossible to tell the difference between an orphan transaction and a double-spend.  Although since neither are fully valid, it might make sense to discard them both, and if the orphan later becomes a non-orphan, hope that it will be retransmitted.

I'm not sure how much space savings this would provide, but I think it would be substantial.

[phillipsjk]

the blockchain right now is not 600mb, its more like 400, excluding the index files. and that can be compressed to at least 80% of the original size. and. 16 gigs should be good for a linux install and 2 more years worth of blockchain worst case.

It is my understanding that the nodes save multiple copies of the block-chain in case of a split or one of the block-chains becomes the "longest" one. I have had a test-node running since June 9, 2011 (0.2.22 and 0.2.23) for a total of 55 days. It ran out of disk space today; consuming 5.8 GB. That works out to 105MB per day. Disk usage dropped to 4.9GB when the client exited. The client had 125 connections during peak times.

[ctoon6]

It is my understanding that the nodes save multiple copies of the block-chain in case of a split or one of the block-chains becomes the "longest" one. I have had a test-node running since June 9, 2011 (0.2.22 and 0.2.23) for a total of 55 days. It ran out of disk space today; consuming 5.8 GB. That works out to 105MB per day. Disk usage dropped to 4.9GB when the client exited. The client had 125 connections during peak times.

I have no idea about the test network, but my entire %appdata%\bitcoin dir has never went over 800mb, yet, but i would assume it will be by the end of next month at most.

[phillipsjk]

I was using the "real" network with the official client in -gen mode.

[kjj]

the blockchain right now is not 600mb, its more like 400, excluding the index files. and that can be compressed to at least 80% of the original size. and. 16 gigs should be good for a linux install and 2 more years worth of blockchain worst case.

It is my understanding that the nodes save multiple copies of the block-chain in case of a split or one of the block-chains becomes the "longest" one. I have had a test-node running since June 9, 2011 (0.2.22 and 0.2.23) for a total of 55 days. It ran out of disk space today; consuming 5.8 GB. That works out to 105MB per day. Disk usage dropped to 4.9GB when the client exited. The client had 125 connections during peak times.

No, it doesn't save multiple copies.  When there is a fork, it keeps both blocks, but it doesn't need to make a copy of the rest of the chain to do it.

Check to see if you have a debug.log.  If I don't clear mine often it gets huge.  Currently around 1.5 GB.

[phillipsjk]

So there is: 4.2GB. (Gedit tried to load the whole thing..)

Code:

ubuntu@ubuntu:/media/803819A438199A6C/bitcoins$ tail debug.log 
StopNode()
Running BitcoinMiner with 2 transactions in block
ThreadBitcoinMiner exiting, 0 threads remaining
DBFlush(true)
blkindex.dat refcount=0
blkindex.dat flush
wallet.dat refcount=0
wallet.dat flush
Bitcoin exiting

I suppose that is what I get for running the "beta" version: It is saving a lot of debugging information. Log rotation would probably help, but I doubt it is a priority if only used for debugging.

[kjj]

http://forum.bitcoin.org/?topic=292.0