Bitcoin Forum
December 07, 2016, 08:47:22 AM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 »  All
  Print  
Author Topic: Bitcoin client operating with a finite amount of disk space  (Read 4163 times)
matt.collier
Member
**
Offline Offline

Activity: 105



View Profile
July 26, 2011, 08:27:45 PM
 #1

Just finished reading some of the topics relating to concerns about the present and eventual size of the block chain.

Presently, the .bitcoin directory which contains the block chain is in excess of 600MB.  In my opinion, this is a non-trivial amount of disk space.  I among others have developed environments designed to assist with client security that involve running the client from a USB memory stick or other devices with resource limitations.

I read in one post, that it not necessary for every client to download/save the entire block chain in order to send/receive bitcoin.  Is a bitcoin client with this feature somewhere on the horizon?

Thanks for considering this issue and all your excellent work!

Matt
1481100442
Hero Member
*
Offline Offline

Posts: 1481100442

View Profile Personal Message (Offline)

Ignore
1481100442
Reply with quote  #2

1481100442
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481100442
Hero Member
*
Offline Offline

Posts: 1481100442

View Profile Personal Message (Offline)

Ignore
1481100442
Reply with quote  #2

1481100442
Report to moderator
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
July 26, 2011, 08:45:15 PM
 #2

The exact shapes of lightweight clients and protocols are not yet known, but lots of people are thinking about / working on them.

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
ctoon6
Sr. Member
****
Offline Offline

Activity: 350



View Profile
July 26, 2011, 09:22:47 PM
 #3

the blockchain right now is not 600mb, its more like 400, excluding the index files. and that can be compressed to at least 80% of the original size. and. 16 gigs should be good for a linux install and 2 more years worth of blockchain worst case.

matt.collier
Member
**
Offline Offline

Activity: 105



View Profile
July 26, 2011, 11:06:24 PM
 #4

the blockchain right now is not 600mb, its more like 400, excluding the index files. and that can be compressed to at least 80% of the original size. and. 16 gigs should be good for a linux install and 2 more years worth of blockchain worst case.

Can you say more about compressing the data?  How could this be accomplished in a transparent way?

I understand that there is a maximum number of transactions that can be included in a single block.  Do we know how much disk space a maxed-out block like this will consume?

You state that 16 gigs should should be good for 2 years worth of block chain data.  Is this a wild-ass guess or based on some reasonable assumptions?  If the estimate is based on some assumptions, would you please share the data you used in your calculations?
Sukrim
Legendary
*
Offline Offline

Activity: 1848


View Profile
July 26, 2011, 11:41:50 PM
 #5

All non-miner clients need are block headers and access to a trusted node that has the transactions cached. The headers are small enough to not be a real issue.

Also you can remove all 0 balance accounts from the local chain again, which also results in very substantial space winnings.

Please write a patch at least for the latter one, it should already be possible to do right now in the current bitcoin client.

https://bitfinex.com <-- leveraged trading of BTCUSD, LTCUSD and LTCBTC (long and short) - 10% discount on fees for the first 30 days with this refcode: x5K9YtL3Zb
Mail me at Bitmessage: BM-BbiHiVv5qh858ULsyRDtpRrG9WjXN3xf
payb.tc
Hero Member
*****
Offline Offline

Activity: 812



View Profile
July 27, 2011, 12:06:51 AM
 #6

but the block chain can even be shrunk with actual ZIP compression (or similar). would there be a way of making it compressed like this while being able to be accessed at the same time?

then when people install bitcoin for the first time, it'd be good if the client could download the blockchain in compressed form
ctoon6
Sr. Member
****
Offline Offline

Activity: 350



View Profile
July 27, 2011, 01:17:18 AM
 #7

Can you say more about compressing the data?  How could this be accomplished in a transparent way?

I understand that there is a maximum number of transactions that can be included in a single block.  Do we know how much disk space a maxed-out block like this will consume?

You state that 16 gigs should should be good for 2 years worth of block chain data.  Is this a wild-ass guess or based on some reasonable assumptions?  If the estimate is based on some assumptions, would you please share the data you used in your calculations?

so far bitcoin has been out for around 30 months and has only just reached about 400mb, 500 if you count indexes. thats 2.5 years.

the past 100,000 blocks have an average size of 4244b
the past 50000 7900b
25000 13558b
10000 22871b
5000 24994b
1000 23705b
500 23170b

you can see that it doubles very roughly every 40000-60000 blocks. but this figure could very easily not work depending on bitcoin growth or death. so the average a year from now would be 40000b, so lets just assume from now on the size per block is 40000b,
40000*6*24*365
now double 40000
80000*6*24*365
add
get 6,307,200,000
i don't know if these figures are bits or bytes, but ill assume bytes.
5.87 gigabytes, assuming worst case scenarios. this means i used numbers that would exist at the end of the, at the beginning. so mathematically the numbers can not be higher than this.

again these numbers are probably wrong because of human behavior, my self and others, but it also seems to look like Moore's law a bit, exept the numbers are doubling sooner than every 18 months.

i got the data from block explorer btw

NetTecture
Full Member
***
Offline Offline

Activity: 140


View Profile
July 27, 2011, 05:07:14 AM
 #8

All non-miner clients need are block headers and access to a trusted node that has the transactions cached.

Ah, no. All a non miner client needs is a prototcol talking to atrusted server.

No need to store anything except addresses and private keys.

You can send signed transactions to the server and get balacnes and new transfers from the server.

Stick it THIN - so you dont need to sync anything. Laptop open, check, finished.
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
July 27, 2011, 05:19:37 AM
 #9

The block chain is not very compressible, since most of it is hashes.  I got 21% space savings with gzip and 22% savings with bzip2.  (436690683 vs. 345345519 vs. 341013976 bytes)

The real magic comes in when you realize that you can prune old transactions.  Since any transaction in the chain can be the input for at most one new transaction, you can delete any transaction that was spent more than X blocks ago, with no ill effects.  Someone wrote a tool for that, and if I recall, he reported that something like 70% of the chain can be pruned already.

All non-miner clients need are block headers and access to a trusted node that has the transactions cached.

Ah, no. All a non miner client needs is a prototcol talking to atrusted server.

No need to store anything except addresses and private keys.

You can send signed transactions to the server and get balacnes and new transfers from the server.

Stick it THIN - so you dont need to sync anything. Laptop open, check, finished.

The trusted server part is currently difficult, so I expect a medium-weight client to pop up and be useful sooner than a fully stripped lightweight client.

That is, unless one of the 3 or more people/groups working on hardware wallets makes some big progress before a serious smartphone developer gets the itch to code up a medium client.

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
timmey
Newbie
*
Offline Offline

Activity: 28


torchat: q23xl6bdgdzhawhf


View Profile
July 27, 2011, 07:42:22 AM
 #10

so far bitcoin has been out for around 30 months and has only just reached about 400mb, 500 if you count indexes. thats 2.5 years.
yes, but bitcoin is also still waiting for it's major break through and doesn't really have many users and shops yet.

I will sign you up anonymously at realitykings.com (http://rk.com)[NSFW] for Bitcoins with 20% discount!
http://timmey.orgfree.com/s.php
read all details in this thread (https://bitcointalk.org/index.php?topic=3242Cool
payb.tc
Hero Member
*****
Offline Offline

Activity: 812



View Profile
July 27, 2011, 11:17:09 AM
 #11

so far bitcoin has been out for around 30 months and has only just reached about 400mb, 500 if you count indexes. thats 2.5 years.
yes, but bitcoin is also still waiting for it's major break through and doesn't really have many users and shops yet.

if the first 2.5 years made 400mb, i bet the *next* 2.5 years would easily make an extra 4000mb (if not compressed or pruned).
Bitsky
Hero Member
*****
Online Online

Activity: 542


View Profile
July 27, 2011, 11:51:46 AM
 #12

the blockchain right now is not 600mb, its more like 400, excluding the index files. and that can be compressed to at least 80% of the original size. and. 16 gigs should be good for a linux install and 2 more years worth of blockchain worst case.

so far bitcoin has been out for around 30 months and has only just reached about 400mb, 500 if you count indexes. thats 2.5 years.

My current bitcoin directory has 705MB right now.
A backup from the 18th has 612MB.
Another from the 14th has 592MB.

So, from 14th->18th, disk space increased by 20MB; that's 4MB/day.
From 18th->27th, another 93MB were saved, which means 9.3MB/day.

Of course it's easy to say "quit whining, diskspace is cheap", but when Bitcoin wants to enter the mobile/smartphone market, diskspace and traffic does matter.
Also, if you are an optimist and hope that Bitcoin will catch on and grow quickly, then the number of transactions will increase which in turn needs even more diskspace.
So, the faster Bitcoin grows, the bigger the storage requirements. This means the initial download/verify time will also increase and most likely anger new users.
Last but not least, when reaching a certain amount of transactions per second, the everyday John Doe simply won't have the bandwith to deal with the blockchain growth.

Bounty: Earn up to 68.7 BTC
Like my post? Feel free to drop a tip to 1BitskyZbfR4irjyXDaGAM2wYKQknwX36Y
netrin
Sr. Member
****
Offline Offline

Activity: 322


FirstBits: 168Bc


View Profile
July 27, 2011, 03:09:13 PM
 #13

I would like to see a ewallet service that uses a finite set of keys for each user wallet and lets the user download a copy.

And/or a service that handles the block chain and protocol allowing my client to deal only with my transactions and keys such as Webcoin and BitcoinJS are attempting:


Greenlandic tupilak. Hand carved, traditional cursed bone figures. Sorry, polar bear, walrus and human remains not available for export.
vector76
Member
**
Offline Offline

Activity: 70


View Profile
July 27, 2011, 04:57:28 PM
 #14

I was thinking about how to remove transactions when all the outputs have been used, but it seems to me that at least the transaction hash must be kept, because without it, it's impossible to tell the difference between an orphan transaction and a double-spend.  Although since neither are fully valid, it might make sense to discard them both, and if the orphan later becomes a non-orphan, hope that it will be retransmitted.

I'm not sure how much space savings this would provide, but I think it would be substantial.
phillipsjk
Legendary
*
Offline Offline

Activity: 1008

Let the chips fall where they may.


View Profile WWW
July 27, 2011, 05:08:33 PM
 #15

the blockchain right now is not 600mb, its more like 400, excluding the index files. and that can be compressed to at least 80% of the original size. and. 16 gigs should be good for a linux install and 2 more years worth of blockchain worst case.

It is my understanding that the nodes save multiple copies of the block-chain in case of a split or one of the block-chains becomes the "longest" one. I have had a test-node running since June 9, 2011 (0.2.22 and 0.2.23) for a total of 55 days. It ran out of disk space today; consuming 5.8 GB. That works out to 105MB per day. Disk usage dropped to 4.9GB when the client exited. The client had 125 connections during peak times.

James' OpenPGP public key fingerprint: EB14 9E5B F80C 1F2D 3EBE  0A2F B3DE 81FF 7B9D 5160
ctoon6
Sr. Member
****
Offline Offline

Activity: 350



View Profile
July 27, 2011, 06:06:25 PM
 #16

It is my understanding that the nodes save multiple copies of the block-chain in case of a split or one of the block-chains becomes the "longest" one. I have had a test-node running since June 9, 2011 (0.2.22 and 0.2.23) for a total of 55 days. It ran out of disk space today; consuming 5.8 GB. That works out to 105MB per day. Disk usage dropped to 4.9GB when the client exited. The client had 125 connections during peak times.
I have no idea about the test network, but my entire %appdata%\bitcoin dir has never went over 800mb, yet, but i would assume it will be by the end of next month at most.

phillipsjk
Legendary
*
Offline Offline

Activity: 1008

Let the chips fall where they may.


View Profile WWW
July 27, 2011, 06:13:08 PM
 #17

I was using the "real" network with the official client in -gen mode.

James' OpenPGP public key fingerprint: EB14 9E5B F80C 1F2D 3EBE  0A2F B3DE 81FF 7B9D 5160
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
July 27, 2011, 06:20:21 PM
 #18

the blockchain right now is not 600mb, its more like 400, excluding the index files. and that can be compressed to at least 80% of the original size. and. 16 gigs should be good for a linux install and 2 more years worth of blockchain worst case.

It is my understanding that the nodes save multiple copies of the block-chain in case of a split or one of the block-chains becomes the "longest" one. I have had a test-node running since June 9, 2011 (0.2.22 and 0.2.23) for a total of 55 days. It ran out of disk space today; consuming 5.8 GB. That works out to 105MB per day. Disk usage dropped to 4.9GB when the client exited. The client had 125 connections during peak times.

No, it doesn't save multiple copies.  When there is a fork, it keeps both blocks, but it doesn't need to make a copy of the rest of the chain to do it.

Check to see if you have a debug.log.  If I don't clear mine often it gets huge.  Currently around 1.5 GB.

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
phillipsjk
Legendary
*
Offline Offline

Activity: 1008

Let the chips fall where they may.


View Profile WWW
July 27, 2011, 06:45:49 PM
 #19

So there is: 4.2GB. (Gedit tried to load the whole thing..)

Code:
ubuntu@ubuntu:/media/803819A438199A6C/bitcoins$ tail debug.log
StopNode()
Running BitcoinMiner with 2 transactions in block
ThreadBitcoinMiner exiting, 0 threads remaining
DBFlush(true)
blkindex.dat refcount=0
blkindex.dat flush
wallet.dat refcount=0
wallet.dat flush
Bitcoin exiting


I suppose that is what I get for running the "beta" version: It is saving a lot of debugging information. Log rotation would probably help, but I doubt it is a priority if only used for debugging.

James' OpenPGP public key fingerprint: EB14 9E5B F80C 1F2D 3EBE  0A2F B3DE 81FF 7B9D 5160
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
July 27, 2011, 08:42:53 PM
 #20

http://forum.bitcoin.org/?topic=292.0

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!