Block chain size/storage and slow downloads for new users

[ngenko]

I'm actually impressed. After opening the client, I was ready to make transactions after a second.
Then, in order to recover the last 60 days in my Qt database, this is another story. It will take ages

I'm waiting for innovation on top of BTC and the extensive use of the Internet of Trust.
What BTC opens is just amazing.

 

[1714871108]

1714871108

[1714871108]

1714871108

[1714871108]

1714871108

[daemondazz]

I've just downloaded and installed MultiBit 0.5.13 on a Win7-64bit PC and it crashes as soon as I try to start it... That's not very new user friendly!

[niko]

I've just downloaded and installed MultiBit 0.5.13 on a Win7-64bit PC and it crashes as soon as I try to start it... That's not very new user friendly!

Hence the 0.5.13. Bitcoin is still in the beta phase.

[Mike Hearn]

It crashed? Could you report that in the MultiBit forum or issue tracker please. I haven't heard people complain of crashes before, but of course, it's always possible.

[btcton]

Good to know there is a solution to the problem.

[Arros]

It's an interesting and informative read, thanks.

[daemondazz]

It crashed? Could you report that in the MultiBit forum or issue tracker please. I haven't heard people complain of crashes before, but of course, it's always possible.

I did a 'Run as Administrator' (which is noted as being needed under Win8 - I'm on Win7 64 bit) and it started up and works.

[tvbcof]

Anyone working on a C implementation, preferably without a GUI?

I'd (with regret) run a SPV client in certain circumstances (compiled by myself on remote systems), but I'll never run anything Java and re-installed my one and only Windows machine from scratch to make sure I got completely rid of it some months ago.  Among other things I was creeped out by that 'Look' toolbar which got on there 'by accident' and could not be removed from Chrome.  That was well before Snowden informed the rest of the world how vulnerable we all are to PRISM machinations and the like.

[Mike Hearn]

I did a 'Run as Administrator' (which is noted as being needed under Win8 - I'm on Win7 64 bit) and it started up and works.

That's weird. You don't have to run the app as administrator on MacOS or Linux. OK, it'd be nice to fix that. I guess the issue is directory permissions somewhere. If you start  a thread in the MultiBit forum, perhaps Jim will explain a bit more.

Anyone working on a C implementation, preferably without a GUI?

Jeff Garzik started on one called picocoin. However you're crazy if you prefer C to Java for this work. The Java installer on Windows is obnoxious and Jim shouldn't rely on a pre-installed JVM, but you can easily bundle a stripped down VM with an app that doesn't have any kind of crapware with it and is self-contained inside the application itself. The JVM is open source and has multiple competitors anyway, so this will always be an option.

Once you fix Oracle's bad distribution habits, the advantages are pretty major - specifically, a guarantee against buffer, heap or stack corruptions that could allow a remote peer to compromise your wallet. Software written in C has a long history of containing such bugs. Also, you don't have to write platform specific code.

The right fix for the Java+SPV wallet problem is just to improve MultiBit so it bundles its own custom JVM instead of using the system provided one.

[FreedomCoin]

i have also found that updating my client to the current blockchain seems to "timeout" someones.. and i have to restart my wallet to get it to start DLing the chain again.

[Moogle]

The more I think I understand about bit coin the less I actually do!
Good read though!

[tvbcof]

The more I think I understand about bit coin the less I actually do!
Good read though!

Unsurprising.  The 'marketing' is not what I would consider to be particularly up-front.  Many of the claims about privacy, peer2peer, scalability, centralization, etc were at best temporarily true.  Many of the legitimate advocates of the solution honestly believed the 'hype',  and very few of those who understood the technical aspects of the solution sufficiently to see the looming issues did much toward disabusing the crowd about some of some of the misconceptions.  We all have at least a financial stake in growing the userbase after all, and it would probably be disingenuous to neglect that as a factor in how the solution has been presented.

[Dabs]

At some point Bitcoin-Qt will change such that it's able to delete old blocks. The details are still being worked out, but most likely you'll be able to say "Use up to 10 GB of disk space" and it will never use more than that. Nodes will broadcast how much of the chain they have and are able to serve. New nodes that are starting from scratch will have to search out other nodes that still have the full chain and sync from them, but any node that just wasn't online for a while and needs to grab the latest parts of the chain will be able to use most of the others. By controlling disk space usage, you can also indirectly control bandwidth usage (you can't upload data you don't have).

What happens, if in a hundred years from now, almost everyone is using the "Use up to 10 GB of disk space" feature? All the old nodes can probably keep up, but new nodes starting from scratch won't be able to find a node with the full chain, or have difficulty doing so.

Or is that a problem we shouldn't worry about for the next 50 years? I can see full nodes "centralizing" then, but it's possible that there will be entities or countries that will maintain a full node independent of other full nodes.

Right now, everyone running QT or the reference client is a full node.

I can see a far distant future where no normal or regular individual has a full node (except for the geeks / enthusiasts / those who can afford), but there will still be at least maybe 1 full node per country with the bigger countries having several, maybe one per university or one per "department" or "agency" or one per "private corporation".

I mean, large companies already maintain servers for all sorts of purposes. What's another one just dedicated to just bitcoind with 500++ gigs of space (or whatever is the then current capacity of hard drives or equivalent)?

[Mike Hearn]

In theory, Bitcoin can still operate even if every copy of the old parts of the chain are destroyed. It means that to start a new node, you have to copy the directory of an existing node that you trust to be correct. This is obviously a problem if you don't have a node that you trust, and it weakens the "trust nobody" aspect of Bitcoin somewhat, but payments would still flow.

In practice, yes, if the chain became absolutely gigantic then you'd get a small number of large organisations that commit to keeping it around as a kind of public service I'm sure. Or they could sell copies of it on stacks of BluRays or magnetic tapes. Look at it this way. Several organisations keep entire histories of the web (google, archive.org, probably microsoft as well). That's a titanic amount of data too. Somehow it happens anyway.

[tvbcof]

In theory, Bitcoin can still operate even if every copy of the old parts of the chain are destroyed. It means that to start a new node, you have to copy the directory of an existing node that you trust to be correct. This is obviously a problem if you don't have a node that you trust, and it weakens the "trust nobody" aspect of Bitcoin somewhat, but payments would still flow.

In practice, yes, if the chain became absolutely gigantic then you'd get a small number of large organisations that commit to keeping it around as a kind of public service I'm sure. Or they could sell copies of it on stacks of BluRays or magnetic tapes. Look at it this way. Several organisations keep entire histories of the web (google, archive.org, probably microsoft as well). That's a titanic amount of data too. Somehow it happens anyway.

Typically through monetizing the intelligence information coming off user access and generally enticing traffic for other properties which do so, though in the case of archive.org I think they may be primarily a philanthropic organization.  One way or another, if they are US based it is a fair assumption that any network interaction will be analyzed by various state and private organizations.  Indeed, unless one is exceedingly careful that assumption should be made about any Internet activity anywhere.

Bringing things back to Bitcoin, I see this sort of utility as a threat to the solution.  Any crypto-currency will be a rich source of intelligence information in proportion to it's success.  There is probably a role for a solution which welcomes such analysis for users who don't care and a role for a different implementation which focuses on hardening against such analysis (and the potential for sister monitoring and legal infrastructure to attack the solution), but each implementation will have one opportunity to decide what camp they wish to be in.  Being the first, Bitcoin get's the 'first round draft choice' here, though it looks to me like the principle movers of Bitcoin have already made the decision.  For my part, I'm falling back into the mode of deciding to live with whatever decision is made...as if I have a choice...

[solex]

In theory, Bitcoin can still operate even if every copy of the old parts of the chain are destroyed. It means that to start a new node, you have to copy the directory of an existing node that you trust to be correct. This is obviously a problem if you don't have a node that you trust, and it weakens the "trust nobody" aspect of Bitcoin somewhat, but payments would still flow.

In practice, yes, if the chain became absolutely gigantic then you'd get a small number of large organisations that commit to keeping it around as a kind of public service I'm sure. Or they could sell copies of it on stacks of BluRays or magnetic tapes. Look at it this way. Several organisations keep entire histories of the web (google, archive.org, probably microsoft as well). That's a titanic amount of data too. Somehow it happens anyway.

Such a data storage storage scenario seems highly unlikely, with this kind of tech emerging:
"The [nanostructured glass] allows unprecedented parameters including 360 TB/disc data capacity, thermal stability up to 1000°C and practically unlimited lifetime."
http://www.southampton.ac.uk/mediacentre/news/2013/jul/13_131.shtml

It really is transmission bandwidth, and the in-memory (RAM) requirements, for handling the blockchain which are the limiting factors.

[Mike Hearn]

Yeah. That tech sounds cool, but even magnetic tape can store massive quantities of data for almost no money at all. You don't get fast access to it of course but to serve up the chain you don't need that, what you need is bulk streaming reads which is exactly what tape robots provide.

tvbcof, much though I dislike industrial-scale spying, it has nothing to do with how the chain is stored or served in future. The chain is a public document by definition.

[tvbcof]

...
tvbcof, much though I dislike industrial-scale spying, it has nothing to do with how the chain is stored or served in future. The chain is a public document by definition.

My comments were directed toward presenting a potential explanation to the 'mystery' of how 'public services' provided by corporate entities come to be.

Many of us never really saw the storage aspect of the block chain as much as a problem, and I'm pretty sure we've been through that before.  Access to the data both locally for functional purposes and for WAN transmission catch-up operations are somewhat more salient concerns, but they are surmountable.  The defining issue here is real-time and near real-time economic activity on the network, and how core Bitcoin is going to evolve to support this.  If it does so natively, I argue that it is almost certain to fall victim to abusive surveillance practices and be under constant threat of technical and legal attacks if it challenges other solutions.

That said, it is a perfectly valid point of view that most potential users don't really care about the privacy issues and the service providers are right to make a dime from their efforts anyway, and also that the solution does not necessarily need to present a challenge to other solutions if implemented 'correctly'.  It's not my point of view, but I can accept it as valid.

[PrintMule]

At some point Bitcoin-Qt will change such that it's able to delete old blocks. The details are still being worked out, but most likely you'll be able to say "Use up to 10 GB of disk space" and it will never use more than that. Nodes will broadcast how much of the chain they have and are able to serve. New nodes that are starting from scratch will have to search out other nodes that still have the full chain and sync from them, but any node that just wasn't online for a while and needs to grab the latest parts of the chain will be able to use most of the others. By controlling disk space usage, you can also indirectly control bandwidth usage (you can't upload data you don't have).

What happens, if in a hundred years from now, almost everyone is using the "Use up to 10 GB of disk space" feature? All the old nodes can probably keep up, but new nodes starting from scratch won't be able to find a node with the full chain, or have difficulty doing so.

Or is that a problem we shouldn't worry about for the next 50 years? I can see full nodes "centralizing" then, but it's possible that there will be entities or countries that will maintain a full node independent of other full nodes.

Right now, everyone running QT or the reference client is a full node.

I can see a far distant future where no normal or regular individual has a full node (except for the geeks / enthusiasts / those who can afford), but there will still be at least maybe 1 full node per country with the bigger countries having several, maybe one per university or one per "department" or "agency" or one per "private corporation".

I mean, large companies already maintain servers for all sorts of purposes. What's another one just dedicated to just bitcoind with 500++ gigs of space (or whatever is the then current capacity of hard drives or equivalent)?

Make paid (like in mining) full nodes, but, to make it really spread around make a rule of no more than x nodes per geographical area (city/county/country) or ISP  or per certain ip adress group. That way it will be as decentralised as possible, with lots of people simply co-hosting from their web servers, not some big company datacenter full of servers having a big chunk of the global node count. Also reward for hosting a node should be very smallish. Just to cover average costs by a 5-10%

[aficionadog]

Are there any stats someone could provide as to how much bandwidth and storage per month are we talking about to run full node like Blockchain?