Very few normal people would wait days for the blockchain to download.

[Jan]

unfortunately java is not my favourite language.
I guess I will try to do it from scratch in python, using bitcoin-abe + bitcoind at the server side.

Good luck with your venture.

[etotheipi]

I may not understand what's under discussion here, but if we're talking about new users and lightweight clients, what's wrong with downloading only the headers (which should take less than 5 minutes), and then only download the parts of the blockchain that have timestamps after the creation date of each key in the wallet?

A new user who doesn't have any keys, will be creating those keys right now, and actually doesn't need any blocks that were created before his own computer just generated them a minute ago.  He only needs the full set of blockheaders to figure out the longest chain and determine "truth" for when he does need to get blocks.  He stores the parts that are relevant to himself, and will always have a full list of available txouts, without any need to trust anyone else. 

Sure, you can't verify other users' txs easily, unless you see the tx in the blockchain with X confirmations.  This may make some people uncomfortable, but I believe the future will eventually require people to trust the longest chain (and all the Tx's in it) since it will eventually be infeasible for people to store the entire blockchain themselves.

Btw, you mentioned python:  check out my codebase, PyBtcEngine.  Right now, the full suite uses the full blockchain, but I do plan to make a lightweight version of it.  There's no networking yet, but it does handle just about everything else (the last thing I need is knapsack optimization to create a set of txOuts to send to my ECDSA signature code).  You might find the python code alone to be useful without any of the C++/SWIG, you just won't have access to the entire blockchain without the C++ (I found it way too slow to juggle the full chain in python).

[kano]

But then how can you accept a payment from a previous transaction?
Each payment has input that is the output of an earlier transaction.
Thus you would need the blockchain back to the earlier transaction to be able to receive it ...

[etotheipi]

But then how can you accept a payment from a previous transaction?
Each payment has input that is the output of an earlier transaction.
Thus you would need the blockchain back to the earlier transaction to be able to receive it ...

This depends entirely on your use-case.  The fact that miners included it in the blockchain is evidence enough that it's valid, as long as it is sufficiently deep.  The miners are checking the validity for you, and the tx would be DOA on broadcast it if it wasn't.  The fact that the world is still building off the blockchain that includes that transaction confirms that it's a legit tx... as long as you don't mind waiting for a few confirmations. 

Of course, you can't "verify" transactions that have no confirmations yet, because (as you said) you don't have the ability to check the inputs.  But even if you did have that information, there's plenty of reasons not to accept 0- or 1-confirmation transactions anyway (unless they are small and you don't mind eating a couple invalids).  But for most users, transferring money between family/friends, or online customer-to-merchant, you will be waiting for 2+ confirmations regardless of whether you have the entire blockchain -- so why waste the hard-drive space? 

Alternatively, when you receive the Tx, you can request the input transactions/merkle trees from your peers, and verify them against the headers in your longest chain (I'm assuming you're at least holding the headers).  Then you can at least know the tx is potentially valid with 0 confirmations, but again, if it's a significant amount of money, you best wait for 2+ anyway.

[MoonShadow]

I may not understand what's under discussion here, but if we're talking about new users and lightweight clients, what's wrong with downloading only the headers (which should take less than 5 minutes), and then only download the parts of the blockchain that have timestamps after the creation date of each key in the wallet?

Nothing at all.  This is actually on of the 'lightweight' client models that have been proposed.

[MatthewLM]

Can compression be used on the blocks when transferring? Could they be included in the software packages? Could they be provided online for fast server downloads?

[MoonShadow]

Can compression be used on the blocks when transferring?

Yes, but the gain would be small.

Could they be included in the software packages?


 Could they be provided online for fast server downloads?

Yes to both, but then it becomes a trust issue.  At least on some level.

[etotheipi]

Can compression be used on the blocks when transferring?

Yes, but the gain would be small.

Could they be included in the software packages?


 Could they be provided online for fast server downloads?

Yes to both, but then it becomes a trust issue.  At least on some level.

You can download the headers in less than 5 minutes.  This gives you a definitive map of the data you should receive to fill in the rest.  Therefore, if the client only uses the network to pick out 15 MB of blockheaders with the longest chain/work, then he can get the other 600 MB from anywhere and be confident he's getting the right data.  This doesn't really require any more trust than downloading the data normally.

And compression wouldn't achieve too much:  most of the blockchain is hashes which are, by design, supposed to "random" sequences of bits.  Random data is not very compressible (in fact, you can use compression algorithms to test your encryption/hashing algorithms:  if the output will compress more than 5%-10%, then it's not sufficiently random).

[Jan]

I may not understand what's under discussion here, but if we're talking about new users and lightweight clients, what's wrong with downloading only the headers (which should take less than 5 minutes), and then only download the parts of the blockchain that have timestamps after the creation date of each key in the wallet?

ThomasV and I are discussing super-light-weight smartphone'ish clients with limited bandwidth that do not need to download/store the block chan at all. They just store the private keys.

[kano]

...
Could they be provided online for fast server downloads?

Yeah I mentioned that on the previous page.
A download of the block chain (independent of the client) with a standard accepted method to get the hashes from the actual chain would easily ensure it's OK also.
The only issue is the client and well that of course (as I have said) should never be included with a block chain download.

[finway]

Yes, and most people close there client, instead, use mtgox.

[Jan]

Yes, and most people close there client, instead, use mtgox.

If you want normal people to participate in Bitcoin and you expect them to have a client running at all times that takes up gigabytes of storage/bandwidth and accepts incoming connections, then you are going to utterly fail. We need secure light-weight clients if we want to hit the masses, not bitcoin banks that run away with your money.

[ShadowOfHarbringer]

Wait, isn't somebody already working on a "Bootstrap" feature ?

Downloading the full blockchain from torrent sites and processing them using client would be definately the fastest option. Torrents also have built-in checksums, so they seem to have enough security.

[DiThi]

Does anybody thought of this before?

- A method for saving a chain of blocks (or its headers) up to a given point, in a single file/stream. No matter where this file is made, it should be always the same, provided it's old enough (6 hours?). I guess this is like getblocks but unlimited and done locally. (edit: and with a canonical reorganization or whatever it's needed to make sure it's equal everywhere)
- A command for querying the hash of this file.

This way you can download a file from anywhere no matter how untrusty it is, then query a bunch of random peers for the hash that this file should have, in the same way you get and verify the authenticity of individual blocks.

[etotheipi]

Does anybody thought of this before?

- A method for saving a chain of blocks (or its headers) up to a given point, in a single file/stream. No matter where this file is made, it should be always the same, provided it's old enough (6 hours?). I guess this is like getblocks but unlimited and done locally.
- A command for querying the hash of this file.

This way you can download a file from anywhere no matter how untrusty it is, then query a bunch of random peers for the hash that this file should have, in the same way you get and verify the authenticity of individual blocks.

This is mostly unnecessary.  It's extraordinarily difficult for someone to give you fake blockheaders, because they would need an extraordinary amount of computing power to give you fake headers that match the difficulty of the block (hash has enough leading zero-bits). i.e. proof-of-work.  You get the headers from a few different peers, and you can verify the leading zeros and accumulate all the difficulty values to get the longest chain.  Unless the attacker has more than 50% of the global computation in his control, he won't be able to feed you a chain of headers longer than the "actual, legit" blockchain headers.

I say mostly unnecessary because technically, if the attacker has a lot of computing power and luck, he might be able to feed you a blockheader list with 1 or 2 fake blockheaders at the top even with less than 50% global computation speed.  But unless he has 51%+, the "actual, legit" blockchain will be extended to be longer within few blocks, and your client will correct itself within an hour. 

tl;dr : It's so difficult to produce fake blockheaders, that, unless you are handling huge amounts of BTC, it is perfectly fine to trust any headers you receive on a lightweight node (as long as its hash has the leading zeros).  I'm sure someone will flame me for this statement...

[John Tobey]

It's so difficult to produce fake blockheaders, that, unless you are handling huge amounts of BTC, it is perfectly fine to trust any headers you receive on a lightweight node (as long as its hash has the leading zeros).

True, with a caveat.  Your untrusted block chain must have "pretty good" total difficulty, and you must acquire a recent estimate of "pretty good" from somewhere.

A lightweight client could produce a graph like http://bitcoin.sipa.be/speed-ever.png and let you compare it to a version of the graph obtained from a trusted source.  Perhaps newspapers will publish the graph, or you could go with your recollection of its shape.  Maybe various organisations will sign and publish statistics, and your lightweight client can ship with their public keys, fetch the signed messages, and tell you who agrees with your data as of N hours ago.  If your untrusted chain's implied hash rate starts to underperform the "real" one at some point in the past, you can be sure you have fake blocks from that point on.

[MoonShadow]

It's so difficult to produce fake blockheaders, that, unless you are handling huge amounts of BTC, it is perfectly fine to trust any headers you receive on a lightweight node (as long as its hash has the leading zeros).

True, with a caveat.  Your untrusted block chain must have "pretty good" total difficulty, and you must acquire a recent estimate of "pretty good" from somewhere.

A lightweight client that only uses block headers would simply have to choose three different sources at random, download the block headers from all three sources, and check them against each other to make certain that they agree.  If they don't dump all data collected from that set of three and start over with another set.  Change your set of three every couple thousand blocks, and you're pretty well protected.  This is similar to what the full client does when accepting a new block.

[etotheipi]

Agreed.  I shouldn't have been so hasty to say "accept any blockheaders."  You should download it from a couple different peers and as long as any one of your peers is honest, you'll be able to receive and quickly determine the longest chain.  This is, by definition, the "correct" chain.    Any longer chain that is invalid will soon be outpaced by the correct chain and any one honest peer will set you straight.

Thus, I would argue if someone shows you a tx, you can "quickly" download the entire blockheaders from the network, get the merkle tree with that tx-hash in it, and confirm it matches a blockheader more than 6 blocks deep in the header list.  I would trust that transaction.

[finway]

The new QT GUI (in git HEAD) has a nifty block-chain-download-progress indicator.

I'd like to pull together a version 0.5 release candidate and start testing it early next week.

Maybe the big feature for version 0.6 can be fast initial download (I'm thinking the best thing to do for brand-new, starting-with-an-empty-wallet installations is to download only block headers, function as a 'lightweight' client, and 'backfill' full blocks until you're caught up with the full chain-- then function as a 'full' client).

High that feature's priority, downloading is annoying.

[ThomasV]

unfortunately java is not my favourite language.
I guess I will try to do it from scratch in python, using bitcoin-abe + bitcoind at the server side.

Good luck with your venture.

alpha version is ready.
see the announcement here: https://bitcointalk.org/index.php?topic=50936.0