Very few normal people would wait days for the blockchain to download.

[Transisto]

Waiting clueless for the block count rise is the most boring thing I ever did.

Seriously can't we have some pop-up saying "transaction may not be accurate until the whole blockchain has been downloaded" ?

,or anything analog to a progress bar so that people don't fall in a coma waiting for their balance to show up.

Also, :

• The HELP button provide no help whatsoever,
  
• The setting/option pane has one page "MAIN" ,,, why have this at all ?
• Seeing   "FILE / EXIT" would puzzle-out any non-geeks. (why file )

I'm quite impressed that the Bitcoin client is still at this "for geek only" stage.

Hope that the default usage of this UI die very soon.

[1715157135]

1715157135

[1715157135]

1715157135

[1715157135]

1715157135

[1715157135]

1715157135

[Andrew Bitcoiner]

Gavin talked about development priorities at the NY conference, you might want to take a look at that.  You may also want to make the changes yourself, the code is open source and I am sure they would appreciate some submissions of UI improvements.

[Transisto]

Gavin talked about development priorities at the NY conference, you might want to take a look at that.  You may also want to make the changes yourself, the code is open source and I am sure they would appreciate some submissions of UI improvements.

Thank you,

http://www.youtube.com/watch?v=0ljx4bbJrYE&feature=player_detailpage#t=1109s

[Gavin Andresen]

The new QT GUI (in git HEAD) has a nifty block-chain-download-progress indicator.

I'd like to pull together a version 0.5 release candidate and start testing it early next week.

Maybe the big feature for version 0.6 can be fast initial download (I'm thinking the best thing to do for brand-new, starting-with-an-empty-wallet installations is to download only block headers, function as a 'lightweight' client, and 'backfill' full blocks until you're caught up with the full chain-- then function as a 'full' client).

[Transisto]

... Maybe the big feature for version 0.6 can be fast initial download (I'm thinking the best thing to do for brand-new, starting-with-an-empty-wallet installations is to download only block headers, function as a 'lightweight' client, and 'backfill' full blocks until you're caught up with the full chain-- then function as a 'full' client).

I just noticed I had a problem and it was taking longer,  

I have a flaky internet connection and it seems like dropped or unresponsive connection are not being retried by the client.
To the point were 56 connection get me a block every 3 seconds.
Restating it made it faster again.

can be quite an issue for people behind NAT.

[AlexWaters]

One of Gavin's recent patches gives a ~30% increase in download time. While it still takes a long time, I think it's manageable for most people.

Additionally, as mentioned - allowing the user to choose a headless version is something I hope will be available soon. There is plenty of innovation that could arise if people can get and start using the client in under 10 minutes.

If you would like to get new features sooner, and don't have a development / C++ background - feel free to contact me to get involved with the testing/QA process.

[etotheipi]

Normal people dont even use the client, they use an online wallet service. The client is for geeks.

I think the future of bitcoin is dependent on making lightweight clients. I am glad to see developement moving that direction.

Lightweight clients will stuff suffer from having to get their initial block scan from somewhere.  Either through a trusted service online, or your computer or device still downloading the entire chain, and throw away the 99.9% that isn't yours.   Although, a lot of this can be avoided by making sure addresses are tagged with a first date and/or block number, so that your client doesn't have to download the entire chain to find its txs.

Btw, my Linux/Ubuntu client was unusable in 0.3.24 because it would take days to update, even if it was just a couple hundred blocks behind.  I just updated to 0.4.0 and now it catches up almost immediately.  I downloaded most of the chain through 0.4.0 and it took less than a day.  Obviously not ideal for the casual user, but it's better than it was before...

[Jan]

Normal people dont even use the client, they use an online wallet service. The client is for geeks.

I think the future of bitcoin is dependent on making lightweight clients. I am glad to see developement moving that direction.

There is one (possibly two) lightweight android clients in the works implemented on top of the BCCAPI. I am very confident that this is the secure way for Bitcoin to his the mass-market. We are talking about clients that are installed and fully functional within minutes, without no risk of the server side going renegade and stealing coins.

[jim618]

As the bitcoinj blockchain from the genesis block is currently less than 16MB, for MultiBit I am just shipping it in the installer.

Gets people started much quicker. The same logic would apply to other bitcoinj clients I think.

[qikaifu]

It seems that Gavin has brought us really good news here. Thanks!

[kano]

I saw a link last week of downloads of the full block-chain up to whatever recent block.
Would that also be a good idea to have available in some reasonably secure place to help people get the full block-chain up and running much faster?
Of course there is the minor issue of the reliability of the source, however, as long as the client isn't included with it, there should be no security issue since the proper client will require it to be valid anyway.

[ThomasV]

Normal people dont even use the client, they use an online wallet service. The client is for geeks.

I think the future of bitcoin is dependent on making lightweight clients. I am glad to see developement moving that direction.

There is one (possibly two) lightweight android clients in the works implemented on top of the BCCAPI. I am very confident that this is the secure way for Bitcoin to his the mass-market. We are talking about clients that are installed and fully functional within minutes, without no risk of the server side going renegade and stealing coins.

I agree, this is the way to go.
ideally, the server-side of this BCCAPI should be performed by any node of the bitcoin network.

[Jan]

Normal people dont even use the client, they use an online wallet service. The client is for geeks.

I think the future of bitcoin is dependent on making lightweight clients. I am glad to see developement moving that direction.

There is one (possibly two) lightweight android clients in the works implemented on top of the BCCAPI. I am very confident that this is the secure way for Bitcoin to his the mass-market. We are talking about clients that are installed and fully functional within minutes, without no risk of the server side going renegade and stealing coins.

I agree, this is the way to go.
ideally, the server-side of this BCCAPI should be performed by any node of the bitcoin network.

This is an interesting idea! Basically this means that the client side can connect to any node in the network and let it be its proxy for creating valid transactions that the client can sign. The worst thing an evil node could do is to misinform the client about its current balance and generate transactions that are not valid/accepted by the network. Both of which can be alliveated by connecting to another node or connecting to several nodes simultaneously and compare the results.

The flipside is that the client side will have to reveal the set of bitcoin addresses belonging to its wallet. For a simple mass-market client anonymity may not be an issue, and it might even just have one key-pair.

[ThomasV]

This is an interesting idea! Basically this means that the client side can connect to any node in the network and let it be its proxy for creating valid transactions that the client can sign. The worst thing an evil node could do is to misinform the client about its current balance and generate transactions that are not valid/accepted by the network. Both of which can be alliveated by connecting to another node or connecting to several nodes simultaneously and compare the results.

The flipside is that the client side will have to reveal the set of bitcoin addresses belonging to its wallet. For a simple mass-market client anonymity may not be an issue, and it might even just have one key-pair.

actually I read a post by Gavin discussing that, but I can't find where it is.
the client does not have to reveal the set of bitcoin addresses to a single server; he could use a different server for each address.

btw, I downloaded your java code; is the protocol documented somewhere?

[Jan]

This is an interesting idea! Basically this means that the client side can connect to any node in the network and let it be its proxy for creating valid transactions that the client can sign. The worst thing an evil node could do is to misinform the client about its current balance and generate transactions that are not valid/accepted by the network. Both of which can be alliveated by connecting to another node or connecting to several nodes simultaneously and compare the results.

The flipside is that the client side will have to reveal the set of bitcoin addresses belonging to its wallet. For a simple mass-market client anonymity may not be an issue, and it might even just have one key-pair.

actually I read a post by Gavin discussing that, but I can't find where it is.
the client does not have to reveal the set of bitcoin addresses to a single server; he could use a different server for each address.

btw, I downloaded your java code; is the protocol documented somewhere?

I have been giving this a bit more thought. This approach requires that the bitcoin node is able to efficiently determine which transactions are associated with any incoming address, and whether the transactions are not already spent. I have not looked at the database layout of the official bitcoin client, but I doubt that the current design has been optimized for this use, we do after all have a -rescan option which takes a significant amount of time.

Regarding the BCCAPI protocol:
The API is pretty small and very well documented.
Look at the interface here: http://code.google.com/p/bccapi/source/browse/trunk/src/com/bccapi/api/BitcoinClientAPI.java
And the implementation here: http://code.google.com/p/bccapi/source/browse/trunk/src/com/bccapi/core/BitcoinClientApiImpl.java
Its simple HTTP requests over a HTTPS connection. There is no separate protocol documentation.

[ThomasV]

I have been giving this a bit more thought. This approach requires that the bitcoin node is able to efficiently determine which transactions are associated with any incoming address, and whether the transactions are not already spent. I have not looked at the database layout of the official bitcoin client, but I doubt that the current design has been optimized for this use, we do after all have a -rescan option which takes a significant amount of time.

yes, you are right, but something like libbitcoin or ABE could do it

Regarding the BCCAPI protocol:
The API is pretty small and very well documented.
Look at the interface here: http://code.google.com/p/bccapi/source/browse/trunk/src/com/bccapi/api/BitcoinClientAPI.java
And the implementation here: http://code.google.com/p/bccapi/source/browse/trunk/src/com/bccapi/core/BitcoinClientApiImpl.java
Its simple HTTP requests over a HTTPS connection. There is no separate protocol documentation.

I looked at these files. but you did not publish the server side, did you?
from looking at the code, and from what you say above, it seems that your server has to keep track of the addresses and to maintain a list  of accounts.
This is not exactly what I had in mind. I am thinking about servers you don't have to be faithful to.

[Jan]

I have been giving this a bit more thought. This approach requires that the bitcoin node is able to efficiently determine which transactions are associated with any incoming address, and whether the transactions are not already spent. I have not looked at the database layout of the official bitcoin client, but I doubt that the current design has been optimized for this use, we do after all have a -rescan option which takes a significant amount of time.

yes, you are right, but something like libbitcoin or ABE could do it

Regarding the BCCAPI protocol:
The API is pretty small and very well documented.
Look at the interface here: http://code.google.com/p/bccapi/source/browse/trunk/src/com/bccapi/api/BitcoinClientAPI.java
And the implementation here: http://code.google.com/p/bccapi/source/browse/trunk/src/com/bccapi/core/BitcoinClientApiImpl.java
Its simple HTTP requests over a HTTPS connection. There is no separate protocol documentation.

I looked at these files. but you did not publish the server side, did you?
from looking at the code, and from what you say above, it seems that your server has to keep track of the addresses and to maintain a list  of accounts.
This is not exactly what I had in mind. I am thinking about servers you don't have to be faithful to.

No, the server side code has not been made public. And yes, the server side does keep track of transactions that have inputs for or outputs to the public keys belonging to various wallets. This bookkeeping is done while downloading the block chain.

The server that you are thinking about would have a database containing all transactions, which is optimized for determining this in close to real-time for any given public key. While this is seems doable, it is not what I have built.

[ThomasV]

No, the server side code has not been made public. And yes, the server side does keep track of transactions that have inputs for or outputs to the public keys belonging to various wallets. This bookkeeping is done while downloading the block chain.

The server that you are thinking about would have a database containing all transactions, which is optimized for determining this in close to real-time for any given public key. While this is seems doable, it is not what I have built.

ok, I have decided to implement this idea.
are you willing to share the server code ?
I would be interested in the communication between your server and the bitcoin network (propagating transactions)

[Jan]

No, the server side code has not been made public. And yes, the server side does keep track of transactions that have inputs for or outputs to the public keys belonging to various wallets. This bookkeeping is done while downloading the block chain.

The server that you are thinking about would have a database containing all transactions, which is optimized for determining this in close to real-time for any given public key. While this is seems doable, it is not what I have built.

ok, I have decided to implement this idea.
are you willing to share the server code ?
I would be interested in the communication between your server and the bitcoin network (propagating transactions)

The server code is based on bitcoinj, sprinkled with my own code. It requires pre-knowledge of public keys before transactions are confirmed, which is of no use to you.
If I were to implement this I would take bitcoinj and throw away all the wallet handling code, as you won't need it for your purpose.
Look for BlockChain.java, which is responsible of all the block chain handling and plug in an persistence layer that allows you to efficiently query for unspent transaction inputs based on a public-key/address. Once this is in place you are half-way there.

[ThomasV]

unfortunately java is not my favourite language.
I guess I will try to do it from scratch in python, using bitcoin-abe + bitcoind at the server side.

[Jan]

unfortunately java is not my favourite language.
I guess I will try to do it from scratch in python, using bitcoin-abe + bitcoind at the server side.

Good luck with your venture.

[etotheipi]

I may not understand what's under discussion here, but if we're talking about new users and lightweight clients, what's wrong with downloading only the headers (which should take less than 5 minutes), and then only download the parts of the blockchain that have timestamps after the creation date of each key in the wallet?

A new user who doesn't have any keys, will be creating those keys right now, and actually doesn't need any blocks that were created before his own computer just generated them a minute ago.  He only needs the full set of blockheaders to figure out the longest chain and determine "truth" for when he does need to get blocks.  He stores the parts that are relevant to himself, and will always have a full list of available txouts, without any need to trust anyone else. 

Sure, you can't verify other users' txs easily, unless you see the tx in the blockchain with X confirmations.  This may make some people uncomfortable, but I believe the future will eventually require people to trust the longest chain (and all the Tx's in it) since it will eventually be infeasible for people to store the entire blockchain themselves.

Btw, you mentioned python:  check out my codebase, PyBtcEngine.  Right now, the full suite uses the full blockchain, but I do plan to make a lightweight version of it.  There's no networking yet, but it does handle just about everything else (the last thing I need is knapsack optimization to create a set of txOuts to send to my ECDSA signature code).  You might find the python code alone to be useful without any of the C++/SWIG, you just won't have access to the entire blockchain without the C++ (I found it way too slow to juggle the full chain in python).

[kano]

But then how can you accept a payment from a previous transaction?
Each payment has input that is the output of an earlier transaction.
Thus you would need the blockchain back to the earlier transaction to be able to receive it ...

[etotheipi]

But then how can you accept a payment from a previous transaction?
Each payment has input that is the output of an earlier transaction.
Thus you would need the blockchain back to the earlier transaction to be able to receive it ...

This depends entirely on your use-case.  The fact that miners included it in the blockchain is evidence enough that it's valid, as long as it is sufficiently deep.  The miners are checking the validity for you, and the tx would be DOA on broadcast it if it wasn't.  The fact that the world is still building off the blockchain that includes that transaction confirms that it's a legit tx... as long as you don't mind waiting for a few confirmations. 

Of course, you can't "verify" transactions that have no confirmations yet, because (as you said) you don't have the ability to check the inputs.  But even if you did have that information, there's plenty of reasons not to accept 0- or 1-confirmation transactions anyway (unless they are small and you don't mind eating a couple invalids).  But for most users, transferring money between family/friends, or online customer-to-merchant, you will be waiting for 2+ confirmations regardless of whether you have the entire blockchain -- so why waste the hard-drive space? 

Alternatively, when you receive the Tx, you can request the input transactions/merkle trees from your peers, and verify them against the headers in your longest chain (I'm assuming you're at least holding the headers).  Then you can at least know the tx is potentially valid with 0 confirmations, but again, if it's a significant amount of money, you best wait for 2+ anyway.

[MoonShadow]

I may not understand what's under discussion here, but if we're talking about new users and lightweight clients, what's wrong with downloading only the headers (which should take less than 5 minutes), and then only download the parts of the blockchain that have timestamps after the creation date of each key in the wallet?

Nothing at all.  This is actually on of the 'lightweight' client models that have been proposed.

[MatthewLM]

Can compression be used on the blocks when transferring? Could they be included in the software packages? Could they be provided online for fast server downloads?

[MoonShadow]

Can compression be used on the blocks when transferring?

Yes, but the gain would be small.

Could they be included in the software packages?


 Could they be provided online for fast server downloads?

Yes to both, but then it becomes a trust issue.  At least on some level.

[etotheipi]

Can compression be used on the blocks when transferring?

Yes, but the gain would be small.

Could they be included in the software packages?


 Could they be provided online for fast server downloads?

Yes to both, but then it becomes a trust issue.  At least on some level.

You can download the headers in less than 5 minutes.  This gives you a definitive map of the data you should receive to fill in the rest.  Therefore, if the client only uses the network to pick out 15 MB of blockheaders with the longest chain/work, then he can get the other 600 MB from anywhere and be confident he's getting the right data.  This doesn't really require any more trust than downloading the data normally.

And compression wouldn't achieve too much:  most of the blockchain is hashes which are, by design, supposed to "random" sequences of bits.  Random data is not very compressible (in fact, you can use compression algorithms to test your encryption/hashing algorithms:  if the output will compress more than 5%-10%, then it's not sufficiently random).

[Jan]

I may not understand what's under discussion here, but if we're talking about new users and lightweight clients, what's wrong with downloading only the headers (which should take less than 5 minutes), and then only download the parts of the blockchain that have timestamps after the creation date of each key in the wallet?

ThomasV and I are discussing super-light-weight smartphone'ish clients with limited bandwidth that do not need to download/store the block chan at all. They just store the private keys.

[kano]

...
Could they be provided online for fast server downloads?

Yeah I mentioned that on the previous page.
A download of the block chain (independent of the client) with a standard accepted method to get the hashes from the actual chain would easily ensure it's OK also.
The only issue is the client and well that of course (as I have said) should never be included with a block chain download.

[finway]

Yes, and most people close there client, instead, use mtgox.

[Jan]

Yes, and most people close there client, instead, use mtgox.

If you want normal people to participate in Bitcoin and you expect them to have a client running at all times that takes up gigabytes of storage/bandwidth and accepts incoming connections, then you are going to utterly fail. We need secure light-weight clients if we want to hit the masses, not bitcoin banks that run away with your money.

[ShadowOfHarbringer]

Wait, isn't somebody already working on a "Bootstrap" feature ?

Downloading the full blockchain from torrent sites and processing them using client would be definately the fastest option. Torrents also have built-in checksums, so they seem to have enough security.

[DiThi]

Does anybody thought of this before?

- A method for saving a chain of blocks (or its headers) up to a given point, in a single file/stream. No matter where this file is made, it should be always the same, provided it's old enough (6 hours?). I guess this is like getblocks but unlimited and done locally. (edit: and with a canonical reorganization or whatever it's needed to make sure it's equal everywhere)
- A command for querying the hash of this file.

This way you can download a file from anywhere no matter how untrusty it is, then query a bunch of random peers for the hash that this file should have, in the same way you get and verify the authenticity of individual blocks.

[etotheipi]

Does anybody thought of this before?

- A method for saving a chain of blocks (or its headers) up to a given point, in a single file/stream. No matter where this file is made, it should be always the same, provided it's old enough (6 hours?). I guess this is like getblocks but unlimited and done locally.
- A command for querying the hash of this file.

This way you can download a file from anywhere no matter how untrusty it is, then query a bunch of random peers for the hash that this file should have, in the same way you get and verify the authenticity of individual blocks.

This is mostly unnecessary.  It's extraordinarily difficult for someone to give you fake blockheaders, because they would need an extraordinary amount of computing power to give you fake headers that match the difficulty of the block (hash has enough leading zero-bits). i.e. proof-of-work.  You get the headers from a few different peers, and you can verify the leading zeros and accumulate all the difficulty values to get the longest chain.  Unless the attacker has more than 50% of the global computation in his control, he won't be able to feed you a chain of headers longer than the "actual, legit" blockchain headers.

I say mostly unnecessary because technically, if the attacker has a lot of computing power and luck, he might be able to feed you a blockheader list with 1 or 2 fake blockheaders at the top even with less than 50% global computation speed.  But unless he has 51%+, the "actual, legit" blockchain will be extended to be longer within few blocks, and your client will correct itself within an hour. 

tl;dr : It's so difficult to produce fake blockheaders, that, unless you are handling huge amounts of BTC, it is perfectly fine to trust any headers you receive on a lightweight node (as long as its hash has the leading zeros).  I'm sure someone will flame me for this statement...

[John Tobey]

It's so difficult to produce fake blockheaders, that, unless you are handling huge amounts of BTC, it is perfectly fine to trust any headers you receive on a lightweight node (as long as its hash has the leading zeros).

True, with a caveat.  Your untrusted block chain must have "pretty good" total difficulty, and you must acquire a recent estimate of "pretty good" from somewhere.

A lightweight client could produce a graph like http://bitcoin.sipa.be/speed-ever.png and let you compare it to a version of the graph obtained from a trusted source.  Perhaps newspapers will publish the graph, or you could go with your recollection of its shape.  Maybe various organisations will sign and publish statistics, and your lightweight client can ship with their public keys, fetch the signed messages, and tell you who agrees with your data as of N hours ago.  If your untrusted chain's implied hash rate starts to underperform the "real" one at some point in the past, you can be sure you have fake blocks from that point on.

[MoonShadow]

It's so difficult to produce fake blockheaders, that, unless you are handling huge amounts of BTC, it is perfectly fine to trust any headers you receive on a lightweight node (as long as its hash has the leading zeros).

True, with a caveat.  Your untrusted block chain must have "pretty good" total difficulty, and you must acquire a recent estimate of "pretty good" from somewhere.

A lightweight client that only uses block headers would simply have to choose three different sources at random, download the block headers from all three sources, and check them against each other to make certain that they agree.  If they don't dump all data collected from that set of three and start over with another set.  Change your set of three every couple thousand blocks, and you're pretty well protected.  This is similar to what the full client does when accepting a new block.

[etotheipi]

Agreed.  I shouldn't have been so hasty to say "accept any blockheaders."  You should download it from a couple different peers and as long as any one of your peers is honest, you'll be able to receive and quickly determine the longest chain.  This is, by definition, the "correct" chain.    Any longer chain that is invalid will soon be outpaced by the correct chain and any one honest peer will set you straight.

Thus, I would argue if someone shows you a tx, you can "quickly" download the entire blockheaders from the network, get the merkle tree with that tx-hash in it, and confirm it matches a blockheader more than 6 blocks deep in the header list.  I would trust that transaction.

[finway]

The new QT GUI (in git HEAD) has a nifty block-chain-download-progress indicator.

I'd like to pull together a version 0.5 release candidate and start testing it early next week.

Maybe the big feature for version 0.6 can be fast initial download (I'm thinking the best thing to do for brand-new, starting-with-an-empty-wallet installations is to download only block headers, function as a 'lightweight' client, and 'backfill' full blocks until you're caught up with the full chain-- then function as a 'full' client).

High that feature's priority, downloading is annoying.

[ThomasV]

unfortunately java is not my favourite language.
I guess I will try to do it from scratch in python, using bitcoin-abe + bitcoind at the server side.

Good luck with your venture.

alpha version is ready.
see the announcement here: https://bitcointalk.org/index.php?topic=50936.0