[new feature discussion] -fastboot

[regis]

Hey guys,

I'm in the middle of implementing new option for official bitcoin client:  -fastboot.
As we know, currently syncing "fresh" client takes really long time - even several hours. This option will allow user to download all blocks in a very short time, by downloading almost all blocks (as compressed file) from one of mirrors (mirror will be choosen randomly by client).  MDhash of downloaded file will be hardcoded into client, to prevent changing blocks. It also mean that this code needs to be updated every few weeks (as currently is with the number of blocks),  but I think it's worth it.

What do you think about that?

Pros:
+ sync you fresh client in few minutes


Cons:
- servers with files are centralized, so in case they doesn't work,  option won't speed up


Thanks!

----
EDIT: I've run small test.

I used Matt's blockchain (thanks Matt for sharing this!!) on fresh client.
(http://bitcoin.bluematt.me/bitcoin-nightly/blockchain-nightly/)

download time: 7 minutes
bunzip2 and tar -  ~ 2 minutes
./bitcoin -rescan :  1 minute


total: 10 minutes
result:  140600 blocks done
-------

[regis]

Any constructive criticism from people who voted no ?  Thanks!

[RaTTuS]

how about loading the blocks backwards?

[regis]

what do you mean? take only last N blocks? but then we don't have the whole history

[sirk390]

It would be nice but I'm not sure you know what you are talking about.
1. You can't update the code every few weeks(How ?)
2. You don't need to as the blockchain is allready "protected" against block changes.

[aq]

Excellent idea! This will be much needed for bitcoin to become useful for the average person.
That being said, I think you should add a -slowboot option and have the fast download as the default behavior.

Maybe you could combine the idea from RaTTuS with the download: have one thread load the block backward from the p2p network, while another thread downloads the cache files from the central servers. Now let those run until they meet each other. That could even solve the validation of the centrally stored mirror files, because if those are bogus the two chains would never match.

[JoelKatz]

I think you're solving the wrong problem. Most of the time is not spend downloading the block chain but processing it. Take a look at your CPU usage.

[regis]

@sirk390

1. you do can -  git push  of course I was talking only about updating md5 string in the code. (so, older clients will still work but have to download more blocks from p2p than newer clients [equal to the number of blocks between releases])

2. You don't need to as the blockchain is allready "protected" against block changes.

It's just another protection. Calculating hash is not a big deal, and I'm not sure how exactly blockchain protection works when attacker can change almost all blocks.

[regis]

I think you're solving the wrong problem. Most of the time is not spend downloading the block chain but processing it. Take a look at your CPU usage.

Is it true also for newer blocks? I know that it's pretty fast to download first  60-70k,  but later it's getting slower and slower....


BTW. perhaps we can store already processed blocks? (as binary blob ),  of course with md5 verification?  then we get everything as fast as download

[zellfaze]

I think that if you just download the block chain and stick it in the bitcoin directory then the Bitcoin client doesn't check if they are valid.  It just assumes they are.

Isn't that what the -rescan option is for?  Re-validating the block-chain?  Or am I mistaken?

[regis]

@zellfaze   hm...but don't we need to run  -rescan? If I download 140k blocks,  what about my transactions which were included in those blocks? Will they show up or I need to rescan?   
Anyway, I believe that still download + rescan is faster than p2p.

[regis]

ok, I did the homework.

I used Matt's blockchain (thanks Matt for sharing this!!) on fresh client.
(http://bitcoin.bluematt.me/bitcoin-nightly/blockchain-nightly/)

download time: 7 minutes
bunzip2 and tar -  ~ 2 minutes
./bitcoin -rescan :  1 minute


total: 10 minutes
result:  140600 blocks done

[Pieter Wuille]

-rescan does not revalidate the block chain. It only synchronizes the block chain with the wallet, adding missing transactions to it.

And indeed, most of the time is spent validating the block chain, not downloading. Before 0.3.24, larger blocks often caused disconnects when sending because of an anti-spam measure. That is/was also a cause of delay when starting up, but is slowly improving, as more nodes (and thus potential download partners) upgrade to 0.3.24.

That said, a -fastboot argument that takes a URL or filename to import a blockchain from may be useful in certain situations.

[ArtForz]

Thinking out loud...
Do we even need to check the ECDSA signatures in blocks older than the last checkpoint?
Assuming the checkpoint hash is okay, you can't change any transactions in blocks before it without breaking the chain.
Assuming the checkpoint hash was replaced (eve hacked the devs box or something)... we're pretty much fucked anyways as the client will reject the legit chain and stay on the attackers chain.

Any input?

[regis]

And indeed, most of the time is spent validating the block chain, not downloading. Before 0.3.24, larger blocks often caused disconnects when sending because of an anti-spam measure. That is/was also a cause of delay when starting up, but is slowly improving, as more nodes (and thus potential download partners) upgrade to 0.3.24.

But here we are not validating blockchain block-by-block (which takes a lot of CPU). I just verify md5um of a file which contains 140000 blocks.  It's  much much faster.

That said, a -fastboot argument that takes a URL or filename to import a blockchain from may be useful in certain situations.

it can take argument, but I think it should also have some default mirrors hardcoded (user usually just download and run a program, and don't care about reading docs and looking for URLs, etc...). I already have 3 people who volunteered to provide mirrors, which may be enough for beta version.

[regis]

Thinking out loud...
Do we even need to check the ECDSA signatures in blocks older than the last checkpoint?
Assuming the checkpoint hash is okay, you can't change any transactions in blocks before it without breaking the chain.
Assuming the checkpoint hash was replaced (eve hacked the devs box or something)... we're pretty much fucked anyways as the client will reject the legit chain and stay on the attackers chain.

Any input?

Hey ArtForz, I think you're right. Anyway, I think it's safer, cleaner and easier to implement this with hardcoded checksums. First, we are discovering very early that downloaded blocks are invalid (because someone changed them or there were problems with download). It allows to retry download from a different mirror as soon as md5hash is calculated.

Secondly, I'm not sure how exactly works the logic inside client (handling broken chain), and I'd prefer to not touch it, just to be on a safe side.  With checksum approach I just write a class with one method, which downloads and verifies files. This method can be run from any point of client, and the rest of the client is untouched.

Last but not least, we get another (perhaps redundant) protection with this md5sum.

[casascius]

I have thought of this... in my ideal world, this is how it would be implemented.

The client has a few public keys which it trusts (developer maintained), or has a screen where trusted keys can be manipulated by the user (developer maintains the defaults but user is ultimately in charge).

At some point, the blocks themselves can contain checkpoint transactions that vouch for everything up through and including the blocks that contain them.  They don't have to be anything special, they could be nothing more than standard Bitcoin transactions involving a signature from the hardcoded key.

When the client sees such a checkpoint, it can assume that everything before it is safe.  With the checkpoints in the block chain, there is no need to keep hard-coding checkpoints in the software (or, at least, the number of hard-coded checkpoints can be vastly reduced for the same effect.)

Finally....(separate feature).... the software ought to support a sort of "flat file import" that contains a recent copy of the blockchain in "blob" format, signed the same way.  If a well-formed and validly-signed "distribution blob" is available, the client should take in everything from the blob and not ask for a single block from a peer until the entire blob has been "downloaded" (read: copied into the preferred format of the client's choice, and indexed the way the client likes it...as the "blob" shall have no indexes).  Given that, the blob should be optimized for space as much as possible (e.g. deduplication of public keys, not wasting bytes wherever possible, etc.)

Blobs, when obtained from a trusted party, could also give rise to pruned blockchains to save space.  When accepting a pruned chain, one must trust that the chain doesn't omit any transactions that would lead the client to believe that spent coins are actually unspent.  A trusted party (developer whose public key is harcoded) could be the person to sign a trusted pruned blockchain, and a third party open source utility could allow the paranoid among us to compare the signed pruned version with the full version to confirm for everyone else that it is indeed not misrepresenting the unspent status of any coins.

[regis]

I have thought of this... in my ideal world, this is how it would be implemented.

The client has a few public keys which it trusts (developer maintained), or has a screen where trusted keys can be manipulated by the user (developer maintains the defaults but user is ultimately in charge).

Hey casascius, I think that's a great idea. Perhaps (for the first version) I can just hardcode signatures of some people (responsible for github/bitcoin), and then - in client - just verify signatures of downloaded files. This will allow to update file with blocks every day (and because of that function should provide even faster boot), without changing client source code.

What do you guys think about hardcoding signatures instead of md5sum of first N blocks?

[regis]

beta version for linux:

https://github.com/kkurach/bitcoin/tree/fastboot

any comments?  (files:  src/fastboot.h  src/fastboot.cpp)

[Pieter Wuille]

And indeed, most of the time is spent validating the block chain, not downloading. Before 0.3.24, larger blocks often caused disconnects when sending because of an anti-spam measure. That is/was also a cause of delay when starting up, but is slowly improving, as more nodes (and thus potential download partners) upgrade to 0.3.24.

But here we are not validating blockchain block-by-block (which takes a lot of CPU). I just verify md5um of a file which contains 140000 blocks.  It's  much much faster.

Of course it is. That's not what I meant. What you are doing is useful, but it does not have the same security guarantees. Right now, through hardcoded block hashes in the client, the maintainer has the ability to force a particular chain onto you, but that chain must still exist and be valid. If the hash of the entire block file is hardcoded in the client, the maintainer can effectively make you get an invalid block database.