Bitcoin Forum
February 27, 2017, 10:53:37 PM *
News: Latest stable version of Bitcoin Core: 0.13.2  [Torrent]. (New!)
 
   Home   Help Search Donate Login Register  
Poll
Question: Do you think that implementing fastboot will be useful?
Yes - 21 (87.5%)
No - 3 (12.5%)
Total Voters: 24

Pages: [1] 2 »  All
  Print  
Author Topic: [new feature discussion] -fastboot  (Read 2064 times)
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 18, 2011, 03:54:19 AM
 #1

Hey guys,

I'm in the middle of implementing new option for official bitcoin client:  -fastboot.
As we know, currently syncing "fresh" client takes really long time - even several hours. This option will allow user to download all blocks in a very short time, by downloading almost all blocks (as compressed file) from one of mirrors (mirror will be choosen randomly by client).  MDhash of downloaded file will be hardcoded into client, to prevent changing blocks. It also mean that this code needs to be updated every few weeks (as currently is with the number of blocks),  but I think it's worth it.

What do you think about that?

Pros:
+ sync you fresh client in few minutes


Cons:
- servers with files are centralized, so in case they doesn't work,  option won't speed up


Thanks!

----
EDIT: I've run small test.

I used Matt's blockchain (thanks Matt for sharing this!!) on fresh client.
(http://bitcoin.bluematt.me/bitcoin-nightly/blockchain-nightly/)

download time: 7 minutes
bunzip2 and tar -  ~ 2 minutes
./bitcoin -rescan :  1 minute


total: 10 minutes
result:  140600 blocks done
-------
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1488236017
Hero Member
*
Offline Offline

Posts: 1488236017

View Profile Personal Message (Offline)

Ignore
1488236017
Reply with quote  #2

1488236017
Report to moderator
1488236017
Hero Member
*
Offline Offline

Posts: 1488236017

View Profile Personal Message (Offline)

Ignore
1488236017
Reply with quote  #2

1488236017
Report to moderator
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 18, 2011, 08:42:15 AM
 #2

Any constructive criticism from people who voted no ?  Thanks!
RaTTuS
Hero Member
*****
Offline Offline

Activity: 792


Bite me


View Profile
August 18, 2011, 08:57:40 AM
 #3

how about loading the blocks backwards?

In the Beginning there was CPU , then GPU , then FPGA then ASIC, what next I hear to ask ....

1RaTTuSEN7jJUDiW1EGogHwtek7g9BiEn
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 18, 2011, 09:00:25 AM
 #4

what do you mean? take only last N blocks? but then we don't have the whole history
sirk390
Full Member
***
Offline Offline

Activity: 153


View Profile
August 18, 2011, 10:19:41 AM
 #5

It would be nice but I'm not sure you know what you are talking about.
1. You can't update the code every few weeks(How ?)
2. You don't need to as the blockchain is allready "protected" against block changes.

aq
Full Member
***
Offline Offline

Activity: 238


View Profile
August 18, 2011, 10:27:29 AM
 #6

Excellent idea! This will be much needed for bitcoin to become useful for the average person.
That being said, I think you should add a -slowboot option and have the fast download as the default behavior.

Maybe you could combine the idea from RaTTuS with the download: have one thread load the block backward from the p2p network, while another thread downloads the cache files from the central servers. Now let those run until they meet each other. That could even solve the validation of the centrally stored mirror files, because if those are bogus the two chains would never match.
JoelKatz
Legendary
*
Offline Offline

Activity: 1442


Democracy is vulnerable to a 51% attack.


View Profile WWW
August 18, 2011, 10:39:26 AM
 #7

I think you're solving the wrong problem. Most of the time is not spend downloading the block chain but processing it. Take a look at your CPU usage.

I am an employee of Ripple.
1Joe1Katzci1rFcsr9HH7SLuHVnDy2aihZ BM-NBM3FRExVJSJJamV9ccgyWvQfratUHgN
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 18, 2011, 10:42:50 AM
 #8

@sirk390

1. you do can -  git push Tongue  of course I was talking only about updating md5 string in the code. (so, older clients will still work but have to download more blocks from p2p than newer clients [equal to the number of blocks between releases])

2. You don't need to as the blockchain is allready "protected" against block changes.

It's just another protection. Calculating hash is not a big deal, and I'm not sure how exactly blockchain protection works when attacker can change almost all blocks.
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 18, 2011, 10:46:50 AM
 #9

I think you're solving the wrong problem. Most of the time is not spend downloading the block chain but processing it. Take a look at your CPU usage.

Is it true also for newer blocks? I know that it's pretty fast to download first  60-70k,  but later it's getting slower and slower....


BTW. perhaps we can store already processed blocks? (as binary blob Smiley ),  of course with md5 verification?  then we get everything as fast as download Smiley
zellfaze
Full Member
***
Offline Offline

Activity: 142


Security Enthusiast


View Profile WWW
August 18, 2011, 03:40:13 PM
 #10

I think that if you just download the block chain and stick it in the bitcoin directory then the Bitcoin client doesn't check if they are valid.  It just assumes they are.

Isn't that what the -rescan option is for?  Re-validating the block-chain?  Or am I mistaken?

A+, CCENT, CCNA
Security Enthusiast
PHP Coder

Not that I expect anyone to, but should you like my post, please donate:
Donate: 1BRbfqii6Sm9tEUE8A16H7QeDmYFjyBZ7V
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 18, 2011, 05:49:33 PM
 #11

@zellfaze   hm...but don't we need to run  -rescan? If I download 140k blocks,  what about my transactions which were included in those blocks? Will they show up or I need to rescan?   
Anyway, I believe that still download + rescan is faster than p2p.
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 18, 2011, 06:08:12 PM
 #12

ok, I did the homework.

I used Matt's blockchain (thanks Matt for sharing this!!) on fresh client.
(http://bitcoin.bluematt.me/bitcoin-nightly/blockchain-nightly/)

download time: 7 minutes
bunzip2 and tar -  ~ 2 minutes
./bitcoin -rescan :  1 minute


total: 10 minutes
result:  140600 blocks done Smiley
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1036


View Profile WWW
August 19, 2011, 02:21:24 PM
 #13

-rescan does not revalidate the block chain. It only synchronizes the block chain with the wallet, adding missing transactions to it.

And indeed, most of the time is spent validating the block chain, not downloading. Before 0.3.24, larger blocks often caused disconnects when sending because of an anti-spam measure. That is/was also a cause of delay when starting up, but is slowly improving, as more nodes (and thus potential download partners) upgrade to 0.3.24.

That said, a -fastboot argument that takes a URL or filename to import a blockchain from may be useful in certain situations.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
ArtForz
Sr. Member
****
Offline Offline

Activity: 406


View Profile
August 19, 2011, 02:39:39 PM
 #14

Thinking out loud...
Do we even need to check the ECDSA signatures in blocks older than the last checkpoint?
Assuming the checkpoint hash is okay, you can't change any transactions in blocks before it without breaking the chain.
Assuming the checkpoint hash was replaced (eve hacked the devs box or something)... we're pretty much fucked anyways as the client will reject the legit chain and stay on the attackers chain.

Any input?

bitcoin: 1Fb77Xq5ePFER8GtKRn2KDbDTVpJKfKmpz
i0coin: jNdvyvd6v6gV3kVJLD7HsB5ZwHyHwAkfdw
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 19, 2011, 11:52:51 PM
 #15

And indeed, most of the time is spent validating the block chain, not downloading. Before 0.3.24, larger blocks often caused disconnects when sending because of an anti-spam measure. That is/was also a cause of delay when starting up, but is slowly improving, as more nodes (and thus potential download partners) upgrade to 0.3.24.

But here we are not validating blockchain block-by-block (which takes a lot of CPU). I just verify md5um of a file which contains 140000 blocks.  It's  much much faster.

That said, a -fastboot argument that takes a URL or filename to import a blockchain from may be useful in certain situations.

it can take argument, but I think it should also have some default mirrors hardcoded (user usually just download and run a program, and don't care about reading docs and looking for URLs, etc...). I already have 3 people who volunteered to provide mirrors, which may be enough for beta version.
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 20, 2011, 03:41:15 AM
 #16

Thinking out loud...
Do we even need to check the ECDSA signatures in blocks older than the last checkpoint?
Assuming the checkpoint hash is okay, you can't change any transactions in blocks before it without breaking the chain.
Assuming the checkpoint hash was replaced (eve hacked the devs box or something)... we're pretty much fucked anyways as the client will reject the legit chain and stay on the attackers chain.

Any input?

Hey ArtForz, I think you're right. Anyway, I think it's safer, cleaner and easier to implement this with hardcoded checksums. First, we are discovering very early that downloaded blocks are invalid (because someone changed them or there were problems with download). It allows to retry download from a different mirror as soon as md5hash is calculated.

Secondly, I'm not sure how exactly works the logic inside client (handling broken chain), and I'd prefer to not touch it, just to be on a safe side.  With checksum approach I just write a class with one method, which downloads and verifies files. This method can be run from any point of client, and the rest of the client is untouched.

Last but not least, we get another (perhaps redundant) protection with this md5sum.
casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1344


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
August 20, 2011, 03:48:44 AM
 #17

I have thought of this... in my ideal world, this is how it would be implemented.

The client has a few public keys which it trusts (developer maintained), or has a screen where trusted keys can be manipulated by the user (developer maintains the defaults but user is ultimately in charge).

At some point, the blocks themselves can contain checkpoint transactions that vouch for everything up through and including the blocks that contain them.  They don't have to be anything special, they could be nothing more than standard Bitcoin transactions involving a signature from the hardcoded key.

When the client sees such a checkpoint, it can assume that everything before it is safe.  With the checkpoints in the block chain, there is no need to keep hard-coding checkpoints in the software (or, at least, the number of hard-coded checkpoints can be vastly reduced for the same effect.)

Finally....(separate feature).... the software ought to support a sort of "flat file import" that contains a recent copy of the blockchain in "blob" format, signed the same way.  If a well-formed and validly-signed "distribution blob" is available, the client should take in everything from the blob and not ask for a single block from a peer until the entire blob has been "downloaded" (read: copied into the preferred format of the client's choice, and indexed the way the client likes it...as the "blob" shall have no indexes).  Given that, the blob should be optimized for space as much as possible (e.g. deduplication of public keys, not wasting bytes wherever possible, etc.)

Blobs, when obtained from a trusted party, could also give rise to pruned blockchains to save space.  When accepting a pruned chain, one must trust that the chain doesn't omit any transactions that would lead the client to believe that spent coins are actually unspent.  A trusted party (developer whose public key is harcoded) could be the person to sign a trusted pruned blockchain, and a third party open source utility could allow the paranoid among us to compare the signed pruned version with the full version to confirm for everyone else that it is indeed not misrepresenting the unspent status of any coins.

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper wallets instead.
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 20, 2011, 07:13:31 AM
 #18

I have thought of this... in my ideal world, this is how it would be implemented.

The client has a few public keys which it trusts (developer maintained), or has a screen where trusted keys can be manipulated by the user (developer maintains the defaults but user is ultimately in charge).


Hey casascius, I think that's a great idea. Perhaps (for the first version) I can just hardcode signatures of some people (responsible for github/bitcoin), and then - in client - just verify signatures of downloaded files. This will allow to update file with blocks every day (and because of that function should provide even faster boot), without changing client source code.

What do you guys think about hardcoding signatures instead of md5sum of first N blocks?
regis
Newbie
*
Offline Offline

Activity: 16


View Profile
August 25, 2011, 09:17:55 AM
 #19

beta version for linux:

https://github.com/kkurach/bitcoin/tree/fastboot

any comments?  (files:  src/fastboot.h  src/fastboot.cpp)
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1036


View Profile WWW
August 25, 2011, 10:23:32 AM
 #20

And indeed, most of the time is spent validating the block chain, not downloading. Before 0.3.24, larger blocks often caused disconnects when sending because of an anti-spam measure. That is/was also a cause of delay when starting up, but is slowly improving, as more nodes (and thus potential download partners) upgrade to 0.3.24.

But here we are not validating blockchain block-by-block (which takes a lot of CPU). I just verify md5um of a file which contains 140000 blocks.  It's  much much faster.

Of course it is. That's not what I meant. What you are doing is useful, but it does not have the same security guarantees. Right now, through hardcoded block hashes in the client, the maintainer has the ability to force a particular chain onto you, but that chain must still exist and be valid. If the hash of the entire block file is hardcoded in the client, the maintainer can effectively make you get an invalid block database.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!