Bitcoin Forum
November 22, 2017, 01:51:00 PM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: what is -checkblocks for, and why does it default so high?  (Read 3024 times)
dooglus
Legendary
*
Offline Offline

Activity: 2352



View Profile
February 05, 2013, 12:10:16 AM
 #1

I notice in the current git version of bitcoin-qt it sets -checkblocks to 2500 by default, which results in:

  "Verifying last 2500 blocks at level 1"

each time I start bitcoin-qt.

What is that for?  Why do blocks need verifying?  Nothing writes blocks to disk except bitcoin-qt, so what's this extra verification step for?  Is it perhaps like a 'rescan', looking for new transactions which may not be in my wallet yet?

Just-Dice                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   Play or Invest                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   1% House Edge
1511358660
Hero Member
*
Offline Offline

Posts: 1511358660

View Profile Personal Message (Offline)

Ignore
1511358660
Reply with quote  #2

1511358660
Report to moderator
1511358660
Hero Member
*
Offline Offline

Posts: 1511358660

View Profile Personal Message (Offline)

Ignore
1511358660
Reply with quote  #2

1511358660
Report to moderator
1511358660
Hero Member
*
Offline Offline

Posts: 1511358660

View Profile Personal Message (Offline)

Ignore
1511358660
Reply with quote  #2

1511358660
Report to moderator
Join ICO Now Coinlancer is Disrupting the Freelance marketplace!
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1511358660
Hero Member
*
Offline Offline

Posts: 1511358660

View Profile Personal Message (Offline)

Ignore
1511358660
Reply with quote  #2

1511358660
Report to moderator
1511358660
Hero Member
*
Offline Offline

Posts: 1511358660

View Profile Personal Message (Offline)

Ignore
1511358660
Reply with quote  #2

1511358660
Report to moderator
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
February 05, 2013, 12:16:03 AM
 #2

"Nothing writes blocks to disk except bitcoin-qt"

Think from a security point of view that is a good assumption to make?  The blockchain is simply an unprotected file on the file system.  Would be downright trivial to design a malware which replaces it with bogus transactions.

By going back x blocks it requires the attacker to produce x blocks.  2500 blocks at current difficulty is beyond the ability of all but the most determined attackers.  They simply can't replace history.
dooglus
Legendary
*
Offline Offline

Activity: 2352



View Profile
February 05, 2013, 12:48:52 AM
 #3

"Nothing writes blocks to disk except bitcoin-qt"

Think from a security point of view that is a good assumption to make?  The blockchain is simply an unprotected file on the file system.

So are and ~/.bitcoin/bitcoin.conf.  Once the attacker is has the ability to write to the local hard drive there's not a whole lot we can do to protect ourselves.  If I'm malware rewriting the blockchain, don't I just add a "checkblocks=1" to bitcoin.conf so my change isn't detected?

Just-Dice                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   Play or Invest                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   1% House Edge
Peter Todd
Legendary
*
expert
Offline Offline

Activity: 1106


View Profile
February 05, 2013, 01:53:42 AM
 #4

FWIW the default has been reduced to 288 blocks (2 days) for 0.8: https://github.com/bitcoin/bitcoin/pull/2222/files

dooglus
Legendary
*
Offline Offline

Activity: 2352



View Profile
February 05, 2013, 02:27:14 AM
 #5

FWIW the default has been reduced to 288 blocks (2 days) for 0.8: https://github.com/bitcoin/bitcoin/pull/2222/files

So is it a security feature primarily?

I was wondering if it might be doing a 'rescan', looking through recent blocks for transactions which aren't in the wallet.dat, but I did a test that seems to suggest rescan is no longer needed.  I sent coins from one wallet to another, waited for the transaction to get 2 confirmations, closed bitcoin-qt, switched the wallet.dat out and replaced it with the recipient wallet, restarted bitcoin-qt with -checkblocks=1 and the transaction showed up in the recipient wallet immediately.

Is that a feature of the new leveldb stuff?

Just-Dice                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   Play or Invest                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   1% House Edge
deepceleron
Legendary
*
Offline Offline

Activity: 1512



View Profile WWW
February 05, 2013, 03:53:24 AM
 #6

FWIW the default has been reduced to 288 blocks (2 days) for 0.8: https://github.com/bitcoin/bitcoin/pull/2222/files

So is it a security feature primarily?

I was wondering if it might be doing a 'rescan', looking through recent blocks for transactions which aren't in the wallet.dat, but I did a test that seems to suggest rescan is no longer needed.  I sent coins from one wallet to another, waited for the transaction to get 2 confirmations, closed bitcoin-qt, switched the wallet.dat out and replaced it with the recipient wallet, restarted bitcoin-qt with -checkblocks=1 and the transaction showed up in the recipient wallet immediately.

Is that a feature of the new leveldb stuff?

The wallet contains a value called "bestblock", which contains the last block seen when the wallet was last used. When starting with an older wallet than blockchain, Bitcoin knows that it needs to look for any wallet transactions after that block height.

Since Bitcoin may previously have crashed or been terminated unceremoniously while it was writing to the blockchain, the startup check ensures that data is in a reasonable state. The default level should really be bumped up to 2:

Quote
-checklevel specifies how thorough the verification must be:
0: only check whether the block exists on disk
1: verify block validity (default)
2: verify transaction index validity
3: check transaction hashes
4: check whether spent txouts were spent within the main chain
5: check whether all prevouts are marked spent
6: check whether spent txouts were spent by a valid transaction that consume them

Code is self-documenting.

Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
February 05, 2013, 09:05:16 AM
 #7

I notice in the current git version of bitcoin-qt it sets -checkblocks to 2500 by default, which results in:

  "Verifying last 2500 blocks at level 1"

each time I start bitcoin-qt.

What is that for?  Why do blocks need verifying?  Nothing writes blocks to disk except bitcoin-qt, so what's this extra verification step for?  Is it perhaps like a 'rescan', looking for new transactions which may not be in my wallet yet?

-rescan is a wallet option, which rescans the block chain for transactions missing from the wallet. It shouldn't be necessary in normal operation since 0.3.21 (since then, wallet auto rescan from the last block they 'saw').

The primary reason for -checkblocks (a feature that has existed for as long as I remember, but has only been made configurable in 0.6) is preventing accidental disk corruption of the block chain file, which could result in rejecting the best chain. It is certainly not a protection against an attacker that is able to write to your block files (though it does make such an attack harder).

Also note that both the meaning of the check levels as the default number of blocks to scan has been changed in git head (only 288 blocks, but with a far more thorough check). This is safe, as block data corruption is no longer a chain forking risk in the new database layout (only the unspent transaction output database, aka coins database, aka chainstate matters).

Here's an explanation of the new -checklevel meanings (for 0.8 ):
  • 0: Validate all block headers + compare (by hash) to blocks on disk for the last -checkblocks blocks
  • 1: In addition, verify (standalong) validity of those -checkblocks blocks
  • 2: In addition, verify that undo data matches checksums
  • 3: In addition, check that the current chainstate matches can reasonably be the result of the last N blocks, where N is limited by both -checkblocks and the amount of in-memory cache (-dbcache); typically it's around 150.
  • 4: In addition, for the last N blocks (see above), do full validation (including signature checks).
Note that the new default (3) is in fact a stronger check than the previous highest level (6).

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
K1773R
Legendary
*
Offline Offline

Activity: 1680


/dev/null


View Profile
February 05, 2013, 09:21:58 AM
 #8

"Nothing writes blocks to disk except bitcoin-qt"

Think from a security point of view that is a good assumption to make?  The blockchain is simply an unprotected file on the file system.

So are and ~/.bitcoin/bitcoin.conf.  Once the attacker is has the ability to write to the local hard drive there's not a whole lot we can do to protect ourselves.  If I'm malware rewriting the blockchain, don't I just add a "checkblocks=1" to bitcoin.conf so my change isn't detected?

if your sure your blockchain wont get corrupted (RAID with brtfs for example) u can sest it to 1 without any problems.

[GPG Public Key]  [Devcoin Builds]  [BBQCoin Builds]  [Multichain Blockexplorer]  [Multichain Blockexplorer - PoS Coins]  [Ufasoft Miner Linux Builds]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
February 05, 2013, 09:32:55 AM
 #9

if your sure your blockchain wont get corrupted (RAID with brtfs for example) u can sest it to 0 without any problems.

-checkblocks=0 actually means "check all blocks" (because of compatibility with the pre-0.6 -checkblocks (without parameter) that checked everything instead of just the last 2500 blocks).

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
K1773R
Legendary
*
Offline Offline

Activity: 1680


/dev/null


View Profile
February 05, 2013, 09:35:53 AM
 #10

if your sure your blockchain wont get corrupted (RAID with brtfs for example) u can sest it to 0 without any problems.

-checkblocks=0 actually means "check all blocks" (because of compatibility with the pre-0.6 -checkblocks (without parameter) that checked everything instead of just the last 2500 blocks).
ouch, edited my post! ty

[GPG Public Key]  [Devcoin Builds]  [BBQCoin Builds]  [Multichain Blockexplorer]  [Multichain Blockexplorer - PoS Coins]  [Ufasoft Miner Linux Builds]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
dooglus
Legendary
*
Offline Offline

Activity: 2352



View Profile
February 05, 2013, 06:37:55 PM
 #11

-rescan is a wallet option, which rescans the block chain for transactions missing from the wallet. It shouldn't be necessary in normal operation since 0.3.21 (since then, wallet auto rescan from the last block they 'saw').

Does this feature know about blockchain reorgs?

For example, the wallet knows it has seen up to block 10000, but since I last loaded the wallet up the blockchain was reorganised, with 'better' blocks 9999 and 10000 replacing the ones my wallet scanned.  Does the wallet just remember "10000" as its bestblock, or does it remember the actual block hash that it last saw?

Just-Dice                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   Play or Invest                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   1% House Edge
K1773R
Legendary
*
Offline Offline

Activity: 1680


/dev/null


View Profile
February 05, 2013, 07:31:47 PM
 #12

-rescan is a wallet option, which rescans the block chain for transactions missing from the wallet. It shouldn't be necessary in normal operation since 0.3.21 (since then, wallet auto rescan from the last block they 'saw').

Does this feature know about blockchain reorgs?

For example, the wallet knows it has seen up to block 10000, but since I last loaded the wallet up the blockchain was reorganised, with 'better' blocks 9999 and 10000 replacing the ones my wallet scanned.  Does the wallet just remember "10000" as its bestblock, or does it remember the actual block hash that it last saw?
it will first load, afterwards it will see the blockchain "fork" and remove the orphaned blocks. in this case it will check from block 10000-2500 to 10000 and validate each of em, if invalid -> redownload, invalid -> just start as usual.

[GPG Public Key]  [Devcoin Builds]  [BBQCoin Builds]  [Multichain Blockexplorer]  [Multichain Blockexplorer - PoS Coins]  [Ufasoft Miner Linux Builds]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
dooglus
Legendary
*
Offline Offline

Activity: 2352



View Profile
February 05, 2013, 08:34:53 PM
 #13

it will first load, afterwards it will see the blockchain "fork" and remove the orphaned blocks. in this case it will check from block 10000-2500 to 10000 and validate each of em, if invalid -> redownload, invalid -> just start as usual.

I'm talking about the case where I have two different wallet.dat files, but only one copy of the blockchain.  I rename the wallet.dat files while bitcoin-qt is shut down.

When the fork happened, I had the other wallet loaded.  When I switch to the first wallet, will it notice that a fork has occurred and act accordingly?

Just-Dice                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   Play or Invest                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   1% House Edge
K1773R
Legendary
*
Offline Offline

Activity: 1680


/dev/null


View Profile
February 05, 2013, 09:35:04 PM
 #14

it will first load, afterwards it will see the blockchain "fork" and remove the orphaned blocks. in this case it will check from block 10000-2500 to 10000 and validate each of em, if invalid -> redownload, invalid -> just start as usual.

I'm talking about the case where I have two different wallet.dat files, but only one copy of the blockchain.  I rename the wallet.dat files while bitcoin-qt is shut down.

When the fork happened, I had the other wallet loaded.  When I switch to the first wallet, will it notice that a fork has occurred and act accordingly?
if its later than 2500 blocks, then i dont think it will. but you can fix this by using -rescan

[GPG Public Key]  [Devcoin Builds]  [BBQCoin Builds]  [Multichain Blockexplorer]  [Multichain Blockexplorer - PoS Coins]  [Ufasoft Miner Linux Builds]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
deepceleron
Legendary
*
Offline Offline

Activity: 1512



View Profile WWW
February 05, 2013, 09:47:04 PM
 #15

or does it remember the actual block hash that it last saw?

The data looks like this:
{
    "bestblock": "00000000000004cce890839ecc7a680b095dd28cce99f964bcfadcd43e4be389",



Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
February 06, 2013, 03:25:26 PM
 #16

Does this feature know about blockchain reorgs?

Yes, it does. It uses the same mechanism the block synchronization mechanism uses to find forks: the wallet stores a series of hashes: the last one, the 9 direct ancestors of the last one, and then the hashes of blocks with exponentially increasing distance back.


aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
dooglus
Legendary
*
Offline Offline

Activity: 2352



View Profile
February 06, 2013, 07:59:15 PM
 #17

Yes, it does. It uses the same mechanism the block synchronization mechanism uses to find forks: the wallet stores a series of hashes: the last one, the 9 direct ancestors of the last one, and then the hashes of blocks with exponentially increasing distance back.

I was trying to get my head around that code yesterday after I asked.  Thanks for explaining it.

Just-Dice                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   Play or Invest                 ██             
          ██████████         
      ██████████████████     
  ██████████████████████████ 
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
██████████████████████████████
    ██████████████████████   
        ██████████████       
            ██████           
   1% House Edge
gmaxwell
Moderator
Legendary
*
qt
Offline Offline

Activity: 2338



View Profile
February 07, 2013, 05:49:01 AM
 #18

if your sure your blockchain wont get corrupted (RAID with brtfs for example) u can sest it to 1 without any problems.
That doesn't make you sure it won't be corrupted. No widely used filesystems give you atomic operations across multiple files (and even if any did— we're not invoking them).  Especially with 0.8 introducing a brand new potentially warty storage engine it's good to have the checking though turning it down is way better than turning it off— most of the benefit comes from checking the most recent blocks.


Bitcoin will not be compromised
K1773R
Legendary
*
Offline Offline

Activity: 1680


/dev/null


View Profile
February 07, 2013, 08:02:02 AM
 #19

if your sure your blockchain wont get corrupted (RAID with brtfs for example) u can sest it to 1 without any problems.
That doesn't make you sure it won't be corrupted. No widely used filesystems give you atomic operations across multiple files (and even if any did— we're not invoking them).  Especially with 0.8 introducing a brand new potentially warty storage engine it's good to have the checking though turning it down is way better than turning it off— most of the benefit comes from checking the most recent blocks.
i got my checkblocks set @ 100, 2.5k is imho overkill since its such a long timerange.

[GPG Public Key]  [Devcoin Builds]  [BBQCoin Builds]  [Multichain Blockexplorer]  [Multichain Blockexplorer - PoS Coins]  [Ufasoft Miner Linux Builds]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
mmortal03
Legendary
*
Offline Offline

Activity: 1606


View Profile
February 25, 2016, 07:12:05 AM
 #20

3: In addition, check that the current chainstate matches can reasonably be the result of the last N blocks, where N is limited by both -checkblocks and the amount of in-memory cache (-dbcache); typically it's around 150.

So, if you raise the value of dbcache, does it lengthen the checkblocks process at startup, assuming that the value of checkblocks stays constant? If it does, this would seem to be a counter-intuitive result.
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!