Blockchain Rescanning

[Nattygirl]

Hi everyone,

Whenever I use the code with the line TheBDM.goOnline(), the blockchain rescans and it takes close to an hour to finish because it starts at 0 to the number of existing blocks(currently 411572).
Sometimes when I execute ArmoryQt.py and then to test the createtransaction.py script for example, because it has the line TheBDM.goOnline, it will initiate a rescan from 0 to current number of blocks. Is this because Armory runs bitcoind in the background, as i have selected in settings or is it suppose to do this?
Can't it just add the new blocks to what is existing on the blockchain instead of rescanning? Or does it have to start from zero if one
is querying the balance or creating a transaction?

Enlightenment  would really help.

Thank you.
Natty.

[1714137136]

1714137136

[1714137136]

1714137136

[goatpig]

You have to register your wallets with the BDM before hitting goOnline. If you registered new addresses it will have to rescan from scratch.

[Nattygirl]

Hi goatpig

So there is no way of optimizing it and making it take less time when it rescans?

[goatpig]

You should register addresses/wallets before you hit goOnline, that's how it is supposed to work. You are using the lazy way, which is: init db -> feed it addresses to monitor.

0.95 has a work around for that. Either use the correct init sequence or wait for 0.95.

[Nattygirl]

Hi Goatpig,

Sorry I responded after such a long time, thank you so much for responding.
Do I use this piece of code to register the wallet? -----"wlt.registerWallet(isNew=False)"---
If yes this line in code come before the goOnline but it still takes long.
Please help because the Blockchain is taking really long when I create a transaction or check the balance using the
developer's code. the create transaction script and the check balance script.

Thank you.

[goatpig]

It would be simpler if I could see your code.

[Nattygirl]

Hi goatpig

Okay here you go, this is the code used to get the balance which is in the extras folder BDMbasics_watchBalance.py, it works but takes close to an hour to return the balance because of the blockchain loading:
_______________________________________________________________________________ __
import os
import sys
sys.path.append('..') # this is for when the script is run in the extras dir
from armoryengine.ALL import *

# Check that user actually supplied a wallet file
if len(CLI_ARGS) < 1:
   print 'Must supply path to wallet file as first argument!'
   exit(1)

# Check that wallet file exists
walletPath = CLI_ARGS[0]
if not os.path.exists(walletPath):
   print 'Wallet file does not exist: "%s"' % walletPath
   exit(1)
  
# Read it into a PyBtcWallet object
wlt = PyBtcWallet().readWalletFile(walletPath)

################################################################################
def printWalletBalance(args):
   # If any args, it's because this is a NEW_BLOCK_ACTION with a new top block
   print 'Current top block:', TheBDM.getTopBlockHeight()

   # Print all three types of balances you can query for a wallet
   for balType in ['full', 'spendable', 'unconfirmed']:
      balStr  = coin2str(wlt.getBalance(balType))
      typeStr = balType.upper().rjust(16)
      print '%s balance for wallet %s: %s BTC' % (typeStr, wlt.uniqueIDB58, balStr)
  


################################################################################
#Register the BDM callback
TheBDM.RegisterEventForSignal(printWalletBalance, FINISH_LOAD_BLOCKCHAIN_ACTION)
TheBDM.RegisterEventForSignal(printWalletBalance, NEW_BLOCK_ACTION)

# Register our wallet with the BDM.
# Pass False during registration because we don't know if the wallet is new.
# The BDM will make sure the history is up to date before signaling our callback
wlt.registerWallet(isNew=False)

#Now start the BDM
TheBDM.goOnline()

'''
The BDM runs on its own thread and will signal our callback when a new event
occurs. All actions take place on a signal basis, while the main thread is
left to perform its own operations. For this reason, our main thread needs it
own loop, otherwise it would exit and shutdown our entire process. Here we have
nothing to do so we'll use an empty loop that sleeps for a second on every
iteration
'''

try:
   while(True):
      time.sleep(1)
except KeyboardInterrupt:
   exit(0)
_______________________________________________________________________________ _


The code does not immediately produce output,  even if (isNew=True)

[goatpig]

you using this on 0.94? Have you dedicated a DB to this use case?

[Nattygirl]

Yes I am using it on 0.94,
I don't understand the question of dedicating a database?
Though I did download the blockchain when I downloaded bitcoinqt and it exists locally.
Can't I load the blockchain as a global variable, so it doesn't have to load every time?

[goatpig]

I'm talking about the Armory DB, not the blockchain. Are you running this sample code with its own DB?

[Nattygirl]

Hi goatpig

Is the Armory db  in the .armory folder. Because in this folder I have a database directory but inside I have the following:

blkdata       headers       history       ssh       stxo       subssh       txhints       zeroconf
blkdata-lock  headers-lock  history-lock  ssh-lock  stxo-lock  subssh-lock  txhints-lock  zeroconf-lock

[Carlton Banks]

Hi goatpig

Is the Armory db  in the .armory folder. Because in this folder I have a database directory but inside I have the following:

blkdata       headers       history       ssh       stxo       subssh       txhints       zeroconf
blkdata-lock  headers-lock  history-lock  ssh-lock  stxo-lock  subssh-lock  txhints-lock  zeroconf-lock

That's the right directory, and the right list of files that comprises the Db, I believe.

[Nattygirl]

Any idea why it has to load the database for an hour when I create a transaction or check the balance?

[goatpig]

Any idea why it has to load the database for an hour when I create a transaction or check the balance?

It you modify address sets in between inits it will rescan from scratch:

You got wallet A and B.

You init the db against wallet A. That's a fresh scan.

You then shutdown Armory and restart with wallet B instead of wallet A. This is again a fresh scan, first time it saw that wallet.

Now you redo this with wallet A, it will start with a fresh scan again because the address sets have changed and you have just registered addresses that are out of sync with the last good DB state.

Same will happen if you scan A, shutdown, extend A's address chain offline, then restart with the longer A address set. It will scan from the start.

This is why I asked about the databases you are using. Seeing you are not familiar with the architecture, I'd suggest you look at the DB code starting here:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/DatabaseBuilder.cpp#L28

[Nattygirl]

Thanks goatpig

Okay let me do that and familiarise myself more.

[Nattygirl]

Hi Goatpig

As promised I familiarized myself with the architecture, and I am just wondering if there is a way then to speed up the
time it takes to do a fresh scan?
Currently it takes close to an hour.

Thank you for your help.

[goatpig]

Hi Goatpig

As promised I familiarized myself with the architecture, and I am just wondering if there is a way then to speed up the
time it takes to do a fresh scan?
Currently it takes close to an hour.

Thank you for your help.

0.95 will fix your issue 100%. You should help testing with that.

[Nattygirl]

Alright,
 I will try that instead but I noticed on your website ( https://www.bitcoinarmory.com/using-armory-python/ ) you have:
"Note that, unless you have built the Armory databases with the –supernode option, Armory will automatically initiate a rescan if there are any addresses in the wallet not there in previous loads" does this still hold?

[goatpig]

Alright,
 I will try that instead but I noticed on your website ( https://www.bitcoinarmory.com/using-armory-python/ ) you have:
"Note that, unless you have built the Armory databases with the –supernode option, Armory will automatically initiate a rescan if there are any addresses in the wallet not there in previous loads" does this still hold?

That's information that holds true for 0.93, the latest version ATI supports. This project is a fork of ATI's work, and 0.94 then 0.95 come with nuances to that old statement. Notably in your case, 0.95 will not rescan.

0.95 keeps constant record of any addresses you show it, as opposed to previous versions that only keep up to date with addresses they are loaded with. This change is due to the database being it's own binary starting 0.95, which means it isn't necessarily loaded with any wallet data at init.

In order to prevent constant rescanning from detached db inits, the db address list is now append only. In your case it simply means it fixes your issue 100%

[Nattygirl]

Hi Goatpig,

That is awesome, okay will be getting it today from git, will let you know how it goes.

Thank you.