CppnotificationHandler problem

[marekNowak]

Hello,

I have c.a 25000 addresses in my watching only wallet. I start armoryd with it. Daemon runs properly.

There is problem with notifications, esepcially with NEW_ZC_ACTION in handleCppNotification()

I recieve 50 new transfers on my wallet (in 10 minutes) - each with different BTC address and in log there is information only about couple of them.
I have found out that only 10% of transfers are visible (logged - which mean the handler is not called (?) ). The same situation with outgoing transfer:

(WARNING) armoryd.py:3173 - Active wallet is set to Y49z82qM
(INFO) armoryd.py:3181 - Initialising RPC server on port 8225
(WARNING) armoryd.py:3586 - Server started...
(WARNING) armoryd.py:3596 - Registering wallet: Y49z82qM
BDM is ready!
New Block:  445336
(INFO) armoryd.py:3299 - New Block! : 445336
(INFO) armoryd.py:3265 - New ZC tx: 2393870
(INFO) armoryd.py:3265 - New ZC tx: 50000
New Block:  445337
(INFO) armoryd.py:3299 - New Block! : 445337
New Block:  445338
(INFO) armoryd.py:3299 - New Block! : 445338
(INFO) armoryd.py:3265 - New ZC tx: 48700
New Block:  445339
(INFO) armoryd.py:3299 - New Block! : 445339
(INFO) armoryd.py:3265 - New ZC tx: 7960018
New Block:  445340
(INFO) armoryd.py:3299 - New Block! : 445340
New Block:  445341
(INFO) armoryd.py:3299 - New Block! : 445341
New Block:  445342
(INFO) armoryd.py:3299 - New Block! : 445342


What may be the reason. All new block are logged correctly (above).

I use armoryd with 0.95.1 version.

Can someone help me?


M.

[1715116955]

1715116955

[1715116955]

1715116955

[goatpig]

Are you using several wallets? In armoryd, you need to select a wallet to be active in order to fetch data for it. At any rate, since I've taken over Armory alone, I have neglected armoryd. I intent to go over it for the upcoming version. Feel free to drop by the IRC channel (#bitcoin-armory on freenode) or create a issue on the github page to go over the bugs you have encountered and I'll make sure to fix them for 0.96.

[marekNowak]

I have only one wallet and its active.

m.

[goatpig]

Let me see the whole log file then.

[marekNowak]

(INFO) ArmoryUtils.py:1137 - C++ block utilities loaded successfully
(INFO) ArmoryUtils.py:651 - Executing popen: free -m
(INFO) ArmoryUtils.py:651 - Executing popen: ['cat', '/proc/cpuinfo']
(INFO) ArmoryUtils.py:1247 -
(INFO) ArmoryUtils.py:1248 -
(INFO) ArmoryUtils.py:1249 -
(INFO) ArmoryUtils.py:1250 - ************************************************************
(INFO) ArmoryUtils.py:1251 - Invoked: /home/armory/BitcoinArmory-0.95.1/armoryd.py --debug
(INFO) ArmoryUtils.py:1252 - ************************************************************
(INFO) ArmoryUtils.py:1253 - Loading Armory Engine:
(INFO) ArmoryUtils.py:1254 -    Armory Version        : 0.95.1
(INFO) ArmoryUtils.py:1255 -    Armory Build:         : None
(INFO) ArmoryUtils.py:1256 -    PyBtcWallet  Version  : 1.35
(INFO) ArmoryUtils.py:1257 - Detected Operating system: Linux
(INFO) ArmoryUtils.py:1258 -    OS Variant            : debian-8.6-
(INFO) ArmoryUtils.py:1259 -    User home-directory   : /home/armory
(INFO) ArmoryUtils.py:1260 -    Satoshi BTC directory : /home/armory/.bitcoin/
(INFO) ArmoryUtils.py:1261 -    Armory home dir       : /home/armory/.armory/
(INFO) ArmoryUtils.py:1262 - Detected System Specs    :
(INFO) ArmoryUtils.py:1263 -    Total Available RAM   : 47.26 GB
(INFO) ArmoryUtils.py:1264 -    CPU ID string         : Intel(R) Core(TM) i7 CPU         950  @ 3.07GHz
(INFO) ArmoryUtils.py:1265 -    Number of CPU cores   : 8 cores
(INFO) ArmoryUtils.py:1266 -    System is 64-bit      : True
(INFO) ArmoryUtils.py:1267 -    Preferred Encoding    : UTF-8
(INFO) ArmoryUtils.py:1268 -    Machine Arch          : x86_64
(INFO) ArmoryUtils.py:1269 -    Available HDD (ARM)   : 69 GB
(INFO) ArmoryUtils.py:1270 -    Available HDD (BTC)   : 69 GB
(INFO) ArmoryUtils.py:1271 -
(INFO) ArmoryUtils.py:1272 - Network Name: Main Network
(INFO) ArmoryUtils.py:1273 - Satoshi Port: 8333
(INFO) ArmoryUtils.py:1274 - Do wlt check: True
(INFO) ArmoryUtils.py:1275 - Named options/arguments to armoryengine.py:
(INFO) ArmoryUtils.py:1277 -     thread_count    : -1
(INFO) ArmoryUtils.py:1277 -     rescan          : False
(INFO) ArmoryUtils.py:1277 -     ignoreAllZC     : False
(INFO) ArmoryUtils.py:1277 -     rescanBalance   : False
(INFO) ArmoryUtils.py:1277 -     disableModules  : False
(INFO) ArmoryUtils.py:1277 -     port            : None
(INFO) ArmoryUtils.py:1277 -     interport       : 8223
(INFO) ArmoryUtils.py:1277 -     skipStatsReport : False
(INFO) ArmoryUtils.py:1277 -     forceWalletCheck: False
(INFO) ArmoryUtils.py:1277 -     regtest         : False
(INFO) ArmoryUtils.py:1277 -     rebuild         : False
(INFO) ArmoryUtils.py:1277 -     nettimeout      : 2
(INFO) ArmoryUtils.py:1277 -     datadir         : DEFAULT
(INFO) ArmoryUtils.py:1277 -     clearMempool    : False
(INFO) ArmoryUtils.py:1277 -     offline         : False
(INFO) ArmoryUtils.py:1277 -     coverageOutputDir: None
(INFO) ArmoryUtils.py:1277 -     armoryDBDir     : DEFAULT
(INFO) ArmoryUtils.py:1277 -     armorydb_port   : 9001
(INFO) ArmoryUtils.py:1277 -     satoshiPort     : DEFAULT
(INFO) ArmoryUtils.py:1277 -     useTorSettings  : False
(INFO) ArmoryUtils.py:1277 -     netlog          : False
(INFO) ArmoryUtils.py:1277 -     keypool         : 100
(INFO) ArmoryUtils.py:1277 -     coverageInclude : None
(INFO) ArmoryUtils.py:1277 -     forceOnline     : False
(INFO) ArmoryUtils.py:1277 -     skipAnnounceCheck: False
(INFO) ArmoryUtils.py:1277 -     redownload      : False
(INFO) ArmoryUtils.py:1277 -     armorydb_ip     : 127.0.0.1
(INFO) ArmoryUtils.py:1277 -     multisigFile    : DEFAULT
(INFO) ArmoryUtils.py:1277 -     ram_usage       : -1
(INFO) ArmoryUtils.py:1277 -     testAnnounceCode: False
(INFO) ArmoryUtils.py:1277 -     mtdebug         : False
(INFO) ArmoryUtils.py:1277 -     logDisable      : False
(INFO) ArmoryUtils.py:1277 -     settingsPath    : /home/armory/.armory/ArmorySettings.txt
(INFO) ArmoryUtils.py:1277 -     db_type         : DB_FULL
(INFO) ArmoryUtils.py:1277 -     doDebug         : True
(INFO) ArmoryUtils.py:1277 -     enableDetSign   : True
(INFO) ArmoryUtils.py:1277 -     disableConfPermis: False
(INFO) ArmoryUtils.py:1277 -     testnet         : False
(INFO) ArmoryUtils.py:1277 -     rpcport         : DEFAULT
(INFO) ArmoryUtils.py:1277 -     satoshiHome     : DEFAULT
(INFO) ArmoryUtils.py:1277 -     satoshiRpcport  : DEFAULT
(INFO) ArmoryUtils.py:1277 -     logFile         : /home/armory/.armory/armoryd.py.log.txt
(INFO) ArmoryUtils.py:1277 -     verbosity       : None
(INFO) ArmoryUtils.py:1278 - Other arguments:
(INFO) ArmoryUtils.py:1281 - ************************************************************
(INFO) ArmoryUtils.py:1684 - C++ block utilities loaded successfully
(INFO) BDM.py:367 - Using the asynchronous/multi-threaded BlockDataManager.
(INFO) BDM.py:368 - Blockchain operations will happen in the background.
(INFO) BDM.py:369 - Devs: check TheBDM.getState() before asking for data.
(INFO) BDM.py:370 - Registering addresses during rescans will queue them for
(INFO) BDM.py:371 - inclusion after the current scan is completed.
(INFO) armoryd.py:3614 - No other armoryd.py instance is running.  We're the first. 8225
(WARNING) armoryd.py:3072 - ************************************************************************
(WARNING) armoryd.py:3074 - * Please note that armoryd v0.95.1 is beta software and is still in
(WARNING) armoryd.py:3075 - * development. Whenever applicable, the interface is designed to match
(WARNING) armoryd.py:3076 - * that of bitcoind, with function parameters and return values closely
(WARNING) armoryd.py:3077 - * matching those of bitcoind. Despite this, the function parameters and
(WARNING) armoryd.py:3078 - * return values may change, both for ported bitcoind function and
(WARNING) armoryd.py:3079 - * Armory-specific functions.
(WARNING) armoryd.py:3080 - ************************************************************************
(WARNING) armoryd.py:3081 -
(WARNING) armoryd.py:3082 - ********************************************************************************
(WARNING) armoryd.py:3084 - * WARNING!  WALLET FILE ACCESS IS NOT INTERPROCESS-SAFE!
(WARNING) armoryd.py:3085 - *           DO NOT run armoryd at the same time as ArmoryQt if
(WARNING) armoryd.py:3086 - *           they are managing the same wallet file.  If you want
(WARNING) armoryd.py:3087 - *           to manage the same wallet with both applications
(WARNING) armoryd.py:3088 - *           you must make a digital copy/backup of the wallet file
(WARNING) armoryd.py:3089 - *           into another directory and point armoryd at that one.
(WARNING) armoryd.py:3090 - *
(WARNING) armoryd.py:3091 - *           As long as the two processes do not share the same
(WARNING) armoryd.py:3092 - *           actual file, there is no risk of wallet corruption.
(WARNING) armoryd.py:3093 - *           Just be aware that addresses may end up being reused
(WARNING) armoryd.py:3094 - *           if you execute transactions at approximately the same
(WARNING) armoryd.py:3095 - *           time with both apps.
(WARNING) armoryd.py:3096 - *
(WARNING) armoryd.py:3097 - ********************************************************************************
(WARNING) armoryd.py:3098 -
(INFO) ArmoryUtils.py:3597 - Using settings file: /home/armory/.armory/ArmorySettings.txt
(WARNING) armoryd.py:3135 - No lockboxes were loaded.
(DEBUG) ArmoryUtils.py:1107 - /home/armory/.armory/databases is a directory.
(INFO) armoryd.py:3155 - Number of wallets read in: 1
(INFO) armoryd.py:3160 -    Wallet (Y49z82qM):    "a_1000 (Watch)                  "   (No Encryption)
(INFO) armoryd.py:3164 - Number of lockboxes read in: 0
(WARNING) armoryd.py:3173 - Active wallet is set to Y49z82qM
(INFO) armoryd.py:3181 - Initialising RPC server on port 8225
(WARNING) armoryd.py:3586 - Server started...
(WARNING) armoryd.py:3596 - Registering wallet: Y49z82qM
BDM is ready!
New Block:  445384
(INFO) armoryd.py:3299 - New Block! : 445384

[goatpig]

Do you get the full history by loading the wallet in ArmoryQt?

[marekNowak]

I am not using ArmoryQt on online machine. I have only shell console - its remote server.

m.

[goatpig]

Ok, what about fetching missing history by address?

[marekNowak]

which function should I use to get this data?

m.

[marekNowak]

was this  TheBDM.bdv().getUnspentTxoutsForAddr160List(scrAddrList, IGNOREZC) deprecated in 0.95.1 version ? Which should I use instead ?

m.

[goatpig]

was this  TheBDM.bdv().getUnspentTxoutsForAddr160List(scrAddrList, IGNOREZC) deprecated in 0.95.1 version ? Which should I use instead ?

m.

It's not that they are deprecated, rather that armoryd has not received much attention since 0.93. You should use the wallet methods instead of relying directly on the BDV:

https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/PyBtcWallet.py#L443

[marekNowak]

Fine,so I have an address with some unconfirmed transactions. This code return 0.0 balance  when I run:

      self.curWlt.getAddrDataFromDB()
      self.curWlt.getBalancesAndCountFromDB()
      atype,a160 = addrStr_to_hash160("1Ne9KrmTLuePACNXND29j3exxxxxxxxx")
      print self.curWlt.getAddrBalance(a160,"unconf")

also getFullUTXOList return only inputs with at least 1 confirmation from the wallet.

How to make it to return also unconfirmed tx data?

m.

[goatpig]

Fine,so I have an address with some unconfirmed transactions. This code return 0.0 balance  when I run:

      self.curWlt.getAddrDataFromDB()
      self.curWlt.getBalancesAndCountFromDB()
      atype,a160 = addrStr_to_hash160("1Ne9KrmTLuePACNXND29j3exxxxxxxxx")
      print self.curWlt.getAddrBalance(a160,"unconf")

Try to get the address like this instead:

https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/PyBtcWallet.py#L3217

You can then use all public methods of the ScrAddrObj class:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/SwigClient.h#L82

These are available to Python through SWIG.

also getFullUTXOList return only inputs with at least 1 confirmation from the wallet.

Armory was never meant to let you spend ZC (only your own ZC change). This is because Armory was developed as a GUI wallet. armoryd was donated by a user, ATI only maintained it because it got some traction with our more advanced users.

I do intent to change that to allow for CPFP and RBF (basically adding double spend features in the expert GUI) in the upcoming version. Under the hood, the methods will let you fetch ZC in bulk.

If you want to fetch ZC right now, you will have to go through a few hops:

1) Fetch the wallet's tx ledger (or the global ledger). These are the getHistoryPage methods.

You can fetch that stuff straight from wallet object:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/SwigClient.h#L115

Or you can use LedgerDelegates, which let you combine several wallets for the DB to output the sorted tx history ledger. Delegates are fairly easy to use, you can see the definition here:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/SwigClient.h#L55

And an example of how to instantiate them here:

https://github.com/goatpig/BitcoinArmory/blob/master/ArmoryQt.py#L6708

Basically, to get the global ledger delegate (for all wallets), you call TheBDM.bdv().getLedgerDelegateForWallets() on a valid bdv object.

2) For ZC, you only ever want to fetch page 0. That page always exists so this call will never fail if the object you are calling it on is valid. All ZC are always prepended to the top of the first history page. You can tell ZC apart by their height, which is always UINT32_MAX (2^32 -1).

Keep in mind that you can also get the history ledger for a given address this way:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/SwigClient.h#L301

Here's an example in code:

https://github.com/goatpig/BitcoinArmory/blob/master/qtdialogs.py#L3688

3) If you want more data, you can get the whole tx. Ledger entries come with txhashes, you can use the txhash to fetch the whole tx with this method:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/SwigClient.h#L311

--------------------

All these methods I am listing are in SwigClient.h, which gets SWIG'd (ie, it is C++ code made available to Python through the use of SWIG). SWIG only parses public members/methods, so only those are made available. If you want to read some private members, usually there will be a public method to do that (instead of just accessing it at is).

SWIG doesn't modify naming, so as long as you instantiate a SWIG'd class, you can use it as a native Python class.

Let me know if you need anything else. If you have some patience (maybe a lot =O), I'm confident I can get armoryd back to a place where you can build your applications around it.

[marekNowak]

Bitcoind 0.13.1 (txindex=1)
Armory 0.95.1
Watchingonly wallet (25000 addresses) - wallet generated with 0.93.1


I have tested function provided by You and I have couple of question:

My code:

 tab =self.curWlt.cppWallet.getHistoryPage(0)

      for tx in tab:
          if tx.getValue()>0:
              print tx.getValue()
              print '\t', binary_to_hex(tx.getTxHash(),BIGENDIAN)
              print '\t', time.ctime(int(tx.getTxTime()))
              addr = tx.getScrAddrList()
              for a160 in addr:
                 print '\t',hash160_to_addrStr(a160)


Belowe some LedgerElelementDatas - output from above code:


300000
        47a99e5e1e58b2c89b604bd6d1da634e4815f527eca2a761d631d7f393a47d22
        Fri Dec 30 09:33:20 2016
11477731
        23f65038e91b7bee1b4f6d734ba848708a19d3c4ab17aeb538a7c95ebbde7ada
        Fri Dec 30 09:33:20 2016
26687823
        1b58c8db6e190f78597ff82985b7c2ba2b00e5b0c017e55c086fad1671082158
        Fri Dec 30 09:33:20 2016
2588182
        819b13d1431940e172c21cdce5c793e8d7f44c2da97d530f7725f704e1bc27ac
        Fri Dec 30 09:33:20 2016


Question 1) Why getScrAddrList() return always empty list? How can I get information aboute addresses (source and destination) from each transaction.

Question 2) getHistoryPage(0) returns:
                       1 unconfirmed transaction when I have 3-6 unconfirmed transaction in wallet
                       4 unconfirmed transatcions when I have 50 unconfirmed transactoin in wallet (I am sending 50 transaction from one address to different addresses in wallet or I am sending out 50 transactions from wallet - each from different address to one destination address).

Is it  a bug? Why all unconfirmed transactions are not visible in history? Can You investigate it or verify whether the problem exists.


I see that getHistoryPage calls c++ code so the problem is rather not in python layer.


  

[goatpig]

Is it  a bug? Why all unconfirmed transactions are not visible in history? Can You investigate it or verify whether the problem exists.

What does the global ledger return? If you are missing transactions there too, I'd strongly suggest you try a rebuild & rescan. This looks like a case of DB desync to me so far.

Why getScrAddrList() return always empty list? How can I get information aboute addresses (source and destination) from each transaction.

After investigation, this call was deprecated. While the list is being constructed on the DB side, the underlying data isn't passed along over the socket. That call was only used to resolve address comments in an expensive way on the Python side, so I gutted it.

In other parts of the code, Tx details are grabbed this way:

1) Grab the tx hash from the ledger entry

2) Get the tx for that ledger entry and create a PyTx object out of it. Your code will look something like this:

Code:

      cppTx = TheBDM.bdv().getTxByHash(txHash)
      if cppTx.isInitialized():
         pytx = PyTx().unserialize(cppTx.serialize())

3) The PyTx object will have a list of PyTxIn and PyTxOut objects, the pretty print methods for both classes show you how to extract data out of those:

https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/Transaction.py#L513
https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/Transaction.py#L577

[marekNowak]

tab = TheBDM.bdv().getLedgerDelegateForWallets().getHistoryPage(0)

for above (is it global ledger for You?) result are exaclty the same as for

tab =self.curWlt.cppWallet.getHistoryPage(0)

I have also called below code. Confirmed transactions looks fine but unconfirmed are absent in result.

tab = TheBDM.bdv().getLedgerDelegateForScrAddr(self.curWlt.uniqueIDB58,Hash160ToScrAddr(a160)).getHistoryPage(0)


My problem is not even to sent ZC inputs but when I get unspent outputs for some address and I wanna prepare transaction from it I am not sure whether maybe they had been already used and transaction which included them is waiting in mempool for confirmation. Of course Inputs are not availabe when transaction get at least one confirmation but sometimes it takes 30 minutes or more time and inputs should not be in "getunspentoutputsforaddres" available during this time.

What exactly do You mean rescan & rebuild database? delete database and start client (or Armorydb) with some parameters? Or maybe I may do this when deamon is running?

m.

[goatpig]

I think I've figured it out.

I've wrote the ZC code so that each transaction gets processed in its own thread. To prevent lockups on older machines, it is meant to skip ZCs if there are no processing threads available to parse them (i.e. the machine is getting overwhelmed with ZC).

There are multiple solutions to this problem and I'll think about adding a CLI arg to force deterministic ZC parsing in the future. For now you can modify this value to increase the amount of parser threads:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/BDM_supportClasses.h#L28

You will have to rebuild the binary for this change to take effect. These threads are sleeping until there is data to parse, so increasing that figure shouldn't tax your CPU unless there is a lot of ZC traffic (or you are pushing 50 ZC in bulk to your node).

[marekNowak]

Thank You. I will test it and let You know.

m.

[marekNowak]

It seems much better after changing  GETZC_THREADCOUNT to 100.

Now I have 3 questions:

1) If I set GETZC_THREADCOUNT for 100 and will have 100 incoming unconfirmed transfers for 10 hours (for example 100 transfers with low fees) does it mean that for 10 hours other ZC transactions will be ignored?

2) Is there any parameter which allow me to spend unconfirmed outputs now (especially I mean unconfirmed change from previous unconfirmed transaction)?

3) For 2-3 times since yesterday I had some error like "POOLIN:0". It was sometking about connection to DB. Do You know it?  I will try to provide You a log for the next time when it will appear.

m. 

[goatpig]

1) If I set GETZC_THREADCOUNT for 100 and will have 100 incoming unconfirmed transfers for 10 hours (for example 100 transfers with low fees) does it mean that for 10 hours other ZC transactions will be ignored?

This isn't how it works. This code consumes a thread in the parser thread FIFO pile whenever it receives a ZC from your node (i.e. it receives an inv_msg_tx packet and follows up with a successful getdata on the hash).

The thread parses the tx, i.e. it performs all tasks that can be parallelized (like deser and hash operations) then sends the processed tx to the main ZC thread for evaluation (since ZC need to be processed in order of appearance). Once a parser thread is done with its task, it re-registers itself with the FIFO pile.

In other words, these threads are constantly recycled, and they do not wait on the tx receiving a confirmation. That's handled by the main scanner code instead.

GETZC_THREADCOUNT defines how many of these threads are created on startup. The current code is built to ignore a ZC if there is no parser thread available to handle it. It is built around a custom container (Stack<T>) that throws if it's empty.

There is another container, BlockingStack<T>, that as the name suggests, will block when it is empty. I could have had you simply swap to that and ignore the thread count, but since this is untested behavior, I'd rather you bump the thread count for now.

2) Is there any parameter which allow me to spend unconfirmed outputs now (especially I mean unconfirmed change from previous unconfirmed transaction)?

All getUtxo methods take a IGNOREALLZC bool argument (usually the last one, defaulting to true). Pass false there and it should return ZC that are your change.

3) For 2-3 times since yesterday I had some error like "POOLIN:0". It was sometking about connection to DB. Do You know it?  I will try to provide You a log for the next time when it will appear.

These are false positives 99% of the time, left that verbose there for my own benefit. Regardless, feel free to post the log if you get to catch another of those.