0.94 preliminary testing phase

[goatpig]

The new DB is about ready. It passes all C++ tests and runs *almost* fine. Turns out my own copy of the mainnet blockchain has a corrupt block (or at least some corner case that displays the same behavior), so I'll be taking extra time putting together a full diagnostic and repair code to get through that bad block. In the mean time I'd like some testing on what I have so far.

Haven't hooked the progress bars yet (so refer to your terminal/console output) but the rest should work (spends and ZC included). The code is in dev (https://github.com/goatpig/BitcoinArmory/tree/dev), pull away. I'm still pushing commits daily to that branch for now. Once it stabilizes and I put out the first testing builds, I'll move that code to a branch named testing.

If your scan fails please repeat and let me know if the error is random or deterministic.

EDIT: haven't tested rebuilds/rescans, they are probably broken. Delete the content of your DB folder manually if you want to rebuild. Obviously use a new folder (this db format is not compatible with ATI's 0.93/0.94 and coexist with it in the same folder). Expected DB size for mainnet is ~150MB.

[1714890389]

1714890389

[1714890389]

1714890389

[Carlton Banks]

  sync'ing & building   

[Carlton Banks]

Code:

(DatabaseBuilder.cpp:47) updated HEADERS db in 4322.26s

Code:

(DatabaseBuilder.cpp:108) scanned new blocks in 2111.9s

All appears well with basic Db build and tx scan (other than the disconnected progress bars). Will test some more tomorrow.

Edit: testing with Haswell i5 laptop, Qubes 3.0/Xen 4.4, Debian 8.3, Bitcoin 0.11.2

[goatpig]

No segfaults? How often have you ran it? Usually I consider 3 successful runs to be enough.

[Carlton Banks]

None. I've run it at least half a dozen times now, no segfaults so far. It does scan the wallet integrity twice, the second time after finishing the tx scan. Other than that it's in good shape on this platform.

[bitpop]

Can't wait thank you

Will you change the donation in it to you?

[achow101]

Great work! I will be testing this and will report back if anything bad happens.

Will 0.94 be just these database changes and RBF signaling?

Edit:
While armory was building the databases it crashed, twice now. I had renamed the old databases folder so armory create a new one for the new databases. This is the error that I got after both crashes:

Code:

terminate called after throwing an instance of 'LMDBException'
  what():  Failed to close env tx (Input/output error)
Aborted (core dumped)

I'm thinking this may also have to do with low disk space on my computer, so once I clean up some junk I will try again.

[goatpig]

Can't wait thank you

Will you change the donation in it to you?

Once I put out a release.

Will 0.94 be just these database changes and RBF signaling?

My plan is to put out a testing release after this phase. I'll go over all the PRs I have received and merge them in while I wait on feedback. Once I'm done with all DB bugs and PR merging, I'll release the rc1 and take it from there.

[goatpig]

I'm thinking this may also have to do with low disk space on my computer, so once I clean up some junk I will try again.

My DB is ~150MB on the mainnet. If you can't get it to scan mainnet, try the testnet, also build and run the C++ unit tests in the /cppForSwig/gtest folder and run those. Let me know if you can pass any of these and if you get errors, whether they are deterministic or random.

[bitpop]

You need a bitcoin address in your profile and sig here

[goatpig]

Not accepting any donations until I have an actual release out there. I'll deal with all that stuff once the release is ready.

[neuronics]

Not accepting any donations until I have an actual release out there. I'll deal with all that stuff once the release is ready.

Respect !

Thank you for your efforts, very much appreciated.   

Waiting for the release (and the donation address)

[RoadStress]

Not accepting any donations until I have an actual release out there. I'll deal with all that stuff once the release is ready.

Respect !

Thank you for your efforts, very much appreciated.   

Waiting for the release (and the donation address)

Thank you too! Words can't describe the appreciation for doing this.

[achow101]

I'm thinking this may also have to do with low disk space on my computer, so once I clean up some junk I will try again.

My DB is ~150MB on the mainnet. If you can't get it to scan mainnet, try the testnet, also build and run the C++ unit tests in the /cppForSwig/gtest folder and run those. Let me know if you can pass any of these and if you get errors, whether they are deterministic or random.

It looks like cleaning up my hard drive did the trick. I think that for those kinds of errors it would be useful to have something say that there isn't enough disk space available, which is what Bitcoin Core does.

[Carlton Banks]

Today's commit produces this compile error:

Code:

BlockchainScanner.cpp: In lambda function:
BlockchainScanner.cpp:235:30: error: invalid initialization of non-const reference of type ‘BlockData&’ from an rvalue of type ‘BlockData’
             return BlockData();
                              ^
Makefile:97: recipe for target 'BlockchainScanner.o' failed
make[1]: *** [BlockchainScanner.o] Error 1
make[1]: Leaving directory '/home/user/BitcoinArmory/cppForSwig'
Makefile:9: recipe for target 'all' failed
make: *** [all] Error 2

Sounds trivial to debug, but I'll leave that to those who code to live. Unless I'm doing something obviously wrong?

[goatpig]

Pushed db auto repair core and progress bar support. Scan and Rebuild should work too now. Fixed the build issue as well. Progress bars are a little wacky for now, I'll look into making them smoother tomorrow.

[droark]

Just FYI, the Mac build process is borked. I plan to do a PR with an update pretty soon.

EDIT: Should have the PR up later tonight.

[achow101]

I am having some build problems that I can't figure out. My build output is on pastebin here: http://pastebin.com/QAVGGuui

[goatpig]

I am having some build problems that I can't figure out. My build output is on pastebin here: http://pastebin.com/QAVGGuui

Looks like you are missing entire parts of the  C++ STL. Which compiler are you using? Also, is it C++11 compatible?

[droark]

I am having some build problems that I can't figure out. My build output is on pastebin here: http://pastebin.com/QAVGGuui

Same problem here. OS X. Latest Xcode (clang). Definitely C++11 compatible. (Hell, you could go up to C++14 and it wouldn't care.)

EDIT: goatpig, the following code should fix the problem.

Code:

   if (fileMap_ == MAP_FAILED) {
      fileMap_ = NULL;
      stringstream errStr;
      errStr << "Failed to create map of file. Error Code: " << errno;
      cout << errStr.str() << endl;
      throw runtime_error(errStr.str());
   }

EDIT 2: Confirmed. Armory now builds properly. Running the OS X build right now. We'll see what happens next....

[goatpig]

That is some weird implementation of operator<< with stringstream on these compilers! I'll push that change and merge the OSX build PR in a minute.

[droark]

Okay. Having two problems with the OS X build.

1)The balance isn't quite right. It's missing a few transactions here and there. Nothing in the logs that I can see.

2)I can't send coins. Am getting the following error.

Code:

(ERROR) Traceback (most recent call last):
  File "/Users/droark/Projects/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/ui/TxFrames.py", line 785, in createTxAndBroadcast
    ustx = self.validateInputsGetUSTX()
  File "/Users/droark/Projects/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/ui/TxFrames.py", line 766, in validateInputsGetUSTX
    utxoSelect, scriptValPairs, pubKeyMap)
  File "/Users/droark/Projects/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/armoryengine/Transaction.py", line 2148, in createFromTxOutSelection
    return self.createFromPyTx(thePyTx, pubKeyMap, txMap, p2shMap)
  File "/Users/droark/Projects/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/armoryengine/Transaction.py", line 2058, in createFromPyTx
    cppPrevTx = TheBDM.bdv().getTxByHash(txhash)
  File "/Users/droark/Projects/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/CppBlockUtils.py", line 2115, in getTxByHash
    def getTxByHash(self, *args): return _CppBlockUtils.BlockDataViewer_getTxByHash(self, *args)
RuntimeError: invalid blkFolder

[Carlton Banks]

Both 16a1569 and 3f02b93 are compiling and Db building OK for me. Now testing with Core 0.12.0 final also (still have a 0.11.2 copy of the blockchain that I will continue to test)

[achow101]

I am having some build problems that I can't figure out. My build output is on pastebin here: http://pastebin.com/QAVGGuui

Looks like you are missing entire parts of the  C++ STL. Which compiler are you using? Also, is it C++11 compatible?

I am using gcc 5.2.1.

It looks like the problem is fixed and now I am running it for testing.

Edit: I'm getting an error about invalid varint

Code:

-DEBUG - 1456328054: (Blockchain.cpp:213) Organizing chain 
-INFO  - 1456328054: (DatabaseBuilder.cpp:47) updated HEADERS db in 0.311487s
-ERROR - 1456328054: (StoredBlockObj.cpp:1346) invalid varint in SSH data
-ERROR - 1456328054: (BDM_mainthread.cpp:429) BDM thread failed: invalid varint

[goatpig]

Lots of changes, everybody will have to rebuild the DB from scratch.

2)I can't send coins. Am getting the following error.

should be fixed now

1)The balance isn't quite right. It's missing a few transactions here and there. Nothing in the logs that I can see.

Are you getting a different balance/ledger scanning the same datadir with another version of Armory? Is there a pattern in the missing transactions (I don't do multisig refs in this db, if you have a lot of lockbox activity you will be missing ledger entries but not balance)? I updated the top block reporting mechanic, is the new db reporting the proper top block now?

Edit: I'm getting an error about invalid varint

Does the error persist with this version?

[goatpig]

Pushed some more changes, make sure you pull from b24dcb2 onwards.

[achow101]

Does the error persist with this version?

Pushed some more changes, make sure you pull from b24dcb2 onwards.

Working well. No problems yet.

Edit: I also get the same coin control problem droark mentioned in the other thread. Here is the traceback:

Code:

(ERROR) Traceback (most recent call last):
  File "/home/andy/bitcoin/armory/BitcoinArmory/ui/TxFrames.py", line 785, in createTxAndBroadcast
    ustx = self.validateInputsGetUSTX()
  File "/home/andy/bitcoin/armory/BitcoinArmory/ui/TxFrames.py", line 602, in validateInputsGetUSTX
    utxoList = self.getUsableTxOutList(totalSend)
  File "/home/andy/bitcoin/armory/BitcoinArmory/ui/TxFrames.py", line 863, in getUsableTxOutList
    utxos = cppAddr.getSpendableTxOutList(IGNOREZC)
  File "/home/andy/bitcoin/armory/BitcoinArmory/CppBlockUtils.py", line 1969, in getSpendableTxOutList
    def getSpendableTxOutList(self, ignoreZC=True): return _CppBlockUtils.ScrAddrObj_getSpendableTxOutList(self, ignoreZC)
RuntimeError: not implemented

[goatpig]

I also get the same coin control problem droark mentioned in the other thread

Should be fixed now.

Introduced a new feature to rescan balances. Let's you fix balance errors without doing a full rescan, takes less than a minute on average (it's actually <1 sec on my machine). No need to wipe the DB with this commit, just try it as is.

[achow101]

I also get the same coin control problem droark mentioned in the other thread

Should be fixed now.

Introduced a new feature to rescan balances. Let's you fix balance errors without doing a full rescan, takes less than a minute on average (it's actually <1 sec on my machine). No need to wipe the DB with this commit, just try it as is.

They both appear to work now.

[Carlton Banks]

An issue switching Core client between 0.12 and 0.11.2: I'm finding that building the Armory db with 0.11.2 initially, then restarting Armory using 0.12 (with all 0.12 validated blocks) renders Armory stuck at the last block the initial build using 0.11.2 had previously reached. Doing the same test, but with 0.12 providing Armory with the initial block data to build the  db recognises the latest block when restarted using 0.11.2. Had this problem with builds previous to b24dcb2 also. Not sure what happens when doing 0.12.0 > 0.11.2 > 0.12.0 etc, haven't tried that

[achow101]

An issue switching Core client between 0.12 and 0.11.2: I'm finding that building the Armory db with 0.11.2 initially, then restarting Armory using 0.12 (with all 0.12 validated blocks) renders Armory stuck at the last block the initial build using 0.11.2 had previously reached. Doing the same test, but with 0.12 providing Armory with the initial block data to build the  db recognises the latest block when restarted using 0.11.2. Had this problem with builds previous to b24dcb2 also. Not sure what happens when doing 0.12.0 > 0.11.2 > 0.12.0 etc, haven't tried that

Perhaps that has something to do with bitcoin core 0.12 obfuscating the chainstate after syncs and reindexs. See the downgrade warning in the release notes.

[goatpig]

I'll test that directly sometimes later.

[goatpig]

Added a couple changes. Maybe that will fix the 0.11.x -> 0.12 swapping.

Going over the RBF PR, it's missing some stuff so I'll have to implement that before merging in into dev.

[achow101]

Added a couple changes. Maybe that will fix the 0.11.x -> 0.12 swapping.

Going over the RBF PR, it's missing some stuff so I'll have to implement that before merging in into dev.

IIRC RBF is only missing the actual mempool replacement stuff and the ability to create RBF transactions.

For the clean up ATI stuff, do you want to remove all of the phone home stuff? In my PR, I only have it commented out because you said earlier that it may be brought back in the future.

[goatpig]

Added a couple changes. Maybe that will fix the 0.11.x -> 0.12 swapping.

Going over the RBF PR, it's missing some stuff so I'll have to implement that before merging in into dev.

IIRC RBF is only missing the actual mempool replacement stuff and the ability to create RBF transactions.

RBF transactions aren't flagged in the main ledger nor the tx info dialog. No detection of RBF inheritence either.

For the clean up ATI stuff, do you want to remove all of the phone home stuff? In my PR, I only have it commented out because you said earlier that it may be brought back in the future.

Deactivating it is fine for now. I will have to evaluate the code and its usefulness eventually.

[goatpig]

RBF handling is actually a lot more complicated than I expected. This is gonna take me a day.

[achow101]

Added a couple changes. Maybe that will fix the 0.11.x -> 0.12 swapping.

Going over the RBF PR, it's missing some stuff so I'll have to implement that before merging in into dev.

IIRC RBF is only missing the actual mempool replacement stuff and the ability to create RBF transactions.

RBF transactions aren't flagged in the main ledger nor the tx info dialog. No detection of RBF inheritence either.

Hm. I thought my PR flagged RBF transactions in the tx info dialog.

[goatpig]

Hm. I thought my PR flagged RBF transactions in the tx info dialog.

It detects RBF in transaction.py, which is a file limited to creating transactions, not processing transaction history. Detection needs to take place at ledger creation (ideally on the C++ side), and it has to deal with inheritance properly, which I am working on.

[goatpig]

Pushed opt-in RBF detection. Regarding inheritence, I went took the simple route: any ZC of ZC is flagged as mempool replaceable, whether the chain of ZC has an RBF input somewhere or not.

If this version passes the test, I'll release the first testing build based on this state.

[achow101]

Pushed opt-in RBF detection. Regarding inheritence, I went took the simple route: any ZC of ZC is flagged as mempool replaceable, whether the chain of ZC has an RBF input somewhere or not.

If this version passes the test, I'll release the first testing build based on this state.

Awesome. Will be building and testing.

Edit: Build error:

Code:

g++  -Icryptopp -Imdb -DUSE_CRYPTOPP -D__STDC_LIMIT_MACROS -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -std=c++11 -O2 -pipe -fPIC -c BDM_supportClasses.cpp
BDM_supportClasses.cpp: In member function ‘std::map<BinaryData, std::vector<BinaryData> > ZeroConfContainer::purge(std::function<bool(const BinaryData&)>)’:
BDM_supportClasses.cpp:620:52: error: invalid initialization of non-const reference of type ‘BinaryData&’ from an rvalue of type ‘BinaryData’
          auto& bhash = lastKnownHeader->getPrevHash();
                                                    ^
Makefile:97: recipe for target 'BDM_supportClasses.o' failed
make[1]: *** [BDM_supportClasses.o] Error 1

Edit 2: Does armoryd indicate if the transaction is RBF? Or is it just GUI right now?

[goatpig]

Build error:

fixed

Does armoryd indicate if the transaction is RBF? Or is it just GUI right now?

only the GUI

[achow101]

Build error:

fixed

Does armoryd indicate if the transaction is RBF? Or is it just GUI right now?

only the GUI

Great. Testing it now.

Also, I used to be able to double click an address in my wallet for more details but it seems that I can't do that now.

[goatpig]

Also, I used to be able to double click an address in my wallet for more details but it seems that I can't do that now.

Fixed.

Ima delay the testing build release for as long it will take me to go over the changes in the cleanupATI branch and merge it in. I'd rather figure out the due diligence before putting out binaries.

[droark]

Edit: Build error:

Interesting. I just built the latest version without any problems. What's your platform?

That said, I wasn't able to send coins on the latest version.

Code:

Traceback (most recent call last):
  File "/Users/droark/Projects/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/ui/TxFrames.py", line 826, in createTxAndBroadcast
    self.main.broadcastTransaction(finalTx)
  File "/Users/droark/Projects/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/ArmoryQt.py", line 3746, in broadcastTransaction
    LOGPPRINT(pytx, logging.INFO)
  File "/Users/droark/Projects/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/armoryengine/ArmoryUtils.py", line 871, in LOGPPRINT
    theObj.pprint()
  File "/Users/droark/Projects/BitcoinArmory/osxbuild/workspace/Armory.app/Contents/MacOS/py/usr/lib/armory/armoryengine/Transaction.py", line 744, in pprint
    if self.optInRBF:
AttributeError: 'PyTx' object has no attribute 'optInRBF'

Had to comment out that line and the previous one. At that point, everything went through.

[achow101]

Edit: Build error:

Interesting. I just built the latest version without any problems. What's your platform?

It's fixed on the latest version. I'm on Ubuntu 15.10

[goatpig]

Merged in knightdk's changes around the ATI references and turning off the phone home code. Give it a spin, try to find broken dialogs (although I think I got them all). I'll be traveling tomorrow so the testing builds will have to wait for Thursday/Friday.

[achow101]

Merged in knightdk's changes around the ATI references and turning off the phone home code. Give it a spin, try to find broken dialogs (although I think I got them all). I'll be traveling tomorrow so the testing builds will have to wait for Thursday/Friday.

What about the links to bitcoinarmory.com like those on the dashboard when armory is offline? What should be done about those?

Edit: Also, I am unable to open the Lockboxes and the Address Book dialogs.

[goatpig]

Also, I am unable to open the Lockboxes and the Address Book dialogs.

Fixed the lockbox issue, can't reproduce the address book one. Do you have some Python error in your terminal when that happens?

What about the links to bitcoinarmory.com like those on the dashboard when armory is offline? What should be done about those?

Comment them out. Just PR to dev if you have minor changes.

[achow101]

Also, I am unable to open the Lockboxes and the Address Book dialogs.

Fixed the lockbox issue, can't reproduce the address book one. Do you have some Python error in your terminal when that happens?

Yeah, I get this error:

Code:

(ERROR) Traceback (most recent call last):
  File "/home/andy/bitcoin/armory/BitcoinArmory/ArmoryQt.py", line 3900, in execAddressBook
    DlgAddressBook(self, self, None, None, None).exec_()
  File "/home/andy/bitcoin/armory/BitcoinArmory/qtdialogs.py", line 8073, in __init__
    rowHeight = tightSizeStr(self.font, 'XygjpHI')[1]
  File "/home/andy/bitcoin/armory/BitcoinArmory/qtdefines.py", line 215, in tightSizeStr
    fm = QFontMetricsF(QFont(obj))
TypeError: QFont(): argument 1 has unexpected type 'builtin_function_or_method'

Traceback (most recent call last):
  File "/home/andy/bitcoin/armory/BitcoinArmory/ArmoryQt.py", line 3900, in execAddressBook
    DlgAddressBook(self, self, None, None, None).exec_()
  File "/home/andy/bitcoin/armory/BitcoinArmory/qtdialogs.py", line 8073, in __init__
    rowHeight = tightSizeStr(self.font, 'XygjpHI')[1]
  File "/home/andy/bitcoin/armory/BitcoinArmory/qtdefines.py", line 215, in tightSizeStr
    fm = QFontMetricsF(QFont(obj))
TypeError: QFont(): argument 1 has unexpected type 'builtin_function_or_method'

What about the links to bitcoinarmory.com like those on the dashboard when armory is offline? What should be done about those?

Comment them out. Just PR to dev if you have minor changes.

Alright, will do that shortly.

[goatpig]

Yeah, I get this error:

That's pretty severe, I should get this error too but I can't reproduce it. Anyone else running into this? Can you git stash git pull and try again?

[achow101]

Yeah, I get this error:

That's pretty severe, I should get this error too but I can't reproduce it. Anyone else running into this? Can you git stash git pull and try again?

Same error. Nothing was stashed and nothing was pulled.

[goatpig]

Same error. Nothing was stashed and nothing was pulled.

The part of the code that is failing has not been changed in over 2 years. If you are getting this error now you should be getting it with other versions. Start 0.93 offline and try.