RAM-Reduction & Backup Center Testing (version 0.89.99.16)

[DobZombie]

Version 0.89.99.5 has been running for 2 days now without crashing.

Gentleman, congratulations!

[etotheipi]

Hey guys, sorry for the lack of updates recently.  I am still trying to get my brain back on track now that my in-laws have left  So I haven't had a chance to fix any of the startup bugs, but goatpig did make me a nice script for fixing the build issues.  It has been tested on a couple versions of Linux and seems to work great.  

Basically, it's a script that is run as part of the makefile, which attempts to locate the correct libpython dependencies and writes the correct values to a pypaths.txt file.  That file is then included in the Makefile.  I mention those details, because if it doesn't work, you'll probably have to manually set the paths in pypaths.txt, and then comment out the line in the Makefile (near the top) that executes the "whereispy.sh" script.  But hopefully no one will have to do that

If you are a linux user, especially one that had compile issues with 0.89.99.5, please try it again.  Checkout a fresh copy and try building.   The last build issue to be resolved on Linux is a redefinition error in log.h, where two functions appear to be the same when compiled on 32-bit systems.  I just have to find the right preprocessor for it and that should be resolved, too.  

So if you are on 64-bit Linux, the new Makefile should work, on all versions of Ubuntu, and I think goatpig tested Fedora, too.  If you are on 32-bit linux, you might have to comment out three "redefined" functions in log.h, but then it should compile fine.  

[etotheipi]

Oh this is fun (for you C++ Linux devs)!  I didn't know this existed:

Code:

touch foo.c ; g++ -dM -E foo.c; rm foo.c;

That prints a full list of all pre-processor symbols defined by default in your g++ compiler.  I noticed a bunch of things I didn't know were normally compiler defined.  Just some samples from my system (Linux Mint 15, 64-bit):

Code:

#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__
#define __INT_LEAST32_TYPE__ int
#define __INT32_TYPE__ int
#define __UINT64_TYPE__ long unsigned int
#define __SIZE_TYPE__ long unsigned int
#define __DBL_EPSILON__ double(2.22044604925031308085e-16L)
#define __DBL_MIN_EXP__ (-1021)
#define __FLT_MIN__ 1.17549435082228750797e-38F
#define __VERSION__ "4.7.3"
#define __SIZEOF_SIZE_T__ 8
#define __SIZEOF_INT__ 4
#define __SIZEOF_POINTER__ 8
#define __SIZE_TYPE__ long unsigned int
#define __x86_64__ 1
#define __LP64__ 1
#define __FLT_HAS_INFINITY__ 1

[pvz]

On my Ubuntu 13.04 i368 running Armory for the first time this error hits every 10 - 15 blocks:

-INFO  - 1382470521: (BlockUtils.cpp:3693) Parsing blockchain file: /home/xxx/.bitcoin/blocks/blk00033.dat
-INFO  - 1382470522: (BlockUtils.cpp:3787) /home/xxx/.bitcoin/blocks/blk00033.dat is 134,217,728 bytes
Unhandled Error
Traceback (most recent call last):
  File "ArmoryQt.py", line 4981, in <module>
    os._exit(QAPP.exec_())
  File "/home/xxx/Downloads/armory/BitcoinArmory/qt4reactor.py", line 231, in _iterate
    self.runUntilCurrent()
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 805, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "ArmoryQt.py", line 4482, in Heartbeat
    sdmState = TheSDM.getSDMState()
  File "/home/xxx/Downloads/armory/BitcoinArmory/armoryengine.py", line 11181, in getSDMState
    state = self.getSDMStateLogic()
  File "/home/xxx/Downloads/armory/BitcoinArmory/armoryengine.py", line 11234, in getSDMStateLogic
    elif not self.bitcoindIsResponsive():
  File "/home/xxx/Downloads/armory/BitcoinArmory/armoryengine.py", line 11169, in bitcoindIsResponsive
    return satoshiIsAvailable(self.bitconf['host'], self.bitconf['rpcport'])
  File "/home/xxx/Downloads/armory/BitcoinArmory/armoryengine.py", line 10703, in satoshiIsAvailable
    s = socket.socket()
  File "/usr/lib/python2.7/socket.py", line 187, in __init__
    _sock = _realsocket(family, type, proto)
socket.error: [Errno 24] Te veel open bestanden (translation: to many open files)
-INFO  - 1382470673: (BlockUtils.cpp:3693) Parsing blockchain file: /home/xxx/.bitcoin/blocks/blk00034.dat
-INFO  - 1382470673: (BlockUtils.cpp:3787) /home/xxx/.bitcoin/blocks/blk00034.dat is 134,217,728 bytes

Armory continues to run without any dump or GUI error (in the case above until block 83 because of error below) and after restart started from block 33.

The second error:

-INFO  - 1382468790: (BlockUtils.cpp:3693) Parsing blockchain file: /home/xxx/.bitcoin/blocks/blk00087.dat
-INFO  - 1382468790: (BlockUtils.cpp:3787) /home/xxx/.bitcoin/blocks/blk00087.dat is 134,089,409 bytes
-INFO  - 1382468834: (BlockUtils.cpp:3693) Parsing blockchain file: /home/xxx/.bitcoin/blocks/blk00088.dat
-INFO  - 1382468834: (BlockUtils.cpp:3787) /home/xxx/.bitcoin/blocks/blk00088.dat is 33,554,432 bytes
-INFO  - 1382468842: (BlockUtils.cpp:3707) Processed 77978 raw blocks DB (3180 seconds)
-INFO  - 1382468844: (BlockUtils.cpp:3736) Starting scan from block height: 0
-ERROR - 1382468844: (StoredBlockObj.cpp:1063) Cannot get tx copy, because don't have full StoredTx!
-ERROR - 1382468844: (BlockUtils.cpp:757) Passed uninitialized STX to regAddrScan
-ERROR - 1382468846: (StoredBlockObj.cpp:1063) Cannot get tx copy, because don't have full StoredTx!
-ERROR - 1382468846: (StoredBlockObj.cpp:1063) Cannot get tx copy, because don't have full StoredTx!
Segmentatiefout (geheugendump gemaakt [translation: memmorydump created] )

Please let me know if I can do some more tests.
Armory keep up the good work!

[etotheipi]

This looks eerily similar to teh guy that just posted here:

https://bitcointalk.org/index.php?topic=316312.0

I wonder if the recent update did something... because so far I haven't ever seen that error until I committed the tiny update a couple hours ago... at least it should be easy to track down what happened...

Can you please try --rebuild.  It will take a while to rebuild the database, but it sounds like your DB is corrupted.  There's some transactions in the DB that were only partially written.  That's bad.

[pvz]

Can you please try --rebuild.  

I'll do that. Should it be default for the first time or a database check before start?

[etotheipi]

Can you please try --rebuild. 

I'll do that. Should it be default for the first time or a database check before start?

It is.  The default behavior is to rebuild if the DB doesn't exist.  Just like I mentioned in that other thread, it looks like you might've interrupted the DB build process in the middle and then it restarted (hence only 77k blocks processed, not 260k).  It's worked on my system before but it may not be reliable yet (to interrupt and then restart the build).

[wheeler]

This looks eerily similar to teh guy that just posted here:

https://bitcointalk.org/index.php?topic=316312.0

Just to update on progress from that thread...

The --rebuild progressed for me until the 18th block file, then raised the same socket error just reported.  Armory GUI was still responsive, but stopped processing further blocks.

Not had chance to look in detail at the satoshiIsAvailable() method, but perhaps socket creation should be within the try block?  Might hack that myself and give it a go.

[etotheipi]

This looks eerily similar to teh guy that just posted here:

https://bitcointalk.org/index.php?topic=316312.0

Just to update on progress from that thread...

The --rebuild progressed for me until the 18th block file, then raised the same socket error just reported.  Armory GUI was still responsive, but stopped processing further blocks.

Not had chance to look in detail at the satoshiIsAvailable() method, but perhaps socket creation should be within the try block?  Might hack that myself and give it a go.

Are you running a 32-bit OS?  I just committed the first version that should build on 32-bit *nix, but haven't had much chance to test online mode (only tested offline mode).  pvz appears to be in 32-bit as well, and I wonder if that's related...

[pvz]

--rebuild does not work.
Deleted .armory directory, started armory again (latest testing branch from a few minutes ago) with this result:

laptop:~/Downloads/armory/BitcoinArmory$ python ArmoryQt.py
********************************************************************************
Loading Armory Engine:
   Armory Version:       0.89.99.6
   PyBtcWallet  Version: 1.35
Detected Operating system: Linux
   OS Variant            : ('Ubuntu', '13.04', 'raring')
   User home-directory   : /home/xxx
   Satoshi BTC directory : /home/xxx/.bitcoin/
   Armory home dir       : /home/xxx/.armory/
   LevelDB directory     : /home/xxx/.armory/databases
   Armory settings file  : /home/xxx/.armory/ArmorySettings.txt
   Armory log file       : /home/xxx/.armory/armorylog.txt
Log file doesn't exist [yet]
(WARNING) armoryengine.py:11197 - Overriding not-available message. This should happen 0-5 times
-INFO  - 1382477339: (BlockUtils.cpp:1582) Set home directory:
-INFO  - 1382477339: (BlockUtils.cpp:1604) Set blkfile dir: /home/xxx/.bitcoin/blocks
-INFO  - 1382477339: (BlockUtils.cpp:1614) Set leveldb dir: /home/xxx/.armory/databases
-INFO  - 1382477339: (BlockUtils.cpp:1570) SetBtcNetworkParams
-INFO  - 1382477339: (BlockUtils.cpp:3542) Executing: doInitialSyncOnLoad
-INFO  - 1382477339: (BlockUtils.cpp:3582) Number of registered addr: 0
-INFO  - 1382477340: (BlockUtils.cpp:1709) Current Top block in HEADERS DB:  0
-INFO  - 1382477340: (BlockUtils.cpp:1710) Current Top block in BLKDATA DB:  0
-INFO  - 1382477340: (BlockUtils.cpp:1711) Current Applied blocks up to hgt: 0
-INFO  - 1382477340: (BlockUtils.cpp:3606) Clearing databases for clean build
-WARN  - 1382477340: (BlockUtils.cpp:3507) Destroying databases;  will need to be rebuilt
pthread lock: Ongeldig argument
Afgebroken (geheugendump gemaakt)

Question: how can I make a fresh Armory restart?

[wheeler]

Are you running a 32-bit OS?

Yes:
Ubuntu 12.04 (precise) 32-bit
AMD Athlon(tm) 64 X2 Dual Core Processor 5000+ × 2
2GB RAM

Will look further tomorrow (on London time here  )

Thanks for your help so far!

[etotheipi]

--rebuild does not work.
Deleted .armory directory, started armory again (latest testing branch from a few minutes ago) with this result:

laptop:~/Downloads/armory/BitcoinArmory$ python ArmoryQt.py
********************************************************************************
Loading Armory Engine:
   Armory Version:       0.89.99.6
   PyBtcWallet  Version: 1.35
Detected Operating system: Linux
   OS Variant            : ('Ubuntu', '13.04', 'raring')
   User home-directory   : /home/xxx
   Satoshi BTC directory : /home/xxx/.bitcoin/
   Armory home dir       : /home/xxx/.armory/
   LevelDB directory     : /home/xxx/.armory/databases
   Armory settings file  : /home/xxx/.armory/ArmorySettings.txt
   Armory log file       : /home/xxx/.armory/armorylog.txt
Log file doesn't exist [yet]
(WARNING) armoryengine.py:11197 - Overriding not-available message. This should happen 0-5 times
-INFO  - 1382477339: (BlockUtils.cpp:1582) Set home directory:
-INFO  - 1382477339: (BlockUtils.cpp:1604) Set blkfile dir: /home/xxx/.bitcoin/blocks
-INFO  - 1382477339: (BlockUtils.cpp:1614) Set leveldb dir: /home/xxx/.armory/databases
-INFO  - 1382477339: (BlockUtils.cpp:1570) SetBtcNetworkParams
-INFO  - 1382477339: (BlockUtils.cpp:3542) Executing: doInitialSyncOnLoad
-INFO  - 1382477339: (BlockUtils.cpp:3582) Number of registered addr: 0
-INFO  - 1382477340: (BlockUtils.cpp:1709) Current Top block in HEADERS DB:  0
-INFO  - 1382477340: (BlockUtils.cpp:1710) Current Top block in BLKDATA DB:  0
-INFO  - 1382477340: (BlockUtils.cpp:1711) Current Applied blocks up to hgt: 0
-INFO  - 1382477340: (BlockUtils.cpp:3606) Clearing databases for clean build
-WARN  - 1382477340: (BlockUtils.cpp:3507) Destroying databases;  will need to be rebuilt
pthread lock: Ongeldig argument
Afgebroken (geheugendump gemaakt)

Question: how can I make a fresh Armory restart?

Ack, this is really not going well on Ubuntu 32-bit (that's what you're in, right?).  You can remove the /home/xxx/.armory/databases directory to retry from scratch.  Though, "--rebuild" should do exactly that.

I noticed the line "pthread lock:  Ongeldig argument", I assume that means "Invalid Argument"?  If so, that may be the source of the problem, perhaps the way leveldb is being compiled here is not playing nicely with the 32-bit libraries.

[etotheipi]

@pvz and @wheeler:

I just ran into the segfault issue when using --rebuild.  It turns out that i screwed up some pointer cleanup, and that was causing the segfaults.  Luckily, the last update was so tiny that it was obvious what did it!  I just fixed it and committed it.  Please do a "git pull origin testing" and then try the --rebuild again.

It might also have been responsible for the other issues you observed, though I won't be sure until you try it. 

[wheeler]

@pvz and @wheeler:

I just ran into the segfault issue when using --rebuild.  It turns out that i screwed up some pointer cleanup, and that was causing the segfaults.  Luckily, the last update was so tiny that it was obvious what did it!  I just fixed it and committed it.  Please do a "git pull origin testing" and then try the --rebuild again.

It might also have been responsible for the other issues you observed, though I won't be sure until you try it. 

Thanks, will give this a try tonight.  FWIW, the last run (not your latest code) failed due to exhausting file handles.  lsof lists lots of ~/.armory/databases/leveldb_blkdata/*.sst files open.

[pvz]

A first quick test seems to work. Because it is not my computer I can't do more tests... to bad...

[cp1]

Thanks, will give this a try tonight.  FWIW, the last run (not your latest code) failed due to exhausting file handles.  lsof lists lots of ~/.armory/databases/leveldb_blkdata/*.sst files open.

If the problem isn't the code, you can always increase the max # of files open, that's a pretty common problem.

[etotheipi]

Okay, I just ran into a problem on windows that wheeler originally observed.   Luckily, this one is isolated, so I should be able to catch it easily.  It causes the "cannot get tx copy" message.  I thought it was a DB-corruption issue, but instead it looks like it has to do with orphan blocks in the block files under certain conditions. 

For me, it keeps triggering at the same block, so I should have no problem isolating it.

One day, this stuff will work...

[wheeler]

Quick update on my attempts:  Got the latest testing branch 0.89.99.6, and also boosted the ulimit on files to 4096 (from 1024).  This time I got much further in the database rebuild (to the 48th block file), but it eventually fell over with a "Bus Error" (OS error, not a Python Exception).  Trying to restart now without the --rebuild option and it seems to have continued from where it left off, so hopefully after a couple of attempts I'll get fully synced!

@etotheipi: Let me know if you want me to e-mail you any cores or logs?

Thanks.

[cypherdoc]

is 0.89.99.5 still the latest Windows version?

i need to try and reinstall since it now refuses to open after about 1 mo of successful usage 

[e4xit]

OK so I am having some difficulty again getting the latest testing branch (0.89.99.6 I believe) to run on Ubuntu 13.10 VM...

I can't actually get past the splash "Armory" splash screen, and I don't think I am throwing any errors, so a bit stuck here... http://imgur.com/CAQNyaD

Here is my logfile http://pastebin.com/hPVigcDn which I don't see anything wrong with. If anyone could offer me some info on how to troubleshoot this I would appreciate it.

Thanks.