Starting preliminary 0.94 testing - "Headless fullnode"

[goatpig]

bad_alloc means the process failed to allocate new memory. Most likely the code is trying to allocate a ridiculous amount of RAM (possibly some badly derserialized varInt).

Download the testnet chain and try to build that. Also, could you give me your first block file? (blk00000.dat)

[trev]

I built the ffreeze version of Armory under Linux and am running it with bitcoind 0.11.0.
So far everything is working fine.

A minor issue happened when I restored a wallet from 0.92 and then
tried "Addresses->View Address Book". Armory threw the following exception:
  File "ArmoryQt.py", line 3902, in execAddressBook
    DlgAddressBook(self, self, None, None, None).exec_()
  File "/home/userid/BitcoinArmory-ffreeze/qtdialogs.py", line 8133, in __init__
    rowHeight = tightSizeStr(self.font, 'XygjpHI')[1]
  File "/home/userid/BitcoinArmory-ffreeze/qtdefines.py", line 215, in tightSizeStr
    fm = QFontMetricsF(QFont(obj))
TypeError: QFont(): argument 1 has unexpected type 'builtin_function_or_method'

However, I keep on getting popups about Armory losing it's connection to bitcoind.
(ERROR) Networking.py:359 - ***Connection to Satoshi client LOST!  Attempting to reconnect...

In ~/.bitcoin/debug.log there are bezillions of lines like this...
....
2015-08-18 00:54:43 keypool reserve 1
2015-08-18 00:54:43 keypool return 1
....
It looks like Armory is polling bitcoind on 127.0.0.1:8332 once a second to get the chain state.
Each poll is a tcp connection; open and close.
The problem is that I get a bucket load of tcp TIME_WAIT entries. Eventually the system runs
out of socket resources, and then I think Armory thows the cannot connect popup.

Is there an option to increase the polling interval to, let's say, 10 second?
Or better still, could Armory open a single tcp connection to bitcoind and keep it open for
the duration of the session?

I like having Armory start bitcoind as a child process, and would prefer to not have to start it
myself, which is the work-around I have seen given in previous posts.

[Carlton Banks]

I like having Armory start bitcoind as a child process, and would prefer to not have to start it
myself, which is the work-around I have seen given in previous posts.

The code Armory uses to launch bitcoind as a child process is already flagged by the devs as starting to look a little stale and old. I'm not sure that they're planning to overhaul it for this particular release ("ffreeze" means "feature freeze" I think), but it will happen eventually. I also get unstable and/or buggy behaviour when Armory manages bitcoind, although not quite the same issues you're getting.

[goatpig]

Is there an option to increase the polling interval to, let's say, 10 second?

There is no command line argument for this purpose afaik. The current stance on this piece of code is that it works enough that we completely ignored it for the past few versions update. As such, I am completely unfamiliar with it.

There is also no plan to fix it or overhaul it, as blocks over P2P would require a C++ implementation of the Bitcoin P2P protocol, in which case all communications with Core would be handled straight from the backend and the Python code dropped entirely, which is another reason I've made a point of ignoring it for so long.


At any rate, if you feel adventurous, you can modify the Python hardcoded value. I think this line should do it:

https://github.com/etotheipi/BitcoinArmory/blob/ffreeze/ArmoryQt.py#L6391

Change nextBeatSec=1 to what you see fit. You can use decimal values to signify smaller time increments (i.e. 1.2 translates to 1200ms)

Or better still, could Armory open a single tcp connection to bitcoind and keep it open for
the duration of the session?

I don't think that's the default reactor behavior. Dropping the socket after every iteration is indeed expensive compared to a keep-alive model, but still marginal in the scope of a localhost connection. The issue isn't so much the amount of sockets opened than the less than stellar Python GC and the implication that no one in Python bothers with explicit clean ups.

[trev]

However, I keep on getting popups about Armory losing it's connection to bitcoind.
(ERROR) Networking.py:359 - ***Connection to Satoshi client LOST!  Attempting to reconnect...

I found that "maxconnections=6" in bitcoin.conf was causing bitcoind to reject connections
from Armory on the localhost interface. When I changed it to "maxconnections=30", Armory
started working as expected, and bitcond says it has 9 open connections.

[goatpig]

However, I keep on getting popups about Armory losing it's connection to bitcoind.
(ERROR) Networking.py:359 - ***Connection to Satoshi client LOST!  Attempting to reconnect...

I found that "maxconnections=6" in bitcoin.conf was causing bitcoind to reject connections
from Armory on the localhost interface. When I changed it to "maxconnections=30", Armory
started working as expected, and bitcond says it has 9 open connections.

Oh right, there's that too. I personally do it the other way and add "addnode=127.0.0.1" to guarantee Core tries to keep the localhost connection with Armory alive.

[trev]

When running ffreeze and bitcoind 0.11.0 in Linux I got a "BDM thread failed: std::bad_alloc" crash.
Python traps the exception so gdb has nothing to report.
I was doing a send from bitcoin-qt to Armory-watch-wallet. The exception happened when Armory received
the broadcast transaction. I restarted Armory and it then saw the transaction when the first confirmation block
arrived. Are there any debug printf's for identifying where the bad_alloc is occuring?

[goatpig]

When running ffreeze and bitcoind 0.11.0 in Linux I got a "BDM thread failed: std::bad_alloc" crash.
Python traps the exception so gdb has nothing to report.
I was doing a send from bitcoin-qt to Armory-watch-wallet. The exception happened when Armory received
the broadcast transaction. I restarted Armory and it then saw the transaction when the first confirmation block
arrived. Are there any debug printf's for identifying where the bad_alloc is occuring?

Are you using a x86 OS?

[trev]

Are you using a x86 OS?

I'm using a home-built Linux running on a HP laptop.
Linux sys 2.6.37.6 #11 SMP Sun Apr 14 21:52:49 EDT 2013 x86_64 Intel(R) Core(TM)2 CPU T5300 @ 1.73GHz GenuineIntel GNU/Linux
I used gcc-4.9.2 to build the c++11 bits of Armory. Packages: swig-3.0.6 sip-4.16.9 PyQt-x11-gpl-4.11.4 psutil-2.1.3 Twisted-15.3.0 qt-4.8.6.

[goatpig]

I'm pretty sure Core 2 can handle both x86 and x64 OSes, so this isn't really helping me. Browse to Armory's binary folder, and run this command and paste the result back:

Code:

file _CppBlockUtils.so

[Carlton Banks]

it does actually say x86_64 on the readout, apologies if that's what you meant.

Could it be something to do with the instruction set on the Core 2 era? I seem to remember that Core 2 was better specified than Core 2 Duo somehow, possibly not

[goatpig]

bad_alloc usually means the process failed to allocate more RAM. I'd like to make sure this is not happening on a x86 machine first before I investigate.

[trev]

I'm pretty sure Core 2 can handle both x86 and x64 OSes, so this isn't really helping me. Browse to Armory's binary folder, and run this command and paste the result back:

Code:

file _CppBlockUtils.so

_CppBlockUtils.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped

The machine only has 2GB dram, but it also has 12GB swap.
I think you can also get this exception from a realloc with a bad ptr.

[goatpig]

We don't use realloc directly, and very little malloc anyways.

It is most likely a std container failing to get enough memory. The issue isn't so much that the addressing failed but whether it is because it ran out of RAM or it was passed a bad ptr. If this is happening in a std container, which we rely heavily upon, then it's the former.

Did the issue happen after scanning the whole chain or was it in a freshly started process?

[trev]

Did the issue happen after scanning the whole chain or was it in a freshly started process?

I've been running Armory on the laptop for about a week. The initial scan happened without incident.
It took me a few days to sort out the maxconnections issue. After that I've been running Armory
continuously. The laptop is only used to run bitcoind and Armory.

I got a first std::bad_alloc crash at 20150821_1449.
I restarted Armory under gdb.
I got the second std::bad_alloc crash at 20150823_2125.
Python had exited cleanly so there was nothing for gdb to display.
I restarted Armory under gdb, set a breakpoint at BDM_mainthread.cpp:453, and did run with --netlog.
I am expecting to have to wait a couple of days for another crash. I'll keep on making small value
transfers between Armory and bitcoin-qt. I want to see if the crash is spontaneous, or if it happens during
some user initiated activity.

[goatpig]

I got a first std::bad_alloc crash at 20150821_1449.
I restarted Armory under gdb.
I got the second std::bad_alloc crash at 20150823_2125.

So you are saying it takes 24h+ to crash? Can you keep an eye on the process memory? I'll run some tests on my own.

[trev]

So you are saying it takes 24h+ to crash? Can you keep an eye on the process memory? I'll run some tests on my own.

I got a third crash at 20150825_2016. It looks like Armory has to have been running for 24++ hrs in order to
get to the condition that triggers the std::bad_alloc. The approximate 48 hr interval that I see is probably due to
the fixed times of day that I use the PC. I sent 1.5BTC from bitcoin-qt to Armory-watch. Armory popped up the
std::bad_alloc window a few seconds after bitcoin-qt said it had sent the TX. I think the problem occurs when
Armory receives the raw TX. After restarting, it gets the TX from the block without a problem (but that's not after
a 24++ hr delay).

The memory usage remained fairly constant for the 48 hrs, including after the exception.
  Armory:   VIRT = 1052M  RES = 288M   SHR = 19M
  Bitcoin-qt  VIRT = 1121M  RES = 404M   SHR = 12M

I'm not having any success debuging ArmoryQt.py with gdb. I'll just try to get a crash in less than 48 hrs.

[goatpig]

So you are saying it takes 24h+ to crash? Can you keep an eye on the process memory? I'll run some tests on my own.

I got a third crash at 20150825_2016. It looks like Armory has to have been running for 24++ hrs in order to
get to the condition that triggers the std::bad_alloc. The approximate 48 hr interval that I see is probably due to
the fixed times of day that I use the PC. I sent 1.5BTC from bitcoin-qt to Armory-watch. Armory popped up the
std::bad_alloc window a few seconds after bitcoin-qt said it had sent the TX. I think the problem occurs when
Armory receives the raw TX. After restarting, it gets the TX from the block without a problem (but that's not after
a 24++ hr delay).

The memory usage remained fairly constant for the 48 hrs, including after the exception.
  Armory:   VIRT = 1052M  RES = 288M   SHR = 19M
  Bitcoin-qt  VIRT = 1121M  RES = 404M   SHR = 12M

I'm not having any success debuging ArmoryQt.py with gdb. I'll just try to get a crash in less than 48 hrs.

What's the total size of your DB? The headers DB alone?

https://github.com/etotheipi/BitcoinArmory/blob/ffreeze/cppForSwig/mdb/mdb.c#L588

Reduce this value to 16MB and rebuild a new DB with that, see if it suffers from the bad_alloc.

[fenwick]

Blockchain not scanned up to the top!

So, I have compiled Armory from github, using the ffreeze branch.
(My exact revision is 65dafec4ad888219abaa8897929364423f71ea05 )

I could successfully sync up my DB, and I also did two successful transactions.
I know they were successful, because they show up (with plenty of confirmations) on blockchain.info.

But then, after restarting Armory, the transactions don't show up.

Here is the full story:

 - I did a successful transaction, which was immediately confirmed a few times.
 - However, I noticed that armory still showed 0 confirmations.
 - I did another transaction, and Armory said that this transaction might have
   been refused by the network. I want to blockchain.info to check, and
   I saw that the transaction waiting there with 0 confirmations.
   A few blocks have gone by, and it wasn't confirmed, so I suspected that
   maybe I should have set higher fees, and so I wanted to "undo" this
   transactions ... so I told Armory (in the help menu) to
   "Clear All Unconfirmed".
 - Meanwhile, I see that eventually, my transaction _were_ confirmed OK
 - However, after restarting armory, these transactions don't show up at all.
   Neither the first one, nor the second one. Likewise, the displayed
   balance is the one before the transactions. According to armory, it's
   as if those transactions have never happened.
 - Also, the label in the bottom right corner says "Connected (353000 blocks)",
   whereas according to the .bitcoin/debug.log, the current block number is
   371598. Ha Armory forgot to scan the last 20000 blocks?

That is going on here?

[fenwick]

On the list of transactions, clicking on the header fields of the table (saying "Date", Wallet", "Comments", etc.), I can see an arrow being moved, switched up/down, moved to the column I clicked, etc, but the displayed transactions are not re-ordered accordingly.

In fact, they don't change at all.