Debugging bitcoind crash when calling sendmany JSON-RPC command

[libertaad]

Recently we have run into a problem with bitcoind crashing on us. After looking into the problem, we have found that the crashes always occur directly after our system issues a `sendmany` JSON-RPC command.

When sending out bitcoins, our system issues the following JSON-RPC commands:

Code:

settxfee 0.0005
walletpassphrase <passphrase>, 5
sendmany "", {"<addr>": <amount>, ... }

After the sendmany command, we occasionally see an error code: "-4", with the message: "Transaction creation failed". However, we also occasionally see a complete crash of bitcoind.

The crashes typically occur when our system is under higher load, and processing a higher volume of deposits and withdrawals. However, we can't really find any strong causation - other than the fact that it always occurs after a sendmany call.

These crashes seemed to occur after upgrading to 0.8. We are currently running 0.8.1, and are still experiencing the problem. However, it is difficult to tell if 0.8 is the culprit, or we just happened to scale to a sufficient size at around the same time we upgraded to 0.8.

We would like to help debug this issue, however, to be honest, we don't really know where to start. So, my questions are:

- Is this a known issue, or has anyone else experienced it? If so, are there any known work-arounds?

- How can we help to get to the bottom of this issue? Ideally, we'd either find what we're doing wrong to cause bitcoind to crash, or we'd help find the bug in bitcoind so it can be patched, and more resilient to whatever is causing this issue.

Thanks!

[1714938642]

1714938642

[1714938642]

1714938642

[1714938642]

1714938642

[2112]

Code:

ulimit -c unlimited

before invoking bitcoind is your friend. Then wait for the crash and do the postmortem:

Code:

gdb bitcoind core.2112

. Post the results.

[libertaad]

Code:

ulimit -c unlimited

before invoking bitcoind is your friend. Then wait for the crash and do the postmortem:

Code:

gdb bitcoind core.2112

. Post the results.

Thanks for the help! We've followed your instructions, and now we're now awaiting a core dump. It usually happens about once a day, so hopefully we don't have to wait too long.

[Nicolai]

I'm pretty sure bitcoind crashes when "walletpassphrase" timeout while "sendmany" is doing it's job.

I can't test it right now, but as far as I remember then 0.7 had this problem (and I don't think it's fixed in 0.8 ).

[libertaad]

I just got a core dump. Unfortunately, my bitcoind isn't compiled with debugging symbols. So, when I run the gdb command, I see this message:

Code:

Reading symbols from /usr/bin/bitcoind...(no debugging symbols found).

The backtrace I get is:

Code:

(gdb) bt
#0  0x00007f0033c22425 in raise ()
   from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f0033c25b8b in abort ()
   from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f003457469d in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f0034572846 in ?? ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f0034572873 in std::terminate() ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f003457296e in __cxa_throw ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00000000004298d3 in ?? ()
#7  0x0000000000561ec9 in ?? ()
#8  0x000000000059fdbe in ?? ()
#9  0x00007f00347c4e9a in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f0033cdfcbd in clone ()
   from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

As you can see, not too useful :/

Please let me know if there is anything I can do to help make this more useful.

I'm pretty sure bitcoind crashes when "walletpassphrase" timeout while "sendmany" is doing it's job.

I can't test it right now, but as far as I remember then 0.7 had this problem (and I don't think it's fixed in 0.8 ).

That would actually make a lot of sense, given the fact that we usually see crashes during high traffic times - which would probably be when bitcoind would take longer to respond. We will try increasing the wallet unlock time from 5 to 20 and see if the crashing behavior is fixed. Is there a bug report that you know of for this bug, so we can track it?

Thanks!

[2112]

Please let me know if there is anything I can do to help make this more useful.

What can I say? Rebuid your bitcoind without stripping the debug symbols. The default makefile.unix already has "-g" to enable debugging.

[libertaad]

Please let me know if there is anything I can do to help make this more useful.

What can I say? Rebuid your bitcoind without stripping the debug symbols. The default makefile.unix already has "-g" to enable debugging.

Sounds good! I will recompile bitcoind with debug symbols included, and try to get a better backtrace.

In the meantime, I will continue to leave things running with a longer wallet unlock timeout for the next few days, or until another crash occurs, since that does seem to be a promising lead.

Thanks again for the help!

[libertaad]

After running with an increased passphrase timeout for the past few days, we have not seen any problems. No more bitcoind crashes, and no more "code: -4, 'Transaction creation failed'" errors.

Specifically, we are now running the code:

Code:

walletpassphrase <passphrase>, 60
sendmany "", {"<addr>": <amount>, ... }

We noticed that sometimes the sendmany command was taking close to 20 seconds, so to be on the safe side, we increased the timeout to 60.

We couldn't seem to find and existing bug report on this issue though. We're planning on collecting some more data, and then submitting a proper bug report.

[danystatic]

After running with an increased passphrase timeout for the past few days, we have not seen any problems. No more bitcoind crashes, and no more "code: -4, 'Transaction creation failed'" errors.

Specifically, we are now running the code:

Code:

walletpassphrase <passphrase>, 60
sendmany "", {"<addr>": <amount>, ... }

We noticed that sometimes the sendmany command was taking close to 20 seconds, so to be on the safe side, we increased the timeout to 60.

We couldn't seem to find and existing bug report on this issue though. We're planning on collecting some more data, and then submitting a proper bug report.

How are your using this code?
Because i'm somewhat new but I can know that your are not typing this in the command line.

So it has to be a bash script, or maybe python, or which method are you using?
I have a problem and I'm using  jsonrpcphp library to sendmany but it is not working.... I could easily change library or go other way....
any words are appreciated