Bitcoin Forum

Bitcoin => Bitcoin Technical Support => Topic started by: jlp on March 11, 2015, 05:08:36 PM

Title: debug.log runCommand error
Post by: jlp on March 11, 2015, 05:08:36 PM

I have used the following successfully in bitcoin.conf for about a year:

Code:

walletnotify=curl https://mydomain.com/walletnotify.php/?tanshash=%s
blocknotify=curl https://mydomain.com/blocknotify.php/?blockhash=%s

They seemed to stop working a few months ago. The following is in debug.log:

Code:

2015-03-11 14:05:53 runCommand error: system(curl https://mydomain.com/blocknotify.php/?blockhash=00000000000000000dd3c33eaf1a2746ebe841418b98cb15f4a0bb71f3330d5e) returned -1

I changed my code in bitcoin.conf to this:

Code:

blocknotify=/usr/bin/php /home/mydomain/public_html/blocknotify.php %s

But I'm still getting the same error:

Code:

2015-03-11 16:28:57 runCommand error: system(/usr/bin/php /home/mydomain/public_html/blocknotify.php 000000000000000015a51b99f54a611b2a88c096d1091e13933346431c17b2be) returned -1

Can anyone tell me what "runCommand error: ... returned -1" means and how to fix this?

Title: Re: debug.log runCommand error
Post by: grue on March 11, 2015, 05:32:57 PM

does it run successfully in in shell? as in, run the command, then do "echo $?" to check the return status.

Title: Re: debug.log runCommand error
Post by: jlp on March 11, 2015, 06:16:56 PM

Quote from: grue on March 11, 2015, 05:32:57 PM

does it run successfully in in shell? as in, run the command, then do "echo $?" to check the return status.

Are you referring to this?:

Code:

https://mydomain.com/blocknotify.php/?blockhash=00000000000000000dd3c33eaf1a2746ebe841418b98cb15f4a0bb71f3330d5e

If so, then yes, that program executes when I invoke it from a browser.

If you are suggesting that I add the following command in bitcoin.conf:

Code:

blocknotify=/home/bitcoin/block.sh %s

...then that means I need to create a shell file. If you are suggesting that I have "echo $?" in the shell file, then I'm not sure how I would see the output. Doesn't the echo output go to the screen? If the shell program is running on the server, how do I see the output?

Title: Re: debug.log runCommand error
Post by: grue on March 11, 2015, 08:25:04 PM

as in, ssh into your server (I seriously hope you have ssh access)
type:

Code:

curl https://mydomain.com/blocknotify.php/?blockhash=00000000000000000dd3c33eaf1a2746ebe841418b98cb15f4a0bb71f3330d5e
echo $?

the echo on the last line is to check the exit status. I'm suspecting that the culprit is your script failing and thus returning -1 to bitcoind when it's executed.

Title: Re: debug.log runCommand error
Post by: jlp on March 11, 2015, 09:07:21 PM

Yes, I have SSH. Sorry, I didn't understand what you meant. I tried your suggestion and I can tell that it executed my PHP program, as I have code in it that sends me an email. However, SSH outputted the following:

$ curl https://mydomain.com/blocknotify.php/?blockhash=1 echo $?
curl: (6) Couldn't resolve host 'echo'
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://0/">here</a>.</p>
</body></html>
$

I don't know what that output means. Do you? Does this mean everything went as planned?

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 11, 2015, 10:09:54 PM

Code:

cat >~/myscript.sh <<EOF
#!/bin/bash
echo hi there, starting >/tmp/log
curl https://mydomain.com/blocknotify.php/?blockhash=42 >>/tmp/log 2>>/tmp/err
echo exit code is $? >>/tmp/log
EOF
$ chmod +x myscript.sh

Then "blocknotify=/home/<your user>/myscript.sh" in bitcoin.conf. After the next block, take a look to /tmp/log and /tmp/err.

Next step: feed the 42 from the "blocknotify=" line, use $@ in the script. Next step: call curl directly with no wrapper.

Title: Re: debug.log runCommand error
Post by: jlp on March 12, 2015, 12:00:16 AM

Quote from: dserrano5 on March 11, 2015, 10:09:54 PM

Code:

cat >~/myscript.sh <<EOF
#!/bin/bash
echo hi there, starting >/tmp/log
curl https://mydomain.com/blocknotify.php/?blockhash=42 >>/tmp/log 2>>/tmp/err
echo exit code is $? >>/tmp/log
EOF
$ chmod +x myscript.sh

Thanks for your suggestions. I created myscript.sh and changed bitcoin.conf as per your suggestions. Then I did this:

Code:

$ bitcoind stop
Bitcoin server stopping
$ bitcoind -daemon
Bitcoin server starting

$ bitcoind getinfo
{
    "version" : 90000,
    "protocolversion" : 70002,
    "walletversion" : 60000,
    "balance" : X.XXXXXX,
    "blocks" : 347231,
    "timeoffset" : 0,
    "connections" : 8,
    "proxy" : "",
    "difficulty" : 47427554950.64830017,
    "testnet" : false,
    "keypoololdest" : 1396812248,
    "keypoolsize" : 84,
    "paytxfee" : 0.00000000,
    "unlocked_until" : 0,
    "errors" : ""
}

When I noticed that "blocks" (from bitcoind getinfo) increased, I checked the /tmp folder. But this folder only had cookies.txt, but log and err do not exist.

I checked /home/myusername/.bitcoin/debug.log and saw this:

Code:

23:35:05 runCommand error: system(/home/myusername/myscript.sh) returned -1

Is it possible that there is something wrong in myscript.sh that is causing the runCommand error?

When you say "call curl directly with no wrapper", are you saying to call curl directly from bitcoin.conf without using myscript.sh?

Title: Re: debug.log runCommand error
Post by: cr1776 on March 12, 2015, 12:04:59 AM

Quote from: jlp on March 11, 2015, 09:07:21 PM

Try the curl command on one line, then the echo command on a second line as it was in grue's example.

It appears from the above that you did it all on one line.

Title: Re: debug.log runCommand error
Post by: jlp on March 12, 2015, 12:15:16 AM

Quote from: cr1776 on March 12, 2015, 12:04:59 AM

Try the curl command on one line, then the echo command on a second line as it was in grue's example.

It appears from the above that you did it all on one line.

Thanks for your keen observation. I indeed entered both commands on one line. Here's what I got now:

Code:

$ curl https://mydomain.com/blocknotify.php/?blockhash=1
$ echo $?
0
$

What does the 0 mean? Should I try putting back the following into bitcoin.conf?:

Code:

blocknotify=curl https://mydomain.com/blocknotify.php/?blockhash=%s

Title: Re: debug.log runCommand error
Post by: cr1776 on March 12, 2015, 05:03:39 PM

Quote from: jlp on March 12, 2015, 12:15:16 AM

Quote from: cr1776 on March 12, 2015, 12:04:59 AM

Try the curl command on one line, then the echo command on a second line as it was in grue's example.

It appears from the above that you did it all on one line.

Thanks for your keen observation. I indeed entered both commands on one line. Here's what I got now:

Code:

$ curl https://mydomain.com/blocknotify.php/?blockhash=1
$ echo $?
0
$

What does the 0 mean? Should I try putting back the following into bitcoin.conf?:

Code:

blocknotify=curl https://mydomain.com/blocknotify.php/?blockhash=%s

0 should mean it returned successfully.

If dserrano5's changes didn't create files in /tmp, that is odd unless there are permission problems there.

Title: Re: debug.log runCommand error
Post by: jlp on March 12, 2015, 05:26:53 PM

Quote from: cr1776 on March 12, 2015, 05:03:39 PM

If dserrano5's changes didn't create files in /tmp, that is odd unless there are permission problems there.

I got the following:

Code:

$ ls -ld /tmp
drwxrwxrwt 4 root root 4096 Mar 12 17:17 /tmp
$

Am I correct to assume that the permissions for /tmp are fine?

Do I need to change something in ~/myscript.sh in order to enable it to write to /tmp ?

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 12, 2015, 09:02:02 PM

You can try "blocknotify=/bin/true". Nothing should happen, not even an error in your log ;).

Then copy /bin/true to your home. "blocknotify=/home/<your username>/true". Nothing should happen again, no errors either.

Another thing worth trying is "blocknotify=/bin/touch /tmp/foo" to weed out problems with spaces and that. After the next block /tmp/foo should exist, and of course no errors in the log.

Title: Re: debug.log runCommand error
Post by: jlp on March 12, 2015, 10:26:03 PM

Quote from: dserrano5 on March 12, 2015, 09:02:02 PM

Thanks for your suggestions.

I tried "blocknotify=/bin/true". I got the following in /home/username/.bitcoin/debug.log:

Code:

2015-03-12 22:12:47 runCommand error: system(/bin/true) returned -1

What does this error mean?

I'm not exactly sure what you mean by "Then copy /bin/true to your home". I assume that you simply want me to try "blocknotify=/home/<your username>/true". I haven't tried this nor your suggestion with /tmp/foo, as I assumed that I'll get the runCommand error again. Let me know if I should anyways or if there isn't else I can try. Thanks.

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 12, 2015, 10:54:31 PM

Quote from: jlp on March 12, 2015, 10:26:03 PM

I tried "blocknotify=/bin/true". I got the following in /home/username/.bitcoin/debug.log:

Code:

2015-03-12 22:12:47 runCommand error: system(/bin/true) returned -1

Interesting. So you had this working for a year, then it suddenly broke "a few months ago" (as per the OP) to the point that not even a simple /bin/true works. Are you using apparmor o something similar?

Title: Re: debug.log runCommand error
Post by: cr1776 on March 12, 2015, 11:37:05 PM

Quote from: jlp on March 12, 2015, 10:26:03 PM

Quote from: dserrano5 on March 12, 2015, 09:02:02 PM

Thanks for your suggestions.

I tried "blocknotify=/bin/true". I got the following in /home/username/.bitcoin/debug.log:

Code:

2015-03-12 22:12:47 runCommand error: system(/bin/true) returned -1

/bin/true should return a 0
/bin/false should return a non-zero, typically 1 (or -1)
e.g.

Code:

 
me@mine:~# /bin/true
me@mine:~# echo $?
0
me@mine:~# /bin/false
me@mine:~# echo $?
1

As dserrano5 said, it seems that "not even a simple /bin/true works."

Try what I have as code above. Do you get something similar to the output above?

/bin/true and /bin/false are useful for testing your logic when needed since you always know the result.

Title: Re: debug.log runCommand error
Post by: cr1776 on March 12, 2015, 11:39:24 PM

Quote from: jlp on March 12, 2015, 12:15:16 AM

Quote from: cr1776 on March 12, 2015, 12:04:59 AM

Try the curl command on one line, then the echo command on a second line as it was in grue's example.

It appears from the above that you did it all on one line.

Thanks for your keen observation. I indeed entered both commands on one line. Here's what I got now:

Code:

$ curl https://mydomain.com/blocknotify.php/?blockhash=1
$ echo $?
0
$

What does the 0 mean? Should I try putting back the following into bitcoin.conf?:

Code:

blocknotify=curl https://mydomain.com/blocknotify.php/?blockhash=%s

Just to respond once more to this, the 0 should indicate that it returned success (like /bin/true), or no error, which is a good thing.

Title: Re: debug.log runCommand error
Post by: jlp on March 13, 2015, 12:49:19 AM

Quote from: dserrano5 on March 12, 2015, 10:54:31 PM

I just checked my system and I have it:

/etc/apparmo
/etc/appmor.d

However, I have never seen it anywhere before you mentioned it. Should I try changing something in it?

Title: Re: debug.log runCommand error
Post by: jlp on March 13, 2015, 12:56:49 AM

Quote from: cr1776 on March 12, 2015, 11:37:05 PM

Code:

 
me@mine:~# /bin/true
me@mine:~# echo $?
0
me@mine:~# /bin/false
me@mine:~# echo $?
1

I got the same output as you did:

Code:

$ /bin/true
$ echo $?
0
$ /bin/false
$ echo $?
1
$

I'm still getting the following in /home/username/.bitcoin/debug.log:

Code:

2015-03-13 00:41:32 runCommand error: system(/bin/true) returned -1

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 13, 2015, 07:20:13 AM

Quote from: jlp on March 13, 2015, 12:56:49 AM

I'm still getting the following in /home/username/.bitcoin/debug.log:

Code:

2015-03-13 00:41:32 runCommand error: system(/bin/true) returned -1

That has nothing to do with the result of the program. The error means that bitcoin core is unable to run /bin/true and that's the problem you have to tackle.

Title: Re: debug.log runCommand error
Post by: jlp on March 13, 2015, 03:57:57 PM

Quote from: dserrano5 on March 13, 2015, 07:20:13 AM

That has nothing to do with the result of the program. The error means that bitcoin core is unable to run /bin/true and that's the problem you have to tackle.

Thanks for your help. Do you have any ideas on what I can try next?

Title: Re: debug.log runCommand error
Post by: cr1776 on March 13, 2015, 04:34:57 PM

Quote from: jlp on March 13, 2015, 03:57:57 PM

Quote from: dserrano5 on March 13, 2015, 07:20:13 AM

That has nothing to do with the result of the program. The error means that bitcoin core is unable to run /bin/true and that's the problem you have to tackle.

Thanks for your help. Do you have any ideas on what I can try next?

In this thread:
https://bitcointalk.org/index.php?topic=385146.msg4154179#msg4154179

there was an error elsewhere in the config file that was messing things up. Perhaps this is a symptom of a different problem that is manifesting itself in an odd way?

Title: Re: debug.log runCommand error
Post by: jlp on March 13, 2015, 07:27:20 PM

Quote from: cr1776 on March 13, 2015, 04:34:57 PM

Thanks for your suggestion.

Automatic had the following in his bitcoin.conf. However, he stated that the notify commands randomly started working and randomly stopped working.

Code:

rpcuser=no.
rpcpassword=no.
maxconnections=1000
checklevel=4
keypool=10000
rpcallowip=127.0.0.1
server=1
blocknotify=/home/bitcoin/block.sh %s
walletnotify=/home/bitcoin/wallet.sh %s
alertnotify=/home/bitcoin/alert.sh %s

I tried the following configs in my bitcoin.conf:

Code:

rpcuser=XXXXXXXXXX
rpcpassword=XXXXXXXXXX
maxconnections=1000
checklevel=4
keypool=10000
rpcallowip=127.0.0.1
server=1
blocknotify=/bin/true
walletnotify=/bin/true
alertnotify=/bin/true

and

Code:

rpcuser=XXXXXXXXXX
rpcpassword=XXXXXXXXXX
blocknotify=/bin/true
walletnotify=/bin/true

After restarting bitcoind each time, I'm still getting the same error in .bitcoin/debug.log:

Code:

2015-03-13 18:45:17 runCommand error: system(/bin/true) returned -1

Any other suggestions that I can try?

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 13, 2015, 10:15:47 PM

Ahem:

Stop bitcoind.
Configure it again to run /bin/true.
Run bitcoind with strace, eg. 'strace -o mytrace -s 99 -ff bitcoind &'. This assumes you usually run bitcoind by simply typing 'bitcoind'. If you run it some other way (for example './bitcoind' or '/path/to/bitcoind', substitute the "bitcoind" at the end accordingly. You may have to install strace first.
Wait for the next block. Find the error in the log.
Stop bitcoind.
Run bitcoind as usual (without strace) so you contribute to the network.
Examine the mytrace.* files you'll find in the current directory. A quick way is 'grep -C 2 bin/true mytrace*'. Post your findings/the output of the grep.

Title: Re: debug.log runCommand error
Post by: jlp on March 14, 2015, 04:03:50 PM

Quote from: dserrano5 on March 13, 2015, 10:15:47 PM

Ahem:

Stop bitcoind.
Configure it again to run /bin/true.
Run bitcoind with strace, eg. 'strace -o mytrace -s 99 -ff bitcoind &'. This assumes you usually run bitcoind by simply typing 'bitcoind'. If you run it some other way (for example './bitcoind' or '/path/to/bitcoind', substitute the "bitcoind" at the end accordingly. You may have to install strace first.
Wait for the next block. Find the error in the log.
Stop bitcoind.
Run bitcoind as usual (without strace) so you contribute to the network.
Examine the mytrace.* files you'll find in the current directory. A quick way is 'grep -C 2 bin/true mytrace*'. Post your findings/the output of the grep.

Here it is:

Code:

$ ls
bitcoin.conf	   debug.log	     mytrace.5729  mytrace.5741  mytrace.5746  mytrace.5751  peers.dat
debug.log.save     mytrace.5730      mytrace.5742  mytrace.5747  mytrace.5752  tempcron
blocks		   debug.log.save.1  mytrace.5731  mytrace.5743  mytrace.5748  mytrace.5753 
chainstate	   mytrace.5727      mytrace.5732  mytrace.5744  mytrace.5749  mytrace.5754
db.log		   mytrace.5728      mytrace.5733  mytrace.5745  mytrace.5750  mytrace.5773
$ grep -C 2 bin/true mytrace.*
mytrace.5773-rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
mytrace.5773-write(4, "2015-03-14 15:45:51 ", 20)    = 20
mytrace.5773:write(4, "runCommand error: system(/bin/true) returned -1\n", 48) = 48
mytrace.5773-madvise(0x7fe25effe000, 8368128, MADV_DONTNEED) = 0
mytrace.5773-_exit(0)                                = ?
$ bitcoind -daemon
Bitcoin server starting
$

I am still running bitcoind version 0.9.0. Will this be a factor? Do I need to upgrade to 0.10.0?

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 14, 2015, 08:48:46 PM

Hmm, nothing there. What if you 'grep execve' or maybe just 'grep exec'?

Title: Re: debug.log runCommand error
Post by: Cryptowatch.com on March 14, 2015, 09:08:32 PM

Apparmor was mentioned.

Some info:
http://en.wikipedia.org/wiki/AppArmor
http://manpages.ubuntu.com/manpages/precise/en/man7/apparmor.7.html

I'm unfamiliar with apparmor, but I think these suggestions might make sense:

If you have root access, try disable or remove apparmor entirely. If block and wallet-notifications now triggers correctly, this was the culprint, and you might decide to have it uninstalled, or you might decide to have it installed but alter it's configuration to suit your needs. From what I understand from the docs, it gives access permissions on application level, rather than user level.

As for how to complete these steps, it's an exercise for you. Good luck.

Title: Re: debug.log runCommand error
Post by: cr1776 on March 14, 2015, 11:58:29 PM

Try looking into AppArmour and see if it could be the culprit. I think we're running out of Bitcoin related things. :-)

As cryptowatch.com said, perhaps it is interfering with app permissions.

Quote from: jlp on March 13, 2015, 07:27:20 PM

Quote from: cr1776 on March 13, 2015, 04:34:57 PM

Thanks for your suggestion.

Automatic had the following in his bitcoin.conf. However, he stated that the notify commands randomly started working and randomly stopped working.

Code:

rpcuser=no.
rpcpassword=no.
maxconnections=1000
checklevel=4
keypool=10000
rpcallowip=127.0.0.1
server=1
blocknotify=/home/bitcoin/block.sh %s
walletnotify=/home/bitcoin/wallet.sh %s
alertnotify=/home/bitcoin/alert.sh %s

I tried the following configs in my bitcoin.conf:

Code:

rpcuser=XXXXXXXXXX
rpcpassword=XXXXXXXXXX
maxconnections=1000
checklevel=4
keypool=10000
rpcallowip=127.0.0.1
server=1
blocknotify=/bin/true
walletnotify=/bin/true
alertnotify=/bin/true

and

Code:

rpcuser=XXXXXXXXXX
rpcpassword=XXXXXXXXXX
blocknotify=/bin/true
walletnotify=/bin/true

After restarting bitcoind each time, I'm still getting the same error in .bitcoin/debug.log:

Code:

2015-03-13 18:45:17 runCommand error: system(/bin/true) returned -1

Any other suggestions that I can try?

Title: Re: debug.log runCommand error
Post by: jlp on March 15, 2015, 12:12:31 AM

Quote from: dserrano5 on March 14, 2015, 08:48:46 PM

Hmm, nothing there. What if you 'grep execve' or maybe just 'grep exec'?

Is this what you are asking for?:

Code:

$ grep -C 2 exec mytrace.*
mytrace.5727:execve("/usr/local/bin/bitcoind", ["bitcoind"], [/* 13 vars */]) = 0
mytrace.5727-brk(0)                                  = 0x7fe2a93af000
mytrace.5727-access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
$

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 15, 2015, 01:21:52 AM

Quote from: jlp on March 15, 2015, 12:12:31 AM

Is this what you are asking for?:

Yes, but the output keeps being unexpected. Look at mine (there are several blocks here, not only one):

Code:

$ grep bin.true footreis.*
footreis.13987:01:05:38.978635 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.13988:01:05:38.988687 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0
footreis.2739:01:12:16.482455 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.2740:01:12:16.491455 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0
footreis.30087:01:00:44.489005 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.30088:01:00:44.542551 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0
footreis.31270:01:03:26.618716 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.31271:01:03:26.627652 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0

Like I and others mentioned, look into apparmor.

Title: Re: debug.log runCommand error
Post by: jlp on March 15, 2015, 01:29:57 AM

Quote from: Cryptowatch.com on March 14, 2015, 09:08:32 PM

Thanks for your suggestion.

Apparmor's profiles are supposed to be in the following directory. A profile does not exist for bitcoind:

Code:

$ ls /etc/apparmor.d/
abstractions  disable	      local	     tunables	      usr.sbin.ntpd	 usr.sbin.tcpdump
cache	      force-complain  sbin.dhclient  usr.sbin.mysqld  usr.sbin.rsyslogd

Nevertheless, I tried stopping apparmor. As per http://www.techytalk.info/disable-and-remove-apparmor-on-ubuntu-based-linux-distributions/ , I did the following:

Code:

$ sudo /etc/init.d/apparmor stop
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."

But it didn't seem to stop when I checked the status:

Code:

$ sudo /etc/init.d/apparmor status
apparmor module is loaded.
6 profiles are loaded.
6 profiles are in enforce mode.
   /sbin/dhclient
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/mysqld
   /usr/sbin/ntpd
   /usr/sbin/tcpdump
0 profiles are in complain mode.
2 processes have profiles defined.
2 processes are in enforce mode.
   /usr/sbin/mysqld (837) 
   /usr/sbin/ntpd (1604) 
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

So, as per https://help.ubuntu.com/lts/serverguide/apparmor.html , I tried the following to stop AppArmor:

Code:

$ sudo service apparmor stop
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
$ sudo update-rc.d -f apparmor remove
 Removing any system startup links for /etc/init.d/apparmor ...
   /etc/rcS.d/S37apparmor

When I checked apparmor status, I got the same as before.

I tried to set bitcoind to complain mode, but it didn't work:

Code:

$ sudo aa-complain /usr/local/bin/bitcoind
sudo: aa-complain: command not found
$

As per https://help.ubuntu.com/community/AppArmor , I did the following to disable AppArmor:

Code:

$ sudo invoke-rc.d apparmor stop
[sudo] password for stockbet: 
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
$ sudo update-rc.d -f apparmor remove
 Removing any system startup links for /etc/init.d/apparmor ...

When I checked apparmor status, I got the same as before.

I stopped and started bitcoind:

Code:

$ bitcoind stop
Bitcoin server stopping
$ bitcoind -daemon
Bitcoin server starting
$

…but I still got the runCommand error in .bitcoin/debug.log:

Code:

2015-03-15 01:18:12 runCommand error: system(/bin/true) returned -1

I have bitcoind 0.9.0. Do I need upgrade it? I read that version 0.10.0 causes frozen blocks.

Title: Re: debug.log runCommand error
Post by: jlp on March 15, 2015, 01:40:41 AM

Quote from: dserrano5 on March 15, 2015, 01:21:52 AM

Yes, but the output keeps being unexpected. Look at mine (there are several blocks here, not only one):

Shortly after starting up bitcoind, I checked debug.log. When I saw the runCommand error, I stopped bitcoind and ran grep. Doesn't this explain why there is only one block in my mytrace files? If I had let bitcoind run for a while before running grep, there would've been several more invocations of blocknotify and therefore several more occurrences of the runCommand error. Doesn't this explain it?

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 15, 2015, 01:46:48 AM

Quote from: jlp on March 15, 2015, 01:40:41 AM

Quote from: dserrano5 on March 15, 2015, 01:21:52 AM

Yes, but the output keeps being unexpected. Look at mine (there are several blocks here, not only one):

Shortly after starting up bitcoind, I checked debug.log. When I saw the runCommand error, I stopped bitcoind and ran grep. Doesn't this explain why there is only one block in my mytrace files?

Yes of course. My parenthesis remark is only to explain why there's more than one pair execve(sh),execve(true). Of course, waiting for one block and actually being there to stop bitcoind will result in only one block :). It's only, I was doing other stuff and forgot about bitcoind so several blocks were added to the chain in the meanwhile.

Title: Re: debug.log runCommand error
Post by: jlp on March 15, 2015, 03:48:55 PM

Quote from: dserrano5 on March 15, 2015, 01:46:48 AM

Sorry for my slowness. Is there anything else that I'm not reading correctly? What part of my output is unexpected, as compared to your output?

Thanks for hanging in here to help me out.

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 15, 2015, 04:03:29 PM

Quote from: jlp on March 15, 2015, 03:48:55 PM

What part of my output is unexpected, as compared to your output?

I was expecting you also had the execve("/bin/true") line, possibly returning -1. The fact that it isn't there means that bitcoind isn't trying to run /bin/true, ie it's finding an error at an earlier stage.

F*ck it, UNINSTALL APPARMOR and try again!

Title: Re: debug.log runCommand error
Post by: Cryptowatch.com on March 15, 2015, 05:47:09 PM

Quote from: dserrano5 on March 15, 2015, 04:03:29 PM

Quote from: jlp on March 15, 2015, 03:48:55 PM

What part of my output is unexpected, as compared to your output?

Some more exercises:

a. Find out the name of the package(s) that contains apparmor on your system.
b. List all files associated with those packages on your system. Make a note of those.
c. Remove apparmor with something like

Code:

apt-get remove apparmor

.
d. ensure it's not running with

Code:

ps aux | grep apparmor

e. verify that the files in b. is now gone.

Check if bitcoind now works as it should..

It might be that apparmour applied a default profile to applications not specifically listed with a profile. I'm about 100% sure that if you remove apparmor as dserrano5 says, everything will work just fine.

Are you running ubuntu? Could it be that apparmor was added with a recent update?

I also found this link, that might be worth reading:
http://ubuntuforums.org/showthread.php?t=1008906

Title: Re: debug.log runCommand error
Post by: jlp on March 15, 2015, 09:23:26 PM

I'm not exactly sure I know what you mean by:

Quote from: Cryptowatch.com on March 15, 2015, 05:47:09 PM

a. Find out the name of the package(s) that contains apparmor on your system.
b. List all files associated with those packages on your system. Make a note of those.

https://wiki.ubuntu.com/DebuggingApparmor seems to shed some light on "package". Based on this, I did the following:

Code:

$ cd /etc/apparmor.d
$ ls
abstractions  force-complain  tunables	       usr.sbin.rsyslogd
cache	      local	      usr.sbin.mysqld  usr.sbin.tcpdump
disable       sbin.dhclient   usr.sbin.ntpd
$ dpkg -S sbin.dhclient
isc-dhcp-client: /etc/apparmor.d/sbin.dhclient
$ dpkg -S usr.sbin.mysqld
mysql-server-5.5: /etc/apparmor.d/usr.sbin.mysqld
$ dpkg -S usr.sbin.ntpd
ntp: /etc/apparmor.d/usr.sbin.ntpd
ntp: /etc/apparmor/init/network-interface-security/usr.sbin.ntpd
$ dpkg -S usr.sbin.rsyslogd
rsyslog: /etc/apparmor.d/usr.sbin.rsyslogd
$ dpkg -S usr.sbin.tcpdump
tcpdump: /etc/apparmor.d/usr.sbin.tcpdump

Note: the following from /etc/apparmor.d are folders:

abstractions
cache
disable
force-complain
local
tunables

I looked inside each of the above folders and did not see "bitcoin". The "abstractions" folder had approximately 73 files. There were a few files in the other folders. Did you want me to find the package for all these as well?

Yes, I'm running Ubuntu. I don't know if AppArmor was added with a recent update. I read somewhere that AppArmor comes with Ubuntu.

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 15, 2015, 09:52:52 PM

Quote from: jlp on March 15, 2015, 09:23:26 PM

I'm not exactly sure I know what you mean by:

Quote from: Cryptowatch.com on March 15, 2015, 05:47:09 PM

a. Find out the name of the package(s) that contains apparmor on your system.
b. List all files associated with those packages on your system. Make a note of those.

b. and e. (not quoted) together are just a paranoid way of making totally sure that apparmor isn't installed. I'd go with simply removing the packages apparmor-* and reboot (rebooting is the easiest way to ensure it's not loaded; inelegant, I know, but meh).

Title: Re: debug.log runCommand error
Post by: Cryptowatch.com on March 16, 2015, 03:32:44 AM

Quote from: dserrano5 on March 15, 2015, 09:52:52 PM

Yes, that's sound advice.

jlp, as for package:

-- https://en.wikipedia.org/wiki/Linux_package_formats

Packages on a linux systems is usually administrated by various tools, but the more usual ones are apt-get and dpkg.

Learn more:

-- http://www.cyberciti.biz/tips/linux-debian-package-management-cheat-sheet.html

So, when I suggest "a. Find out the name of the package(s) that contains apparmor on your system.", the way to go about solving that task is to understand what to do, then figure out how to do it.

If you don't know what a package is, you search for "What is a package in linux?" The first hit on duckduckgo.com is the wikipedia link a few lines above. Now you know what a package is, and you need to find all packages associated with apparmor.

If you search a bit more, for instance on stackoverflow.com, you will find postings like:

http://stackoverflow.com/questions/12740452/ubuntu-command-to-list-the-installed-software

From there you can learn you can run a command like

Code:

sudo dpkg --get-selections

to view all packages. Now you could either save the output to a text file by doing

Code:

sudo dpkg --get-selections > list.txt

and then look at list.txt for apparmor entries, or you could run a command like

Quote

sudo dpkg --get-selections | grep apparmor

It's also possible to user apt-cache:

-- http://www.howtogeek.com/howto/ubuntu/search-for-install-packages-from-the-ubuntu-command-line/?PageSpeed=noscript

Or you can go directly to Ubuntu, and search for packages:

-- http://packages.ubuntu.com/

When I search for apparmor there, I get this result:
-- http://packages.ubuntu.com/search?keywords=apparmor&searchon=names&suite=trusty&section=all

As exact hit, I get apparmor.

This means it should be possible to uninstall it with

Code:

sudo apt-get remove apparmor

.

Then to ensure, the system is in a "fresh" state, the easiest thing is to just reboot. I assume that apparmor is the main package for apparmor daemon, and once it's removed, bitcoind should function normally for you again.

Linux can be a bit complicated, so you need to be patient, and spend time learning.

If you do not understand a sentence posing a problem then break the problem down in sub-parts, and make sure you understand every sub-part, if there's a word you do not understand, google it. Once you understand the problem, finding a solution is much simpler. Just trying stuff blindly, seldom works.

There's plenty of people around here wanting to help you, but it's also important that you demonstrate that you've done a real effort to try solve the problem. But don't worry, problem solving skills is something that can be improved.

Let us know how this turns out, I'm sure everyone rots for you to solve the issue with bitcoind not running the external scripts for wallet and blocknotify events.

Title: Re: debug.log runCommand error
Post by: jlp on March 16, 2015, 02:22:01 PM

dserrano5:

I did the following

Code:

$ sudo apt-get remove apparmor
[sudo] password for stockbet: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libxxf86dga1 x11-utils libxxf86vm1 xterm libgl1-mesa-dri libxcb-glx0 libgl1-mesa-glx
  libx11-xcb1 libglapi-mesa xbitmaps libxaw7 linux-headers-3.5.0-23-generic libxmu6
  linux-headers-3.5.0-23 libfontenc1 libxtst6 libutempter0 tk8.5 libxcb-shape0 libxv1
  libllvm3.0
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  apparmor
0 upgraded, 0 newly installed, 1 to remove and 55 not upgraded.
After this operation, 1,081 kB disk space will be freed.
Do you want to continue [Y/n]? Y
(Reading database ... 257432 files and directories currently installed.)
Removing apparmor ...
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
Processing triggers for man-db ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot

Should I run 'apt-get autoremove' to remove the above packages that are no longer required?

Code:

$ ps aux | grep apparmor
username  9244  0.0  0.0   9384   936 pts/0    S+   23:29   0:00 grep apparmor

As per http://stackoverflow.com/questions/9375711/more-elegant-ps-aux-grep-v-grep , grep is included in the results, and so I did the following as well, which returned nothing:

Code:

$ ps aux | grep '[a]pparmor'
$

I rebooted.

blocknotify is now working!!

Cryptowatch.com:

I had done the above already before I saw your posting.

I ran these commands:

Code:

$ sudo dpkg --get-selections | grep apparmor
apparmor					deinstall
dh-apparmor					install
$ 

$ apt-cache search apparmor
apparmor - User-space parser utility for AppArmor
apparmor-docs - Documentation for AppArmor
apparmor-notify - AppArmor notification system
apparmor-profiles - Profiles for AppArmor Security policies
apparmor-utils - Utilities for controlling AppArmor
dh-apparmor - AppArmor debhelper routines
libapache2-mod-apparmor - changehat AppArmor library as an Apache module
libapparmor-dev - AppArmor development libraries and header files
libapparmor-perl - AppArmor library Perl bindings
libapparmor1 - changehat AppArmor library
libpam-apparmor - changehat AppArmor library as a PAM module
python-libapparmor - AppArmor library Python bindings
$

To find the files installed with a package, I ran:

Code:

$ dpkg -L apparmor

From a glance, it seems that all of the files shown by the above command are still on my system, even though I had uninstalled apparmor. There are some files in /etc/apparmor.d/local/ that were not listed by the above command.

All of the files shown below still exist on my system:

Code:

$ dpkg -L dh-apparmor
/.
/usr
/usr/bin
/usr/bin/dh_apparmor
/usr/share
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/dh_apparmor.1.gz
/usr/share/doc
/usr/share/doc/dh-apparmor
/usr/share/doc/dh-apparmor/changelog.Debian.gz
/usr/share/doc/dh-apparmor/copyright
/usr/share/debhelper
/usr/share/debhelper/autoscripts
/usr/share/debhelper/autoscripts/postrm-apparmor
/usr/share/debhelper/autoscripts/postinst-apparmor
$

I don't know if any of the above files matter anymore because blocknotify now works.

dserrano5 and Cryptowatch.com:

Thank you so much for your help.

What about AppArmor? Without it, isn't my system insecure?

Is there any way to find out the offending AppArmor profile and to fix it? I googled (startpage.com) for "default profile in apparmor" and cannot find much, other than to assume that apparmor runs the profiles in /etc/apparmor.d. I looked at some of the profiles in /etc/apparmor.d and cannot figure out how any of them can interfere with bitcoind.

I didn't upgrade my operating system. I didn't install AppArmor. What do you think prompted AppArmor to cause the problem?

Title: Re: debug.log runCommand error
Post by: cr1776 on March 16, 2015, 03:12:57 PM

Quote from: jlp on March 16, 2015, 02:22:01 PM

...
blocknotify is now working!!
...

Glad that is now working.

The odd thing to me is that apparmor is (at least on my three servers - one bitcoin related, two not) there by default (Ubuntu 14.04 on them). But it causes no problems and I have no profile for bitcoind.

I am not sure if I would auto remove it. Since it is part of the mainline, now that you know it was the problem, you might want to see if you can figure out why it was causing an issue and fix it that way.

Ubuntu? Debian? version?

e.g.

Quote

AppArmor is an established technology first seen in Immunix and later integrated into Ubuntu, Novell/SUSE, and Mandriva. Core AppArmor functionality is in the mainline Linux kernel from 2.6.36 onwards; work is ongoing by AppArmor, Ubuntu and other developers to merge additional AppArmor functionality into the mainline kernel.
https://wiki.ubuntu.com/AppArmor

Title: Re: debug.log runCommand error
Post by: Cryptowatch.com on March 16, 2015, 03:50:34 PM

jlp, I am very glad to hear everything now works, as I suspected.

The remaining apparmor files listed on the system should give no issues.

As to whether your system is insecure without apparmor? Every program run as a user has the rights of that user, which has been normal linux behaviour for ages, so I wouldn'ẗ worry too much.

If you worry, you should reinstall apparmor, and then explicitly make an application profile or allow it the required access.

But if everything works now, I'd just leave it alone for now..

Great to see the issue solved!

Title: Re: debug.log runCommand error
Post by: jlp on March 16, 2015, 04:11:55 PM

Quote from: cr1776 on March 16, 2015, 03:12:57 PM

Ubuntu? Debian? version?

Ubuntu 12.04

Title: Re: debug.log runCommand error
Post by: cr1776 on March 16, 2015, 05:06:04 PM

Quote from: jlp on March 16, 2015, 04:11:55 PM

Quote from: cr1776 on March 16, 2015, 03:12:57 PM

Ubuntu? Debian? version?

Ubuntu 12.04

I wonder if there is something different happening between 12.04 or 14.04. Odd.

At least it is working.

Title: Re: debug.log runCommand error
Post by: dserrano5 on March 16, 2015, 07:03:33 PM

Quote from: jlp on March 16, 2015, 02:22:01 PM

Should I run 'apt-get autoremove' to remove the above packages that are no longer required?

Every piece of software that you have installed is a potential security hole.

The packages in that list were installed in the past as dependencies of some other software you installed. Later you removed this software but the dependencies remain there. Probably you aren't using those, so it's safe to remove them.

Quote from: jlp on March 16, 2015, 02:22:01 PM

blocknotify is now working!!

Glad to hear that!

Quote from: jlp on March 16, 2015, 02:22:01 PM

Code:

$ dpkg -L dh-apparmor
/.
/usr
/usr/bin
/usr/bin/dh_apparmor
[...]

dh-apparmor is a tool for debian developers ("dh" = "debian helper") and has nothing to do with the actual execution of apparmor.