debug.log runCommand error

[cr1776]

That has nothing to do with the result of the program. The error means that bitcoin core is unable to run /bin/true and that's the problem you have to tackle.

Thanks for your help.  Do you have any ideas on what I can try next?

In this thread:
https://bitcointalk.org/index.php?topic=385146.msg4154179#msg4154179

there was an error elsewhere in the config file that was messing things up.  Perhaps this is a symptom of a different problem that is manifesting itself in an odd way?

[jlp]

In this thread:
https://bitcointalk.org/index.php?topic=385146.msg4154179#msg4154179

there was an error elsewhere in the config file that was messing things up.  Perhaps this is a symptom of a different problem that is manifesting itself in an odd way?

Thanks for your suggestion.

Automatic had the following in his bitcoin.conf.  However, he stated that the notify commands randomly started working and randomly stopped working.

Code:

rpcuser=no.
rpcpassword=no.
maxconnections=1000
checklevel=4
keypool=10000
rpcallowip=127.0.0.1
server=1
blocknotify=/home/bitcoin/block.sh %s
walletnotify=/home/bitcoin/wallet.sh %s
alertnotify=/home/bitcoin/alert.sh %s

I tried the following configs in my bitcoin.conf:

Code:

rpcuser=XXXXXXXXXX
rpcpassword=XXXXXXXXXX
maxconnections=1000
checklevel=4
keypool=10000
rpcallowip=127.0.0.1
server=1
blocknotify=/bin/true
walletnotify=/bin/true
alertnotify=/bin/true

and

Code:

rpcuser=XXXXXXXXXX
rpcpassword=XXXXXXXXXX
blocknotify=/bin/true
walletnotify=/bin/true

After restarting bitcoind each time, I'm still getting the same error in .bitcoin/debug.log:

Code:

2015-03-13 18:45:17 runCommand error: system(/bin/true) returned -1

Any other suggestions that I can try?

[dserrano5]

Ahem:

• Stop bitcoind.
• Configure it again to run /bin/true.
• Run bitcoind with strace, eg. 'strace -o mytrace -s 99 -ff bitcoind &'. This assumes you usually run bitcoind by simply typing 'bitcoind'. If you run it some other way (for example './bitcoind' or '/path/to/bitcoind', substitute the "bitcoind" at the end accordingly. You may have to install strace first.
• Wait for the next block. Find the error in the log.
• Stop bitcoind.
• Run bitcoind as usual (without strace) so you contribute to the network.
• Examine the mytrace.* files you'll find in the current directory. A quick way is 'grep -C 2 bin/true mytrace*'. Post your findings/the output of the grep.

[jlp]

Ahem:

• Stop bitcoind.
• Configure it again to run /bin/true.
• Run bitcoind with strace, eg. 'strace -o mytrace -s 99 -ff bitcoind &'. This assumes you usually run bitcoind by simply typing 'bitcoind'. If you run it some other way (for example './bitcoind' or '/path/to/bitcoind', substitute the "bitcoind" at the end accordingly. You may have to install strace first.
• Wait for the next block. Find the error in the log.
• Stop bitcoind.
• Run bitcoind as usual (without strace) so you contribute to the network.
• Examine the mytrace.* files you'll find in the current directory. A quick way is 'grep -C 2 bin/true mytrace*'. Post your findings/the output of the grep.

Here it is:

Code:

$ ls
bitcoin.conf	   debug.log	     mytrace.5729  mytrace.5741  mytrace.5746  mytrace.5751  peers.dat
debug.log.save     mytrace.5730      mytrace.5742  mytrace.5747  mytrace.5752  tempcron
blocks		   debug.log.save.1  mytrace.5731  mytrace.5743  mytrace.5748  mytrace.5753 
chainstate	   mytrace.5727      mytrace.5732  mytrace.5744  mytrace.5749  mytrace.5754
db.log		   mytrace.5728      mytrace.5733  mytrace.5745  mytrace.5750  mytrace.5773
$ grep -C 2 bin/true mytrace.*
mytrace.5773-rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
mytrace.5773-write(4, "2015-03-14 15:45:51 ", 20)    = 20
mytrace.5773:write(4, "runCommand error: system(/bin/true) returned -1\n", 48) = 48
mytrace.5773-madvise(0x7fe25effe000, 8368128, MADV_DONTNEED) = 0
mytrace.5773-_exit(0)                                = ?
$ bitcoind -daemon
Bitcoin server starting
$ 

I am still running bitcoind version 0.9.0.  Will this be a factor?  Do I need to upgrade to 0.10.0?

[dserrano5]

Hmm, nothing there. What if you 'grep execve' or maybe just 'grep exec'?

[Cryptowatch.com]

Apparmor was mentioned.

Some info:
http://en.wikipedia.org/wiki/AppArmor
http://manpages.ubuntu.com/manpages/precise/en/man7/apparmor.7.html

I'm unfamiliar with apparmor, but I think these suggestions might make sense:

If you have root access, try disable or remove apparmor entirely. If block and wallet-notifications now triggers correctly, this was the culprint, and you might decide to have it uninstalled, or you might decide to have it installed but alter it's configuration to suit your needs. From what I understand from the docs, it gives access permissions on application level, rather than user level.

As for how to complete these steps, it's an exercise for you. Good luck.

[cr1776]

Try looking into AppArmour and see if it could be the culprit.  I think we're running out of Bitcoin related things. :-)

As cryptowatch.com said, perhaps it is interfering with app permissions.

In this thread:
https://bitcointalk.org/index.php?topic=385146.msg4154179#msg4154179

there was an error elsewhere in the config file that was messing things up.  Perhaps this is a symptom of a different problem that is manifesting itself in an odd way?

Thanks for your suggestion.

Automatic had the following in his bitcoin.conf.  However, he stated that the notify commands randomly started working and randomly stopped working.

Code:

rpcuser=no.
rpcpassword=no.
maxconnections=1000
checklevel=4
keypool=10000
rpcallowip=127.0.0.1
server=1
blocknotify=/home/bitcoin/block.sh %s
walletnotify=/home/bitcoin/wallet.sh %s
alertnotify=/home/bitcoin/alert.sh %s

I tried the following configs in my bitcoin.conf:

Code:

rpcuser=XXXXXXXXXX
rpcpassword=XXXXXXXXXX
maxconnections=1000
checklevel=4
keypool=10000
rpcallowip=127.0.0.1
server=1
blocknotify=/bin/true
walletnotify=/bin/true
alertnotify=/bin/true

and

Code:

rpcuser=XXXXXXXXXX
rpcpassword=XXXXXXXXXX
blocknotify=/bin/true
walletnotify=/bin/true

After restarting bitcoind each time, I'm still getting the same error in .bitcoin/debug.log:

Code:

2015-03-13 18:45:17 runCommand error: system(/bin/true) returned -1

Any other suggestions that I can try?

[jlp]

Hmm, nothing there. What if you 'grep execve' or maybe just 'grep exec'?

Is this what you are asking for?:

Code:

$ grep -C 2 exec mytrace.*
mytrace.5727:execve("/usr/local/bin/bitcoind", ["bitcoind"], [/* 13 vars */]) = 0
mytrace.5727-brk(0)                                  = 0x7fe2a93af000
mytrace.5727-access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
$

[dserrano5]

Is this what you are asking for?:

Yes, but the output keeps being unexpected. Look at mine (there are several blocks here, not only one):

Code:

$ grep bin.true footreis.*
footreis.13987:01:05:38.978635 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.13988:01:05:38.988687 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0
footreis.2739:01:12:16.482455 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.2740:01:12:16.491455 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0
footreis.30087:01:00:44.489005 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.30088:01:00:44.542551 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0
footreis.31270:01:03:26.618716 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.31271:01:03:26.627652 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0

Like I and others mentioned, look into apparmor.

[jlp]

Apparmor was mentioned.

Some info:
http://en.wikipedia.org/wiki/AppArmor
http://manpages.ubuntu.com/manpages/precise/en/man7/apparmor.7.html

I'm unfamiliar with apparmor, but I think these suggestions might make sense:

If you have root access, try disable or remove apparmor entirely. If block and wallet-notifications now triggers correctly, this was the culprint, and you might decide to have it uninstalled, or you might decide to have it installed but alter it's configuration to suit your needs. From what I understand from the docs, it gives access permissions on application level, rather than user level.

As for how to complete these steps, it's an exercise for you. Good luck.

Thanks for your suggestion.

Apparmor's profiles are supposed to be in the following directory.  A profile does not exist for bitcoind:

Code:

$ ls /etc/apparmor.d/
abstractions  disable	      local	     tunables	      usr.sbin.ntpd	 usr.sbin.tcpdump
cache	      force-complain  sbin.dhclient  usr.sbin.mysqld  usr.sbin.rsyslogd

Nevertheless, I tried stopping apparmor.  As per http://www.techytalk.info/disable-and-remove-apparmor-on-ubuntu-based-linux-distributions/ , I did the following:

Code:

$ sudo /etc/init.d/apparmor stop
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."

But it didn't seem to stop when I checked the status:

Code:

$ sudo /etc/init.d/apparmor status
apparmor module is loaded.
6 profiles are loaded.
6 profiles are in enforce mode.
   /sbin/dhclient
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/mysqld
   /usr/sbin/ntpd
   /usr/sbin/tcpdump
0 profiles are in complain mode.
2 processes have profiles defined.
2 processes are in enforce mode.
   /usr/sbin/mysqld (837) 
   /usr/sbin/ntpd (1604) 
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

So, as per https://help.ubuntu.com/lts/serverguide/apparmor.html , I tried the following to stop AppArmor:

Code:

$ sudo service apparmor stop
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
$ sudo update-rc.d -f apparmor remove
 Removing any system startup links for /etc/init.d/apparmor ...
   /etc/rcS.d/S37apparmor

When I checked apparmor status, I got the same as before.

I tried to set bitcoind to complain mode, but it didn't work:

Code:

$ sudo aa-complain /usr/local/bin/bitcoind
sudo: aa-complain: command not found
$ 

As per https://help.ubuntu.com/community/AppArmor , I did the following to disable AppArmor:

Code:

$ sudo invoke-rc.d apparmor stop
[sudo] password for stockbet: 
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
$ sudo update-rc.d -f apparmor remove
 Removing any system startup links for /etc/init.d/apparmor ...

When I checked apparmor status, I got the same as before.

I stopped and started bitcoind:

Code:

$ bitcoind stop
Bitcoin server stopping
$ bitcoind -daemon
Bitcoin server starting
$ 

…but I still got the runCommand error in .bitcoin/debug.log:

Code:

2015-03-15 01:18:12 runCommand error: system(/bin/true) returned -1

I have bitcoind 0.9.0.  Do I need upgrade it?  I read that version 0.10.0 causes frozen blocks.

[jlp]

Yes, but the output keeps being unexpected. Look at mine (there are several blocks here, not only one):

Shortly after starting up bitcoind, I checked debug.log.  When I saw the runCommand error, I stopped bitcoind and ran grep.  Doesn't this explain why there is only one block in my mytrace files?  If I had let bitcoind run for a while before running grep, there would've been several more invocations of blocknotify and therefore several more occurrences of the runCommand error.  Doesn't this explain it?

[dserrano5]

Yes, but the output keeps being unexpected. Look at mine (there are several blocks here, not only one):

Shortly after starting up bitcoind, I checked debug.log.  When I saw the runCommand error, I stopped bitcoind and ran grep.  Doesn't this explain why there is only one block in my mytrace files?

Yes of course. My parenthesis remark is only to explain why there's more than one pair execve(sh),execve(true). Of course, waiting for one block and actually being there to stop bitcoind will result in only one block . It's only, I was doing other stuff and forgot about bitcoind so several blocks were added to the chain in the meanwhile.

[jlp]

Yes of course. My parenthesis remark is only to explain why there's more than one pair execve(sh),execve(true). Of course, waiting for one block and actually being there to stop bitcoind will result in only one block . It's only, I was doing other stuff and forgot about bitcoind so several blocks were added to the chain in the meanwhile.

Sorry for my slowness.  Is there anything else that I'm not reading correctly?  What part of my output is unexpected, as compared to your output?

Thanks for hanging in here to help me out.

[dserrano5]

What part of my output is unexpected, as compared to your output?

I was expecting you also had the execve("/bin/true") line, possibly returning -1. The fact that it isn't there means that bitcoind isn't trying to run /bin/true, ie it's finding an error at an earlier stage.

F*ck it, UNINSTALL APPARMOR and try again!

[Cryptowatch.com]

What part of my output is unexpected, as compared to your output?

I was expecting you also had the execve("/bin/true") line, possibly returning -1. The fact that it isn't there means that bitcoind isn't trying to run /bin/true, ie it's finding an error at an earlier stage.

F*ck it, UNINSTALL APPARMOR and try again!

Some more exercises:

a. Find out the name of the package(s) that contains apparmor on your system.
b. List all files associated with those packages on your system. Make a note of those.
c. Remove apparmor with something like

Code:

apt-get remove apparmor

.
d. ensure it's not running with

Code:

ps aux | grep apparmor

e. verify that the files in b. is now gone.

Check if bitcoind now works as it should..

It might be that apparmour applied a default profile to applications not specifically listed with a profile. I'm about 100% sure that if you remove apparmor as dserrano5 says, everything will work just fine.

Are you running ubuntu? Could it be that apparmor was added with a recent update?

I also found this link, that might be worth reading:
http://ubuntuforums.org/showthread.php?t=1008906

[jlp]

I'm not exactly sure I know what you mean by:

a. Find out the name of the package(s) that contains apparmor on your system.
b. List all files associated with those packages on your system. Make a note of those.

https://wiki.ubuntu.com/DebuggingApparmor  seems to shed some light on "package".  Based on this, I did the following:

Code:

$ cd /etc/apparmor.d
$ ls
abstractions  force-complain  tunables	       usr.sbin.rsyslogd
cache	      local	      usr.sbin.mysqld  usr.sbin.tcpdump
disable       sbin.dhclient   usr.sbin.ntpd
$ dpkg -S sbin.dhclient
isc-dhcp-client: /etc/apparmor.d/sbin.dhclient
$ dpkg -S usr.sbin.mysqld
mysql-server-5.5: /etc/apparmor.d/usr.sbin.mysqld
$ dpkg -S usr.sbin.ntpd
ntp: /etc/apparmor.d/usr.sbin.ntpd
ntp: /etc/apparmor/init/network-interface-security/usr.sbin.ntpd
$ dpkg -S usr.sbin.rsyslogd
rsyslog: /etc/apparmor.d/usr.sbin.rsyslogd
$ dpkg -S usr.sbin.tcpdump
tcpdump: /etc/apparmor.d/usr.sbin.tcpdump

Note:  the following from /etc/apparmor.d are folders:

abstractions
cache
disable
force-complain
local
tunables

I looked inside each of the above folders and did not see "bitcoin".  The "abstractions" folder had approximately 73 files.  There were a few files in the other folders.  Did you want me to find the package for all these as well?

Yes, I'm running Ubuntu.  I don't know if AppArmor was added with a recent update.  I read somewhere that AppArmor comes with Ubuntu.

[dserrano5]

I'm not exactly sure I know what you mean by:

a. Find out the name of the package(s) that contains apparmor on your system.
b. List all files associated with those packages on your system. Make a note of those.

b. and e. (not quoted) together are just a paranoid way of making totally sure that apparmor isn't installed. I'd go with simply removing the packages apparmor-* and reboot (rebooting is the easiest way to ensure it's not loaded; inelegant, I know, but meh).

[Cryptowatch.com]

b. and e. (not quoted) together are just a paranoid way of making totally sure that apparmor isn't installed. I'd go with simply removing the packages apparmor-* and reboot (rebooting is the easiest way to ensure it's not loaded; inelegant, I know, but meh).

Yes, that's sound advice.

jlp, as for package:

-- https://en.wikipedia.org/wiki/Linux_package_formats

Packages on a linux systems is usually administrated by various tools, but the more usual ones are apt-get and dpkg.

Learn more:

-- http://www.cyberciti.biz/tips/linux-debian-package-management-cheat-sheet.html

So, when I suggest "a. Find out the name of the package(s) that contains apparmor on your system.", the way to go about solving that task is to understand what to do, then figure out how to do it.

If you don't know what a package is, you search for "What is a package in linux?" The first hit on duckduckgo.com is the wikipedia link a few lines above. Now you know what a package is, and you need to find all packages associated with apparmor.

If you search a bit more, for instance on stackoverflow.com, you will find postings like:

http://stackoverflow.com/questions/12740452/ubuntu-command-to-list-the-installed-software

From there you can learn you can run a command like

Code:

sudo dpkg --get-selections 

to view all packages. Now you could either save the output to a text file by doing

Code:

sudo dpkg --get-selections > list.txt

and then look at list.txt for apparmor entries, or you could run a command like

sudo dpkg --get-selections | grep apparmor

It's also possible to user apt-cache:

-- http://www.howtogeek.com/howto/ubuntu/search-for-install-packages-from-the-ubuntu-command-line/?PageSpeed=noscript

Or you can go directly to Ubuntu, and search for packages:

-- http://packages.ubuntu.com/

When I search for apparmor there, I get this result:
-- http://packages.ubuntu.com/search?keywords=apparmor&searchon=names&suite=trusty&section=all

As exact hit, I get apparmor.

This means it should be possible to uninstall it with

Code:

sudo apt-get remove apparmor

.

Then to ensure, the system is in a "fresh" state, the easiest thing is to just reboot. I assume that apparmor is the main package for apparmor daemon, and once it's removed, bitcoind should function normally for you again.

Linux can be a bit complicated, so you need to be patient, and spend time learning.

If you do not understand a sentence posing a problem then break the problem down in sub-parts, and make sure you understand every sub-part, if there's a word you do not understand, google it. Once you understand the problem, finding a solution is much simpler. Just trying stuff blindly, seldom works.

There's plenty of people around here wanting to help you, but it's also important that you demonstrate that you've done a real effort to try solve the problem. But don't worry, problem solving skills is something that can be improved.

Let us know how this turns out, I'm sure everyone rots for you to solve the issue with bitcoind not running the external scripts for wallet and blocknotify events.

[jlp]

dserrano5:

I did the following

Code:

$ sudo apt-get remove apparmor
[sudo] password for stockbet: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libxxf86dga1 x11-utils libxxf86vm1 xterm libgl1-mesa-dri libxcb-glx0 libgl1-mesa-glx
  libx11-xcb1 libglapi-mesa xbitmaps libxaw7 linux-headers-3.5.0-23-generic libxmu6
  linux-headers-3.5.0-23 libfontenc1 libxtst6 libutempter0 tk8.5 libxcb-shape0 libxv1
  libllvm3.0
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  apparmor
0 upgraded, 0 newly installed, 1 to remove and 55 not upgraded.
After this operation, 1,081 kB disk space will be freed.
Do you want to continue [Y/n]? Y
(Reading database ... 257432 files and directories currently installed.)
Removing apparmor ...
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
Processing triggers for man-db ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot

Should I run 'apt-get autoremove' to remove the above packages that are no longer required?

Code:

$ ps aux | grep apparmor
username  9244  0.0  0.0   9384   936 pts/0    S+   23:29   0:00 grep apparmor

As per http://stackoverflow.com/questions/9375711/more-elegant-ps-aux-grep-v-grep , grep is included in the results, and so I did the following as well, which returned nothing:

Code:

$ ps aux | grep '[a]pparmor'
$ 

I rebooted.

blocknotify is now working!!

Cryptowatch.com:

I had done the above already before I saw your posting.

I ran these commands:

Code:

$ sudo dpkg --get-selections | grep apparmor
apparmor					deinstall
dh-apparmor					install
$ 

$ apt-cache search apparmor
apparmor - User-space parser utility for AppArmor
apparmor-docs - Documentation for AppArmor
apparmor-notify - AppArmor notification system
apparmor-profiles - Profiles for AppArmor Security policies
apparmor-utils - Utilities for controlling AppArmor
dh-apparmor - AppArmor debhelper routines
libapache2-mod-apparmor - changehat AppArmor library as an Apache module
libapparmor-dev - AppArmor development libraries and header files
libapparmor-perl - AppArmor library Perl bindings
libapparmor1 - changehat AppArmor library
libpam-apparmor - changehat AppArmor library as a PAM module
python-libapparmor - AppArmor library Python bindings
$ 

To find the files installed with a package, I ran:

Code:

$ dpkg -L apparmor

From a glance, it seems that all of the files shown by the above command are still on my system, even though I had uninstalled apparmor.  There are some files in /etc/apparmor.d/local/ that were not listed by the above command.

All of the files shown below still exist on my system:

Code:

$ dpkg -L dh-apparmor
/.
/usr
/usr/bin
/usr/bin/dh_apparmor
/usr/share
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/dh_apparmor.1.gz
/usr/share/doc
/usr/share/doc/dh-apparmor
/usr/share/doc/dh-apparmor/changelog.Debian.gz
/usr/share/doc/dh-apparmor/copyright
/usr/share/debhelper
/usr/share/debhelper/autoscripts
/usr/share/debhelper/autoscripts/postrm-apparmor
/usr/share/debhelper/autoscripts/postinst-apparmor
$ 

I don't know if any of the above files matter anymore because blocknotify now works.

dserrano5 and Cryptowatch.com:

Thank you so much for your help.

What about AppArmor?  Without it, isn't my system insecure?

Is there any way to find out the offending AppArmor profile and to fix it?  I googled (startpage.com) for "default profile in apparmor" and cannot find much, other than to assume that apparmor runs the profiles in /etc/apparmor.d.  I looked at some of the profiles in /etc/apparmor.d and cannot figure out how any of them can interfere with bitcoind.

I didn't upgrade my operating system.  I didn't install AppArmor.  What do you think prompted AppArmor to cause the problem?

[cr1776]

...
blocknotify is now working!!
...

Glad that is now working.

The odd thing to me is that apparmor is (at least on my three servers - one bitcoin related, two not) there by default (Ubuntu 14.04 on them).  But it causes no problems and I have no profile for bitcoind.

I am not sure if I would auto remove it.  Since it is part of the mainline, now that you know it was the problem, you might want to see if you can figure out why it was causing an issue and fix it that way.

Ubuntu?  Debian?  version?

e.g.

AppArmor is an established technology first seen in Immunix and later integrated into Ubuntu, Novell/SUSE, and Mandriva. Core AppArmor functionality is in the mainline Linux kernel from 2.6.36 onwards; work is ongoing by AppArmor, Ubuntu and other developers to merge additional AppArmor functionality into the mainline kernel.
https://wiki.ubuntu.com/AppArmor