Last week, we discussed how to configure AppArmor for MongoDB Replica Sets which basically has the same concepts applicable when configuring this for your MySQL-based systems. Indeed, security is very important because you have to make sure that your data is very well protected and encapsulated against unwanted data gathering of information from your business domain.
A little bit of history about AppArmor
AppArmor was first used in Immunix Linux 1998–2003. At the time, AppArmor was known as SubDomain, a reference to the ability for a security profile for a specific program to be segmented into different domains, which the program can switch between dynamically. AppArmor was first made available in SLES and openSUSE, and was first enabled by default in SLES 10 and in openSUSE 10.1.
In May 2005 Novell acquired Immunix and rebranded SubDomain as AppArmor and began code cleaning and rewriting for the inclusion in the Linux kernel. From 2005 to September 2007, AppArmor was maintained by Novell. Novell was taken over by SUSE who are now the legal owners of the trademarked name AppArmor.
AppArmor was first successfully ported/packaged for Ubuntu in April 2007. AppArmor became a default package starting in Ubuntu 7.10, and came as a part of the release of Ubuntu 8.04, protecting only CUPS by default. As of Ubuntu 9.04 more items such as MySQL have installed profiles. AppArmor hardening continued to improve in Ubuntu 9.10 as it ships with profiles for its guest session, libvirt virtual machines, the Evince document viewer, and an optional Firefox profile.
Why do we need AppArmor?
In our previous blog, we have tackled a bit of what is the use of AppArmor. It is a Mandatory Access Control (MAC) system, implemented upon the Linux Security Modules (LSM). It is used and mostly enabled by default in systems such as Ubuntu, Debian (since Buster), SUSE, and other distributions. It is comparable to RHEL/CentOS SELinux, which requires good userspace integration to work properly. SELinux attaches labels to all files, processes, and objects and is therefore very flexible. However, configuring SELinux is considered to be very complicated and requires a supported filesystem. AppArmor, on the other hand, works using file paths and its configuration can be easily adapted.
AppArmor, like most other LSMs, supplements rather than replaces the default Discretionary Access Control (DAC). As such it is impossible to grant a process more privileges than it had in the first place.
AppArmor proactively protects the operating system and applications from external or internal threats and even zero-day attacks by enforcing a specific rule set on a per-application basis. Security policies completely define what system resources individual applications can access, and with what privileges. Access is denied by default if no profile says otherwise. A few default policies are included with AppArmor and using a combination of advanced static analysis and learning-based tools, AppArmor policies for even very complex applications can be deployed successfully in a matter of hours.
Every breach of policy triggers a message in the system log, and AppArmor can be configured to notify users with real-time violation warnings.
AppArmor for MySQL
I have setup a MySQL-replication-based cluster using ClusterControl to my target database nodes running in Ubuntu Bionic. You can further follow this blog on how to deploy it or follow this video tutorial. Take note that ClusterControl checks or disables the AppArmor during deployment. You might have to uncheck this according to your setup just like below:
ClusterControl will just issue a warning that it is not touching your current AppArmor configuration. See below:
Managing your AppArmor profiles
Standard installation of your AppArmor in Ubuntu would not include utilities that would help manage the profiles efficiently. So let’s install these packages like so:
$ apt install apparmor-profiles apparmor-utils
Once installed, check the status of your AppArmor in the system by running aa-status command. In the node I am using, I have the following output without MySQL 8 installed.
$ aa-statusapparmor module is loaded.15 profiles are loaded.15 profiles are in enforce mode. /sbin/dhclient /usr/bin/lxc-start /usr/bin/man /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/NetworkManager/nm-dhcp-helper /usr/lib/connman/scripts/dhclient-script /usr/lib/snapd/snap-confine /usr/lib/snapd/snap-confine//mount-namespace-capture-helper /usr/sbin/tcpdump lxc-container-default lxc-container-default-cgns lxc-container-default-with-mounting lxc-container-default-with-nesting man_filter man_groff0 profiles are in complain mode.0 processes have profiles defined.0 processes are in enforce mode.0 processes are in complain mode.0 processes are unconfined but have a profile defined.
Since I am using ClusterControl to deploy my MySQL-replication based cluster with AppArmor (i.e. ClusterControl won’t touch my current AppArmor config), the deployment shall have the MySQL profile in place and that shows up in the list running aa-status.
$ aa-statusapparmor module is loaded.56 profiles are loaded.19 profiles are in enforce mode. ... /usr/sbin/mysqld ...37 profiles are in complain mode. ...1 processes have profiles defined.1 processes are in enforce mode. /usr/sbin/mysqld (31501)0 processes are in complain mode.0 processes are unconfined but have a profile defined.
It is worth noting that a profile is in one of the following modes:
Enforce – Default setting. Applications are prevented from taking actions restricted by the profile rules.
Complain – Applications are allowed to take restricted actions, and the actions are logged.
Disabled – Applications are allowed to take restricted actions, and the actions are not logged.
You can also mix enforce and complain profiles in your server.
Based on the output above, let’s elaborate more about the profile complain. AppArmor will allow it to perform (almost as complain mode status will still enforce any explicit deny rules in a profile) all tasks without restriction but it will log them in the audit log as events. This is useful when you are attempting to create a profile for an application but are not sure what things it needs access to. Whereas the unconfined status, on the other hand, will allow the program to perform any task and will not log it. This usually occurs if a profile was loaded after an application is started, meaning it runs without restrictions from AppArmor. It is also important to note that only processes that have profiles are listed under this unconfined status. Any other processes that run on your system but do not have a profile created for them will not be listed under aa-status.
If you have disabled AppArmor but then realize you wanted to enhance your security or comply with security regulations, you can use this MySQL 8.0 profile that is provided by MySQL itself. To apply that, just run the following command:
$ cat /etc/apparmor.d/usr.sbin.mysqld | sudo apparmor_parser -a
It is worth noting that AppArmor profiles are stored by default in /etc/apparmor.d/. It is a good practice to add your profiles in that directory.
Diagnosing your AppArmor profiles
AppArmor logs can be found in the systemd journal, in /var/log/syslog and /var/log/kern.log (and /var/log/audit.log when auditd is installed). What you need to look for is the following:
ALLOWED (logged when a profile in complain mode violates the policy)
DENIED (logged when a profile in enforce mode actually blocks an operation)
The full log message should provide more information on what exact access has been denied. You can use this to edit profiles before turning them on in enforce mode.
$ grep -i -rn -E 'apparmor=.*denied|apparmor=.*allowed' /var/log//var/log/kern.log:503:Jun 18 18:54:09 ubuntu-bionic kernel: [ 664.680141] audit: type=1400 audit(1624042449.006:19): apparmor="DENIED" operation="capable" profile="/usr/sbin/mysqld" pid=30349 comm="mysqld" capability=2 capname="dac_read_search"Binary file /var/log/journal/877861ee473c4c03ac1512ed369dead1/system.journal matches/var/log/syslog:1012:Jun 18 18:54:09 ubuntu-bionic kernel: [ 664.680141] audit: type=1400 audit(1624042449.006:19): apparmor="DENIED" operation="capable" profile="/usr/sbin/mysqld" pid=30349 comm="mysqld" capability=2 capname="dac_read_search"
Customizing your AppArmor profile
Profiles prepared by Oracle MySQL shall not be a one-size-fits-all pattern. In that case, you might decide to change, for example, the data directory where your MySQL instance data is located. After you apply the changes to your configuration file and restart your MySQL instances, AppArmor will deny this action. For example,
$ egrep -i -rn 'apparmor=.*denied|apparmor=.*allowed' /var/log//var/log/kern.log:503:Jun 18 18:54:09 ubuntu-bionic kernel: [ 664.680141] audit: type=1400 audit(1624042449.006:19): apparmor="DENIED" operation="capable" profile="/usr/sbin/mysqld" pid=30349 comm="mysqld" capability=2 capname="dac_read_search"/var/log/kern.log:522:Jun 18 19:46:26 ubuntu-bionic kernel: [ 3801.151770] audit: type=1400 audit(1624045586.822:67): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/mysql-data/mysql.sock.lock" pid=5262 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=1002 ouid=1002Binary file /var/log/journal/877861ee473c4c03ac1512ed369dead1/system.journal matches/var/log/syslog:1012:Jun 18 18:54:09 ubuntu-bionic kernel: [ 664.680141] audit: type=1400 audit(1624042449.006:19): apparmor="DENIED" operation="capable" profile="/usr/sbin/mysqld" pid=30349 comm="mysqld" capability=2 capname="dac_read_search"/var/log/syslog:1313:Jun 18 19:46:26 ubuntu-bionic kernel: [ 3801.151770] audit: type=1400 audit(1624045586.822:67): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/mysql-data/mysql.sock.lock" pid=5262 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=1002 ouid=1002
Together with the error I had earlier, now it adds that I had just decided to use the /mysql-data directory but that is denied.
To apply the changes, let’s do the following. Edit the file /etc/apparmor.d/usr.sbin.mysqld. You might find these lines:
# Allow data dir access /var/lib/mysql/ r, /var/lib/mysql/** rwk,Those flags such as r, rwk are the so-called access modes. These mean the following: r - read w - write -- conflicts with append k - lock
The man page explains those flags in more detail.
Now, I have changed it to the following:
# Allow data dir access /mysql-data/ r, /mysql-data/** rwk,
Then I reload the profiles as follows:
$ apparmor_parser -r -T /etc/apparmor.d/usr.sbin.mysqld
Restart the MySQL server:
$ systemctl restart mysql.service
What if I set my mysqld to a complain mode?
As mentioned earlier, complain mode status will still enforce any explicit deny rules in a profile. Though this works for example:
$ aa-complain /usr/sbin/mysqld
Setting /usr/sbin/mysqld to complain mode.
$ aa-statusapparmor module is loaded.56 profiles are loaded.18 profiles are in enforce mode. ...38 profiles are in complain mode. ...1 processes have profiles defined.0 processes are in enforce mode.1 processes are in complain mode. /usr/sbin/mysqld (23477)0 processes are unconfined but have a profile defined.
After you restarted MySQL, it will run and will show log files such as:
/var/log/syslog:1356:Jun 18 19:58:51 ubuntu-bionic kernel: [ 4545.427074] audit: type=1400 audit(1624046331.098:83): apparmor="ALLOWED" operation="open" profile="/usr/sbin/mysqld" name="/mysql-data/mysql.sock.lock" pid=5760 comm="mysqld" requested_mask="wrc" denied_mask="wrc" fsuid=1002 ouid=1002/var/log/syslog:1357:Jun 18 19:58:51 ubuntu-bionic kernel: [ 4545.432077] audit: type=1400 audit(1624046331.102:84): apparmor="ALLOWED" operation="mknod" profile="/usr/sbin/mysqld" name="/mysql-data/mysql.sock" pid=5760 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=1002 ouid=1002/var/log/syslog:1358:Jun 18 19:58:51 ubuntu-bionic kernel: [ 4545.432101] audit: type=1400 audit(1624046331.102:85): apparmor="ALLOWED" operation="mknod" profile="/usr/sbin/mysqld" name="/mysql-data/mysql.pid" pid=5760 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=1002 ouid=1002And it will work. However, it will probably have issues with networking as it still denies entries as what we have in /etc/apparmor.d/usr.sbin.mysqld. For example, my replica is not able to connect to the primary: Last_IO_Error: error connecting to master '[emailprotected]:3306' - retry-time: 10 retries: 1 message: Host '192.168.40.247' is not allowed to connect to this MySQL server Last_SQL_Errno: 0
In that case, using enforce and reloading your profile shall be an efficient and easy approach to manage your MySQL profiles with AppArmor.
MariaDB is more scalable and offers a higher query speed when compared to MySQL. This makes it good for managing large-sized data. You will also find more features in MariaDB that MySQL doesn't have, like sequence storage engines and virtual columns. You can also use multiple engines in one table.How to disable AppArmor for MySQL? ›
- Check the status. $ sudo apparmor_status.
- Stop and disable AppArmor. $ sudo systemctl stop apparmor $ sudo systemctl disable apparmor.
Yes, it is. Just as two MySQL instances can coexist. MariaDB is still a drop-in replacement. By port or by different socket path, and install MariaDB to a non-standard directory.What is the disadvantage of MariaDB? ›
Disadvantages. MariaDB is somewhat liable to bloating. Its central IDX log file, in particular, tends to become very large after protracted use, ultimately slowing performance. Caching is another area where MariaDB could use work—it is not as fast as it could be, which can be frustrating.How do I set up an AppArmor? ›
How to enable/disable. If AppArmor is not the default security module it can be enabled by passing security=apparmor on the kernel's command line. If AppArmor is the default security module it can be disabled by passing apparmor=0, security=XXXX (where XXXX is valid security module), on the kernel's command line.How do I know if AppArmor is enabled? ›
Detect the state of AppArmor by inspecting /sys/kernel/security/apparmor/profiles . If cat /sys/kernel/security/apparmor/profiles reports a list of profiles, AppArmor is running. If it is empty and returns nothing, AppArmor is stopped.Is AppArmor enabled by default? ›
AppArmor is installed and loaded by default. It uses profiles of an application to determine what files and permissions the application requires. Some packages will install their own profiles, and additional profiles can be found in the apparmor-profiles package.Should I learn MySQL or MariaDB? ›
MySQL is more widely used, has a more mature feature set, and is more reliable. If you are looking for a general-purpose database to use in your applications, MySQL is probably the better choice. If you are looking for a database that has been specifically optimized for performance, MariaDB is a better option.Is MariaDB a replacement for MySQL? ›
Yes, it is possible to replace MySQL with MariaDB. MariaDB is considered to be a drop-in replacement for the corresponding MySQL version (i.e MariaDB 10.2 and MySQL 5.7). The vice versa replacement, MariaDB to MySQL is also possible.Why was MySQL replaced by MariaDB? ›
On this basis, Fedora developers replaced MySQL with MariaDB in Fedora 19, out of concerns that Oracle was making MySQL a more closed software project.
When MariaDB Server's predecessor, MySQL, was bought by Oracle in 2009, MySQL founder Michael “Monty” Widenius forked the project due to concerns about Oracle's stewardship, naming the new project MariaDB. MySQL was named after his first daughter, My, while MariaDB is named after his second daughter, Maria.