mysql service fails to start/hangs up - timeout (Ubuntu, MariaDB)

Question

I set up my first Ubuntu Server with Ubuntu 16.04, nginx, php7.0, MariaDB, nextcloud and external DynDNS using this tutorial here: Install Nextcloud 9 on Ubuntu 16.04

Everything worked fine but since I restarted the server the next day, nextcloud just shows me a blank page. After clicking through all logs of nginx, MariaDB and nextcloud, I found out that the mysql service just doesn't start. So run service mysql start and everything worked fine again (calling nextcloud from server as well as other workstations). I just wondered that the terminal didn't "close" the line. Like it was still working on the command. After about 5 minutes, the line "closes" and the following message appears:

"Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details."

Then the clients again just get a blank page in nextcloud. When I run the command and close the terminal immediately, clients get the access as well but still loses it after 5 minutes.

I tried backing up the nextcloud, sql and run apt-get purge --auto-remove mariadb-server. Then again run the MariaDB installation steps out of the tutorial with importing the backup sql instead of creating a new one. Didn't change everything.

Next try was update-rc.d mysql defaults and update-rc.d mysql enable. But after a restart just the blank page again. Access is only possible for 5 minutes by starting the service manual.

I also tried the BUM - BootUpManager but the service seems to be enabled. I saw you can start services out oft it manually as well. So tried it with mysql and surprise: nextcloud available for 5 minutes while BUM just hangs up.

I found mariadb.com/kb/en/mariadb/starting-and-stopping-mariadb-automatically/ as well but tried nothing of it because it seems like there is something else really wrong.

root@s1:~# systemctl status mariadb.service:

\u25cf mariadb.service - MariaDB database server    Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset:    Drop-In: /etc/systemd/system/mariadb.service.d            \u2514\u2500migrated-from-my.cnf-settings.conf    Active: failed (Result: timeout) since Di 2016-12-06 14:52:51 CET; 55s ago   Process: 3565 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WS   Process: 3415 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR   Process: 3409 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START   Process: 3405 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/ru  Main PID: 3565 (code=exited, status=0/SUCCESS)  Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] /usr/sbin Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] Event Sch Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 2147785536 [Note] InnoDB: F Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] InnoDB: S Dez 06 14:52:49 s1 mysqld[3565]: 2016-12-06 14:52:49 3067387712 [Note] InnoDB: W Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] InnoDB: S Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] /usr/sbin Dez 06 14:52:51 s1 systemd[1]: Failed to start MariaDB database server. Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Unit entered failed state. Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Failed with result 'timeout'. 

root@s1:~# journalctl -xe:

Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] Event Scheduler: Purging the queue. 0 events Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 2147785536 [Note] InnoDB: FTS optimize thread exiting. Dez 06 14:52:48 s1 mysqld[3565]: 2016-12-06 14:52:48 3067387712 [Note] InnoDB: Starting shutdown... Dez 06 14:52:49 s1 mysqld[3565]: 2016-12-06 14:52:49 3067387712 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer po Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] InnoDB: Shutdown completed; log sequence number 111890806 Dez 06 14:52:50 s1 mysqld[3565]: 2016-12-06 14:52:50 3067387712 [Note] /usr/sbin/mysqld: Shutdown complete Dez 06 14:52:50 s1 audit[3648]: AVC apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profi Dez 06 14:52:50 s1 kernel: audit: type=1400 audit(1481032370.973:29): apparmor="DENIED" operation="sendmsg" info="Failed name lookup -  Dez 06 14:52:50 s1 audit[3565]: AVC apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profi Dez 06 14:52:50 s1 kernel: audit: type=1400 audit(1481032370.973:30): apparmor="DENIED" operation="sendmsg" info="Failed name lookup -  Dez 06 14:52:51 s1 systemd[1]: Failed to start MariaDB database server. -- Subject: Unit mariadb.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel --  -- Unit mariadb.service has failed. --  -- The result is failed. Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Unit entered failed state. Dez 06 14:52:51 s1 systemd[1]: mariadb.service: Failed with result 'timeout'. Dez 06 14:54:54 s1 x11vnc[2665]: 06/12/2016 14:54:54 cursor_noshape_updates_clients: 1 Dez 06 14:55:16 s1 ntpd[1244]: 46.4.1.155 local addr 192.168.178.50 -> <null> Dez 06 14:57:30 s1 ntpd[1244]: 89.238.66.98 local addr 192.168.178.50 -> <null> 

Content in /ect/init.d (if useful):

#!/bin/bash # ### BEGIN INIT INFO # Provides:          mysql # Required-Start:    $remote_fs $syslog # Required-Stop:     $remote_fs $syslog # Should-Start:      $network $named $time # Should-Stop:       $network $named $time # Default-Start:     2 3 4 5 # Default-Stop:      0 1 6 # Short-Description: Start and stop the mysql database server daemon # Description:       Controls the main MariaDB database server daemon "mysqld" #                    and its wrapper script "mysqld_safe". ### END INIT INFO # set -e set -u ${DEBIAN_SCRIPT_DEBUG:+ set -v -x}  test -x /usr/sbin/mysqld || exit 0  . /lib/lsb/init-functions  SELF=$(cd $(dirname $0); pwd -P)/$(basename $0) CONF=/etc/mysql/my.cnf MYADMIN="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf"  # priority can be overriden and "-s" adds output to stderr ERR_LOGGER="logger -p daemon.err -t /etc/init.d/mysql -i"  # Safeguard (relative paths, core dumps..) cd / umask 077  # mysqladmin likes to read /root/.my.cnf. This is usually not what I want # as many admins e.g. only store a password without a username there and # so break my scripts. export HOME=/etc/mysql/  # Source default config file. [ -r /etc/default/mariadb ] && . /etc/default/mariadb  ## Fetch a particular option from mysql's invocation. # # Usage: void mysqld_get_param option mysqld_get_param() {     /usr/sbin/mysqld --print-defaults \         | tr " " "\n" \         | grep -- "--$1" \         | tail -n 1 \         | cut -d= -f2 }  ## Do some sanity checks before even trying to start mysqld. sanity_checks() {   # check for config file   if [ ! -r /etc/mysql/my.cnf ]; then     log_warning_msg "$0: WARNING: /etc/mysql/my.cnf cannot be read. See README.Debian.gz"     echo                "WARNING: /etc/mysql/my.cnf cannot be read. See README.Debian.gz" | $ERR_LOGGER   fi    # check for diskspace shortage   datadir=`mysqld_get_param datadir`   if LC_ALL=C BLOCKSIZE= df --portability $datadir/. | tail -n 1 | awk '{ exit ($4>4096) }'; then     log_failure_msg "$0: ERROR: The partition with $datadir is too full!"     echo                "ERROR: The partition with $datadir is too full!" | $ERR_LOGGER     exit 1   fi }  ## Checks if there is a server running and if so if it is accessible. # # check_alive insists on a pingable server # check_dead also fails if there is a lost mysqld in the process list # # Usage: boolean mysqld_status [check_alive|check_dead] [warn|nowarn] mysqld_status () {     ping_output=`$MYADMIN ping 2>&1`; ping_alive=$(( ! $? ))      ps_alive=0     pidfile=`mysqld_get_param pid-file`     if [ -f "$pidfile" ] && ps `cat $pidfile` >/dev/null 2>&1; then ps_alive=1; fi      if [ "$1" = "check_alive"  -a  $ping_alive = 1 ] ||        [ "$1" = "check_dead"   -a  $ping_alive = 0  -a  $ps_alive = 0 ]; then     return 0 # EXIT_SUCCESS     else     if [ "$2" = "warn" ]; then         echo -e "$ps_alive processes alive and '$MYADMIN ping' resulted in\n$ping_output\n" | $ERR_LOGGER -p daemon.debug     fi     return 1 # EXIT_FAILURE     fi }  # # main() #  case "${1:-''}" in   'start')     sanity_checks;     # Start daemon     log_daemon_msg "Starting MariaDB database server" "mysqld"     if mysqld_status check_alive nowarn; then        log_progress_msg "already running"        log_end_msg 0     else         # Could be removed during boot         test -e /var/run/mysqld || install -m 755 -o mysql -g root -d /var/run/mysqld          # Start MariaDB!          /usr/bin/mysqld_safe "${@:2}" > /dev/null 2>&1 &          # 6s was reported in #352070 to be too little         for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-60}"); do                 sleep 1             if mysqld_status check_alive nowarn ; then break; fi         log_progress_msg "."         done         if mysqld_status check_alive warn; then                 log_end_msg 0             # Now start mysqlcheck or whatever the admin wants.             output=$(/etc/mysql/debian-start)         [ -n "$output" ] && log_action_msg "$output"         else             log_end_msg 1         log_failure_msg "Please take a look at the syslog"         fi     fi     ;;    'stop')     # * As a passwordless mysqladmin (e.g. via ~/.my.cnf) must be possible     # at least for cron, we can rely on it here, too. (although we have      # to specify it explicit as e.g. sudo environments points to the normal     # users home and not /root)     log_daemon_msg "Stopping MariaDB database server" "mysqld"     if ! mysqld_status check_dead nowarn; then       set +e       shutdown_out=`$MYADMIN shutdown 2>&1`; r=$?       set -e       if [ "$r" -ne 0 ]; then         log_end_msg 1         [ "$VERBOSE" != "no" ] && log_failure_msg "Error: $shutdown_out"         log_daemon_msg "Killing MariaDB database server by signal" "mysqld"         killall -15 mysqld             server_down=         for i in `seq 1 600`; do               sleep 1               if mysqld_status check_dead nowarn; then server_down=1; break; fi             done           if test -z "$server_down"; then killall -9 mysqld; fi       fi         fi          if ! mysqld_status check_dead warn; then       log_end_msg 1       log_failure_msg "Please stop MariaDB manually and read /usr/share/doc/mariadb-server-10.1/README.Debian.gz!"       exit -1     else       log_end_msg 0         fi     ;;    'restart')     set +e; $SELF stop; set -e     $SELF start      ;;    'reload'|'force-reload')     log_daemon_msg "Reloading MariaDB database server" "mysqld"     $MYADMIN reload     log_end_msg 0     ;;    'status')     if mysqld_status check_alive nowarn; then       log_action_msg "$($MYADMIN version)"     else       log_action_msg "MariaDB is stopped."       exit 3     fi     ;;    'bootstrap')     # Bootstrap the cluster, start the first node     # that initiates the cluster     log_daemon_msg "Bootstrapping the cluster" "mysqld"     $SELF start "${@:2}" --wsrep-new-cluster     ;;    *)     echo "Usage: $SELF start|stop|restart|reload|force-reload|status|bootstrap"     exit 1     ;; esac 

Unfortunately, Google can't help me. I tried to explain as much as I can maybe this helps you in helping me. Thanks a lot!

Answer 1

In case you are bitten by this bug, the solution is given as a suggestion in the bug report (all of these have to be done as root, so either with sudo -i as a zeroth command or with sudo prefixed):

1. echo "/usr/sbin/mysqld { }" > /etc/apparmor.d/usr.sbin.mysqld (the second part with sudo is ... | sudo tee /etc/apparmor.d/usr.sbin.mysqld, thank you @dvlcube)
2. apparmor_parser -v -R /etc/apparmor.d/usr.sbin.mysqld
3. systemctl restart mariadb

Background

If you previously had MySQL installed, it activated an AppArmor profile which is incompatible with MariaDB. apt-get remove --purge only removes the profile, but does not deactivate/unload it. Only manually unloading it lets MariaDB work unhindered by AppArmor.