Status: "MariaDB server is down"

Question

Hope someone can help me:

I have many sites running on my server and I am using mariadb. when I do:

sudo service mysql restart

I get:

Job for mariadb.service failed because the control process exited with 
error code. See "systemctl status mariadb.service" and "journalctl -xe" 
for details.

systemctl status mariadb.service

    mariadb.service - MariaDB database server

    Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)

    Drop-In: /etc/systemd/system/mariadb.service.d
             └─migrated-from-my.cnf-settings.conf
    Active: failed (Result: exit-code) since Thu 2017-09-14 03:16:51 UTC; 1min 6s ago
    Process: 13247 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=
    Process: 13086 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/gale
    Process: 13070 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, s
    Process: 13033 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, st
   Main PID: 13247 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"
   CGroup: /system.slice/mariadb.service
           └─3803 /usr/sbin/mysqld

sudo journalctl -xe

   Sep 14 03:35:51 ubuntu systemd[1]: Failed to start MariaDB database server.
   -- Subject: Unit mariadb.service has failed
   -- Defined-By: systemd
   -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
   -- 
   -- Unit mariadb.service has failed.
   -- 
   -- The result is failed.
   Sep 14 03:35:51 ubuntu systemd[1]: mariadb.service: Unit entered failed state.
   Sep 14 03:35:51 ubuntu sudo[13566]: pam_unix(sudo:session): session closed for user root
   Sep 14 03:35:51 ubuntu systemd[1]: mariadb.service: Failed with result 'exit-code'.
   Sep 14 03:35:52 ubuntu sshd[13753]: Failed password for root from 123.183.209.136 port 38816 ssh2
   Sep 14 03:35:52 ubuntu sshd[13753]: Received disconnect from 123.183.209.136 port 38816:11:  [preauth]
   Sep 14 03:35:52 ubuntu sshd[13753]: Disconnected from 123.183.209.136 port 38816 [preauth]
   Sep 14 03:35:52 ubuntu sshd[13753]: PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser
   Sep 14 03:36:34 ubuntu sshd[13756]: Connection closed by 123.183.209.136 port 60477 [preauth]
   Sep 14 03:36:36 ubuntu postfix/pickup[12161]: 703AA5C0C5: uid=0 from=<root>
   Sep 14 03:36:36 ubuntu postfix/cleanup[12162]: 703AA5C0C5: message-id=<[email protected]
   Sep 14 03:36:36 ubuntu postfix/cleanup[12162]: warning: 703AA5C0C5: write queue file: No space left on dev
   Sep 14 03:36:36 ubuntu postfix/pickup[12161]: warning: maildrop/5152F5C0C4: error writing 703AA5C0C5: queu
   Sep 14 03:37:17 ubuntu sshd[13760]: Received disconnect from 123.183.209.136 port 25567:11:  [preauth]
   Sep 14 03:37:17 ubuntu sshd[13760]: Disconnected from 123.183.209.136 port 25567 [preauth]
   Sep 14 03:37:36 ubuntu postfix/pickup[12161]: 7764B5C0C5: uid=0 from=<root>
   Sep 14 03:37:36 ubuntu postfix/cleanup[12162]: 7764B5C0C5: message-id=<[email protected]
   Sep 14 03:37:36 ubuntu postfix/cleanup[12162]: warning: 7764B5C0C5: write queue file: No space left on dev
   Sep 14 03:37:36 ubuntu postfix/pickup[12161]: warning: maildrop/5152F5C0C4: error writing 7764B5C0C5: queu
   Sep 14 03:37:47 ubuntu sudo[13765]:      bob : TTY=pts/0 ; PWD=/home/bob ; USER=root ; COMMAND=/bin/journa
   Sep 14 03:37:47 ubuntu sudo[13765]: pam_unix(sudo:session): session opened for user root by bob(uid=0)
   lines 1238-1265/1265 (END)

I really don't want to lose my db. What should I do next?

Many thanks

this is what df gives me:

   Filesystem     1K-blocks     Used Available Use% Mounted on
   /dev/root       49362256 46854648         0 100% /
   devtmpfs         2019828        0   2019828   0% /dev
   tmpfs            2021768        0   2021768   0% /dev/shm
   tmpfs            2021768    17968   2003800   1% /run
   tmpfs               5120        0      5120   0% /run/lock
   tmpfs            2021768        0   2021768   0% /sys/fs/cgroup
   tmpfs             404356        0    404356   0% /run/user/1000
   tmpfs             404356        0    404356   0% /run/user/0

Answer 1

See the first filesystem (/dev/root) is 100% full! And there are no other volumes mounted besides tmpfs volumes. So mariadb is trying to write something to disk as it starts up, and can't, so it gives up and shuts itself down.

You have to figure out what's filling up your disk, and prune what you can. For example, as a guess I'd start by looking in the http server logs, since you say you host many sites on this server. I've seen many servers that filled up disk space by letting httpd log everything for months and months.

In the future, you might want to provision this server with more than one disk volume, so if one volume fills up it doesn't interfere with other volumes.

Also consider automating log rotation.

If the culprit for disk space usage is not logs, but is actually the mariadb data directory, then you might need to get an additional disk volume immediately, and move the mariadb content to it before you can bring your sites back online.

Unless you can find a little bit of other stuff elsewhere on the server that you can delete and make enough space for a short while.