Amazon EC2 MySQL Failed to Start

Question

I'm having issues starting MySQL after it randomly stopped working a few minutes ago. I'm getting this error while trying to connect:

Connect failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

So I tried to restart MySQL (this had worked for me before) and I got this:

Stopping mysqld:                                           [  OK  ]
MySQL Daemon failed to start.
Starting mysqld:                                           [FAILED]

Here's my error log:

130414 20:03:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130414 20:03:45 [Note] Plugin 'FEDERATED' is disabled.
130414 20:03:45 InnoDB: The InnoDB memory heap is disabled
130414 20:03:45 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130414 20:03:45 InnoDB: Compressed tables use zlib 1.2.5
130414 20:03:45 InnoDB: Using Linux native AIO
130414 20:03:45 InnoDB: Initializing buffer pool, size = 128.0M
130414 20:03:45 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
130414 20:03:45  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
130414 20:03:46  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
130414 20:03:46  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables

Can anyone offer some tips? I'm pretty noobish at this server stuff :P

Many thanks!

Answer 1

I also had faced the same challenge this is what I explored, the reason for restarting can be one of the below:

1. There have been updates applied at your EC2 instance, causing the MySql service to stop and not able to restart again.
2. There may be other processes running along which cause a crunch in memory, thus not allowing the MySql to restart.

To Tackle this you can either use one of the below:

1. If your MySql service is going down because of memory issue -- Upgrade your instance.

2. If your mysqld restart command fails, try to restart the httpd service first and then your mysql service. Here are the commands:

   • sudo service httpd restart
   • sudo service mysqld restart

3. If none of the above work restart your EC2 Instance. Not a permanent fix, but help if you want your services to be up and running, and later want to do an RCA of the issue

If you want you can create a script shown below, and execute it after via a cron-job every 5-10 mins depending on your requirement :

#!/bin/bash
dateFormat=`date "+%Y-%m-%d %T"`
log_file_path="/home/ec2-user/mysql_restart_log.dat"

sudo service mysqld status | grep "is running"
if [ $? -ne 0 ]; then
   echo "HTTPD restart attempted ${dateFormat}" >> ${log_file_path}

   sudo service httpd restart
   if [ $? -ne 0 ]; then
     echo "HTTPD restart failed... ${dateFormat}" >> ${log_file_path}
   else
     echo "HTTPD restart success... ${dateFormat}" >> ${log_file_path}
   fi

   echo "MYSQL restart attempted ${dateFormat}" >> ${log_file_path}
   sudo service mysqld restart
   if [ $? -ne 0 ]; then
     echo "MYSQL restart failed... ${dateFormat}" >> ${log_file_path}
   else
     echo "MYSQL restart success... ${dateFormat}" >> ${log_file_path}
   fi
fi

Also found this link helpful: mysql on amazon linux - MySQL Daemon failed to start