MAMP PRO crashes; MySQL will not start on reboot

Question

Today at work my computer randomly froze/crashed. On reboot MAMP refuses to start mysql and I can't figure out why. There are definitely no other mysql processes running; I’ve checked several times. So, the killall -9 mysqld is not the solution. I’ve actually completely re-installed MAMP, also.

This is what seems to be the important part of my MySQL log, but I’m not very seasoned in reading these things, so maybe the answer is right in front of me.

140527 15:06:58 mysqld_safe Starting mysqld daemon with databases from /Library/Application Support/appsolute/MAMP PRO/db/mysql
140527 15:06:58 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
140527 15:06:58 [Warning] Setting lower_case_table_names=2 because file system for /Library/Application Support/appsolute/MAMP PRO/db/mysql/ is case insensitive
140527 15:06:58 [Note] Plugin 'FEDERATED' is disabled.
140527 15:06:58 InnoDB: The InnoDB memory heap is disabled
140527 15:06:58 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140527 15:06:58 InnoDB: Compressed tables use zlib 1.2.3
140527 15:06:58 InnoDB: Initializing buffer pool, size = 128.0M
140527 15:06:58 InnoDB: Completed initialization of buffer pool
140527 15:06:58 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 791075520
140527 15:06:58  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 791076717
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 8402.
InnoDB: You may have to recover from a backup.
140527 15:06:58  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex ....
InnoDB: End of page dump
140527 15:06:58  InnoDB: Page checksum 3802906200, prior-to-4.0.14-form checksum 786607151
InnoDB: stored checksum 3802906200, prior-to-4.0.14-form stored checksum 1787456768
InnoDB: Page lsn 0 791046088, low 4 bytes of lsn at page end 790720183
InnoDB: Page number (if stored to page already) 8402,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an insert undo log page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 8402.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
140527 15:06:58  InnoDB: Assertion failure in thread 140735261836048 in file buf0buf.c line 3619
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
19:06:58 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 134084 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
0   mysqld                              0x000000010028081c my_print_stacktrace + 44
1   mysqld                              0x0000000100021624 handle_fatal_signal + 692
2   libsystem_platform.dylib            0x00007fff91e625aa _sigtramp + 26
3   ???                                 0x0000000000000000 0x0 + 0
4   libsystem_c.dylib                   0x00007fff9355bb1a abort + 125
5   mysqld                              0x00000001002b30af buf_page_io_complete + 959
6   mysqld                              0x00000001002b9892 buf_read_page_low + 610
7   mysqld                              0x00000001002b9fa5 buf_read_page + 85
8   mysqld                              0x00000001002b4431 buf_page_get_gen + 673
9   mysqld                              0x000000010034fdd5 trx_undo_lists_init + 373
10  mysqld                              0x0000000100348e2e trx_rseg_mem_create + 334
11  mysqld                              0x0000000100348fed trx_rseg_list_and_array_init + 157
12  mysqld                              0x000000010034a147 trx_sys_init_at_db_start + 215
13  mysqld                              0x000000010033eafd innobase_start_or_create_for_mysql + 5805
14  mysqld                              0x00000001003114c1 _ZL13innobase_initPv + 1473
15  mysqld                              0x0000000100027028 _Z24ha_initialize_handlertonP13st_plugin_int + 104
16  mysqld                              0x000000010017f0f1 _ZL17plugin_initializeP13st_plugin_int + 97
17  mysqld                              0x0000000100181810 _Z11plugin_initPiPPci + 3776
18  mysqld                              0x00000001000ccce5 _Z11mysqld_mainiPPc + 10405
19  mysqld                              0x0000000100001804 start + 52
20  ???                                 0x000000000000000a 0x0 + 10
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140527 15:06:58 mysqld_safe mysqld from pid file /Applications/MAMP/tmp/mysql/mysql.pid ended

And when trying to do a db dump I am getting: (EDIT: I got this part of it fixed)

mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) when trying to connect

Answer 1

PREFACE: This sounds bad, but please be sure to read everything in this answer before acting. You can’t break things worse by taking your time. Read each step & hopefully this will be clear enough for you to follow & get your MySQL database server in MAMP Pro up and running again.

So, it seems like your InnoDB databases crashed. Not the app itself. The key is here in the log:

140527 15:06:58 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 791075520
140527 15:06:58  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 791076717
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 8402.
InnoDB: You may have to recover from a backup.

And it looks like you are using MAMP PRO over here:

/Library/Application Support/appsolute/MAMP PRO/db/mysql

So the question is, do you have a backup of the MAMP Pro databases? Either via mysqldump or something else? Do you have other InnoDB databases in your MAMP install?

Also, you say you were able to run mysqldump, but it’s really not possible of the database crashed. So I am assuming when you ran mysqldump that was another, separate install of MySQL on your system. The MySQL binaries such as mysqldump in MAMP or MAMP Pro are not the same as the systemwide mysqldump. They are two 100% different installs. You can check which mysqldump is being used by typing in this command:

which mysqldump

To see the full path of what you believe you were using. The MAMP install of mysqldump—and other related binaries—is located here:

/Applications/MAMP/Library/bin/

And to run it directly without modifying your $PATH value (a whole other thing) is to run it like this:

/Applications/MAMP/Library/bin/mysqldump

PLEASE READ CAREFULLY: Please note the advice I am giving you below is me presenting every way I would deal with a situation like this. If the InnoDB database is not important, just do my first suggestion of trashing the InnoDB specific DB files. If you have a mysqldump backup, do the same thing but recover the mysqldump backup.

Also, InnoDB is not a default storage engine. You have to go out of your way to set that. The default is MyISAM. Any new DB created in MySQL will be MyISAM. So this will help you. You need to put on your thinking cap an figure out which databases have InnoDB storage engines set. If you say you have 25 but only 1 has InnoDB, easy solution. But also if you have 25 databases, you should get into the habit of making regular mysqldump backups. If you had backups, this would be a headache but a straight forward thing to solve.

ONE OPTION: Delete the corrupted InnoDB stuff & recover from a mysqldump backup.

The first thing I would do if I were you is backup the mysql directory in /Library/Application Support/appsolute/MAMP PRO/db/ so you can at least have a backup of the corrupted files just in case.

Then I would delete the following files:

/Library/Application Support/appsolute/MAMP PRO/db/mysql/ib_logfile0
/Library/Application Support/appsolute/MAMP PRO/db/mysql/ib_logfile1
/Library/Application Support/appsolute/MAMP PRO/db/mysql/ibdata1

Those are InnoDB specific files. Delete them and then attempt to start MAMP again. It should come up. But any InnoDB database in MAMP will be in some “zombie” state. You should delete those databases & recreate from backup. Or from scratch if you can.

ANOTHER OPTION: Try to get the MySQL server up & running again with innodb_force_recovery.

Now on the offhand chance you need to recover that DB, you can run attempt to set a innodb_force_recovery as described here.

For MAMP Pro it seems you can edit your MySQL configuration file as per these instructions:

1. Start MAMP Pro.
2. Stop the MAMP Pro server if it is running.
3. Select File -> Edit Template -> MySQL my.cnf
4. An editor window appears.
5. If a warning message appears confirm with OK.
6. Find the section "[mysqld]"
7. Beneath the last line of this section add this line: innodb_force_recovery = 1

And as the MySQL documentation explains, this is strictly to get the database up and running so you can make a backup via mysqldump:

In such cases, use the innodb_force_recovery option to force the InnoDB storage engine to start up while preventing background operations from running, so that you can dump your tables.

Now there are about 6 different values for innodb_force_recovery but you should really only attempt with 1 for now. If you want to attempt each of the 6, here is a breakdown:

1 (SRV_FORCE_IGNORE_CORRUPT)

Lets the server run even if it detects a corrupt page. Tries to make SELECT * FROM tbl_name jump over corrupt index records and pages, which helps in dumping tables.

2 (SRV_FORCE_NO_BACKGROUND)

Prevents the master thread and any purge threads from running. If a crash would occur during the purge operation, this recovery value prevents it.

3 (SRV_FORCE_NO_TRX_UNDO)

Does not run transaction rollbacks after crash recovery.

4 (SRV_FORCE_NO_IBUF_MERGE)

Prevents insert buffer merge operations. If they would cause a crash, does not do them. Does not calculate table statistics.

5 (SRV_FORCE_NO_UNDO_LOG_SCAN)

Does not look at undo logs when starting the database: InnoDB treats even incomplete transactions as committed.

6 (SRV_FORCE_NO_LOG_REDO)

Does not do the redo log roll-forward in connection with recovery.

With this value, you might not be able to do queries other than a basic SELECT * FROM t, with no WHERE, ORDER BY, or other clauses. More complex queries could encounter corrupted data structures and fail.

If corruption within the table data prevents you from dumping the entire table contents, a query with an ORDER BY primary_key DESC clause might be able to dump the portion of the table after the corrupted part.

If you happen to get the database up and running and then can do a mysqldump then congratulations! You are in the clear! The best next steps is to

1. Stop the MySQL database server
2. Remove the innodb_force_recovery option from the MySQL config so the database server can operate normally.
3. Restart the MySQL database server.
4. Delete the corrupted MySQL database from the server (Don’t delete the dump file! That is your backup!)
5. Create a new database you want to recover.
6. Import the mysqldump backup into the new database.

And you should be done.