Menu
Environment Details:
Server: Amazon ec2 Linux
Web Server: Apache
Web Framework: Django with mod_wsgi
Following I have found in the mysql_err.log file.
The InnoDB memory heap is disabled
120823 3:21:40 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120823 3:21:40 InnoDB: Compressed tables use zlib 1.2.3
120823 3:21:40 InnoDB: Using Linux native AIO
120823 3:21:41 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
120823 3:21:41 InnoDB: Completed initialization of buffer pool
120823 3:21:41 InnoDB: Fatal error: cannot allocate memory for the buffer pool
120823 3:21:41 [ERROR] Plugin 'InnoDB' init function returned error.
120823 3:21:41 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
120823 3:21:41 [ERROR] Unknown/unsupported storage engine: InnoDB
120823 3:21:41 [ERROR] Aborting
Looks like the system memory is not sufficient to allocate memory to buffer pool. Same error happens when I was using Amazon ec2 micro instance, So I moved to the small instance. It works fine for some days but now it is breaking once in a day again. Is there a permanent fix for that? I can move to medium instance but the issue is will that be fixed or not? Should I decrease the innodb_buffer_pool_size, what is the preferred size?
The result of cat /proc/meminfo is following (may be it will help):
MemTotal: 1697824 kB
MemFree: 125744 kB
Buffers: 109704 kB
Cached: 481408 kB
SwapCached: 0 kB
Active: 1212396 kB
Inactive: 266840 kB
Active(anon): 888192 kB
Inactive(anon): 76 kB
Active(file): 324204 kB
Inactive(file): 266764 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 888144 kB
Mapped: 15604 kB
Shmem: 144 kB
Slab: 63752 kB
SReclaimable: 53680 kB
SUnreclaim: 10072 kB
KernelStack: 800 kB
PageTables: 16436 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 848912 kB
Committed_AS: 1417140 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 10988 kB
VmallocChunk: 34359725168 kB
DirectMap4k: 1748992 kB
DirectMap2M: 0 kB
OS version (uname -a):
Linux ip-10-246-134-149 3.2.21-1.32.6.amzn1.x86_64 #1 SMP Sat Jun 23 02:32:15 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
I checked the ps aux command whet the server has like only 15MB of memory left and these are the httpd process running at that time:
The result of free -m
total used free shared buffers cached
Mem: 1657 1628 29 0 3 19
-/+ buffers/cache: 1605 51
Swap: 895 875 20
The result of ps aux
apache 21123 0.1 1.2 394652 20464 ? S 19:35 0:06 /usr/sbin/httpd
apache 21146 0.1 1.2 394280 20796 ? S 19:38 0:06 /usr/sbin/httpd
apache 21152 0.1 1.2 394284 21560 ? S 19:38 0:05 /usr/sbin/httpd
apache 21155 0.2 1.4 396244 24528 ? S 19:38 0:06 /usr/sbin/httpd
apache 21156 0.1 1.1 392552 20344 ? S 19:38 0:06 /usr/sbin/httpd
apache 21157 0.1 1.1 394284 18884 ? S 19:38 0:05 /usr/sbin/httpd
apache 21159 0.1 1.4 396200 25040 ? S 19:38 0:06 /usr/sbin/httpd
apache 21161 0.1 1.2 394856 21724 ? S 19:38 0:06 /usr/sbin/httpd
apache 21162 0.1 1.3 394864 22400 ? S 19:38 0:06 /usr/sbin/httpd
apache 21163 0.1 1.3 394860 22204 ? S 19:38 0:06 /usr/sbin/httpd
apache 21164 0.1 1.1 392560 19204 ? S 19:38 0:06 /usr/sbin/httpd
apache 21165 0.1 1.3 394832 22280 ? S 19:38 0:06 /usr/sbin/httpd
apache 21166 0.1 1.3 395276 22932 ? S 19:38 0:06 /usr/sbin/httpd
apache 21172 0.2 1.4 396320 24820 ? S 19:38 0:06 /usr/sbin/httpd
apache 21174 0.2 1.7 400672 29452 ? S 19:39 0:06 /usr/sbin/httpd
apache 21178 0.1 1.4 400540 25304 ? S 19:39 0:06 /usr/sbin/httpd
apache 21179 0.2 1.6 400580 27856 ? S 19:39 0:06 /usr/sbin/httpd
apache 21184 0.1 1.7 400628 29320 ? S 19:39 0:06 /usr/sbin/httpd
apache 21185 0.1 1.6 397944 27292 ? S 19:39 0:05 /usr/sbin/httpd
apache 21186 0.1 1.5 397960 25648 ? S 19:39 0:05 /usr/sbin/httpd
apache 21187 0.1 1.7 400576 29120 ? S 19:39 0:06 /usr/sbin/httpd
apache 21191 0.1 1.4 400576 24400 ? S 19:39 0:06 /usr/sbin/httpd
apache 21193 0.1 1.4 400536 24940 ? S 19:39 0:05 /usr/sbin/httpd
apache 21194 0.1 1.5 400572 26096 ? S 19:39 0:06 /usr/sbin/httpd
apache 21203 0.1 1.6 400580 28808 ? S 19:39 0:05 /usr/sbin/httpd
apache 21206 0.1 1.7 400584 29732 ? S 19:39 0:06 /usr/sbin/httpd
apache 21207 0.1 1.6 400576 27940 ? S 19:39 0:06 /usr/sbin/httpd
apache 21224 0.1 1.2 400624 20768 ? S 19:39 0:06 /usr/sbin/httpd
apache 21225 0.1 1.6 400576 28468 ? S 19:39 0:05 /usr/sbin/httpd
apache 21226 0.1 1.6 400576 28048 ? S 19:39 0:06 /usr/sbin/httpd
apache 21228 0.1 1.4 400572 23880 ? S 19:39 0:06 /usr/sbin/httpd
apache 21237 0.1 1.5 400628 26124 ? S 19:39 0:06 /usr/sbin/httpd
apache 21265 0.1 1.6 400536 28592 ? S 19:39 0:06 /usr/sbin/httpd
apache 21276 0.1 1.2 400544 21456 ? S 19:39 0:05 /usr/sbin/httpd
apache 21277 0.1 1.3 400624 22676 ? S 19:39 0:05 /usr/sbin/httpd
apache 21278 0.1 1.6 400536 27360 ? S 19:39 0:06 /usr/sbin/httpd
apache 21282 0.1 1.4 400612 24996 ? S 19:39 0:06 /usr/sbin/httpd
apache 21292 0.1 1.4 400532 24780 ? S 19:39 0:05 /usr/sbin/httpd
apache 21302 0.2 1.2 400540 21332 ? S 19:39 0:06 /usr/sbin/httpd
apache 21303 0.1 1.3 400628 22228 ? S 19:39 0:06 /usr/sbin/httpd
apache 21305 0.2 1.2 400536 21116 ? S 19:39 0:06 /usr/sbin/httpd
apache 21306 0.1 1.3 400572 22380 ? S 19:39 0:06 /usr/sbin/httpd
apache 21307 0.1 1.1 397956 20056 ? S 19:39 0:05 /usr/sbin/httpd
apache 21308 0.1 1.2 400624 21520 ? S 19:39 0:06 /usr/sbin/httpd
apache 21319 0.1 1.1 400540 19468 ? S 19:39 0:05 /usr/sbin/httpd
apache 21320 0.1 1.3 400628 22712 ? S 19:39 0:05 /usr/sbin/httpd
apache 21335 0.1 1.0 400540 17236 ? S 19:39 0:05 /usr/sbin/httpd
apache 21336 0.1 1.3 400628 22188 ? S 19:39 0:06 /usr/sbin/httpd
apache 21352 0.1 1.1 394276 18972 ? S 19:40 0:04 /usr/sbin/httpd
apache 21356 0.1 1.1 394280 19028 ? S 19:40 0:05 /usr/sbin/httpd
apache 21358 0.1 1.1 394280 19004 ? S 19:40 0:05 /usr/sbin/httpd
apache 21361 0.2 0.7 400452 12632 ? S 19:40 0:06 /usr/sbin/httpd
apache 21610 0.2 1.6 400536 27660 ? S 19:46 0:06 /usr/sbin/httpd
apache 21643 0.2 1.3 400156 23272 ? S 19:55 0:04 /usr/sbin/httpd
apache 21647 0.2 1.0 400544 17556 ? S 19:57 0:05 /usr/sbin/httpd
apache 21654 0.2 1.5 400188 26884 ? S 19:58 0:05 /usr/sbin/httpd
apache 21719 0.3 1.9 400192 32264 ? S 20:14 0:03 /usr/sbin/httpd
apache 21725 0.2 2.0 400044 35340 ? S 20:15 0:03 /usr/sbin/httpd
apache 21738 0.0 0.8 257648 13792 ? S 20:26 0:00 /usr/sbin/httpd
Can any one one has an idea about it why there is so much httpd process??
919
You can decrease the innodb_buffer_pool_size very low to see if it helps:
#/etc/my.cnf
innodb_buffer_pool_size = 1M
A rule of thumb is to set innodb_buffer_pool_size to 50% of available RAM for your low memory testing. This means you start the server and everything except MySQL InnoDB. See how much RAM you have. Then use 50% of that for InnoDB.
To try many low-memory settings at once:
A more likely culprit is whatever else is on that server, such as a webserver.
Are you using Apache and/or another webserver? If so, try to decrease its RAM usage. For example in Apache conf, consider low RAM settings like these:
StartServers 1
MinSpareServers 1
MaxSpareServers 5
MaxClients 5
And cap the requests like this:
MaxRequestsPerChild 300
Then restart Apache.
If you're using Apache with mod_python, switch to Apache with mod_wsgi.
If it's still happening, possibly your Django is steadily growing. Try Django memory profiling with Pympler:
Your report of once-per-day failures, then once-per-week failures, could point to some kind of cron job running daily or weekly. For example, perhaps there's a batch process that takes up a lot of RAM, or a database dump, etc.
To track RAM use and look for RAM spikes in the hour before MySQL dies, take a look at SAR, which is a great tool: http://www.thegeekstuff.com/2011/03/sar-examples/
89
You have to decrease you innodb_buffer_pool_size = <60-80% of your main memory)
Solution for Innodb Error:
110603 7:34:15 [ERROR] Plugin ‘InnoDB’ init function returned error.
110603 7:34:15 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed.
110603 7:34:15 [ERROR] Unknown/unsupported storage engine: InnoDB
110603 7:34:15 [ERROR] Aborting
10603 7:34:15 [Note] /usr/sbin/mysqld: Shutdown complete
I moved the ib_logfile0 and ib_logfile01 to bak and start Mysql again. Now this time, it is working fine
[root@xxx mysql]# mv ib_logfile0 ib_logfile0-bak
[root@xxx mysql]# mv ib_logfile1 ib_logfile1-bak
Source: http://www.onaxer.com/tag/error-plugin-innodb-init-function-returned-error/
23
Like others have mentioned, the problem appears to be your system running low on RAM and MySQL is blowing up due to that. Below is how to go about narrowing down where your system's memory is being used and how to recover from the database going down.
Take a look at collectd and its plugins. Some of the applicable ones may be the processes plugin and the memory plugin. With those you can see your systems' memory usage and what processes are taking up most of it.
Depending on how you are running Django, you can configure the worker processes to only process a certain number of requests and then terminate. That way if there is some sort of memory leak in your application it will not persist past that number of requests. For example, if you use Gunicorn, you can use the --max-requests option. Setting it to 500 will drop the worker after it has processed 500 requests.
The above combined with stats collection will show you some interesting memory usage trends.
As for the database going down, you can setup process supervision so that if MySQL does die, it will be relaunched automatically. MySQL in latest version of Ubuntu uses Upstart to do just that. If the process dies, Upstart will bring it back up immediately. If you're using another distro that doesn't have this built-in, take a look at Supervisor. While this doesn't fix the problem it will at least mitigate its effects. This should not be seen as the fix but rather a way to keep your application running in case something does go wrong.
3
Once I got stuck in similar issues, I was really frustrated that my users see this ugly message that Error Establishing DB Connection. Instead of resolving the exact issues I found this repo to work like a charm for me (temporarily). After that I got debugged it by my friend and he just fine tuned my server with some configuration changes. But I've still added this script to my crontab every 10 minutes and then check if the server is crashed (which for my case crashed eventually whenever I run VNCServer on my server) and then restart it
2
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
