I am experiencing a recurring issue why my ubuntu server complete becomes unavailable around the same time for the past two days. I cant even SSH into it during the crash.
The Ubuntu server is on a amazon aws t2.micro instance, running on 1 gb of ram , Ubuntu 16.04 The apache error logs are shown below
[Wed Aug 30 18:02:23.710072 2017] [autoindex:error] [pid 7505] [client 60.191.38.77:57957] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.cgi,$
[Wed Aug 30 18:02:23.710126 2017] [:error] [pid 7505] [client 60.xx.xx.xx:57957] script '/var/www/html/404.php' not found or unable to stat
[Wed Aug 30 19:11:54.375001 2017] [autoindex:error] [pid 11307] [client 45.55.21.189:52050] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.cgi$
[Wed Aug 30 19:11:54.376134 2017] [:error] [pid 11307] [client 45.xx.xx.xx:52050] script '/var/www/html/404.php' not found or unable to stat
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[Wed Aug 30 20:50:00.570286 2017] [core:notice] [pid 31139] AH00051: child pid 17670 exit signal Aborted (6), possible coredump in /etc/apache2
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[Wed Aug 30 21:03:27.234926 2017] [core:notice] [pid 31139] AH00051: child pid 18507 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:27.354905 2017] [core:notice] [pid 31139] AH00051: child pid 18511 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:27.354927 2017] [core:notice] [pid 31139] AH00051: child pid 18512 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:42.865027 2017] [core:notice] [pid 31139] AH00051: child pid 18506 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:46.984235 2017] [core:notice] [pid 31139] AH00051: child pid 18529 exit signal Aborted (6), possible coredump in /etc/apache2
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[Wed Aug 30 21:14:50.194072 2017] [core:notice] [pid 31139] AH00051: child pid 18605 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Aug 30 21:14:50.482541 2017] [core:notice] [pid 31139] AH00051: child pid 18618 exit signal Aborted (6), possible coredump in /etc/apache2
[Thu Aug 31 07:09:50.271441 2017] [mpm_prefork:notice] [pid 1321] AH00163: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Thu Aug 31 07:09:50.274044 2017] [core:notice] [pid 1321] AH00094: Command line: '/usr/sbin/apache2'
The server becomes inaccessible around this time Wed Aug 30 19:11:54 based on the logs and traffic monitor on AWS.
I have little understanding of how Apache works in terms of memory and concurrent connections. The website I have in it has low traffic. Is there some config file I might have messed up? .BTW memory_limit is -1 no limit as per the PHP docs Or what can I do to find the cause of the problem, I am thinking of migrating the server files to a AWS beanstalk instance (A deploying aide plus scaling aide,AFAIK) a fresh start.
Is there away I could know the script(s) that might have caused the crash.I have too many scripts plus running on laravel PHP framework + third party scripts would really make it hard to outline which one is the suspect.
I would greatly appreciate an answer that explains the error log.i.e
A script did this...
Apache seems to have done this...
Ubuntu then did this...
Which led to this...
Borrowing from what @arkascha said in the comments the crash was caused by some script that was using a lot of memory and not well optimised. The following steps will help others facing similar problems.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With