-bash: fork: Cannot allocate memory

Question

When I run any command in bash shell, it returns:

$ free -m
-bash: fork: Cannot allocate memory

Then I tried debugging for memory leaks with the ps command. It returns:

$ ps --sort -rss -eo rss,pid,command | head
RSS   PID COMMAND
518116 1310 /usr/bin/influxd -pidfile /var/run/influxdb/influxd.pid -config /etc/influxdb/influxdb.conf
281700 1912 /opt/td-agent/embedded/bin/ruby /usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
68260 23896 /home/alphauser/Envs/vogo-alpha/bin/python ./manage.py runserver 0.0.0.0:8008
43720 20201 python ./manage.py runserver 0.0.0.0:8008
36928  2223 rsyslogd
27432  1909 /opt/td-agent/embedded/bin/ruby /usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
25296 15009 /usr/bin/telegraf -pidfile /var/run/telegraf/telegraf.pid -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d
24428 32529 /usr/sbin/grafana-server --pidfile=/var/run/grafana-server.pid --config=/etc/grafana/grafana.ini cfg:default.paths.data=/var/lib/grafana cfg:default.paths.logs=/var/log/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins
20792  2314 /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9025/bin/mdsd -A -C -c /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.9025/./xmlCfg.xml -p 29131 -R -r lad_mdsd -e /var/log/azure/Microsoft.OSTCExtensions.LinuxDiagnostic/2.3.9025/mdsd.err -w /var/log/azure/Microsoft.OSTCExtensions.LinuxDiagnostic/2.3.9025/mdsd.warn -o /var/log/azure/Microsoft.OSTCExtensions.LinuxDiagnostic/2.3.9025/mdsd.info

How to debug? What is the core problem?

Answer 1

I also faced this issue with my Ubuntu 14.04 desktop.

 free -m

Even these basic command showed Can't allocate memory error. On investigating, found that system is using all the memory for Caching and is not freeing up memory. This is called Cache Ballooning and solved this by clearing the cache.

Answer 2

In my case, OS was running out of PID instead of memory, the error message was the same though.

The default value of max PID number is 32768, to view the value, run

cat  /proc/sys/kernel/pid_max

To change the max pid number, run

echo 100000 > /proc/sys/kernel/pid_max

In my scenario, the root cause was that one java prcocess spawned 18k+ threads( in linux kernel, thread is essentially a process), to find out the thread count of each process, run

ps -eo nlwp,pid,args --sort nlwp

Answer 3

You might first want to check your system memory usage to see if there's enough free memory left.

If not, which is my case, please check your ulimit by typing ulimit -a to see if you've reached the limit of max open files (mostly caused by some certain processes that occupies a lot of file descriptors). In this case adjusting the ulimit will solve your problem.

Answer 4

I had the same problem. In my Case, after knowing about the detail of memory with "proc/meminfo", I found the PIDs that they used a lot of CPU and memory with "TOP".After that, I checked how long they have run with "ps -o etime= -p "PID" ". Then I kill the PIDs with "kill -9 PID".