bash fork error (Resource temporarily unavailable) does not stop, and keeps showing up every time I try to kill/reboot

Question

I mistakenly used a limited server as an iperf server for 5000 parallel connections. (limit is 1024 processes) Now every time I log in, I see this:

-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

Then, I try to kill them, but when I do ps, I get this:

-bash-4.1$ ps
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

Same happens when I do a killall or similar things. I have even tried to reboot the system but again this is what I get after reboot:

-bash-4.1$ sudo reboot
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
-bash-4.1$ 

So Basically I cannot do anything. all the commands get this error :/ I can, however, do "exit".

This is an off-site server that I do not have physical access to, so I cannot turn it off/on physically.

Any ideas how I can fix this problem? I highly appreciate any help.

Answer 1

Given that you can login, you may want to try using exec to execute all your commands. After executing exec you will have to log in again, since exec will kill your shell (by replacing it with the command you run).

exec won't take up an extra process slot because it will replaces the running shell with the program to run. Thus, it should be able to bypass the ulimit restriction.

Answer 2

I had the same issue recently. In my case the reason was there was code that was executing under my ownership and consumed almost all the resources leaving nothing for my commands. Here's what I did, "exec top" to identify the PID thats consuming maximum resources "exec kill -9 " killing the PID identified by above command.

After killing the PID, everything came back to normal and I was able to login back.