php is locking a deleted session file

Question

Our php app, running under apache, is occaisionally trying to lock a deleted session file, this hangs the apache process and eventually we run out of processes.

The evidence :

strace : flock(89, LOCK_EX                   -----stuck in flock for File Descriptor 89

and then :

lsof, for the same process :

httpd 22161 apache 89u ... /var/lib/php/session/sess_mf7svvirg7rbu9l0m5999pm6h4 (deleted)

Any ideas as to why this is occurring? And what we could do to prevent it?

Answer 1

I've experienced a problem with identical symptoms to yours, except whether the session file was deleted or not was irrelevant (and may be for you too—do you have evidence the deletion of the session file is related to the flock?).

In my case, it was a race condition for access to the same session file between two scripts:

1. /page1.php executes fine when loaded. It includes javascript that does a XMLHttpRequest to /background.php.
2. When /background.php is accessed it runs readfile(http://remote.url), which is a non-existent URL.
3. When navigating to /page2.php the script is stalled. A strace shows flock(89, LOCK_EX, and lsof indicates it is waiting for read/write access on a session file.

In this scenario, both /page2.php and /background.php are both waiting on the same session file, but the latter was unable to do so because it was held up waiting for the readfile() to timeout. I got this in a php_error_log:

PHP Warning:  readfile(http://remote.url): failed to open stream: Connection timed out in […]/background.php on line […]

So the problem was with an entirely different script than the perceived culprit.

You can check for this problem quickly by grepping all opened files for the php session file to see which httpd process is using it:

$ sudo lsof | grep sess_it9q6kkvf83gcj72vu05p9fcad7
httpd     31325    apache   74u      REG                8,5       410   11634061 /var/lib/php/session/sess_it9q6kkvf83gcj72vu05pfa543
httpd     31632    apache   74uW     REG                8,5       410   11634061 /var/lib/php/session/sess_it9q6kkvf83gcj72vu05pfa543

If two httpd processes are accessing the same session file, probably this is your problem. Now you can check what those processes are doing with strace, or in my case it was enough to use apachectl fullstatus:

$ sudo apachectl fullstatus | grep 31632
5-7  31632 0/1/1169  _ 0.05 6     30692 0.0  0.02  3.45 111.222.333.444 www.example.com.com     /background.php