Random session data loss in PHP

Question

Here is the problem we have been facing for the past few weeks.

1/ Our setup

• PHP 5.4 + MySQL
• 2 dedicated servers, load-balanced
• Sessions are replicated between the 2 servers using memcached
• 3 applications running on these servers :
  • One custom-developped application, using default php session settings
  • Another custom-developped application, using different session settings (cookie name, path)
  • One Wordpress CMS

2/ The problem

The problem occurs on our first application.

Some of our users reported that they sometimes get disconnected after a few minutes (when the session is setup to last 3 hours). It can happen to them several time in the same day, then no disconnection for a few days, but the problem always comes back. So far the fraction of users impacted is small, but I would like to solve this before it "spreads" to other users.

The problem seems to occur in different places of the application, though we have identified 3 scenarii where most of the errors occur :

• Some involve submitting a form ($_SESSION variable is modified)
• Other simply involve opening a popup page, with no modification of the session data

We have tried to reproduce the different scenarii described by the users : sometimes we have been able to, but most of the time we don't have any problem, which makes it hard to debug.

Other notes :

• The problem is recent, this application had been running for years without any problem.
• It doesn't seem to be related to our server load, because the problem still occured during the summer break when our trafic was low
• It only affects one session/users at a time: all the other users logged in at the same time don't experience this problem
• The problem occured on all the different browsers (IE, Firefox, Chrome)

3/ Technical analysis

When a disconnect occurs, the user is redirected to a page "Your session has expired or you don't have the right to view". When this page is loaded, we get a technical email with a dump of the $_SESSION variable.

When a session expires the normal way, the email we get shows that the $_SESSION variable is empty (normal behavior). When an unexpected disconnect occurs, what is interesting is that the $_SESSION is not entirely empty : out of the ~20 elements the array contained, only one is left (always the same).

So this would mean the session is not expired, but not enough data is left to "identify" the user, hence the "no rights" page displayed. As a confirmation when this occurs, we can check in memcached that this session still holds some data.

These are the potential problem causes we have identified so far, and what we have done to rule them out :

• Memcached indicates between 70 et 80% freespace, so we don't think it is the problem.
• We removed Memcached and went back to using a NFS shared directory for session files: the problem actually got worse. This would point to an applicative bug, because NFS being slower to write data, session loss would occur more often.
• We have browsed all the different forums (including SO) talking about PHP session data loss, and reviewed our code accordingly. The code base is big, but we have used automated tools and scripts to avoid missing a file.
  • session_start() is called at the beginning of each page.
  • exit() is called after each header("Location...")
  • register_globals is Off
• We have tested the possible interractions between our 2 other applications and the problematic one, though they don't share any code, database or session handling. Nothing identified there.
• We have analyzed our access logs around the times of the disconnections, to check for behavior patterns : no luck here either.

So we have no idea what causes this problem, as it seems to occur randomly, so my questions are :

• The problem could come from our code: did we miss anything to check ? This solutions seems unlikely as the code works most of the time for all our users, but I am still considering it.
• The problem could come from another application/process that would "empty" part of the session variable array. We have also reviewed the code from the other applications, but didn't find anything that could cause this. And if another process is doing this, why would it only empty some sessions and not all of them ?

Thanks for your help.

Answer 1

I don't think you'll get a definitive answer to your question. There are too many probable causes and you haven't shown any code.

Still, my guess is that you have memcached.sess_locking turned Off, or if you have a custom session implementation - that it doesn't implement locking at all.
Eventually, this leads to a race condition between two simultaneous HTTP requests.

My guess is based on the often seen bad advice to turn off locks or free them as soon as possible, in order to achieve higher performance.