PHP File Upload in Sharded Server Configuration

Question

We use multiple servers to handle incoming web requests which are load-balanced in a round-robin fashion. I've run into an issue that I'm not sure how to solve.

Using AJAX (qqFileUploader), I am uploading a file. By default it goes into the /tmp folder which is fine. The problem is when I try to retrieve that file, that retrieval request gets handled by the next server in line which does not have the file I uploaded. If I keep repeating the request over and over again, it will eventually reach the original server (via round robin load balancing) where the file was stored and then I can open it. Obviously that is not a good solution.

Here is essentially the code: http://jsfiddle.net/Ap27Z/. I removed some of it for brevity. You will see that the uploader object makes a call to a PHP file to do the file upload and then after the file upload is complete, another AJAX call is made to a script to process the .csv file. This is where the process is getting lost in the round-robin.

I read a few questions here on SO relating to uploading files to memory and it seems that it is essentially not currently feasible. Is there another option I can use to upload a file and handle it all within the same request?

Answer 1

The classic solution to this type of problem is to use sticky sessions on your load balancer. That may not be a good solution for you as it would be modifying your whole setup to fix a small problem.

I would suggest adding a sub-domain prefix for each machine e.g. upload goes to www.example.com and then each server is allocated an additional subdomain www1.example.com, www2.example.com which are always passed directly to that server, rather than round robin DNS.

As part of the success result, you could pass back the server name that points to the exact server, rather than the load-balanced name, and then all subsequent Ajax calls that reference the uploaded data use that server specific domain name, rather than the generic load balanced domain name.

Is there another option I can use to upload a file and handle it all within the same request?

Sure, why not? The code that handles the POSTing of the data can do whatever you want it to do.

Answer 2

There are (at least) 2 solutions to your problem:

1. You change the load-balancing.

There are several load balancing proxies out there which support session affinity a.k.a. "sticky sessions". That means that a user always gets the same server within a session.

Two programs that can act in this way are HAProxy (related question here on SO) and nginx with a custon module (tutorial here).

2. You chance the files' location.

The other choice would be to change the location of your stored files to some place that all of your servers can access via the same location. This could be, for example, an NFS mount or a database (with the files stored as BLOBS). This way, it doesn't matter which server processes the request, as all of them have access to the file.