Error with time-consuming REST service in Python

Question

I'm writing the code to attend a service. I receive POST requests to a particular service and start processing. The whole process is rather simple; I loop trough items in the request and add each item to the database. The problem arise when I have to process a lot of items and the loop takes like three minutes to finish, then when I try to respond:

status = '200 OK'
headers = [('Content-type', 'application/json'),('Access-Control-Allow-Origin','*')]
start_response(status, headers)

return json.dumps(response)

I get this error:

Exception happened during processing of request from ('XXX.XXX.XXX.XXX', 49172)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 310, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python2.7/SocketServer.py", line 640, in __init__
    self.finish()
  File "/usr/local/lib/python2.7/SocketServer.py", line 693, in finish
    self.wfile.flush()
  File "/usr/local/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe

I don't know if this helps, but the POST request is a forwarded POST made from a browser to a different domain (that's why the a Access-Control-Allow-Origin) and all the accesses to the database are made using a single object that interacts with the database using SQLAlchemy (can be seen similar to a Java EE DAO pattern).

How do I avoid this error?

Answer 1

You maybe are violating the idea behind REST.

If the proccessing could take some time, the service may want to answer it with an 202 Accepted Response! For a full overview of http response codes follow this link: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

202 Accepted

The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.

The 202 response is intentionally non-committal. Its purpose is to allow a server to accept a request for some other process (perhaps a batch-oriented process that is only run once per day) without requiring that the user agent's connection to the server persist until the process is completed. The entity returned with this response SHOULD include an indication of the request's current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.