Best practice for implementing long-running searches with REST

Question

As part of a REST Service, I need to implement a search call. The basic idea is that the user can POST a new search, and GET the results:

POST http://localhost/api/search
GET http://localhost/api/search?id=123

However, my search may run a few minutes, and return partial results until it is done. That is, the GET-Request would return something like:

status: running
results: a, b, c.

while the next GET-Request might return

status: completed
results: a, b, c, d, e.

This contradicts the semantics of a RESTful GET request. The request should always return the same result when called several times. For example, when the user uses a caching proxy, the full results might never be delivered to the user.

Question: Is there any way to provide a truly RESTful implementation for long running searches with partial results?

Answer 1

While the search is executing, you could set the appropriate response headers (e.g. Expires or max-age) to indicate that the response should not be cached (HTTP/1.1 14.9.3, 13.4).

Once the search result is complete, you could then send a more appropriate Expires / max-age header to allow or extend the cacheability of the result.

The burden would be on the client to re-query the resource until its search status is complete. The client could maybe use the value of the Expires header to determine when it should re-query for updated results.

Alongside of this, you could also use a custom 2XX status code to indicate that the result is not yet complete. Maybe a HTTP/1.1 299 In Progress, or whatever makes sense. The spec indicates that HTTP status codes are extensible.

---

For the record, your statement:

This contradicts the semantics of a RESTful GET request. The request should always return the same result when called several times.

is not true for GET requests - resources can change. That GET requests are idempotent only means that "...the side-effects of N > 0 identical requests is the same as for a single request". ^[spec]

Answer 2

A few days ago I happend to stumble upon a blog post over at reddit that deals with your problem. You might want to check it out: Bill Higgin's RESTy long-ops.

Happy reading.

Answer 3

It's not a problem if the first GET request returns partial results, and the second GET request returns the full results. That's because the first GET request doesn't cause the result of the second request to change: That request would have returned the full results, even if the first GET hadn't been issued. "idempotent" doesn't mean identical results. It means that the first GET doesn't affect the second GET.

It would be a problem if the first GET request returned partial results, and the second GET would return the remaining results (first GET returns A, B, C; second GET returns D, E, F). Here the first GET changes the second result, so it's not RESTful.