I have written my own Restful API and am wondering about the best way to deal with large amounts of records returned from the API.
For example, if I use GET
method to myapi.co.uk/messages/
this will bring back the XML for all message records, which in some cases could be 1000's. This makes using the API very sluggish.
Can anyone suggest the best way of dealing with this? Is it standard to return results in batches and to specify batch size in the request?
To handle 'millions of request' the system must be deployed on multiple web servers behind a load-balancer that would round robin between each. if the system is hitting a datastore, a second level cache(ehcache, memcache,etc.) should be used to reduce load on the datastore.
The maximum data payload size for requests to this endpoint is 128 mb. The maximum rate limit is 60 files per hour per account (aka profile).
REST API call limits The maximum payload limit for a single API call is 45 MB, so ensure that the aggregate size of the records in a request do not exceed this limit.
You can change your API to include additional parameters to limit the scope of data returned by your application.
For instance, you could add limit
and offset
parameters to fetch just a little part. This is how pagination can be done in accordance with REST. A request like this would result in fetching 10 resources from the messages collection, from 21st to 30th. This way you can ask for a specific portion of a huge data set:
myapi.co.uk/messages?limit=10&offset=20
Another way to decrease the payload would be to only ask for certain parts of your resources' representation. Here's how facebook does it:
/joe.smith/friends?fields=id,name,picture
Remember that while using either of these methods, you have to provide a way for the client to discover each of the resources. You can't assume they'll just look at the parameters and start changing them in search of data. That would be a violation of the REST paradigm. Provide them with the necessary hyperlinks to avoid it.
I strongly recommend viewing this presentation on RESTful API design by apigee (the screencast is called "Teach a Dog to REST"). Good practices and neat ideas to approach everyday problems are discussed there.
EDIT: The video has been updated a number of times since I posted this answer, you can check out the 3rd edition from January 2013
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With