Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Bulk Collection Manipulation through a REST (RESTful) API

Tags:

rest

http

I'd like some advice on designing a REST API which will allow clients to add/remove large numbers of objects to a collection efficiently.

Via the API, clients need to be able to add items to the collection and remove items from it, as well as manipulating existing items. In many cases the client will want to make bulk updates to the collection, e.g. adding 1000 items and deleting 500 different items. It feels like the client should be able to do this in a single transaction with the server, rather than requiring 1000 separate POST requests and 500 DELETEs.

Does anyone have any info on the best practices or conventions for achieving this?

My current thinking is that one should be able to PUT an object representing the change to the collection URI, but this seems at odds with the HTTP 1.1 RFC, which seems to suggest that the data sent in a PUT request should be interpreted independently from the data already present at the URI. This implies that the client would have to send a complete description of the new state of the collection in one go, which may well be very much larger than the change, or even be more than the client would know when they make the request.

Obviously, I'd be happy to deviate from the RFC if necessary but would prefer to do this in a conventional way if such a convention exists.

like image 689
mattwynne Avatar asked Nov 20 '08 18:11

mattwynne


People also ask

CAN REST API handle bulk data?

Bulk operations on REST resources Sometimes only a single operation needs to support bulk data. In such a case, we can simply create a new resource that accepts a collection of bulk entries.

What is difference between REST API and RESTful API?

Put simply, there are no differences between REST and RESTful as far as APIs are concerned. REST is the set of constraints. RESTful refers to an API adhering to those constraints. It can be used in web services, applications, and software.

What is a REST API collection?

REST API collection resources provide access to information about a list of IBM® Streams objects of the same type. For example, you can use a collection resource to access information about a list of jobs. Collection resources can be paged, sorted, and filtered.


3 Answers

You might want to think of the change task as a resource in itself. So you're really PUT-ing a single object, which is a Bulk Data Update object. Maybe it's got a name, owner, and big blob of CSV, XML, etc. that needs to be parsed and executed. In the case of CSV you might want to also identify what type of objects are represented in the CSV data.

List jobs, add a job, view the status of a job, update a job (probably in order to start/stop it), delete a job (stopping it if it's running) etc. Those operations map easily onto a REST API design.

Once you have this in place, you can easily add different data types that your bulk data updater can handle, maybe even mixed together in the same task. There's no need to have this same API duplicated all over your app for each type of thing you want to import, in other words.

This also lends itself very easily to a background-task implementation. In that case you probably want to add fields to the individual task objects that allow the API client to specify how they want to be notified (a URL they want you to GET when it's done, or send them an e-mail, etc.).

like image 72
Jamie Flournoy Avatar answered Sep 23 '22 19:09

Jamie Flournoy


Yes, PUT creates/overwrites, but does not partially update.

If you need partial update semantics, use PATCH. See http://greenbytes.de/tech/webdav/draft-dusseault-http-patch-14.html.

like image 41
Julian Reschke Avatar answered Sep 20 '22 19:09

Julian Reschke


You should use AtomPub. It is specifically designed for managing collections via HTTP. There might even be an implementation for your language of choice.

like image 22
dowski Avatar answered Sep 20 '22 19:09

dowski