Adapter Proxy for Restful APIs

Question

this is a general 'what technologies are available' question.

My company provides a web application with a RESTful API. However, it is too slow for my needs and some of the results are in an awkward format.

I want to wrap their restful server with a proxy/adapter server, so when you connect to the proxy you get the RESTful API I wish the real one provides.

So it needs to do a few things:

• passthrough most requests
• cache some requests
• do some extra requests on the original server to detect if a request is cacheable

for instance: there is a request for a field in a record: GET /records/id/field which might be slow, but there is a fingerprint request GET /records/id/fingerprint which is always fast. If there exists a cache of GET /records/1/field2 for the fingerprint feedbeef, then I need to check the original server still has the fingerprint feed beef before serving the cached version.

• fix headers for some responses - e.g. content-type, based upon the path
• do stream processing on some large content, for instance

GET /records/id/attachments/1234

returns a 100Mb log file in text format

remove null characters from files optionally recode the log to filter out irrelevant lines, reducing the load on the client cache the filtered version for later requests.

---

While I could modify the client to achieve this functionality, such code would not be re-usable for other clients (different languages), and complicates the client logic.

---

I had a look at whether clojure/ring could do it, and while there is a nice little proxy middleware for it, it doesn't handle streaming content as far as I can tell - the whole 100Mb would have to be downloaded. Also it doesn't include any cache logic yet.

---

I took a look at whether squid could do it, but I'm not familiar with the technology, and it seems mostly concerned with passing through requests rather than modifying them on the fly.

---

I'm looking for hints where I might find the correct technology to implement this. I'm mostly language agnostic if learning a new language gets me access to a really simple way to do it.

Answer 1

I believe you should choose a platform that is easier for you to implement your custom business logic on. The following web application frameworks provide easy connectivity with REST APIs, and allow you to create a web application that could work as a REST proxy:

• Play framework (Java + Scala)
• express + Node.js (Javascript)
• Sinatra (Ruby)

I'm more familiar with Play, of which I know it provides utilities for caching you could find useful, and is also extendable by a number of plugins.

If you are familiar with Scala, you could have a also have a look at Finagle. It is a framework build be Twitter's infrastructure team to provide protocol-agnostic connectivity. It might be an overkill for REST to REST proxy, but it provides abstractions you might find useful.

You could also look at some 3rd party services like Apitools, which allows to create a proxy programmatically (in lua). Apirise is a similar service (of which I'm a co-founder) that intends to do provide similar functionalities with a user-friendly UI.

Answer 2

Beeceptor does exactly what you want. It plugs in-between your web-app and original API to route requests.

• For your use-case of caching a few responses, you can create a rule. That way it shall not hit the original endpoint.
• The requests to original APIs can be mocked, and you can inspect response
• You can simulate delays.

(Note: it is a shameless plug, I am the author of Beeceptor and thought it should help you and other developers.)