Having a public API but only allowing access to requests sent from my website

Question

I have been searching on Google and SO for hours now but without finding the someone with the same challenge as I now face so here goes:

We have a database with which we put a lot of money and effort into maintaining. The data from the database is publicly available through an REST-API. We also have a public javascript web app that consumes this API and which we sell to some 30-40 customers. As the data in the API is quite valuable to us we want to try to secure it so that no one can scrape the content from it and make their own copy of our database. Nor do we want anyone building services using our API without our consent. At the same time, we need our web app at http://www.example.com/theApp, http://www.example2.com/theApp, http://www.example3.com/theApp etc to be able to access the API. There are no users involved. Everyone can go to http://www.example.com/theApp and get the full feature of the site. The API is also read only so we are not concerned by anyone trying to pollute our data.

The javascript web app is built with react.js with a node.js server. SSL will, of course, be used for all communication between servers and client.

Things that I think will NOT work:

• Making the users log in to use http://www.example.com/theApp. (There are no extra features for logging in so this will just annoy the users.)
• Storing a password/token in the web app. ( Nothing in javascript is safe however mush obfuscating one does.)
• Making the web app client request a token from an authentication server. (This request can also be done by hostile clients.)
• Whitelisting allowed IPs. (Everyone can access the sites...)
• Whitelisting URLs. (URL is sent as a header. Headers can be manipulated.)

Things that might work (or at least be part of the solution):

• Having the node.js server-app request a token from the authentication server on behalf of the client. (But I still do not know how this will prevent a hostile client making the same request to the node.js server-app)
• Obfuscating. (This could put off most hostile users, but the more persistent (and most skilled/dangerous?) would just be excited by the challenge and will eventually be able to crack it.)

As this is quite a complicated thing we are trying to achieve, heck I am starting to believe it might be impossible, I would greatly appreciate if anyone has some advice about what to do. "Don't do it" is a perfectly good answer if good reasons are given. I am more of the conceptual solution here, but if anyone wants to be concrete on software we have a Linux environment with node.js, Nginx and PHP.

Answer 1

If you don't want to authenticate your users (you don't want them to log in), you have no way to tell who is consuming your API, or what requests come from which user. All the information needed to make an API request is already in the javascript client, anybody can make another client, or a valid request to the API, and you can't even tell if the request is from a new client or the same as in a previous request (somebody that wants to download your database could just distribute downloading across many client computers).

Daniel's answer is probably the closest you can get to your goal this way. If you issue tokens, those at least can be revoked. However, you would still have no way to stop an attacker from requesting a new one.

My thought after reading your question was that a javascript web application is probably not what you want. You are saying you have relatively few clients and you don't want login at all. Would that be an option to give them something like a desktop/mobile client? It could still be Javascript wrapped in some kind of a container, but for each client you could compile one with their own key included. An obvious risk is that they could still extract the key, but this way not anyone could have a client with a key, in fact, a leaked key would be associated with its rightful owner, and you could have contractual clauses to cover that scenario (prohibiting reverse engineering, etc). You could also monitor and control mass-downloads of your database and revoke offending keys, or implement additional security measures like restricting key access to certain client IP addresses if that's possible in your scenario.

This risk of course may or may not be acceptable in your case, just an idea. Obviously, it would be easy to get a key from a client, the point is not everybody would have a client to get a valid key from. A better way would probably be to just distribute keys separately as license files for the client, it's essentially the same, but without the concept of "hard-coded keys" (which I think they are not really in this case). It would make your life easier as you would not need to distribute a full client in case of a revoked key, only a new license file.

This is of course authentication, but in a way that could probably be more user-friendly in your case, as users would not need to do anything.

Another thought for that number of clients is client certificates. To any client that wants to consume your API, you would give a client certificate. Anyone could download your web app, but client certificates could be used to authenticate callers to the API. It's the same as keys above, but at a different level. It's more difficult (and possibly more expensive) to revoke and issue a new one, and you could still not prevent downloading your whole database, but again, you would have proof who did that, and could revoke their key and cover this in your client contracts.

Answer 2

Obfuscation is your only option in the scenario described. You want to make it very hard for an attacker to reverse engineer your web client.

If the client is required to make a complicated computation in order to make a successful API request, then an attacker has to reverse engineer that computation and replicate it in another application.

Some things that would help:

• Minified JavaScript
• A token (such as a CSRF token) received from the server, which you then hash or transform in some way in the web client code
• Including the current date somewhere in the computation

None of the above would stop a sufficiently motivated attacker.