What is the best way to design a HTTP request when somewhat complex parameters are needed?

Question

I have some web services that I am writing and I am trying to be as RESTful as possible. I am hosting these web services using a HTTPHandler running inside of IIS/ASP.NET/SharePoint.

Most of my services expect a HTTP GET. I have two of these that are simply returning some data (i.e., a query) and will be Idempotent, but the parameters may be somewhat complex. Both of them could include characters in the parameters of the service that are not allowed for at least the PATH portion of the URL.

Using IIS, ASP.NET, and SharePoint I have found that the following characters in the URL path don't even make it to my HttpHandler even if Url encoded (the request blows up and I don't have any easy control over this):

• % (%25)
• & (%26)
• * (%2a, but didn't Url encode)
• + (%2b)
• : (%3a)
• < (%3c)

• (%3e)

The following characters made it to my HttpHandler, but the UriTemplate could not handle them properly even if Url encoded:

• (%23)

• . (%2e, but didn't Url encode; UriTemplate removed the "." if is is the last character before a /)
• ? (%3f)
• / (%2f - UriTemplate fails for obvious reasons even if UrlEncoded)
• \ (%5c)

So, I've been somewhat thorough, but I need to test these url encoded characters in the query string. It appears that this will work for the most part there.

In one of my services, the special characters that are a parameter are semantically part of a query/filter (actually search terms for a search service), but in another they are not really part of a query/filter so ideally they are part of the path and not the query string.

My question is, what option is best? Here are some I know of:

1. Use HTTP GET and query string. Anything that may use special characters should be on the query string and Url Encoded. This is where I am leaning, but I am concerned about extremely long query strings (IE has a 2083 limit)

2. Use HTTP GET and base64 encoding within path. Use a Modified Base64 for URL for any parameters that might use special characters and keep them as part of the path if preferred. I have tried this and it works, but it is kind of ugly. Still a concern about extremely long query strings.

3. Use HTTP POST and message body. Anything that may use special characters should be in the body of the request. Seems like a decent solution, but posts are understood to not be Idempotent and (I thought) are generally meant for changes (whereas no change is occurring here).

4. Use HTTP GET and message body. Anything that may use special characters should be in the body of the request. This seems like a bad idea according to SO: HTTP GET with request body and Roy Fielding.

5. Use a combination of #3 and either #1 or #2 above depending on how large the request can be.

6. Other???

Note that in some cases I may be able to change things around to prevent special characters (and I may do that), but I won't be able to do this in all cases.

---

Regarding URI length, RFC2616 Sec3.2.1 says the following:

The HTTP protocol does not place any a priori limit on the length of a URI. Servers MUST be able to handle the URI of any resource they serve, and SHOULD be able to handle URIs of unbounded length if they provide GET-based forms that could generate such URIs. A server SHOULD return 414 (Request-URI Too Long) status if a URI is longer than the server can handle (see section 10.4.15).

  Note: Servers ought to be cautious about depending on URI lengths   above 255 bytes, because some older client or proxy   implementations might not properly support these lengths. 

In addition the Maximum URL length is 2,083 characters in Internet Explorer.

Related: How to pass complex queries in REST?

Answer 1

There's no perfect way to do this.

The correct HTTP/REST way would be to use a GET and have all your parameters in the URL as query arguments. You've identified two practical problems with this approach

1. Your server software is not correctly passing some characters to you, even if URL encoded. That surprises me, actually, and you should look more closely at what's going on that you can't even get a % through the URL. Does your framework give you raw access to PATH_INFO or otherwise unprocessed characters? That may give you a workaround.
2. Your query strings may be too long. You mention the 2083 byte limit in MSIE. That may or may not be a practical problem for you, depending on whether MSIE is a client of your API. (Ie: via Javascript making calls to a JSON API). But in my experience very long URLs will end up breaking mysteriously in several places; proxy caches along the path, even a stateful firewall. If you have absolute control over the clients and network path you can probably live with the dangers of long URLs. If it's a public API, forget it.

Hopefully you can make the straightforward GET work in your environment. You may even want to consider refactoring your API to make the query data smaller.

But what if you can't make the GET work? You propose several alternatives. I would immediately dismiss two of them. Don't put content in the GET request body; too much software will break if you try that, and anyway it violates the very REST spirit you're trying to capture. And I wouldn't use base64 encoding. It may help you work around problem 1, your server not handling some characters in URLs right. But if applied wrong it will actually make your URLs longer, not shorter, compounding problem 2. Even if you do base64 right and include some compression it won't make URLs significantly shorter, and will make the client much more complicated.

Your most practical solution is probably option 3, an HTTP POST. This isn't RESTful; you should be using GETs for read-only queries. And you'll lose some advantages of the REST approach with caching of GETs and the like. On the other hand it will work correctly, and simply, with a large variety of Internet infrastructure and software libraries. You can then pass as much data you want in the POST body either via multipart/form-data encoding, JSON, or XML. (I've built two major public web services using SOAP, which is just XML on POSTs. It's ugly and not RESTful, but it does work reliably.)

REST is a great design paradigm. It's a guideline. If it doesn't fit your app, don't feel you need to stick with it. HTTP is not good at passing large amounts of data to the server with a GET. If you need have giant query parameters, do something else.