REST and complex search queries

Question

I'm looking for a robust way to model search queries in a REST api.

In my api, you can specify the search criteria in the URI of a resource using query parameters.

For example:

/cars?search=color,blue;AND;doors,4 --> Returns a list of blue cars with 4 doors

/cars?search=color,blue;OR;doors,4 --> Returns a list of cars that are blue or have 4 doors

On the server side, the search string is mapped to the desired underlying technology. Depending on the rest resource, this can be a SQL query, the Hibernate Criteria api, another webservice call, ...

The 2 examples are simple enough to support, but I also need more complex search features like substring search, searching before/after dates, NOT, ...

This is a common problem I think. Is there a library (or a pattern) that I can use that:

• Maps search queries specified as a string to a generic Criteria model. The search format does not have to be the same as I listed above.
• Allows me to map that Criteria model to any technology I need to use.
• Offers mapping support for Hibernate/JPA/SQL, but this is a bonus ;)

Kind regards,

Glenn

Answer 1

Whenever I face these problems, I ask myself "How would I present this to a user, if I was creating a traditional webpage"? The simple answer is that I wouldn't present those sort of options in a single page. The interface would be too complex; however what I could do is provide an interface that allowed users to build up more and more complex queries over a number of pages and that's the solution I think you should go for in this case.

The HATEOAS constraint specifies that we must include the hypermedia controls (links and forms) in our responses. So let's say we have a paginated collections of cars at /cars with a search option, so that when you get /cars it returns something like (BTW I'm using a custom media-type here, but the forms and links should be pretty obvious. Let me know if it isn't):

<cars href="/cars">
    <car href="/cars/alpha">...</car>
    <car href="/cars/beta">...</car>
    <car href="/cars/gamma">...</car>
    <car href="/cars/delta">...</car>
    ...
    <next href="/cars?page=2"/>
    <search-color href="/cars" method="GET">
        <color type="string" cardinality="required"/>
        <color-match type="enumeration" cardinality="optional" default="substring">
            <option name="exact"/>
            <option name="substring"/>
            <option name="regexp"/>
        </color-match>
        <color-logic type="enumeration" cardinality="optional" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </color-logic>
    </search>
    <search-doors href="/cars" method="GET">
        <doors type="integer" cardinality="required"/>
        <door-logic type="enumeration" cardinality="required" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </door-logic>
    </search>
</cars>

So just say we search for white cars, we would GET /cars?color=white and we might get back something like:

<cars href="/cars?color=white">
    <car href="/cars/beta">...</car>
    <car href="/cars/delta">...</car>
    ...
    <next href="/cars?color=white&page=2"/>
    <search-color href="/cars?color=white" method="GET">
        <color2 type="string" cardinality="required"/>
        <color2-match type="enumeration" cardinality="optional" default="substring">
            <option name="exact"/>
            <option name="substring"/>
            <option name="regexp"/>
        </color2-match>
        <color2-logic type="enumeration" cardinality="optional" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </color2-logic>
    </search>
    <search-doors href="/cars?color=white" method="GET">
        <doors type="integer" cardinality="required"/>
        <door-logic type="enumeration" cardinality="required" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </door-logic>
    </search>
</cars>

This result then let's us refine our query. So just say we wanted white cars but not "off-white" cars, we could then GET '/cars?color=white&color2=off-white&color2-logic=not', which might return

<cars href="/cars?color=white&color2=off-white&color2-logic=not">
    <car href="/cars/beta">...</car>
    <car href="/cars/delta">...</car>
    ...
    <next href="/cars?color=white&color2=off-white&color2-logic=not&page=2"/>
    <search-color href="/cars?color=white&color2=off-white&color2-logic=not" method="GET">
        <color3 type="string" cardinality="required"/>
        <color3-match type="enumeration" cardinality="optional" default="substring">
            <option name="exact"/>
            <option name="substring"/>
            <option name="regexp"/>
        </color3-match>
        <color3-logic type="enumeration" cardinality="optional" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </color3-logic>
    </search>
    <search-doors href="/cars?color=white&color2=off-white&color2-logic=not" method="GET">
        <doors type="integer" cardinality="required"/>
        <door-logic type="enumeration" cardinality="required" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </door-logic>
    </search>
</cars>

We could then further refine our query, but the point is that at each step along the way, the hypermedia controls tells us what is possible.

Now, if we think about the search options for cars, the colors, doors, makes and models aren't unbounded, so we could make the options more explicit by providing enumerations. For instance

<cars href="/cars">
    ...
    <search-doors href="/cars" method="GET">
        <doors type="enumeration" cardinality="required">
            <option name="2"/>
            <option name="3"/>
            <option name="4"/>
            <option name="5"/>
        </doors>
        <door-logic type="enumeration" cardinality="required" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </door-logic>
    </search>
</cars>

However the only white cars that we have might be 2 and 4 door, in which case GETing /cars?color=white might give us

<cars href="/cars?color=white">
    ...
    <search-doors href="/cars?color=white" method="GET">
        <doors type="enumeration" cardinality="required">
            <option name="2"/>
            <option name="4"/>
        </doors>
        <door-logic type="enumeration" cardinality="required" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </door-logic>
    </search>
</cars>

Similarly, as we refine the colors, we might find their are only a few options, in which case we can switch from providing a string search, to providing an enumeration search. e.g., GETing /cars?color=white might give us

<cars href="/cars?color=white">
    ...
    <search-color href="/cars?color=white" method="GET">
        <color2 type="enumeration" cardinality="required">
            <option name="white"/>
            <option name="off-white"/>
            <option name="blue with white racing stripes"/>
        </color2>
        <color2-logic type="enumeration" cardinality="optional" default="and">
            <option name="and"/>
            <option name="or"/>
            <option name="not"/>
        </color2-logic>
    </search>
    ...
</cars>

You could do the same for other search categories. For instance, initially you wouldn't want to enumerate all the makes, so you would provide some sort of text search. Once the collection has been refined, and there are only a few models to pick from, then it makes sense to provide an enumeration. The same logic applies to other collections. For instance you wouldn't want to enumerate all of the cities in the world, but once you have refined the area to 10 or so cities, then enumerating them can be very helpful.

Is there a library that will do this for you? None that I know of. Most I've seen don't even support hypermedia controls (i.e., you've got to add the links and forms yourself). Is there a pattern you can use? Yes, I believe the above is a valid pattern for solving this sort of problem.

Answer 2

hmmm... this is a tricky area. There is no framework out there that is tailored for your problem. Even ODATA that @p0wl mentions has certain limitations about the way fields are mapped to domain models. It gets weird after a point.

The simplest way to get around this is to accept the query as a String which you then validate against a domain specific language using a AST or whatever it is that you want. This allows you to be flexible while accepting the query while still validating it for correctness. What you lose is the ability to describe the query in a more meaningful manner through the URL.

Take a look at chapter 8 of the Restful recipes cookbook. It highlights one of the solutions I proposed and also recommends other approaches. Choose one based on the flexibility needed by your client versus the expressiveness of your query. There is a balance you can strike somewhere.