Alternatives to YQL

Question

This is a multi-part question. I just watched a very interesting presentation on YQL by the lead developer (a graduate of my MS program). While it was very compelling, and I am looking forward to trying it out, I am wondering if anyone knows of alternative frameworks for querying multiple web service APIs to make them appear seamless, the apparent purpose of YQL?

Yahoo's strategy has been to create XML schema definitions that bind a given web service's parameters into their YQL Open Table query parameters, which I think is very clever. Is there any tool that attempts (perhaps I am naive here) to automate the discovery of parameters in say a REST API? I am aware that with SOAP APIs, because there is a published WSDL, it makes automation easier, but is there yet no way to do this with REST? Is anyone trying?

Answer 1

Yes people are trying to produce description languages for REST. The most popular effort is WADL. There are lots of questions about WADL here on SO. Is it a good idea? In my opinion no.

REST does not need a discovery model beyond what it already has with hypermedia, because is trying to solve a problem at a different architectural layer than web services. Web services deliver data to an application's business logic/domain model. REST is about delivering content and behaviour to a presentation layer.

How about an analogy? Think of the different between an object and struct in C++. A struct is just simple data that some client process is going to manipulate. That's what a web service does, it returns a chunk of data, a struct. Sure maybe it did a bunch of server side processing to produce the result, but the end result is a lump of data. A REST interface delivers an object. i.e. It contains both data and the methods that can be used to manipulate that object. By definition, if you understand the uniform interface and you understand the returned media type, you already know what you can do with the response. Discovery mechanisms are redundant.

If you find this hard to believe, the think about the web. How does a web browser discover web pages? The web has no formalized discovery mechanism, and yet there is a world of information out there that we can discover with a web browser.

Answer 2

There is this little website http://zachgrav.es/yql/tablesaw/ which indeed auto-discovers parameters in a REST api and turns it into a YQL compatible table.

Answer 3

There are two ways to find information. Either you use a 100% unambiguous language or you use a natural language. Anything in between like YQL is doomed to fail because it delivers neither and works well only with the examples its authors tout.

I blogged about this at http://zscraper.wordpress.com/2012/05/30/enough-with-crawling-2. My personal stance is that you'll always get the most accurate results if you do your homework first, i.e. study the target domain and figure out how to query it unambiguously.

To answer your question and give you an alternative -- try Bobik. This is a cloud-backed scraping service that you control via REST API. Compose your "queries" in traditional syntax (Bobik supports Javascript, JQuery, XPATH and CSS) and call Bobik to run them from any client-side environment (webpages, mobile apps, or your server).

Hope this helps.