Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Rightmove API and scraping technical and legal

Tags:

I'm looking to build an app using property data. Nestoria has a free API and rules of use and Zoopla an API you register for. OnTheMarket and Rightmove have same terms of use to the letter (bizarre for competitors?). Rightmove advertise an API for upload but not download - I can't find anything for OnTheMarket.

I've discovered that Rightmove does have an API although the post code search is obfuscated by their own outcode mappings...

https://api.rightmove.co.uk/api/sale/find?index=0&sortType=1&numberOfPropertiesRequested=2&locationIdentifier=OUTCODE%5E1&apiApplication=IPAD

I'm wary of using an API that's not promoted. The alternative is scraping, which is harder technically and legally questionable, although from what I read the data is in the public domain and so free to use.

I've contacted Rightmove but got no response.

Is anyone using the Rightmove api and had this authorised by them? Seems most strange that it's open and available but barely mentioned when searching for it.

Can anyone clarify what rules/law/ethics are in place for scraping data?

like image 333
Failing Coder Avatar asked Apr 16 '16 09:04

Failing Coder


People also ask

Is it legal to scrape data from Rightmove?

Yesterday afternoon, a Rightmove spokesperson said: “It is against Rightmove's terms and conditions to conduct scraping of the content on our site. “If scraping is detected or we find or are alerted to job adverts requesting people to scrape Rightmove we take appropriate action.”

Is scraping data Legal UK?

Is data scraping illegal? As things currently stand, many acts of data scraping are potentially illegal under UK law. The exact nature of the illegal activity depends on a variety factors. Unfortunately, therefore, every situation needs to be analysed on its own facts.

Is there a Rightmove API?

I've discovered that Rightmove does have an API although the post code search is obfuscated by their own outcode mappings...

Do you need permission to scrape a website?

Web scraping is legal if you scrape data publicly available on the internet. But some kinds of data are protected by international regulations, so be careful scraping personal data, intellectual property, or confidential data. Respect your target websites and use empathy to create ethical scrapers.


2 Answers

Don't query their hidden API. But you can run a web crawler on RightMove.co.uk website, and it is perfectly legal as outlined in their Terms of Service under section 3.3 :

You must not use or attempt to use any automated program unless the automated program identifies itself uniquely in the User Agent field and is fully compliant with the Robots Exclusion Protocol

A web crawler like Apache Nutch perfectly follows the Robots Exclusion Protocol. From their robots.txt file I found they have elaborate nested sitemap.xml files, and hence they rather promote organized but polite crawling of their website. I was myself wanting to get their data, so I am beginning with my endeavour to crawl them with my resources - do let me know if you need access to this data.

like image 193
Tushar Goswami Avatar answered Sep 19 '22 00:09

Tushar Goswami


You are not allowed to scrape their data, here what their terms&conditions say about it:

"You must not use or attempt to use any automated program (including, without limitation, any spider or other web crawler) to access our system or this Site. You must not use any scraping technology on the Site. Any such use or attempted use of an automated program shall be a misuse of our system and this Site. Obtaining access to any part of our system or this Site by means of any such automated programs is strictly unauthorised."

like image 34
user12627702 Avatar answered Sep 22 '22 00:09

user12627702