Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What's the IP address range of Facebook's Open Graph crawler?

In order to test the Open Graph API on our preview environment, we need to poke a hole in our firewall to allow Facebook to scrape our object pages. What IP ranges should we allow?

like image 280
the1plummie Avatar asked Jan 14 '12 00:01

the1plummie


1 Answers

EDIT

Facebook has been showing some love and is now making the IP block public for anyone to have

http://developers.facebook.com/docs/ApplicationSecurity/#facebook_scraper https://developers.facebook.com/docs/sharing/best-practices#crawl

Facebook Scraper

A number of Platform services such as Social Plugins and the Open Graph require our systems to be able to reach your Web Pages. We recognize that there are situations where you might not want these pages on the public Internet, during testing or for other security reasons.

To facilitate this, you should make exceptions in your security systems to allow Facebook to scrape these pages by adding the following IP ranges, accurate as of April 2012.

31.13.24.0/21
31.13.64.0/18
66.220.144.0/20
69.63.176.0/20
69.171.224.0/19
74.119.76.0/22
103.4.96.0/22
173.252.64.0/18
204.15.20.0/22

Instead of IP, you can also use the user agent for your firewall.

http://developers.facebook.com/docs/reference/plugins/like/

When does Facebook scrape my page?

Facebook needs to scrape your page to know how to display it around the site.

Facebook scrapes your page every 24 hours to ensure the properties are up to date. The page is also scraped when an admin for the Open Graph page clicks the Like button and when the URL is entered into the Facebook URL Linter. Facebook observes cache headers on your URLs - it will look at "Expires" and "Cache-Control" in order of preference. However, even if you specify a longer time, Facebook will scrape your page every 24 hours.

The user agent of the scraper is: "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"

like image 110
DMCS Avatar answered Sep 21 '22 22:09

DMCS