Going from Ruby to Python : Crawlers [closed]

Question

I've started to learn python the past couple of days. I want to know the equivalent way of writing crawlers in python.

so In ruby I use:

1. nokogiri for crawling html and getting content through css tags
2. Net::HTTP and Net::HTTP::Get.new(uri.request_uri).body for getting JSON data from a url

what are equivalents of these in python?

Answer 1

1. Between lxml and beautiful soup, lxml is more equivalent to nokogiri because it is based on libxml2 and it has xpath/css support.
2. The equivalent of net/http is urllib2

Answer 2

Well

Mainly you have to separate the 'scraper'/crawler the python lib/program/function that will download the files/data from the webserver and the Parser that will read this data and interpret the data. In my case I had to scrap and get some govt info that is 'open' but not download/data friendly. For this project I used scrapy[1].

Mainly I set the 'starter_urls' that are the urls my robot will crawl/get and after I use a function 'parser' to retrieve/parse this data.

For parsing/retrieving you are going to need some html,lxml extractor as the 90% of your data will be that.

Now focusing in your question:

For data crawling

1. Scrapy
2. Requests [2]
3. Urllib [3]

For parsing data

1. Scrapy/lxml or scrapy+other
2. lxml[4]
3. beatiful soup [5]

And please remember 'crawling' and scrapping is not only for web, emails too. you can check another question about that here [6]

[1] = http://scrapy.org/

[2] - http://docs.python-requests.org/en/latest/

[3] - http://docs.python.org/library/urllib.html

[4] - http://lxml.de/

[5] - http://www.crummy.com/software/BeautifulSoup/

[6] - Python read my outlook email mailbox and parse messages