Using Scrapy to parse sitemaps

Question

I want to be able to use scrapy to crawl links on a sitemap. I don't know much about this application, so I would be interested in any links/info/documentation you could provide.

Thanks

Answer 1

A new generic spider has just been added to Scrapy trunk, for this purpose. It will be available on next release (Scrapy 0.14)

• Code here: http://snippets.scrapy.org/snippets/20/
• Documentation here: http://readthedocs.org/docs/scrapy/en/latest/topics/spiders.html#sitemapspider