Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Downloading images in scrapy

I am trying to download image in via scrapy. Here are my different files :

items.py

class DmozItem(Item):
        title = Field()
        image_urls = Field()
        images = Field() 

settings.py

BOT_NAME = 'tutorial'

SPIDER_MODULES = ['tutorial.spiders']
NEWSPIDER_MODULE = 'tutorial.spiders'
ITEM_PIPELINES = ['scrapy.contrib.pipeline.images.ImagesPipeline']
IMAGES= '/home/mayank/Desktop/sc/tutorial/tutorial'

spider

class DmozSpider(BaseSpider):
    name = "wikipedia"
    allowed_domains = ["wikipedia.org"]
    start_urls = [
        "http://en.wikipedia.org/wiki/Pune"
    ]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        items = []
        images=hxs.select('//a[@class="image"]')
        for image in images:
                item = DmozItem()
                link=image.select('@href').extract()[0]
                link = 'http://en.wikipedia.com'+link
                item['image_urls']=link
                items.append(item)

In spite of all these setting I my pipeline is not getting activated.Please help. I am new to this framework.

like image 868
Mayank Jain Avatar asked Apr 16 '13 18:04

Mayank Jain


People also ask

Can Scrapy download files?

Scrapy provides reusable item pipelines for downloading files attached to a particular item (for example, when you scrape products and also want to download their images locally).

Can Scrapy scrape dynamic content?

Let's suppose we are reading some content from a source like websites, and we want to save that data on our device. We can copy the data in a notebook or notepad for reuse in future jobs.


1 Answers

First, settings.py: IMAGES -> IMAGES_STORE

Second, spider: You should return an item so that ImagesPipeline could download those images.

item = DmozItem()
image_urls = hxs.select('//img/@src').extract()
item['image_urls'] = ["http:" + x for x in image_urls]
return item
like image 90
imwilsonxu Avatar answered Oct 13 '22 17:10

imwilsonxu