Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Scrapyd jobid value inside spider

Framework Scrapy - Scrapyd server.

I have some problem with getting jobid value inside the spider.

After post data to http://localhost:6800/schedule.json the response is

status = ok
jobid = bc2096406b3011e1a2d0005056c00008

But I need use this jobid inside the current spider during the process. It can be used for open {jobid}.log file or other dynamic reasons.

class SomeSpider(BaseSpider):
    name = "some"
    start_urls = ["http://www.example.com/"]
    def parse(self, response):
        items = []
        for val in values:
            item = SomeItem()
            item['jobid'] = self.jobid # ???!
            items.append(item)
        return items

But I see this jobid only after the task is finihed :( Thanks!

like image 553
fcmax Avatar asked Mar 11 '12 04:03

fcmax


1 Answers

In the spider.py -->

class SomeSpider(BaseSpider):
    name = "some"
    start_urls = ["http://www.example.com/"]

    def __init__(self, *args, **kwargs):
        super(SomeSpider, self).__init__(*args, **kwargs)
        self.jobid = kwargs.get('_job')

    def parse(self, response):
        items = []
        for val in values:
           item = SomeItem()
           item['jobid'] = self.jobid # ???!
           items.append(item)
        return items
like image 200
Sadia Avatar answered Oct 23 '22 17:10

Sadia