Framework Scrapy - Scrapyd server.
I have some problem with getting jobid value inside the spider.
After post data to http://localhost:6800/schedule.json the response is
status = ok
jobid = bc2096406b3011e1a2d0005056c00008
But I need use this jobid inside the current spider during the process. It can be used for open {jobid}.log file or other dynamic reasons.
class SomeSpider(BaseSpider):
name = "some"
start_urls = ["http://www.example.com/"]
def parse(self, response):
items = []
for val in values:
item = SomeItem()
item['jobid'] = self.jobid # ???!
items.append(item)
return items
But I see this jobid only after the task is finihed :( Thanks!
In the spider.py -->
class SomeSpider(BaseSpider):
name = "some"
start_urls = ["http://www.example.com/"]
def __init__(self, *args, **kwargs):
super(SomeSpider, self).__init__(*args, **kwargs)
self.jobid = kwargs.get('_job')
def parse(self, response):
items = []
for val in values:
item = SomeItem()
item['jobid'] = self.jobid # ???!
items.append(item)
return items
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With