Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

An array field in scrapy.Item

I want to add a field to scrapy.Item so that it's an array:

class MyItem(scrapy.Item):
    field1 = scrapy.Field()
    field2 = scrapy.Field()
    field3_array = ???

How can I do that?

like image 348
Mario Honse Avatar asked Mar 24 '15 07:03

Mario Honse


People also ask

What is field in Scrapy?

Field([arg]) class scrapy. Field([arg]) The Field class is just an alias to the built-in dict class and doesn't provide any extra functionality or attributes. In other words, Field objects are plain-old Python dicts. A separate class is used to support the item declaration syntax based on class attributes.

What is pipeline in Scrapy?

Each item pipeline component (sometimes referred as just “Item Pipeline”) is a Python class that implements a simple method. They receive an item and perform an action over it, also deciding if the item should continue through the pipeline or be dropped and no longer processed.


1 Answers

You just create a filed

field3_array = scrapy.Field()

But while parsing the scraped items do like this

items['field3_array'] = []

items['field3_array'][0] ='one'
items['field3_array'][1] ='two'

in this way you can achieve this.

Have a look

like image 189
backtrack Avatar answered Oct 12 '22 00:10

backtrack