Can not get simplest pipeline example to work in scrapy

Question

This is my simple code and i am not getting it work.

I am subclassing from initspider

This is my code

class MytestSpider(InitSpider):
    name = 'mytest'
    allowed_domains = ['example.com']
    login_page = 'http://www.example.com'
    start_urls = ["http://www.example.com/ist.php"]

    def init_request(self):
        #"""This function is called before crawling starts."""
        return Request(url=self.login_page, callback=self.parse)

    def parse(self, response):
        item = MyItem()
        item['username'] = "mytest"
        return item

Pipeline

class TestPipeline(object):
    def process_item(self, item, spider):
            print item['username']

i am geting same error if try to print the item

The error i get is

 File "crawler/pipelines.py", line 35, in process_item
            myitem.username = item['username']
        exceptions.TypeError: 'NoneType' object has no attribute '__getitem__'

I the problem is with InitSpider . My pieplines are not getting item objects

items.py

class MyItem(Item):
    username = Field()

setting.py

BOT_NAME = 'crawler'

SPIDER_MODULES = ['spiders']
NEWSPIDER_MODULE = 'spiders'


DOWNLOADER_MIDDLEWARES = {

    'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware': 700 # <-
}

COOKIES_ENABLED = True
COOKIES_DEBUG = True


ITEM_PIPELINES = [

'pipelines.TestPipeline',


]

IMAGES_STORE = '/var/www/htmlimages'

Answer 1

pipelines.TestPipeline is missing order number. It should be something like ITEM_PIPELINES = {'pipelines.TestPipeline': 900}.

Answer 2

There's another issue with your process_item function. According to the official documentation:

This method is called for every item pipeline component and must either return a dict with data, Item (or any descendant class) object or raise a DropItem exception. Dropped items are no longer processed by further pipeline components.

In your case, you could add a return statement at the end of your function:

def process_item(self, item, spider):
    print item['username']
    return item

If you don't include a return statement, the return value of this pipeline is None. That's why the following pipeline complains--you can't do item['username'] when item is None.