Scrapy + Eclipse PyDev : how to setup the debugger?

Question

I've successfully setup Eclipse with my Scrapy project.

I did it by setting a new Run/Debug configuration :

• Whose main module links to Scrapy /usr/local/bin/scrapy for me (I've found suggestion to use cmdline.py but that failed on my computer (OSX Lion & scrapy installed through easy_install)
• Defining the arguments to send "crawl ny" in my case as I would if I used the Scrapy command line
• Setting the correct working directory (${workspace_loc:My Project/src} in my case)

Eclipse can successfully launch my project, but I've no debbuger. I'm missing my breakpoints and variable inspection, does anyone know how to setup the debbugger with this environment ?

Answer 1

• Keep the whole scrapy project folder under PyDev.
• You Need to set the main module to scrapy/cmdline.py
• Set arg to crawl ny in your case

Answer 2

None of the suggestions above worked for me. Things would run, but no break points would fire.

I added a main.py file to my local project and hooked into the scrapy command line like so:

import scrapy.cmdline

def main():
    scrapy.cmdline.execute(argv=['scrapy', 'crawl', 'wiki'])

if  __name__ =='__main__':
    main()

This could easily be further refined to pass in the spider name to have different debug configs per spider.