Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Determine the scrapy root directory

I'd like to be able to read a file relative to the scrapy root directory, as described in the documentation.

Is there a way of determining this path from inside the spider?

like image 776
Dragon Dave Avatar asked Jan 15 '13 15:01

Dragon Dave


2 Answers

I don't know if scrapy is aware of where it is located on the filesystem, but in my settings.py i add

PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))

then in spider you can say

from yourscrapyprojecthere import settings

settings.PROJECT_ROOT

Here I have assumed my project root is wherethe settings.py file is located. If you want to get the directory where scrapy.cfg is you have to go up a directory

like image 51
dm03514 Avatar answered Oct 06 '22 01:10

dm03514


Very old question but for those often looking for the answer, scrapy has a built-in way of finding the project root:

from scrapy.utils.conf import closest_scrapy_cfg

proj_root = closest_scrapy_cfg()

will return the absolute path of the folder containing the scrapy.cfg file

EDIT: will return the absolute path of the scrapy.cfg file for the project

like image 32
Lynx-Lab Avatar answered Oct 06 '22 00:10

Lynx-Lab