I have created a spider using Scrapy but I cannot figure out how to turn off the default logging. From the documentation it appears that I should be able to turn it off by doing
logging.basicConfig(level=logging.ERROR)
But this has no effect. From looking at the code for logging.basicConfig() I'm guessing this is because "the root logger has handlers configured" but perhaps I'm wrong about that. At any rate, can anyone explain what I need to do to get Scrapy to not output the usual
2015-10-18 17:42:00 [scrapy] INFO: Scrapy 1.0.3 started (bot: EF)
2015-10-18 17:42:00 [scrapy] INFO: Scrapy 1.0.3 started (bot: EF)
2015-10-18 17:42:00 [scrapy] INFO: Optional features available: ssl, http11, boto
2015-10-18 17:42:00 [scrapy] INFO: Optional features available: ssl, http11, boto
etc.?
EDIT: As suggested by sirfz below, the line
logging.getLogger('scrapy').setLevel(logging.WARNING)
can be used to set the logging level. However, it appears that you must do this in the init method (or later) in your spider.
While working with Scrapy, one needs to create scrapy project. In Scrapy, always try to create one spider which helps to fetch data, so to create one, move to spider folder and create one python file over there. Create one spider with name gfgfetch.py python file. Move to the spider folder and create gfgfetch.py .
You can start by running the Scrapy tool with no arguments and it will print some usage help and the available commands: Scrapy X.Y - no active project Usage: scrapy <command> [options] [args] Available commands: crawl Run a spider fetch Fetch a URL using the Scrapy downloader [...]
You can simply change the logging level for scrapy (or any other logger):
logging.getLogger('scrapy').setLevel(logging.WARNING)
This disables all log messages less than the WARNING
level.
To disable all scrapy log messages you can just set propagate
to False
:
logging.getLogger('scrapy').propagate = False
This prevents scrapy's log messages from propagating to the root logger (which prints to console when configured using basicConfig()
)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With