Menu
I am trying to save the output of the scrapy crawl command I have tried
scrapy crawl someSpider -o some.json -t json >> some.text
But it doesn't worked ...can some body tell me how i can save output to a text file....I mean the logs and information printed by scrapy...
789
While working with Scrapy, one needs to create scrapy project. In Scrapy, always try to create one spider which helps to fetch data, so to create one, move to spider folder and create one python file over there. Create one spider with name gfgfetch.py python file. Move to the spider folder and create gfgfetch.py .
Finally you hit Ctrl-D (or Ctrl-Z in Windows) to exit the shell and resume the crawling: >>> ^D 2014-01-23 17:50:03-0400 [scrapy.
Using the scrapy tool You can start by running the Scrapy tool with no arguments and it will print some usage help and the available commands: Scrapy X.Y - no active project Usage: scrapy <command> [options] [args] Available commands: crawl Run a spider fetch Fetch a URL using the Scrapy downloader [...]
The scrapy.cfg file is a project root directory, which includes the project name with the project settings. For instance − [settings] default = [name of the project].settings [deploy] #url = http://localhost:6800/ project = [name of the project]
You can add these lines to your settings.py:
LOG_STDOUT = True
LOG_FILE = '/tmp/scrapy_output.txt'
And then start your crawl normally:
scrapy crawl someSpider
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
