Menu
I want to download pages in Chinese, So i opened the command line and the python, and typed these code
from scrapy.conf import settings
settings.overrides['DEFAULT_REQUEST_HEADERS'] = {'Accept':'text/heml,application/xhtml+xml;q=0.9,*/*;q=0.8','Accept-Language':'ch',}
but it didn't work. The 'Accept-Language' remained 'en'. So i tried other settings, same thing happened. I just can't change the settings. Someone plz help me out, it drives me crazy! i spent 3 hours on it but got nothing. Thanks a lot!
456
Designating the settings When you use Scrapy, you have to tell it which settings you're using. You can do this by using an environment variable, SCRAPY_SETTINGS_MODULE . The value of SCRAPY_SETTINGS_MODULE should be in Python path syntax, e.g. myproject.
Open the configuration file of your Scrapy project using your preferred text editor. Search for the USER_AGENT option. Uncomment the line and set the value to the user-agent of your choice to permanently set the user agent for your Scrapy spider.
In the latest version of Scrapy, available on GitHub, you can raise a CloseSpider exception to manually close a spider. It succeeds to force stop, but not fast enough. It still lets some Request running.
In the context of Scrapy, this means to send out “concurrent” requests instead of sending them one by one. In other words, this means that the Scrapy spider will send a X number of (simultaneous) requests to the web server at the same time.
is
settings.overrides['DEFAULT_REQUEST_HEADERS'] = {'Accept':'text/heml,application/xhtml+xml;q=0.9,*/*;q=0.8','Accept-Language':'ch',}
supposed to say text/heml or html?
50
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
