I would like to try send requests.get to this website:
requests.get('https://rent.591.com.tw')
and I always get
<Response [404]>
I knew this is a common problem and tried different way but still failed. but all of other website is ok.
any suggestion?
404 error means the service is not returning any response. If you can observe between this tag it will return page not found text which means it returns a web page.
If you have ever wanted to check whether a page has returned a 404 for any reason, one of the easiest ways is to use this little helper function UrlExists() with the current url, given by window. location. href.
Webservers are black boxes. They are permitted to return any valid HTTP response, based on your request, the time of day, the phase of the moon, or any other criteria they pick. If another HTTP client gets a different response, consistently, try to figure out what the differences are in the request that Python sends and the request the other client sends.
That means you need to:
I usually point my requests to a http://httpbin.org endpoint, have it record the request, and then experiment.
For requests
, there are several headers that are set automatically, and many of these you would not normally expect to have to change:
Host
; this must be set to the hostname you are contacting, so that it can properly multi-host different sites. requests
sets this one.Content-Length
and Content-Type
, for POST requests, are usually set from the arguments you pass to requests
. If these don't match, alter the arguments you pass in to requests
(but watch out with multipart/*
requests, which use a generated boundary recorded in the Content-Type
header; leave generating that to requests
).Connection
: leave this to the client to manageCookies
: these are often set on an initial GET request, or after first logging into the site. Make sure you capture cookies with a requests.Session()
object and that you are logged in (supplied credentials the same way the browser did).Everything else is fair game but if requests
has set a default value, then more often than not those defaults are not the issue. That said, I usually start with the User-Agent header and work my way up from there.
In this case, the site is filtering on the user agent, it looks like they are blacklisting Python
, setting it to almost any other value already works:
>>> requests.get('https://rent.591.com.tw', headers={'User-Agent': 'Custom'})
<Response [200]>
Next, you need to take into account that requests
is not a browser. requests
is only a HTTP client, a browser does much, much more. A browser parses HTML for additional resources such as images, fonts, styling and scripts, loads those additional resources too, and executes scripts. Scripts can then alter what the browser displays and load additional resources. If your requests
results don't match what you see in the browser, but the initial request the browser makes matches, then you'll need to figure out what other resources the browser has loaded and make additional requests with requests
as needed. If all else fails, use a project like requests-html
, which lets you run a URL through an actual, headless Chromium browser.
The site you are trying to contact makes an additional AJAX request to https://rent.591.com.tw/home/search/rsList?is_new_list=1&type=1&kind=0&searchtype=1®ion=1
, take that into account if you are trying to scrape data from this site.
Next, well-built sites will use security best-practices such as CSRF tokens, which require you to make requests in the right order (e.g. a GET request to retrieve a form before a POST to the handler) and handle cookies or otherwise extract the extra information a server expects to be passed from one request to another.
Last but not least, if a site is blocking scripts from making requests, they probably are either trying to enforce terms of service that prohibit scraping, or because they have an API they rather have you use. Check for either, and take into consideration that you might be blocked more effectively if you continue to scrape the site anyway.
In my case this was due to fact that the website address was recently changed, and I was provided the old website address. At least this changed the status code from 404 to 500, which, I think, is progress :)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With