How to solve the 10054 error

Question

I'm using an app provided by some website to collect some data from the website periodly, say, 30s a time. The returned response is then recorded in database.

I use the requests modular by import requests and write codes to catch Exception. The codes for the main function are as following:

def get_response(self):
    try:
       response = requests.get(self.request_url)
       if response.status_code == 200:
          return response.json()
       except Exception as e:
          msg = "Exception is:\n %s \n" % e
          print msg

The above function works quite well for the first several hours. The function can also recover from some exceptions like: ('Connection aborted.', BadStatusLine("''",))
or ('Connection aborted.', error(10053, '')) It omits the exception (by recording a Null in database) and continues to get the response of next period.

However, the function stops working when encoutering a 10054 error.

Exception is:
 ('Connection aborted.', error(10054, '')) 

I check the database to find that all the response is Null after the time that the 10054 error comes. I firstly guess that the website may breakdown, thus no response is received. But when I manually restart the function, it starts to get response again. So that's no realated with breakdown of the website.

I search in stackoverflow and find: Errno 10054 An existing connection was forcibly closed by the remote host. But I don't know how to resolve it.

Could you please provide some solotion to this problem?(ideally speaking) or provide some solution to restart the function without manually restarting? (It looks like once I restart the funtion and it works again.)

Thanks in advance.

Answer 1

The web server actively rejected your connection. That's usually because it is congested, has rate limiting or thinks that you are launching a denial of service attack. If you get this from a server, you should sleep a bit before trying again. In fact, if you don't sleep before retry, you are a denial of service attack. The polite thing to do is implement a progressive sleep of, say, (1,2,4,8,16,32) seconds.

Answer 2

You can try it. It solved my problem with ConnectionResetError 10054.

session = requests.Session()
session.headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.1.2222.33 Safari/537.36",
    "Accept-Encoding": "*",
    "Connection": "keep-alive"
}
response = session.get(self.request_url)