Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python requests hangs when script launched through crontab

I've got a Python script which downloads data in json format through HTTP. If I run the script through command-line using the requests module, the HTTP connection is successful and data is downloaded without any issues. But when I try to launch the script as a crontab job, the HTTP connection throws a timeout after a while. Could anyone please tell me what is going on here? I am currently downloading data via a bash script first and then running the Python script from within that bash. But this is nonsense! Thank you so much!

Using: 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:09:58) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]

P.S.: I haven't found any posts regarding this issue. If there is already an answer for this on some other post, then please accept my apologies.

This is an excerpt from my code. It times out when running requests.get(url):

try:
   response = requests.get(url)
   messages = response.json()["Messages"]
except requests.exceptions.Timeout:
   logging.critical("TIMEOUT received when connecting to HTTP server.")
except requests.exceptions.ConnectionError:
   logging.critical("CONNECTION ERROR received when connecting to HTTP server.")
like image 591
Erik Campo Avatar asked Nov 03 '17 17:11

Erik Campo


People also ask

Can I run Python script from cron?

You'll learn in a bit what this table refers to. You should use Cron any time you want to automate something, like an OS job or a Python script. Needless to say, but an automated Python script can do basically anything. On Linux and macOS, the Crontab consists of six fields.

How do you stop a Python script from crontab?

Print a process tree by ps axjf and look over the "COMMAND" column for the script name respectively the "PID" column to get its PID. kill the corresponding process: kill placeholder_for_pid_of_your_script.

Does Python requests wait for response?

It will wait until the response arrives before the rest of your program will execute. If you want to be able to do other things, you will probably want to look at the asyncio or multiprocessing modules. Chad S. Chad S.

What is Python crontab?

The crontab is a list of commands that you want to run on a regular schedule, and also the name of the command used to manage that list. cron is the system process that will automatically perform tasks for you according to a set schedule.


1 Answers

I just found the answer to my question. I've defined the proxy being used and then used it like this in my code:

HTTP_PROXY="http://your_proxy:proxy_port"
PROXY_DICT={"http":HTTP_PROXY}

response = requests.get(url, proxies=PROXY_DICT)

Reference:

Proxies with Python 'Requests' module

Thank you all for your comprehension. I guess I should have done a thorough search before posting. Sorry.

like image 123
Erik Campo Avatar answered Oct 26 '22 15:10

Erik Campo