Using curl in Popen in Python

Question

I run this curl command in the unix shell and it works (see below). I was able to redirect the returned data to a file but now I want to process the data in my code instead of wasting a bunch of space in a file.

curl -k -o outputfile.txt 'obfuscatedandVeryLongAddress'
#curl command above, python representation below
addr = "obfuscatedandVeryLongAddress"
theFile = subprocess.Popen(["curl", "-k", addr], stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True)

theFile.stdout is empty after this. The data returned in the curl command should be something like 4,000 lines (verified when running the command in the shell). Is the size breaking theFile.stdout? Am I doing something else wrong? I tried using:

out, err = theFile.communicate()

and then printing the out variable but still nothing

edit: formatting and clarification

Answer 1

You need to remove shell=True.

theFile = subprocess.Popen(["curl", "-k", addr], stdout = subprocess.PIPE, stderr = subprocess.PIPE)

Should work.

If you do shell=True, you should pass a string. Otherwise, what you're actually doing is passing those arguments -k, and addr as arguments to the shell. So if your shell is sh, what you're doing is sh 'curl' -k addr.