What is correct method to execute curl from a subprocess?

Question

I was trying to invoke curl from subprocess to download images, but kept getting curl error (error code 2 ..which from doc refers to CURL_FAILED_INIT). I am not using urllib as i will eventually be executing a script using subprocess. Following is the code snippet

import subprocess
import multiprocessing

def worker(fname, k):
    f = open(fname, 'r')
    i = 0
    for imgurl in f:
        try:
            op = subprocess.call(['curl', '-O', imgurl], shell=False)
        except:
            print 'problem downloading image - ', imgurl

def main():
    flist = []
    flist.append(sys.argv[1])
    flist.append(sys.argv[2])
    ...

    for k in range(1):
        p = multiprocessing.Process(target=worker, args=(flist[k],k))
        p.start()

O/P:

curl: try 'curl --help' or 'curl --manual' for more information

2

curl: try 'curl --help' or 'curl --manual' for more information

2

....

Answer 1

If you want to run a shell command, subprocess is the way to go. As this can start a shell command in its own process, use of multiprocessing is at best redundant. Multiprocessing comes in handy when you want to run a function of your python program in distinct process. You appear to intend to run a shell command, not a python function.

I am not familiar with curl. If your intent is to get the standard output from curl, use subprocess.Popen(). subprocess.call() returns the program return code, not stdout.

See http://docs.python.org/release/3.2/library/subprocess.html

Something like:

subp = subprocess.Popen(['curl', '-O', imgurl], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

curlstdout, curlstderr = subp.communicate()

op = str(curlstdout)

might be closer. Not familiar with curl as I said, so your program may vary.