curl works fine, except if I call it with subprocess

Question

I have a curl that looks a bit like this:

curl -1 -X POST --user "xxx:yyy" -d "status=new&content=issue+details+at%3A+http%3A%2F%2Flocalhost%3A6543%2Ftest%2Fsubmit%2F16-07-03-H-20-18-&kind=bug&title=QA+Fail&responsible=xxx&priority=critical"  "https://api.bitbucket.org/1.0/repositories/my/repo/issues"

If I open up a terminal and just execute it it works fine (an issue is created in bitbucket)

if I try to execute the same curl via subprocess it just fails:

sCmd = "curl....etc"
lCmd = [s for s in sCmd.split() if s]
subprocess.call(lCmd)

I get the error message:

curl: (1) Protocol "https not supported or disabled in libcurl

I don't get why the exact same command works so differently in Python. Any ideas?

This is without use of a virtualenv, by the way. And I know for a fact that the contents of lCmd are valid

PS: yes, I know I should be using requests. Unfortunately requests was giving me similar problems.

Answer 1

The only point of having a shell, in this scenario, is to split your string into a set of arguments.

Thing is, you don't need anything to do that for you -- you can do it more accurately and consistently yourself!

subprocess.call([
  'curl',
  '-1',
  '-X', 'POST', 
  '--user', 'xxx:yyy',
  '-d', 'status=new&content=issue+details+at%3A+http%3A%2F%2Flocalhost%3A6543%2Ftest%2Fsubmit%2F16-07-03-H-20-18-&kind=bug&title=QA+Fail&responsible=xxx&priority=critical',
  'https://api.bitbucket.org/1.0/repositories/my/repo/issues'
])

This also avoids confusion between literal and syntactic quotes: When specifying a literal argv, the literal Python string you pass is the literal string the child process receives -- there's no need for quoted quotes to be passed to the shell.

---

This is much more reliable than starting with a string and splitting it, even with Python builtins: Consider a case where your username contains whitespace. A list containing

'--user', 'user name:password'

...is completely unambiguous, whereas

--user user name:password

...will parse incorrectly.

Even worse would be a username containing $(rm -rf $HOME) -- if you use os.system() or subprocess.Popen(..., shell=True), that username could be executed as a command; with a literal string, you're entirely safe.

---

That said -- the specific error, re: libcurl not being compiled with SSL support, isn't going to be calling-convention-specific. I'd suggest inspecting your environment -- if you have different versions of curl first in the PATH, or different versions of libcurl in LD_LIBRARY_PATH between your environments, that would explain the error in question.

Answer 2

I can't answer your question about using the curl command in subprocess, but it might work if you just call os.system Like this:

import os
os.system ("curl -1 -X POST --user \"xxx:yyy\" -d \"status=new&content=issue+details+at%3A+http%3A%2F%2Flocalhost%3A6543%2Ftest%2Fsubmit%2F16-07-03-H-20-18-&kind=bug&title=QA+Fail&responsible=xxx&priority=critical\"  \"https://api.bitbucket.org/1.0/repositories/my/repo/issues\"")

This most likely works knowing that curl runs via the terminal directly.

This might help with your question about curl not running in subprocess (probably has something to do with the environment of subprocess being different then the environment that curl was configured): https://curl.haxx.se/docs/faq.html#curl_1_SSL_is_disabled_https