Print executed command for Python subprocess.Popen

Question

I have a script that is automating author re-writes on a number of git repositories.

def filter_history(old, new, name, repoPath):

command = """--env-filter '
        an="$GIT_AUTHOR_NAME"
        am="$GIT_AUTHOR_EMAIL"
        cn="$GIT_COMMITTER_NAME"
        cm="$GIT_COMMITTER_EMAIL"

        if [[ "$GIT_COMMITTER_NAME" = "|old|" ]]
        then
            cn="|name|"
            cm="|new|"
        fi

        if [[ "$GIT_AUTHOR_NAME" = "|old|" ]]
        then
            an="|name|"
            am="|new|"
        fi

        export GIT_AUTHOR_NAME="$an"
        export GIT_AUTHOR_EMAIL="$am"
        export GIT_COMMITTER_NAME="$cn"
        export GIT_COMMITTER_EMAIL="$cm"
'
"""

#DO string replace
command = command.replace("|old|", old)
command = command.replace("|new|", new)
command = command.replace("|name|", name)

print "git filter-branch -f " + command

process = subprocess.Popen(['git filter-branch -f', command],cwd=os.path.dirname(repoPath), shell=True)
process.wait()

The command executes fine, but tells me that nothing changed in the repo history. However, if I take the command that is printed out (which should be what is being executed), drop it in a shell script, and execute it, it changes the history fine. I think that the command is somehow not being executed correctly. Is there any way for be to see exactly what command the subprocess module is executing?

Answer 1

When you use shell = True, subprocess.Popen expects a string as its first argument. It is better not to use shell = True if you can help it, since it can be a security risk (see the Warning.

When you omit shell = True, or use shell = False, subprocess.Popen expects a list of arguments. You can generate that list of arguments from a string using shlex.split:

import shlex
import subprocess

def filter_history(old, new, name, repoPath):
    """Change author info
    """
    # http://help.github.com/change-author-info/
    # http://stackoverflow.com/a/3880493/190597
    command = """git filter-branch -f --env-filter '
        an="$GIT_AUTHOR_NAME"
        am="$GIT_AUTHOR_EMAIL"
        cn="$GIT_COMMITTER_NAME"
        cm="$GIT_COMMITTER_EMAIL"

        if [[ "$GIT_COMMITTER_NAME" = "{old}" ]]
        then
            cn="{name}"
            cm="{new}"
        fi

        if [[ "$GIT_AUTHOR_NAME" = "{old}" ]]
        then
            an="{name}"
            am="{new}"
        fi

        export GIT_AUTHOR_NAME="$an"
        export GIT_AUTHOR_EMAIL="$am"
        export GIT_COMMITTER_NAME="$cn"
        export GIT_COMMITTER_EMAIL="$cm"
      '
      """.format(old = old, new = new, name = name)

    process = subprocess.Popen(
        shlex.split(command),
        cwd = os.path.dirname(repoPath))
    process.communicate()

Answer 2

If your application is running in a Windows environment, as stated in the following answer, subprocess has an undocumented function called subprocess.list2cmdline which you could use. subprocess.list2cmdline translates a sequence of arguments into a command line string, using the same rules as the MS C runtime.

if you are using Python > 3.3 you could also get the args list directly from the subprocess object using .args:

import subprocess

process = subprocess.Popen(...)
subprocess.list2cmdline(process.args)

Since Python 3.8 there is also a possibility to use the shlex.join() function:

Keep in mind though that subprocess does everything via IPC, so the best approach would be to simply examine the args list, as they will be passed to argv in the called program.