Python Popen fails in compound command (PowerShell)

Question

I am trying to use Python's Popen to change my working directory and execute a command.

pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()

However, powershell returns "The system cannot find the path specified." The path does exist.

If I run

 pg = subprocess.Popen("cd c:/mydirectory ;", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)

it returns the same thing.

However, if i run this: (without the semicolon)

pg = subprocess.Popen("cd c:/mydirectory",stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)

The command returns without an error. This leads me to believe that the semicolon is issue. What is the cause for this behavior and how can I get around it?

I know I can just do c:/mydirectory/runExecutable.exe --help, but I would like to know why this is happening.

UPDATE :

I have tested passing the path to powershell as the argument for Popen's executable parameter. Just powershell.exe may not be enough. To find the true absolute path of powershell, execute where.exe powershell. Then you can pass it into Popen. Note that shell is still true. It will use the default shell but pass the command to powershell.exe

powershell = C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True, executable=powershell)
buff,buffErr = pg.communicate()
//It works!

Answer 1

In your subprocess.Popen() call, shell=True means that the platform's default shell should be used.

While the Windows world is - commendably - moving from CMD (cmd.exe) to PowerShell, Python determines what shell to invoke based on the COMSPEC environment variable, which still points to cmd.exe, even in the latest W10 update that has moved toward PowerShell in terms of what the GUI offers as the default shell.

For backward compatibility, this will not change anytime soon, and will possibly never change.

Therefore, your choices are:

• Use cmd syntax, as suggested in Maurice Meyer's answer.

• Do not use shell = True and invoke powershell.exe explicitly - see below.

• Windows only: Redefine environment variable COMSPEC before using shell = True - see below.

---

A simple Python example of how to invoke the powershell binary directly, with command-line switches followed by a single string containing the PowerShell source code to execute:

import subprocess

args = 'powershell', '-noprofile', '-command', 'set-location /; $pwd'
subprocess.Popen(args)

Note that I've deliberately used powershell instead of powershell.exe, because that opens up the possibility of the command working on Unix platforms too, once PowerShell Core is released.

---

Windows only: An example with shell = True, after redefining environment variable COMSPEC to point to PowerShell first:

import os, subprocess    

os.environ["COMSPEC"] = 'powershell'

subprocess.Popen('Set-Location /; $pwd', shell=True)

Note:

• COMSPEC is only consulted on Windows; on Unix platforms, the shell executable is invariably /bin/sh

• As of Windows PowerShell v5.1 / PowerShell Core v6-beta.3, invoking powershell with just -c (interpreted as -Command) still loads the profiles files by default, which can have unexpected side effects (with the explicit invocation of powershell used above, -noprofile suppresses that).

  • Changing the default behavior to not loading the profiles is the subject of this GitHub issue, in an effort to align PowerShell's CLI with that of POSIX-like shells.