Maven command mvn runs without errors from terminal but not from python

Question

I am trying to run a maven project from a python script. I have installed apache maven. Running the command: mvn exec:java -D"exec.mainClass"="org.matsim.project.RunMatsim" from terminal in the project folder where the pom.xml is, creates no errors and the project runs correctly.

But when running the following code from my python script

import subprocess as sp

def execute(cmd):
    popen = sp.Popen(cmd, stdout=sp.PIPE, universal_newlines=True,shell=True)
    for stdout_line in iter(popen.stdout.readline, ""):
        yield stdout_line 
    popen.stdout.close()
    return_code = popen.wait()
    if return_code:
        raise sp.CalledProcessError(return_code, cmd)

for path in execute(["mvn", "exec:java" ,'-D"exec.mainClass"="org.matsim.project.MatsimRun"']):
    print(path, end="")

I got the following error:

[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format : or :[:]:. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify`

Why this is the case? What could be wrong?

The occurred warnings are the same for both cases (terminal, python script).

Answer 1

Generally when I run into issues with args when running a program from java/python/etc. I wrap the command I want to use in an sh -c, like so:

sp.Popen(['/bin/sh', '-c', "mvn exec:java -D\"exec.mainClass\"=\"org.matsim.project.MatsimRun\""])

The above seems to work properly for me. Please be aware there may be security implications of doing this, make sure that the command entered as the third parameter of sh is never hooked up to the internet of hate.

Okay so now why was your code not working?

The python docs say that shell=True is equivalent to:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

So a good hypothesis is that sh is ignoring arg[1] and further arguments. You can test this by doing:

<censored>@debian:~$ sh -c echo "sfaklg"

<censored>@debian:~$ 

Observe how echo doesn't get the args it expects. This consistent across many different quoting methods/shells:

<censored>@debian:~$ dash -c echo "sfaklg"

<censored>@debian:~$ bash -c echo "sfaklg"

<censored>@debian:~$ zsh -c echo "sfaklg"

<censored>@debian:~$ zsh -c echo sfaklg

<censored>@debian:~$ bash -c echo sfaklg

<censored>@debian:~$ 

The man page for bash says:

If the -c option is present, then commands are read from the first non-option argument command_string. If there are arguments after the command_string, the first argument is assigned to $0 and any remaining arguments are assigned to the positional parameters. The assignment to $0 sets the name of the shell, which is used in warning and error messages.

In other words sh -c is intended for running shell scripts which expect args to be passed with $1 , $2, ... .

So what should you do?

Is suspect you want to run maven in a shell because it needs JAVA_HOME or similar environment vars to be set. This question gives examples of how to set/modify the environment vars, though you may still need shell=True. This answer also suggests using the env command to override environment variables.

Lastly you may find this question helpful.