Why does Runtime.exec(String) work for some but not all commands?

Question

When I try to run Runtime.exec(String), certain commands work, while other commands are executed but fail or do different things than in my terminal. Here is a self-contained test case that demonstrates the effect:

public class ExecTest {   static void exec(String cmd) throws Exception {     Process p = Runtime.getRuntime().exec(cmd);      int i;     while( (i=p.getInputStream().read()) != -1) {       System.out.write(i);     }     while( (i=p.getErrorStream().read()) != -1) {       System.err.write(i);     }   }    public static void main(String[] args) throws Exception {     System.out.print("Runtime.exec: ");     String cmd = new java.util.Scanner(System.in).nextLine();     exec(cmd);   } } 

The example works great if I replace the command with echo hello world, but for other commands -- especially those involving filenames with spaces like here -- I get errors even though the command is clearly being executed:

myshell$ javac ExecTest.java && java ExecTest Runtime.exec: ls -l 'My File.txt' ls: cannot access 'My: No such file or directory ls: cannot access File.txt': No such file or directory 

meanwhile, copy-pasting to my shell:

myshell$ ls -l 'My File.txt' -rw-r--r-- 1 me me 4 Aug  2 11:44 My File.txt 

Why is there a difference? When does it work and when does it fail? How do I make it work for all commands?

Answer 1

Why do some commands fail?

This happens because the command passed to Runtime.exec(String) is not executed in a shell. The shell performs a lot of common support services for programs, and when the shell is not around to do them, the command will fail.

When do commands fail?

A command will fail whenever it depends on a shell features. The shell does a lot of common, useful things we don't normally think about:

1. The shell splits correctly on quotes and spaces

   This makes sure the filename in "My File.txt" remains a single argument.

   Runtime.exec(String) naively splits on spaces and would pass this as two separate filenames. This obviously fails.

2. The shell expands globs/wildcards

   When you run ls *.doc, the shell rewrites it into ls letter.doc notes.doc.

   Runtime.exec(String) doesn't, it just passes them as arguments.

   ls has no idea what * is, so the command fails.

3. The shell manages pipes and redirections.

   When you run ls mydir > output.txt, the shell opens "output.txt" for command output and removes it from the command line, giving ls mydir.

   Runtime.exec(String) doesn't. It just passes them as arguments.

   ls has no idea what > means, so the command fails.

4. The shell expands variables and commands

   When you run ls "$HOME" or ls "$(pwd)", the shell rewrites it into ls /home/myuser.

   Runtime.exec(String) doesn't, it just passes them as arguments.

   ls has no idea what $ means, so the command fails.

What can you do instead?

There are two ways to execute arbitrarily complex commands:

Simple and sloppy: delegate to a shell.

You can just use Runtime.exec(String[]) (note the array parameter) and pass your command directly to a shell that can do all the heavy lifting:

// Simple, sloppy fix. May have security and robustness implications String myFile = "some filename.txt"; String myCommand = "cp -R '" + myFile + "' $HOME 2> errorlog"; Runtime.getRuntime().exec(new String[] { "bash", "-c", myCommand }); 

Secure and robust: take on the responsibilities of the shell.

This is not a fix that can be mechanically applied, but requires an understanding the Unix execution model, what shells do, and how you can do the same. However, you can get a solid, secure and robust solution by taking the shell out of the picture. This is facilitated by ProcessBuilder.

The command from the previous example that requires someone to handle 1. quotes, 2. variables, and 3. redirections, can be written as:

String myFile = "some filename.txt"; ProcessBuilder builder = new ProcessBuilder(     "cp", "-R", myFile,        // We handle word splitting        System.getenv("HOME")); // We handle variables builder.redirectError(         // We set up redirections     ProcessBuilder.Redirect.to(new File("errorlog"))); builder.start();