Bash inserting quotes into string before execution

Question

I have managed to track done a weird problem in an init script I am working on. I have simplified the problem down in the following example:

> set -x                           # <--- Make Bash show the commands it runs
> cmd="echo \"hello this is a test\""
+ cmd='echo "hello this is a test"'
> $cmd
+ echo '"hello' this is a 'test"'  # <--- Where have the single quotes come from?
"hello this is a test"

Why is bash inserting those extra single quotes into the executed command?

The extra quotes don't cause any problems in the above example, but they are really giving me a headache.

For the curious, the actual problem code is:

cmd="start-stop-daemon --start $DAEMON_OPTS \
    --quiet \
    --oknodo \
    --background \
    --make-pidfile \
    $* \
    --pidfile $CELERYD_PID_FILE
    --exec /bin/su -- -c \"$CELERYD $CELERYD_OPTS\" - $CELERYD_USER"

Which produces this:

start-stop-daemon --start --chdir /home/continuous/ci --quiet --oknodo --make-pidfile --pidfile /var/run/celeryd.pid --exec /bin/su -- -c '"/home/continuous/ci/manage.py' celeryd -f /var/log/celeryd.log -l 'INFO"' - continuous

And therefore:

/bin/su: invalid option -- 'f'

Note: I am using the su command here as I need to ensure the user's virtualenv is setup before celeryd is run. --chuid will not provide this

Answer 1

Because when you try to execute your command with

$cmd

only one layer of expansion happens. $cmd contains echo "hello this is a test", which is expanded into 6 whitespace-separated tokens:

1. echo
2. "hello
3. this
4. is
5. a
6. test"

and that's what the set -x output is showing you: it's putting single quotes around the tokens that contain double quotes, in order to be clear about what the individual tokens are.

If you want $cmd to be expanded into a string which then has all the bash quoting rules applied again, try executing your command with:

bash -c "$cmd"

or (as @bitmask points out in the comments, and this is probably more efficient)

eval "$cmd"

instead of just

$cmd

Answer 2

Use Bash arrays to achieve the behavior you want, without resorting to the very dangerous (see below) eval and bash -c.

Using arrays:

declare -a CMD=(echo --test-arg \"Hello\ there\ friend\")
set -x
echo "${CMD[@]}"
"${CMD[@]}"

outputs:

+ echo echo --test-arg '"Hello there friend"'
echo --test-arg "Hello there friend"
+ echo --test-arg '"Hello there friend"'
--test-arg "Hello there friend"

Be careful to ensure that your array invocation is wrapped by double-quotes; otherwise, Bash tries to perform the same "bare-minimum safety" escaping of special characters:

declare -a CMD=(echo --test-arg \"Hello\ there\ friend\")
set -x
echo "${CMD[@]}"
${CMD[@]}

outputs:

+ echo echo --test-arg '"Hello there friend"'
echo --test-arg "Hello there friend"
+ echo --test-arg '"Hello' there 'friend"'
--test-arg "Hello there friend"

ASIDE: Why is eval dangerous?

eval is only safe if you can guarantee that every input passed to it will not unexpectedly change the way that the command under eval works.

Example: As a totally contrived example, let's say we have a script that runs as part of our automated code deployment process. The script sorts some input (in this case, three lines of hardcoded text), and outputs the sorted text to a file whose named is based on the current directory name. Similar to the original SO question posed here, we want to dynamically construct the --output= parameter passed to sort, but we must (must? not really) rely on eval because of Bash's auto-quoting "safety" feature.

echo $'3\n2\n1' | eval sort -n --output="$(pwd | sed 's:.*/::')".txt

Running this script in the directory /usr/local/deploy/project1/ results in a new file being created at /usr/local/deploy/project1/project1.txt.

So somehow, if a user were to create a project subdirectory named owned.txt; touch hahaha.txt; echo, the script would actually run the following series of commands:

echo $'3\n2\n1'
sort -n --output=owned.txt; touch hahaha.txt; echo .txt

As you can see, that's totally not what we want. But you may ask, in this contrived example, isn't it unlikely that the user could create a project directory owned.txt; touch hahaha.txt; echo, and if they could, aren't we already in trouble already?

Maybe, but what about a scenario where the script is parsing not the current directory name, but instead the name of a remote git source code repository branch? Unless you plan to be extremely diligent about restricting or sanitizing every user-controlled artifact whose name, identifier, or other data is used by your script, stay well clear of eval.