Reading quoted/escaped arguments correctly from a string

Question

I'm encountering an issue passing an argument to a command in a Bash script.

poc.sh:

#!/bin/bash

ARGS='"hi there" test'
./swap ${ARGS}

swap:

#!/bin/sh
echo "${2}" "${1}"

The current output is:

there" "hi

Changing only poc.sh (as I believe swap does what I want it to correctly), how do I get poc.sh to pass "hi there" and test as two arguments, with "hi there" having no quotes around it?

Answer 1

A Few Introductory Words

If at all possible, don't use shell-quoted strings as an input format.

• It's hard to parse consistently: Different shells have different extensions, and different non-shell implementations implement different subsets (see the deltas between shlex and xargs below).
• It's hard to programmatically generate. ksh and bash have printf '%q', which will generate a shell-quoted string with contents of an arbitrary variable, but no equivalent exists to this in the POSIX sh standard.
• It's easy to parse badly. Many folks consuming this format use eval, which has substantial security concerns.

NUL-delimited streams are a far better practice, as they can accurately represent any possible shell array or argument list with no ambiguity whatsoever.

---

xargs, with bashisms

If you're getting your argument list from a human-generated input source using shell quoting, you might consider using xargs to parse it. Consider:

array=( )
while IFS= read -r -d ''; do
  array+=( "$REPLY" )
done < <(xargs printf '%s\0' <<<"$ARGS")

swap "${array[@]}"

...will put the parsed content of $ARGS into the array array. If you wanted to read from a file instead, substitute <filename for <<<"$ARGS".

---

xargs, POSIX-compliant

If you're trying to write code compliant with POSIX sh, this gets trickier. (I'm going to assume file input here for reduced complexity):

# This does not work with entries containing literal newlines; you need bash for that.
run_with_args() {
  while IFS= read -r entry; do
    set -- "$@" "$entry"
  done
  "$@"
}
xargs printf '%s\n' <argfile | run_with_args ./swap

---

These approaches are safer than running xargs ./swap <argfile inasmuch as it will throw an error if there are more or longer arguments than can be accommodated, rather than running excess arguments as separate commands.

---

Python shlex -- rather than xargs -- with bashisms

If you need more accurate POSIX sh parsing than xargs implements, consider using the Python shlex module instead:

shlex_split() {
  python -c '
import shlex, sys
for item in shlex.split(sys.stdin.read()):
    sys.stdout.write(item + "\0")
'
}
while IFS= read -r -d ''; do
  array+=( "$REPLY" )
done < <(shlex_split <<<"$ARGS")

Answer 2

Embedded quotes do not protect whitespace; they are treated literally. Use an array in bash:

args=( "hi there" test)
./swap "${args[@]}"

In POSIX shell, you are stuck using eval (which is why most shells support arrays).

args='"hi there" test'
eval "./swap $args"

As usual, be very sure you know the contents of $args and understand how the resulting string will be parsed before using eval.