Invalid string: control characters from U+0000 through U+001F must be escaped using Bash? [duplicate]

Question

Should or should I not wrap quotes around variables in a shell script?

For example, is the following correct:

xdg-open $URL
[ $? -eq 2 ]

or

xdg-open "$URL"
[ "$?" -eq "2" ]

And if so, why?

Answer 1

General rule: quote it if it can either be empty or contain spaces (or any whitespace really) or special characters (wildcards). Not quoting strings with spaces often leads to the shell breaking apart a single argument into many.

$? doesn't need quotes since it's a numeric value. Whether $URL needs it depends on what you allow in there and whether you still want an argument if it's empty.

I tend to always quote strings just out of habit since it's safer that way.

Answer 2

In short, quote everything where you do not require the shell to perform word splitting and wildcard expansion.

Single quotes protect the text between them verbatim. It is the proper tool when you need to ensure that the shell does not touch the string at all. Typically, it is the quoting mechanism of choice when you do not require variable interpolation.

$ echo 'Nothing \t in here $will change'
Nothing \t in here $will change

$ grep -F '@&$*!!' file /dev/null
file:I can't get this @&$*!! quoting right.

Double quotes are suitable when variable interpolation is required. With suitable adaptations, it is also a good workaround when you need single quotes in the string. (There is no straightforward way to escape a single quote between single quotes, because there is no escape mechanism inside single quotes -- if there was, they would not quote completely verbatim.)

$ echo "There is no place like '$HOME'"
There is no place like '/home/me'

No quotes are suitable when you specifically require the shell to perform word splitting and/or wildcard expansion.

Word splitting (aka token splitting);

 $ words="foo bar baz"
 $ for word in $words; do
 >   echo "$word"
 > done
 foo
 bar
 baz

By contrast:

 $ for word in "$words"; do echo "$word"; done
 foo bar baz

(The loop only runs once, over the single, quoted string.)

 $ for word in '$words'; do echo "$word"; done
 $words

(The loop only runs once, over the literal single-quoted string.)

Wildcard expansion:

$ pattern='file*.txt'
$ ls $pattern
file1.txt      file_other.txt

By contrast:

$ ls "$pattern"
ls: cannot access file*.txt: No such file or directory

(There is no file named literally file*.txt.)

$ ls '$pattern'
ls: cannot access $pattern: No such file or directory

(There is no file named $pattern, either!)

In more concrete terms, anything containing a filename should usually be quoted (because filenames can contain whitespace and other shell metacharacters). Anything containing a URL should usually be quoted (because many URLs contain shell metacharacters like ? and &). Anything containing a regex should usually be quoted (ditto ditto). Anything containing significant whitespace other than single spaces between non-whitespace characters needs to be quoted (because otherwise, the shell will munge the whitespace into, effectively, single spaces, and trim any leading or trailing whitespace).

When you know that a variable can only contain a value which contains no shell metacharacters, quoting is optional. Thus, an unquoted $? is basically fine, because this variable can only ever contain a single number. However, "$?" is also correct, and recommended for general consistency and correctness (though this is my personal recommendation, not a widely recognized policy).

Values which are not variables basically follow the same rules, though you could then also escape any metacharacters instead of quoting them. For a common example, a URL with a & in it will be parsed by the shell as a background command unless the metacharacter is escaped or quoted:

$ wget http://example.com/q&uack
[1] wget http://example.com/q
-bash: uack: command not found

(Of course, this also happens if the URL is in an unquoted variable.) For a static string, single quotes make the most sense, although any form of quoting or escaping works here.

wget 'http://example.com/q&uack'  # Single quotes preferred for a static string
wget "http://example.com/q&uack"  # Double quotes work here, too (no $ or ` in the value)
wget http://example.com/q\&uack   # Backslash escape
wget http://example.com/q'&'uack  # Only the metacharacter really needs quoting

The last example also suggests another useful concept, which I like to call "seesaw quoting". If you need to mix single and double quotes, you can use them adjacent to each other. For example, the following quoted strings

'$HOME '
"isn't"
' where `<3'
"' is."

can be pasted together back to back, forming a single long string after tokenization and quote removal.

$ echo '$HOME '"isn't"' where `<3'"' is."
$HOME isn't where `<3' is.

This isn't awfully legible, but it's a common technique and thus good to know.

As an aside, scripts should usually not use ls for anything. To expand a wildcard, just ... use it.

$ printf '%s\n' $pattern   # not ``ls -1 $pattern''
file1.txt
file_other.txt

$ for file in $pattern; do  # definitely, definitely not ``for file in $(ls $pattern)''
>  printf 'Found file: %s\n' "$file"
> done
Found file: file1.txt
Found file: file_other.txt

(The loop is completely superfluous in the latter example; printf specifically works fine with multiple arguments. stat too. But looping over a wildcard match is a common problem, and frequently done incorrectly.)

A variable containing a list of tokens to loop over or a wildcard to expand is less frequently seen, so we sometimes abbreviate to "quote everything unless you know precisely what you are doing".

Answer 3

Here is a three-point formula for quotes in general:

Double quotes

In contexts where we want to suppress word splitting and globbing. Also in contexts where we want the literal to be treated as a string, not a regex.

Single quotes

In string literals where we want to suppress interpolation and special treatment of backslashes. In other words, situations where using double quotes would be inappropriate.

No quotes

In contexts where we are absolutely sure that there are no word splitting or globbing issues or we do want word splitting and globbing.

---

Examples

Double quotes

• literal strings with whitespace ("StackOverflow rocks!", "Steve's Apple")
• variable expansions ("$var", "${arr[@]}")
• command substitutions ("$(ls)", "`ls`")
• globs where directory path or file name part includes spaces ("/my dir/"*)
• to protect single quotes ("single'quote'delimited'string")
• Bash parameter expansion ("${filename##*/}")

Single quotes

• command names and arguments that have whitespace in them
• literal strings that need interpolation to be suppressed ( 'Really costs $$!', 'just a backslash followed by a t: \t')
• to protect double quotes ('The "crux"')
• regex literals that need interpolation to be suppressed
• use shell quoting for literals involving special characters ($'\n\t')
• use shell quoting where we need to protect several single and double quotes ($'{"table": "users", "where": "first_name"=\'Steve\'}')

No quotes

• around standard numeric variables ($$, $?, $# etc.)
• in arithmetic contexts like ((count++)), "${arr[idx]}", "${string:start:length}"
• inside [[ ]] expression which is free from word splitting and globbing issues (this is a matter of style and opinions can vary widely)
• where we want word splitting (for word in $words)
• where we want globbing (for txtfile in *.txt; do ...)
• where we want ~ to be interpreted as $HOME (~/"some dir" but not "~/some dir")

---

See also:

• Difference between single and double quotes in Bash
• What are the special dollar sign shell variables?
• Quotes and escaping - Bash Hackers' Wiki
• When is double quoting necessary?