Zsh: Why is \n interpreted within single quotes?

Question

From my understanding, a \n should be interpreted as a new line in

"a\nb"
$'a\nb'

but not in

'a\nb'

However, I see the following:

(transscript from my zsh session follows)

-0-1- ~  > zsh --version
zsh 5.1.1 (x86_64-unknown-cygwin)
-0-1- ~  > echo 'a\nb'
a
b
-0-1- ~  >

Answer 1

zsh itself does not interpret the \n in 'a\nb', but the builtin echo does. 'a\nb' and "a\nb" are equivalent and do not contain a newline but the (literal) character sequence \n. $'a\nb' on the other hand actually does contains a newline.

There are two things at work here:

1. Quoting:

Quoting is used to tell the shell that you want a character itself, not any special meaning it may have in the shells syntax. zsh has four types of quoting, some of which may retain or add special meaning to a few characters or character sequences:

1. \: Quoting of single characters by prepending \. For zsh saying \n only means "I want the character n". As n has no special meaning, it is the same as writing just n. This changes with characters like *: Without quoting * is used as a wildcard for globbing, writing \* prevents that (for example: compare outputs of echo /* and echo /\*). If you want to pass the character \ literally, you have to quote it, for example with another \: \\.
2. '...': Just like with \ any character withing '...' is taken literally, this includes other methods of quoting. 'foo* bar?' is essentially equivalent to foo\*\ bar\? (or \f\o\o\*\ \b\a\r\? if one wants to be pedantic). Only ' itself cannot appear inside a string quoted with '...' as it would mark the end of the qoute. (This can be changed by setting RC_QUOTES. If set a pair of single quote is taken as a single quote: 'foo''bar' → foo\'bar)
3. "...": Allows for parameter and command substitution. That means words after a $ are taken as parameter names (e.g. "path: $PATH") and strings wrapped in $(...) or `...` are used for command substitution (e.g. echo "Date: $(date)"). Otherwise it behaves like '...' with the exception that it allows the quoting of `, $, \ and " with \.
4. $'...': strings insid $'...' are treated like string arguments of the print builtin. Only here some alphabetic characters have indeed special meaning: \n for newline, \b for backspace, \a for the bell character, etc.. \ and ' can be quoted with \\ and \' respectively. The resulting string is considered fully quoted. There is no parameter or command substitution inside $'...'.

2. The zsh-builtin echo:

Other than the echo binary or the bash-builtin echo the zsh-builtin by default recognizes (some) escape sequences like \n or \t. While this behavior usually needs to be explicitly enabled with /bin/echo or the bash-builtin (usually by passing the -e flag: echo -e "foo\nbar"), for the zsh-builtin it need to be explicitly disabled. Either by passing the -E flag (echo -E 'foo\nbar') or by setting the BSD_ECHO option (setopt bsdecho) in which case the -e flag can be used to re-enable the feature like with the other types of echo.

Conclusion

That means that both 'a\nb' and "a\nb" (and a\\nb for that matter) are passed as a\nb (literally), but the zsh-builtin echo then interprets the \n, leading to an output with a newline. On the other hand $'a\nb' contains a literal newline already before it is passed to echo.

Running

for quoted_string in a\\nb 'a\nb' "a\nb" $'a\nb'; do
    echo '>' $quoted_string '<'
    /bin/echo -e '>' $quoted_string '<'
    echo -E '>' $quoted_string '<'
    /bin/echo '>' $quoted_string '<'
    echo
done

should get you the following output:

> a
b <
> a
b <
> a\nb <
> a\nb <

> a
b <
> a
b <
> a\nb <
> a\nb <

> a
b <
> a
b <
> a\nb <
> a\nb <

> a
b <
> a
b <
> a
b <
> a
b <

As you can see there is no difference between the first three kinds of quoting, while the fourth always prints with a newline.

---

BTW: perl (at least version 5; I do not know about Perl 6) behaves in the way you describe the expected behavior. In perl '...' behaves like it does in zsh. On the other hand"..." in perl behaves like a combination of "..." and $'...' of zsh: variables are replaced by their value and character sequences like \n and \t are treated specially.

Answer 2

You are wrong about "a\nb", which does not replace \n with a newline (well, not portably, anyway). So there is no standard quoting mechanism that allows a newline in a string without using a literal newline, as in

x="a
b"

This is why the $'a\nb' notation was invented to do just that. It's not POSIX as of 2016, but the POSIX folks are thinking about adding it to the shell specification. As always, don't hold your breath :-)