Newlines at the end "$( .... )" get removed in shell scripts. Why?

Question

Can someone explain why the output of these two commands is different?

$ echo "NewLine1\nNewLine2\n"
NewLine1
NewLine2
                         <-- Note 2nd newline here
$ echo "$(echo "NewLine1\nNewLine2\n")"
NewLine1
NewLine2
$                        <-- No second newline

Is there any good way that I can keep the new lines at the end of the output in "$( .... )" ? I've thought about just adding a dummy letter and removing it, but I'd quite like to understand why those new lines are going away.

Answer 1

Because that's what POSIX specifies and has always been like that in Bourne shells:

2.6.3 Command Substitution

Command substitution allows the output of a command to be substituted in place of the command name itself. Command substitution shall occur when the command is enclosed as follows:

$(command)

or (backquoted version):

`command`

The shell shall expand the command substitution by executing command in a subshell environment (see Shell Execution Environment) and replacing the command substitution (the text of command plus the enclosing "$()" or backquotes) with the standard output of the command, removing sequences of one or more <newline> characters at the end of the substitution. Embedded <newline> characters before the end of the output shall not be removed; however, they may be treated as field delimiters and eliminated during field splitting, depending on the value of IFS and quoting that is in effect. If the output contains any null bytes, the behavior is unspecified.

One way to keep the final newline(s) would be

VAR="$(command; echo x)"   # Append x to keep newline(s).
VAR=${VAR%x}               # Chop x.

Vis.:

$ x="$(whoami; echo x)" ; printf '<%s>\n' "$x" "${x%x}"
<user
x>
<user
>

But why remove trailing newlines? Because more often than not you want it that way. I'm also programming in perl and I can't count the number of times where I read a line or variable and then need to chop the newline:

while (defined ($string = <>)) {
    chop $string;
    frobnitz($string);
} 

Answer 2

command substitution removes every trailing newline.

It makes sense to remove one. For instance:

basename foo/bar

outputs bar\n. In:

var=$(basename foo/bar)

you want $var to contain bar, not bar\n.

However in

var=$(basename $'foo/bar\n')

You would like $var to contain bar\n (after all, newline is as valid a character as any in a file name on Unix). But all shells remove every trailing newline character. That misfeature was in the original Bourne shell and even rc which has fixed most of Bourne's flaws has not fixed that one. (though rc has the ``(){cmd} syntax to not strip any newline character).

In POSIX shells, to work around the issue, you can do:

var=$(basename -- "$file"; echo .)
var=${var%??}

Though you're then losing the exit status of basename. Which you can fix with:

var=$(basename -- "$file" && echo .) && var=${var%??}

${var%??} is to remove the last two characters. The first one is the . that we added above, the second is the one newline character added by basename, we're not removing any more as command substitution would do as the other newline characters, if any, would be part of the filename we want to get the base of, so we do want them.

In the Bourne shell which doesn't have the ${var%x} operator, you had to go a long and convoluted way to work around it.