Exactly how do backslashes work within backticks?

Question

From the Bash FAQ:

Backslashes (\) inside backticks are handled in a non-obvious manner:

 $ echo "`echo \\a`" "$(echo \\a)"
 a \a
 $ echo "`echo \\\\a`" "$(echo \\\\a)"
 \a \\a

But the FAQ does not break down the parsing rules that lead to this difference. The only relevant quote from man bash I found was:

When the old-style backquote form of substitution is used, backslash retains its literal meaning except when followed by $, `, or .

The "$(echo \\a)" and "$(echo \\\\a)" cases are easy enough: Backslash, the escape character, is escaping itself into a literal backlash. Thus every instance of \\ becomes \ in the output. But I'm struggling to understand the analogous logic for the backtick cases. What is the underlying rule and how does the observed output follow from it?

Finally, a related question... If you don't quote the backticks, you get a "no match" error:

$ echo `echo \\\\a`
-bash: no match: \a

What's happening in this case?

update

Re: my main question, I have a theory for a set of rules that explains all the behavior, but still don't see how it follows from any of the documented rules in bash. Here are my proposed rules....

Inside backticks, a backslash in front of a character simply returns that character. Ie, a single backslash has no effect. And this is true for all characters, except backlash itself and backticks. In the case of backslash itself, \\ becomes an escaping backslash. It will escape its next character.

Let's see how this plays out in an example:

a=xx
echo "`echo $a`"      # prints the value of $a
echo "`echo \$a`"     # single backslash has no effect: equivalent to above
echo "`echo \\$a`"    # escaping backslash make $ literal

prints:

xx
xx
$a

Try it online!

Let's analyze the original examples from this perspective:

echo "`echo \\a`"

Here the \\ produces an escaping backslash, but when we "escape" a we just get back a, so it prints a.

echo "`echo \\\\a`"

Here the first pair \\ produces an escaping backslash which is applied to \, producing a literal backslash. That is, the first 3 \\\ become a single literal \ in the output. The remaining \a just produces a. Final result is \a.

Answer 1

The logic is quite simple as such. So we look at bash source code (4.4) itself

subst.c:9273

case '`': /* Backquoted command substitution. */
{
    t_index = sindex++;

    temp = string_extract(string, &sindex, "`", SX_REQMATCH);
    /* The test of sindex against t_index is to allow bare instances of
        ` to pass through, for backwards compatibility. */
    if (temp == &extract_string_error || temp == &extract_string_fatal)
    {
    if (sindex - 1 == t_index)
    {
        sindex = t_index;
        goto add_character;
    }
    last_command_exit_value = EXECUTION_FAILURE;
    report_error(_("bad substitution: no closing \"`\" in %s"), string + t_index);
    free(string);
    free(istring);
    return ((temp == &extract_string_error) ? &expand_word_error
                                            : &expand_word_fatal);
    }

    if (expanded_something)
    *expanded_something = 1;

    if (word->flags & W_NOCOMSUB)
    /* sindex + 1 because string[sindex] == '`' */
    temp1 = substring(string, t_index, sindex + 1);
    else
    {
    de_backslash(temp);
    tword = command_substitute(temp, quoted);
    temp1 = tword ? tword->word : (char *)NULL;
    if (tword)
        dispose_word_desc(tword);
    }
    FREE(temp);
    temp = temp1;
    goto dollar_add_string;
}

As you can see calls a function de_backslash(temp); on the string which updates the string in c. The code the same function is below

subst.c:1607

/* Remove backslashes which are quoting backquotes from STRING.  Modifies
   STRING, and returns a pointer to it. */
char *
    de_backslash(string) char *string;
{
  register size_t slen;
  register int i, j, prev_i;
  DECLARE_MBSTATE;

  slen = strlen(string);
  i = j = 0;

  /* Loop copying string[i] to string[j], i >= j. */
  while (i < slen)
  {
    if (string[i] == '\\' && (string[i + 1] == '`' || string[i + 1] == '\\' ||
                              string[i + 1] == '$'))
      i++;
    prev_i = i;
    ADVANCE_CHAR(string, slen, i);
    if (j < prev_i)
      do
        string[j++] = string[prev_i++];
      while (prev_i < i);
    else
      j = i;
  }
  string[j] = '\0';

  return (string);
}

The above just does simple thing if there is \ character and the next character is \ or backtick or $, then skip this \ character and copy the next character

So if convert it to python for simplicity

text = r"\\\\$a"

slen = len(text)
i = 0
j = 0
data = ""
while i < slen:
    if (text[i] == '\\' and (text[i + 1] == '`' or text[i + 1] == '\\' or
                             text[i + 1] == '$')):
        i += 1
    data += text[i]
    i += 1

print(data)

The output of the same is \\$a. And now lets test the same in bash

$ a=xxx

$ echo "$(echo \\$a)"
\xxx

$ echo "`echo \\\\$a`"
\xxx

Answer 2

Did some more research to find the reference and rule of what is happening. From the GNU Bash Reference Manual it states

When the old-style backquote form of substitution is used, backslash retains its literal meaning except when followed by ‘$’, ‘`’, or ‘\’. The first backquote not preceded by a backslash terminates the command substitution. When using the $(command) form, all characters between the parentheses make up the command; none are treated specially.

In other words \, \$, and ` inside of `` are processed by the CLI parser before the command substitution. Everything else is passed to the command substitution for processing.

Let's step through each example from the question. After the # I put how the command substitution was processed by the CLI parser before `` or $() is executed.

Your first example explained.

$ echo "`echo \\a`"   # echo \a
 a 
$ echo "$(echo \\a)"  # echo \\a
 \a

Your second example explained:

$ echo "`echo \\\\a`"   # echo \\a
 \a 
$ echo "$(echo \\\\a)"  # echo \\\\a
 \\a

Your third example:

a=xx
$ echo "`echo $a`"    # echo xx 
xx
$ echo "`echo \$a`"   # echo $a
xx
echo "`echo \\$a`"    # echo \$a
$a

Your third example using $()

$ echo "$(echo $a)"     # echo $a
xx
$ echo "$(echo \$a)"    # echo \$a
$a
$ echo "$(echo \\$a)"   # echo \\$a
\xx