Why is it necessary to escape the bar Ex-command separator in mapping definitions but not in command line in Vim?

Question

I don’t understand why the following commands do different things.

Pasted in the .vimrc file, both of the commands define two versions of a mapping triggered by pressing t in Normal mode:

nnoremap t :call search('\m\(a\|b\)', 'W')<CR>
nnoremap t :call search('\m\(a\\|b\)', 'W')<CR>

Compare the effects of the above mappings with the results of running those search calls directly from the command line:

:call search('\m\(a\|b\)', 'W')
:call search('\m\(a\\|b\)', 'W')

To be specific, the “intended” behavior requires \\| in the nnoremap example, but it requires \| in the call search example.

I’m aware that the special treatment of the vertical bar character (:help :bar) is one of those traps that Vim has laid out for me, but it still doesn’t make sense. The documentation clearly says that “this list of commands will see bar as part of their argument”, but none of those exceptions apply here.

All the commands involved in this example treat bar as a meta concatenate character. Moreover, in this situation, the bar is inside a string, and—I think?—being parsed as part of a string takes priority over meta concatenate syntax.

Answer 1

Indeed, the issue is caused by the special treatment of the bar character by the mapping-creating commands.

The key-mapping mechanism in Vim is a way of making a sequence of key presses to be interpreted as another sequence of keys; no semantic interpretation of Vimscript language happens at this level. Since, in order to create a mapping, it is necessary to separate both of the key-sequence arguments to be mapped, commands of the :map-family start by determining the boundaries of the two arguments. To make use of characters that could interfere with this process in a mapping, one must use escaping syntax provided for those characters (among which are the carriage return, space, backslash, and bar characters).

Because the bar character can be used to separate a mapping command from the next Ex command and, therefore, to define the ending boundary of the right-hand-side of the mapping, it cannot be used as-is in a key sequence. According to :help map_bar, depending on settings, the bar character can be escaped as <bar>, \|, or ^V| (where ^V denotes the literal Ctrl+V key code).

Keeping that in mind, let us follow the mappings in question (around the \|/\\| part) the way they are interpreted in the default configuration. In the first mapping, the \| sequence is treated as a single | character. Therefore, after that mapping command is executed, pressing t will be the same as typing

:call search('\m\(a|b\)', 'W')Enter

When the second mapping command is run, the \\| string is interpreted as a literal backslash character (there is no need to escape \ in the right-hand-side of mappings, except for nested ones) followed by the \| specifier representing a bar character. Thus, this command maps t to the following:

:call search('\m\(a\|b\)', 'W')Enter

However, when one types the mapped search calls in Command-line mode, unlike key sequences in mappings, they are interpreted as Ex commands right away. Those bar characters occur in string literals, so there is no possibility of misinterpreting them as separators for Ex commands. When directly typed in, the commands are sent to execution as they are written. Thus, the difference between them are due to the meaning of the regular expressions \m\(a\|b\) and \m\(a\\|b\), not due to any kind of escaping behavior.