Doubled escape character

Question

I am currently trying to do the following:

cmd = r'sudo sed -irn "1!N; s/<ip>127.0.0.1<\/ip>(\n.*4000.*)/<ip>0.0.0.0<\/ip>\1/" /usr/something.conf'
subprocess.Popen(cmd)

However Popen is complaining that \\1 is an invalid reference. Upon inspecting it in pdb I see this,

'sudo sed -irn "1!N; s/<ip>127.0.0.1<\\/ip>(\\n.*4000.*)/<ip>0.0.0.0<\\/ip>\\1/" /usr/something.conf'

It appears as though python is adding an extra \. Is there any way to prevent that so that I can run the command as is using Popen?

Also, for simplification I left it out of the example but this is actually being wrapped in an SSH call before being passed to Popen, so yes... it does need to be done with Popen and sed.

For reference here is the full chain of steps the string goes through to be run...

def _formatCmd(cmdString, host=None, user=None, keyfile=None):
    cmd = []
    if host:
        cmd.append('ssh')
        keyfile = keyfile or getKeyFile()
        if keyfile:
            cmd.append('-i')
            cmd.append(keyfile)
        cmd.append("%s@%s" % (user, host))
        cmd.append(cmdString)
    else:
        cmd += cmdString.split()

    return cmd


def runCmd(host, user, cmd, timeout=None, cleanup=False):
    try:
        cmd = _formatCmd(cmd, host=host, user=user)
    except:
        pass

    #create cmd and add it to list of running cmds
    proc = Popen(cmd, stdout=PIPE, stderr=PIPE)
    runningCmds[proc.pid] = proc

    t = threading.Timer(timeout, proc.kill)
    t.start()
    stdout, stderr = proc.communicate()
    t.cancel()

    del runningCmds[proc.pid]
    return (proc.returncode, stdout, stderr)

cmd = r'sudo sed -irn "1!N; s/<ip>127.0.0.1<\/ip>(\n.*4000.*)/<ip>0.0.0.0<\/ip>\1/" /usr/something.conf'
runCmd('1.1.1.1', 'username', cmd)

The exact error message returned is:

sed: -e expression #1, char 59: invalid reference \\1 on `s' command's RHS

Answer 1

The problem is that the shell is doing its own interpreting/escaping. I found something similar when using cygwin (in the cygwin case the shell was bash).

The fact that you get the error:

sed: -e expression #1, char 59: invalid reference \\1 on `s' command's RHS

implies that it is the brackets () that are the problem rather than the escaping of the \1. Essentially, it is not able to find the group, so you need to escape the brackets \(...\).

To find the cause, the trick is to use echo to debug what is being sent:

1. Simplify the regex to just a group and the match. Something like:

   s/(one)/\1\1/
   

   where the input string is 'one' and the expected output is oneone

2. Change your cmd to echo so that what you pass to the shell is echo s/(one)/\1\1/

3. I am guessing you will see something like bash: syntax error near unexpected token '('. This gives us our clue. Basically, we need to escape the brackets.

4. So now try echo s/\(one\)/\1\1. In my case I see something like

   s/(one)/\1\1
   

5. With any luck that should do it and you should be able to apply it to your problematic expression.

It may be simplest to use strong quoting (surrounding command in single quotes) which tells bash not to interpret the string, though you will probably still have to escape the brackets ().

As an aside, for cygwin things need to be twice escaped, so the actual correct expression is:

sed s/\\\(one\\\)/\\1\\1/

so

echo one | sed s/\\\(one\\\)/\\1\\1/

gives

oneone

The equivalent using strong quoting is:

 echo one | sed 's/\(one\)/\1\1/'