sed not working from within bash script

Question

I have read all the similar questions on this topic, but didn't find a matching question to what I am experiencing. I apologise if this has been answered already.

Inside a bash script I wrote, there is a very simple sed command, which does not seem to be working. There are no errors, and the command works perfectly when run from the command line.

In the output from set -x I can see the sed command executing perfectly.

GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)

Bash script: (toned down for easier comprehension)

#!/bin/bash -x

# This script has the exact same sed command as used on cli

contact='"[email protected]"'

sed -i "/$contact/d" /home/tim/Desktop/file.txt

exit

Shell output:

tim@ubuntu:~/Desktop$ cat file.txt
t,b,[email protected]
tim@ubuntu:~/Desktop$ ./test.sh 
+ contact='"[email protected]"'
+ sed -i '/"[email protected]"/d' /home/tim/Desktop/file.txt
+ exit
tim@ubuntu:~/Desktop$ cat file.txt
t,b,[email protected]
tim@ubuntu:~/Desktop$ sed -i "/"[email protected]"/d" /home/tim/Desktop/file.txt
tim@ubuntu:~/Desktop$ cat file.txt
tim@ubuntu:~/Desktop$

I assume I am missing something very obvious, but I am done staring at it hoping for the answer to jump off the screen and slap me in the face. Please help :-)

Tim

Answer 1

There are double quotes around the mail address in the $contact script variable that are missing from the command line call:

# case 1 - works
# only the sed pattern delimiters are enclosed in quotes and these quotes will be stripped by the shell. 
sed -i "/"[email protected]"/d" ./file.txt; cat file.txt

# case 2 - fails
# escaping with \ turns dquotes #2,3 from shell-level delimiters to char literals w/o special  semantics.
sed -i "/\"[email protected]\"/d" ./file.txt; cat file.txt

# case 3 - fails
# Single quotes enclose the complete sed pattern spec which comprises double quotes enclosing the mail address
sed -i '/"[email protected]"/d' ./file.txt; cat file.txt

# case 4 - works
sed -i "/[email protected]/d" ./file.txt; cat file.txt

# case 5 - works
sed -i '/[email protected]/d' ./file.txt; cat file.txt

This would explain the different behavior of the script vs. the cli call.

The OP pointed out that he needs the double quotes in the real script. that may be so, however, if these doublequotes aren't present in the file, there will be no match.

A solution would be to preprocess the file ( if necessary, work on a copy ) with sed:

sed -i 's/,/","/g; s/^/"/; s/$/"/' ./file.txt

This command assumes a comma-separated list of items on each line with no item containing double quotes. It will wrap each item in double quotes so they will match against the search pattern in the original script's $contact variable.

Alternative (adapted from this SO answer [that I have not been the author of])

Another option is changing the relevant portion of the script be deriving a second variable from $contact:

contact='"[email protected]"'
c2=$(echo $contact | tr -d '"')

sed -i "/$c2/d"  /home/tim/Desktop/file.txt

Answer 2

This is just an addition to collapsar's answer which already solved the issue.

When we use sed in bash script, bash script acts as a wrapper for sed. This has two purposes

• sed command can be executed as if it is executed outside the bash.

• bash script wrapper helps sed to communicate with the outside world, using environment variables.

For example, suppose that the file testfile contains two lines

[email protected]
[email protected]

Now If I would like to write a bash script which helps sed replace the lines that contain [email protected] my script sedscript would be like this :

#!/bin/bash
contact='[email protected]'
sed -i "/$contact/d" $1

Now I would execute the script like below

./sedscript testfile

to remove all the lines containing [email protected].

In fact you can replace $1 with the actual file name. But the important point to note, as mentioned in the previous answer, is that whenever we use a bash variable inside the sed command, always enclose the command in double quotes. Only then bash would replace the variable with corresponding string before passing it to sed.