How do I run the sed command with input and output as the same file?

Question

I'm trying to do use the sed command in a shell script where I want to remove lines that read STARTremoveThisComment and lines that read removeThisCommentEND.

I'm able to do it when I copy it to a new file using

sed 's/STARTremoveThisComment//' > test

But how do I do this by using the same file as input and output?

Answer 1

sed -i (or the extended version, --in-place) will automate the process normally done with less advanced implementations, that of sending output to temporary file, then renaming that back to the original.

The -i is for in-place editing, and you can also provide a backup suffix for keeping a copy of the original:

sed -i.bak fileToChange
sed --in-place=.bak fileToChange

Both of those will keep the original file in fileToChange.bak.

Keep in mind that in-place editing may not be available in all sed implementations but it is in GNU sed which should be available on all variants of Linux, as per your tags.

If you're using a more primitive implementation, you can use something like:

cp oldfile oldfile.bak && sed 'whatever' oldfile >newfile && mv newfile oldfile

Answer 2

You can use the flag -i for in-place editing and the -e for specifying normal script expression:

sed -i -e 's/pattern_to_search/text_to_replace/' file.txt

To delete lines that match a certain pattern you can use the simpler syntax. Notice the d flag:

sed -i '/pattern_to_search/d' file.txt

Answer 3

You really should not use sed for that. This question seems to come up ridiculously often, and it seems very strange that it does since the general solution is so trivial. It seems bizarre that people want to know how to do it in sed, and in python, and in ruby, etc. If you want to have a filter operate on an input and overwrite it, use the following simple script:

#!/bin/sh -e
in=${1?No input file specified}
mv $in ${bak=.$in.bak}
shift
"$@" < $bak > $in

Put that in your path in an executable file name inline, and then the problem is solved in general. For example:

inline input-file sed -e s/foo/bar/g

Now, if you want to add logic to keep multiple backups, or if you have some options to change the backup naming scheme, or whatever, you fix it in one place. What's the command line option to get 1-up counters on the backup file when processing a file in-place with perl? What about with ruby? Is the option different for gnu-sed? How does awk handle it? The whole friggin' point of unix is that tools do one thing only. Handling logic for backup files is a second thing, and needs to be factored out. If you are implementing a tool, do not add logic to create backup files. Tell your users to use a 2nd tool for that. Integration is bad. Modularity is good. That is the unix way.

Notice that this script has several problems. The permissions/mode of the input file may be changed, for example. I'm sure there are innumerable other issues. However, by putting the backup logic in a wrapper script, you localize all of these issues and don't have to worry that sed overwrites the files and changes mode, while python keeps the file in place and does not change the inode (I made up those two cases, the point being that not all tools will use the same logic, while the wrapper script will.)