What should every Perl hacker know about perl -ne?

Question

I have been using the Perl command line with a -ne option for years, largely to process text files in ways that sed can't. Example:

cat in.txt | perl -ne "s/abc/def/; s/fgh/hij/; print;" > out.txt

I have no idea where I learned this, and have only today read perlrun and found there are other forms (perl -pe for example).

What else should I know about perl -ne?

Answer 1

perl -ne 'CODE' is equivalent to the program

while (<>) {     CODE } 

perl -ane 'CODE' and perl -F/PATTERN/ -ane are also good idioms to know about. They are equivalent to

while (<>) {     @F = split /\s+/, $_;     CODE } 

and

while (<>) {     @F = split /PATTERN/, $_;     CODE } 

Example: advanced grep:

perl -ne 'print if/REGEX1/&&!/REGEX2/&&(/REGEX3/||/REGEX4/&&!/REGEX5/)' input  perl -F/,/ -ane 'print if $F[2]==4&&$F[3]ge"2009-07-01"&&$F[3]lt"2009-08-01"' file.csv 

---

A particularly clever example that uses mismatched braces is here.

Answer 2

There is one important thing to know about perl -ne and perl -pe scripts: they implicitly use <>.

"Why is that important?" you might ask.

The magic <> operator uses the 2 arg form of open. If you recall, 2 arg open includes the specification of mode with the filename in one argument. An old style call to open FILE, $foo is vulnerable to manipulation of the file mode. A particularly interesting mode in this context is |--you open a handle to a pipe to a process you execute.

You might be thinking "Big deal!", but it is.

• Imagine a cron job executed by root to munge log files in some directory.
• The script is invoked as script *.
• Imagine a file in that directory named |rm -rf /.

What happens?

1. The shell expands the * and we get script file_1 file_2 '|rm -rf /' file_4
2. The script processes file_1 and file_2.
3. Next it opens a handle to STDIN of rm -rf /.
4. Lots of disk activity follows.
5. file_4 no longer exists, so we can't open it.

Of course, the possibilities are endless.

You can read more discussion of this issue at Perlmonks.

The moral of the story: be careful with the <> operator.

FWIW, I just confirmed that this is still an issue with perl 5.10.0.

Answer 3

You can specify more than one -e clause. Sometimes I have a command line that starts growing as I refine a search / extract / mangulation operation. if you mistype something, you will get a "line number" telling you which -e has the error.

Of course, some might argue that if you have more than one or two -e clauses, maybe you should put whatever it is into a script, but some stuff really is just throw away, so why bother.

perl -n -e 'if (/good/)' -e '{ system "echo $_ >> good.txt"; }' \
-e 'elsif (/bad/)' -e '{ system "echo $_ >> bad.txt"; }' \
-e 'else' -e '{ system "echo $_ >> ugly.txt"; }' in.txt another.txt etc.txt

Presumably you would do something less trivial than grep / egrep into 3 files :-)

Answer 4

The -i option lets you do the changes inline:

 perl -i -pe 's/abc/def/; s/fgh/hij/' file.txt

or save a backup:

 perl -i.bak -pe 's/abc/def/; s/fgh/hij/' file.txt