Using Unix tools to process text: search and replace all text that are not between some lines

Question

I'm looking to do some text processing on a bunch of *.org files. I would like to change the following in each file:

[my description](link)

to

[[link][my description]]

,

`some text`

to

=some text=

,

## some heading

to

** some heading

,

*some italics*

to

/some italics/

, and

**some bold**

to

*some bold*

. Yes, this IS markdown syntax to org-mode syntax. I AM aware of pandoc. The caveat is that I want the above changes, except when they occur in the following block:

#+BEGIN_EXAMPLE
don't want above changes to take place in this block
...
#+END_EXAMPLE

Hence, I can't use pandoc. I'd like to process these files according to the above requirements using some kind of unix script: awk, sed, python, perl, bash, etc. Once I have a working script, I can modify it and learn from it.

Thanks for your help!

Answer 1

Perl Solution

This is the result of the simplifying changes I suggested for @jkerian’s script: use the flipflop operator and -p. I’ve also fixed his regexes to use the correct $1 and $2 in the RHS, altered delimiters from s/// to s::: to avoid LTS (“Leaning Toothpick Syndrome”), and added /x to improve readability. There was a logic error in dealing with bold and italics, which I’ve fixed. I added comments showing what the transform should be in each case, corresponding to the original problem description, and aligned the RHS of the transforms to make them easier to read.

#!/usr/bin/perl -p
#
# the -p option makes this a pass-through filter
#####################################################

# omit protected region
next if /^#\+BEGIN_EXAMPLE/ .. /^#\+END_EXAMPLE/;

# `some text`                      ⇒   =some text=
s: ` ( [^`]* ) `                       :=$1=:gx;

# [desc](link)                     ⇒   [[link][desc]]
s: \[ ( [^]]* ) \] \( ( [^)]* ) \)     :[[$2][$1]]:gx;

# ^## some heading ⇒ ** some heading
#      NB: can't use /x here or would have to use ugly \#
s:^##:**:;   

# *some italics*                   ⇒   /some italics/
s: (?!< \* ) \* ( [^*]+ ) \* (?! \*)   :/$1/:gx;

# **some bold**                    ⇒   *some bold*
s: \*{2} ( [^*]+ ) \*{2}               :*$1*:gx;

See how easy that is? Just 6 simple lines of eminently readable code in Perl. It’s easy in Perl because Perl is specifically designed to make writing this sort of filter super-easy, and Python is not. Python has separate design goals.

Although you could certainly rewrite this in Python, it wouldn’t be worth the bother because Python simply isn’t designed for this sort of thing. Python’s missing the -p “make-me-a-filter” flag for an implicit loop and implicit print. Python is missing the implicit accumulator variable. Python is missing built-in regexes. Python is missing the s/// operator. And Python is missing the stateful flipflop operator. All those contribute to making the Perl solution much much much easier to read, write, and maintain than the Python solution.

However, you should not get the idea that this always holds. It doesn’t. In other areas, you can come up with problems that Python comes out ahead in those areas. But not here. It’s because this filter thing is a focused specialty area for Perl, and it isn’t for Python.

The Python solution would consequently be much longer, noisier, and harder to read — and therefore harder to maintain — than this easy Perl version, all because Perl was designed to make easy things easy, and this is one of its target application areas. Try rewriting this in Python and notice how nasty it is. Sure it’s possible, but not worth the hassle, or the maintenance nightmare.

Python Version

#!/usr/bin/env python3.2

from __future__ import print_function

import sys
import re

if (sys.version_info[0] == 2):
    sys.stderr.write("%s: legacy Python detected! Please upgrade to v3+\n"
                   % sys.argv[0] )
    ##sys.exit(2)

if len(sys.argv) == 1:
    sys.argv.append("/dev/stdin")

flip_rx = re.compile(r'^#\+BEGIN_EXAMPLE')
flop_rx = re.compile(r'^#\+END_EXAMPLE')

#EG# `some text`  -->   =some text=
lhs_backticks = re.compile(r'` ( [^`]* ) `', re.VERBOSE)
rhs_backticks =            r'=\1='

#EG# [desc](link)  -->  [[link][desc]]
lhs_desclink  = re.compile(r' \[ ( [^]]* ) \] \( ( [^)]* ) \) ', re.VERBOSE)
rhs_desclink  =            r'[[\2][\1]]'

#EG# ^## some heading  -->  ** some heading
lhs_header    = re.compile(r'^##')
rhs_header    =            r'**'

#EG# *some italics*  -->  /some italics/
lhs_italics   = re.compile(r' (?!< \* ) \* ( [^*]+ ) \* (?! \*)  ', re.VERBOSE)
rhs_italics   =            r'/\1/'

## **some bold**  -->  *some bold*
lhs_bold      = re.compile(r'\*{2} ( [^*]+ ) \*{2}', re.VERBOSE)
rhs_bold      =            r'*\1*'

errcnt = 0

flipflop = "flip"

for filename in sys.argv[1:]:
    try:
        filehandle = open(filename, "r")
    except IOError as oops:
        errcnt = errcnt + 1
        sys.stderr.write("%s: can't open '%s' for reading: %s\n"
                      % ( sys.argv[0],    filename,        oops) )
    else:
        try:
            for line in filehandle:

                new_flipflop = None

                if flipflop == "flip":
                    if flip_rx.search(line):
                        new_flipflop = "flop"
                elif flipflop == "flop":
                    if flop_rx.search(line):
                        new_flipflop = "flip"
                else:
                    raise FlipFlop_SNAFU

                if flipflop != "flop":
                    line = lhs_backticks . sub ( rhs_backticks, line)
                    line = lhs_desclink  . sub ( rhs_desclink,  line)
                    line = lhs_header    . sub ( rhs_header,    line)
                    line = lhs_italics   . sub ( rhs_italics,   line)
                    line = lhs_bold      . sub ( rhs_bold,      line)                        
                print(line, end="")

                if new_flipflop != None:
                    flipflop = new_flipflop

        except IOError as oops:
            errcnt = errcnt + 1
            sys.stderr.write("%s: can't read '%s': %s\n"
              % ( sys.argv[0],    filename,        oops) )
        finally:
            try:
                filehandle.close()
            except IOError as oops:
                errcnt = errcnt + 1
                sys.stderr.write("%s: can't close '%s': %s\n"
                  % ( sys.argv[0],    filename,        oops) )

if errcnt == 0:
    sys.exit(0)
else:
    sys.exit(1)

Summary

It’s important to use the right tool for the right job. For this task, that tool is Perl, which took only 7 lines. There are only 7 things to do, but don’t try telling Python that. It’s like being back to assembly language with too many interrupt stacks. Python at 72 lines is clearly not cut out for this kind of work, and all the painful complexity and noisey unreadable code shows you just exactly why. Bug rate per line of code is the same no matter the language, so if you have your choice between writing N lines of code or 10*N lines of code, there is no choice.