Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Sed replace with the output of a bash command using capture group as argument

Tags:

bash

sed

I'm trying to make some base64 substitution with sed.

What I'm trying to do is this:

sed -i "s|\(some\)\(pattern\)|\1 $(echo "\2" | base64 -d)|g" myFile

In English that would be:

  • Math a pattern
  • Capture groups
  • Use the captured group in a bash command
  • Use the output of this command as a replacement string

So far my command doesn't work since \2 is known only by sed and not by the bash command I'm calling.

What elegant solution to I have to pass a capture group to a command of which I want to use the output?


Edit

Here is a minimal example of what I'm trying to do:

I have the following file:

someline
someline
Base64Expression stringValue="Zm9v"
someline
Base64Expression stringValue="YmFy"

And I want to replace the base64 by plain text:

someline
someline
Base64Expression stringValue="foo"
someline
Base64Expression stringValue="bar"

In the future I'll have to do the backward operation (encoding string in base64 on the decoded file)

I've started using awk but I though it could get simpler (and much more elegant) with sed. So far with awk I have this (where $bundle is the file I'm editing):

#For each line containing "Base64Expression"
#Put in the array $substitutions[]:
# The number of the line (NR)
# The encoded expression ($2)
# The decoded expression (x)
substitutions=($(awk -v bd=$bundle '
    BEGIN {
        # Change the separator from default
        FS="""
        ORS=","
        OFS=","
    }
    /Base64Expression/ {
        #Decode the base64 lines
        cmd="echo -ne \""$2"\" | base64 -d"
        cmd | getline x

        if ( (cmd | getline) == 0 ){
            print NR, $2, x
        }
    }
' $bundle))

# Substitute the encoded expressions by the decoded ones
# Use the entries of the array 3 by 3
# Create a sed command which takes the lines numbers
for ((i=0; i<${#substitutions[@]}; i+=3))
do
    # Do the substitution only if the string is not empty
    # Allows to handle properly the empty variables
    if [ ${substitutions[$((i+1))]} ]
    then
        sed -i -e "${substitutions[$i]}s#${substitutions[$((i+1))]}#${substitutions[$((i+2))]}#" $bundle
    fi
done
like image 810
statox Avatar asked Sep 06 '16 12:09

statox


People also ask

How do you replace something with sed?

Find and replace text within a file using sed command The procedure to change the text in files under Linux/Unix using sed: Use Stream EDitor (sed) as follows: sed -i 's/old-text/new-text/g' input.txt. The s is the substitute command of sed for find and replace.

Can you use sed in a bash script?

We can manipulate text in streams and files using sed from the command line in Bash and other command-line shells.

Which sed command is used to modify all text matches in a line of text?

'g' option is used in `sed` command to replace all occurrences of matching pattern. Create a text file named python.

What does sed bash do?

sed is the Stream EDitor. It can do a whole pile of really cool things, but the most common is text replacement. The s,%,$,g part of the command line is the sed command to execute. The s stands for substitute, the , characters are delimiters (other characters can be used; / , : and @ are popular).


2 Answers

You can use e in GNU sed to pass the substitution string to a shell for evaluation. This way, you can say:

printf "%s %s" "something" "\1"

Where \1 holds a captured group. All together:

$ sed -r 's#match_([0-9]*).*#printf "%s %s" "something" "\1"#e' <<< "match_555 hello"
something 555

This comes handy when you want to perform some shell action with a captured group, like in this case.

So, let's capture the first part of the line, then the part that needs to be encoded and finally the rest. Once this is done, let's print those pieces back with printf triggering the usage of base64 -d against the second slice:

$ sed -r '/^Base64/s#(.*;)([^\&]*)(&.*)# printf "%s%s%s" "\1" $(echo "\2" | base64 -d) "\3";#e' file
someline
someline
Base64Expression stringValue=&quot;foo&quot;
someline
Base64Expression stringValue=&quot;bar&quot;

Step by step:

sed -r '/^Base64/s#(.*;)([^\&]*)(&.*)# printf "%s%s%s" "\1" $(echo "\2" | base64 -d) "\3";#e' file
#        ^^^^^^^    ^^^  ^^^^^^  ^^^                        ^^^^^^^^^^^^^^^^^^^^^^^^       ^
#           |   first part  |   the rest                encode the 2nd captured group      |
#           |               |                                                              |
#           |           important part                                      execute the command
#           |
# on lines starting with Base64, do...

The idea comes from this superb answer by anubhava on How to change date format in sed?.

like image 88
fedorqui 'SO stop harming' Avatar answered Oct 11 '22 02:10

fedorqui 'SO stop harming'


It sounds like this is what you're trying to do:

$ cat tst.awk
BEGIN { FS=OFS="&quot;" }
/^Base64Expression/ {
    cmd="echo -ne \""$2"\" | base64 -d"
    if ( (cmd | getline x) > 0 ) {
        $2 = x
    }
    close(cmd)
}
{ print }

$ awk -f tst.awk file
someline
someline
Base64Expression stringValue=&quot;foo&quot;
someline
Base64Expression stringValue=&quot;bar&quot;

assuming your echo | base64 is the right approach.

like image 34
Ed Morton Avatar answered Oct 11 '22 00:10

Ed Morton