Using sed to replace string in file with contents of variable of that name

Question

I'm trying to use sed to replace template strings in files of the form %XXX% with the value of a variable called XXX in my shell script.

e.g. The following works perfectly

sed "s/%user_home%/$user_home/gi"

So if user_home=fred the following,

NameVirtualHost *:80

<VirtualHost *:80>
  ServerName %server_name%

  ErrorLog /var/log/apache2/%user_home%_webapp_error.log
  CustomLog /var/log/apache2/%user_home%_webapp.log common

  DocumentRoot /home/%user_home%/web_app/public
</VirtualHost>

becomes,

NameVirtualHost *:80

<VirtualHost *:80>
  ServerName %server_name%

  ErrorLog /var/log/apache2/fred_webapp_error.log
  CustomLog /var/log/apache2/fred_webapp.log common

  DocumentRoot /home/fred/web_app/public
</VirtualHost>

The problem is that I want to run the sed command without explicitly knowing the template strings and their variables up front. That is, it looks for %XXX% and then replaces that with the contents of $XXX without caring what the actual name of the variable is.

I know its got something to do with back-references but I can't figure out how to use the content of a back-reference as the variable name.

I tried,

sed "s/%\([a-z_]\)%/$(\1)/gi"

but this failed to work because it seems to be a looking for a variable called $\1.

Answer 1

The problem here is that by the time the sed command is actually run (and therefore by the time it retrieves the variable-name), the sed command must have been fully assembled (including substituting the Bash variable's value into the replacement string); so everything happens in the wrong order.

Or, taking a higher-level view, the problem is that sed doesn't know about Bash variables, so you need Bash to provide the details of the variables, but Bash doesn't know about sed replacements, so it doesn't have any way of knowing what variables you need the details of.

The fix, as long as you want to use Bash variables, is to use more Bash: you need to identify the relevant variable-names before you first call sed. The below shows how you can do that.

---

To get the list of all variable-names in your file, you can write something like this:

grep -o '%[a-z_][a-z_]*%' FILE | grep -o '[a-z_][a-z_]*' | sort -u

(The first grep gets all expressions of the form %...%. The second grep filters out the percent-signs; or you can use sed for that, if you prefer. The sort -u eliminates the duplicates, since you only need the list of distinct variable-names.)

Armed with that, you can assemble a sed command that performs all the necessary replacements:

sed_args=()
while read varname ; do
    sed_args+=(-e "s/%$varname%/${!varname}/g")
done < <(grep -o '%[a-z_][a-z_]*%' FILE | grep -o '[a-z_][a-z_]*' | sort -u)
sed "${sed_args[@]}" FILE

(Note the use of ${!varname} to mean "take the value of $varname as a variable-name, and return the value of that variable." This is what §3.5.3 "Shell Parameter Expansion" of the Bash Reference Manual calls "indirect expansion".)

You can wrap this in a function:

function replace_bash_variables () {
    local file="$1"
    local sed_args=()
    local varname
    while read varname ; do
        sed_args+=(-e "s/%$varname%/${!varname}/g")
    done < <(grep -o '%[a-z_][a-z_]*%' "$file" | grep -o '[a-z_][a-z_]*' | sort -u)
    if [[ "${#sed_args[@]}" = 0 ]] ; then
        # if no variables to replace, just cat the file:
        cat -- "$file"
    else
        sed "${sed_args[@]}" -- "$file"
    fi
}

replace_bash_variables OLD_FILE > NEW_FILE

You can also adjust the above to do line-by-line processing, so that it doesn't need to read the file twice. (That gives you more flexibility, since reading the file twice means you have to pass in the actual file, and can't (say) apply this to the output of a pipeline.)