sed | replace value with environment variable containing double quotes

Question

my environment variable contained double-quotes as follows:

$echo $CONNECT_SASL_JAAS_CONFIG
org.apache.kafka.common.security.plain.PlainLoginModule required username="USER" password="XXXX/YYY";

output from "export" command:

declare -x CONNECT_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"USER\" password=\"XXXX/YYY\";

my template file content is as follows (props.txt):

sasl.jaas.config = JAAS_CONF

I try to replace the JAAS_CONF with the env variable using following command :

 sed  -e  "s/JAAS_CONF/$CONNECT_SASL_JAAS_CONFIG/g;" props.txt

This gives the following error :

/tmp# sed -e "s/JAAS_CONF/$CONNECT_CONSUMER_SASL_JAAS_CONFIG/g;" props.txt sed: -e expression #1, char 168: unknown option to `s'

Any tips on how to fix this ?

UPDATE:

Note:

• "password" contains "/" character
• There is a new-line char at the end of CONNECT_CONSUMER_SASL_JAAS_CONFIG
• I need to need escape the double-doubles back again.

So finally my solution was:

  export TMP_JAAS=${SASL_JAAS_CONFIG//$'\"'/'\\"'}
  export PARSED_JAAS=${TMP_JAAS//$'\n'/\ }
  sed  -e  "s|JAAS_CONF|${PARSED_JAAS//$'\n'/\ }|g" file.cong > /tmp/app.conf

Answer 1

Using envsubst:

Assume you have a string STRING which you want to replace with the variable VARIABLE_STRING. Then I would suggest the following approach:

1. substitute all variables of interest with a proper shell variable name (include the $)
2. export all variables of interest
3. Use envsubst to perform the substitution

Here is an example:

$ export VARIABLE_STRING='foo "bar" qux'
$ echo "This is the string 'STRING'" | sed 's/\bSTRING\b/$VARIABLE_STRING/g' | envsubst '$VARIABLE_STRING'
This is the string 'foo "bar" qux'

You can adapt this to your liking. Please read [U&L] Replacing only specific variables with envsubst for further information.

Using GNU awk:

If you cannot make use of envsubst, you can write something in awk. GNU awk can access the environment using the ENVIRON array which is a map from variable name to variable value. Example:

awk 'BEGIN{print ENVIRON["HOME"]}' 

This will print the content of $HOME

So now you could something like this:

1. Create a map from string to replace into environment
2. Loop over the names:

This would look something like this:

awk 'BEGIN{ map["NAME1"]="ENVIRONMENT_NAME1"
            map["NAME2"]="ENVIRONMENT_NAME2"
            map["NAME3"]="ENVIRONMENT_NAME3" }
     { for(i in map) gsub("/<" i "/>",ENVIRON[map[i]]) }
     { print }' file

Similar to the sed in the first solution, I've added a way to ensure that the string NAME1FOO is not being replaced by ENVIRONMENT_NAME1FOO. This by adding word_boundaries /< and /> in the regex.

**Note: this method might create loops. Imagine you have an environment variable:

ENVIRONMENT_NAME1="NAME2"

Then the string NAME1 might be replaced with the content of ENVIRONMENT_NAME2. I say might because everything depends on the order of transversal in the for loop which is awk dependent.