Bash escapes tilde and wildcards but not space

Question

(EDIT in solution with sed)

I have a list of filenames and directories, including tilde and wildcards. E.g.:

~/docs/file.*    
~/my docs/*.txt
...

I read the rows and pass them to a command (e.g. rsync):

while read ROW
do
   rsync $ROW /my/dest/
done < list.txt

The problem is handling with spaces in filenames. If I put $ROW in double quotes like this

rsync "$ROW" /my/dest/

of course bash doesn't escape the wildcards nor the tilde. But if I don't use quotes, the space breaks the row.

One possible solution is change IFS (carefully: the script is more complicated than the one I reported). Another solution (thanks for the fix to Patrick Echterbruch) is to pre-escape spaces. However, the following code doesn't work for me:

while read ROW
do
    export NEWROW=$(echo $ROW | sed -e 's/ /\\ /g') 
    echo "Value: $NEWROW"
    ls -1d $NEWROW
done < list.txt

Please note no quotes are passed to ls. The file "~/a b c/test.txt" exists, but I get:

Value: ~/saver/a\ b\ c/*.txt
ls: impossibile accedere a ~/saver/a\: Nessun file o directory
ls: impossibile accedere a b\: Nessun file o directory
ls: impossibile accedere a c/*.txt: Nessun file o directory

Seems like NEWROW is passed to ls as a string, so wildcards are not expanded but space keeps breaking the filename, though escaped.

Any tip? Thanks.

Answer 1

As far as I can see, the sed script you posted is not going to have any effect - the "search" and "replace" parts are the same.

What you could do could do:

ROW=$(echo $ROW | sed -e "s/ /\\\\ /g")

or (better):

ROW=$(echo $ROW | sed -e 's/ /\\ /g')

Latter example uses single quotes, so that the shell doesn't treat the backslash specially.

Concerning the second problem that arose: The special characters (like ~) are expanded by bash when they are encountered in parameters to a program. They will not be expanded when storing them in a variable, nor will bash inspect and expand the contents of a variable when passing it as parameter to a command.

This is why rsync $NEWROW ... does not work - rsync receives the unmodified content of $NEWROW as first parameter and would then have to do shell-like expansion on it's own.

The only workaround I known of is to run the command in a subshell, ie:

ROW="~/a b c"
$NEWROW=$(echo $ROW | sed -e 's/ /\\ /g')
bash -c "ls -ld $f"

This will cause the currently running bash to extract the variables content (i.e. ~/a\ b\ c and pass it to the second shell it is about to invoke. This "inner" bash will receive the command along with the parameters and will then of course to parameter expansion.

Sorry for missing this point in the beginning, I focused only on the regex part of the question.

Found another possible solution: After converting the path using sed (e.g. NEWROW=$(echo $ROW | sed -e 's/ /\\ /g') try:

eval ls -ld $NEWROW

or eval rsync $NEWROW /other/path

This should work as well, without the performance impact of starting subshells.

//EDIT: Added missing g to the sed script

//EDIT: Added workaround for the pathname expansion problem

//EDIT: Addes eval solution