Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Preserving leading white space while reading>>writing a file line by line in bash

I am trying to loop through a directory of text files and combine them into one document. This works great, but the text files contain code snippets, and all of my formatting is getting collapsed to the left. All leading whitespace on a line is stripped.

#!/bin/sh
OUTPUT="../best_practices.textile"
FILES="../best-practices/*.textile"
for f in "$FILES"
do
  echo "Processing $f file..."
  echo "">$OUTPUT

  cat $f | while read line; do 
      echo "$line">>$OUTPUT
  done
  echo >>$OUTPUT
  echo >>$OUTPUT
done

I am admittedly a bash noob, but after searching high and low I couldn't find a proper solution. Apparently BASH hates the leading white space in general.

like image 376
Joel Hooks Avatar asked Oct 30 '09 04:10

Joel Hooks


People also ask

How do you read a file line by line in Unix while loop?

Syntax: Read file line by line on a Bash Unix & Linux shell The -r option passed to read command prevents backslash escapes from being interpreted. Add IFS= option before read command to prevent leading/trailing whitespace from being trimmed. while IFS= read -r line; do COMMAND_on $line; done < input. file.

How read a file line by line in Linux?

We can use the read command to read the contents of a file line by line. We use the -r argument to the read command to avoid any backslash-escaped characters. In the following example, we can see that we have an iteration over a file line by line and we store the contents of a single line in the variable “line“.


3 Answers

As others have pointed out, using cat or awk instead of a read-echo loop is a much better way to do this -- avoids the whitespace-trimming problem (and a couple of others you haven't stumbled upon), runs faster, and at least with cat, is simply cleaner code. Nonetheless, I'd like to take a stab at getting the read-echo loop to work right.

First, the whitespace-trimming problem: the read command automatically trims leading and trailing whitespace; this can be fixed by changing its definition of whitespace by setting the IFS variable to blank. Also, read assumes that a backslash at the end of the line means the next line is a continuation, and should be spliced together with this one; to fix this, use its -r (raw) flag. The third problem here is that many implementations of echo interpret escape sequences in the string (e.g. they may turn \n into an actual newline); to fix this, use printf instead. Finally, just as a general scripting hygiene rule, you shouldn't use cat when you don't actually need to; use input redirection instead. With those changes, the inner loop looks like this:

while IFS='' read -r line; do 
  printf "%s\n" "$line">>$OUTPUT
done <$f

...there are also a couple of other problems with the surrounding script: the line that tries to define FILES as the list of available .textile files has quotes around it, meaning it never gets expanded into an actual list of files. The best way to do this is to use an array:

FILES=(../best-practices/*.textile)
...
for f in "${FILES[@]}"

(and all occurrences of $f should be in double-quotes in case any of the filenames have spaces or other funny characters in them -- should really do this with $OUTPUT as well, though since that's defined in the script it's actually safe to leave off.)

Finally, there's a echo "">$OUTPUT near the top of the loop-over-files that's going to erase the output file every time through (i.e. at the end, it only contains the last .textile file); this needs to be moved to before the loop. I'm not sure if the intent here was to put a single blank line at the beginning of the file, or three blank lines between files (and one at the beginning and two at the end), so I'm not sure exactly what the appropriate replacement is. Anyway, here's what I can up with after fixing all of these problems:

#!/bin/sh
OUTPUT="../best_practices.textile"
FILES=(../best-practices/*.textile)

: >"$OUTPUT"
for f in "${FILES[@]}"
do
  echo "Processing $f file..."
  echo >>"$OUTPUT"

  while IFS='' read -r line; do 
    printf "%s\n" "$line">>"$OUTPUT"
  done <"$f"

  echo >>"$OUTPUT"
  echo >>"$OUTPUT"
done
like image 80
Gordon Davisson Avatar answered Oct 04 '22 17:10

Gordon Davisson


Instead of:

cat $f | while read line; do 
    echo "$line">>$OUTPUT
done

Do this:

cat $f >>$OUTPUT

(If there's a reason you need to do things line by line it'd be good to include that in the question.)

like image 27
Laurence Gonsalves Avatar answered Oct 04 '22 19:10

Laurence Gonsalves


that's an overly expensive way of combining files.

cat ../best-practices/*.textile >  ../best_practices.textile

if you want to add a blank( newline) to each file as you concatenate, use awk

awk 'FNR==1{print "">"out.txt"}{print > "out.txt" }' *.textile

OR

awk 'FNR==1{print ""}{print}' file* > out.txt
like image 38
ghostdog74 Avatar answered Oct 04 '22 19:10

ghostdog74