Why can't you use cat to read a file line by line where each line has delimiters

Question

I have a text file that contains something like this:

abc 123, comma
the quick brown fox
jumped over the lazy dog
comma, comma

I wrote a script

for i in `cat file`
do
   echo $i
done

For some reason, the output of the script doesn't output the file line by line but breaks it off at the commas, as well as the newline. Why is cat or "for blah in cat xyz" doing this and how can I make it NOT do this? I know I can use a

while read line
do
   blah balh blah
done < file

but I want to know why cat or the "for blah in" is doing this to further my understanding of unix commands. Cat's man page didn't help me and looking at for or looping in the bash manual didn't yield any answers (http://www.gnu.org/software/bash/manual/bashref.html). Thanks in advance for your help.

Answer 1

The problem is not in cat, nor in the for loop per se; it is in the use of back quotes. When you write either:

for i in `cat file`

or (better):

for i in $(cat file)

or (in bash):

for i in $(<file)

the shell executes the command and captures the output as a string, separating the words at the characters in $IFS. If you want lines input to $i, you either have to fiddle with IFS or use the while loop. The while loop is better if there's any danger that the files processed will be large; it doesn't have to read the whole file into memory all at once, unlike the versions using $(...).

IFS='
'
for i in $(<file)
do echo "$i"
done

The quotes around the "$i" are generally a good idea. In this context, with the modified $IFS, it actually isn't critical, but good habits are good habits even so. It matters in the following script:

old="$IFS"
IFS='
'
for i in $(<file)
do
   (
   IFS="$old"
   echo "$i"
   )
done

when the data file contains multiple spaces between words:

$ cat file
abc                  123,         comma
the   quick   brown   fox
jumped   over   the   lazy   dog
comma,   comma
$ 

Output:

$ sh bq.sh
abc                  123,         comma
the   quick   brown   fox
jumped   over   the   lazy   dog
comma,   comma
$

Without the double quotes:

$ cat bq.sh
old="$IFS"
IFS='
'
for i in $(<file)
do
   (
   IFS="$old"
   echo $i
   )
done
$ sh bq.sh
abc 123, comma
the quick brown fox
jumped over the lazy dog
comma, comma
$

Answer 2

You can use IFS variable to specific you want a newline as the field separator:

IFS=$'\n'
for i in `cat file`
do
   echo $i
done