Confused about Common LISP `loop for var =` syntax

Question

I checked the post here on how to loop over a file. I checked several places online trying to understand what is going on in that code snipped posted here again:

(defun get-file (filename)
  (with-open-file (stream filename)
    (loop for line = (read-line stream nil)
          while line
          collect line)))

This syntax seems to be so arcane, and the references are not trivial to understand. According to the reference here " the fourth of seven for/as syntaxes", the syntax is:

for var [type-spec] = expr1 [then expr2]

And if there is no expr2, then expr1 is used.

What is confusing me:

I am lost at the lines without knowing what to ask other than how does it work:

      while line
      collect line

I wouldn't know how to substitute that with other code. It does not look like a list at all. My intuition would tell me that it should've looked something like:

(while line (collect line))

Also, isn't that also same as:

while line collect line

And what is the expr1:

(read-line stream nil)
              while line
              collect line)

or

      while line
      collect line

What if I had much more code instead of collect line? Will there be no lists? I don't see the structure.

I know these are more than 1 question, but probably there is something big I am missing, preventing me from asking a good question.

Answer 1

It's a loop. Stuff which runs multiple times.

Basically the loop has two parts:

1. part one defines a variable line and what happens to it: on each loop iteration line is set to the result of evaluating the expression (read-line stream nil). This means that lines get read one by one.
2. the second part are the clauses which are run on each iteration. This part says: as long as the variable line is not nil collect the value of line into a list.

Thus each line is read and as long there is a line, collect it into a list. If the end of the input stream (which is a file stream) is reached, the (read-line stream nil) form returns nil, the while sees that line is nil. nil is false and thus the loop is terminated. Then the so far collected lines will be returned as the result.

The whole loop form returns the collected list of lines.

Here loop is convenient because:

• it conditionally collects a value
• we don't need to give the result list a variable name, because there is only one accumulation going on: the single collect ... clause.
• the result list is returned in the correct order: first collected item is the first element in the returned list

Background

Generally LOOP is a macro, which is based on the similar FOR macro from Interlisp of the early 70s. There the idea was introduced to have program constructs which could be used like normal english: for item in list collect item. This was called conversational Lisp. Similar ideas are for example used in Applescript, a simple scripting language from Apple. This style was never very popular in Lisp (though there are other macros using it), but the loop macro remained, because it turned out to be useful.