Common Lisp reading from file and storing as list

Question

So doing some common lisp exercises and everything was going well until I encountered this strange behaviour. I read text from file (brown.txt) into a variable corpus, and it's supposed to be stored as a list. However, i suspect that it's not, even though it sometimes works like one, but fails at other times.

Here is the basic read from file -> append for a list -> store list in corpus stuff (split / tokenized on whitespace):

(defun tokenize (string)
  (loop
     for start = 0 then (+ space 1)
     for space = (position #\space string :start start)
     for token = (subseq string start space)
     unless (string= token "") collect token
     until (not space)))

(defparameter *corpus*
   (with-open-file (stream "./brown.txt" :direction :input)
     (loop for line = (read-line stream nil)
           while line
           append (tokenize line))))

And below are 2 expressions that should both work, yet only the latter does (the corpus one). The first one returns NIL.

(loop for token in *corpus* do
     (print token))
*corpus*

I suspect that it has to do with reading from file as a stream object, and that the (append ...) does not create a list from this stream, but instead lazy waits until i want to evaluate it later or sumth, and at that later time it just decides not to work anymore?? (makes little sense to me).

Answer 1

This expression:

(loop for token in *corpus* do
     (print token))

returns NIL because it has no RETURN clause or an accumulation clause (e.g. COLLECT or APPEND). It simply calls PRINT repeatedly, but discards its return value.