Are Lisp source code files themselves lists?

Question

No matter the Lisp dialect, it looks like every source code file containing Lisp functions isn't itself a list (the first time I was "surprised" by this was when working on my Emacs .el files).

I've got a few questions but they're all related to the same "issue" and it's probably just me misunderstanding a few things.

Is there a reason why source code files for the various Lisp dialects seems to be a bunch of "disorganized" functions like this:

(function1 ...)
(function2 ...)
(function3 ...)

Instead of a "Lisp list" of functions, maybe like this:

(
  '(function1 ...)
  '(function2 ...)
  '(function3 ...)
)

I'm a bit surprised in this whole "code is data, data is code" thing to see that source code file themselves apparently aren't neat lists... Or are they!?

Are the source code files something you're supposed to "manipulate" or not?

What if I wanted to, say, convert one of my .clj (Clojure) source file to some CSS+HTML webpage, isn't it a "problem" that the source code file apparently isn't itself a list?

I'm beginning with Lisp so I don't know if my question makes sense or not and any explanation would be welcome.

Answer 1

In Common Lisp a source file contains lisp forms and comments. Lisp forms are either data or Lisp code. Typical operations on a source file are done by the functions LOAD and COMPILE-FILE.

LOAD would read forms from a file and execute them one by one.

COMPILE-FILE is much more complex. It typically reads forms and compiles them to some other representation (machine code, byte code, C code, ...). It does not execute the code.

What would it help you if the file contain one list of forms instead of just multiple forms below each other?

• you would have one level of added parentheses
• you would have to read the whole list before you can do anything with it (or alternatively you need a different reader mechanism)
• adding forms to the end of a file by a program would be a pain
• you can't add something into the file which changes the reader interpretation of the rest of the file
• files can't be infinitively long for LOAD

Now for a example a compiler would read lisp forms from a file stream and compile them piece by piece.

If you want all forms you can do

CL-USER 170 > (defun read-forms (file)
               (with-open-file (stream file)
                 (loop for form = (read stream nil nil)
                       while form
                       collect form)))
READ-FORMS

CL-USER 171 > (read-forms (capi:prompt-for-file "source file"))
((DEFPARAMETER *UNITS-TO-SHOW* 4.1)
 (DEFPARAMETER *TEXT-WIDTH-IN-PICAS* 28.0)
 (DEFPARAMETER *DEVICE-PIXELS-PER-INCH* 300)
 (DEFPARAMETER *PIXELS-PER-UNIT* (* (/ (/ *TEXT-WIDTH-IN-PICAS* 6)
                                       (* *UNITS-TO-SHOW* 2))
                                    *DEVICE-PIXELS-PER-INCH*))
...

If you want to put parentheses around everything use PROGN:

 (progn
   'form-1
   (defun function-defintion-form () )
   42)

PROGN preserves also the 'top-level-ness' of its sub forms.

Side note: alternatives to this have been explored in Lisp for decades. The most prominent example is the now defunct Interlisp-D from Xerox. Interlisp-D was developed in parallel to Smalltalk by Xerox PARC. Interlisp-D originally used an structure editor to edit Lisp data and the source code was edited as such Lisp data. The development environment was based on this idea. But in the long run the 'source as text' won. Still you can emulate some of that in many current Lisp environments. For example many Lisp systems allow to write an 'image' of the current execution memory - this image includes all data and all code (also the compiled code). So you can work on this data/code and save an image from time to time.

Answer 2

In Lisp there are two levels of source code, or there is no source code at all depending on how you define source code.

The two levels are present because two separate conceptual steps are performed (normally) by a Lisp interpreter/compiler.

First step: "reading"

In this step the source code is a sequence of characters, for example coming from a file. Here the parenthesis, quoted strings, numbers, symbols, quote signs and even part of quasiquoting syntax is processed and transformed into Lisp data structures. At this level the syntax rules are about parenthesis, digits, pipes, quotes, semicolons, sharp signs, commas, at-signs and so on.

Second step: "compiling"/"interpreting"

In this step the input are Lisp data structures and the output is either machine code, byte code or possibly the source is directly executed by an interpreter. At this level the syntax is about the meaning of special forms... e.g. (if ...), (labels ...), (symbol-macrolet ...) and so on. The structure is uniform in Lisp code (just lists and atoms) but the semantic isn't (if forms look like function calls, but they are not).

So in this view the question to your answer is yes and no. No for step 1, yes for step 2. If you consider only files then the answer is no... files contain characters, not lists. Those characters can be transformed by a reader into lists.

Lisp has no syntax

Why then someone says that Lisp has no syntax when in fact has two different syntax levels? The reason is that both of these levels are under the control of the programmer.

You can customize level 1 by defining reader macros, and you can customize level 2 by defining macros. So Lisp has no fixed syntax, and therefore a source file can begin with a "lispy" look and can end looking exactly like Python code.

A source file can contain anything (from a certain point on) because the initial forms could define some new reading rules that will change the meaning of following characters.

Normally Lisp programmers don't do crazy things with the reading level so most Lisp source code files look just like sequences of Lisp forms and they remain "lispy".

But this is not an hard constraint... for example I was not joking about Lisp syntax morphing into Python: someone did exactly that.