Compiling Lisp code with read macros

Question

I'm having a bit of trouble understanding what becomes of read macros when compiling a file of lisp code into a bytecode or raw assembly (or a fasl file for that matter). Or maybe I do understand it but don't know. I'm just really confused.

When you use a read macro, don't you have to have the source available?

If you do, then you have to be executing the source code that makes up the function of the read macro. If you don't, then how can they work when you can do stuff like read-char?

To do that, if you want to have the read macro use aforedefined variables, you have to be executing all the code before it, so this becomes runtime which messes up everything.

If you don't run the code before it, then the stuff defined above it won't be available.

What about functions or compiler macros that define read macros? I would assume they wouldn't work at all unless you require or load a file or something that isn't compiled. But if they were compiled, then they wouldn't be able to use them?

If some of my guesses are correct, then it means there is a large difference in "what data will be available to macros" and "what macros will be available to functions" depending on whether you are compiling an entire file to be run later or interpreting a file one line at a time (that is, reading, compiling, and evaluating one expression after another).

In short, it seems like that to compile one line to a form where it can be executed without further macro processing or whatever, you have to read, compile, and run the previous lines.

Remember again that these questions apply to compiling lisp, not interpreting it where you can run each line as it comes in.

Sorry for my rambling, but I am new to lisp and want to know more how it works.

Answer 1

This is actually an interesting question, and something that a lot of beginning Lisp programmers struggle with. One of the main reasons for this is that everything mostly works "as expected" and you only really start thinking about these things when you start using the more advanced features of Lisp.

The short answer to your question is, yes, in order for the code to be properly compiled, some of the previous code has to have been executed. Note the word some, that's the key. Let's make a small example. Consider a file with the following content:

(print 'a)

(defmacro bar (x) `(print ,x))

(bar 'b)

As you have already figured out, if you run COMPILE-FILE on this file, the resulting .fasl file will simply contain the compiled version of the following code:

(print 'a)
(print 'b)

"But", you might ask, "Why was the DEFMACRO form executed during compilation, but the PRINT form wasn't?". The answer is explained in the Hyperspec section 3.2.3. It contains the following sentence:

Normally, the top level forms appearing in a file compiled with compile-file are evaluated only when the resulting compiled file is loaded, and not when the file is compiled. However, it is typically the case that some forms in the file need to be evaluated at compile time so the remainder of the file can be read and compiled correctly.

There is a form that can be used to control exactly when a form is evaluated. You use EVAL-WHEN for this purpose. In fact, this is exactly how the Lisp compiler implements DEFMACRO itself. You can see how your Lisp implements it by typing the following from the REPL:

(macroexpand '(defmacro bar (x) `(print ,x)))

Obviously different Lisp implementations will implement this differently, but the key important thing is that it wraps the definition in a form: (eval-when (:compile-toplevel :load-toplevel :execute) ...). This tells the compiler that the form should be evaluated both when the file is compiled, but also when the file is loaded. If it didn't do this, you wouldn't be able to use the macro in the same file as it was defined. If the form was only evaluated when the file was compiled, you wouldn't be able to use the macro in a different file after loading it.

Answer 2

Compilation of files is defined in Common Lisp: CLHS Section 3.2.3 File Compilation

While compiling: to make use of a form using a read macro, you have to make that read macro implementation available to the compiler.

Typically such dependencies are handled with a defsystem facility, where the dependencies between various files of a system (something like a project) is described. In order to compile a certain file, another file (preferably the compiled version) must be loaded into the compiling Lisp.

Now, if you want to define the read macro and have forms using its notation in the same file, then you again need to make sure the compiler knows about the read macro and its implementation. The file compiler has compilation environment. It does not load the compiled functions of the same file into this environment by default.

To make the compiler aware of certain code in a file it compiles Common Lisp provides EVAL-WHEN.

Let's look at a read macro example:

(set-syntax-from-char #\] #\)) 

(defun reader-example (stream char)
  (declare (ignore char))
  (let ((class (read stream t nil t))
        (args (read-delimited-list #\] stream t)))
    (apply #'make-instance
           class
           args)))

(set-macro-character #\[ 'reader-example)

(defclass example ()
  ((name :initarg :name)))

(defvar *examples*
  (list [example :name e1]
        [example :name e2]
        [example :name e3]))

If you load the above source, everything is fine. But if we use a file compiler it does not compile without loading it first. The file compiler for example is invoked by calling the function COMPILE-FILE with a pathname.

Now compiling the file:

(set-syntax-from-char #\] #\)) 

Above will not be executed at compile time. The new syntax change will not be available at compile time.

(defun reader-example (stream char)
  (declare (ignore char))
  (let ((class (read stream t nil t))
        (args (read-delimited-list #\] stream t)))
    (apply #'make-instance
           class
           args)))

The above function gets compiled, but not loaded. It implementation is not available to the the compiler in later steps.

(set-macro-character #\[ 'reader-example)

Again, the above form is not executed - just code for it gets generated.

(defclass example ()
  ((name :initarg :name)))

The compiler notes the class, but can't make instances of it later on.

(defvar *examples*
  (list [example :name e1]
        [example :name e2]
        [example :name e3]))

Above code triggers an error, since the read macro is not available at compile time - unless it has been loaded before.

There are now two easy solutions:

• put the implementation of the read macro in a separate file and make sure that it is compiled and loaded before any file which uses the read macro.

• put an EVAL-WHEN around the code which needs to have effect at compile time:

Example:

(EVAL-WHEN (:compile-toplevel :load-toplevel :execute)
  (do-something-also-at-compile-time))

Above will be seen by the compiler and also executed then. Now you have to make sure that the code has everything it calls (all needed definitions) at compile time.

Needless to say: it is good style to reduce such compilation dependencies as much as possible. Typically put the needed functionality in a separate file and make sure that this file gets compiled and loaded into the compiling Lisp before you compile a file which uses it.

Answer 3

Macros (including read macros) are nothing but functions, and they're treated just the same way as all the other functions. You do not need to retain a source code once a function or a macro had been compiled.

Many Lisp implementations would not do any interpretation at all. For example, SBCL by default will only compile, without switching to an interpretation mode even for eval. An important nuance is that Common Lisp compilation is incremental (opposed to the separate compilation, common in many Scheme implementations and languages like C and Java), which allows you to compile a function or a macro and use it straight away, in the same "compilation unit".