Lisp unit tests for macros conventions and best practices

Question

I find it hard to reason about macro-expansion and was wondering what the best practices were for testing them.

So if I have a macro, I can perform one level of macro expansion via macroexpand-1.

(defmacro incf-twice (n)
  `(progn
     (incf ,n)
     (incf ,n)))

for example

(macroexpand-1 '(incf-twice n))

evaluates to

(PROGN (INCF N) (INCF N))

It seems simple enough to turn this into a test for the macro.

(equalp (macroexpand-1 '(incf-twice n))
  '(progn (incf n) (incf n)))

Is there an established convention for organizing tests for macros? Also, is there a library for summarizing differences between s-expressions?

Answer 1

Generally testing macros is not one of the strong parts of Lisp and Common Lisp. Common Lisp (and Lisp dialects in general) uses procedural macros. The macros can depend on the runtime context, the compile-time context, the implementation and more. They also can have side effects (like registering things in the compile-time environment, registering things in the development environment and more).

So one might want to test:

• that the right code gets generated
• that the generated code actually does the right thing
• that the generated code actually works in code contexts
• that the macro arguments are actually parsed correctly in case of complex macros. Think loop, defstruct, ... macros.
• that the macro detects wrongly formed argument code. Again, think of macros like loop and defstruct.
• the side effects

From above list on can infer that it is best to minimize all these problem areas when developing a macro. BUT: there are really really complex macros out there. Really scary ones. Especially those who are used to implemented new domain specific languages.

Using something like equalp to compare code works only for relatively simple macros. Macros often introduce new, uninterned and unique symbols. Thus equalp will fail to work with those.

Example: (rotatef a b) looks simple, but the expansion is actually complicated:

CL-USER 28 > (pprint (macroexpand-1 '(rotatef a b)))

(PROGN
  (LET* ()
    (LET ((#:|Store-Var-1234| A))
      (LET* ()
        (LET ((#:|Store-Var-1233| B))
          (PROGN
            (SETQ A #:|Store-Var-1233|)
            (SETQ B #:|Store-Var-1234|))))))
  NIL)

#:|Store-Var-1233| is a symbol, which is uninterned and newly created by the macro.

Another simple macro form with a complex expansion would be (defstruct s b).

Thus one would need a s-expression pattern matcher to compare the expansions. There are a few available and they would be useful here. One needs to make sure in the test patterns that the generated symbols are identical, where needed.

There are also s-expression diff tools. For example diff-sexp.