lisp: when to use a function vs. a macro

Question

In my ongoing quest to learn lisp, I'm running into a conceptual problem. It's somewhat akin to the question here, but maybe it's thematically appropriate to lisp that my question is a level of abstraction up.

As a rule, when should you create a macro vs. a function? It seems to me, maybe naively, that there would be very few cases where you must create a macro instead of a function, and that in most remainder cases, a function would generally suffice. Of these remainder cases, it seems like the main additional value of a macro would be in clarity of syntax. And if that's the case, then it seems like not just the decision to opt for macro use but also the design of their structures might be fundamentally idiosyncratic to the individual programmer.

Is this wrong? Is there a general case outlining when to use macros over functions? Am I right that the cases where a macro is required by the language are generally few? And lastly, is there a general syntactic form that's expected of macros, or are they generally used as shorthands by programmers?

Answer 1

Yes, the first rule is: don't use a macro where a function will do.

There are a few things you can't do with functions, for example conditional evaluation of code. Others become quite unwieldy.

In general I am aware of three recurring use cases for macros (which doesn't mean that there aren't any others):

• Defining forms (e. g. defun, defmacro, define-frobble-twiddle)

  These often have to take some code snippet, wrap it (e. g. in a lamdba form), and register it somewhere, maybe even multiple places. The users (programmers) should only concern themselves with the code snippet. This is thus mostly about removing boilerplate. Additionally, the macro can process the body, e. g. registering docstrings, handle declarations etc.

  Example: Imagine that you are writing a sort of event mini-framework. Your event handlers are pure functions that take some input and produce an effect declaration (think re-frame from the Clojure world). You want these functions to be normal named functions so that you can just test them with the usual testing frameworks, but also register them in a lookup table for your event loop mechanism. You'd maybe want to have something like a define-handler macro:

  (defvar *handlers* (make-hash-table)) ; internal for the framework
  
  (defmacro define-handler (&whole whole name lambda-list &body body)
    `(progn (defun ,@(rest whole))
            (setf (gethash ,name *handlers*)
                  (lambda ,lambda-list ,@body))))  ; could also be #',name
  

• Control constructs (e. g. case, cond, switch, some->)

  These use conditional evaluation and convenient re-arrangement of the expression.

• With- style wrappers

  This is an idiom to provide unwind-protect functionality to some arbitrary resource. The difference to a general with construct (as in Clojure) is that the resource type can be anything, you don't have to reify it with something like a Closable interface.

  Example:

  (defmacro with-foo-bar-0 (&body body)
    (let ((foo-bar (gensym "FOO-BAR")))
      `(let (,foo-bar))
         (shiftf ,foo-bar (aref (gethash :foo *buzz*) 0) 0)
         (unwind-protect (progn ,@body)
           (setf (aref (gethash :foo *buzz*) 0) ,foo-bar)))))
  

  This sets something inside a nested data structure to 0, and ensures that it is reset to the value it had before on any, even non-local, exit.

Answer 2

I found a detailed answer, from Paul Graham's On Lisp, bold emphases added:

---

Macros can do two things that functions can’t: they can control (or prevent) the evaluation of their arguments, and they are expanded right into the calling context. Any application which requires macros requires, in the end, one or both of these properties.

...

Macros use this control in four major ways:

1. Transformation. The Common Lisp setf macro is one of a class of macros which pick apart their arguments before evaluation. A built-in access function will often have a converse whose purpose is to set what the access function retrieves. The converse of car is rplaca, of cdr, rplacd, and so on. With setf we can use calls to such access functions as if they were variables to be set, as in (setf (car x) ’a), which could expand into (progn (rplaca x ’a) ’a). To perform this trick, setf has to look inside its first argument. To know that the case above requires rplaca, setf must be able to see that the first argument is an expression beginning with car. Thus setf, and any other operator which transforms its arguments, must be written as a macro.

2. Binding. Lexical variables must appear directly in the source code. The first argument to setq is not evaluated, for example, so anything built on setq must be a macro which expands into a setq, rather than a function which calls it. Likewise for operators like let, whose arguments are to appear as parameters in a lambda expression, for macros like do which expand into lets, and so on. Any new operator which is to alter the lexical bindings of its arguments must be written as a macro.

3. Conditional evaluation. All the arguments to a function are evaluated. In constructs like when, we want some arguments to be evaluated only under certain conditions. Such flexibility is only possible with macros.

4. Multiple evaluation. Not only are the arguments to a function all evaluated, they are all evaluated exactly once. We need a macro to define a construct like do, where certain arguments are to be evaluated repeatedly.

There are also several ways to take advantage of the inline expansion of macros. It’s important to emphasize that the expansions thus appear in the lexical context of the macro call, since two of the three uses for macros depend on that fact. They are:

5. Using the calling environment. A macro can generate an expansion containing a variable whose binding comes from the context of the macro call. The behavior of the following macro: (defmacro foo (x) ‘(+ ,x y)) depends on the binding of y where foo is called. This kind of lexical intercourse is usually viewed more as a source of contagion than a source of pleasure. Usually it would be bad style to write such a macro. The ideal of functional programming applies as well to macros: the preferred way to communicate with a macro is through its parameters. Indeed, it is so rarely necessary to use the calling environment that most of the time it happens, it happens by mistake...

6. Wrapping a new environment. A macro can also cause its arguments to be evaluated in a new lexical environment. The classic example is let, which could be implemented as a macro on lambda. Within the body of an expression like (let ((y 2)) (+ x y)), y will refer to a new variable.

7. Saving function calls. The third consequence of the inline insertion of macro expansions is that in compiled code there is no overhead associated with a macro call. By runtime, the macro call has been replaced by its expansion. (The same is true in principle of functions declared inline.)

...

What about those operators which could be written either way [i.e. as a function or a macro]?... Here are several points to consider when we face such choices:

THE PROS

1. Computation at compile-time. A macro call involves computation at two times: when the macro is expanded, and when the expansion is evaluated. All the macro expansion in a Lisp program is done when the program is compiled, and every bit of computation which can be done at compile-time is one bit that won’t slow the program down when it’s running. If an operator could be written to do some of its work in the macro expansion stage, it will be more efficient to make it a macro, because whatever work a smart compiler can’t do itself, a function has to do at runtime. Chapter 13 describes macros like avg which do some of their work during the expansion phase.

2. Integration with Lisp. Sometimes, using macros instead of functions will make a program more closely integrated with Lisp. Instead of writing a program to solve a certain problem, you may be able to use macros to transform the problem into one that Lisp already knows how to solve. This approach, when possible, will usually make programs both smaller and more efficient: smaller because Lisp is doing some of your work for you, and more efficient because production Lisp systems generally have had more of the fat sweated out of them than user programs. This advantage appears mostly in embedded languages, which are described starting in Chapter 19.

3. Saving function calls. A macro call is expanded right into the code where it appears. So if you write some frequently used piece of code as a macro, you can save a function call every time it’s used. In earlier dialects of Lisp, programmers took advantage of this property of macros to save function calls at runtime. In Common Lisp, this job is supposed to be taken over by functions declared inline. By declaring a function to be inline, you ask for it to be compiled right into the calling code, just like a macro. However, there is a gap between theory and practice here; CLTL2 (p. 229) says that “a compiler is free to ignore this declaration,” and some Common Lisp compilers do. It may still be reasonable to use macros to save function calls, if you are compelled to use such a compiler...

THE CONS

4. Functions are data, while macros are more like instructions to the compiler. Functions can be passed as arguments (e.g. to apply), returned by functions, or stored in data structures. None of these things are possible with macros. In some cases, you can get what you want by enclosing the macro call within a lambda-expression. This works, for example, if you want to apply or funcall certain macros:> (funcall #’(lambda (x y) (avg x y)) 1 3) --> 2. However, this is an inconvenience. It doesn’t always work, either: even if, like avg, the macro has an &rest parameter, there is no way to pass it a varying number of arguments.

5. Clarity of source code. Macro definitions can be harder to read than the equivalent function definitions. So if writing something as a macro would only make a program marginally better, it might be better to use a function instead.

6. Clarity at runtime. Macros are sometimes harder to debug than functions. If you get a runtime error in code which contains a lot of macro calls, the code you see in the backtrace could consist of the expansions of all those macro calls, and may bear little resemblance to the code you originally wrote. And because macros disappear when expanded, they are not accountable at runtime. You can’t usually use trace to see how a macro is being called. If it worked at all, trace would show you the call to the macro’s expander function, not the macro call itself.

7. Recursion. Using recursion in macros is not so simple as it is in functions. Although the expansion function of a macro may be recursive, the expansion itself may not be. Section 10.4 deals with the subject of recursion in macros...

Having considered what can be done with macros, the next question to ask is: in what sorts of applications can we use them? The closest thing to a general description of macro use would be to say that they are used mainly for syntactic transformations. This is not to suggest that the scope for macros is restricted. Since Lisp programs are made from lists, which are Lisp data structures, “syntactic transformation” can go a long way indeed...

Macro applications form a continuum between small general-purpose macros like while, and the large, special-purpose macros defined in the later chapters. On one end are the utilities, the macros resembling those that every Lisp has built-in. They are usually small, general, and written in isolation. However, you can write utilities for specific classes of programs too, and when you have a collection of macros for use in, say, graphics programs, they begin to look like a programming language for graphics. At the far end of the continuum, macros allow you to write whole programs in a language distinctly different from Lisp. Macros used in this way are said to implement embedded languages.