Exporting anaphoric macros in common lisp packages

Question

i am having problem exporting a macro, it works when in it is declared in the same package, but not when it is imported. I use Emacs, SLIME, Clozure on Windows.

Package file

(defpackage :tokenizer
  (:use :common-lisp)
  (:export :tokenize-with-symbols 
       :current-token 
       :advanze-token 
       :peek-token
       :with-token-and-peek
       :with-token))

(defpackage :csharp-parser
  (:use :common-lisp :tokenizer)
  (:import-from :tokenizer :with-token-and-peek :with-token))

Tokenizer file

(in-package :tokenizer)

(defmacro with-token-and-peek (&body body) 
  `(let ((token (current-token tokenizer))
     (peek (peek-token tokenizer)))
     ,@body))

Parser file

(in-package :csharp-parser)

(defun expression (tokenizer node-stack)
  (with-token-and-peek
   (cond ((is-number? token) (make-value-node "number" token))
         ((is-bool? token) (make-value-node "bool" token))
         ((is-identifier? token peek) (make-identifier-node tokenizer node-stack))
         (t (make-ast-node :identifier "bla")))))

Gives the errors on compile:

csharpParser.lisp:265:3:
  warning: Undeclared free variable TOKENIZER::TOKENIZER (2 references)
           style-warning: Unused lexical variable TOKENIZER::PEEK
           style-warning: Unused lexical variable TOKENIZER::TOKEN
csharpParser.lisp:266:14:
  warning: Undeclared free variable TOKEN
etc etc etc

If i try a macroexpansion in package :csharp-parser

(macroexpand-1 '(with-token-and-peek tok))

(LET ((TOKENIZER::TOKEN (CURRENT-TOKEN TOKENIZER::TOKENIZER))
      (TOKENIZER::PEEK (PEEK-TOKEN TOKENIZER::TOKENIZER)))
  TOK)
T

Now like i said if i move the macros to the parser file, it compiles and works perfectly. But when i try to refactor it to the tokenizer file and export it via the package system it gives these errors, because it seems to internalize the symbol to the calling package. I have tried multiple ways via the colons, but can't get it to work.

If anybody could help me with this i would be very thankful.

Answer 1

The symbols TOKEN and PEEK in the macro are interned in the TOKENIZER package, while the code inside the COND uses symbols interned in the CSHARP-PARSER package. There are two ways around this.

1. Have the expansion use a symbol interned in the package where the code is. This can be done by manually interning a symbol in the current package while expanding the macro. For example:

   (defpackage #:foo
     (:use #:cl)
     (:export #:aif))
   
   (in-package #:foo)
   
   (defmacro aif (test then &optional else)
     (let ((it (intern (symbol-name 'it))))
       `(let ((,it ,test))
          (if ,it ,then ,else))))
   
   (in-package :cl-user)
   (use-package :foo)
   (aif (+ 3 3) it) ;=> 6
   

   Using (intern (symbol-name 'it)) instead of just (intern "IT") is a way of avoiding problems in case the lisp doesn't convert symbols to uppercase.

2. Have the code use the symbol interned in the tokenizer package. This can be done by exporting the symbol.

   (defpackage #:foo
     (:use #:cl)
     (:export #:aif
              #:it))
   
   (in-package #:foo)
   
   (defmacro aif (test then &optional else)
     `(let ((it ,test))
        (if it ,then ,else)))
   
   (in-package :cl-user)
   (use-package :foo)
   (aif (+ 3 3) it) ;=> 6
   

   The drawback is that the user of the macro must import the symbol, so they can't use the package qualified name for the macro.

   (defpackage #:foo
     (:use #:cl)
     (:export #:aif
              #:it))
   
   (in-package #:foo)
   
   (defmacro aif (test then &optional else)
     `(let ((it ,test))
        (if it ,then ,else)))
   
   (in-package :cl-user)
   (foo:aif (+ 3 3) it) ; Fails
   

Answer 2

This happened because the macro with-interned-symbols expanded to code which includes symbols interned in TOKENIZER, while the cond expression has only symbols interned in CSHARP-PARSER. Any symbols (other than keywords or gensyms) that a macro expansion includes should be interned.

The following macro will intern a list of symbols to variables of the same name:

(defmacro with-interned-symbols (symbol-list &body body)
  "Interns a set of symbols in the current package to variables of the same (symbol-name)."
  (let ((symbol-list (mapcar (lambda (s)
                               (list s `(intern (symbol-name ',s))))
                             symbol-list)))
    `(let ,symbol-list ,@body)))

The macro with-token-and-peek can be redefined this way using the above to avoid this mismatch:

(with-interned-symbols (token peek)
  (defmacro with-token-and-peek (&body body) 
    `(let ((,token (current-token tokenizer))
       (,peek (peek-token tokenizer)))
       ,@body)))

Note that while anaphoric macros might be the most obvious special case here, this can happen in any macro that introduces new symbols to the expansion, with the exception of keywords as they are always in the keyword package.