Why is it possible to use unquote-splicing on a non-list at the end of a quasiquoted list?

Question

The quasiquoted list `(1 ,@2 3) is invalid because 2 is not a list. However, `(1 2 ,@3) is valid and will return a dotted list: (1 2 . 3). I observe this result in Common Lisp and Scheme. Why is it possible to use unquote-splicing for non-lists at the end of a quasiquoted list? Why is the result a dotted list?

Answer 1

The expression `(1 2 ,@3) is not valid in either Scheme or Common Lisp.

---

Scheme

In R6RS Scheme (and similarly for R5RS), the behavior is not specified for operating on a non-list with unquote-splicing. The R6RS Scheme Standard requires (11.17 Quasiquotation):

If an (unquote-splicing <expression> ...) form appears inside a <qq template>, then the <expression>s must evaluate to lists....

---

Common Lisp

The Common Lisp HyperSpec says first that (2.4.6 Backquote):

If a comma is immediately followed by an at-sign, then the form following the at-sign is evaluated to produce a list of objects. These objects are then "spliced" into place in the template.

In the sub-expression ,@3, 3 does not evaluate to a list. This seems a pretty strong argument that the expression is not valid. Even if the 3 were magically placed in a list before splicing, this would not result in a dotted list. The HyperSpec goes on to provide a formal summary of backquote syntax. The part of interest is:

• `(x1 x2 x3 ... xn . atom) may be interpreted to mean

  (append [ x1] [ x2] [ x3] ... [ xn] (quote atom))

  where the brackets are used to indicate a transformation of an xj as follows:

  -- [form] is interpreted as (list `form), which contains a backquoted form that must then be further interpreted.

  -- [,form] is interpreted as (list form).

  -- [,@form] is interpreted as form.

So in Common Lisp the original expression, which is equivalent to `(1 2 ,@3 . nil), can be interpreted as:

(append (list `1) (list `2) 3 (quote nil))

But, this is not a valid call to append, which requires proper lists for all arguments except the last one. So, there seems to be no support for the idea that the original expression was valid.

The fact that it worked for the OP in both Scheme and Common Lisp probably comes down to similar definitions for the backquote macro across different implementations. These definitions all seem to expect that the form following ,@ will evaluate to a list; when that is not the case, the observed behavior (production of a dotted list) can't be relied upon, according to the standard. That said, I tested Chez Scheme, Guile Scheme, MIT Scheme, SBCL, CCL, and CLisp: all of them exhibited the same behavior reported by OP.

---

An Interesting Case

I also tested against an implementation of the backquote macro by Guy Steele and published in CLTL2. This case is more interesting. This backquote implementation in CLTL2 is meant for exploring the behavior of backquote expressions and has an optional code simplification phase. Here $ corresponds to a backquote, and %@ corresponds to ,@. Without code simplification, the result of expanding the original expression is:

CL-USER> (setf *bq-simplify* nil)
NIL
CL-USER> (try '("$(1 2 %@3)"))
`(1 2 ,@3) = (APPEND (LIST '1) (LIST '2) 3 'NIL)

This corresponds to the expression which was expected above from reading the description in the HyperSpec. But note that this expression will not compile:

CL-USER> (append (list 1) (list 2) 3 nil)

The value
  3
is not of type
  LIST
[Condition of type TYPE-ERROR]

But, when code simplification is turned on:

CL-USER> (setf *bq-simplify* t)
T
CL-USER> (try '("$(1 2 %@3)"))
`(1 2 ,@3) = (LIST* '1 '2 3)

This "simplified" expression is valid, and evaluates to a dotted list:

CL-USER> (list* 1 2 3)
(1 2 . 3)

My conclusion is that the expression following ,@ must be a list per the Common Lisp standard, but some common implementations either do some form of code simplification similar to what is shown in CLTL2 or otherwise expand the backquote form in such a way that it appears that a non-list form can follow ,@. Don't rely on this, as it is difficult to say when it won't work.