Lexical eval in emacs24

Question

Can anybody explain me how eval works with emacs24? From eval description:

eval is a built-in function in `C source code'.

(eval FORM &optional LEXICAL)

Evaluate FORM and return its value.
If LEXICAL is t, evaluate using lexical scoping.

Does that mean, that something like this should work?

(setq lexical-binding t)
(let ((myvarr 42)) (eval 'myvarr t)) ; (void-variable myvarr)

Update:

(setq lexical-binding nil)
;; => nil
(let ((myvarr 42)) (eval 'myvarr))
;; => 42 (#o52, #x2a, ?*)
(setq lexical-binding t)
;; => t
(let ((myvarr 42)) (eval 'myvarr))
;; Debugger entered--Lisp error: (void-variable myvarr)
;;   eval(myvarr)
;;   (let ((myvarr 42)) (eval (quote myvarr)))
;;   (progn (let ((myvarr 42)) (eval (quote myvarr))))
;;   eval((progn (let ((myvarr 42)) (eval (quote myvarr)))) t)
;;   eval-last-sexp-1((4))
;;   eval-last-sexp((4))
;;   call-interactively(eval-last-sexp nil nil)
;;   call-last-kbd-macro(nil kmacro-loop-setup-function)
;;   kmacro-call-macro(nil nil)
;;   kmacro-end-or-call-macro(nil)
;;   call-interactively(kmacro-end-or-call-macro nil nil)
(ignore-errors (let ((myvarr 42)) (eval 'myvarr)))
;; => nil
(setq lexical-binding nil)
;; => nil
(eval (let ((myvarr 42)) (eval 'myvarr)) t)
;; => 42
(eval '(let ((myvarr 42)) (eval 'myvarr)) t)
;; Debugger entered--Lisp error: (void-variable myvarr)
;;   eval(myvarr)
;;   (let ((myvarr 42)) (eval (quote myvarr)))
;;   eval((let ((myvarr 42)) (eval (quote myvarr))) t)
;;   eval((eval (quote (let ((myvarr 42)) (eval (quote myvarr)))) t) nil)
;;   eval-last-sexp-1((4))
;;   eval-last-sexp((4))
;;   call-interactively(eval-last-sexp nil nil)

Emacs version: GNU Emacs 24.1.1 (i386-mingw-nt6.1.7600) of 2012-06-10 on MARVIN

Answer 1

Since lexical binding breaks much existing elisp code, it is an opt-in feature.

lexical scoping can best be understood with a simple example:

(defun some-func (callback)
 (let ((a 5))
  (funcall callback)))

(let ((a 3))
 (some-func (lambda () a)))

Under most languages, this would return 3 since the a in some-func doesn't seem visible from the bottom form. However, in emacs before 24 or without lexical scope this program returns 5.

This has led to many unexpected surprises and subtle and often hidden bugs between interacting functions; to fix this emacs 24 introduced lexical scoping, but as mentioned previously, is backwards compatible.

The mechanism to opt-in to dynamic scoping is either the file variable lexical-binding (which turns it on per source file) or as the option you see to eval

So if we rewrite the example to use eval:

(eval '(let ((a 3))
        (some-func (lambda () a))) nil) ; => 5

(eval '(let ((a 3))
        (some-func (lambda () a))) t) ; => 3

In your example, what is making the difference is not whether the code inside eval is dynamically scoped, but whether the code surrounding it is. Variable binding is what is affected by lexical scoping, not variable lookup. I'm not completely certain of eval's semantics here, but what seems to be happening (and what makes the most sense) is eval evaluates the expression in an entirely new lexical context. So the outer lexical scope is hidden from the inside of eval, but the dynamic scope is still visible (so the lookup succeeds when the file is dynamically scoped, but not otherwise).

Answer 2

I could not find the formal semantics of the eval function in Emacs 24, but all of your examples make sense when assuming that it works in the same way as that in Common Lisp (as indicated by cobbal). The Common Lisp HyperSpec says:

Syntax:

eval form

Description:

Evaluates form in the current dynamic environment and the null lexical environment.

Let's look at your examples one by one with the description in mind.

(setq lexical-binding t)
(let ((myvarr 42)) (eval 'myvarr t)) ; Lisp error: (void-variable myvarr)

Lexical binding is enabled by setting t to lexical-binding, so myvarr turns to be a lexically bound variable, which is not available inside the eval function as stated above. The eval function's option, t, is irrelevant here.

(setq lexical-binding nil)
(let ((myvarr 42)) (eval 'myvarr)) ; 42

Lexical binding is disabled by setting nil to lexical-binding, so myvarr turns to be a dynamically bound variable, which is available inside the eval function. The eval function's option, implicilty nil, is irrelevant here.

(setq lexical-binding t)
(let ((myvarr 42)) (eval 'myvarr)) ; Lisp error: (void-variable myvarr)

Lexical binding is enabled by setting t to lexical-binding, so myvarr turns to be a lexically bound variable, which is not available inside the eval function. The eval function's option, implicilty nil, is irrelevant here.

(ignore-errors (let ((myvarr 42)) (eval 'myvarr))) ; nil

Ditto.

(setq lexical-binding nil)
(eval (let ((myvarr 42)) (eval 'myvarr)) t) ; 42

Lexical-binding is disabled by setting nil to lexical-binding, so myvar turns to be a dynamically bound variable, which is available inside the inner eval function. Note that the let form, including the inner eval function, is evaluated as argument preparation before the outer eval is called. Neither the outer nor inner eval function's option is relevenat here.

(eval '(let ((myvarr 42)) (eval 'myvarr)) t) ; Lisp error: (void-variable myvarr)

myvarr turns to be a lexically bound variable, which is not available inside the inner eval. Note that, because of ', the let form is evaluated by the outer eval function with lexical binding enabled. The outer eval function's option is relevant here while the inner eval function's is not.

---

Why the null lexical environment?

That is, I think, because that alpha-equivalence would not hold anymore if the current lexical environment were used.

Alpha-equivalence is a formal way to say that the names of function parameters are not important. For example, (lambda (x) x) and (lambda (y) y) are alpha-equivalent, and we have regarded them as the same. Alpha-equivalence allows us to change a function parameter's name as we wish at any time. We take it for granted, and we will be very surprised if it does not hold. See Lambda calculus and Alpha-equivalence for more formal explanation.

But alpha-equivalence turns out to have some problem when a code value involving a free variable (called open code) can be passed around as in Lisp. Let's see the following example:

;;; -*- lexical-binding: t -*-
(lambda (x) (lambda (y) (eval x))) '(1+ y)

If eval evaluated under the current lexical environment, the form would be equivalent to (lambda (y) (1+ y)). Now see the following program:

(lambda (x) (lambda (z) (eval x))) '(1+ y)

This program is different from the previous one only in its parameter's name, z in place of y, so we naturally expect them to behave in the same way. But the latter program evaluates to (lambda (z) (1+ y)), which is definitely different from (lambda (y) (1+ y). Think about what will happen when the resulting function values are applied to the same argument. The point is that the name of a function parameter DOES matter when a code value containing a free variable is allowed.

Here we have two choices in order to preserve alpha-equivalence: we cannot give up alpha-equivalence because it feels so natural and we have been so much accustomed to it. The first option is to have eval evaluated under the null lexical environment as (Common) Lisp does. With this option, the free variable y in (1+ y) does not bind to the formal parameter y in (lambda (y) ...). So those two program behave consistently and alpha-equivalence is preserved well. The other choice is to preclude the problem by not allowing open code value from the beginning. I have heard that it is a method chosen by MetaOCaml

Someone may ask "if so, why the current dynamic environment?". For dynamically bound (a.k.a. special) variables, we already acknowledge well that their names are so important and that, if you change them carelessly, your program will be broken.