Why prohibit a function object in function position (but allow lambda forms)?

Question

lambda form expression in function position

A lambda expression in function position compiles just fine:

> ((lambda (n) (> n 10)) 42)
T

Closure built from a lambda in function position

On the other hand:

> (defun greater-than (x)
     (lambda (n) (> n x)))
GREATER-THAN

> ((greater-than 10) 42)
Compile-time error:  illegal function call

doesn't work.

(I obviously need to call FUNCALL to make it work: (funcall (greater-than 10) 42) => T.

Why this design?

I understand why a SYMBOL with a function object as value binding must not work, e.g.: (let ((foo (lambda () 42))) (foo)). Separate namespaces and all that.

But why prohibit a function object itself in function position? What was the rationale behind this decision?

Answer 1

What is a lambda form?

A lambda-form in function position compiles just fine:

In Common Lisp wording a lambda expression is not a form. A list with a lambda expression and zero or more arguments is a form, specifically a lambda form. Forms can be evaluated, lambda expressions not. Note: there is also a macro operator lambda, which expands the macro form (lambda ...) into (function (lambda ...)) - which is a valid form, using the special operator function.

(                       ; the whole list is a valid lambda form

 (lambda (n) (> n 10))   ; a lambda expression, not a form

 42)                     ; 42 is a form, too

The lambda form above is mostly similar to:

(let ((n 42))
  (> n 10))

Calling function objects of evaluation results at runtime

But why prohibit a function object itself in function position? What was the rationale behind this decision?

Common Lisp prefers to have known (or unknown) functions in function position.

(sin 3)
((lambda (x) (sin 3)) 3)
(unknown-function 3)

Above all three functions can't be anything else: they can't be numbers, strings or other data objects. There is no way to put another data object there. Operators like DEFUN, FLET and others refuse to define functions, which are of some other data type (numbers, strings, ...).

If we would have a computation in function position we could write:

((if (> foo bar) 42 #'sin) 3)

Depending on the evaluation of (> foo bar) this could be similar to (sin 3) or (42 3), where the latter is not a valid form, since a number is not a function.

Common Lisp requires one to explicitly use one of FUNCALL, APPLY, MULTIPLE-VALUE-CALL to call function objects. This makes it clear in the source code that a function object is being computed/retrieved and called. They also do the necessary checks that the thing passed really is a function or a symbol denoting a function.

Background

For some details see Technical Issues of Separation in Function Cells and Value Cells by Gabriel/Pitman.