What are the limitations of forward declaring in Clojure? Why can't I use comp in this example?

Question

I like my code to have a "top-down" structure, and that means I want to do exactly the opposite from what is natural in Clojure: functions being defined before they are used. This shouldn't be a problem, though, because I could theoretically declare all my functions first, and just go on and enjoy life. But it seems in practice declare cannot solve every single problem, and I would like to understand what is exactly the reason the following code does not work.

I have two functions, and I want to define a third by composing the two. The following three pieces of code accomplish this:

1

(defn f [x] (* x 3))
(defn g [x] (+ x 5))
(defn mycomp [x] (f (g x)))
(println (mycomp 10))

2

(defn f [x] (* x 3))
(defn g [x] (+ x 5))
(def mycomp (comp f g))

3

(declare f g)
(defn mycomp [x] (f (g x)))
(defn f [x] (* x 3))
(defn g [x] (+ x 5))

But what I would really like to write is

(declare f g)
(def mycomp (comp f g))
(defn f [x] (* x 3))
(defn g [x] (+ x 5))

And that gives me

Exception in thread "main" java.lang.IllegalStateException: Attempting to call unbound fn: #'user/g,

That would mean forward declaring works for many situations, but there are still some cases I can't just declare all my functions and write the code in any way and in whatever order I like. What is the reason for this error? What does forward declaring really allows me to do, and what are the situations I must have the function already defined, such as for using comp in this case? How can I tell when the definition is strictly necessary?

Answer 1

You can accomplish your goal if you take advantage of Clojure's (poorly documented) var behavior:

(declare f g)
(def mycomp (comp #'f #'g))
(defn f [x] (* x 3))
(defn g [x] (+ x 5))

(mycomp 10) => 45

Note that the syntax #'f is just shorthand (technically a "reader macro") that translates into (var f). So you could write this directly:

(def mycomp (comp (var f) (var g)))

and get the same result.

Please see this answer for a more detailed answer on the (mostly hidden) interaction between a Clojure symbol, such as f, and the (anonymous) Clojure var that the symbol points to, namely either #'f or (var f). The var, in turn, then points to a value (such as your function (fn [x] (* x 3)).

When you write an expression like (f 10), there is a 2-step indirection at work. First, the symbol f is "evaluated" to find the associated var, then the var is "evaluated" to find the associated function. Most Clojure users are not really aware that this 2-step process exists, and nearly all of the time we can pretend that there is a direct connection between the symbol f and the function value (fn [x] (* x 3)).

The specific reason your original code doesn't work is that

(declare f g)

creates 2 "empty" vars. Just as (def x) creates an association between the symbol x and an empty var, that is what your declare does. Thus, when the comp function tries to extract the values from f and g, there is nothing present: the vars exist but they are empty.

---

P.S.

There is an exception to the above. If you have a let form or similar, there is no var involved:

(let [x 5
      y (* 2 x) ]
  y)  

;=> 10

In the let form, there is no var present. Instead, the compiler makes a direct connection between a symbol and its associated value; i.e. x => 5 and y => 10.

Answer 2

I think Alan's answer addresses your questions very well. Your third example works because you aren't passing the functions as arguments to mycomp. I'd reconsider trying to define things in "reverse" order because it works against the basic language design, requires more code, and might be harder for others to understand.

But... just for laughs and to demonstrate what's possible with Clojure macros, here's an alternative (worse) implementation of comp that works for your preferred syntax, without dealing directly in vars:

(defn- comp-fn-arity [variadic? args f & fs] ;; emits a ([x] (f (g x)) like form
  (let [args-vec (if variadic?
                   (into (vec (butlast args)) ['& (last args)])
                   (apply vector args))
        body (reduce #(list %2 %1)
                     (if variadic?
                       (apply list 'apply (last fs) args)
                       (apply list (last fs) args))
                     (reverse (cons f (butlast fs))))]
    `(~args-vec ~body)))

(defmacro momp
  ([] identity)
  ([f] f)
  ([f & fs]
   (let [num-arities 5
         args-syms (repeatedly num-arities gensym)]
     `(fn ~@(map #(apply comp-fn-arity (= % (dec num-arities)) (take % args-syms) f fs)
                 (range num-arities))))))

This will emit something kinda like comp's implementation:

(macroexpand '(momp f g))
=>
(fn*
 ([] (f (g)))
 ([G__1713] (f (g G__1713)))
 ([G__1713 G__1714] (f (g G__1713 G__1714)))
 ([G__1713 G__1714 G__1715] (f (g G__1713 G__1714 G__1715)))
 ([G__1713 G__1714 G__1715 & G__1716] (f (apply g G__1713 G__1714 G__1715 G__1716))))

This works because your (unbound) functions aren't being passed as values to another function; during compilation the macro expands "in place" as if you'd written the composing function by hand, as in your third example.

(declare f g)
(def mycomp (momp f g))
(defn f [x] (* x 3))
(defn g [x] (+ x 5))
(mycomp 10) ;; => 45
(apply (momp vec reverse list) (range 10)) ;; => [9 8 7 6 5 4 3 2 1 0]

This won't work in some other cases, e.g. ((momp - dec) 1) fails because dec gets inlined and doesn't have a 0-arg arity to match the macro's 0-arg arity. Again, this is just for the sake of example and I wouldn't recommend it.