Clojure macros weirdness when running jars

Question

Below is a simple Clojure app example created with lein new mw:

(ns mw.core
  (:gen-class))

(def fs (atom {}))

(defmacro op []
  (swap! fs assoc :macro-f "somevalue"))

(op)

(defn -main [& args]
  (println @fs))

and in project.clj I have

:profiles {:uberjar {:aot [mw.core]}}
:main mw.core

When run in REPL, evaluating @fs returns {:macro-f somevalue}. But, running an uberjar yields {}. If I change op definition to defn instead of defmacro, then fs has again proper content when run from an uberjar. Why is that?

I vaguely realize that this has something to do with AOT compilation and the fact that macro-expansion occurs before the compilation phase, but clearly my understanding of these things is lacking.

I ran into this issue while trying to deploy an application that uses a very nice mixfix library, in which mixfix operators are defined using a global atom. It took me quite a long time to isolate the issue to the example presented above.

Any help will be greatly appreciated.

Thanks!

Answer 1

The real problem here is that your macro is incorrect. You forgot to add backquote character:

(defmacro op []
  `(swap! fs assoc :macro-f "somevalue"))
; ^ syntax-quote ("backquote")

This operation is called syntax-quote and it's very important here, because macros in clojure modify your code during its compilation.

So, as a result you've got an impure macro, modifying fs atom whenever your code is compiled.

Since your macro doesn't produce any code, (op) call in your example does nothing at all (only it's compilation do). It appear to be working in REPL because compilation and execution is handled by the same clojure instance (see Timur's answer for details).

Answer 2

This is indeed related to the AOT, and the fact that some side effects are expected when a top-level code is executed - here at macro expansion time. The difference between the lein repl (or lein run) and the uberjar is in when exactly this happens.

When lein repl is executed, REPL starts and then loads the mw.core namespace automatically, if it is defined in project.clj, or one does it manually. When namespace is loaded, first the atom is defined, then macro is expanded and this expansion changes the value of the atom. All this happens in same runtime environment (in REPL process), and after the module is loaded, atom has an updated value in this REPL. Executing lein run will do pretty much the same - load namespace and then execute -main function in the same process.

And when lein uberjar is executed - same thing happens and this is the problem now. Compiler, in order to compile the clj file will first load it and evaluate the top level (I learned it myself from this SO answer). So the module is loaded, top level is evaluated, macro is expanded, reference value is changed and then, after compilation completes, the compiler process, the one where reference value just changed, ends. Now, when the uberjar is executed with java -jar this spawns the new process, with a compiled code, where the macro is already expended (so (op) is already "replaced" with the code the op macro generated, which is none in this case). Therefore, atom value is unchanged.

In my opinion, good fix would be to not rely on side effects in a macro.

If stick to the macro anyway, the way to make this idea work is to skip the AOT for the module where macro expansion happens and load it lazily from the main module (again, same solution as in the other SO answer I mentioned). For example:

project.clj:

; ...
:profiles {:uberjar {:aot [mw.main]}}) ; note, no `mw.core` here
; ...

main.clj:

(ns mw.main
  (:gen-class))

(defn get-fs []
  (require 'mw.core)
  @(resolve 'mw.core/fs))

(defn -main [& args]
  (println @(get-fs)))

core.clj:

(ns mw.core
  (:gen-class))

(def fs (atom {}))

(defmacro op []
  (swap! fs assoc :macro-f "somevalue"))

(op)

I'm not sure myself, however, if this solution is stable enough and that there are no edge cases. It does work though on this simple example.