Reloading code on a production ring-clojure server

Question

What's the best way to push new code to a production ring server without restarting the whole JVM?

Currently I use wrap-reload in production, but this doesn't quite work for me because sometimes I want to run commands in the repl (doing database migrations for example) before ring starts handling requests with the new code. Also various blogs and tutorials out there say not to use wrap-reload in production, though I don't understand why not.

I have come up with the following solution, but I confess I don't have a deep understanding of what's going on under the hood. I was wondering if I could get a sanity check by someone who does. Does this technique seem reasonable?

The idea is to have a path (/admin/reload-clj) that causes all the clojure code to be reloaded.

(defonce ^:dynamic *jetty*)
(declare reload-clj)

(defn app [req]
 ...
 (when (= (req :uri) "/admin/reload-clj") (reload-clj req))
 ...)

(defn start-jetty []
 (let [j (run-jetty app {:port (http-port) :join? false :max-threads 16})]
   (dosync (ref-set *jetty* j))
   j))

(defn reload-clj [req]
 (future
    (log/info "Reloading clojure code...")
    (require '(whrusrv admin main utils wdb) :reload-all)
    (.stop @*jetty*)
    (start-jetty)
    (log/info "Clojure reload success!"))
 {:status 200
  :headers {"Content-Type" "text/plain"}
  :body "Reloading..."})

(defn -main [& args]
 (start-jetty))

Answer 1

The code you have will work, though you should be aware that :reload-all only loads a namespace and that namespaces dependencies. It does not recursively load dependencies of those namespaces.

I should add that reloading in this way is strongly not reccomended in a production system.

Newly deployed code might have errors that aren't apparent until the system is restarted (e.g, they depend on a var that is still defined from the running system but whose declaration was removed). The system will work fine but then fail on restart.

Loading code can also have side effects which could screw up your production environment. Although its good style to avoid these, the only way to truly be sure something unexpected won't happen is to do a JVM restart.

The best way to do a zero-downtime deploy on the JVM is a rolling deploy using a load balancer.

Answer 2

I had an intranet web service in common lisp at a previous job, so I don't know if that qualifies as production. Also, being in CL my answer is neither ring nor clojure specific. Still, it's useful for reloading code.

What I did is to start a swank server (part of slime) on the production lisp instance and connect to it even remotely from my desktop. That way I could write new code, rewrite code, debug, reload, etc. Obviously you'll have to be extra carefull in a real production system.

You can do similarly in clojure, you even have alternatives to swank, like nrepl.

You'll have to take security into account as well and allow only local connections, or whatever works for you.