Editing programs “while they are running”? How?

Question

This question is a corollary to: Editing programs “while they are running”? Why?

I'm only recently being exposed to the world of Clojure and am fascinated by a few examples I've seen of "live coding". The question linked above discusses the "why."

My question is: How is this live coding technique possible? Is it a characteristic of the clojure language which makes it possible? Or is it just a pattern that they applied which could be applied to any language? I've got a background in python and java. Would it be possible to "live code" in either of these languages like it is possible in clojure?

Answer 1

Some language implementations have that for a long time, especially many Lisp variants and Smalltalk.

Lisp has identifiers as a data structure, called symbols. These symbols can be reassigned and they are looked up at runtime. This principle is called late binding. Symbols name functions and variables.

Additionally Lisp implementations either have at runtime an interpreter or even a compiler. The interface are the functions EVALand COMPILE. Plus there is a function LOAD, which allows loading of source code and compiled code.

Next a language like Common Lisp has an object system which allows changes to the class hierarchy, classes themselves, can add/update/remove methods and propagates these changes to already existing objects. So the object-oriented software and code can be updated itself. With the Meta-object Protocol one can even re-program the object system at runtime.

It's also important that Lisp implementations then can garbage collect removed code. That way the running Lisp will not grow in runtime size just because code is replaced.

Lisp often also has an error system which can recover from errors and allows replacing defective code from within the debugger.

Answer 2

JRebel is one solution for Java. Here's a brief passage from their FAQ:

JRebel integrates with the JVM and application servers mainly on the class loader level. It does not create any new class loaders, instead, it extends the existing ones with the ability to manage reloaded classes.

Answer 3

There are a lot of good answers here, and I'm not sure I can improve on any of them, but I wanted to add some comments around Clojure and Java.

First off, Clojure is written in Java, so you can definitely build a live-coding environment in Java. Just think of Clojure as a specific flavor of live-coding environment.

Basically, live coding in Clojure works via the read function in main.clj and the eval function in core.clj (src/clj/clojure/main.clj and src/clj/clojure/core.clj in the github repository). You read in the forms and pass them to eval, which calls the clojure.lang.Compiler (src/jvm/clojure/lang/Compiler.java in the repo).

Compiler.java converts Clojure forms into JVM bytecode using the ASM library (ASM website here, documentation here). I'm not sure what version of the ASM library is used by Clojure. This bytecode (an array of bytes => byte[] bytecode is the member of the Compiler class that will ultimately hold the bytes generated by the clojure.asm.ClassWriter class via ClassWriter#toByteArray) must then be converted to a class and linked into the running process.

Once you have a representation of a class as a byte array, it's a matter of getting ahold of a java.lang.ClassLoader, calling defineClass to turn those bytes into a Class, and then passing the resulting Class to the resolve method of the ClassLoader to link it to the Java runtime. This is basically what happens when you define a new function, and you can see the internals of the compiler in Compiler$FnExpr which is the inner class that generates the bytecode for function expressions.

There's more going on than that with respect to Clojure, such as the way in which it handles namespace and symbol interning. I'm not completely sure how it gets around the fact that the standard ClassLoader will not replace a linked Class with a new version of that Class, but I suspect it has to do with how classes are named and how symbols are interned. Clojure also defines its own ClassLoader, a certain clojure.lang.DynamicClassLoader, which inherits from java.net.URLClassLoader, so that might have something to do with it; I'm not sure.

In the end, all the pieces are there to do live-coding in Java between ClassLoaders and bytecode generators. You just have to provide a way to input forms into a running instance, eval the forms, and link them up.

Hope this sheds a little more light on the subject.