How to stream program code?

Question

I've been learning Lisp recently (Scheme, Racket and Clojure to various extents) and have read the corresponding literature on famous Playstation developer Naughty Dog.

This Gamasutra article mentions that their streaming engine actually streams in game code in addition to game data. Now, they have a dialect of Lisp called GOAL which they use extensively. I know of Lisp's "Code as Data", although I have yet to grok this concept fully. So, I was wondering if the ability to stream code is a property of their code being Lisp data or compiled machine code, as opposed to being a generic design pattern that can be used in many languages?

Can someone point me to literature on this topic?

Bonus: Can someone point me to literature on long-term-listening-compilers as mentioned in the Wikipedia article?

Answer 1

A 'Listener' in Lisp speak is a REPL (Read Eval Print Loop). Listeners usually provide several services like integrated debugger, command histories, command interpreters, and more. In many cases the Listener runs inside the Lisp one is using. In some cases all code typed to a REPL/Listener is compiled before executing (examples are Clozure CL, SBCL). If that is not the case, the code can be compiled on demand (see the COMPILE or COMPILE-FILE functions in Common Lisp).

Here I would guess that the development environment was implemented in Allegro Common Lisp and the runtime runs on a Playstation. The development environment provides an interface to talk to the runtime and allows to compile code and to download it into the runtime. I would guess that the compiler does not run on the Playstation, but inside the development environment. The Listener then also provides functionality to look at the state of the running application on the Playstation, including the symbol table. So, one connects the development environment to the Playstation's running application and one can interact with that, including changing data and code.

Since Lisps typical can load both source and compiled code at runtime, 'streaming' code is not that difficult. It just means that one loads the code (possibly in the background) while the player moves around in the game. If the code is organized in a certain way, it can also be garbage collected once it is no longer needed. For example if a 'room' in a game is an object and the room points to various functions by name, the functions can be autoloaded on demand if the player is in the room. If the player leaves the room the code can be automatically or semi-automatically unloaded. In a typical Lisp the garbage collector would collect unreferenced code. In a game that might not be a task of the GC, but of some other more 'manual' process. This would be a functionality of the GOAL runtime. Compare that with the stock JVM which could not garbage collect unloaded code (AFAIK).

Answer 2

Their streaming engine more or less replicates what DLL (windows), SO(Linux), or Dylib loading does on more general OSes, and isn't all that interesting outside of the fact that they implemented it themselves. It doesn't have anything to do with the language they use. As far as long term listening compilers go that is the first time I have heard edit and continue ascribed to the compiler it is usually a feature of the debugger.