Compile-file with in-memory data instead of real file in Common Lisp

Question

Is there a way to achieve the effects of compile-file but not with a regular file on disk, but with a stream or just an in-memory string? (I.e. if I don't have a file and don't want to create temporary files out of this in-memory data)

EDIT

I'm thinking of the following use case: loading code from some other places, than the file system. For example from archives (similar to Java's jars or Python zip handling capabilities) or from the network. Maybe there may be alternative approaches to this, than just bending compile-file machinery.

Answer 1

The first comment of Rainer Joswig nails it:

Not in standard Common Lisp.

You can search in each implementation for the calls made from compile-file to see if the underlying implementation takes a stream.

If you want to achieve this by yourself, you'll be implementing a quite inefficient version of compile-file for input and output streams that generates loadable forms suitable for load or your own version of it.

Wrapping the compilation with with-compilation-unit, you'll have to take special care in macroexpanding top-level forms, make sure that progn forms are treated like top-level forms, that locally, macrolet and symbol-macrolet forms establish bindings and process their forms as top-level with a non-null lexical enviroment (the manipulation of which is not portable either), and process eval-when forms, where some forms may be evaluated, processed or both. Then, generate externalizable forms, having care about what can be coalesced and what cannot (e.g. symbols and packages), calling make-load-form for standard-object, structure-object and condition instances, and taking special care for symbols and packages, such as checking if the stream to compile has a first non-atomic in-package form or not as to be able to signal an error if load is called with a different value of *package*.

And this are just the topmost steps for minimal compilation, more or less, without mentioning when does compilation actually happen (compile-time or load-time), load-time-value forms and many details. Some of the steps require code-walking.

My advice: just extract and save whatever source code you have from compressed data and sockets, then call compile-file followed by load, if you want to preserve compilation and loading semantics. For simpler things, you may do something like this:

(let* ((*package* *package*)
       (*readtable* *readtable*)
       (eof (copy-symbol 'eof))
       (form nil))
  (loop
    (setf form (read stream nil eof))
    (when (eq form eof)
      (return))
    (funcall (compile nil `(lambda () ,form)))))

If you don't care much about portability, make your implementation be able to handle compressed files by, for instance, supplying a decompressing external format to compile-file, and make your implementation be able to open URL's (URI's, IRI's, whatever you care about) instead of just pathnames. AFAIK, ABCL can open URL's for input. Yet, what would be the output file of compile-file when called with such an external resource? Maybe make your own compile-and-load which states the temporary file to use to both compile-file's output-file key argument and to load.