Loading external packages in Common Lisp with SLIME on Debian

Question

I'm using SBCL 1.0.56 on Debian squeeze, with cl-swank/slime 1:20120420-2 (Debian version number). These are all the current versions in unstable.

I've been having problems with loading third party CL packages. The documentation for using CL on Debian (and indeed the more general CL usage documentation on Linux) is sketchy, contradictory, and out-of-date, so I'll summarize what I know. Here is where I am at.

Debian installs binary packages (example, cl-split-sequence) in /usr/share/common-lisp/source. In the case of split-sequence, this is /usr/share/common-lisp/source/cl-split-sequence.

The .asd file (here /usr/share/common-lisp/source/cl-split-sequence/split-sequence.asd), which, as I understand it, gives instructions to the CL implementation about version and dependencies, looks like

;;; -*- Lisp -*- mode
(defpackage #:split-sequence-system (:use #:cl #:asdf))
(in-package :split-sequence-system)

(defsystem :split-sequence
    :version "20011114.1"
    :components ((:file "split-sequence")))

Now, when running slime, entering the following two lines at the REPL works without a problem

(require :split-sequence)
(split-sequence:SPLIT-SEQUENCE  #\, "foo,bar")

The (require :split-sequence) invokes (I think) the builtin copy of ASDF inside SBCL, which presumably looks at split-sequence.asd. This is probably SBCL-specific, see Common Lisp in Debian Manual Chapter 3 -Libraries. It is worth noting that this page, which is as useful and detailed as anything I've come across, makes frequent reference to CLC (Common Lisp Controller), but it seems Debian is moving away from this. See Redesign of Common Lisp Controller, which I don't fully understand. At any rate, none of the documented commands for using CLC work for me. However, CLC is still available on Debian. Also, the users mailing list is dead - see The Clc-users Archives

The first time (require :split-sequence) is invoked, it is compiled, and (on my system, possibly Debian-specific) the resulting fasl is placed in

~/.cache/common-lisp/sbcl-1.0.56.0.debian-linux-x86/usr/share/common-lisp/source/cl-split-sequence/split-sequence.fasl

I.e. the file is placed under the cache in a filesystem mirroring the location of the original source. The obvious question is, how does the system know where to look for the package? This is one thing I am not sure about. It looks like the search paths should be given in /etc/common-lisp/source-registry.conf.d, which is part of the ASDF Debian package, but the closest thing is 01-common-lisp-controller.conf, which is just

(:directory  #p"/usr/share/common-lisp/systems/")

Maybe this is hard-wired somewhere, but I'd like to know.

Anyway, once this ASDF file is in the cache, it is not compiled again, but the REPL doesn't see it after slime starts up unless one does require again.

Now, if I put the lines

(require :split-sequence)
(split-sequence:SPLIT-SEQUENCE  #\, "foo,bar")

In a file, say seq.lisp, and load it into the REPL with C-c C-k, I get an error and traceback, starting with

The name "SPLIT-SEQUENCE" does not designate any package.
   [Condition of type SB-KERNEL:SIMPLE-PACKAGE-ERROR]

so I conclude that the package has not been loaded correctly. I tried variations like

(asdf:oos 'asdf:load-op :split-sequence)

and

(asdf:load-system :split-sequence)

but no dice. Weirdly, suppose we just have the line

(require :split-sequence)

in a file - call this require.lisp for clarity. Then loading require.lisp doesn't give an error, and then typing

(split-sequence:SPLIT-SEQUENCE  #\, "foo,bar")

at the REPL works! However, without loading require.lisp, typing the preceding line doesn't work at the REPL.

So, in conclusion, how can one successfully load a package in a script? I'd also be interested in the issue mentioned above, about how ASDF is finding the /usr/share/common-lisp/source/ location, but that is more a side issue.

Answer 1

C-c C-k compiles the source file and then loads the compiled file.

A Lisp source file contains definitions and calls. The file compiler walks through the file and creates code for it. But it does not execute it.

If your file contains a call to REQUIRE, it is not executed during compilation. It will be executed if you later load the compiled file. If the next Lisp form in the file then is using some package which would be available after the call to REQUIRE, it is just not there during compilation, since the REQUIRE has not been executed yet - thus the error during reading while compiling.

There are basically two solutions to that:

• execute all the necessary REQUIRE operations before you compile a particular file using the required functionality.
• execute the REQUIRE statement during compilation. That's where EVAL-WHEN :COMPILE-TOPLEVEL tells the compile to execute the subforms during compilation.

#+:sbcl(foo) means that (foo) is read only when :SBCL is a symbol present in the list CL:*FEATURES*. It is used so that the compiler sees this code only when it is SBCL.

The compilation of Common Lisp code takes more care to prepare, because:

• one can execute Lisp code during file compilation and thus change the behavior of the file compiler.

• packages (which are namespaces for symbols), which are used in the source code, need to be known to the file compiler's reader.

Answer 2

I came across the following post, Zach Beane's Blog - Making a small Common Lisp project, which had a link to http://common-lisp.net/~loliveira/ediware/ediware.lisp

Copying the code at that top of that file works. So running C-c C-k in slime on this file loads split-sequence, and runs the code int (split-sequence:SPLIT-SEQUENCE #\, "foo,bar"))

#+:sbcl
(eval-when (:compile-toplevel :load-toplevel :execute)
  (require :asdf))

(eval-when (:compile-toplevel :load-toplevel :execute)
  (asdf:oos 'asdf:load-op :split-sequence))

(defpackage #:seq
  (:use #:cl #:split-sequence))
(in-package #:seq)

(print (split-sequence:SPLIT-SEQUENCE  #\, "foo,bar"))

However, I've no idea what these magical incantations do. Would anyone care to explain, perhaps in another answer? I'm putting this in an answer, because it is an answer to the question, but it is not complete, because I don't understand why it works. I'll gladly accept an explanation of this script.