How do I iterate through a directory in Common Lisp?

Question

I'm using OpenMCL on Darwin, and I'd like to do something like:

(loop for f in (directory "somedir")   collect (some-per-file-processing f)) 

But I can't get directory to return anything other than NIL, and I can't seem to find any good explanation online (other than "its different for each system").

Any pointers?

Answer 1

There are basically two ways to specify pathnames:

• using strings

Strings are obviously depending on the platform: Unix syntax vs. Windows syntax for example.

"/Users/foo/bar.text"  is a valid pathname "/Users/foo/*/foo.*"   is a valid pathname with two wildcards 

You can create a pathname object from a string:

? (pathname "/Users/bar/foo.text") #P"/Users/bar/foo.text" 

The #p above assures that a pathname object (and not a string) is created, when you read it back.

? #P"/Users/bar/foo.text" #P"/Users/bar/foo.text" 

So, internally Common Lisp works with pathname objects, but it lets you use normal strings and makes pathname objects from them if needed.

When Common Lisp sees a pathname that has not all components specified (for example the directory is missing), then it fills in the components from the pathname object that is the value of the variabel *DEFAULT-PATHNAME-DEFAULTS* .

With the function DESCRIBE you can look at the components of a pathname (here Clozure CL):

? (describe (pathname "/Users/bar/*.text")) #P"/Users/bar/*.text" Type: PATHNAME Class: #<BUILT-IN-CLASS PATHNAME> TYPE: (PATHNAME . #<CCL::CLASS-WRAPPER PATHNAME #x3000401D03BD>) %PATHNAME-DIRECTORY: (:ABSOLUTE "Users" "bar") %PATHNAME-NAME: :WILD %PATHNAME-TYPE: "text" %PHYSICAL-PATHNAME-VERSION: :NEWEST %PHYSICAL-PATHNAME-DEVICE: NIL 

• using the Lisp functions creating pathname objects

MAKE-PATHNAME is the function and it takes a few keyword arguments to specify the components.

Sometimes it is also useful to create a new pathname based on an existing one:

(make-pathname :name "foo" :defaults (pathname "/Users/bar/baz.text")) 

If you use DIRECTORY it is useful to use a pathname with wildcards. DIRECTORY then will return a list of matching pathnames. The name 'DIRECTORY' is slightly misleading, since DIRECTORY does not list the contents of a directory, but lists the matching pathnames for (usually) a pathname with wildcards. The wildcards can match a sequences of characters in components like /foo/s*c/list*.l*". There is also the wild card ** , which is used to match parts of a directory hierachy like /foo/**/test.lisp , which matches all files test.lisp under the directory foo and its subdirectories.

(directory "/Users/foo/Lisp/**/*.lisp") 

Above should return a list of all 'lisp' files in '/Users/foo/Lisp/' and all its subdirectories.

To return the .c files in a single directory use:

(directory "/Users/foo/c/src/*.c") 

Note that DIRECTORY returns a list of pathname objects (not a list of strings).

? (directory (make-pathname                :name "md5"                :type :wild                :directory '(:absolute "Lisp" "cl-http" "cl-http-342" "server"))) (#P"/Lisp/cl-http/cl-http-342/server/md5.lisp"  #P"/Lisp/cl-http/cl-http-342/server/md5.xfasl") 

Above uses a pathname object that is created by MAKE-PATHNAME. It returns all the files that match /Lisp/cl-http/cl-http-342/server/md5.* .

This is the same as:

(directory "/Lisp/cl-http/cl-http-342/server/md5.*") 

which is shorter, but depends on the Unix pathname syntax.

Answer 2

Does your pathname specification contain a wildcard? Common Lisp's pathname stuff is somewhat hard to grasp at first - at least for me it was... As the CLHS states on the directory function:

If the pathspec is not wild, the resulting list will contain either zero or one elements.

In order to have your pathname include a wildcard, you might try the make-pathname function, like

(directory (make-pathname :directory '(:absolute "srv" "hunchentoot") :name :wild :type "lisp")) 

Or even

(directory (make-pathname :directory '(:absolute "srv" "hunchentoot") :name :wild :type :wild)) 

I found the CL-FAD library a great help for dealing with pathnames and the file system. In particular, its list-directory function might be easier to use than the plain standard directory function.