How to search for files with a wildcard in Common Lisp?

Question

I am not satisfied to find files matching a string like this:

(remove-if-not (lambda (it)
                   (search "wildcard" (namestring it)))
               (uiop:directory-files "./"))
;; I'll ignore case with str:contains?
;; https://github.com/vindarel/cl-str

How would one search for files with unix-style wildcards ?

If it is not built-in, I'd enjoy a solution with uiop. Maybe there is with Osicat or cl-fad (with which it doesn't seem so, the documentation oftentimes says "non-wild pathname").

Bonus if it is possible to use the double wildcard to traverse directories recursively (./**/*.jpg).

edit: I have tried variants of (directory #p"./**/*.jpg") and it returns nil :( Also tried #p".*jpg", #p"./.*jpg",…

(wild-pathname-p (pathname "*.jpg"))
(:WILD :WILD-INFERIORS)


(make-pathname :name :wild :type "jpg")
#P"*.jpg"

The following gets me files by jpg extension, but it isn't a proper wildcard yet:

(directory *)
(#P"/home/vince/cl-cookbook/AppendixA.jpg"
 #P"/home/vince/cl-cookbook/AppendixB.jpg"
 #P"/home/vince/cl-cookbook/AppendixC.jpg")

Documentation on pathnames and make-pathname: http://gigamonkeys.com/book/files-and-file-io.html (no mentions of wildcards)

Answer 1

SBCL

SBCL supports wildcards in names. First, create some files:

(loop 
  with stem = #P"/tmp/stack/_.txt"
  initially (ensure-directories-exist stem)
  for name in '("abc" "def" "cadar" "cdadr" "cddr")
  for path = (make-pathname :name name :defaults stem)
  do (open path :direction :probe :if-does-not-exist :create))

Then, list all files that contains an "a":

CL-USER> (directory #P"/tmp/stack/*a*.txt")
(#P"/tmp/stack/abc.txt" #P"/tmp/stack/cadar.txt" #P"/tmp/stack/cdadr.txt")

The pathname contains an implementation-specific (valid) name component:

CL-USER> (describe #P"/tmp/stack/*a*.txt")
#P"/tmp/stack/*a*.txt"
  [structure-object]

Slots with :INSTANCE allocation:
  HOST       = #<SB-IMPL::UNIX-HOST {10000F3FF3}>
  DEVICE     = NIL
  DIRECTORY  = (:ABSOLUTE "tmp" "stack")
  NAME       = #<SB-IMPL::PATTERN :MULTI-CHAR-WILD "a" :MULTI-CHAR-WILD>
  TYPE       = "txt"
  VERSION    = :NEWEST
; No value

SBCL also defines sb-ext:map-directory, which process files one by one, instead of first collecting all files in a list.

Portable solutions

If you need to stick to standard pathname components, you can first call directory with normal wildcards, and filter the resulting list:

CL-USER> (remove-if-not (wildcard "*a*")
                        (directory #P"/tmp/stack/*.txt")
                        :key #'pathname-name)

(#P"/tmp/stack/abc.txt" #P"/tmp/stack/cadar.txt" #P"/tmp/stack/cdadr.txt")

... where wildcard might be based on regex (PPCRE):

(defun parse-wildcard (string)
  (delete ""
          (map 'list
               (lambda (string)
                 (or (cdr (assoc string
                                 '(("*" . :wild)
                                   ("?" . :char))
                                 :test #'string=))
                     string))
               (ppcre:split '(:sequence
                              (:negative-lookbehind #\\)
                              (:register (:alternation #\* #\?)))
                            string
                            :with-registers-p t))
          :test #'string=))

(note: the above negative lookbehind does not eliminate escaped backslashes)

(defun wildcard-regex (wildcard)
  `(:sequence
    :start-anchor
    ,@(loop
        for token in wildcard
        collect (case token
                  (:char :everything)
                  (:wild '(:greedy-repetition 0 nil :everything))
                  (t token)))
    :end-anchor))

(defun wildcard (string)
  (let ((scanner (ppcre:create-scanner
                  (wildcard-regex (parse-wildcard string)))))
    (lambda (string)
      (ppcre:scan scanner string))))

Intermediate functions:

CL-USER> (parse-wildcard "*a*a\\*a?\\?a")
(:WILD "a" :WILD "a\\*a" :CHAR "\\?a")

CL-USER> (wildcard-regex (parse-wildcard "*a*a\\*a?\\?a"))
(:SEQUENCE :START-ANCHOR #1=(:GREEDY-REPETITION 0 NIL :EVERYTHING) "a" #1# "a\\*a" :EVERYTHING "\\?a" :END-ANCHOR)

Answer 2

no current directory and no home directory characters

The concept of . denoting the current directory does not exist in portable Common Lisp. This may exist in specific filesystems and specific implementations.

Also ~ to denote the home directory does not exist. They may be recognized by some implementations as non-portable extensions.

In pathname strings you have * and ** as wildcards. This works in absolute and relative pathnames.

defaults for the default pathname

Common Lisp has *default-pathname-defaults* which provides a default for some pathname operations.

Examples

CL-USER 46 > (directory "/bin/*")
(#P"/bin/[" #P"/bin/bash" #P"/bin/cat"   ....   )

Now in above it is already slightly undefined or diverging what implementations do on Unix:

• resolve symbolic links?
• include 'hidden' files?
• include files with types?

Next:

CL-USER 47 > (directory "/bin/*sh")
(#P"/bin/zsh" #P"/bin/tcsh" #P"/bin/sh" #P"/bin/ksh" #P"/bin/csh" #P"/bin/bash")

Using a relative pathname:

CL-USER 48 > (let ((*default-pathname-defaults* (pathname "/bin/")))
               (directory "*sh"))
(#P"/bin/zsh" #P"/bin/tcsh" #P"/bin/sh" #P"/bin/ksh" #P"/bin/csh" #P"/bin/bash")

Files in your home directory:

CL-USER 49 > (let ((*default-pathname-defaults* (user-homedir-pathname)))
              (directory "*"))

The same:

CL-USER 54 > (directory (make-pathname :name "*"
                                       :defaults (user-homedir-pathname)))

Finding all files ending with sh in /usr/local/ and below:

CL-USER 54 > (directory "/usr/local/**/*sh")

Constructing pathnames with MAKE-PATHNAME

Three ways to find all .h files under /usr/local/:

(directory "/usr/local/**/*.h")

(directory (make-pathname :name :wild
                          :type "h"
                          :defaults "/usr/local/**/")

(directory
   (make-pathname :name :wild
                  :type "h"
                  :directory '(:ABSOLUTE "usr" "local" :WILD-INFERIORS)))

Problems

There are a lot of different interpretations of implementations across platforms ('windows', 'unix', 'mac', ...) and even on the same platform (especially 'windows' or 'unix'). Stuff like unicode in pathnames creates additional complexity - not describe in the CL standard.

We still have a lot of different filesystems ( https://en.wikipedia.org/wiki/List_of_file_systems ), but they are different or different in capabilities from what was typical when Common Lisp was designed. Implementations may have tracked the changes, but not necessarily in portable ways.