How to make defsystem use "everything"?

Question

I am working on project euler problems in SBCL and keep a short file for every solution. Every problem has some 5am-based tests, which are referenced from a "main" test suite. These tests are run when "tests.lisp" is run. Since I got bored with maintaining the list of files by hand, I wrote some code to do it for me:

(defpackage #:euler/asdf
  (:use :cl :asdf))
(in-package #:euler/asdf)

;; directory containing the problem files
(defparameter +dir+ "/home/stefan/quicklisp/local-projects/euler")

;; build file list for package components
(defun files-for-problems (dir)
  (mapcar #'(lambda (p) (list :file (pathname-name p) :depends-on '("package")))
      (directory (concatenate 'string dir "/e????.lisp"))))

;; build dependency list for all tests component
(defun depends-on-problems (dir)
  (mapcar #'pathname-name
      (directory (concatenate 'string dir "/e????.lisp"))))

;; define euler system
(defsystem euler
    :name "euler"
    :author "Stefan Schmiedl"
    :description "Solutions to problems at http://projecteuler.net"
    :depends-on ("iterate" "fiveam" "cl-csv")
    :components #.`((:file "package")
            ,@(files-for-problems +dir+)
         #.`(:file "tests" :depends-on ,(depends-on-problems +dir+))))

In short, defsystem euler uses all e????.lisp files as components and tests.lisp depends on all of these files.

Is this a good idea? Is there an "official" way to make defsystem use all files in a directory or all files matching a given filename pattern?

I feel like I'm missing something elementary here, especially after reading some ELS slides on github about a "more declarative defsystem" where the thing I've done above would probably be frowned upon.

---

After some fiddling with Fare's suggestion here is what I now have:

;; define private package for defsystem
(defpackage #:euler-system
  (:use :cl :uiop :asdf))
(in-package #:euler-system)


;; define euler system
(defsystem "euler"
  :author "Stefan Schmiedl"
  :description "Solutions to problems at http://projecteuler.net"
  :depends-on ("iterate" "fiveam" "cl-csv")
  :components ((:module "package"
                        :pathname ""
                        :components ((:file "package")))
               (:module "problems"
                        :pathname ""
                        :depends-on ("package")
                        :components #.(mapcar #'(lambda (p) (list :file (pathname-name p)))
                                              (directory-files (pathname-directory-pathname
                                                                (uiop/lisp-build:current-lisp-file-pathname))
                                                               "e*.lisp")))
               (:module "tests"
                        :pathname ""
                        :depends-on ("package" "problems")
                        :components ((:file "tests")))))

Thanks for the feedback.

Answer 1

For the directory part, I recommend using relative pathnames. You could do it several ways.

1- Thou shalt not use an absolute pathname. Use relative pathname like that, possibly via a variable: (subpathname (current-file-pathname) #p"e????.lisp")

2- I'm not sure how portable ? is as a wildcard character — if you can live with it, * is much more portable.

3- uiop:directory-files is safer than cl:directory in this and many contexts.

4- for an "official" way of handling wildcard patterns without #. or (eval `...), get inspiration from asdf/contrib/wild-modules.lisp — that said, for a one-off, #. is totally acceptable, especially since we're so far from purely declarative .asd files.

5- for grouped dependencies, you could use

(defsystem "euler"
  :depends-on ("iterate" "fiveam" "cl-csv")
  :serial t
  :components
   ((:module "package" :pathname ""
       :components ((:file "package")))
    (:module "problems" :pathname "" :depends-on ("package")
       :components #.(mapcar ...))
    (:module "tests" :pathname ""
       :components ((:file "tests")))))

6- Instead of modules, you could be using secondary systems, at which point system-relative-pathname would be available:

(defsystem "euler" :depends-on ("euler/tests"))
(defsystem "euler/tests"
  :depends-on ("euler/package")
  :components ((:file "package")))
(defsystem "euler/problems"
  :depends-on ("euler/package")
  :components
    #.(mapcar ... (directory-files (system-relative-pathname "euler" #p"e*.lisp")))))
(defsystem "euler/tests"
  :depends-on ("euler/problems")
  :components ((:file "tests")))

7- In the above I assume asdf3, and that you use uiop without prefix:

(defpackage :euler-system (:use :cl :uiop :asdf))
(in-package :euler-system)

If you don't define any function or variable or class, you could directly (in-package :asdf)

I'm glad you enjoyed my talk at ELS 2013. I gave another one at ELS 2014, in the same repository.

Answer 2

ASDF provides three built-in component-types, you're using the simple :file component type only in your system definition. Typically for grouping some files together, one would introduce separate modules (which pretty much directly translate to different directories), but modules still require you to specify the (sub-)components and then you're back to where you started. I looked briefly if there would be an ASDF extension that supports the functionality you have built, but haven't found anything. So, while there might be some minor problems with your code (such as the wildcard syntax probably not being portable across implementations), your general approach looks fine to me.

To address your second question of whether this is a good idea: Looking at make's implicit rules, I think that having something like this might be useful. However, more often than not you have dependencies between the various files and as soon as you need to specify these, you're basically back to having to list the components and their dependencies. The entire idea of defsystem(s) is to be able to specify dependencies and required serializations. So your use case might not be too common which might explain why you don't find an easily provided solution for this.