Deploying Common Lisp Web Applications

Question

I am wondering how one goes about deploying a Common Lisp web application written in, say, Hunchentoot, Wookie, Woo, or even Clack.

That is, suppose I write an app that contains some files, packages, etc. Typically when I am working locally, I simply run a command in REPL that starts the server and then visit it using localhost:8000 or something like that.

However, I am a bit puzzled as to what the process is for deploying an app to a production server like AWS EC2. In what form should I deploy the Lisp code? Are there different options? What happens if the server needs to be restarted or is experiencing problems?

Answer 1

I've figured something out lately by building self-contained executables for web apps and I wrote about it on lisp-journey/web-dev (shipping and deployment sections), as well as for the building part on the Common Lisp Cookbook/scripting#for-web-apps.

I copy the interesting parts here, there's a bit more on each resource. Edits are welcome, primarily on those resources thanks !

edit july 2019: I contributed a page on the Cookbook: https://lispcookbook.github.io/cl-cookbook/web.html

edit: see also a list of tools and platforms that provide professional CL support: https://github.com/CodyReichert/awesome-cl#deployment

(edited) How to run the web app as a script

I explain below how to build and run executables, but we can of course run the application as a script. In a lisp file, say run.lisp, ensure:

• to load your project's asd file: (load "my-project.asd")
• to load its dependencies: (ql:quickload :my-project)
• to call its main function: (my-project:start) (given start is an exported symbol, otherwise ::start).

In doing so, the application starts and gives you back a Lisp REPL. You can interact with the running application. You can update it and even install new Quicklisp libraries as it runs.

How to build a self-contained executable

See also https://github.com/CodyReichert/awesome-cl#interfaces-to-other-package-managers for bindings to Homebrew and Debian packages.

With SBCL

How to build (self-contained) executables is implementation-specific (see below Buildapp and Rowsell). With SBCL, as says its documentation, it is a matter of:

(sb-ext:save-lisp-and-die #P"path/name-of-executable" :toplevel #'my-app:main-function :executable t)

sb-ext is an SBCL extension to run external processes. See other SBCL extensions (many of them are made implementation-portable in other libraries).

:executable t tells to build an executable instead of an image. We could build an image to save the state of our current Lisp image, to come back working with it later. Specially useful if we made a lot of work that is computing intensive.

If you try to run this in Slime, you'll get an error about threads running:

Cannot save core with multiple threads running.

Run the command from a simple SBCL repl.

I suppose your project has Quicklisp dependencies. You must then:

• ensure Quicklisp is installed and loaded at Lisp startup (you completed Quicklisp installation)
• load the project's .asd
• install dependencies
• build the executable.

That gives:

(load "my-app.asd")
(ql:quickload :my-app)
(sb-ext:save-lisp-and-die #p"my-app-binary" :toplevel #'my-app:main :executable t)

From the command line, or from a Makefile, use --load and --eval:

build:
    sbcl --non-interactive \
         --load my-app.asd \
         --eval '(ql:quickload :my-app)' \
         --eval "(sb-ext:save-lisp-and-die #p\"my-app\" :toplevel #my-app:main :executable t)"

With ASDF

Now that we'seen the basics, we need a portable method. Since its version 3.1, ASDF allows to do that. It introduces the make command, that reads parameters from the .asd. Add this to your .asd declaration:

:build-operation "program-op" ;; leave as is
:build-pathname "<binary-name>"
:entry-point "<my-system:main-function>"

and call asdf:make :my-system.

So, in a Makefile:

LISP ?= sbcl

build:
    $(LISP) --non-interactive \
        --load my-app.asd \
        --eval '(ql:quickload :my-app)' \
        --eval '(asdf:make :my-system)' 

With Roswell or Buildapp

Roswell, an implementation manager and much more, also has the ros build command, that should work for many implementations.

We can also make our app installable with Roswell by a ros install my-app. See its documentation.

We'll finish with a word on Buildapp, a battle-tested and still popular "application for SBCL or CCL that configures and saves an executable Common Lisp image".

Many applications use it (for example, pgloader), it is available on Debian: apt install buildapp, but you shouldn't need it now with asdf:make or Roswell.

For web apps

We can similarly build a self-contained executable for our web-app. It would thus contain a web server and would be able to run on the command line:

$ ./my-web-app
Hunchentoot server is started.
Listening on localhost:9003.

Note that this runs the production webserver, not a development one, so we can run the binary on our VPS right away and access the app from outside.

We have one thing to take care of, it is to find and put the thread of the running web server on the foreground. In our main function, we can do something like this:

(defun main ()
  (start-app :port 9003) ;; our start-app, for example clack:clack-up
  ;; let the webserver run.
  ;; warning: hardcoded "hunchentoot".
  (handler-case (bt:join-thread (find-if (lambda (th)
                                            (search "hunchentoot" (bt:thread-name th)))
                                         (bt:all-threads)))
    ;; Catch a user's C-c
    (#+sbcl sb-sys:interactive-interrupt
      #+ccl  ccl:interrupt-signal-condition
      #+clisp system::simple-interrupt-condition
      #+ecl ext:interactive-interrupt
      #+allegro excl:interrupt-signal
      () (progn
           (format *error-output* "Aborting.~&")
           (clack:stop *server*)
           (uiop:quit)))
    (error (c) (format t "Woops, an unknown error occured:~&~a~&" c))))

We used the bordeaux-threads library ((ql:quickload "bordeaux-threads"), alias bt) and uiop, which is part of ASDF so already loaded, in order to exit in a portable way (uiop:quit, with an optional return code, instead of sb-ext:quit).

Parsing command line arguments

see the Cookbook here. TLDR; use uiop:command-line-arguments to get a list of the arguments. To parse them for real, there are libraries.

Deployment

Straightforward with an executable. The web app is visible from the outside right away.

On Heroku

See this buildpack.

Daemonizing, restarting in case of crashes, handling logs

See how to do that on your system.

Most GNU/Linux distros now come with Systemd.

Examples search result:

• https://seanmcgary.com/posts/deploying-nodejs-applications-with-systemd/

It is as simple as writing a configuration file:

# /etc/systemd/system/my-app.service
[Unit]
Description=stupid simple example

[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/local/bin/sthg sthg
Type=simple
Restart=always
RestartSec=10

running a command to start it:

sudo systemctl start my-app.service

a command to check its status:

systemctl status my-app.service

and Systemd can handle logging (we write to stdout or stderr, it writes logs):

journalctl -f -u my-app.service

and it handles crashes and restarts the app:

Restart=always

and it can start the app after a reboot:

[Install]
WantedBy=basic.target

to enable it:

sudo systemctl enable my-app.service

Debugging SBCL error: ensure_space: failed to allocate n bytes

If you get this error with SBCL on your server:

mmap: wanted 1040384 bytes at 0x20000000, actually mapped at 0x715fa2145000
ensure_space: failed to allocate 1040384 bytes at 0x20000000
(hint: Try "ulimit -a"; maybe you should increase memory limits.)

then disable ASLR:

sudo bash -c "echo 0 > /proc/sys/kernel/randomize_va_space"

Connecting to a remote Swank server

Little example here: http://cvberry.com/tech_writings/howtos/remotely_modifying_a_running_program_using_swank.html.

Demo project here: https://lisp-journey.gitlab.io/blog/i-realized-that-to-live-reload-my-web-app-is-easy-and-convenient/

It defines a simple function that prints forever:

;; a little common lisp swank demo
;; while this program is running, you can connect to it from another terminal or machine
;; and change the definition of doprint to print something else out!
;; (ql:quickload :swank)
;; (ql:quickload :bordeaux-threads)

(require :swank)
(require :bordeaux-threads)

(defparameter *counter* 0)

(defun dostuff ()
  (format t "hello world ~a!~%" *counter*))

(defun runner ()
  (bt:make-thread (lambda ()
                    (swank:create-server :port 4006)))
  (format t "we are past go!~%")
  (loop while t do
       (sleep 5)
       (dostuff)
       (incf *counter*)
       ))

(runner)

On our server, we run it with

sbcl --load demo.lisp

we do port forwarding on our development machine:

ssh -L4006:127.0.0.1:4006 [email protected]

this will securely forward port 4006 on the server at example.com to our local computer's port 4006 (swanks accepts connections from localhost).

We connect to the running swank with M-x slime-connect, typing in port 4006.

We can write new code:

(defun dostuff ()
  (format t "goodbye world ~a!~%" *counter*))
(setf *counter* 0)

and eval it as usual with M-x slime-eval-region for instance. The output should change.

There are more pointers on CV Berry's page.

Hot reload

Example with Quickutil. See notes on lisp-journey.

It has to be run on the server (a simple fabfile command can call this through ssh). Beforehand, a fab update has run git pull on the server, so new code is present but not running. It connects to the local swank server, loads the new code, stops and starts the app in a row.

Continuous Integration, continuous delivery of executables, Docker

See https://lispcookbook.github.io/cl-cookbook/testing.html#continuous-integration