Run commands in Emacs asynchronously, but display output incrementally

Question

I have a utility function:

(defun execute-in-buffer (command-with-args buffer)
  "Execute a string COMMAND-WITH-ARGS representing a shell command with arguments,
inserting the results in BUFFER."
  (switch-to-buffer buffer)
  (insert (format ">>> %s\n" command-with-args))
  (let* ((command-args-list (s-split " " command-with-args))
         (command (car command-args-list))
         (args (cdr command-args-list)))
    (apply 'call-process command nil buffer t args)))

This allows me to do things like (execute-in-buffer "ls /" (get-buffer-create "*my-output*"). However, it's not well suited for slow commands. If I call a series of slow commands, I don't get any output until the very end:

(let ((buf (get-buffer-create "*my-output")))
  (execute-in-buffer "sleep 10" buf)
  (execute-in-buffer "ls /" buf))

I want to be able to call synchronously, so the next command only runs after the previous one finishes. However, I want to see the output from my commands as they run. How would I do this?

(Example code is just for show, I'm happy to drop it in favour of something else.)

Answer 1

Using synchronous processes

If you want to stick with synchronous processes (e.g. using call-process), you need to call (redisplay) after each call to execute-in-buffer for the display to be updated and the output to be visible (also see this question for more details). However, the output of each command will not be visible until the process terminates, and emacs will hang while the external processes are running.

Using asynchronous processes

Using asynchronous processes is a bit more complicated, but avoids hanging Emacs while the commands are running, which also solves the redisplay issue. The tricky part here is to sequentially chain all commands. Here is a bit of elisp which should do the trick:

(defun execute-commands (buffer &rest commands)
  "Execute a list of shell commands sequentially"
  (with-current-buffer buffer
    (set (make-local-variable 'commands-list) commands)
    (start-next-command)))

(defun start-next-command ()
  "Run the first command in the list"
  (if (null commands-list)
      (insert "\nDone.")
    (let ((command  (car commands-list)))
      (setq commands-list (cdr commands-list))
      (insert (format ">>> %s\n" command))
      (let ((process (start-process-shell-command command (current-buffer) command)))
        (set-process-sentinel process 'sentinel)))))

(defun sentinel (p e)
  "After a process exited, call `start-next-command' again"
  (let ((buffer (process-buffer p)))
    (when (not (null buffer))
      (with-current-buffer buffer
        ;(insert (format "Command `%s' %s" p e) )
        (start-next-command)))))

;; Example use
(with-current-buffer (get-buffer-create "*output*") (erase-buffer))
(execute-commands "*output*"
                  "echo 1"
                  "sleep 1"
                  "echo 2; sleep 1; echo 3"
                  "ls /")

Answer 2

This works for me:

(async-shell-command "echo 1; sleep 10; echo 2; sleep 10; ls /" "*abcd*")

This can be adapted to do what you need?