Generate and stream a zip-file in a Ring web app in Clojure

Question

I have a Ring handler that needs to:

• Zip a few files
• Stream the Zip to the client.

Now I have it sort of working, but only the first zipped entry gets streamed, and after that it stalls/stops. I feel it has something to do with flushing/streaming that is wrong.

Here is my (compojure) handler:

(GET "/zip" {:as request}
            :query-params [order-id   :- s/Any]
            (stream-lessons-zip (read-string order-id) (:db request) (:auth-user request)))

Here is the stream-lessons-zip function:

(defn stream-lessons-zip
  []
  (let [lessons ...];... not shown

  {:status 200
   :headers {"Content-Type" "application/zip, application/octet-stream"
             "Content-Disposition" (str "attachment; filename=\"files.zip\"")
   :body (futil/zip-lessons lessons)}))

And i use a piped-input-stream to do the streaming like so:

(defn zip-lessons
 "Returns an inputstream (piped-input-stream) to be used directly in Ring HTTP responses"
[lessons]
(let [paths (map #(select-keys % [:file_path :file_name]) lessons)]
(ring-io/piped-input-stream
  (fn [output-stream]
    ; build a zip-output-stream from a normal output-stream
    (with-open [zip-output-stream (ZipOutputStream. output-stream)]
      (doseq [{:keys [file_path file_name] :as p} paths]
        (let [f (cio/file file_path)]
          (.putNextEntry zip-output-stream (ZipEntry. file_name)) 
          (cio/copy f zip-output-stream)
          (.closeEntry zip-output-stream))))))))

So I have confirmed that the 'lessons' vector contains like 4 entries, but the zip file only contains 1 entry. Furthermore, Chrome doesn't seem to 'finalize' the download, ie. it thinks it is still downloading.

How can I fix this?

Answer 1

It sounds like producing a stateful stream using blocking IO is not supported by http-kit. Non-stateful streams can be done this way:

http://www.http-kit.org/server.html#async

A PR to introduce stateful streams using blocking IO was not accepted:

https://github.com/http-kit/http-kit/pull/181

It sounds like the option to explore is to use a ByteArrayOutputStream to fully render the zip file to memory, and then return the buffer that produces. If this endpoint isn't highly trafficked and the zip file it produces is not large (< 1 gb) then this might work.

Answer 2

So, it's been a few years, but that code still runs in production (ie. it works). So I made it work back then, but forgot to mention it here (and forgot WHY it works, to be honest,.. it was very much trial/error).

This is the code now:

(defn zip-lessons
  "Returns an inputstream (piped-input-stream) to be used directly in Ring HTTP responses"
  [lessons {:keys [firstname surname order_favorite_name company_name] :as annotation
            :or {order_favorite_name ""
                 company_name ""
                 firstname ""
                 surname ""}}]
  (debug "zipping lessons" (count lessons))
  (let [paths (map #(select-keys % [:file_path :file_name :folder_number]) lessons)]
    (ring-io/piped-input-stream
      (fn [output-stream]
        ; build a zip-output-stream from a normal output-stream
        (with-open [zip-output-stream (ZipOutputStream. output-stream)]
          (doseq [{:keys [file_path file_name folder_number] :as p} paths]
            (let [f (cio/as-file file_path)
                  baos (ByteArrayOutputStream.)]
              (if (.exists f)
                (do
                  (debug "Adding entry to zip:" file_name "at" file_path)
                  (let [zip-entry (ZipEntry. (str (if folder_number (str folder_number "/") "") file_name))]
                    (.putNextEntry zip-output-stream zip-entry)

                   
                    (.close baos)
                    (.writeTo baos zip-output-stream)
                    (.closeEntry zip-output-stream)
                    (.flush zip-output-stream)
                    (debug "flushed")))
                (warn "File '" file_name "' at '" file_path "' does not exist, not adding to zip file!"))))
          (.flush zip-output-stream)
          (.flush output-stream)
          (.finish zip-output-stream)
          (.close zip-output-stream))))))