Lazy concatenation of sequence in Clojure

Question

Here's a beginner's question: Is there a way in Clojure to lazily concatenate an arbitrary number of sequences? I know there's lazy-cat macro, but I can't think of its correct application for an arbitrary number of sequences.

My use case is lazy loading data from an API via paginated (offseted/limited) requests. Each request executed via request-fn below retrieves 100 results:

(map request-fn (iterate (partial + 100) 0))

When there are no more results, request-fn returns an empty sequence. This is when I stop the iteration:

(take-while seq (map request-fn (iterate (partial + 100) 0)))

For example, the API might return up to 500 results and can be mocked as:

(defn request-fn [offset] (when (< offset 500) (list offset)))

If I want to concatenate the results, I can use (apply concat results) but that eagerly evaluates the results sequence:

(apply concat (take-while seq (map request-fn (iterate (partial + 100) 0))))

Is there a way how to concatenate the results sequence lazily, using either lazy-cat or something else?

Answer 1

For the record, apply will consume only enough of the arguments sequence as it needs to determine which arity to call for the provided function. Since the maximum arity of concat is 3, apply will realize at most 3 items from the underlying sequence.

If those API calls are expensive and you really can't afford to make unnecessary ones, then you will need a function that accepts a seq-of-seqs and lazily concatenates them one at a time. I don't think there's anything built-in, but it's fairly straightforward to write your own:

(defn lazy-cat' [colls]
  (lazy-seq
    (if (seq colls)
      (concat (first colls) (lazy-cat' (next colls))))))