Idiomatic Clojure way of mimicking Python's yield

Question

I'm iterating through a list, building up state as I go, and occasionally when I encounter a certain sentinel, I return a result. If I was doing this in Python, I would lazily yield the results, tracking state in the function's local scope as I go:

# this is simplified for illustration
def yielder(input_list):
    state = 0
    for item in input_list:
        if item = 'SENTINEL':
            yield state * 2
            state = 0
        else:
            state += item

yielder([1, 5, 2, 5, 'SENTINEL', 4, 6, 7]) # [26, 34]

My first implementation uses reduce, but that's not as good as yield because:

• The value I pass between iterations has both the loop state, and the items's i want to yield, which seems clunky
• It's not lazy

iterate could be used to mitigate the latter, but i don't actually want to return something for every input item, so it would require more munging.

What's an idiomatic way to do this in Clojure?

Answer 1

You can build this yourself using lazy-seq as you mention or you could use partition and reduce to split the problem into phases then thread them together. I'll use the thread-last macro to show each step on it's own:

user> (->> [1, 5, 2, 5, :SENTINEL, 4, 6, 7] ;; start with data
           (partition-by #(= :SENTINEL %))  ;; ((1 5 2 5) (:SENTINEL) (4 6 7))
           (take-nth 2)                     ;; ((1 5 2 5) (4 6 7))
           (map #(* 2 (reduce + %))))       ;; the map here keeps it lazy
(26 34)

and here it is usin lazy-seq directly:

user>  (defn x [items]
         (when (seq items)
           (lazy-seq (cons (* 2 (reduce + (take-while #(not= :SENTINEL %) items)))
                           (x (rest (drop-while #(not= :SENTINEL %) items)))))))
#'user/x
user> (x [1, 5, 2, 5, :SENTINEL, 4, 6, 7])
(26 34)

Answer 2

The Tupelo library has a way to do this using lazy-gen/yield which mimics a Python generator function:

(ns xyz
  (:require [tupelo.core :as t] ))

(def data-1 [1 5 2 5 :SENTINEL 4 6 7] )
(def data-2 [1 5 2 5 :SENTINEL 4 6 7 :SENTINEL] )

(defn yielder [vals]
  (t/lazy-gen
    (let [state (atom 0)]
      (doseq [item vals]
        (if (= :SENTINEL item)
          (do
            (t/yield (* 2 @state))
            (reset! state 0))
          (swap! state + item))))))

(yielder data-1) => (26)
(yielder data-2) => (26 34)

Note that the original problem description had a bug, since the cumulative state is only output when the :SENTENEL tag is encountered. The different outputs for data-1 and data-2 illustrate the problem.