Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to forget head(GC'd) for lazy-sequences in Clojure?

Let's say I have a huge lazy seq and I want to iterate it so I can process on the data that I get during the iteration.

The thing is I want to lose head(GC'd) of lazy seq(that processed) so I can work on seqs that have millions of data without having OutofMemoryException.

I have 3 examples that I'm not sure.

Could you provide best practices(examples) for that purpose?

Do these functions lose head?

Example 1

(defn lose-head-fn
  [lazy-seq-coll]
  (when (seq (take 1 lazy-seq-coll))
    (do
      ;;do some processing...(take 10000 lazy-seq-coll)
      (recur (drop 10000 lazy-seq-coll)))))

Example 2

(defn lose-head-fn
  [lazy-seq-coll]
  (loop [i lazy-seq-coll]
    (when (seq (take 1 i))
      (do
        ;;do some processing...(take 10000 i)
        (recur (drop 10000 i))))))

Example 3

(doseq [i lazy-seq-coll]
  ;;do some processing...
  )

Update: Also there is an explanation in this answer here

like image 211
Ertuğrul Çetin Avatar asked Jun 16 '17 09:06

Ertuğrul Çetin


1 Answers

copy of my above comments

As far as I know, all of the above would lose head (first two are obvious, since you manually drop the head, while doseq's doc claims that it doesn't retain head).

That means that if the lazy-seq-coll you pass to the function isn't bound somewhere else with def or let and used later, there should be nothing to worry about. So (lose-head-fn (range)) won't eat all your memory, while

(def r (range)) 

(lose-head-fn r) 

probably would.

And the only best practice I could think of is not to def possibly infinite (or just huge) sequences, because all of their realized items would live forever in the var.

like image 98
leetwinski Avatar answered Oct 22 '22 11:10

leetwinski