Are persistent collections garbage collected?

Question

If I hold e reference to a part of a persistent collection, can the whole collection be garbage collected? Do I understand this correctly?

The function gctest is just to test the behavior of collections.

(defn gctest
  "A shot about testing the gc-ability of persistent thingies."
  [n]
  (take 5 (drop 100 (vec (range n)))))

main=> (def a (gctest 1e7))
main=> (def b (gctest 1e7))
main=> (def c (gctest 1e7))
main=> (def d (gctest 1e7))
main=> (def e (gctest 1e7))
main=> (def f (gctest 1e7))
main=> (def g (gctest 1e7))

OutOfMemoryError GC overhead limit exceeded  clojure.lang.ArrayChunk.dropFirst     (ArrayChunk.java:54)

I already heard about head retention, but this seems a bit more general or isn't it?

What I want to understand is how I can use big changing collection. I expect big parts of the collections change over time, in a way that big parts can be garbage collected in principle, but not all of them.

Is there a standard way to cope with this?

Answer 1

Standard GC rule remains: as long as you keep the reference to a part of collection, all objects accessible from your references stay in memory. So only the part of collection that is accessible from your references will be hold, the rest will be collected. In particular, if you refer to the last 50 elements of 100 element list, first 50 elements will be collected and the rest will stay in memory.

However, in your case all elements of each collection starting from 100th will be kept. And the reason for it is lazy evaluation. Function take produces lazy sequence of (in your case) 5 elements. Lazy sequence object itself is not real sequence, instead it is special generator object (though it's not Clojure's, but rather Python's term). When you need element of lazy sequence, generator object generates and returns it. But if you do not ask for the element, generator just keeps references to all objects it may need to generate an element.

In your example you create large vector and ask for 5 elements from it, and then save result to variables a, b, c, etc. Clojure makes large vector and generator object, pointing to 100th element. Reference to collection itself is lost, but reference to generator object is saved on top level. You never evaluate generator objects and thus never make real 5 element sequences. REPL refers to vars a, b, c, etc., these vars refer to generator objects, and generator objects refer to the collections they need to produce real 5 element sequences. Thus all elements (except may be first 100 of them) of all collections have to stay in memory.

On other hand, if you evaluate generator objects, they will produce real 5 element sequences and forget reference to the rest of collection. Try this:

user> (def a (gctest 1e7))
#'user/a                                                                                                                                               
user> (println a)
(100 101 102 103 104)                                                                                                                                  
nil                                                                                                                                                    
user> (def b (gctest 1e7))
#'user/b                                                                                                                                               
user> (println b)
(100 101 102 103 104)                                                                                                                                  
nil                                                                                                                                                    
user> (def c (gctest 1e7))
#'user/c                                                                                                                                               
user> (println c)
(100 101 102 103 104)                                                                                                                                  
nil                                                                                                                                                    
user> (def d (gctest 1e7))
#'user/d                                                                                                                                               
user> (println d)
(100 101 102 103 104)                                                                                                                                  
nil                                                                                                                                                    
user> (def e (gctest 1e7))
#'user/e                                                                                                                                               
user> (println e)
(100 101 102 103 104)                                                                                                                                  
nil                                                                                                                                                    
user> (def f (gctest 1e7))
#'user/f                                                                                                                                               
user> (println f)
(100 101 102 103 104)                                                                                                                                  
nil                                                                                                                                                    
user> (def g (gctest 1e7))
#'user/g                                                                                                                                               
user> (println g)
(100 101 102 103 104)                                                                                                                                  
nil                                                                                                                                                    
user> (def h (gctest 1e7))
#'user/h                                                                                                                                               
user> (println h)
(100 101 102 103 104)                                                                                                                                  
nil                                                                                                                                                    
user> (def i (gctest 1e7))
#'user/i                                                                                                                                               
user> (println i)
(100 101 102 103 104)                                                                                                                                  
nil 

There's no OutOfMemory! Vars a, b, c, etc. now store real lists of 5 elements, and thus there are no more references to large collections, so they can be collected.

Answer 2

These kinds of problems can be solved using lazy sequences. In your case you have used vec function which actually create a vector in memory by going though each element generated by the range function (range returns a lazy sequence).

Below code (without vec call won't have the memory problem)

(defn gctest
  "A shot about testing the gc-ability of persistent thingies."
  [n]
  (take 5 (drop 100 (range n))))

UPDATE:

When you use the vec call, all the elements in the memory will be retained and not collected by GC because these elements are referenced (and required) by the sequence object returned from the function gctest so that it can fetch the required elements (i.e skipping 100 elements and taking 5 elements) when the sequence object is asked for elements.