How to clear out/delete tensors in tensorflow?

Question

As I understand, tf.reset_default_graph() only creates a new graph and sets it equal to the default graph. So, the previously created tensors would just be lying around occupying the memory. I have also read the unreferenced tensors are not garbage collected (like normal variables in Python are).

If I am running a cross-validation to search for a set of hyperparameters and thus creating the same graph, again and again, how do I get rid of the previously created tensors?

Answer 1

I had the same problem when designing experiments, after researching about this problem, the only solution that worked for me is this one. As you can read in that link, it seems to be a design flaw and the TF team doesn't seem to care about fixing.

The solution is to create a new process for each cross-validation iteration. So when the process finishes the system kills it and releases the resources automatically.

import multiprocessing

def evaluate(...):
    import tensorflow as tf
    # Your logic

for ... in cross_valiadtion_loop:
    process_eval = multiprocessing.Process(target=evaluate, args=(...))
    process_eval.start()
    process_eval.join()