Menu
I have a really large table need to calculate (10 billion + rows). It's too long to calculate all of them at one time. Therefore, I create a list of break points, and I calculate and store at the end of each step. However, each step costs longer time. I think the reason is the memory or cache, do you have any idea how to deal with in this situation or just how to clear cache or memory?
I reuse the variable for output in the loop, so the variable will not become bigger and bigger.
Thank you so much.
204
We will learn how to clear memory for a variable, list, and array using two different methods. The two different methods are del and gc.collect (). del and gc.collect () are the two different methods to delete the memory in python. The clear memory method is helpful to prevent the overflow of memory.
We can overcome this problem, by using Python’s inbuilt garbage collection module to collect all objects that have LRU Cache Wrappers, and iteratively clear each object’s cache. Writing code in comment? Please use ide.geeksforgeeks.org , generate link and share the link here.
Memory Management¶ Overview¶ Memory management in Python involves a private heap containing all Python objects and data structures. The management of this private heap is ensured internally by the Python memory manager.
Memory link will cause because of lingering large objects which are not released and reference cycles within the code. GC module is helpful to debug the memory leaks in python. It provides the garbage collector, has a list of all objects. This module will give an idea of where the program’s memory is being used. Then the objects are filtered.
I use to solve this issue using a line that reset the variable at the end of the process, cleaning the cache:
MyVariable = None
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
