Menu
As part of one of my programs, I want to catch any MemoryError and log them. Currently, I'm using traceback.format_exception to format all other exceptions; I would like to do that for MemoryError as well.
However, using format_exception requires using more memory, which is exactly what I cannot do when I'm out of memory.
What do?
What can I do when I'm thrown a MemoryError besides terminating? Ideally, I would log them and then resume the program on the assumption that freeing the part of the stack between the handler and the thrower* has freed up sufficient memory for me to do so.
(* and also those heap objects kept alive only by the references emanating from that stack section)
787
Attention to Large Nested Loops In these cases, the best practice is often to break the work into batches, allowing the memory to be freed in between calls. As an example, in the code below, we have broken out earlier nested loops into 3 separate loops, each running for 333,333,333 iterations.
Python doesn't limit memory usage on your program. It will allocate as much memory as your program needs until your computer is out of memory. The most you can do is reduce the limit to a fixed upper cap. That can be done with the resource module, but it isn't what you're looking for.
Debugging. Once you detect that there is an unusual memory consumption pattern in your app, the next step to take is to debug your app to locate the cause. Python's inbuilt garbage collector enables you to debug your app's memory usage. You can view a list of objects in the memory that the garbage collector is aware of ...
del and gc. collect() are the two different methods to delete the memory in python. The clear memory method is helpful to prevent the overflow of memory. We can delete that memory whenever we have an unused variable, list, or array using these two methods.
It really has little to do with "the stack". Python raises MemoryError when the system malloc() (or similar platform function) returns NULL. In that case, no new memory is actually allocated! So you have just as much free memory remaining as you did before the request that triggered MemoryError. For example, in an interactive shell on a 32-bit box here:
>>> x = [None] * 10**9
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
MemoryError
>>>
That's harmless. Yes, in a realistic program, other memory may be freed as the stack unwinds.
But there must be something unusual about your program if "just continuing" won't just lead to another MemoryError when the same (or similar) task is attempted again. The sanest thing is indeed usually exiting the program and starting over again.
@FredMitchell's idea of reserving some "emergency memory" may be worth pursuing. E.g.,
emergency = 'x' * 10**7
consumes about 10 MB (in Python 2), mostly in "one big chunk", and your handler could do
emergency = ""
to release that back to the system. Then malloc() would have a new contiguous block of 10MB to work with.
But, in the end, I bet it will prove a lot easier to just quit ;-)
84
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
