How to deal with "MemoryError" in Python code

Question

I have some piece of python code which generates a MemoryError after a while. I know that it consumes a lot of memory. So, I decided to put the code within a try/except block so that the skeleton looks like the following:

while True:

      while True:

            try:
            #---- do some stuff

            except MemoryError as err:
                   print(err)
                   break

So, my idea is to break out of the first while-loop if a MemoryError occurs and since I have an outer while-loop, it will start the program again.

It seems that it works for the moment but I am not sure. After a while, it stops again and I need to restart the program again. Does somebody know a better solution so that the program can run after the MemoryError again?

Answer 1

Note that Python only throws the MemoryError when it realizes it will overuse the memory beforehand. If it happens by accident (or "unnoticed" by Python) then you're out of luck. The documentation already mentions this:

MemoryError

Raised when an operation runs out of memory but the situation may still be rescued (by deleting some objects). The associated value is a string indicating what kind of (internal) operation ran out of memory. Note that because of the underlying memory management architecture (C’s malloc() function), the interpreter may not always be able to completely recover from this situation; it nevertheless raises an exception so that a stack traceback can be printed, in case a run-away program was the cause.

So if there is nothing to rescue or if the interpreter can't recover there is no MemoryError.

---

A good approach would require knowing what you're doing and how. In a majority of cases generators (see for example PEP 289 on generator expressions) or map-reduce approaches can save you a lot of memory. These might be applicable here as well.

Answer 2

It is hard to assess what to do without knowing what do you do inside this try but I will try.

Frstrly, regarding continuing the try-except block. I am afraid you cannot do this.

So short answer is: you cannot go back to try block to place where exception occured, you can go to first line of try

What you can do:

I usually handle my exceptions like the following. Create while True loop as such:

while True:
    try:
        #some code here
    except SomeException:
        continue

Now you can continue to try after exception occured.

Second way (but not reccomended) is to embedd your code using some function and recursively execute it.

def foo():
    try:
        while True:
            foo2()
    except StopIteration:
        #end code

def foo2():
    while True:
        try:
            #some code here
        except MemoryError:
            foo2()
            raise StopIteration()

However this is very DANGEROUS however if your memory is being exceeded not many times (<1000) this can be okay way to go if you need to do something before while True however you need to watch solution closely.