Reason and solution of "*** Error in `python': free(): corrupted unsorted chunks: 0x0000000000ff2460 ***" in Python

Question

I created a service using Python which connects to SQL Azure using pymssql and only makes SELECT queries. After 1 day (or a bit more), the connection begins to fail in queries and finally the service ends with the error:

Error in `python': free(): corrupted unsorted chunks: 0x0000000000ff2460

I am not be sure if there is only one error or more (maybe the first error provokes more errors).

The code of connection is here:

  connectionDb = pymssql.connect(host=self.HOST_DATA_BASE, user=self.USER_DATA_BASE, password=self.PASSWORD_DATA_BASE, database=self.DATA_BASE_NAME)    

and I execute the query in the next form:

  cursor=connectionDb.cursor()
  cursor.execute("select * from vehicles") 
  rows = cursor.fetchall()

  if (rows!=None):
     return rows

Initially, the connection works fine. The problem occurs after a time of inactivity.

I tried to simplify the queries but I don´t believe that is the reason of error.

*Maybe it is possible bug in pymssql?

Answer 1

Okay so I was able to verify that this is a memory issue within pymssql. There is an open ticket out for this.

I used the python garbage collection library (gc) to print how much memory has not been freed up by Python.

Below is the garbage collection outputs. As you can see 'collections' continues to increase, meaning that pymssql did not free the memory even though it was no longer in use.

garbage collector output is [{'collections': 1822693, 'collected': 54744, 'uncollectable': 0}, {'collections': 165699, 'collected': 863905, 'uncollectable': 0}, {'collections': 4173, 'collected': 359393, 'uncollectable': 0}]
garbage collector output is [{'collections': 1823867, 'collected': 54744, 'uncollectable': 0}, {'collections': 165806, 'collected': 863905, 'uncollectable': 0}, {'collections': 4176, 'collected': 359393, 'uncollectable': 0}]
garbage collector output is [{'collections': 1825043, 'collected': 54744, 'uncollectable': 0}, {'collections': 165912, 'collected': 863905, 'uncollectable': 0}, {'collections': 4178, 'collected': 359393, 'uncollectable': 0}]
garbage collector output is [{'collections': 1826218, 'collected': 54744, 'uncollectable': 0}, {'collections': 166019, 'collected': 863905, 'uncollectable': 0}, {'collections': 4180, 'collected': 359393, 'uncollectable': 0}]
garbage collector output is [{'collections': 1827393, 'collected': 54744, 'uncollectable': 0}, {'collections': 166126, 'collected': 863905, 'uncollectable': 0}, {'collections': 4182, 'collected': 359393, 'uncollectable': 0}]
garbage collector output is [{'collections': 1828567, 'collected': 54744, 'uncollectable': 0}, {'collections': 166233, 'collected': 863905, 'uncollectable': 0}, {'collections': 4185, 'collected': 359393, 'uncollectable': 0}]
garbage collector output is [{'collections': 1829741, 'collected': 54744, 'uncollectable': 0}, {'collections': 166340, 'collected': 863905, 'uncollectable': 0}, {'collections': 4188, 'collected': 359393, 'uncollectable': 0}]
garbage collector output is [{'collections': 1830917, 'collected': 54744, 'uncollectable': 0}, {'collections': 166446, 'collected': 863905, 'uncollectable': 0}, {'collections': 4190, 'collected': 359393, 'uncollectable': 0}]
garbage collector output is [{'collections': 1832092, 'collected': 54744, 'uncollectable': 0}, {'collections': 166553, 'collected': 863905, 'uncollectable': 0}, {'collections': 4192, 'collected': 359393, 'uncollectable': 0}]
garbage collector output is [{'collections': 1833267, 'collected': 54744, 'uncollectable': 0}, {'collections': 166660, 'collected': 863905, 'uncollectable': 0}, {'collections': 4194, 'collected': 359393, 'uncollectable': 0}]
garbage collector output is [{'collections': 1834441, 'collected': 54744, 'uncollectable': 0}, {'collections': 166767, 'collected': 863905, 'uncollectable': 0}, {'collections': 4197, 'collected': 359393, 'uncollectable': 0}]


free(): corrupted unsorted chunks

When I switched to pyodbc I ran the same code and got the following:

garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]
garbage collector output is [{'collections': 218, 'collected': 806, 'uncollectable': 0}, {'collections': 19, 'collected': 115, 'uncollectable': 0}, {'collections': 1, 'collected': 5, 'uncollectable': 0}]

As you can see 'collections' element stays constant. This proves that there is a memory leak issue with pymssql. My recommendation is to use pyodbc, pytds, or ctds. I made a point about this on the github ticket

Answer 2

@APRocha, It seems to be not a bug of pymssql, it's an error from glibc when python free some malloc memory.

Did you close the connection after done the sql opertion per request? If not, I suggest you do, or using SQLAlchemy with pymssql to manage the connections in a pool.

Otherwise, I think you can try to use gc.collect() to release the unreferenced memory at intervals, please refer to the document as reference.

Hope it helps.