Reconnecting MySQL on timeout

Question

I have a Python program which runs on background for weeks, and does database queries every once in a while. For that, I am using the ORM peewee (version 2.2.1). I am using MySQL as a backend.

Lately I've encountered a recurring problem with accessing the DB, usually after days of running the program. The error which is raised by peewee is

peewee.OperationalError: (2006, 'MySQL server has gone away')

The traceback is deep in peewee. I post it here, but as my virtualenv makes filenames too long, I am shortening them:

  File ".../local/lib/python2.7/site-packages/peewee.py", line 2910, in save
    ret_pk = self.insert(**field_dict).execute()
  File ".../local/lib/python2.7/site-packages/peewee.py", line 2068, in execute
    return self.database.last_insert_id(self._execute(), self.model_class)
  File ".../local/lib/python2.7/site-packages/peewee.py", line 1698, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File ".../local/lib/python2.7/site-packages/peewee.py", line 2232, in execute_sql
    self.commit()
  File ".../local/lib/python2.7/site-packages/peewee.py", line 2104, in __exit__
    reraise(new_type, new_type(*exc_value.args), traceback)
  File ".../local/lib/python2.7/site-packages/peewee.py", line 2223, in execute_sql
    res = cursor.execute(sql, params or ())
  File ".../local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File ".../local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
peewee.OperationalError: (2006, 'MySQL server has gone away')

Possible solution attempts I've found:

• In this question, one of the comments suggest pinging the MySQL server every once in a while to keep it (the connection?) alive. I am not sure how to do it via the ORM, though. (should I simply SELECT 1 every hour, say?)
• In this github peewee issue, which was opened 4 months ago, the same error is referred, though, it is claimed there that it is solved (and I am using a newer version).
• In a 7 year old issue of trac, one suggestion is to increase the timeout of MySQL for 3 days.
• In this forum discussion, the option of increasing MySQL's timeout is suggested, but an alternative of "using the autoReconnect option for the MySQL JDBC connector" is offered. I tried to figure out if such an option exists for Python's MySQLdb module, but couldn't find.
• I have found this MySQL reference page on reconnection behaviour, but it's a bit complicated for my understanding of MySQL (usually I work only with ORMs), and I don't know how to apply any of it from peewee.

Even if I'm able to ping the database to keep the connection alive for longer periods, I think it is considered a bad practice to keep a connection alive when one doesn't really need it. Is there any way to reopen the connection via the ORM? I consider both pinging and increasing the timeout of MySQL as workarounds, while a real solution would be to reconnect when needed (and a real solution is what I'm asking for).

Answer 1

I had the same problem and for peewee using MySQLdb I got the following solution when initialize the MySQL database instance:

db = MySQLDatabase(db_name, user=db_username, passwd=db_password, host=db_host, port=db_port)
db.get_conn().ping(True)

where for the ping function there is:

Checks whether or not the connection to the server is working. If it has gone down, an automatic reconnection is attempted.

This function can be used by clients that remain idle for a long while, to check whether or not the server has closed the connection and reconnect if necessary.

New in 1.2.2: Accepts an optional reconnect parameter. If True, then the client will attempt reconnection. Note that this setting is persistent. By default, this is on in MySQL<5.0.3, and off thereafter.

Non-standard. You should assume that ping() performs an implicit rollback; use only when starting a new transaction. You have been warned.

in the db.get_conn().ping.__doc__. Mind that db.get_conn().ping(True) has to be used if you create another connection again. So if you reconnect (through db.connect() for example) you must repeat the ping.

Answer 2

You have to catch the exception and based on which error, reconnect or to do something else. Whether it is a connection time out, or a network problem or the MySQL had to be restarted.

The below (pseudoish) code shows how you could do that, but there is more to it. You'll want to try a few times and then bail out, or maybe try every 2 minutes or so.

while True:
    try:
        # do your database stuff
    except peewee.OperationalError as exc:
        # Oops! We have to try to reconnect

Does not really matter whether you use an ORM or not. However, an ORM might offer this functionality.