I have a Google App Engine app that uses a Google Cloud SQL instance for storing data. I need my instance to be able to serve hundreds of clients at a time, via restful calls, which each result in one or a handful of DB queries. I've wrapped the methods that need DB access and store the handle to the DB connection in os.environ. See this SO question/answer for basically how I'm doing it.
However, as soon as a couple hundred clients connect to my app and trigger database calls, I start getting these errors in the Google App Engine error logs (and my app returns 500, of course):
could not connect: ApplicationError: 1033 Instance has too many concurrent requests: 100 Traceback (most recent call last): File "/base/python27_run
Any tips from experienced users of Google App Engine and Google Cloud SQL? Thanks in advance.
Here's the code for the decorator I use around methods that require DB connection:
def with_db_cursor(do_commit = False):
""" Decorator for managing DB connection by wrapping around web calls.
Stores connections and open connection count in the os.environ dictionary
between calls. Sets a cursor variable in the wrapped function. Optionally
does a commit. Closes the cursor when wrapped method returns, and closes
the DB connection if there are no outstanding cursors.
If the wrapped method has a keyword argument 'existing_cursor', whose value
is non-False, this wrapper is bypassed, as it is assumed another cursor is
already in force because of an alternate call stack.
Based mostly on post by : Shay Erlichmen
At: https://stackoverflow.com/a/10162674/379037
"""
def method_wrap(method):
def wrap(*args, **kwargs):
if kwargs.get('existing_cursor', False):
#Bypass everything if method called with existing open cursor
vdbg('Shortcircuiting db wrapper due to exisiting_cursor')
return method(None, *args, **kwargs)
conn = os.environ.get("__data_conn")
# Recycling connection for the current request
# For some reason threading.local() didn't work
# and yes os.environ is supposed to be thread safe
if not conn:
conn = _db_connect()
os.environ["__data_conn"] = conn
os.environ["__data_conn_ref"] = 1
dbg('Opening first DB connection via wrapper.')
else:
os.environ["__data_conn_ref"] = (os.environ["__data_conn_ref"] + 1)
vdbg('Reusing existing DB connection. Count using is now: {0}',
os.environ["__data_conn_ref"])
try:
cursor = conn.cursor()
try:
result = method(cursor, *args, **kwargs)
if do_commit or os.environ.get("__data_conn_commit"):
os.environ["__data_conn_commit"] = False
dbg('Wrapper executing DB commit.')
conn.commit()
return result
finally:
cursor.close()
finally:
os.environ["__data_conn_ref"] = (os.environ["__data_conn_ref"] -
1)
vdbg('One less user of DB connection. Count using is now: {0}',
os.environ["__data_conn_ref"])
if os.environ["__data_conn_ref"] == 0:
dbg("No more users of this DB connection. Closing.")
os.environ["__data_conn"] = None
db_close(conn)
return wrap
return method_wrap
def db_close(db_conn):
if db_conn:
try:
db_conn.close()
except:
err('Unable to close the DB connection.', )
raise
else:
err('Tried to close a non-connected DB handle.')
Cloud Run limits Cloud Run services are limited to 100 connections to a Cloud SQL database. This limit applies per service instance.
In the Google Cloud console, go to the Cloud SQL Instances page. To open the Overview page of an instance, click the instance name. Select Connections from the SQL navigation menu. In the Authorized networks section, click Add network and enter the IP address of the machine where the client is installed.
SQL Server allows a maximum of 32,767 user connections.
I read from the documentation and noticed there's a 12 connection / instance limit:
Look for "Each App Engine instance cannot have more than 12 concurrent connections to a Google Cloud SQL instance." in https://developers.google.com/appengine/docs/python/cloud-sql/
Short answer: Your queries are probably too slow and the mysql server doesn't have enough threads to process all of the requests you are trying to send it.
Long Answer:
As background, Cloud SQL has two limits that are relevant here:
It doesn't sound like the connection limit is affecting you, but I wanted to mention it just in case.
When it comes to Concurrent Requests, increasing the limit might help, but it usually makes the problem worse. There are two cases we've seen in the past:
You should use EXPLAIN and SHOW ENGINE INNODB STATUS to see which of the two problems is going on.
Of course, it is also possible that you are just driving a ton of traffic at your instance and there just aren't enough threads. In that case, you'll probably be maxing out the cpu for the instance anyway, so adding more threads won't help.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With