AWS Aurora server sometimes responds with "Unknown Database: ..."

Question

I have a .NET application that executes queries on a MySQL database in Amazon Aurora RDS. The database is a MySQL 5.6 database - the version that comes with Aurora.

I execute the queries against a schema (name: flight) within the database (name: SSIMLoader). I am using the MySQL 6.9.8.0 libraries.

edit: The connection string points to the master server in the cluster, not to a specific server.

The error occurs when doing inserts, selects, and updates so there's no real pattern to it. Even simple select statements fail randomly.

However, we receive no such error when executing the queries through MySQL workbench.

After restarting the server, the error disappears for a few days and then returns.

We have some very intense queries that do hike the CPU up to 90% for extended periods, but the error persists even after the CPU has dropped back down to 10%.

Has anyone else experienced this and if so, how did you get past it?

Thanks in advance.

• edit: Finally got the error again:

  MySql.Data.MySqlClient.MySqlException (0x80004005): Unknown database 'flight' at MySql.Data.MySqlClient.MySqlStream.ReadPacket() at MySql.Data.MySqlClient.NativeDriver.ReadOk(Boolean read) at MySql.Data.MySqlClient.MySqlConnection.ChangeDatabase(String databaseName) at MySql.Data.MySqlClient.MySqlConnection.Open() at MySql.Data.MySqlClient.MySqlConnection.Open() at Universal.Access.Database.BaseDatabaseConnection.EstablishFactoryConnection() at Universal.Access.Database.BaseDatabaseConnection.ExecuteReader(CommandType commandType, CommandText commandText) at Universal.Core.DataModel.ExecutableDatabaseConnection.ExecuteQuery(String sqlQuery) in C:\tc-projects\universal-platform\Universal.Core\DataModel\ExecutableDatabaseConnection.cs:line 65

This is the query that was running:

UP-DEV | Universal.Core.Data.Importer Error finding current record, SQL: SELECT aircraft_config,updated_by,asa_flight_id,atd_utc,bay,terminal,is_cancelled,flight_number,created_at,from_iata,flight_date_utc,std_local,takeoff_time_local,is_international,is_aerocare,cobt_local,customer_atd_local,takeoff_time_utc,std_utc,atd_local,asa_state,aircraft_type,customer_atd_utc,carrier_code,updated_at,gate,service_type,etd_local,cobt_utc,flight_status,etd_utc,aircraft_rego,created_by,id,flight_date,to_iata FROM flight.departure_flight WHERE carrier_code='JQ' AND flight_date_utc='2017-01-20T00:00:00' AND flight_number='57' AND from_iata='MEL'. Database returned: Unknown database 'flight'

Answer 1

While I'm not sure what specifically triggers the problem (it appears that an open connection can somehow end up in a broken state on the RDS cluster side), it appears that the problem is exasperated by the aggressive connection pooling and re-use on the client side happening in the MySQL .NET Connector.

Once the problem occurs, the affected connections appear to just stay in the pool, getting arbitrarily picked back up from the pool and re-used over and over. (Quite likely related to how you see the problem continuing to randomly occur once it has started.)

Adding Connection Lifetime=x to the connection string, x being the time in seconds that a connection is kept alive for re-use, seems to at the very least reduce the symptoms.

Answer 2

So it seems that this error occurs when the Aurora instance cpu is at 100%. The connection cannot find the database specified because it cannot connect to the instance as there are no threads left with which to establish a connection.