Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Number of active connections on the server reached to max

I am working with mongodb and nodejs. I have mongodb hosted on Atlas.

My backend had been working perfectly but now it is sometimes getting stuck and when I see the analytics on mongodb atlas it shows maximum number of active connections reached to 100.

enter image description here

Can someone please explain why this is happening? Can I reboot the connections and make it 0?

@Stennie I have used mongoose to connect to database

Here is my configuration file

const mongooseOptions = {
  useNewUrlParser: true,
  autoReconnect: true,
  poolSize: 25,
  connectTimeoutMS: 30000,
  socketTimeoutMS: 30000
} 

exports.register = (server, options, next) => {
  defaults = Hoek.applyToDefaults(defaults, options)

  if (Mongoose.connection.readyState) {
    return next()
  }

  if (!Mongoose.connection.readyState) {
    server.log(`${process.env.NOED_ENV} server connecting to ${defaults.url} ${defaults.url}`)

    return Mongoose.connect(defaults.url, mongooseOptions).then(() => {
      return next() // call the next item in hapi bootstrap
    })
  }
}
like image 233
Profer Avatar asked May 30 '19 07:05

Profer


People also ask

What is the maximum number of connections a server can handle?

On the TCP level the tuple (source ip, source port, destination ip, destination port) must be unique for each simultaneous connection. That means a single client cannot open more than 65535 simultaneous connections to a single server. But a server can (theoretically) serve 65535 simultaneous connections per client.

How do I find the maximum number of connections in SQL Server?

In Object Explorer, right-click a server and select Properties. Select the Connections node. Under Connections, in the Max number of concurrent connections box, type or select a value from 0 through 32767 to set the maximum number of users that are allowed to connect simultaneously to the instance of SQL Server.

What is Max connections per host?

So depending on what browser is used, the limit on the number of connections per host will range anywhere from 2 to 13, with a maximum number of connections ranging up to 60.

How many connections SQL Server can handle?

By default, SQL Server allows a maximum of 32767 concurrent connections which is the maximum number of users that can simultaneously log in to the SQL server instance.


3 Answers

There are so many factors affecting the max connection limit. You have mongoDB hosted on Atlas and as you mentioned the backend is lamda means you have a serverless environment.

  1. Serverless environments spawn new container on the new connection and destroy a connection when it's no longer being used. The peak connection shows that there are so many new instances being initialized or so many concurrent requests from the user connection. The best practice is to terminate database connection once it's no longer needed. You can terminate the connection mongoose.connection.close(); as you have used mongoose. It will release the connection from the connection pool. Rather exhausting the concurrent connection limit, you should release connection once it's idle.
  2. Your configuration forces the database driver to reconnect after the connection is dropped by the database. You are explicitly setting the autoReconnect as true so the driver will quickly instantiate connection request once the connection is dropped. That may affect the concurrent connection limit. You should avoid setting it explicitly.
  3. cluster mode can optimize the requests according to the load, you can change the server uri to the replica of database. it may help to migrate the load.
  4. There is a small initial startup cost of approximately 5 to 10 seconds when the Lambda function is invoked for the first time and the MongoDB client in your AWS Lambda function connects to MongoDB. Connections to a mongos for a sharded cluster are faster than connecting to a replica set. Subsequent connections will be significantly faster for the duration of the lifecycle of the Lambda function. so Each invocation will leave a container idle to prevent cold start or cold boot, or use an existing one if available.
  5. Atlas sets the limit for concurrent incoming connections to a cluster based on the cluster tier. If you try to connect when you are at this limit, MongoDB displays an error stating “connection refused because too many open connections”. You can close any open connections to your cluster not currently in use. scaling down to a higher tier to support more concurrent connections. as mentioned in best practice you may restart the application. To prevent this issue in the future, consider utilizing the maxPoolSize connection string option to limit the number of connections in the connection pool.
  6. Final Solution to this issue is Upgrading to a larger Atlas cluster tier which allows a greater number of connections. if your user base is too large for your current cluster tier.
like image 144
Kiran Maniya Avatar answered Oct 13 '22 05:10

Kiran Maniya


Assuming your backend is deployed on lambda since serverless tag.

Each invocation will leave a container idle to prevent cold start, or use an existing one if available. You are leaving the connection open to reuse it between invocation, like advertised in best practices.

With a poolSize of 25 (?) and 100 max connections, you should limit your function concurrency to 4.

Reserve concurrency to prevent your function from using all the available concurrency in the region, or from overloading downstream resources.

More reading: https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs

like image 43
Gabriel Bleu Avatar answered Oct 13 '22 04:10

Gabriel Bleu


You could try couple of things:

  1. In a serverless environment, as already suggested by @Gabriel Bleu, why have such a high connectionLimit. Serverless environment keeps spawning new containers and stopping as per requests. If multiple instances spawn concurrently, it would exhaust the MongoDB server limit very quickly.

  2. The concept of connectionPool is, x number of connections are established every time from every node (instance). But that does not mean all the connections are automatically released after querying. After completing ALL the DB operation, you should release each connection individual after use: mongoose.connection.close();

Note: Mongoose connection close will close all the connections of connection pool. So ideally, this should be run just before returning the response.

  1. Why are you setting explicity autoReconnect to true. MongoDB driver internally reconnects whenever the connection is lost and certainly is not recommended for short lifespan instances such as serverless containers.

  2. If you are running in cluster mode, to optimize for performance, change the serverUri to replica set URL format: MONGODB_URI=mongodb://<username>:<password>@<hostOne>,<hostTwo>,<hostThree>...&ssl=true&authSource=admin.

like image 6
Sagar Chilukuri Avatar answered Oct 13 '22 06:10

Sagar Chilukuri