go/golang + redis too many open files error

Question

I'm using redis in Golang with the Redigo connector (https://github.com/garyburd/redigo) suggested by the Redis website.

I have:

• After every Dial() I defer a Close()
• Set fs.file-max = 100000
• Set vm.overcommit_memory = 1
• Disabled saving
• Set maxclients = 100000

I run a high traffic website, and everything runs great for about 10 minutes, from which I get

error: dial tcp 127.0.0.1:6379: too many open files

Then I can't access redis at all from my application.

I see nothing in the redis logs to suggest any errors or problems. What do I do to fix this?

Answer 1

I don't know which driver you use but with redigo you can define a number of open connections in a pool, all you have to do then is in every query to redis, first get a client from the pool, then close it, so it goes back to the pool and gets re-used, just like this:

redisPool = &redis.Pool{
        MaxIdle: 3,
        MaxActive: 10, // max number of connections
        Dial: func() (redis.Conn, error) {
                c, err := redis.Dial("tcp", ":6379")
                if err != nil {
                        panic(err.Error())
                }
                return c, err
        },
}
r := redisPool.Get() // get a client from the pool
_, err = r.Do("GET one") // use the client
if err != nil {
        panic(err.Error())
}
r.Close() // close the client so the connection gets reused

Answer 2

Your problem is that Redis cant open new connections and then becomes unresponsive, you need to increase the file descriptors limit of your operating system (ubuntu defaults to 1024 which can be a problem) which right now is dominating the redis maxclients setting.

The way you tweak this limit depends on the os redis is running, on linux you need something like this:

ulimit -n 99999