Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Reconnect with Redis after Puma fork

I am using a global variable in a rails application to store a redis client using the redis gem. In a config/initializers/redis.rb, I have

$redis = Redis.new(host: "localhost", port: 6379)

Then in application code, I use $redis to work the data in the Redis store.

I also use puma as the web server in production environment, and capistrano to deploy code. In the deploy process, capistrano restarts puma.

Every time I start or restart the puma web servers, I always get an "Internal Server Error" when I first use $redis to access data in the Redis store. I saw errors like Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking.)

Searching around with google and stackoverflow led me to think that I needed to reconnect to Redis after puma forks child processes. So, I added in my config/puma.rb:

on_worker_boot do
  $redis.ping
end

But I was still getting the "Internal Server Error" caused by Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking.).

I saw this post http://qiita.com/yaotti/items/18433802bf1720fc0c53. I then tried adding in config/puma.rb:

on_restart do
  $redis.quit
end

That did not work.

I tried in config/initializers/redis.rb to $redis.ping right after Redis.new. That did not work either.

I got this error if puma was started with no puma processes running, or restarted when an instance of puma process was running.

Refreshing the page would get me past this error. But I want to get rid of this even on the first attempt to use $redis. I was thinking that I did not use the redis gem or configure its reconnection correctly. Could someone tell me:

  1. Is that the right way to use redis gem in a rails application?
  2. How should the redis connection be reconnected in puma?

puma gem documentation says, "You should place code to close global log files, redis connections, etc in this block so that their file descriptors don't leak into the restarted process. Failure to do so will result in slowly running out of descriptors and eventually obscure crashes as the server is restart many times." It was talking about the on_restart block. But it did not say how that should be done.

like image 294
hau Avatar asked Oct 22 '22 01:10

hau


1 Answers

I was able to fix the error with a monkeypatch. This changes the behaviour so it just reconnects instead of throwing the Redis::InheritedError

###### MONKEYPATCH redis-rb 
# https://github.com/redis/redis-rb/issues/364
# taken from https://github.com/redis/redis-rb/pull/389/files#diff-597c124889a64c18744b52ef9687c572R314
class Redis
  class Client
   def ensure_connected
      tries = 0

      begin
        if connected?
          if Process.pid != @pid
            reconnect
          end
        else
          connect
        end

        tries += 1

        yield
      rescue ConnectionError
        disconnect

        if tries < 2 && @reconnect
          retry
        else
          raise
        end
      rescue Exception
        disconnect
        raise
      end
    end
  end
end
## MONKEYPATCH end
like image 119
Simon Woker Avatar answered Oct 24 '22 11:10

Simon Woker