Using ruby timeout in a thread making a database call

Question

I am using Ruby 1.9.2.

I have a thread running which makes periodic calls to a database. The calls can be quite long, and sometimes (for various reasons) the DB connection disappears. If it does disappear, the thread just silently hangs there forever.

So, I want to wrap it all in a timeout to handle this. The problem is, on the second time through when a timeout should be called (always second), it still simply hangs. The timeout never takes effect. I know this problem existed in 1.8, but I was lead to believe timeout.rb worked in 1.9.

t = Thread.new do
  while true do
    sleep SLEEPTIME
    begin
      Timeout::timeout(TIMEOUTTIME) do
        puts "About to do DB stuff, it will hang here on the second timeout"
        db.do_db_stuff()
        process_db_stuff()
      end
    rescue Timeout::Error
      puts "Timed out"
      #handle stuff here
    end
  end
end

Any idea why this is happening and what I can do about it?

Answer 1

One possibility is that your thread does not hang, it actually dies. Here's what you should do to figure out what's going on. Add this before you create your worker thread:

Thread.abort_on_exception = true

When an exception is raised inside your thread that is never caught, your whole process is terminated, and you can see which exception was raised. Otherwise (and this is the default), your thread is killed.

If this turns out not to be the problem, read on...

Ruby's implementation of timeouts is pretty naive. It sets up a separate thread that sleeps for n seconds, then blindly raises a Timeout exception inside the original thread.

Now, the original code might actually be in the middle of a rescue or ensure block. Raising an exception in such a block will silently abort any kind of cleanup code. This might leave the code that times out in an improper state.

It's quite difficult to tell if this is your problem exactly, but seeing how database handlers might do a fair bit of locking and exception handling, it might be very likely. Here's an article that explains the issue in more depth.

Is there any way you can use your database library's built-in timeout handling? It might be implemented on a lower level, not using Ruby's timeout implementation.

A simple alternative is to schedule the database calls in a separate process. You can fork the main process each time you do the heavy database-lifting. Or you could set up a simple cronjob to execute a script that executes it. This will be slightly more difficult if you need to communicate with your main thread. Please leave some more details if you want any advice on which option might suit your needs.

---

Based on your comments, the thread is dying. This might be a fault in libraries or application code that you may or may not be able to fix. If you wish to trap any arbitrary error that is generated by the database handling code and subsequently retry, you can try something like the following:

t = Thread.new do
  loop do
    sleep INTERVAL
    begin
      # Execute database queries and process data
    rescue StandardError
      # Log error or recover from error situation before retrying
    end
  end
end

You can also use the retry keyword in the rescue block to retry immediately, but you probably should keep a counter to make sure you're not accidentally retrying indefinitely when an unrecoverable error keeps occurring.