Ruby on Rails loopback requests cause deadlock?

Question

Solution

I am switching to unicorn in development mode. One worker process for each level of recursion is needed to prevent the deadlock situation, so I am running with 2 worker processes.

Problem

I am working on the Thin server in my development environment. I use port 3000 (the default in dev environment). My problem is getting the server to make requests to itself.

Let's say I have the following controller:

# app/controllers/recursions_controller.rb
class RecursionsController < ApplicationController

    # /recursions
    def index

        # synchronously call recursions#show
        RestClient.get("http://localhost:3000/recursions/1") 

        # finish!
        render :text => 'index'

    end

    # /recursions/:id
    def show

        # finish immediately
        render :text => 'show'

    end

end

Here is the corresponding route:

# config/routes.rb
resources :recursions

Here is output from the request log when I initially request recursions#index:

[INFO] 2013-01-15 12:09:05 -0800 Started GET "/recursions" for 127.0.0.1 at 2013-01-15 12:09:05 -0800
Processing by RecursionsController#index as HTML
Completed 500 Internal Server Error in 60049ms

[FATAL] 2013-01-15 12:10:05 -0800 RestClient::RequestTimeout (Request Timeout):
  app/controllers/recursions_controller.rb:8:in `index'
  Rendered /usr/local/lib/ruby/gems/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/templates/rescues/_trace.erb (0.8ms)
  Rendered /usr/local/lib/ruby/gems/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (0.7ms)
  Rendered /usr/local/lib/ruby/gems/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/templates/rescues/diagnostics.erb within rescues/layout (7.4ms)

[INFO] 2013-01-15 12:10:05 -0800 

[INFO] 2013-01-15 12:10:05 -0800 

[INFO] 2013-01-15 12:10:05 -0800 Started GET "/recursions/1" for 127.0.0.1 at 2013-01-15 12:10:05 -0800
Processing by RecursionsController#show as XML
  Parameters: {"id"=>"1"}
  Rendered text template (0.0ms)
Completed 200 OK in 7ms (Views: 5.5ms)

I suspect what is going on here is some kind of deadlock situation. Request A cannot return until request B returns (recursive ordering, can't help it), but request B cannot be processed until request A returns (apparent limitation built into my webserver?). The deadlock is resolved when RestClient times out, causing an exception and terminating request A with a 500. Only then is request B handled, though it's moot at this point.

It seems to me like my webserver cannot handle concurrent requests. That said, here are my questions:

1. Is there a webserver I can switch to in my dev environment that is not limited this way? We use Unicorn in production which can spawn multiple worker processes and therefore handle concurrent requests, but Unicorn seems too heavy for a dev environment. The same thing that makes Unicorn a solution to my problem could make it difficult to read the log output. This is my last resort solution.

2. Is there a tricky way to make a request to the Rails/Rack frameworks that circumvents the apparent concurrent request limitation?

3. Can anyone provide me with docs stating this limation explicitly? I don't know if it's a limitation inherent to all single-process Ruby on Rails webservers or just Thin.

Note: This is just a toy problem to demonstrate the blocking issue I am having. If you need to know the actual reason for this, it is that I am moving some HTTP services from another part of our infrastructure into our RoR app. Our RoR app consumes these service at many different points in our code, so I am trying to leave the client code for these services untouched and only change the implementation. That means making circular HTTP requests. This will be optimized at a later date, when everything has stabilized.

Answer 1

You are correct that by default your app will only process one request at a time. This limitation is implemented by the Rack::Lock middleware - it will be there no matter what webserver you use.

You can change this by calling config.thread_safe! in the appropriate environment.rb file. However rails' development mode code reloading is not thread safe and is disabled by this setting, which makes it not really usable in development. Exactly what config.thread_safe! does is described in the rails configuration guide.

passenger, pow, unicorn all make it easy to run multiple instances.

Answer 2

I ran into this exact problem with one of my clients. It's exactly as you describe - since you're running a single-threaded webserver with only one process, you are limited to 1 client at a time. If you make a second request, or a loopback request as it were, your 2nd request is queued by the OS until the first times out.

Switching to unicorn doesn't intrinsically fix the problem, though in our case we were on unicorn and we simply bumped the worker count to '2'. I'm not familiar with thin enough to recommend how to increase the worker count, but no doubt there is a way. Otherwise, use unicorn - you'd also get better dev/prod parity if you made the switch, so it might be worthwhile.