perform_async not working in sidekiq

Question

I'm trying to use sidekiq to process some requests in background (it's integrated with padrino framework).

My expectation was that once the worker got called, the request handler would return immediately the answer to user.

In order to be sure it was working properly I implemented a worker that would print some messages and sleep for some 44s prior to finish it's processing (to simulate a long processing in background).

For my surprise, the request got stuck until the worker had the job finished. Only after that the request handler could return an answer to user.

At first I thought that the sleep function could be the problem, so I replaced the sleep function by a busy while but I have the same behavior: the request handler got hanged until the worker finished its task.

Any idea why is this happening?

You can see the following :

request handler:

get :hardworker, map: '/hardworker' do
  logger.info "I'll call HardWorker"
  HardWorker.perform_async(44)
  logger.info "HardWorker was called"
  return "The job should still be running in background."
end

Sidekiq worker:

class HardWorker
  include Sidekiq::Worker

  def perform(count)

    logger.info "Doing hard work"
    Sidekiq.redis { |c| logger.info "redis location: [#{c.client.location }]" }

    redis_info = Sidekiq.redis { |conn| conn.info }
    logger.info "connected clients: [#{redis_info['connected_clients']}]"

    sleep count

    logger.info "hard work was done"
  end
end

The redis server is running:

ps -ef | grep redis

returns

redis     1232     1  0 16:54 ?        00:00:09 /usr/bin/redis-server 127.0.0.1:6379

as well as sidekiq server:

bundle exec sidekiq -r ./config/boot.rb -v
2015-06-06T20:31:26.207Z 3343 TID-8614g INFO: Booting Sidekiq 3.3.4 with redis options {:url=>"redis://127.0.0.1:6379/0", :concurrency=>25}

Also, from logs I put in the worker we can see that client apparently is connected to redis:

INFO -  redis location: [127.0.0.1:6379]
INFO -  connected clients: [3]

sidekiq version: 3.3.4 redis server: 2.8.4 ruby: 2.1.2p95

Note: I mounted sidekiq web tool on my padrino and it shows 0 for all stats (Processed, Failed, Busy, Enqueued, Retries, Scheduled, Dead) both prior and after the worker has executed.

The sample project can be downloaded from here: https://github.com/plicatibu/sidekiq-problem.git

Answer 1

Well I found your bug, in your boot.rb file you have this line:

require 'sidekiq/testing/inline' if RACK_ENV == 'development'

This bit of code, uses Sidekiq's Testing framework which bypasses redis and runs it almost as if it's a ruby class (ie it doesn't ever get queued on redis). Remove that line and only use it in your test suite (if you need it).

You'll also need to remove that pid file from your YML file, you don't need that and if you do, it's probably only for production. My 2cents - remove it.

Once you do that, you're good to go:

Here's your rackup with a get request to /hardworker:

[2015-06-07 07:54:25] INFO  WEBrick 1.3.1
[2015-06-07 07:54:25] INFO  ruby 2.2.2 (2015-04-13) [x86_64-darwin14]
[2015-06-07 07:54:25] INFO  WEBrick::HTTPServer#start: pid=74128 port=9292
   INFO -  I'll call HardWorker
   INFO -  HardWorker was called
  DEBUG -       GET (0.0131s) /hardworker - 200 OK
::1 - - [07/Jun/2015:07:55:32 -0400] "GET /hardworker HTTP/1.1" 200 46 0.0291

Here's sidekiq processing that job in the background:

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
          ss
     sss  sss         ss
     s  sss s   ssss sss   ____  _     _      _    _
     s     sssss ssss     / ___|(_) __| | ___| | _(_) __ _
    s         sss         \___ \| |/ _` |/ _ \ |/ / |/ _` |
    s sssss  s             ___) | | (_| |  __/   <| | (_| |
    ss    s  s            |____/|_|\__,_|\___|_|\_\_|\__, |
    s     s s                                           |_|
          s s
         sss
         sss

   INFO -  Running in ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-darwin14]
   INFO -  See LICENSE and the LGPL-3.0 for licensing details.
   INFO -  Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org/pro
   INFO -  Starting processing, hit Ctrl-C to stop
  DEBUG -
   INFO -
   INFO -  Doing hard work
   INFO -  redis location: [127.0.0.1:6379]
   INFO -  connected clients: [3]
   INFO -  hard work was done
   INFO -