execute ruby as pipeline input for logstash

Question

Logstash allows executing arbitrary commands as inputs for a pipeline. Here is my example pipeline:

input {
  exec {
    command => '/usr/bin/ruby -e "puts RUBY_VERSION"'
    interval => 10
  }
}

output {
  stdout { codec => rubydebug }
}

With this I get the following error:

/opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:328:in `block in verify_gemfile_dependencies_are_found!': Could not find gem 'logstash-core (= 2.4.0) ruby' in any of the gem sources listed in your Gemfile or installed on this machine. (Bundler::GemNotFound)
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:307:in `each'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:307:in `verify_gemfile_dependencies_are_found!'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:199:in `start'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:182:in `resolve'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:192:in `resolve'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:132:in `specs'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:177:in `specs_for'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:166:in `requested_specs'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/environment.rb:18:in `requested_specs'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/runtime.rb:13:in `setup'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler.rb:122:in `setup'
        from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/setup.rb:18:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'

I also tried running logstash's very own JRuby inside the exec like this

input {
  exec {
    command => '/opt/logstash/vendor/jruby/bin/jruby -e "puts RUBY_VERSION"'
    interval => 10
  }
}

output {
  stdout { codec => rubydebug }
}

and it creates another Bundler error:

Bundler::GemNotFound: Could not find gem 'ci_reporter_rspec (= 1.0.0) java' in any of the gem sources listed in your Gemfile or installed on this machine.
  verify_gemfile_dependencies_are_found! at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:328
                                    each at org/jruby/RubyArray.java:1613
  verify_gemfile_dependencies_are_found! at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:307
                                   start at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:199
                                 resolve at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:182
                                 resolve at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:192
                                   specs at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:132
                               specs_for at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:177
                         requested_specs at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:166
                         requested_specs at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/environment.rb:18
                                   setup at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/runtime.rb:13
                                   setup at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler.rb:122
                                  (root) at /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/setup.rb:18
                                 require at org/jruby/RubyKernel.java:1040
                                  (root) at /opt/logstash/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1

Logstash keeps running and the pipeline executes every 10 seconds but the message is always empty. Trying to execute other commands like echo "hello" works without problems.

Running on logstash 2.4.0, openjdk version "1.8.0_102", system ruby is MRI 2.3.1 on Archlinux ARM.

Any ideas?

Answer 1

It looks like you might need to switch to JRuby instead of MRI, which is experimental for Logstash and not currently supported. You might consider RVM or rbenv which can help manage different Ruby versions and switch between JRuby and native as needed in the shell.

Once you are using JRuby + bundler (gem install bundler), you can bundle install from the project root and your dependency issues should be resolved.