Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

God starts too many processes

Tags:

I have a god script that is supposed to keep an eye on two stalker processes. The problem is that after 24h it starts way too many processes.

This is the god script.

rails_root = File.expand_path("../..", __FILE__)  2.times do |n|   God.watch do |w|     w.group = "app-scripts"     w.name  = "run-#{n}"     w.interval = 30.seconds     w.dir      = File.dirname(__FILE__)      w.env = {       "BUNDLE_GEMFILE" => "#{rails_root}/Gemfile",       "RAILS_ENV" => "production",       "BEANSTALK_URL" => "beanstalk://127.0.0.1:54132"     }      w.start = "bbundle exec stalk #{File.join(rails_root, "config/jobs.rb")}"      w.start_grace = 5.seconds     w.stop_grace  = 5.seconds      w.start_if do |start|       start.condition(:process_running) { |c| c.running = false }     end      w.restart_if do |restart|       restart.condition(:memory_usage) do |c|         c.above = 200.megabytes         c.times = [3, 5]       end        restart.condition(:cpu_usage) do |c|         c.above = 95.percent         c.times = 5       end     end      w.lifecycle do |on|       on.condition(:flapping) do |c|         c.to_state = [:start, :restart]         c.times = 5         c.within = 5.minute         c.transition = :unmonitored         c.retry_in = 10.minutes         c.retry_times = 5         c.retry_within = 2.hours       end     end   end end 

ps aux | grep stalk returns the following.

root      3178  0.2  2.7 417580 117284 ?       Sl   Oct28   2:22 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root      3179  0.2  3.3 506068 138740 ?       Sl   Oct28   2:26 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root      4588  0.2  2.9 497932 121664 ?       Sl   Oct25  16:10 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root      4794  0.2  3.0 497792 128084 ?       Sl   Oct25  15:57 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     10391  0.2  2.8 496784 121388 ?       Sl   Oct25  15:44 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     10392  0.2  2.8 497624 121528 ?       Sl   Oct25  15:31 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     18874 75.0  2.0 214116 83948 ?        Rl   15:49   0:09 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     18875 75.0  2.0 214944 84868 ?        Rl   15:49   0:09 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     20649  0.2  2.6 410636 110012 ?       Sl   Oct28   2:44 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     20650  0.2  3.0 439284 128996 ?       Sl   Oct28   2:47 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     23272  0.2  2.7 414452 115772 ?       Sl   Oct28   2:44 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     23273  0.2  2.7 417728 117152 ?       Sl   Oct28   2:44 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     25919  0.2  3.1 436276 131876 ?       Sl   Oct28   2:28 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     25920  0.2  3.3 503236 138676 ?       Sl   Oct28   2:29 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     28782  0.2  2.8 431836 121108 ?       Sl   Oct25  16:58 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     30687  0.2  2.7 415908 117008 ?       Sl   Oct28   2:39 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb root     30688  0.2  2.6 476184 111844 ?       Sl   Oct28   2:37 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb 

This is the /usr/bin/bbundle script.

#!/usr/bin/env bash  if [[ -s "/home/webmaster/.rvm/environments/ruby-1.9.2-p320@webmaster" ]] then   source "/home/webmaster/.rvm/environments/ruby-1.9.2-p320@webmaster"   bundle  "$@" else   echo "ERROR: Missing RVM environment file: '/home/webmaster/.rvm/environments/ruby-1.9.2-p320@webmaster'" >&2   exit 1 fi 
  • Running sudo god stop app-scripts won't kill any processes.

  • I've tried adding w.uid = "webmaster" to the god script, but the problem remains.

  • I'm running god version 0.12.1, ruby version 1.9.3p286 and stalker version 0.9.0.

What am I doing wrong?

like image 453
Linus Oleander Avatar asked Oct 24 '12 22:10

Linus Oleander


People also ask

What does it mean to be processed by God?

A process is a series of actions or steps taken in order to achieve a particular end. God knew that some things needed to happen in David's life before he could be king. Likewise, there are some things that need to happen in our lives before we can step into what God has for us.

What does it mean when God is refining you?

God sees us not full of impurity, but full of potential. He, in the refining imagery, is the refiner and we are the lump of unrefined gold, full of impurity and full of potential beauty. Sometimes we expect God simply zaps the junk out of our lives when we enter into relationship with him.

Is God's timing perfect?

Lesson 1: God's timing is always perfect. God's never early, never late, but always on time. Our timing isn't God's timing. For us, God's timing often feels like a long, desperate delay.

What is the meaning of Matthew 6 25?

Analysis. Jesus has just told his followers that it is impossible to serve both God and wealth. In this verse he begins a discussion of why one should not be over anxious about all material things. The word “more than” in this context means that the first object being compared exists even without the second.


2 Answers

It seems like god is trying to follow bbundle and not stalk. You need to let god know where to find the PID of the actual process you want to follow with w.pid_file. You might also need to tell it how to kill the process, if a standard kill won't do the trick. For that you can use w.stop_signal for a different signal (as simonmenke suggested) or w.stop for a whole other command.

The log file should shed more light on what's going on. Call god -D to print it to stdout or god -l /var/log/god.log.

like image 197
kichik Avatar answered Sep 27 '22 01:09

kichik


Stacker stops when it receives a INT signal (not a TERM signal). Try adding a stop signal:

# ... w.stop_signal = 'INT' # ... 
like image 36
simonmenke Avatar answered Sep 23 '22 01:09

simonmenke