How to debug an upstart script that intermittently fails?

Question

I have a process that I want to start as soon my system is rebooted by whatever means so I was using upstart script for that but sometimes what I am noticing is my process doesn't get started up during hard reboot (plugging off and starting the machine) so I think my upstart script is not getting kicked in after hard reboot. I believe there is no runlevel for Hard Reboot.

I am confuse that why sometimes during reboot it works, but sometimes it doesn't work. And how can I debug this out?

Below is my upstart script:

# sudo start helper
# sudo stop helper
# sudo status helper
start on runlevel [2345]
stop on runlevel [!2345]

chdir /data
respawn

pre-start script
  echo "[`date`] Agent Starting" >> /data/agent.log
  sleep 30
end script

post-stop script
  echo "[`date`] Agent Stopping" >> /data/agent.log
  sleep 30
end script

limit core unlimited unlimited
limit nofile 100000 100000
setuid goldy
exec python helper.py

Is there any way to debug this out what's happening? I can easily reproduce this I believe. Any pointers on what I can do here?

Note:

During reboot sometimes I see the logging that I have in pre-start script but sometimes I don't see the logging at all after reboot and that means my upstart script was not triggered. Is there anything I need to change on runlevel to make it work?

I have a VM which is running in a Hypervisor and I am working with Ubuntu.

Answer 1

Your process running nicely, BUT during system startup many things go parallel.

IF mount (which makes available the /data folder) runs later than your pre-start script you will not see the "results" of pre-start script.

I suggest to move sleep 30 earlier (BTW 30 secs seems too looong):

pre-start script
  sleep 30 # sleep 10 should be enough
  echo "[`date`] Agent Starting" >> /data/agent.log
end script