capturing stdout from python in an upstart job

Question

I'm struggling to save the stdout output of a python2.6 script into a file when I run it as an upstart job.

This is my current test script:

#!/usr/bin/python
from time import sleep
import sys

while 1:
    print "Hello from stdout"
    print >> sys.stderr, "Hello from stderr"
    sleep(1)

if I do $ myscript >> /var/log/myscript that works fine. I see both.

However, as an upstart script, if I use the following conf

1. /etc/init/myscript.conf:

   exec /path/to/myscript >> /var/log/myscript 2>&1
   

   I see this:

   Hello from stderr
   Hello from stderr
   Hello from stderr
   Hello from stderr
   Hello from stderr
   Hello from stderr
   

If I use a different file for stderr and stdout, they are both created, but the stdout is always empty.

What am I doing wrong?

Answer 1

Buffering and flushing would be the guess.

stderr is typically an unbuffered stream while stdout is buffered. When you send it to a file it will be buffered in 4k chunks (typically).

If you make the following change:

while 1:
    print "hello from stdout"
    sys.stdout.flush()
    print >> sys.stderr, "Hello from stderr"
    sleep(1)

You should see output in your file.