Jenkins not printing output of python script in console

Question

I have a python script(myscript.py) as follows:

#!/bin/python
import os
import optparse
import subprocess
import sys
sys.stdout.flush()

print("I can see this message on Jenkins console output")
cmd="sshpass -p 'xxx' ssh test@testmachine 'cmd /c cd C:\stage && test.bat'"
retval=subprocess.call(cmd,shell=True)
print retval

In jenkins, I have a job with execute shell as follows:

#!/bin/sh
./myscript.py

Problem: Jenkins console shows only "I can see this message on Jenkins console output". If there is any output from the subprocess call, it does not print it out on the console.

If I putty to Server A and run the same command (./myscript.py) on shell, I can see the output of subprocess call.

How can I print this output of subprocess call on Jenkins console?

FYI: As you can see from my command, the subprocess call is running a batch file on windows; Jenkins is running on Linux; There is ssh setup between the two machines..

Edit: My test.bat looks like this:

echo off
RMDIR /S /Q C:\Test

IF %ERRORLEVEL% NEQ 0 (
  ECHO Could not delete
  EXIT /b %ERRORLEVEL%
)

if I run this batch file locally on windows server, it returns a 1 ( because am holding a file open in Test folder )

But when the python script calls this batch file using the subprocess call, all i get is a Zero for retval.

Why is this and how to fix this? If I can capture the correct retval, I can make the Jenkins job fail.

Edit 12/12: Helllo!! Anybody! Somebody! Help!

Answer 1

I wonder if it has to do anything with stdout being buffered Can you try setting PYTHONUNBUFFERED before running your command?

export PYTHONUNBUFFERED=true

Answer 2

In my Jenkins environment, executing python scripts with the unbuffered argument makes the output appear immediately. Like this:

python3 -u some_script.py

More information comes from the help menu (python3 --help):

-u     : force the stdout and stderr streams to be unbuffered;
         this option has no effect on stdin; also PYTHONUNBUFFERED=x

Answer 3

TL; DR

The fix is to use some conditional execution (the || operator) on rmdir to fix the errorlevel being returned.

Investigation

This was a corker of a bug, with quite a few twists and turns! We initially suspected that the stdout chain was broken somehow, so looked into that through explicit use of pipes in Popen and then removing sshpass from your command and so using the output from ssh directly.

However, that didn't do the trick, so we moved on to looking at the return code of the command. With sshpass removed, ssh should return the result of the command that was run. However, this was always 0 for you.

At this point, I found a known bug in Windows that rmdir (which is the same as rd) doesn't always set errorlevel correctly. The fix is to use some conditional execution (the || operator) on rmdir to fix up the errorlevel.

See batch: Exit code for "rd" is 0 on error as well for full details.

Answer 4

When you execute your script in a shell, Python sets your shell's STDOUT as the subprocess's STDOUT, so everything that gets executed gets printed to your terminal. I'm not sure why, but when you're executing in Jenkins the subprocess is not inheriting the shell's STDOUT so its output is not displayed.

In all likelihood, the best way to solve your problem will be to PIPE the STDOUT (and STDERR for good measure) and print it after the process ends. Also, if you exit with the exit code of your subprocess and the exit code is not 0, it will likely terminate your Jenkins job.

p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                     stderr=subprocess.PIPE, shell=True)
exit_code = p.wait()  # wait for it to end
print('Got the following output from the script:\n', p.stdout.read().decode())
print('Got the following errors from the script:\n', p.stderr.read().decode())
print('Script returned exit code:', exit_code)

sys.exit(exit_code)