What is the Python equivalent of `set -x` in shell?

Question

Please suggest Python command which is equivalent of set -x in shell scripts.

Is there a way to print/log each source file line executed by Python?

Answer 1

You can use the trace module:

python -m trace -t your_script.py

The command line above will display every line of code as it is executed.

Answer 2

to get a proper equivalent of bash -x using the trace module, one needs to use --ignore-dir to block the printing of source lines of every module imported, e.g. python -m trace --trace --ignore-dir /usr/lib/python2.7 --ignore-dir /usr/lib/pymodules repost.py, adding more --ignore-dir directives as necessary for other module locations.

this becomes important when attempting to locate slow-loading modules such as requests which spit out millions of source lines for several minutes on a slow machine. the proper use of --ignore-dir cuts the time down to a few seconds, and shows only the lines from your own code.

$ time python -m trace --trace repost.py 2>&1 | wc
3710176 16165000 200743489

real    1m54.302s
user    2m14.360s
sys 0m1.344s

vs.

$ time python -m trace --trace --ignore-dir /usr/lib/python2.7 --ignore-dir /usr/lib/pymodules repost.py 2>&1 | wc
     42     210    2169

real    0m12.570s
user    0m12.421s
sys 0m0.108s

this doesn't really answer your question; you asked for a Python equivalent of set -x. a simple way to approximate that is with sys.settrace():

jcomeau@aspire:/tmp$ cat test.py
#!/usr/bin/python -OO
'''
test program for sys.settrace
'''
import sys, linecache
TRACING = []

def traceit(frame, event, arg):
    if event == "line":
        lineno = frame.f_lineno
        line = linecache.getline(sys.argv[0], lineno)
        if TRACING:
            print "%d: %s" % (lineno, line.rstrip())
    return traceit

def test():
    print 'this first line should not trace'
    TRACING.append(True)
    print 'this line and the following ought to show'
    print "that's all folks"
    TRACING.pop()
    print 'this last line should not trace'

if __name__ == '__main__':
    sys.settrace(traceit)
    test()

which, when run, gives:

jcomeau@aspire:/tmp$ ./test.py
this first line should not trace
19:     print 'this line and the following ought to show'
this line and the following ought to show
20:     print "that's all folks"
that's all folks
21:     TRACING.pop()
this last line should not trace

eliminating the line 'TRACING.pop()' from the trace output is left as an exercise for the reader.

sources: https://pymotw.com/2/sys/tracing.html and http://www.dalkescientific.com/writings/diary/archive/2005/04/20/tracing_python_code.html

Answer 3

I liked @jcomeau_ictx's answer very much, but it has a small flaw, which is why I extended on it a bit. The problem is that jcomeau_ictx's 'traceit' function only works correctly if all code to be traced is within the file that is called with python file.py (let's call it the host file). If you call any imported functions, you get a lot of line numbers without code. The reason for this is that line = linecache.getline(sys.argv[0], lineno) always tries to get the line of code from the host file (sys.argv[0]). This can easily be corrected, as the name of the file that actually contains the traced line of code can be found in frame.f_code.co_filename. This now may produce a lot of output, which is why one probably would want to have a bit more control.

There is also another bit to notice. According to the sys.settrace() documentation:

The trace function is invoked (with event set to 'call') whenever a new local scope is entered

In other words, the code to be traced has to be inside a function.

To keep everything tidy, I decided to put everything into an own file called setx.py. The code should be pretty self-explanatory. There is, however, one piece of code that is needed for Python 3 compatibility, which deals with the differences between Python 2 and 3 with respect to how modules are imported. This is explained here. The code should now also work both with Python 2 and 3.

##setx.py
from __future__ import print_function
import sys, linecache

##adapted from https://stackoverflow.com/a/33449763/2454357
##emulates bash's set -x and set +x

##for turning tracing on and off
TRACING = False

##FILENAMES defines which files should be traced
##by default this will on only be the host file 
FILENAMES = [sys.argv[0]]

##flag to ignore FILENAMES and alwas trace all files
##off by default
FOLLOWALL = False

def traceit(frame, event, arg):
    if event == "line":
        ##from https://stackoverflow.com/a/40945851/2454357
        while frame.f_code.co_filename.startswith('<frozen'):
            frame = frame.f_back
        filename = frame.f_code.co_filename
##        print(filename, FILENAMES)
        if TRACING and (
            filename in FILENAMES or
            filename+'c' in FILENAMES or
            FOLLOWALL
        ):
            lineno = frame.f_lineno
            line = linecache.getline(filename, lineno)
            print("{}, {}: {}".format(filename, lineno, line.rstrip()))
    return traceit

sys.settrace(traceit)

I then test the functionality with this code:

##setx_tester.py
from __future__ import print_function
import os
import setx
from collections import OrderedDict

import file1
from file1 import func1
import file2
from file2 import func2

def inner_func():
    return 15

def test_func():

    x=5
    print('the value of x is', x)

    ##testing function calling:
    print('-'*50)
    ##no further settings
    print(inner_func())
    print(func1())
    print(func2())

    print('-'*50)
    ##adding the file1.py to the filenames to be traced
    ##it appears that the full path to the file is needed:
    setx.FILENAMES.append(file1.__file__)
    print(inner_func())
    print(func1())
    print(func2())

    print('-'*50)
    ##restoring original:
    setx.FILENAMES.pop()

    ##setting that all files should be traced:
    setx.FOLLOWALL = True
    print(inner_func())
    print(func1())
    print(func2())

##turn tracing on:
setx.TRACING = True
outer_test = 42  ##<-- this line will not show up in the output
test_func()

The files file1.py and file2.py look like this:

##file1.py
def func1():
    return 7**2

and

##file2.py
def func2():
    return 'abc'*3

The output then looks like this:

setx_tester.py, 16:     x=5
setx_tester.py, 17:     print('the value of x is', x)
the value of x is 5
setx_tester.py, 20:     print('-'*50)
--------------------------------------------------
setx_tester.py, 22:     print(inner_func())
setx_tester.py, 12:     return 15
15
setx_tester.py, 23:     print(func1())
49
setx_tester.py, 24:     print(func2())
abcabcabc
setx_tester.py, 26:     print('-'*50)
--------------------------------------------------
setx_tester.py, 29:     setx.FILENAMES.append(file1.__file__)
setx_tester.py, 30:     print(inner_func())
setx_tester.py, 12:     return 15
15
setx_tester.py, 31:     print(func1())
**path to file**/file1.py, 2:     return 7**2
49
setx_tester.py, 32:     print(func2())
abcabcabc
setx_tester.py, 34:     print('-'*50)
--------------------------------------------------
setx_tester.py, 36:     setx.FILENAMES.pop()
setx_tester.py, 39:     setx.FOLLOWALL = True
setx_tester.py, 40:     print(inner_func())
setx_tester.py, 12:     return 15
15
setx_tester.py, 41:     print(func1())
**path to file**/file1.py, 2:     return 7**2
49
setx_tester.py, 42:     print(func2())
**path to file**/file2.py, 2:     return 'abc'*3
abcabcabc