comparing Python code for equivalence

Question

Is there a reliable, automatic way (such as a command-line utility) to check if two Python files are equivalent modulo whitespace, semicolons, backslash continuations, comments, etc.? In other words, that they are identical to the interpreter?

For example, this:

import sys
sys.stdout.write('foo\n')
sys.stdout.write('bar\n')

should be considered equivalent to this:

import   sys
sys.stdout.\
    write('foo\n'); sys.stdout.\
    write(

    'bar\n') # This is an unnecessary comment

Answer 1

Use the ast module.

Example (for Python 2):

import ast

x = r'''import sys
sys.stdout.write('foo\n')
sys.stdout.write('bar\n')'''

y = r'''import   sys
sys.stdout.\
    write('foo\n'); sys.stdout.\
    write(

    'bar\n') # This is an unnecessary comment'''

xd = ast.dump(ast.parse(x))
yd = ast.dump(ast.parse(y))
print xd == yd

You can of course read in the source code from actual files instead of string literals.

Edit:

So that the comments make sense, I'd like to note that I originally proposed using the built-in compile() function. However, @Jian found a simple case that it didn't handle well. Perhaps it could be adapted, as suggested by @DSM, but then the solution becomes a little less tidy. Maybe not unreasonably so, but if the ast parse-and-dump works as well or better, it's the more straightforward way.

Answer 2

Use python's parser:

In [1]: import parser

In [2]: with open('file1.py', 'r') as f1:
    st1 = parser.suite(f1.read())

In [3]: with open('file2.py', 'r') as f2:
    st2 = parser.suite(f2.read())

In [4]: st1.compile() == st2.compile()
Out[4]: True

Answer 3

Python includes its own parser. Apply it to both files, then check that the result is structurally equivalent.