Parsing Python Code From Within Python? [closed]

Question

We have an older C++ tool that generates some python code automatically. I tried to slog through the C++ source tool, today and pretty much wanted to shoot my self. The thing is what i want to do, is clean up the source created by the tool and link the classes to our internal documentation system via adding sphinx tags.

Now what i am wondering is there some sort of wonderful tool for parsing python code within python?

There is alot of stuff like this:

foo._methods_.append()

Snip 500 lines

foo._methods_.append()

ANy suggestions?

Basically i have a functional but insanely messy code structure, i basically want to extract certain chunks, move them to their own files. And cleanup all the miscellanous things that get created.

I looked quickyl at both parser and AST but i cannot find any real examples of it being used.

Answer 1

You may tokenize python code to parse individual tokens using tokenize module. e.g. Script to remove Python comments/docstrings

or you can use the parser module

or use ast module

Answer 2

Depending on your needs, you may also want to check out the 2to3 library. It was written to automatically facilitate the conversion of Python 2.x apps to Python 3.0, so its main use case is taking one Python source file, performing some transformations on it, and then spitting out the result source file.

One benefit that lib2to3 has over the ast module is that ast does not preserve whitespace and comments, whereas lib2to3 does. If you're already dealing with autogenerated code this might not be a problem for you.