I found an ANTLRv4 Python3 grammer, but it generates a parse-tree, which generally has many useless nodes.
I'm looking for a known package to get a Python AST from that parse tree.
Does something like this exist?
EDIT: Clarification regarding use of the Python ast
package: my project is in Java and I need to parse Python files.
EDIT 2: By 'AST' I mean http://docs.python.org/2/library/ast.html#abstract-grammar, while by 'parse tree' I mean http://docs.python.org/2/reference/grammar.html.
The following could be a start:
public class AST {
private final Object payload;
private final List<AST> children;
public AST(ParseTree tree) {
this(null, tree);
}
private AST(AST ast, ParseTree tree) {
this(ast, tree, new ArrayList<AST>());
}
private AST(AST parent, ParseTree tree, List<AST> children) {
this.payload = getPayload(tree);
this.children = children;
if (parent == null) {
walk(tree, this);
}
else {
parent.children.add(this);
}
}
public Object getPayload() {
return payload;
}
public List<AST> getChildren() {
return new ArrayList<>(children);
}
private Object getPayload(ParseTree tree) {
if (tree.getChildCount() == 0) {
return tree.getPayload();
}
else {
String ruleName = tree.getClass().getSimpleName().replace("Context", "");
return Character.toLowerCase(ruleName.charAt(0)) + ruleName.substring(1);
}
}
private static void walk(ParseTree tree, AST ast) {
if (tree.getChildCount() == 0) {
new AST(ast, tree);
}
else if (tree.getChildCount() == 1) {
walk(tree.getChild(0), ast);
}
else if (tree.getChildCount() > 1) {
for (int i = 0; i < tree.getChildCount(); i++) {
AST temp = new AST(ast, tree.getChild(i));
if (!(temp.payload instanceof Token)) {
walk(tree.getChild(i), temp);
}
}
}
}
@Override
public String toString() {
StringBuilder builder = new StringBuilder();
AST ast = this;
List<AST> firstStack = new ArrayList<>();
firstStack.add(ast);
List<List<AST>> childListStack = new ArrayList<>();
childListStack.add(firstStack);
while (!childListStack.isEmpty()) {
List<AST> childStack = childListStack.get(childListStack.size() - 1);
if (childStack.isEmpty()) {
childListStack.remove(childListStack.size() - 1);
}
else {
ast = childStack.remove(0);
String caption;
if (ast.payload instanceof Token) {
Token token = (Token) ast.payload;
caption = String.format("TOKEN[type: %s, text: %s]",
token.getType(), token.getText().replace("\n", "\\n"));
}
else {
caption = String.valueOf(ast.payload);
}
String indent = "";
for (int i = 0; i < childListStack.size() - 1; i++) {
indent += (childListStack.get(i).size() > 0) ? "| " : " ";
}
builder.append(indent)
.append(childStack.isEmpty() ? "'- " : "|- ")
.append(caption)
.append("\n");
if (ast.children.size() > 0) {
List<AST> children = new ArrayList<>();
for (int i = 0; i < ast.children.size(); i++) {
children.add(ast.children.get(i));
}
childListStack.add(children);
}
}
}
return builder.toString();
}
}
and can be used to create an AST for the input "f(arg1='1')\n"
as follows:
public static void main(String[] args) {
Python3Lexer lexer = new Python3Lexer(new ANTLRInputStream("f(arg1='1')\n"));
Python3Parser parser = new Python3Parser(new CommonTokenStream(lexer));
ParseTree tree = parser.file_input();
AST ast = new AST(tree);
System.out.println(ast);
}
which would print:
'- file_input |- stmt | |- small_stmt | | |- atom | | | '- TOKEN[type: 35, text: f] | | '- trailer | | |- TOKEN[type: 47, text: (] | | |- arglist | | | |- test | | | | '- TOKEN[type: 35, text: arg1] | | | |- TOKEN[type: 53, text: =] | | | '- test | | | '- TOKEN[type: 36, text: '1'] | | '- TOKEN[type: 48, text: )] | '- TOKEN[type: 34, text: \n] '- TOKEN[type: -1, text: ]
I realize this still contains nodes you might not want, but you could even add a set of token types you'd like to exclude. Feel free to hack away!
Here is a Gist containing a version of the code above with the proper import statements and some JavaDocs and inline comments.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With