Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python parser for Python-like language

I'm looking to write a Python import filter or preprocessor for source files that are essentially Python with extra language elements. The goal is to read the source file, parse it to an abstract syntax tree, apply some transforms in order to implement the new parts of the language, and write valid Python source which can then be consumed by CPython. I want to write this thing in Python and am looking for the best parser for the task.

The parser built in to Python is not appropriate because it requires the source files be actual Python, which these will not be. There are tons of parsers (or parser generators) that will work with Python, but it's hard to tell which is the best for my needs without a whole bunch of research.

In summary, my requirements are:

  1. Parser is written in Python or has Python bindings.
  2. Comes with a Python grammar that I can tweak, or can easily consume a tweakable Python grammar available elsewhere (such as http://docs.python.org/reference/grammar.html).
  3. Can re-serialize the AST after transforming it.
  4. Should not be too horrific to work with API-wise.

Any suggestions?

like image 646
kindall Avatar asked Feb 23 '12 20:02

kindall


People also ask

Does Python have a parser?

The parser module provides an interface to Python's internal parser and byte-code compiler. The primary purpose for this interface is to allow Python code to edit the parse tree of a Python expression and create executable code from this.

Which parser is used in Python?

Making experiments. As the generated C parser is the one used by Python, this means that if something goes wrong when adding some new rules to the grammar you cannot correctly compile and execute Python anymore.

How do you get parser in Python?

Creating a parser>>> parser = argparse.ArgumentParser(description='Process some integers.') The ArgumentParser object will hold all the information necessary to parse the command line into Python data types.


1 Answers

I would recommend that you check out my library: https://github.com/erezsh/lark

It can parse ALL context-free grammars, automatically builds an AST (with line & column numbers), and accepts the grammar in EBNF format, which is considered the standard.

It can easily parse a language like Python, and it can do so faster than any other parsing library written in Python.

In fact, there's already an example python grammar and parser

like image 104
Erez Avatar answered Sep 26 '22 20:09

Erez