Parsing a lightweight language in Python

Question

Say I define a string in Python like the following:

my_string = "something{name1, name2, opt1=2, opt2=text}, something_else{name3, opt1=58}"

I would like to parse that string in Python in a way that allows me to index the different structures of the language.

For example, the output could be a dictionary parsing_result that allows me to index the different elements in a structred manner.

For example, the following:

parsing_result['names'] 

would hold a list of strings: ['name1', 'name2']

whereas parsing_result['options'] would hold a dictionary so that:

• parsing_result['something']['options']['opt2'] holds the string "text"
• parsing_result['something_else']['options']['opt1'] holds the string "58"

My first question is: How do I approach this problem in Python? Are there any libraries that simplify this task?

For a working example, I am not necessarily interested in a solution that parses the exact syntax I defined above (although that would be fantastic), but anything close to it would be great.

Update

1. It looks like the general right solution is using a parser and a lexer such as ply (thank you @Joran), but the documentation is a bit intimidating. Is there an easier way of getting this done when the syntax is lightweight?

2. I found this thread where the following regular expression is provided to partition a string around outer commas:

   r = re.compile(r'(?:[^,(]|\([^)]*\))+')
   r.findall(s)
   

   But this is assuming that the grouping character are () (and not {}). I am trying to adapt this, but it doesn't look easy.

Answer 1

I highly recommend pyparsing:

The pyparsing module is an alternative approach to creating and executing simple grammars, vs. the traditional lex/yacc approach, or the use of regular expressions.

The Python representation of the grammar is quite readable, owing to the self-explanatory class names, and the use of '+', '|' and '^' operator definitions. The parsed results returned from parseString() can be accessed as a nested list, a dictionary, or an object with named attributes.

Sample code (Hello world from the pyparsing docs):

from pyparsing import Word, alphas
greet = Word( alphas ) + "," + Word( alphas ) + "!" # <-- grammar defined here
hello = "Hello, World!"
print (hello, "->", greet.parseString( hello ))

Output:

Hello, World! -> ['Hello', ',', 'World', '!']

Edit: Here's a solution to your sample language:

from pyparsing import *
import json

identifier = Word(alphas + nums + "_")
expression = identifier("lhs") + Suppress("=") + identifier("rhs")
struct_vals = delimitedList(Group(expression | identifier))
structure = Group(identifier + nestedExpr(opener="{", closer="}", content=struct_vals("vals")))
grammar = delimitedList(structure)

my_string = "something{name1, name2, opt1=2, opt2=text}, something_else{name3, opt1=58}"
parse_result = grammar.parseString(my_string)
result_list = parse_result.asList()

def list_to_dict(l):
    d = {}
    for struct in l:
        d[struct[0]] = {}
        for ident in struct[1]:
            if len(ident) == 2:
                d[struct[0]][ident[0]] = ident[1]
            elif len(ident) == 1:
                d[struct[0]][ident[0]] = None
    return d

print json.dumps(list_to_dict(result_list), indent=2)

Output: (pretty printed as JSON)

{
  "something_else": {
    "opt1": "58", 
    "name3": null
  }, 
  "something": {
    "opt1": "2", 
    "opt2": "text", 
    "name2": null, 
    "name1": null
  }
}

Use the pyparsing API as your guide to exploring the functionality of pyparsing and understanding the nuances of my solution. I've found that the quickest way to master this library is trying it out on some simple languages you think up yourself.