Manipulating a Python file from C#

Question

I'm working on some tools for a game I'm making. The tools serve as a front end to making editing game files easier. Several of the files are python scripting files. For instance, I have an Items.py file that contains the following (minimalized for example)

from ItemModule import *

import copy

class ScriptedItem(Item):
    def __init__(self, name, description, itemtypes, primarytype, flags, usability, value, throwpower):
        Item.__init__(self, name, description, itemtypes, primarytype, flags, usability, value, throwpower, Item.GetNextItemID())
    def Clone(self):
        return copy.deepcopy(self)

ItemLibrary.AddItem(ScriptedItem("Abounding Crystal", "A colourful crystal composed of many smaller crystals. It gives off a warm glow.", ItemType.SynthesisMaterial, ItemType.SynthesisMaterial, 0, ItemUsage.Unusable, 0, 50))

As I Mentioned, I want to provide a front end for editing this file without requring an editor to know python/edit the file directly. My editor needs to be able to:

1. Find and list all the class types (in this example, it'd be only Scripted Item)
2. Find and list all created items (in this case there'd only be one, Abounding Crystal). I'd need to find the type (in this caseScriptedItem) and all the parameter values
3. Allow editing of parameters and the creation/removal of items.

To do this, I started writing my own parser, looking for the class keyword and when these recorded classes are use to construct objects. This worked for simple data, but when I started using classes with complex constructors (lists, maps, etc.) it became increasing difficult to correctly parse.

After searching around, I found IronPython made it easy to parse python files, so that's what I went about doing. Once I built the Abstract Syntax Tree I used PythonWalkers to identify and find all the information I need. This works perfectly for reading in data, but I don't see an easy way to push updated data into the Python file. As far as I can tell, there's no way to change the values in the AST and much less so to convert the AST back into a script file. If I'm wrong, I'd love for someone to tell me how I could do this. What I'd need to do now is search through the file until I find the correctly line, then try to push the data into the constructor, ensuring correct ordering.

Is there some obvious solution I'm not seeing? Should I just keeping working on my parser and make it support more complex data types? I really though I had it with the IronPython parser, but I didn't think about how tricky it'd be to push modified data back into the file.

Any suggestions would be appreciated

Answer 1

You want a source-to-source program transformation tool.

Such a tool parses a language to an internal data structure (invariably an AST), allows you to modify the AST, and then can regenerate source text from the modified AST without changing essentially anything about the source except where the AST changes were made.

Such a program transformation tool has to parse text to ASTs, and "anti-parse" (called "Prettyprint") ASTs to text. If IronPython has a prettyprinter, that's what you need. If it doesn't, you can build one with some (maybe a lot) of effort; as you've observed, this isn't as easy as one might think. See my answer Compiling an AST back to source code

If that doesn't work, our DMS Software Reengineering Toolkit with its Python front end might do the trick. It has all the above properties.