.NET/C# Interop to Python

Question

My backend is written in .NET/C#, I have a requirement that I need to execute python scripts passing context from the .net side of the house. These are queued up in a background task engine called hangfire running as a windows service.

I did a little digging and found IronPython, however, after implementing it failed to support many of the pypi python packages that I need to execute inside my script.

Secondly, I looked at Python.Net which is a embedded interpreter that embeds or extends CPython. CPython can run all the scripts/etc that I needed, however, I found that opening/closing the python interpreter all the time can create quite a few memory leaks and there is always threading constraints there too. See docs for some of the details.

I'm wondering if this interopt and embedding python in .net is even a good idea. I'm wondering if making python its own execution engine using something like celery and marshalling data between the two using something like protobufs would be a better solution? This adds much more complexity and an additional task engine to my stack too.

I was wondering if anyone else had any ideas/feedback/experiences trying to accomplishing something similar? Thanks!

Answer 1

I would argue that pythonnet is your best option:

1. Embedded Python interpreter is not the same as Python sub-interpreter. The sub-interpreter is used much less, while CPython has been embedded in major applications. Moreover memory leaks should be very unlikely due to pythonnet taking care of this for you.
2. The embedded interpreter does not have to be shutdown between calls to it - in fact this can get very expensive, you can keep it persistent as long as your application is alive.
3. Threading issues can be avoided if you enclose the code requiring the GIL with using (Py.GIL()) {} blocks.
4. The added benefit of debugging CPython/.NET code in mixed-mode using PTVS.