pythonnet Embedding Python in .net example failing to load module

Question

I'm trying to run the Embedding Python in .NET example from https://github.com/pythonnet/pythonnet. I've followed troubleshooting articles to set the proper %PYTHONPATH% and %PYTHONHOME% to my anaconda environment in the program base directory.

After activating my anaconda environment, I have successfully imported sys, and imp as a test, and also sucessfully used PythonEngine.RunSimpleString(), but the numpy example fails with Python.Runtime.PythonException: ImportError : No module named 'numpy'

importing numpy from python in this environment was successful, but this and other packages fail to import in pythonnet.

Pythonnet version: 2.3 x64 (installed using conda install -c pythonnet pythonnet)

Python version: Python 3.5 x64 (anaconda)

Operating System: Windows 10

The following code produces the error:

static void Main(string[] args)
{
    string envPythonHome = AppDomain.CurrentDomain.BaseDirectory + "cntk-py35";
    string envPythonLib = envPythonHome + @"\Lib";
    Environment.SetEnvironmentVariable("PYTHONHOME", envPythonHome, EnvironmentVariableTarget.Process);
    Environment.SetEnvironmentVariable("PATH", envPythonHome + ";" + Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine), EnvironmentVariableTarget.Process);
    Environment.SetEnvironmentVariable("PYTHONPATH", envPythonLib, EnvironmentVariableTarget.Process);

    PythonEngine.PythonHome = envPythonHome;
    PythonEngine.PythonPath = Environment.GetEnvironmentVariable("PYTHONPATH");

    using (Py.GIL())
    {
        dynamic np = Py.Import("numpy");
        Console.WriteLine(np.cos(np.pi * 2));

        dynamic sin = np.sin;
        Console.WriteLine(sin(5));

        double c = np.cos(5) + sin(5);
        Console.WriteLine(c);

        dynamic a = np.array(new List<float> { 1, 2, 3 });
        Console.WriteLine(a.dtype);

        dynamic b = np.array(new List<float> { 6, 5, 4 }, dtype: np.int32);
        Console.WriteLine(b.dtype);

        Console.WriteLine(a * b);
        Console.ReadKey();
    }
}

It seems that any package under site-packages in my environment similarly fail. Adding to %PATH% did not work. Is there a way to get pythonnet to recognize and load these modules?

Answer 1

Setting up your python environment in .NET is a bit cumbersome.This issue is not well detailed on pythonnet website or most of suggested solutions I have found on the internet did not work for my computer. The reason is that every computer may have a different python setup environment (depending how you have installed python and the libraries). It took me a while as well but finally I have succeeded to call python modules and .py scripts from .NET. Here is what I did.

Pythonnet version: 2.4.0 x64 (installed using pip install @ Anaconda CMD prompt)

Python version: Python 3.7 x64 (Anaconda)

Operating System: Windows 10

Keep in mind that everyone has a different Python environment, that is why you have to configure your environment first (in your VS project). First, we need to assign "PATH", "PYTHONHOME" and "PYTHONPATH" variables.

in C# use:

string pythonPath1 = @"C:\Users\<your username>\Anaconda3";
        string pythonPath2 = @"C:\Users\<your username>\Anaconda3\lib\site-packages";
        Environment.SetEnvironmentVariable("PATH", pythonPath1, EnvironmentVariableTarget.Process);
        Environment.SetEnvironmentVariable("PYTHONHOME", pythonPath1, EnvironmentVariableTarget.Process);
        Environment.SetEnvironmentVariable("PYTHONPATH", pythonPath2, EnvironmentVariableTarget.Process); 

I have used Anaconda to install python runtime and packages. Note that default Anaconda installation is under C:\Users(your username assigned in your computer)\Anaconda3. (You can find yours using code by (How can I get the current user directory?). If you did not use Anaconda, you need to locate the directory where python packages are installed on your computer.

1-C:\Users\\Anaconda3 directory must have your version of python DLL (i.e.python37.dll).

2-C:\Users\\Anaconda3\lib\site-packages has the "modules" (i.e. python frameworks like 'numpy').

3-You must reference python runtime in your project (Python.Runtime.dll).(use Windows File Explorer to find the file. If you use Anaconda the runtime is under C:\Users\\Anaconda3 directory).

4-Add following on top of your code:

using Python.Runtime;

5-In VS Solution Explorer right click on your project and select 'Properties'

and make sure to set 'Platform Target' (either x64 or x86)

6-After doing all this, if you can NOT run some of the 'modules' and receive a "Can not load module" or "can not find module" exception message, then follow the instruction here (https://github.com/microsoft/vscode-python/issues/9218) Usually, uninstalling/re-installing the module will resolve the issue by updating the version. (i.e. pip uninstall numpy/pip install numpy).

NOTE: The code still works, even if you do not set PYTHONPATH variable above. PYTHONPATH variable is used, when you need to call a custom .py script, where you identify the directory where your file resides. A descriptive example can be found at https://stackoverflow.com/a/57910578/7675537

UPDATE: I have realized that one easy way to configure your Python Environment is using Visual Studio (I use VS 2017 Community version). Just go to Python in Visual Studio and go through the example. In VS under 'Python Environments' you can observe all the setups you have in your computer. I had problems to run .py scripts, like not being able to use 'import matplotlib.pyplot' and spent several hours reading articles on the internet but could not find a solution. Finally I switched my environment to 'C:\Users\\AppData\Local\Programs\Python\Python37\' and installed all the missing packages from visual studio's list (of suggestions) and it worked. I think calling Anaconda environment from .NET (via pythonnet or else) has problems. I would suggest not to use your Anaconda python environment if you make .NET calls to python. I use:

    private static string pythonPath1 = @"C:\Users\<your name>\AppData\Local\Programs\Python\Python37";
    static void Main(string[] args)
    {
        Test();
    }

    private static void Test()
    {
        string pathToPython = pythonPath1;
        string path = pathToPython + ";" +
                      Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine);
        Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
        Environment.SetEnvironmentVariable("PYTHONHOME", pathToPython, EnvironmentVariableTarget.Process);

        var lib = new[]
        {
            @"C:\Users\<your name>\<your python code is here>",
            Path.Combine(pathToPython, "Lib"),
            Path.Combine(pathToPython, "DLLs")

        };
        string paths = string.Join("; ", lib);
        Environment.SetEnvironmentVariable("PYTHONPATH", paths, EnvironmentVariableTarget.Process);

        using (Py.GIL()) //Initialize the Python engine and acquire the interpreter lock
        {
            try
            {
                // import your script into the process
                dynamic sampleModule = Py.Import("yourpythoncode");
            }
            catch (PythonException error)
            {
                Console.WriteLine("Error occured: ", error.Message);
            }

        }
    }