What files are required for Py_Initialize to run?

Question

I am working on a simple piece of code that runs a Python function from a C/C++ application. In order to do this I set the PYTHONPATH and run initialize as follows:

Py_SetPythonHome("../Python27");
Py_InitializeEx(0);

Then I import my module and run my function. It works great.

I am now attempting to build an installer for my colleagues to run my code. I want to minimize the number of files I need to include in this installer, for obvious reasons.

Googling around the subject tells me that I should be able to include the files "Python27.lib" and "Python27.dll", then zip the "DLLs" and "Lib" folders and include them. However, when I attempt this, Py_Initialize fails.

A quick examination of what is causing this failure shows that Py_Initialize appears to depend upon a number of .pyc files in the Lib folder including (but not limited to warnings.pyc, _abcoll.pyc, _future_.pyc and the contents of the "encodings" folder.

I cannot understand why this would be. Any advice?

Answer 1

At the beginning, I wanted to say that there's no module required (at least no non-builtin one) for Py_InitializeEx, so the only requirement was python27.dll (BTW: python27.lib is not required, unless your colleagues want to link something against it - but that wouldn't be very easy w / o Python's Include dir).
I had this code (using Python 2.7.10 that I built using VStudio 10 (2010)):

#include <stdio.h>
#include <conio.h>
#include <Python.h>

int main()
{
    int i = 0;
    char *pyCode = 
        "s = \"abc\"\n"
        "print s, 1234";
    Py_InitializeEx(0);
    i = PyRun_SimpleString(pyCode);
    Py_Finalize();
    printf("PyRun_SimpleString returned: %d\nPress a key to exit...\n", i);
    _getch();
    return 0;
}

It built fine, it ran OK from VStudio, and from the command line (after copying the .dll in its folder). But then I copied the .exe and .dll to another computer and when running, bang!!!

ImportError: No module named site

Considering that:

• I have no PYTHON* env vars set in neither of the consoles on the 2 machines where I ran the .exe (with different results)
• On both machines the Python installation is on the same path (I previously (years ago) modified it on the machine that doesn't work)

I don't know why it doesn't behave the same (one thing that I haven't check is that there might be some registry key on the machine that works?).

Note: site is a (.py(c)) module located under %PYTHON_INSTALL_DIR%\Lib.

Then, I browsed Python's source code and I came across [GitHub]: python/cpython - (v2.7.10) cpython/Python/pythonrun.c (function Py_InitializeEx - currently line #141) - this is how I'm going to refer a point in the source code):

if (!Py_NoSiteFlag)
    initsite(); /* Module site */

while in initsite:

m = PyImport_ImportModule("site");

which is pretty obvious (Py_NoSiteFlag is 0).

Then I noticed that Py_NoSiteFlag is declared as extern __declspec(dllexport) ([MS.Docs]: Using extern to Specify Linkage, [MS.Docs]: dllexport, dllimport), so I modified my code to:

#include <stdio.h>
#include <conio.h>
#include <Python.h>

extern __declspec(dllimport) int Py_NoSiteFlag;


int main()
{
    int i = 0;
    char *pyCode = 
        "s = \"abc\"\n"
        "print s, 1234";
    Py_NoSiteFlag = 1;
    Py_InitializeEx(0);
    i = PyRun_SimpleString(pyCode);
    Py_Finalize();
    printf("PyRun_SimpleString returned: %d\nPress a key to exit...\n", i);
    _getch();
    return 0;
}

and it works! Yay!

So, at this point only the .dll is required in order to run a piece of code. But I imagine that your code is "a little bit" more complex than that (it has imports ([Python 2.Docs]: The import statement). To solve the import problem, you can use this nice module: [Python 2.Docs]: modulefinder - Find modules used by a script (part of Python 2.7's standard modules). To make use of it:

• Save the code that you execute from C in a .py file
• Run modulefinder against it

Here's an example for my code (pyCode contents in my C program, saved in a file).

code00.py:

s = "abc"
print s, 1234

Running:

${PYTHON_INSTALL_DIR}\python.exe -m modulefinder code00.py

yields:

Name                      File
----                      ----
m __main__                code00.py

But, if I add an import os (which is a pretty common module) statement in the file, the above command yields:

Name                        File
----                        ----
m StringIO                  %PYTHON_INSTALL_DIR%\lib\StringIO.py
m UserDict                  %PYTHON_INSTALL_DIR%\lib\UserDict.py
m __builtin__
m __future__                %PYTHON_INSTALL_DIR%\lib\__future__.py
m __main__                  a.py
m _abcoll                   %PYTHON_INSTALL_DIR%\lib\_abcoll.py
m _codecs
m _collections
m _functools
m _hashlib                  %PYTHON_INSTALL_DIR%\DLLs\_hashlib.pyd
m _heapq
m _io
m _locale
m _random
m _sre
m _struct
m _subprocess
m _threading_local          %PYTHON_INSTALL_DIR%\lib\_threading_local.py
m _warnings
m _weakref
m _weakrefset               %PYTHON_INSTALL_DIR%\lib\_weakrefset.py
m abc                       %PYTHON_INSTALL_DIR%\lib\abc.py
m array
m atexit                    %PYTHON_INSTALL_DIR%\lib\atexit.py
m bdb                       %PYTHON_INSTALL_DIR%\lib\bdb.py
m binascii
m cPickle
m cStringIO
m cmd                       %PYTHON_INSTALL_DIR%\lib\cmd.py
m codecs                    %PYTHON_INSTALL_DIR%\lib\codecs.py
m collections               %PYTHON_INSTALL_DIR%\lib\collections.py
m copy                      %PYTHON_INSTALL_DIR%\lib\copy.py
m copy_reg                  %PYTHON_INSTALL_DIR%\lib\copy_reg.py
m difflib                   %PYTHON_INSTALL_DIR%\lib\difflib.py
m dis                       %PYTHON_INSTALL_DIR%\lib\dis.py
m doctest                   %PYTHON_INSTALL_DIR%\lib\doctest.py
m dummy_thread              %PYTHON_INSTALL_DIR%\lib\dummy_thread.py
P encodings                 %PYTHON_INSTALL_DIR%\lib\encodings\__init__.py
m encodings.aliases         %PYTHON_INSTALL_DIR%\lib\encodings\aliases.py
m errno
m exceptions
m fnmatch                   %PYTHON_INSTALL_DIR%\lib\fnmatch.py
m functools                 %PYTHON_INSTALL_DIR%\lib\functools.py
m gc
m genericpath               %PYTHON_INSTALL_DIR%\lib\genericpath.py
m getopt                    %PYTHON_INSTALL_DIR%\lib\getopt.py
m gettext                   %PYTHON_INSTALL_DIR%\lib\gettext.py
m hashlib                   %PYTHON_INSTALL_DIR%\lib\hashlib.py
m heapq                     %PYTHON_INSTALL_DIR%\lib\heapq.py
m imp
m inspect                   %PYTHON_INSTALL_DIR%\lib\inspect.py
m io                        %PYTHON_INSTALL_DIR%\lib\io.py
m itertools
m keyword                   %PYTHON_INSTALL_DIR%\lib\keyword.py
m linecache                 %PYTHON_INSTALL_DIR%\lib\linecache.py
m locale                    %PYTHON_INSTALL_DIR%\lib\locale.py
P logging                   %PYTHON_INSTALL_DIR%\lib\logging\__init__.py
m marshal
m math
m msvcrt
m nt
m ntpath                    %PYTHON_INSTALL_DIR%\lib\ntpath.py
m opcode                    %PYTHON_INSTALL_DIR%\lib\opcode.py
m operator
m optparse                  %PYTHON_INSTALL_DIR%\lib\optparse.py
m os                        %PYTHON_INSTALL_DIR%\lib\os.py
m os2emxpath                %PYTHON_INSTALL_DIR%\lib\os2emxpath.py
m pdb                       %PYTHON_INSTALL_DIR%\lib\pdb.py
m pickle                    %PYTHON_INSTALL_DIR%\lib\pickle.py
m posixpath                 %PYTHON_INSTALL_DIR%\lib\posixpath.py
m pprint                    %PYTHON_INSTALL_DIR%\lib\pprint.py
m random                    %PYTHON_INSTALL_DIR%\lib\random.py
m re                        %PYTHON_INSTALL_DIR%\lib\re.py
m repr                      %PYTHON_INSTALL_DIR%\lib\repr.py
m select                    %PYTHON_INSTALL_DIR%\DLLs\select.pyd
m shlex                     %PYTHON_INSTALL_DIR%\lib\shlex.py
m signal
m sre_compile               %PYTHON_INSTALL_DIR%\lib\sre_compile.py
m sre_constants             %PYTHON_INSTALL_DIR%\lib\sre_constants.py
m sre_parse                 %PYTHON_INSTALL_DIR%\lib\sre_parse.py
m stat                      %PYTHON_INSTALL_DIR%\lib\stat.py
m string                    %PYTHON_INSTALL_DIR%\lib\string.py
m strop
m struct                    %PYTHON_INSTALL_DIR%\lib\struct.py
m subprocess                %PYTHON_INSTALL_DIR%\lib\subprocess.py
m sys
m tempfile                  %PYTHON_INSTALL_DIR%\lib\tempfile.py
m textwrap                  %PYTHON_INSTALL_DIR%\lib\textwrap.py
m thread
m threading                 %PYTHON_INSTALL_DIR%\lib\threading.py
m time
m token                     %PYTHON_INSTALL_DIR%\lib\token.py
m tokenize                  %PYTHON_INSTALL_DIR%\lib\tokenize.py
m traceback                 %PYTHON_INSTALL_DIR%\lib\traceback.py
m types                     %PYTHON_INSTALL_DIR%\lib\types.py
P unittest                  %PYTHON_INSTALL_DIR%\lib\unittest\__init__.py
m unittest.case             %PYTHON_INSTALL_DIR%\lib\unittest\case.py
m unittest.loader           %PYTHON_INSTALL_DIR%\lib\unittest\loader.py
m unittest.main             %PYTHON_INSTALL_DIR%\lib\unittest\main.py
m unittest.result           %PYTHON_INSTALL_DIR%\lib\unittest\result.py
m unittest.runner           %PYTHON_INSTALL_DIR%\lib\unittest\runner.py
m unittest.signals          %PYTHON_INSTALL_DIR%\lib\unittest\signals.py
m unittest.suite            %PYTHON_INSTALL_DIR%\lib\unittest\suite.py
m unittest.util             %PYTHON_INSTALL_DIR%\lib\unittest\util.py
m warnings                  %PYTHON_INSTALL_DIR%\lib\warnings.py
m weakref                   %PYTHON_INSTALL_DIR%\lib\weakref.py

 Missing modules:
? _emx_link imported from os
? ce imported from os
? fcntl imported from subprocess, tempfile
? org.python.core imported from copy, pickle
? os.path imported from os, shlex
? os2 imported from os
? posix imported from os
? pwd imported from posixpath
? readline imported from cmd, pdb
? riscos imported from os
? riscosenviron imported from os
? riscospath imported from os

As you can see, there is an awfully lot of modules (I modified the output a little bit, instead of the actual path I placed the ${PYTHON_INSTALL_DIR} env var placeholder). In order for the Python code to work, you'll have to include all of those modules/packages in the installer.

Notes about modulefinder's output (that I've noticed while playing with it):

• It searches for modules recursively, so here is the whole module dependency tree
• It searches for import statements located in functions (so, not only the ones at module level)
• It doesn't search for dynamic imports (e.g. [Python 2.Docs]: __import__(name[, globals[, locals[, fromlist[, level]]]]))

So, looking at the modules that are required by os, I'm not sure that taking out the site import from C, makes much of a difference.

IMPORTANT NOTE: To make sure your .exe works on any computer, you might consider including VStudio C Runtime Library or VCRTLib (msvcr##(#).dll: [MS.Docs]: Run-Time Library Reference) (where *#*s are placeholders for digits - representing VStudio version) in your installer.