Py_Initialize fails - unable to load the file system codec

Question

I am attempting to put together a simple c++ test project that uses an embedded python 3.2 interpreter. The project builds fine but Py_Initialize raises a fatal error:

Fatal Python error: Py_Initialize: unable to load the file system codec
LookupError: no codec search functions registered: can't find encoding

Minimal code:

#include <Python.h>

int main (int, char**)
{
  Py_Initialize ();
  Py_Finalize ();
  return 0;
}

The OS is 32bit Vista.

The python version used is a python 3.2 debug build, built from sources using VC++ 10.

The python_d.exe file from the same build runs without any problems.

Could someone explain the problem and how to fix it? My own google-fu fails me.

EDIT 1

After going through the python source code I've found that, as the error says, no codec search functions have been registered. Both codec_register and PyCodec_Register are as they should be. It's just that nowhere in the code are any of these functions called.

I don't really know what this means as I still have no idea when and from where these functions should have been called. The code that raises the error is entirely missing from the source of my other python build (3.1.3).

EDIT 2

Answered my own question below.

Answer 1

Check the PYTHONPATH and PYTHONHOME environment variables and make sure they don't point to Python 2.x.

http://bugs.python.org/issue11288

Answer 2

Parts of this have been mentioned before, but in a nutshell this is what worked for my environment where I have multiple Python installs and my global OS environment set-up to point to a different install than the one I attempt to work with when encountering the problem.

Make sure your (local or global) environment is fully set-up to point to the install you aim to work with, e.g. you have two (or more) installs of, let's say a python27 and python33 (sorry these are windows paths but the following should be valid for equivalent UNIX-style paths just as well, please let me know about anything I'm missing here (probably the DLLs path might differ)):

C:\python27_x86

C:\python33_x64

Now, if you intend to work with your python33 install but your global environment is pointing to python27, make sure you update your environment as such (while PATH and PYTHONHOME may be optional (e.g. if you temporarily work in a local shell)):

PATH="C:\python33_x64;%PATH%"

PYTHONPATH="C:\python33_x64\DLLs;C:\python33_x64\Lib;C:\python33_x64\Lib\site-packages"

PYTHONHOME=C:\python33_x64

Note, that you might need/want to append any other library paths to your PYTHONPATH if required by your development environment, but having your DLLs, Lib and site-packages properly set-up is of prime importance.

Hope this helps.

Answer 3

The core reason is quite simple: Python does not find its modules directory, so it can of course not load encodings, too

Python doc on embedding says "Py_Initialize() calculates the module search path based upon its best guess" ... "In particular, it looks for a directory named lib/pythonX.Y"

Yet, if the modules are installed in (just) lib - relative to the python binary - above guess is wrong.

Although docs says that PYTHONHOME and PYTHONPATH are regarded, we observed that this was not the case; their actual presence or content was completely irrelevant.

The only thing that had an effect was a call to Py_SetPath() with e.g. [path-to]\lib as argument before Py_Initialize().

Sure this is only an option for an embedding scenario where one has direct access and control over the code; with a ready-made solution, special steps may be necessary to solve the issue.