ctypes vs _ctypes - why does the latter exist?

Question

I recently learned that Python has not only a module named ctypes, which has a docs page, but also a module named _ctypes, which doesn't (but is nonetheless mentioned a few times in the docs). Some code on the internet, like the snippet in this Stack Overflow answer, uses this mysterious undocumented _ctypes module.

A little experimentation indicates that the two modules have similar but non-identical docstrings and overlapping but non-identical attribute lists:

Python 3.7.4 (default, Sep  7 2019, 18:27:02) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes, _ctypes
>>> print(ctypes.__doc__)
create and manipulate C data types in Python
>>> print(_ctypes.__doc__)
Create and manipulate C compatible data types in Python.
>>> dir(ctypes)
['ARRAY', 'ArgumentError', 'Array', 'BigEndianStructure', 'CDLL', 'CFUNCTYPE', 'DEFAULT_MODE', 'LibraryLoader', 'LittleEndianStructure', 'POINTER', 'PYFUNCTYPE', 'PyDLL', 'RTLD_GLOBAL', 'RTLD_LOCAL', 'SetPointerType', 'Structure', 'Union', '_CFuncPtr', '_FUNCFLAG_CDECL', '_FUNCFLAG_PYTHONAPI', '_FUNCFLAG_USE_ERRNO', '_FUNCFLAG_USE_LASTERROR', '_Pointer', '_SimpleCData', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_c_functype_cache', '_calcsize', '_cast', '_cast_addr', '_check_size', '_ctypes_version', '_dlopen', '_endian', '_memmove_addr', '_memset_addr', '_os', '_pointer_type_cache', '_reset_cache', '_string_at', '_string_at_addr', '_sys', '_wstring_at', '_wstring_at_addr', 'addressof', 'alignment', 'byref', 'c_bool', 'c_buffer', 'c_byte', 'c_char', 'c_char_p', 'c_double', 'c_float', 'c_int', 'c_int16', 'c_int32', 'c_int64', 'c_int8', 'c_long', 'c_longdouble', 'c_longlong', 'c_short', 'c_size_t', 'c_ssize_t', 'c_ubyte', 'c_uint', 'c_uint16', 'c_uint32', 'c_uint64', 'c_uint8', 'c_ulong', 'c_ulonglong', 'c_ushort', 'c_void_p', 'c_voidp', 'c_wchar', 'c_wchar_p', 'cast', 'cdll', 'create_string_buffer', 'create_unicode_buffer', 'get_errno', 'memmove', 'memset', 'pointer', 'py_object', 'pydll', 'pythonapi', 'resize', 'set_errno', 'sizeof', 'string_at', 'wstring_at']
>>> dir(_ctypes)
['ArgumentError', 'Array', 'CFuncPtr', 'FUNCFLAG_CDECL', 'FUNCFLAG_PYTHONAPI', 'FUNCFLAG_USE_ERRNO', 'FUNCFLAG_USE_LASTERROR', 'POINTER', 'PyObj_FromPtr', 'Py_DECREF', 'Py_INCREF', 'RTLD_GLOBAL', 'RTLD_LOCAL', 'Structure', 'Union', '_Pointer', '_SimpleCData', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '__version__', '_cast_addr', '_memmove_addr', '_memset_addr', '_pointer_type_cache', '_string_at_addr', '_unpickle', '_wstring_at_addr', 'addressof', 'alignment', 'buffer_info', 'byref', 'call_cdeclfunction', 'call_function', 'dlclose', 'dlopen', 'dlsym', 'get_errno', 'pointer', 'resize', 'set_errno', 'sizeof']

I momentarily thought that perhaps what I was seeing was an "accelerator module", but I think it can't be, because the current implementation of ctypes unconditionally imports stuff from _ctypes. Nor is it clear that _ctypes is just an implementation detail; it exposes at least one public member, PyObj_FromPtr, that is useful, not available via the ctypes module, and not used anywhere within the CPython source - perhaps suggesting that it's meant for us to import and use when writing Python code?

How come Python has these two modules with basically the same name? What's the division of responsibilities between the two, and when would I want to use one over the other? Should I regard _ctypes as a part of the standard library, or as an implementation detail that I shouldn't be touching?

Answer 1

_ctypes exists because a very large amount of ctypes has to be written in C. ctypes and _ctypes both exist because not all of ctypes has to be written in C. ctypes contains the parts that were more convenient to write in Python.

The fact that _ctypes has things in it with no leading underscore, not exposed by ctypes, does not mean that those things are meant to be used by anything. Sometimes programs just have stuff left lying around like that. There might have been an intent to make it public at some point in development.

_ctypes is an implementation detail. Use ctypes.