Is a portable determination of appropriate mapping for off_t in Python for use in ctypes possible?

Question

Background: The POSIX-defined off_t data type is a signed integer of variable-size depending on the environment. For 64-bit build environments it seems to be consistently a 64-bit off_t. For 32-bit systems the size of off_t depends on the environment (usually controlled with _FILE_OFFSET_BITS and related).

I'm using Python's ctypes to access some library calls for functions that use the off_t data type. ctypes does not have a type for off_t so mapping such an API via a Structure or otherwise requires choosing some other types that are defined by ctypes, namely one of c_int, c_int32, c_int64, c_long and c_longlong.

In searching around for solutions by others I've seen all sorts, most of them guesses and limited to one of the environments, some of them flat-out wrong (using an unsigned type) and none of them portable.

Is there a robust, portable way to determine the size of off_t for the build of a given Python interpreter?

Thanks!

Answer 1

There is no support for this in a standard, and as you yourself point out, it is going to vary on 32 bit machines based on the flags that were used to compile the library, not Python. (That is, you need to know if the library routines you are using were compiled with -D_FILE_OFFSET_BITS=64 or not...)

GCC seems to define off_t as long int pretty consistently. So that's probably the "safest" choice. Or you could just go with 64 bits by default. In either case, an option to override the behavior and specify the number of bits might be valuable.

Is there a function you can call that will return an off_t? Maybe you could fill a buffer with 0xFF, then seek(0) and ftell() to see how many zeroes get written?

Answer 2

I'm not 100% sure, but sysconfig might be just that (for python 3.2+):

Python 3.9.1 (default, Dec  8 2020, 02:26:20) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('SIZEOF_OFF_T')
8
>>> sysconfig.get_config_var('SIZEOF_PID_T')
4