Handling 128-bit integers with ctypes

Question

What is the best way of supporting 128-bit integers (currently __uint128_t) with Python ctypes?

A user-defined struct of two uint64_t's perhaps, but this will create alignment issues where that is required.

Any thoughts on why ctypes has not been extended to support 128-bit integers?

Answer 1

If you really want to work with 128-bit integers then you don't need to worry about alignment. No current architecture, no machine that Python runs on, supports 128-bit native integer arithmetic. So no machine would require or benefit from having 128-bit integers 16-byte aligned. Just use that user defined struct and you'll be fine.

If what you're really asking for is support for 128-bit vector types then you'll probably need to have them aligned. That is, you need them aligned if you're creating them in Python code and passing them by reference to C/C++ code. You can't reliably pass them by value there's no way to get ctypes to align them properly on the stack (if that's required by the architecture ABI). Vectors that are passed from C/C++ to Python will presumably already be aligned properly. So, if you can arrange it so all your vectors are allocated in C/C++ code you should also be fine with your user defined struct.

Assuming that you do really need to create aligned vectors in Python code then I've included code for aligned ctypes arrays. I also have code to align other ctypes types that I haven't included to the code size reasonable. Arrays should be enough for most purposes. These aligned arrays have a couple of limitations. They won't be aligned properly if you pass them by value to C/C++ functions or if you include them as members in a struct or union. You can make aligned arrays of aligned arrays using the * operator.

Use aligned_array_type(ctypes-type, length, alignment) to create new aligned array types. Use aligned_type(ctypes-type, alignment) to create an aligned version of an already existing array type.

import ctypes

ArrayType = type(ctypes.Array)

class _aligned_array_type(ArrayType):
    def __mul__(self, length):
        return aligned_array_type(self._type_ * self._length_,
                      length, self._alignment_)

    def __init__(self, name, bases, d):
        self._alignment_ = max(getattr(self, "_alignment_", 1), 
                       ctypes.alignment(self))

def _aligned__new__(cls):
    a = cls._baseclass_.__new__(cls)
    align = cls._alignment_
    if ctypes.addressof(a) % align == 0:
        return a
    cls._baseclass_.__init__(a) # dunno if necessary
    ctypes.resize(a, ctypes.sizeof(a) + align - 1)
    addr = ctypes.addressof(a)
    aligned = (addr + align - 1) // align * align
    return cls.from_buffer(a, aligned - addr)

class aligned_base(object):
    @classmethod
    def from_address(cls, addr):
        if addr % cls._alignment_ != 0:
            raise ValueError, ("address must be %d byte aligned"
                       % cls._alignment_)
        return cls._baseclass_.from_address(cls, addr)

    @classmethod
    def from_param(cls, addr):
        raise ValueError, ("%s objects may not be passed by value"
                   % cls.__name__)

class aligned_array(ctypes.Array, aligned_base):
    _baseclass_ = ctypes.Array
    _type_ = ctypes.c_byte
    _length_ = 1
    __new__ = _aligned__new__

_aligned_type_cache = {}

def aligned_array_type(typ, length, alignment = None):
    """Create a ctypes array type with an alignment greater than natural"""

    natural = ctypes.alignment(typ)
    if alignment == None:
        alignment = typ._alignment_
    else:
        alignment = max(alignment, getattr(typ, "_alignment_", 1))

    if natural % alignment == 0:
        return typ * length
    eltsize = ctypes.sizeof(typ)
    eltalign = getattr(typ, "_alignment_", 1)
    if eltsize % eltalign != 0:
        raise TypeError("type %s can't have element alignment %d"
                " in an array" % (typ.__name__, alignment))
    key = (_aligned_array_type, (typ, length), alignment)
    ret = _aligned_type_cache.get(key)
    if ret == None:
        name = "%s_array_%d_aligned_%d" % (typ.__name__, length,
                           alignment)
        d = {"_type_": typ,
             "_length_": length,
             "_alignment_": alignment}
        ret = _aligned_array_type(name, (aligned_array,), d)
        _aligned_type_cache[key] = ret
    return ret

def aligned_type(typ, alignment):
    """Create a ctypes type with an alignment greater than natural"""

    if ctypes.alignment(typ) % alignment == 0:
        return typ
    if issubclass(typ, ctypes.Array):
        return aligned_array_type(typ._type_, typ._length_,
                      alignment)
    else:
        raise TypeError("unsupported type %s" % typ)