How to use 128 bit integers in Cython

Question

On my 64 bit computer the long long type has 64 bits.

print(sizeof(long long))
# prints 8

I need to use 128 bit integers and luckily GCC supports these. How can I use these within Cython?

The following doesn't work. Compiling foo.pyx containing just

cdef __int128_t x = 0

yields

$ cython foo.pyx 

Error compiling Cython file:
------------------------------------------------------------
...

cdef __int128_t x = 0
    ^
------------------------------------------------------------

foo.pyx:2:5: '__int128_t' is not a type identifier

Answer 1

EDIT: this is NOT a workaround anymore, this is the right way to do it. Refer also to @IanH's answer.

Now, the problem you have is that cython does not recognize your type, while gcc does. So we can try to trick cython.

File helloworld.pyx:

cdef extern from "header_int128.h":
    # this is WRONG, as this would be a int64. it is here
    # just to let cython pass the first step, which is generating
    # the .c file.
    ctypedef unsigned long long int128

print "hello world"

cpdef int foo():
    cdef int128 foo = 4
    return 32

File header_int128.h:

typedef __int128_t int128;

File setup.py:

from distutils.core import setup
from Cython.Build import cythonize

setup(ext_modules = cythonize("helloworld.pyx"))

Now, on my machine, when I run python setup.py build_ext --inplace, the first step passes, and the file helloworld.c is generated, and then the gcc compilation passes as well.

Now if you open the file helloworld.c, you can check that your variable foo is actually declared as an int128.

Be very careful with using this workaround. In particular, it can happen that cython will not require a cast in the C code if you assign an int128 to a int64 for example, because at that step of the process it actually does not distinguish between them.

Answer 2

I'll throw my two cents in here.

First, the solution proposed in the other answers saying to use an external typedef is not just a workaround, that is the way the Cython docs say things like this should be done. See the relevant section. Quote: "If the header file uses typedef names such as word to refer to platform-dependent flavours of numeric types, you will need a corresponding ctypedef statement, but you don’t need to match the type exactly, just use something of the right general kind (int, float, etc). For example ctypedef int word will work okay whatever the actual size of a word is (provided the header file defines it correctly). Conversion to and from Python types, if any, will also be used for this new type."

Also, it isn't necessary to actually create a header file with a typedef for a type you've already included somewhere else along the way. Just do this

cdef extern from *:
    ctypedef int int128 "__int128_t"

Or, if you feel like keeping the name the same in Cython as it is in C,

cdef extern from *:
    ctypedef int __int128_t

Here's a test to demonstrate that this is working. If the 128 bit arithmetic is working, a > 1, and a is representable as a 64 bit integer, the first function will print the same number back again. If it is not, integer overflow should cause it to print 0. The second function shows what happens if 64 bit arithmetic is used.

Cython file

# cython: cdivision = True

cdef extern from *:
    ctypedef int int128 "__int128_t"

def myfunc(long long a):
    cdef int128 i = a
    # set c to be the largest positive integer possible for a signed 64 bit integer
    cdef long long c = 0x7fffffffffffffff
    i *= c
    cdef long long b = i / c
    print b

def myfunc_bad(long long a):
    cdef long long i = a
    # set c to be the largest positive integer possible for a signed 64 bit integer
    cdef long long c = 0x7fffffffffffffff
    i *= c
    cdef long long b = i / c
    print b

In Python, after both functions have been imported, myfunc(12321) prints the correct value while myfunc_bad(12321) prints 0.

Answer 3

Here is an example for using the hack proposed by @Giulio Ghirardo.

The file cbitset.px contains:

typedef unsigned __int128 bitset;

The file bitset.pyx contains:

from libc.stdlib cimport malloc
from libc.stdio cimport printf

cdef extern from "cbitset.h":
    ctypedef unsigned long long bitset

cdef char* bitset_tostring(bitset n):
    cdef char* bitstring = <char*>malloc(8 * sizeof(bitset) * sizeof(char) + 1)
    cdef int i = 0
    while n:
        if (n & <bitset>1):
            bitstring[i] = '1'
        else:
            bitstring[i] = '0'

        n >>= <bitset>1
        i += 1
    bitstring[i] = '\0'
    return bitstring

cdef void print_bitset(bitset n):
    printf("%s\n", bitset_tostring(n))

The file main.pyx contains:

from bitset cimport print_bitset

cdef extern from "cbitset.h":
    ctypedef unsigned long long bitset

# x contains a number consisting of more than 64 1's
cdef bitset x = (<bitset>1 << 70) - 1

print_bitset(x)
# 1111111111111111111111111111111111111111111111111111111111111111111111

The file setup.py contains:

from distutils.core import setup
from Cython.Build import cythonize

setup(
    name="My app that used 128 bit ints",
    ext_modules=cythonize('main.pyx')
)

Compile this using the command

python3 setup.py build_ext --inplace

and run using the command

python3 -c 'import main'