Fast way to split an int into bytes

Question

If I have an int that fits into 32 bits, what is the fastest way to split it up into four 8-bit values in python? My simple timing test suggests that bit masking and shifting is moderately faster than divmod(), but I'm pretty sure I haven't thought of everything.

>>> timeit.timeit("x=15774114513484005952; y1, x =divmod(x, 256);y2,x = divmod(x, 256); y3, y4 = divmod(x, 256)")
0.5113952939864248
>>> timeit.timeit("x=15774114513484005952; y1=x&255; x >>= 8;y2=x&255; x>>=8; y3=x&255; y4= x>>8")
0.41230630996869877

Before you ask: this operation will be used a lot. I'm using python 3.4.

Answer 1

If you're doing it a lot, the fastest approach is to create a specialized Struct instance and pre-bind the pack method:

# Done once
int_to_four_bytes = struct.Struct('<I').pack

# Done many times (you need to mask here, because your number is >32 bits)
y1, y2, y3, y4 = int_to_four_bytes(x & 0xFFFFFFFF)

Using struct.pack directly would use a cached Struct object after the first use, but you'd pay cache lookup costs to go from format string to cached Struct every time, which is suboptimal. By creating and prebinding the pack of a Struct object (which is implemented in C in CPython), you bypass all Python byte code execution beyond the actual function call, and spend no time on cache lookups. On my machine, this runs in about 205 ns, vs. 267 ns for shift and mask (without reassigning x).

An alternate approach (for more general, not struct compatible sizes) is using int.to_bytes; for example, in this case:

y1, y2, y3, y4 = (x & 0xFFFFFFFF).to_bytes(4, 'big')

which takes about the same amount of time as the manually shifting and masking approach (it took 268 ns per loop), but scales to larger numbers of bytes better.

Answer 2

If you need something very fast, you should look into writing a C extension (see this). You can do this with or without cython. If you write a lot of these, I definitely recommend looking into cython.

This type of things is exactly what Python is awesome for: you can write your speed critical components in C directly, and it will interact (almost) seamlessly with your python code.