How can I convert a (big endian) variable-sized binary byte array to an (unsigned) integer/long? As an example, '\x11\x34'
, which represents 4404
Right now, I'm using
def bytes_to_int(bytes): return int(bytes.encode('hex'), 16)
Which is small and somewhat readable, but probably not very efficient. Is there a better (more obvious) way?
The BigInteger class has a longValue() method to convert a byte array to a long value: long value = new BigInteger(bytes). longValue();
The intValue() method of Byte class is a built in method in Java which is used to return the value of this Byte object as int.
The Ints class also has a toByteArray() method that can be used to convert an int value to a byte array: byte[] bytes = Ints. toByteArray(value);
A byte is 8 bits. The . length of a byte array is its length in bytes, which is always a whole number of bytes, which is a multiple of 8 bits. A byte array can be any whole number of bytes long, so b.
Python doesn't traditionally have much use for "numbers in big-endian C layout" that are too big for C. (If you're dealing with 2-byte, 4-byte, or 8-byte numbers, then struct.unpack
is the answer.)
But enough people got sick of there not being one obvious way to do this that Python 3.2 added a method int.from_bytes
that does exactly what you want:
int.from_bytes(b, byteorder='big', signed=False)
Unfortunately, if you're using an older version of Python, you don't have this. So, what options do you have? (Besides the obvious one: update to 3.2, or, better, 3.4…)
First, there's your code. I think binascii.hexlify
is a better way to spell it than .encode('hex')
, because "encode" has always seemed a little weird for a method on byte strings (as opposed to Unicode strings), and it's in fact been banished in Python 3. But otherwise, it seems pretty readable and obvious to me. And it should be pretty fast—yes, it has to create an intermediate string, but it's doing all the looping and arithmetic in C (at least in CPython), which is generally an order of magnitude or two faster than in Python. Unless your bytearray
is so big that allocating the string will itself be costly, I wouldn't worry about performance here.
Alternatively, you could do it in a loop. But that's going to be more verbose and, at least in CPython, a lot slower.
You could try to eliminate the explicit loop for an implicit one, but the obvious function to do that is reduce
, which is considered un-Pythonic by part of the community—and of course it's going to require calling a function for each byte.
You could unroll the loop or reduce
by breaking it into chunks of 8 bytes and looping over struct.unpack_from
, or by just doing a big struct.unpack('Q'*len(b)//8 + 'B' * len(b)%8)
and looping over that, but that makes it a lot less readable and probably not that much faster.
You could use NumPy… but if you're going bigger than either 64 or maybe 128 bits, it's going to end up converting everything to Python objects anyway.
So, I think your answer is the best option.
Here are some timings comparing it to the most obvious manual conversion:
import binascii import functools import numpy as np def hexint(b): return int(binascii.hexlify(b), 16) def loop1(b): def f(x, y): return (x<<8)|y return functools.reduce(f, b, 0) def loop2(b): x = 0 for c in b: x <<= 8 x |= c return x def numpily(b): n = np.array(list(b)) p = 1 << np.arange(len(b)-1, -1, -1, dtype=object) return np.sum(n * p)
In [226]: b = bytearray(range(256)) In [227]: %timeit hexint(b) 1000000 loops, best of 3: 1.8 µs per loop In [228]: %timeit loop1(b) 10000 loops, best of 3: 57.7 µs per loop In [229]: %timeit loop2(b) 10000 loops, best of 3: 46.4 µs per loop In [283]: %timeit numpily(b) 10000 loops, best of 3: 88.5 µs per loop
For comparison in Python 3.4:
In [17]: %timeit hexint(b) 1000000 loops, best of 3: 1.69 µs per loop In [17]: %timeit int.from_bytes(b, byteorder='big', signed=False) 1000000 loops, best of 3: 1.42 µs per loop
So, your method is still pretty fast…
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With