Ascii string of bytes packed into bitmap/bitstring back to string?

Question

I have a string that is packed such that each character was originally an unsigned byte but is stored as 7 bits and then packed into an unsigned byte array. I'm trying to find a quick way to unpack this string in Python but the function I wrote that uses the bitstring module works well but is very slow. It seems like something like this should not be so slow but I'm probably doing it very inefficiently...

This seems like something that is probably trivial but I just don't know what to use, maybe there is already a function that will unpack the string?

from bitstring import BitArray

def unpackString(raw):
    msg = ''

    bits = BitArray(bytes=raw)
    mask = BitArray('0b01111111')

    i = 0
    while 1:
        try:
            iByte = (bits[i:i + 8] & mask).int
            # value of 0 denotes a line break
            if iByte == 0:
                msg += '\n'
            elif iByte >= 32 and iByte <= 126:
                msg += chr(iByte)
            i += 7
        except:
            break

    return msg

Answer 1

This took me a while to figure out, as your solution seems to ignore the first bit of data. Given the input byte of 129 (0b10000001) I would expect to see 64 '1000000' printed by the following, but your code produces 1 '0000001' -- ignoring the first bit.

bs = b'\x81' # one byte string, whose value is 129 (0x81)
arr = BitArray(bs)
mask = BitArray('0b01111111')
byte = (arr[0:8] & mask).int
print(byte, repr("{:07b}".format(byte)))

Simplest solution would be to modify your solution to use bitstring.ConstBitStream -- I got an order of magnitude speed increase with the following.

from bitstring import ConstBitStream
def unpack_bitstream(raw):
    num_bytes, remainder = divmod(len(raw) * 8 - 1, 7)
    bitstream = ConstBitStream(bytes=raw, offset=1) # use offset to ignore leading bit
    msg = b''
    for _ in range(num_bytes):
        byte = bitstream.read("uint:7")
        if not byte:
            msg += b'\n'
        elif 32 <= byte <= 126:
            msg += bytes((byte,))
            # msg += chr(byte) # python 2
    return msg

However, this can be done quite easily using only the standard library. This makes the solution more portable and, in the instances I tried, faster by another order of magnitude (I didn't try the cythonised version of bitstring).

def unpack_bytes(raw, zero_replacement=ord("\n")):
    # use - 1 to ignore leading bit
    num_bytes, remainder = divmod(len(raw) * 8 - 1, 7)

    i = int.from_bytes(raw, byteorder="big")
    # i = int(raw.encode("hex"), 16) # python 2
    if remainder:
        # remainder means there are unused trailing bits, so remove these
        i >>= remainder

    msg = []
    for _ in range(num_bytes):
        byte = i & 127
        if not byte:
            msg.append(zero_replacement)
        elif 32 <= byte <= 126:
            msg.append(byte)
        i >>= 7
    msg.reverse()

    return bytes(msg)
    # return b"".join(chr(c) for c in msg) # python 2

I've used python 3 to create these methods. If you're using python 2 then there are a number of adjustments you'll need to make. I've added these as comments after the line they are intended to replace and marked them python 2.