Bit-operations on large number of bytes

Question

Doing some exercises with simple file encryption/decryption and am currently just reading in a bunch of bytes and performing the appropriate bit-operations on each byte one at a time, then writing them to the output file.

This method seems pretty slow. For example, if I want to XOR every byte by 0xFF, I would loop over each byte and XOR by 0xFF, rather than do some magic and every byte is XOR'd quickly.

Are there better ways to perform bit operations rather than a byte at a time?

Answer 1

Using the bitwise array operations from numpy may be what you're looking for.

Answer 2

No matter what, it appears that each byte would have to be

• read from memory,
• modified in some fashion, and
• written back to memory.

You can save a bit (no pun intended) of time by operating on multiple bytes at a time, for example by performing the XOR operation on 4, or even 8 bytes integers, hence dividing the overhead associated with the management of the loop by, roughly, a factor of 4 or 8, but this improvement would likely not amount to a significant gain for the overall algorithm.

Additional improvements can be found by replacing the "native" bit operations (XOR, Shifts, Rotations and the like) of the CPU/Language by reading pre-computed values in a table. Beware however that these native operations are typically rather optimized, and that you must be very diligent in designing the equivalent operations externally, and in measuring precisely the relative performance of these operations.

Edit: Oops, I just noted the [Python] tag, and also the reference to numpy in another response.
Beware... while the Numpy bitwise array suggestion is plausible, it all depends on actual parameters of the problem at hand. For example a fair amount of time may be lost in lining up the underlying arrays implied by numpy's bitwise function. See this Stack Overflow question which seems quite relevant. While focused on the XOR operation, this question provides quite a few actionable hints for both improving loops etc. and for profiling in general.