Difference between different hex types/representations in Python

Question

Working my way through some some Python code, I'm noticing there are a number of different representations for hexadecimal values. For example, if I choose a number like so:

xx = '\x03\xff'

Then the following command (a version of which I'm using to convert little endian to big endian)

yy = hex(struct.unpack('>H', xx)[0])

will return:

'0x3ff'

However, this command

zz = xx.encode('hex')

will return:

'03ff'

Finally, printing just the value out will return this

'\x03\xff'

From the looks of it there are three different types of hex then.

1. '\xFF'
2. '0xFF'
3. 'FF'

What's the difference?

Bonus points if someone could suggest a better way of converting a little endian to a big endian number. The above method for yy won't work for numbers larger than two bytes obstinately enough and I'm working with some hex strings that are 16 bytes long (including values that don't correspond to an ascii/integer value)

Answer 1

Anything using \x is a string escape code, which happens to use hex notation; other escape codes include \n for newlines, \' for a literal quote, etc. A python string is a sequence of bytes and you can specify literal values outside the ASCII printable range using such characters. When Python echoes a string value back at you in the interpreter, or you print the result of a repr() call on a string, Python will use such escapes to represent any byte that cannot be printed as a ASCII character instead:

>>> chr(65)
'A'
>>> chr(11)
'\x0b'

The hex() function returns a very specific string representation, as does .encode('hex') with the difference being that the former includes the 0x prefix. There are two more methods to produce such string representations; using the '%x' and '%X' string formatters, which use lowercase or uppercase letters for the representation.

>>> hex(11)
'0xb'
>>> '\x0b'.encode('hex')
'0b'
>>> '%x' % (11,)
'b'
>>> '%X' % (11,)
'B'

These are all string representations though (a series of ASCII characters), and have the same relation to the original data as str(number) is to integer data; you have changed the type and are further away from your goal of changing the byte ordering.

Changing a piece of binary information from little-ending to big-endian requires that you know the size of that piece of information. If all you have are short integers, then you need to flip every two bytes around, but if you have normal (long) integers, then you have 4 bytes per value and you need to reverse each 4 bytes.

Using the struct module is, I think, an excellent approach because you have to specify the value type. The following would interpret xx as a big-endian unsigned short int, then pack it back to a binary representation as a little-endian unsigned short int:

>>> import struct
>>> xx = '\x03\xff'
>>> struct.pack('<H', *struct.unpack('>H', xx))
'\xff\x03'

Answer 2

'\xFF' represents the string containing the character with ASCII code 255.

E.g.: print '\x41' gives 'A' (because this is the character with ASCII code 41)

the xx.encode('hex') and hex(struct.unpack('>H', xx)[0]) just give a human readable hexadecimal representation of the ASCII codes the string xx contains. This means that the resulting string contains a number of characters between a and f or 0 and 9.