Parsing Yann LeCun's MNIST IDX file format

Question

I would like to understand how to open this version of the MNIST data set. For example, the training set label file train-labels-idx1-ubyte is defined as:

TRAINING SET LABEL FILE (train-labels-idx1-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  60000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label

And I found some code online that seems to work, but do not understand how it works:

with open('train-labels-idx1-ubyte', 'rb') as f:
    bytes = f.read(8)
    magic, size = struct.unpack(">II", bytes)

print(magic) # 2049
print(size)  # 60000

My understanding is that struct.unpack interprets the second argument as a big-endian byte string of two 4-byte integers (See here). When I actually print the value of bytes, though, I get:

b'\x00\x00\x08\x01\x00\x00\xea`'

The first four-byte integer makes sense:

b'\x00\x00\x08\x01'

The first two bytes are 0. The next indicates the data are unsigned bytes. And 0x01 indicates a 1-dimensional vector of labels. Assuming my understanding is correct so far, what is happening with the next three (four?) bytes:

...\x00\x00\xea`

How does this translate to 60,000?

Answer 1

I wrote the following code in case anyone needs to parse the whole dataset of images (as it appears in the question's title), and not just the first two bytes.

import numpy as np
import struct

with open('samples/t10k-images-idx3-ubyte','rb') as f:
    magic, size = struct.unpack(">II", f.read(8))
    nrows, ncols = struct.unpack(">II", f.read(8))
    data = np.fromfile(f, dtype=np.dtype(np.uint8).newbyteorder('>'))
    data = data.reshape((size, nrows, ncols))

And just to check, show the first digit. In my case it's a 7.

import matplotlib.pyplot as plt
plt.imshow(data[0,:,:], cmap='gray')
plt.show()

Answer 2

To understand how it works, you need to convert it to its binary representation.

As you mentioned, Python is correctly extracting the right information:

>>> import struct
>>> with open('train-labels-idx1-ubyte', 'rb') as f:
...     data = f.read(8)
... 
>>> print(data)
b'\x00\x00\x08\x01\x00\x00\xea`'
>>> print(struct.unpack('>II', data))
(2049, 60000)

In the header of the string, there are two 4-bytes integers. We can see their binary and decimal representation if we iterate over data:

>>> for char in data:
...     print('{0:08b} - {0:3d} - {1:s}'.format(char, str(bytes([char]))))
... 
00000000 -   0 - b'\x00'
00000000 -   0 - b'\x00'
00001000 -   8 - b'\x08'
00000001 -   1 - b'\x01'
00000000 -   0 - b'\x00'
00000000 -   0 - b'\x00'
11101010 - 234 - b'\xea'
01100000 -  96 - b'`'

The easy part is to know that the first 4 bytes are the first integer (the magic number), and the next 4 bytes are the second integer (the number of items).

Then, given these last 4 bytes, there are two ways one can construct the integer value they represent.

The first option (the one used in MNIST), is big or high endian. Which means, that the MOST significant bytes are found first:

00000000 00000000 11101010 01100000

If you check the decimal value of this binary number, it is 60,000, the number of items in the MNIST dataset.

Also, we could interpret this as little endian. In this case, the LESS significant bytes are found first:

01100000 11101010 00000000 00000000

Which in its decimal representation, is the number 1,625,948,160.

So, if you simply convert each byte in \x00\x00\xea` to binary, and you find the decimal representation of that whole binary number (reverting the order of the bytes if little endian), you have the integer value they represent.