Reading MNIST dataset with javascript/node.js

Question

I'm trying to decode the dataset from this source: http://yann.lecun.com/exdb/mnist/

There is a description of the "very simple" IDX file type in the bottom, but I cannot figure it out.

What I'm trying to achieve is something like:

var imagesFileBuffer = fs.readFileSync(__dirname + '/train-images-idx3-ubyte');
var labelFileBuffer  = fs.readFileSync(__dirname + '/train-labels-idx1-ubyte');
var pixelValues      = {};

Do magic

pixelValues are now like:

// {
//   "0": [0,0,200,190,79,0... for all 784 pixels ... ],
//   "4": [0,0,200,190,79,0... for all 784 pixels ... ],

etc for all image entries in the dataset. I've tried to figure out the structure of the binary files, but failed.

Answer 1

I realized there would be duplicate keys in my structure of the pixelValues object, so I made an array of objects of it instaed. The following code will create the structure I'm after:

var dataFileBuffer  = fs.readFileSync(__dirname + '/train-images-idx3-ubyte');
var labelFileBuffer = fs.readFileSync(__dirname + '/train-labels-idx1-ubyte');
var pixelValues     = [];

// It would be nice with a checker instead of a hard coded 60000 limit here
for (var image = 0; image <= 59999; image++) { 
    var pixels = [];

    for (var x = 0; x <= 27; x++) {
        for (var y = 0; y <= 27; y++) {
            pixels.push(dataFileBuffer[(image * 28 * 28) + (x + (y * 28)) + 15]);
        }
    }

    var imageData  = {};
    imageData[JSON.stringify(labelFileBuffer[image + 8])] = pixels;

    pixelValues.push(imageData);
}

The structure of pixelValues is now something like this:

[
    {5: [28,0,0,0,0,0,0,0,0,0...]},
    {0: [0,0,0,0,0,0,0,0,0,0...]},
    ...
]

There are 28x28=784 pixel values, all varying from 0 to 255.

To render the pixels, use my for loops like I did above, rendering the first pixel in the upper left corner, then working towards the right.

Answer 2

Just a small improvement:

for (var image = 0; image <= 59999; image++) {

with 60000 there is an "entry" with null's at the very end of your pixelValues.

EDIT:

I got a little obsessed with details because I wanted to convert the MNIST dataset back to real and separate image files. So I have found more mistakes in your code.

1. it is definitely +16 because you have to skip the 16 Bytes of header data. This little mistake is reflected in your answer where the first pixel value of the first digit (with is a 5) is '28'. Which is actually the value that tells how many columns the image has - not the first pixel of the image.

2. Your nested for loops has to be turned inside-out to get you the right pixel order - asuming you will "rebuild" your image from the upper left corner down to the lower right corner. With your code the image will be flipped along the axis that goes from the upper left to the lower right corner.

So your code should be:

var dataFileBuffer  = fs.readFileSync(__dirname + '/train-images-idx3-ubyte');
var labelFileBuffer = fs.readFileSync(__dirname + '/train-labels-idx1-ubyte');
var pixelValues     = [];

// It would be nice with a checker instead of a hard coded 60000 limit here
for (var image = 0; image <= 59999; image++) { 
    var pixels = [];

    for (var y = 0; y <= 27; y++) {
        for (var x = 0; x <= 27; x++) {
            pixels.push(dataFileBuffer[(image * 28 * 28) + (x + (y * 28)) + 16]);
        }
    }

    var imageData  = {};
    imageData[JSON.stringify(labelFileBuffer[image + 8])] = pixels;

    pixelValues.push(imageData);
}

Those little details wouldn't be an issue if you stay consistent and use those extracted data to -for example- train neural networks, because you will do the same with the testing dataset. But if you want to take that MNIST trained neural network and try to verify it with real life hand written digits, you will get bad results because the real images are not flipped.