Reading a sparse array from a binary file on Matlab

Question

I have to write a function that reads a two-dimensional array of doubles from a binary file (.dat) structured in one single column, whose name is provided by the single input argument of the function. The format of the file is the following: First of all, there are two uint32 numbers which correspond to the number of rows and columns of the array. After that, there is a double number which is the number of non-zero elements on the array. And then each non-zero element of the array is represented by two uint32 scalars and a double scalar in the file in this order: its row index (uint32), its column index (uint32), and its value (double). An example could be:

5
4
2
1
1
8
2
2
9

Which means that the array has 5 rows and 4 columns, a total of 2 non-zero elements. These elements will be found in position (1,1) (with a value of 8) and position (2,2) (with a value of 9). All the other elements are equal to 0. Therefore, the array would be:

8  0  0  0
0  9  0  0
0  0  0  0
0  0  0  0
0  0  0  0

The function must return the two-dimensional array that it reads from the file as an output argument, and if there is a problem opening the file, the function returns an empty array. For the moment I have tried with this code:

function A = sparse_array_in( filename )
fid = fopen( filename,'rt' );
if fid < 0
    A = [];
    return;
end
% Get total number of elements on the file
n = 0;
while (fgets(fid) ~= -1),
  n = n+1;
end
% Close then reopen
fclose(fid);
fid = fopen( filename,'rt' );
% Read size of array and number of non-zero elements
rows = fread( fid,1,'uint32' );
cols = fread( fid,1,'uint32' );
dims = [ rows,cols ];
non_zero = fread( fid,1,'uint32' );
% Create array of zeros
A = zeros( dims );   
% Fill array A with the values from the file
for i = 1:non_zero
    r = fread( fid,1,'uint32' );
    c = fread( fid,1,'uint32' );
    v = fread( fid,1,'double' );
    A(r,c) = v;
end
fclose( fid );
end

But it seems that it is not working. What am I missing?

Answer 1

There are a few of problems with your code:

1. If the file ID is negative, you should not only return the empty array (as you're doing), but also make sure the code doesn't proceed from that point. I would stick a return statement right after the assignment to A:

   if fid < 0
       A = [];
       return;
   end
   

2. Next when you're determining the number of lines in your text file, be advised that you're advancing the file pointer each time you call fgets and so when you finally finish reading all of the lines, the file pointer points at the end of the text file. Any further calls to fread, fgets, or any of the f* family that reads things from files will not give you anything because you're at the end of the file. What you need to do is close the file and open it again so that you can start reading from the file again. Therefore, close the file when you read the number of lines, then open it up again:

   % Get total number of elements on the file
   n = 0;
   while (fgets(fid) ~= -1),
     n = n+1;
   end 
   %// Close then reopen
   fclose(fid);
   fid = fopen( filename,'rt');
   

3. You're not using fread right. The second parameter tells you how many numbers of a particular type you want to read in. You are using 1, 2 or 3 which means that you're reading 1, 2 or 3 numbers of a particular type. Because you're using fread to read in individual numbers per call, they should all be 1. Also, determining the total number of lines seems rather superfluous to me. If you are given the total number of non-zero elements, why bother figuring out how many lines there are? You already know how many non-zero elements there are, so just iterate from 1 up to as many non-zero numbers. Therefore, try this:

   % Read size of array and number of non-zero elements
   rows = double(fread( fid,1,'uint32' )); %// Change
   cols = double(fread( fid,1,'uint32' )); %// Change
   dims = [ rows,cols ];
   non_zero = fread( fid,1,'uint32' ); %// Change
   % Create array of zeros
   A = zeros( dims );   
   % Fill array A with the values from the file
   for i = 1 : non_zero %// Change
       r = fread( fid,1,'uint32' ); %// Change
       c = fread( fid,1,'uint32' ); %// Change
       v = fread( fid,1,'double' ); %// Change
       A(r,c) = v;
   end
   %A = reshape( A,dims' ); %// Why are you reshaping?
   fclose( fid );
   

Minor comment: Why are you reshaping the matrix? If you already know the row and column locations of your matrix and are putting them exactly where they're supposed to go, why bother transposing?

---

Therefore, with the above comments, your code would look like this:

function A = sparse_array_in( filename )
fid = fopen( filename,'rt' );
if fid < 0
    A = [];
    return; %// Change
end

% Read size of array and number of non-zero elements
rows = fread( fid,1,'uint32'); %// Change
cols = fread( fid,1,'uint32'); %// Change
dims = [ rows,cols ];
non_zero = fread( fid,1,'uint32' ); %// Change
% Create array of zeros
A = zeros( dims );   
% Fill array A with the values from the file
for i = 1:non_zero
    r = fread( fid,1,'uint32' ); %// Change
    c = fread( fid,1,'uint32' ); %// Change
    v = fread( fid,1,'double' ); %// Change
    A(r,c) = v;
end
%// Change - remove reshape
fclose( fid );
end

---

Here's an example that shows that it works for a binary file. I've created the following scenario:

5
7
4
1
1
1
2
2
2
3
3
3
4
4
4

This is a 5 x 7 matrix where (1,1) = 1, (2,2) = 2, (3,3) = 3, (4,4) = 4 with 4 non-zero values. I create a binary file, then use the function I fixed above to get the results:

fid = fopen('sparse_binary.dat', 'w');
fwrite(fid, 5, 'uint32');
fwrite(fid, 7, 'uint32');
fwrite(fid, 4, 'uint32');
fwrite(fid, 1, 'uint32');
fwrite(fid, 1, 'uint32');
fwrite(fid, 1, 'double');
fwrite(fid, 2, 'uint32');
fwrite(fid, 2, 'uint32');
fwrite(fid, 2, 'double');
fwrite(fid, 3, 'uint32');
fwrite(fid, 3, 'uint32');
fwrite(fid, 3, 'double');
fwrite(fid, 4, 'uint32');
fwrite(fid, 4, 'uint32');
fwrite(fid, 4, 'double');
fclose(fid);
A = sparse_array_in('sparse_binary.dat');

I get for A:

A =

     1     0     0     0     0     0     0
     0     2     0     0     0     0     0
     0     0     3     0     0     0     0
     0     0     0     4     0     0     0
     0     0     0     0     0     0     0

... which is what we expect.