how to vectorize array reformatting?

Question

I have a .csv file with data on each line in the format (x,y,z,t,f), where f is the value of some function at location (x,y,z) at time t. So each new line in the .csv gives a new set of coordinates (x,y,z,t), with accompanying value f. The .csv is not sorted.

I want to use imagesc to create a video of this data in the xy-plane, as time progresses. The way I've done this is by reformatting M into something more easily usable by imagesc. I'm doing three nested loops, roughly like this

M = csvread('file.csv');
uniqueX = unique(M(:,1));
uniqueY = unique(M(:,2));
uniqueT = unique(M(:,4));

M_reformatted = zeros(length(uniqueX), length(uniqueY), length(uniqueT));

for i = 1:length(uniqueX)
    for j = 1:length(uniqueY)
        for k = 1:length(uniqueT)
            M_reformatted(i,j,k) = M( ...
                M(:,1)==uniqueX(i) & ...
                M(:,2)==uniqueY(j) & ...
                M(:,4)==uniqueT(k), ...
                5 ...
            );
         end
     end
end

once I have M_reformatted, I can loop through timesteps k and use imagesc on M_reformatted(:,:,k). But doing the above nested loops is very slow. Is it possible to vectorize the above? If so, an outline of the approach would be very helpful.

edit: as noted in answers/comments below, I made a mistake in that there are several possible z-values, which I haven't taken into account. If only a single z-value, the above would be ok.

Answer 1

This vectorized solution allows for negative values of x and y and is many times faster than the non-vectorized solution (close to 20x times for the test case at the bottom).

The idea is to sort the x, y, and t values in lexicographical order using sortrows and then using reshape to build the time slices of M_reformatted.

---

The code:

idx = find(M(:,3)==0);            %// find rows where z==0
M2 = M(idx,:);                    %// M2 has only the rows where z==0
M2(:,3) = [];                     %// delete z coordinate in M2
M2(:,[1 2 3]) = M2(:,[3 1 2]);    %// change from (x,y,t,f) to (t,x,y,f)

M2 = sortrows(M2);                 %// sort rows by t, then x, then y

numT = numel(unique(M2(:,1)));     %// number of unique t values
numX = numel(unique(M2(:,2)));     %// number of unique x values
numY = numel(unique(M2(:,3)));     %// number of unique y values

%// fill the time slice matrix with data
M_reformatted = reshape(M2(:,4), numY, numX, numT);

---

Note: I am assuming y refers to the columns of the image and x refers to the rows. If you want these flipped, use M_reformatted = permute(M_reformatted,[2 1 3]) at the end of the code.

---

The test case I used for M (to compare the result to other solutions) has a NxNxN space with T times slices:

N = 10;
T = 10;

[x,y,z]     = meshgrid(-N:N,-N:N,-N:N);
numPoints   = numel(x);
x=x(:); y=y(:); z=z(:);
s = repmat([x,y,z],T,1);
t = repmat(1:T,numPoints,1);
M = [s, t(:), rand(numPoints*T,1)];
M = M( randperm(size(M,1)), : );

Answer 2

I don't think you need to vectorize. I think you change your algorithm.

You only need one loop to step through the lines of the CSV file. For every line, you have (x,y,z,t,f) so just store it in M_reformatted where it belongs. Something like this:

M_reformatted = zeros(max(M(:,1)), max(M(:,2)), max(M(:,4)));
for line = 1:size(M,2)
    z = M(line, 3);
    if z ~= 0, continue; end;
    x = M(line, 1);
    y = M(line, 2);
    t = M(line, 4);
    f = M(line, 5);
    M_reformatted(x, y, t) = f;
end

Also note that pre-allocating M_reformatted is a very good idea, but your code may have been getting the size wrong (depending on the data). I think using max like I did will always do the right thing.