MatLab Bottleneck

Question

I am working with big arrays (~6x40million) and my code is showing great bottlenecks. I am experienced programming in MatLab, but don't know much about the inner processes (like memory and such...).

My code looks as follows(Just the essentials, of course all variables are initialized, specially the arrays in loops, I just don't want to bomb you all with code ):

First I read the file,

 disp('Point cloud import and subsampling')
    tic
    fid=fopen(strcat(Name,'.dat'));
    C=textscan(fid, '%d%d%f%f%f%d'); %<= Big!
    fclose(fid);

then create arrays out of the contents,

    y=C{1}(1:Subsampling:end)/Subsampling;
    x=C{2}(1:Subsampling:end)/Subsampling;
    %... and so on for the other rows    

    clear C %No one wants 400+ millon doubles just lying around.

And clear the cell array (1), and create some images and arrays with the new values

for i=1:length(x)

    PCImage(y(i)+SubSize(1)-maxy+1,x(i)+1-minx)=Reflectanse(i);     
    PixelCoordinates(y(i)+SubSize(1)-maxy+1,x(i)+1-minx,:)=Coordinates(i,:);

end
toc

Everything runs more or less smoothly until here, but then I manipulate some arrays

disp('Overlap alignment')
   tic

    PCImage=PCImage(:,[1:maxx/2-Overlap,maxx/2:end-Overlap]); %-30 overlap?
    PixelCoordinates=PixelCoordinates(:,[1:maxx/2-Overlap,maxx/2:end-Overlap],:);
    Sphere=Sphere(:,[1:maxx/2-Overlap,maxx/2:end-Overlap],:);

toc 

and this is a big bottleneck, but it gets worst at the next step

disp('Planar view and point cloud matching')
   tic

   CompImage=zeros(max(SubSize(1),PCSize(1)),max(SubSize(2),PCSize(2)),3);

   CompImage(1:SubSize(1),1:SubSize(2),2)=Subimage; %ExportImage Cyan
   CompImage(1:SubSize(1),1:SubSize(2),3)=Subimage;
   CompImage(1:PCSize(1),1:PCSize(2),1)=PCImage; %PointCloudImage Red

  toc

Output

Point cloud import and subsampling

Elapsed time is 181.157182 seconds.

Overlap alignment

Elapsed time is 408.750932 seconds.

Planar view and point cloud matching

Elapsed time is 719.383807 seconds.

My questions are: will clearing unused objects like C in 1 have any effect? (it doesn't seem like that)

Am I overseeing any other important mechanisms or rules of thumb, or is the whole thing just too much and supposed to happen like this?

Answer 1

When subsref is used, matlab makes a copy of the sub referenced elements. This may be costly for large arrays. Often it will be faster to catenate vectors like

res = [a,b,c];

This is not possible with the current code as written above, but if the code could be modified to make this work, it may save some time.

EDIT For multi-dimensional arrays you need to use cat

CompImage = cat(dim,Subimage,Subimage,PCImage);

where dim is 3 for this example.