Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Improving stepping through an array twice (Nested loop on same array)

I have a large set of data that I want to cycle through in order to determine various statistics on the data set from a point in time 'D1' to a point in time in the future 'D2'. Basically, I want to add to a database each time the difference between the values is larger than 10. For example:

Datum[] data = x;
for( Datum d1 : data ){
    Datum[] tail = y; //From d1 up to 10 elements ahead
    for( Datum d2 : tail ){
        //Calculate difference
        if( (d2.val - d1.val) > 10 ){
            //Insert into database
        }
    }
}

My question is, is there a better algorithm/method to doing this? Since 9 elements from tail are reused in the next iteration of the outer loop, can I benefit somehow from that? My goal was to get this down to much less than (Big-O Notation) O(n2), but I can't wrap my head around it. Typically finding a D1, D2 pair that satisfies the criteria means that the next D1 element will have a greater chance of matching as well. Can I use that to my advantage?

I'm trying to get this to be as efficient as possible because the data set is just so large.

like image 323
Dave Avatar asked Aug 30 '11 00:08

Dave


1 Answers

An index-based for loop might perform much better than an iterator, since you can index the original array directly and avoid copying to a new array. You'd have much better memory locality, less chance of false sharing, etc.

like image 69
Ben Voigt Avatar answered Sep 30 '22 19:09

Ben Voigt