Process for comparing two datasets

Question

I have two datasets at the time (in the form of vectors) and I plot them on the same axis to see how they relate with each other, and I specifically note and look for places where both graphs have a similar shape (i.e places where both have seemingly positive/negative gradient at approximately the same intervals). Example:

So far I have been working through the data graphically but realize that since the amount of the data is so large plotting each time I want to check how two sets correlate graphically it will take far too much time.

Are there any ideas, scripts or functions that might be useful in order to automize this process somewhat?

Answer 1

The first thing you have to think about is the nature of the criteria you want to apply to establish the similarity. There is a wide variety of ways to measure similarity and the more precisely you can describe what you want for "similar" to mean in your problem the easiest it will be to implement it regardless of the programming language.

Having said that, here is some of the thing you could look at :

• correlation of the two datasets
• difference of the derivative of the datasets (but I don't think it would be robust enough)
• spectral analysis as mentionned by @thron of three
• etc. ...

Knowing the origin of the datasets and their variability can also help a lot in formulating robust enough algorithms.