how to identify turning points in stock price data

Question

This question is a continuation of this one.

My goal is to find the turning points in stock price data.

So far I:

Tried differentiating the smoothed price set, with the help of Dr. Andrew Burnett-Thompson using the centered five-point method, as explained here.

I use the EMA20 of tick data for smoothing the data set.

For each point on the chart I get the 1st derivative (dy/dx). I create a second chart for the turning points. Each time the dy/dx is between [-some_small_value] and [+some_small_value] - I add a point to this chart.

The problems are: I don't get the real turning points, I get something close. I get too much or too little points - depening on [some_small_value]

I tried a second method of adding a point when dy/dx turns from negative to positive, which also creates too many points, maybe because I use EMA of tick data (and not of 1 minute closing price)

A third method is to divide the data set into slices of n points, and to find the minimum and maximum points. This works fine (not ideal), but it's lagging.

Anyone has a better method?

I attached 2 pictures of the output (1st derivative and n points min/max)

Answer 1

You could take the second derivative into account, meaning you should additionally (to your first derivative) evaluate (y_{i-1} + y_{i+1} - 2y_i) / (dx)². If this is above a certain threshold you have a maximum, if it is below you have a minimum and else you can discard it. This should throw out a lot of points that you keep using your method of finding extrema (y' = 0), because this condition is also valid for saddle points.

Answer 2

Here's just an idea, sort of an idea from a different angle, and possibly a very bad idea, but since differentiation isn't working, something like this might be a thought.

First, you need to determine a minimum meaningful X-axis interval. In your figure, if you take this to be too small, you will get false positives from the bumps. This is conceptually similar to the idea of smoothing your data. Call this interval dx.

Next, using a sliding window of size dx, generate a moving average curve corresponding to your curve. There are lots of different ways you could think about doing this (to remove statistical outliers, or to use more or fewer points in the window). Call this curve g(x), and your original curve f(x). Additionally, make a curve h(x) which gives some measure of the variability of data in the sliding window which you use to compute g(x) (standard deviation should work fine if you're using a few points from the interval).

Now, begin computing curves of the form c_m(x) = |f(x) - g(x)| - m * h(x). You can start with m = 1. Any points x for which c_m(x) is positive are candidates for a local min/max. Depending on how many hits you get, you can begin increasing or decreasing m. You can do this in a way similar to binarys search: if you want more points, make m = (min + m) / 2, and if you want fewer points, make m = (max + m) / 2 (adjusting min and max accordingly).

So here's an example of what I'm suggesting. Let's say we have the following series:

f(x) = [  1,   2,   4,   3,   2,   3,   6,   7,   8,   7, 
          5,   4,   3,   2,   2,   3,   2,   3,   5,   8,   9]

We choose dx = 5. We construct g(x) by taking a simple average of the points around x:

g(x) = [2.3, 2.5, 2.4, 2.8, 3.6, 4.2, 5.2, 6.2, 6.6, 6.2, 
        5.4, 4.2, 3.2, 2.8, 2.4, 2.4, 3.0, 4.2, 5.4, 6.3, 7.3]

h(x) = [1.2, 1.1, 1.0, 0.7, 1.4, 2.4, 2.3, 1.7, 1.0, 1.5,
        1.9, 1.7, 1.2, 0.7, 0.5, 0.6, 1.1, 2.1, 2.7, 2.4, 1.7]

With m = 1 we get:

c(x) = [0.1, xxx, 0.6, xxx, 0.2, xxx, xxx, xxx, 0.4, xxx,
        xxx, xxx, xxx, 0.1, xxx, 0.0, xxx, xxx, xxx, xxx, 0.0]

This seems to have worked fairly well, actually. Feel free to share thoughts. Note that this might be more or less the equivalent of differentiation, given the mean value theorem.