What is a pythonic method to find a step (or spike) shape in a time series?

Question

I have a pandas dataframe (e.g. df) in which some values are suddenly jumping (like a step or spike). What is the best way to identify them?

I have written a very naive code by which the differences of value with a couple of the next and previous values are calculated. Then by comparing those, the programs will decide whether is a step or spike.

# to create a dataframe
df=pd.DataFrame(np.random.randn(25), index=pd.date_range(start='2010-1-1',end='2010-1-2',freq='H'), columns=['value'])

# to manipulate the dataframe
df[10:11] = -0.933463
df[11:12] = 15
df[12:13] = 15
df[13:14] = 15

# to calculated the differnces of a value with a couple next and previous values
df_diff = pd.DataFrame()
df_diff['p1'] = df['value'].diff(periods=1).abs()
df_diff['p2'] = df['value'].diff(periods=2).abs()
df_diff['n1'] = df['value'].diff(periods=-1).abs()
df_diff['n2'] = df['value'].diff(periods=-2).abs()

max=5  # as an eligible maximum value
results =  (df_diff['n1'] >max) & (df_diff['n1'] == df_diff['n2']) & (df_diff['p1']==0)

What I expect is:

2010-01-01 00:00:00    False
2010-01-01 01:00:00    False
2010-01-01 02:00:00    False
2010-01-01 03:00:00    False
2010-01-01 04:00:00    False
2010-01-01 05:00:00    False
2010-01-01 06:00:00    False
2010-01-01 07:00:00    False
2010-01-01 08:00:00    False
2010-01-01 09:00:00    False
2010-01-01 10:00:00    True
2010-01-01 11:00:00    True
2010-01-01 12:00:00    True
2010-01-01 13:00:00    True
2010-01-01 14:00:00    True
2010-01-01 15:00:00    False
2010-01-01 16:00:00    False
2010-01-01 17:00:00    False
2010-01-01 18:00:00    False
2010-01-01 19:00:00    False
2010-01-01 20:00:00    False
2010-01-01 21:00:00    False
2010-01-01 22:00:00    False
2010-01-01 23:00:00    False
2010-01-02 00:00:00    False

Answer 1

The value you choose for the down peak (df[10:11] = -0.933463) is too low to differentiate it from the other lows without more information.

So I changed this value to -7.

from scipy.signal import find_peaks
import pandas as pd
import numpy as np

# to create a dataframe
np.random.seed(42)
df=pd.DataFrame(np.random.randn(25), index=pd.date_range(start='2010-1-1',end='2010-1-2',freq='H'), columns=['value'])

# to manipulate the dataframe
df[10:11] = -7
df[11:12] = 15
df[12:13] = 15
df[13:14] = 15

peaks_up = find_peaks(df.value, prominence=4, plateau_size=1)
peaks_down = find_peaks(-df.value, prominence=4, plateau_size=1)

peaks_idx = np.unique(
    np.concatenate(
        [peaks_up[1]['left_edges'], peaks_up[0], peaks_up[1]['right_edges'],
         peaks_down[1]['left_edges'], peaks_down[0], peaks_down[1]['right_edges']],
        axis=0))
peaks_df = df.iloc[peaks_idx ]

To plot:

import matplotlib.pyplot as plt
import seaborn as sns

sns.lineplot(df.index, df.value)
plt.scatter(peaks_df.index, peaks_df.value, color="red")