How to perform time series analysis that contains multiple groups in Python using fbProphet or other models?

Question

All,

My dataset looks like following. I am trying to predict the 'amount' for next 6 months using either the fbProphet or other model. But my issue is that I would like to predict amount based on each groups i.e A,B,C,D for next 6 months. I am not sure how to do that in python using fbProphet or other model ? I referenced official page of fbprophet, but the only information I found is that "Prophet" takes two columns only One is "Date" and other is "amount" .

I am new to python, so any help with code explanation is greatly appreciated!

import pandas as pd
data = {'Date':['2017-01-01', '2017-02-01', '2017-03-01', '2017-04-01','2017-05-01','2017-06-01','2017-07-01'],'Group':['A','B','C','D','C','A','B'],
       'Amount':['12.1','13','15','10','12','9.0','5.6']}
df = pd.DataFrame(data)
print (df)

output:

         Date Group Amount
0  2017-01-01     A   12.1
1  2017-02-01     B     13
2  2017-03-01     C     15
3  2017-04-01     D     10
4  2017-05-01     C     12
5  2017-06-01     A    9.0
6  2017-07-01     B    5.6

Answer 1

fbprophet requires two columns ds and y, so you need to first rename the two columns

df = df.rename(columns={'Date': 'ds', 'Amount':'y'})

Assuming that your groups are independent from each other and you want to get one prediction for each group, you can group the dataframe by "Group" column and run forecast for each group

from fbprophet import Prophet
grouped = df.groupby('Group')
for g in grouped.groups:
    group = grouped.get_group(g)
    m = Prophet()
    m.fit(group)
    future = m.make_future_dataframe(periods=365)
    forecast = m.predict(future)
    print(forecast.tail())

Take note that the input dataframe that you supply in the question is not sufficient for the model because group D only has a single data point. fbprophet's forecast needs at least 2 non-Nan rows.

EDIT: if you want to merge all predictions into one dataframe, the idea is to name the yhat for each observations differently, do pd.merge() in the loop, and then cherry-pick the columns that you need at the end:

final = pd.DataFrame()
for g in grouped.groups:
    group = grouped.get_group(g)
    m = Prophet()
    m.fit(group)
    future = m.make_future_dataframe(periods=365)
    forecast = m.predict(future)    
    forecast = forecast.rename(columns={'yhat': 'yhat_'+g})
    final = pd.merge(final, forecast.set_index('ds'), how='outer', left_index=True, right_index=True)

final = final[['yhat_' + g for g in grouped.groups.keys()]]

Answer 2

import pandas as pd
import numpy as np
from statsmodels.tsa.statespace.sarimax import SARIMAX
from statsmodels.tsa.arima_model import ARIMA
from statsmodels.tsa.stattools import adfuller
from matplotlib import pyplot as plt
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_squared_log_error  



# Before doing any modeling using ARIMA or SARIMAS etc Confirm that
# your time-series is stationary by using Augmented Dick Fuller test
# or other tests.

# Create a list of all groups or get from Data using np.unique or other methods
groups_iter = ['A', 'B', 'C', 'D']

dict_org = {}
dict_pred = {}
group_accuracy = {}

# Iterate over all groups and get data 
# from Dataframe by filtering for specific group
for i in range(len(groups_iter)):
    X = data[data['Group'] == groups_iter[i]]['Amount'].values
    size = int(len(X) * 0.70)
    train, test = X[0:size], X[size:len(X)]
    history = [x for in train]

    # Using ARIMA model here you can also do grid search for best parameters
    for t in range(len(test)):
        model = ARIMA(history, order = (5, 1, 0))
        model_fit = model.fit(disp = 0)
        output = model_fit.forecast()
        yhat = output[0]
        predictions.append(yhat)
        obs = test[t]
        history.append(obs)
        print("Predicted:%f, expected:%f" %(yhat, obs))
    error = mean_squared_log_error(test, predictions)
    dict_org.update({groups_iter[i]: test})
    dict_pred.update({group_iter[i]: test})

    print("Group: ", group_iter[i], "Test MSE:%f"% error)
    group_accuracy.update({group_iter[i]: error})
    plt.plot(test)
    plt.plot(predictions, color = 'red')
    plt.show()

Answer 3

I know this is old but I was trying to predict outcomes for different clients and I tried to use Aditya Santoso solution above but got into some errors, so I added a couple of modifications and finally this worked for me:

df = pd.read_csv('file.csv')
df = pd.DataFrame(df)
df = df.rename(columns={'date': 'ds', 'amount': 'y', 'client_id': 'client_id'})
#I had to filter first clients with less than 3 records to avoid errors as prophet only works for 2+ records by group
df = df.groupby('client_id').filter(lambda x: len(x) > 2)

df.client_id = df.client_id.astype(str)

final = pd.DataFrame(columns=['client','ds','yhat'])

grouped = df.groupby('client_id')
for g in grouped.groups:
    group = grouped.get_group(g)
    m = Prophet()
    m.fit(group)
    future = m.make_future_dataframe(periods=365)
    forecast = m.predict(future)
    #I added a column with client id
    forecast['client'] = g
    #I used concat instead of merge
    final = pd.concat([final, forecast], ignore_index=True)

final.head(10)