SettingWithCopyWarning using pandas apply [duplicate]

Question

Trying to figure out why the below function is returning the dreaded SettingWithCopyWarning... Here is my function that intends to modify the dataframe df by reference.

def remove_outliers_by_group(df, cols):
    """
    Removes outliers based on median and median deviation computed using cols
    :param df: The dataframe reference
    :param cols: The columns to compute the median and median dev of
    :return:
    """
    flattened = df[cols].as_matrix().reshape(-1, )
    median = np.nanmedian(flattened)
    median_dev = np.nanmedian(np.abs(flattened) - median)
    for col in cols:
        df[col] = df[col].apply(lambda x: np.nan if get_absolute_median_z_score(x, median, median_dev) >= 2 else x)

And the offending line is df[col] = df[col].apply(lambda x: np.nan if get_absolute_median_z_score(x, median, median_dev) >= 2 else x) as per this error:

A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy df[col] = df[col].apply(lambda x: np.nan if get_absolute_median_z_score(x, median, median_dev) >= 2 else x)

What I don't understand is that I see this pattern all over the place, using something like df['a'] = df['a'].apply(lambda x: ...), so I can't imagine all of them are doing it wrong.

Am I doing it wrong? What is the best way to do this? I want to modify the original dataframe.

Thanks for your help.

Answer 1

The problem is due to the reassignement and not the fact that you use apply.

SettingWithCopyWarning is a warning that chained-indexing has been detected in an assignment. It does not necessarily mean anything has gone wrong.

To avoid, the warning, as adviced use .loc like this

df.loc[:, col] = df[col].apply(...)