Python Pandas SettingWithCopyWarning copies vs new objects

Question

I'm working with a dataframe 'copy' created by sub-setting a previous one - see below:

import random
import pandas as pd
df = pd.DataFrame({'data':list(random.sample(range(10,100),25))})
df_filtered = df.query('data > 20 and data < 80')
df_filtered.rename(columns={'data':'observations'},inplace=True)

The problem is, when the rename method is called I receive a SettingWithCopy warning that, as I understand it, means I'm operating on a copy of the original (df in this case) object. The warning text is: "A value is trying to be set on a copy of a slice from a DataFrame"

I found this question that was answered using a different approach to subsetting. I prefer the Dataframe.query() method myself (syntax-wise). Is there a way I can create a new Dataframe object using the.query() method rather than the method suggested in the question I linked? I've tried a few options with iloc but haven't been successful thus-far.

Answer 1

You can always explicitly make a copy by calling .copy() on your filtered dataframe. Concretely, replace

df_filtered = df.query('data > 20 and data < 80')

with

df_filtered = df.query('data > 20 and data < 80').copy()

Does that get rid of the warning?

Answer 2

try this instead of using inplace=True:

In [12]: df_filtered = df.query('data > 20 and data < 80')

In [13]: df_filtered = df_filtered.rename(columns={'data':'observations'})

.rename() function returns a new object, so you can simply overwrite your DF with the returned new DF

if you use inplace the following is happening

from docs:

inplace : boolean, default False

Whether to return a new DataFrame. If True then value of copy is ignored.

Returns:

renamed : DataFrame (new object)

PS basically you should try to avoid using inplace=True and use df = df.function(...) technique instead