Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Append not working with DataFrames in for loop

I have a time series with 3 columns. I need to copy column 2 after the last row and the same with the column 3.

I created a for loop and append the dataframes, but the append doesn't seem to work. There is no error or warning, just doesn't work.

Initial DataFrame dataImport_selVar100:

    val01_ambient_temperature   val01_ambient_winddir   val01_ambient_windspeed
measure_time            
2019-03-24 07:30:00 12.956060   108.200005  4.166667
2019-03-24 07:40:00 12.999207   103.000000  3.666667
2019-03-24 07:50:00 12.761206   106.500000  4.533333
2019-03-24 08:00:00 12.523205   98.413330   3.916667
2019-03-24 08:10:00 12.285204   97.853333   4.055000

Code:

counterTest=0
for column in dataImport_selVar100:
    if counterTest==0: #initialize
        result0=pd.DataFrame(dataImport_selVar100.iloc[:,counterTest])
    else:
        result1=pd.DataFrame(dataImport_selVar100.iloc[:,counterTest])
        result0.append(result1,ignore_index=True,sort=False)

    #print(result[column])
    counterTest +=1

The actual results are just the ones from result0 (100 rows)

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 100 entries, 2019-03-24 07:30:00 to 2019-03-25 00:00:00
Data columns (total 1 columns):
val01_ambient_temperature    100 non-null float64
dtypes: float64(1)
memory usage: 6.6 KB

The expected results are the sum of all the rows

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 300 entries, 2019-03-24 07:30:00 to 2019-03-25 00:00:00
Data columns (total 3 columns):
val01_ambient_temperature    100 non-null float64
val01_ambient_winddir        100 non-null float64
val01_ambient_windspeed      100 non-null float64
dtypes: float64(2)
memory usage: 7.0 KB
like image 867
JuanFT Avatar asked Jun 09 '19 21:06

JuanFT


People also ask

Does append work with DataFrames?

append() function is used to append rows of other dataframe to the end of the given dataframe, returning a new dataframe object. Columns not in the original dataframes are added as new columns and the new cells are populated with NaN value. Parameters: other : DataFrame or Series/dict-like object, or list of these.

Can you append DataFrames?

Concatenating DataFrames. We can use the concat function in pandas to append either columns or rows from one DataFrame to another.

How do you concatenate a loop in a DataFrame?

To append pandas DataFrame generated in a for a loop, we will first create an empty list and then inside the loop, we will append the modified value inside this empty list, and finally, outside the loop, we will concat all the values of the new list to create DataFrame.


1 Answers

result0.append(result1,ignore_index=True,sort=False)

Append returns the new dataframe. It does not happen inplace. You'll need:

result0 = result0.append(result1,ignore_index=True,sort=False)

Also be aware that append is very costly. Might be worth it to look into pd.concat.

like image 89
DSC Avatar answered Nov 08 '22 22:11

DSC