Interpolate PANDAS df

Question

I know this subject was brought up a few times on stack overflow, however I'm still stumbling upon an interpolation problem.

I have a complex dataframe of a set of columns, which could look something like this if simplified:

df_new = pd.DataFrame(np.random.randn(5,7), columns=[402.3, 407.2, 412.3, 415.8, 419.9, 423.5, 428.3])
wl     = np.array([400.0, 408.2, 412.5, 417.2, 420.5, 423.3, 425.0])

So what I need to do is to interpolate column-wise, to the new assigned values of cols (wl), for each row.

And how to get the new dataframe with columns ONLY containing values presented in the wl array?

Answer 1

Use reindex to include wl as new columns (whose values will be filled with NaNs). Then use interpolate(axis=1) to interpolate across the columns. Strictly speaking interpolation is only done between known values. You could, however, use limit_direction='both' to fill NaN edge values in both the forward and backward directions:

>>> df_new.reindex(columns=df_new.columns.union(wl)).interpolate(axis=1, limit_direction='both')
      400.0     402.3     407.2     408.2     412.3     412.5     415.8     417.2     419.9     420.5     423.3     423.5     425.0     428.3
0  0.342346  0.342346  1.502418  1.102496  0.702573  0.379089  0.055606 -0.135563 -0.326732 -0.022298  0.282135  0.586569  0.164917 -0.256734
1 -0.220773 -0.220773 -0.567199 -0.789194 -1.011190 -0.485832  0.039526 -0.426771 -0.893069 -0.191818  0.509432  1.210683  0.414023 -0.382636
2  0.078147  0.078147  0.335040 -0.146892 -0.628824 -0.280976  0.066873 -0.881153 -1.829178 -0.960608 -0.092038  0.776532  0.458758  0.140985
3 -0.792214 -0.792214  0.254805  0.027573 -0.199659 -1.173250 -2.146841 -1.421482 -0.696124 -0.073018  0.550088  1.173194 -0.049967 -1.273128
4 -0.485818 -0.485818  0.019046 -1.421351 -2.861747 -1.020571  0.820605  0.097722 -0.625160 -0.782700 -0.940241 -1.097781 -0.809617 -0.521453

Note that Pandas DataFrames store values in a primarily column-based data structure. So computations are generally more efficient when done column-wise, not row-wise. Therefore, it might be better to transpose your dataframe:

df = df_new.T

and then proceed similarly as described above:

df = df.reindex(index=df.index.union(wl))
df = df.interpolate(limit_direction='both')

---

If you want to extrapolate edge values, you could use scipy.interpolate.interp1d with : fill_value='extrapolate':

import numpy as np
import pandas as pd
import scipy.interpolate as interpolate
np.random.seed(2018)

df_new = pd.DataFrame(np.random.randn(5,7), columns=[402.3, 407.2, 412.3, 415.8, 419.9, 423.5, 428.3])
wl = np.array([400.0, 408.2, 412.5, 417.2, 420.5, 423.3, 425.0, 500])

x = df_new.columns
y = df_new.values
newx = x.union(wl)
result = pd.DataFrame(
    interpolate.interp1d(x, y, fill_value='extrapolate')(newx),
    columns=newx)

yields

      400.0     402.3     407.2     408.2     412.3     412.5     415.8     417.2     419.9     420.5     423.3     423.5     425.0     428.3      500.0
0 -0.679793 -0.276768  0.581851  0.889017  2.148399  1.952520 -1.279487 -0.671080  0.502277  0.561236  0.836376  0.856029  0.543898 -0.142790 -15.062654
1  0.484717  0.110079 -0.688065 -0.468138  0.433564  0.437944  0.510221  0.279613 -0.165131 -0.362906 -1.285854 -1.351779 -0.758526  0.546631  28.904127
2  1.303039  1.230655  1.076446  0.628001 -1.210625 -1.158971 -0.306677 -0.563028 -1.057419 -0.814173  0.320975  0.402057  0.366778  0.289165  -1.397156
3  2.385057  1.282733 -1.065696 -1.191370 -1.706633 -1.618985 -0.172797 -0.092039  0.063710  0.114863  0.353577  0.370628 -0.246613 -1.604543 -31.108665
4 -3.360837 -2.165729  0.380370  0.251572 -0.276501 -0.293597 -0.575682 -0.235060  0.421854  0.469009  0.689062  0.704780  0.498724  0.045401  -9.804075

---

If you wish to create a DataFrame containing only the wl columns, you could sub-select those columns using result[wl], or you could simplying interpolate only at the wl values:

result_wl = pd.DataFrame(
    interpolate.interp1d(x, y, fill_value='extrapolate')(wl),
    columns=wl)