Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how to select all columns that starts with a common label

I have a dataframe in Spark 1.6 and want to select just some columns out of it. The column names are like:

colA, colB, colC, colD, colE, colF-0, colF-1, colF-2

I know I can do like this to select specific columns:

df.select("colA", "colB", "colE")

but how to select, say "colA", "colB" and all the colF-* columns at once? Is there a way like in Pandas?

like image 791
user299791 Avatar asked Feb 11 '16 13:02

user299791


People also ask

How do I select a column with certain names in R?

To select a column in R you can use brackets e.g., YourDataFrame['Column'] will take the column named “Column”. Furthermore, we can also use dplyr and the select() function to get columns by name or index. For instance, select(YourDataFrame, c('A', 'B') will take the columns named “A” and “B” from the dataframe.

How do I select specific columns in a data frame?

To select a single column, use square brackets [] with the column name of the column of interest.

How do I select a column by name in python?

This is the most basic way to select a single column from a dataframe, just put the string name of the column in brackets. Returns a pandas series. Passing a list in the brackets lets you select multiple columns at the same time.


2 Answers

First grab the column names with df.columns, then filter down to just the column names you want .filter(_.startsWith("colF")). This gives you an array of Strings. But the select takes select(String, String*). Luckily select for columns is select(Column*), so finally convert the Strings into Columns with .map(df(_)), and finally turn the Array of Columns into a var arg with : _*.

df.select(df.columns.filter(_.startsWith("colF")).map(df(_)) : _*).show

This filter could be made more complex (same as Pandas). It is however a rather ugly solution (IMO):

df.select(df.columns.filter(x => (x.equals("colA") || x.startsWith("colF"))).map(df(_)) : _*).show 

If the list of other columns is fixed you could also merge a fixed array of columns names with filtered array.

df.select((Array("colA", "colB") ++ df.columns.filter(_.startsWith("colF"))).map(df(_)) : _*).show
like image 92
Michael Lloyd Lee mlk Avatar answered Oct 21 '22 08:10

Michael Lloyd Lee mlk


Python (tested in Azure Databricks)

selected_columns = [column for column in df.columns if column.startswith("colF")]
df2 = df.select(selected_columns)
like image 37
Eugene Lycenok Avatar answered Oct 21 '22 08:10

Eugene Lycenok