Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Pyspark: Select all columns except particular columns

Tags:

I have a large number of columns in a PySpark dataframe, say 200. I want to select all the columns except say 3-4 of the columns. How do I select this columns without having to manually type the names of all the columns I want to select?

like image 489
Tshilidzi Mudau Avatar asked Jun 13 '18 13:06

Tshilidzi Mudau


1 Answers

In the end, I settled for the following :

  • Drop:

    df.drop('column_1', 'column_2', 'column_3')

  • Select :

    df.select([c for c in df.columns if c not in {'column_1', 'column_2', 'column_3'}])

like image 139
Tshilidzi Mudau Avatar answered Sep 20 '22 03:09

Tshilidzi Mudau