Menu
I have a dataframe with 15 columns (4 categorical and the rest numeric).
I have created dummy variables for every categorical variable. Now I want to find the number of variables in my new dataframe.
I tried calculating length of printSchema(), but is NoneType:
print type(df.printSchema())
984
Using countDistinct() SQL Function if you want to get count distinct on selected multiple columns, use the PySpark SQL function countDistinct() . This function returns the number of distinct elements in a group.
In this article, we will discuss how to get the number of rows and the number of columns of a PySpark dataframe. For finding the number of rows and number of columns we will use count() and columns() with len() function respectively. df. count(): This function is used to extract number of rows from the Dataframe.
Count the number of rows and columns of Dataframe using len() function. The len() function returns the length rows of the Dataframe, we can filter a number of columns using the df. columns to get the count of columns.
In Pyspark, there are two ways to get the count of distinct values. We can use distinct() and count() functions of DataFrame to get the count distinct of PySpark DataFrame. Another way is to use SQL countDistinct() function which will provide the distinct value count of all the selected columns.
You are finding it wrong way, Here is sample example for this and about printSchema:-
df = sqlContext.createDataFrame([
(1, "A", "X1"),
(2, "B", "X2"),
(3, "B", "X3"),
(1, "B", "X3"),
(2, "C", "X2"),
(3, "C", "X2"),
(1, "C", "X1"),
(1, "B", "X1"),
], ["ID", "TYPE", "CODE"])
# Python 2:
print len(df.columns) #3
# Python 3
print(len(df.columns)) #3
columns provides list of all columns and we can check len. Instead printSchema prints schema of df which have columns and their data type, ex below:-
root
|-- ID: long (nullable = true)
|-- TYPE: string (nullable = true)
|-- CODE: string (nullable = true)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
