Menu
I have this dataframe in Spark I want to count the number of available columns in it. I know how to count the number of rows in column but I want to count number of columns.
val df1 = Seq(
("spark", "scala", "2015-10-14", 10,"rahul"),
("spark", "scala", "2015-10-15", 11,"abhishek"),
("spark", "scala", "2015-10-16", 12,"Jay"),
("spark","scala",null,13,"Kiran"))
.toDF("bu_name","client_name","date","patient_id","paitent _name")
df1.show
Can anybody tell me how I can count number of column count in this dataframe? I am using the Scala language.
438
Using the count () method, we can get the number of rows from the column, and finally, we can use the collect() method to get the count from the column. Where, df is the input PySpark DataFrame. column_name is the column to get the total number of rows (count).
To get the number of rows from the PySpark DataFrame use the count() function. This function returns the total number of rows from the DataFrame.
Similar to Python Pandas you can get the Size and Shape of the PySpark (Spark with Python) DataFrame by running count() action to get the number of rows on DataFrame and len(df. columns()) to get the number of columns.
To count the number of columns, simply do:
df1.columns.size
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
