Plotting Histogram for all columns in a Data Frame

Question

I am trying to draw histograms for all of the columns in my data frame. I imported pyspark and matplotlib. df is my data frame variable. plt is matplotlib.pyplot variable

I was able to draw/plot histogram for individual column, like this:

bins, counts = df.select('ColumnName').rdd.flatMap(lambda x: x).histogram(20)
plt.hist(bins[:-1], bins=bins, weights=counts)

But when I try to plot it for all variables I am having issues. Here is the for loop I have so far:

for x in range(0, len(df.columns)):
    bins, counts = df.select(x).rdd.flatMap(lambda x: x).histogram(20)
    plt.hist(bins[:-1], bins=bins, weights=counts)

How would I do it? Thanks in advance.

Answer 1

As an alternative to the for loop approach, I think you can try this:

df.hist(bins=30, figsize=(15, 10))

This will plot a histogram for each numerical attribute in the df DataFrame. Here, the bins and figsize arguments are just for customizing the output.