Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Spark Dataframe Maximum Column Count

What is the maximum column count of spark Dataframe? I tried getting it from data frame documentation but unable to find it.

like image 726
Saran Avatar asked Sep 07 '16 14:09

Saran


1 Answers

From the architectural perspective, they are scalable, so there should not be any limit on the column count, but it can give rise to uneven load on the nodes & may affect the overall performance of your transformations.

like image 191
KiranM Avatar answered Oct 04 '22 11:10

KiranM