Menu
I like to know the maximum number of columns I can have in the dataframe,Is there any limitations in maintaining number of columns in dataframes. Thanks.
386
We can get the maximum value from the column in the dataframe using the select() method. Using the max() method, we can get the maximum value from the column.
By default, only the first 20 rows will be printed out. In case you want to display more rows than that, then you can simply pass the argument n , that is show(n=100) .
You can add multiple columns to Spark DataFrame in several ways if you wanted to add a known set of columns you can easily do by chaining withColumn() or on select(). However, sometimes you may need to add multiple columns after applying some transformations n that case you can use either map() or foldLeft().
Sparing you the details, the answer is Yes, there is a limit for the size the number of columns in Apache Spark.
Theoretically speaking, this limit depends on the platform and the size of element in each column.
Don't forget that Java is limited by the size of the JVM and an executor is also limited by that size - Java largest object size in Heap.
I would go back an refer to this Why does Spark RDD partition has 2GB limit for HDFS? which refers to the limitation with HDFS on block/partition size.
So there is actually lots of restriction to take into account.
This means that you can easily find a hard limit (Int.MaxValue par ex.) but what is more important Spark scales well only long and relatively thin data. (like stated by pault).
Finally, you need to remember that fundamentally you cannot split a single record between executors/partitions. And there is a number of practical limitations (GC, disk IO) which make very wide data impractical. Not to mention some known bugs.
Note : I mention @pault and @RameshMaharjan as this answer is actually the fruit of the discussion we had. (And ofc @zero323 for his comment from the other answer).
69
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
