Menu
I am using CassandraSQLContext from spark-shell to query data from Cassandra. So, I want to know two things one how to fetch more than 20 rows using CassandraSQLContext and second how do Id display the full value of column. As you can see below by default it append dots in the string values.
Code :
val csc = new CassandraSQLContext(sc) csc.setKeyspace("KeySpace") val maxDF = csc.sql("SQL_QUERY" ) maxDF.show Output:
+--------------------+--------------------+-----------------+--------------------+ | id| Col2| Col3| Col4| +--------------------+--------------------+-----------------+--------------------+ |8wzloRMrGpf8Q3bbk...| Value1| X| K1| |AxRfoHDjV1Fk18OqS...| Value2| Y| K2| |FpMVRlaHsEOcHyDgy...| Value3| Z| K3| |HERt8eFLRtKkiZndy...| Value4| U| K4| |nWOcbbbm8ZOjUSNfY...| Value5| V| K5| 
731
By default Spark with Scala, Java, or with Python (PySpark), fetches only 20 rows from DataFrame show() but not all rows and the column value is truncated to 20 characters, In order to fetch/display more than 20 rows and column full value from Spark/PySpark DataFrame, you need to pass arguments to the show() method.
The only way to show the full column content we are using show() function. show(): Function is used to show the Dataframe. n: Number of rows to display. truncate: Through this parameter we can tell the Output sink to display the full column content by setting truncate option to false, by default this value is true.
In Spark/PySpark, you can use show() action to get the top/first N (5,10,100 ..) rows of the DataFrame and display them on a console or a log, there are also several Spark Actions like take() , tail() , collect() , head() , first() that return top and last n rows as a list of Rows (Array[Row] for Scala).
If you want to print the whole value of a column, in scala, you just need to set the argument truncate from the show method to false :
maxDf.show(false) and if you wish to show more than 20 rows :
// example showing 30 columns of // maxDf untruncated maxDf.show(30, false) For pyspark, you'll need to specify the argument name :
maxDF.show(truncate = False) You won't get in nice tabular form instead it will be converted to scala object.
maxDF.take(50) If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
