how to loop through each row of dataFrame in pyspark

Question

E.g

sqlContext = SQLContext(sc)  sample=sqlContext.sql("select Name ,age ,city from user") sample.show() 

The above statement prints theentire table on terminal. But I want to access each row in that table using for or while to perform further calculations.

Answer 1

You simply cannot. DataFrames, same as other distributed data structures, are not iterable and can be accessed using only dedicated higher order function and / or SQL methods.

You can of course collect

for row in df.rdd.collect():     do_something(row) 

or convert toLocalIterator

for row in df.rdd.toLocalIterator():     do_something(row) 

and iterate locally as shown above, but it beats all purpose of using Spark.

Answer 2

To "loop" and take advantage of Spark's parallel computation framework, you could define a custom function and use map.

def customFunction(row):     return (row.name, row.age, row.city)  sample2 = sample.rdd.map(customFunction) 

or

sample2 = sample.rdd.map(lambda x: (x.name, x.age, x.city)) 

The custom function would then be applied to every row of the dataframe. Note that sample2 will be a RDD, not a dataframe.

Map may be needed if you are going to perform more complex computations. If you just need to add a simple derived column, you can use the withColumn, with returns a dataframe.

sample3 = sample.withColumn('age2', sample.age + 2)