How to modify a column value in a row of a spark dataframe?

Question

I am working with data frame with following structure

Here I need to modify each record so that if a column is listed in post_event_list I need to populate that column with corresponding post_column value. So in the above example for both records I need to populate col4 and col5 with post_col4 and post_col5 values. Can someone please help me to do this in pyspark.

Answer 1

Maybe this is what you want in pyspark2

suppose df is the DataFrame

row = df.rdd.first()

d = row.asDict()
d['col4'] = d['post_col4']
new_row = pyspark.sql.types.Row(**d) 

now we has a new Row object;

put these codes in a map function can help to change all df.

Answer 2

You can use when/otherwise in pyspark.sql.functions. Something likes:

import pyspark.sql.functions as sf
from pyspark.sql.types import BooleanType

contains_col4_udf = udf(lambda x: 'col4' in x, BooleanType())
df.select(sf.when(contains_col4_udf('post_event_list'), sf.col('post_col4')).otherwise(sf.col('col_4')).alias('col_4'))

Here is the doc: https://spark.apache.org/docs/1.6.2/api/python/pyspark.sql.html#pyspark.sql.Column.otherwise