Menu
I'm using Spark 1.3.0 and Python. I have a dataframe and I wish to add an additional column which is derived from other columns. Like this,
>>old_df.columns [col_1, col_2, ..., col_m] >>new_df.columns [col_1, col_2, ..., col_m, col_n] where
col_n = col_3 - col_4 How do I do this in PySpark?
340
In PySpark, to add a new column to DataFrame use lit() function by importing from pyspark. sql. functions import lit , lit() function takes a constant value you wanted to add and returns a Column type, if you wanted to add a NULL / None use lit(None) .
A new column could be added to an existing Dataset using Dataset. withColumn() method. withColumn accepts two arguments: the column name to be added, and the Column and returns a new Dataset<Row>. The syntax of withColumn() is provided below.
Let's create a new column with constant value using lit() SQL function, on the below code. The lit() function present in Pyspark is used to add a new column in a Pyspark Dataframe by assigning a constant or literal value.
Using concat() Function to Concatenate DataFrame Columns Spark SQL functions provide concat() to concatenate two or more DataFrame columns into a single Column. It can also take columns of different Data Types and concatenate them into a single column. for example, it supports String, Int, Boolean and also arrays.
One way to achieve that is to use withColumn method:
old_df = sqlContext.createDataFrame(sc.parallelize( [(0, 1), (1, 3), (2, 5)]), ('col_1', 'col_2')) new_df = old_df.withColumn('col_n', old_df.col_1 - old_df.col_2) Alternatively you can use SQL on a registered table:
old_df.registerTempTable('old_df') new_df = sqlContext.sql('SELECT *, col_1 - col_2 AS col_n FROM old_df') If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
