Spark withColumn() performing power functions

Question

I have a data frame df with columns "col1" and "col2". I want to create a third column which uses one of the columns as in an exponent function.

df = df.withColumn("col3", 100**(df("col1")))*df("col2")

However, this always results in:

TypeError: unsupported operand type(s) for ** or pow(): 'float' and 'Column'

I understand that this is due to the function taking df("col1") as a "Column" instead of the item at that row.

If I perform

results = df.map(lambda x : 100**(df("col2"))*df("col2"))

this works, but I can't append to my original data frame.

Any thoughts?

This is my first time posting, so I apologize for any formatting problems.

Answer 1

Since Spark 1.4 you can usepow function as follows:

from pyspark.sql import Row
from pyspark.sql.functions import pow, col

row = Row("col1", "col2")
df = sc.parallelize([row(1, 2), row(2, 3), row(3, 3)]).toDF()

df.select("*", pow(col("col1"), col("col2")).alias("pow")).show()

## +----+----+----+
## |col1|col2| pow|
## +----+----+----+
## |   1|   2| 1.0|
## |   2|   3| 8.0|
## |   3|   3|27.0|
## +----+----+----+

If you use an older version a Python UDF should do the trick:

import math
from pyspark.sql.functions import udf
from pyspark.sql.types import DoubleType

my_pow = udf(lambda x, y: math.pow(x, y), DoubleType())