How do I use multiple conditions with pyspark.sql.functions.when()?

Question

I have a dataframe with a few columns. Now I want to derive a new column from 2 other columns:

from pyspark.sql import functions as F new_df = df.withColumn("new_col", F.when(df["col-1"] > 0.0 & df["col-2"] > 0.0, 1).otherwise(0)) 

With this I only get an exception:

py4j.Py4JException: Method and([class java.lang.Double]) does not exist 

It works with just one condition like this:

new_df = df.withColumn("new_col", F.when(df["col-1"] > 0.0, 1).otherwise(0)) 

Does anyone know to use multiple conditions?

I'm using Spark 1.4.

Answer 1

Use parentheses to enforce the desired operator precedence:

F.when( (df["col-1"]>0.0) & (df["col-2"]>0.0), 1).otherwise(0) 

Answer 2

when in pyspark multiple conditions can be built using &(for and) and | (for or), it is important to enclose every expressions within parenthesis that combine to form the condition

%pyspark dataDF = spark.createDataFrame([(66, "a", "4"),                                  (67, "a", "0"),                                  (70, "b", "4"),                                  (71, "d", "4")],                                 ("id", "code", "amt")) dataDF.withColumn("new_column",        when((col("code") == "a") | (col("code") == "d"), "A")       .when((col("code") == "b") & (col("amt") == "4"), "B")       .otherwise("A1")).show() 

when in spark scala can be used with && and || operator to build multiple conditions

//Scala val dataDF = Seq(           (66, "a", "4"), (67, "a", "0"), (70, "b", "4"), (71, "d", "4"           )).toDF("id", "code", "amt")     dataDF.withColumn("new_column",            when(col("code") === "a" || col("code") === "d", "A")           .when(col("code") === "b" && col("amt") === "4", "B")           .otherwise("A1"))           .show() 

Output:

+---+----+---+----------+ | id|code|amt|new_column| +---+----+---+----------+ | 66|   a|  4|         A| | 67|   a|  0|         A| | 70|   b|  4|         B| | 71|   d|  4|         A| +---+----+---+----------+