PySpark: TypeError: 'Column' object is not callable

Question

I'm loading data from HDFS, which I want to filter by specific variables. But somehow the Column.isin command does not work. It throws this error:

TypeError: 'Column' object is not callable

from pyspark.sql.functions import udf, col
variables = ('852-PI-769', '812-HC-037', '852-PC-571-OUT')
df = sqlContext.read.option("mergeSchema", "true").parquet("parameters.parquet")
same_var = col("Variable").isin(variables)
df2 = df.filter(same_var)

The schema looks like this:

df.printSchema()
root
 |-- Time: timestamp (nullable = true)
 |-- Value: float (nullable = true)
 |-- Variable: string (nullable = true)

Any idea what am I doing wrong? PS: It's Spark 1.4 with Jupyter Notebook.

Answer 1

The problem is that isin was added to Spark in version 1.5.0 and therefore not yet avaiable in your version of Spark as seen in the documentation of isin here.

There is a similar function in in the Scala API that was introduced in 1.3.0 which has a similar functionality (there are some differences in the input since in only accepts columns). In PySpark this function is called inSetinstead. Usage examples from the documentation:

df[df.name.inSet("Bob", "Mike")]
df[df.age.inSet([1, 2, 3])]

Note: inSet is depricated in version 1.5.0 and forward, isin should be used in newer versions.