Filter Spark Dataframe with a variable

Question

Is this even possible in spark dataframe (1.6/2.1)

val data="some variable"

df.filter("column1"> data)

I can do this with a static value but cant figure out how to do filter by a variable.

Answer 1

import org.apache.spark.sql.functions._

val data="some variable"
df.filter(col("column1") > lit(data))

Answer 2

I'm not sure how you accomplished that with a literal either since what you have doesn't match any of the filter method signatures.

So yes, you can work with a non-literal, but try this:

import sparkSession.implicits._
df.filter($"column1" > data)

Note the $, which uses implicit conversion to turn the String into the Column named with that String. Meanwhile, this Column has a > method that takes an Any and returns a new Column. That Any will be your data value.