Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Spark 2.0, DataFrame, filter a string column, unequal operator (!==) is deprecated

Tags:

I am trying to filter a DataFrame by keeping only those rows that have a certain string column non-empty.

The operation is the following:

df.filter($"stringColumn" !== "") 

My compiler shows that the !== is deprecated since I moved to Spark 2.0.1

How can I check if a string column value is empty in Spark > 2.0?

like image 560
Rami Avatar asked Oct 20 '16 12:10

Rami


1 Answers

Use =!= as a replacement:

df.filter($"stringColumn" =!= "") 
like image 153
user6022341 Avatar answered Sep 23 '22 15:09

user6022341