Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

pyspark: counter part of like() method in dataframe

Is there any counter method for like() in spark dataframe (something as notLike())?

Or is there any other way to do it except using the traditonal SQL query?

I want to do just the opposite of the following:

df.where(col("_c2").like("XY6%")).show(5)
like image 678
Mrinal Avatar asked May 25 '17 09:05

Mrinal


People also ask

How do you use like in DataFrame PySpark?

In Spark & PySpark like() function is similar to SQL LIKE operator that is used to match based on wildcard characters (percentage, underscore) to filter the rows. You can use this function to filter the DataFrame rows by single or multiple conditions, to derive a new column, use it on when().


2 Answers

It worked :)

I had to use the negation operator (~) instead of the 'not' keyword.

df.where(~ col("_c2").like("XY6%")).show(5)
like image 119
Mrinal Avatar answered Nov 05 '22 23:11

Mrinal


Or you can do :

df.where( col("_c2").like("XY6%") == False ).show(5)
like image 20
Ahmed Avatar answered Nov 05 '22 23:11

Ahmed