How do I check for equality using Spark Dataframe without SQL Query?

Question

I want to select a column that equals to a certain value. I am doing this in scala and having a little trouble.

Heres my code

df.select(df("state")==="TX").show() 

this returns the state column with boolean values instead of just TX

Ive also tried

df.select(df("state")=="TX").show()  

but this doesn't work either.

Answer 1

I had the same issue, and the following syntax worked for me:

df.filter(df("state")==="TX").show() 

I'm using Spark 1.6.

Answer 2

There is another simple sql like option. With Spark 1.6 below also should work.

df.filter("state = 'TX'") 

This is a new way of specifying sql like filters. For a full list of supported operators, check out this class.