Joining two dataframes in Spark

Question

When I'm trying to join two data frames using

DataFrame joindf = dataFrame.join(df, df.col(joinCol)); //.equalTo(dataFrame.col(joinCol)));

My program is throwing below exception

org.apache.spark.sql.AnalysisException: join condition 'url' of type string is not a boolean.;

Here joinCol value is url Need inputs as what could possibly cause these exceptions

Answer 1

join variants which take as a second argument Column expect that it can be evaluated as a boolean expression.

If you want a simple equi-join based on a column name use a version which takes a column name as a String:

String joinCol = "foo";
dataFrame.join(df, joinCol);

Answer 2

What that means is that the join condition should evaluate to an expression. Lets say we want to join 2 dataframes based on id, so what we can do is :

With Python:

df1.join(df2, df['id'] == df['id'], 'left')  # 3rd parameter is type of join which in this case is left join

With Scala:

df1.join(df2, df('id') === df('id'))    // create inner join based on id column