Cannot recognize the DataFrame for Java on spark in the Intellij platform

Question

I am quite new to Spark and I tried to manipulate some data using DataFrame library included in the spark library.

While doing the task, I have been facing the error indicating that my IDE cannot resolve the symbol DataFrame. I searched the related issues on the Internet and followed the solution they recommended, but the result didn't resolve my case.

Below is my code and the error throw Cannot resolve symbol DataFrame.

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.SQLContext;


public class Test {
    public static void main(String[] args) throws Exception{
        String inputFile = args[0];
        String outputFile = args[1];
        SparkConf conf = new SparkConf().setAppName("Data Transformation")
                .set("spark.serializer","org.apache.spark.serializer.KryoSerializer")
                .setMaster("local[*]");

        JavaSparkContext sc = new JavaSparkContext(conf);

        SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);

        //below statement throw error in the DataFrame declaration.
        DataFrame df = sqlContext.read().json("examples/src/main/resources/people.json");

        sc.stop();
        sc.close();
    }
}

and my pom.xml is as below.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.bistel</groupId>
    <artifactId>demo-maven</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>2.0.1</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>2.0.1</version>
        </dependency>
    </dependencies>
</project>

I don't know why I got this error although I specified the all the relevant libraries such as spark-core and spark-sql. I resolved the same issues on the Scala language but I couldn't for the Java.

Answer 1

You are missing the import for Dataframe:

import org.apache.spark.sql.DataFrame

Answer 2

The problem was version issue. The problem is the version of spark library I used. The version of 2.0.1 does not provide the DataFrame. I downgraded my version to 1.6.0 and I am able to include import org.apache.spark.sql.DataFrame.

Below is my new pom.xml.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.bistel</groupId>
    <artifactId>demo-maven</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>1.6.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.11 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>1.6.0</version>
        </dependency>

    </dependencies>
</project>