Apache Spark Maven Dependencies for release and develop an app

Question

I have to follow this tutorial to create a uber jar for my Apache Spark application with maven.

I have set all Spark Dependencies in the pom with <scope>provided</scope>. This works very well, but now when I run the application locally, I get an error for missing Spark dependencies.

At the moment I had to remove provided tag from the pom.

How can I make provided spark dependencies only when building app for release?

I use Intellij as IDE for developing applications.

Answer 1

You can create seperate Maven profiles.

Best option is to have dependencyManagment section in POM where you'll specify versions, then in profiles you will have only groupId + artifactId + scope

For example:

<profile>
    <id>dev</id>
    <activation>
        <activeByDefault>false</activeByDefault>
    </activation>
    <dependencies>
    <!-- Here Spark deps without provided -->
        <dependency>
            <groupId>...</groupId>
            <artifactId>...</artifactId>
            <scope>compile</scope>
        </dependency>
    </dependencies>
</profile>
<profile>
    <id>prod</id>
    <dependencies>
    <!-- Here Spark deps with provided -->
        <dependency>
            <groupId>...</groupId>
            <artifactId>...</artifactId>
            <scope>provided</scope>
        </dependency>
    </dependencies>
</profile>