Menu
Is it possible to get spark to fetch the primaryResource jar from Nexus given the maven coordinates for it? I know it is possible to fetch dependencies from Nexus, but it doesn't appear to be possible to fetch the job jar from Nexus. I'm wondering if anyone else has tried it or has a workaround for the same. Thanks.
284
By default, Spark on YARN will use Spark jars installed locally, but the Spark jars can also be in a world-readable location on HDFS.
Adding jars to your applicationUse –jars or SparkContext. addJar to add jar to your application. Note that this option will make the jars available on the nodes in the cluster but the jars will not be added to the classpath.
Spark JAR files let you package a project into a single file so it can be run on a Spark cluster. A lot of developers develop Spark code in brower based notebooks because they're unfamiliar with JAR files.
Looks like spark-submit works with just --packages and --repositories and a dummy primaryResource.
The example:
--packages someGroupId:someArtifactId:someVersion \
--repositories http://nexus.company.com/nexus nonExistentFakeName.jar
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With