Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Use library in Spark-shell

I want to use this library in spark-shell and/or in a .scala file to manipulate some data. How do I do that? I cannot use maven.

EDIT for possible dupl: I also do not have a jar; if that is part of the solution, how do I make a jar from that library?

like image 298
Dimebag Avatar asked Dec 24 '22 00:12

Dimebag


1 Answers

The library you reference is available on Maven Central, and spark-shell can automatically download libraries from Maven Central and a few other popular repositories if you give it the correct Maven coordinates. You don't need to explicitly use Maven. (In fact, it even lets you specify your own additional Maven repositories and searches those as well.) See http://spark.apache.org/docs/latest/rdd-programming-guide.html#using-the-shell

In your case specifically, the command should be something like

./bin/spark-shell --master local[4] --packages "dk.tbsalling:aismessages:2.2.1"

Note: You can browse https://spark-packages.org/ to find spark packages.

like image 193
Yawar Avatar answered Jan 12 '23 00:01

Yawar