Menu
I want to do testing of Spark programs on a Mac. Spark is running and my spark scala program compiles: but there is a library (mesos.so ?) error at runtime:
Exception in thread "main" java.lang.UnsatisfiedLinkError: no mesos in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1758)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1045)
at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:46)
at spark.SparkContext.<init>(SparkContext.scala:170)
at com.blazedb.scala.ccp.spark.LoadRDD$.main(LoadRDD.scala:14)
What setup is required on os/x beyond the spark server itself for mesos in order to run a spark client program?
851
Spark can run on hardware clusters managed by Apache Mesos. The advantages of deploying Spark with Mesos include: dynamic partitioning between Spark and other frameworks. scalable partitioning between multiple instances of Spark.
Steps to use the cluster modeStart the MesosClusterDispatcher in your cluster: ./sbin/start-mesos-dispatcher.sh -master mesos://mesosmaster:5050. This will generally start the dispatcher at port 7077. From the client, submit a job to the mesos cluster by Spark-submit specifying the dispatcher URL.
Mesos Runtime Configuration Each option can be set in two ways: By passing it to the binary using --option_name=value , either specifying the value directly, or specifying a file in which the value resides ( --option_name=file://path/to/file ). The path can be absolute or relative to the current working directory.
You need to set 'MESOS_NATIVE_LIBRARY' environment variable, which is the location of libmesos.so. It's typically /usr/local/lib/libmesos.so.
# For Linux
$ export MESOS_NATIVE_LIBRARY='/usr/local/lib/libmesos.so'
# For OSX
$ export MESOS_NATIVE_LIBRARY='/usr/local/lib/libmesos.dylib'
I would recommend adding that line to your .bashrc as well, to avoid doing that every time.
200
If you want to use Spark with Mesos, there are instructions on the project website, including notes on how to find the path to the Mesos library on OS X.
As you've noticed, there are other deployment modes, including the local modes, that don't require Mesos to be installed.
Based on your stacktrace, it looks like you might be using an older version of Spark. Since Spark 0.8.0+, the packages have been moved into the org.apache.spark namespace, so you might need to use earlier versions of the docs if you don't want to upgrade.
23
if you build mesos from sources then all generated libs will be generated inside [MESOS_HOME]/src/.libs folder. You must delete the empty [MESOS_HOME]/.libs folder and create symbolic link to [MESOS_HOME]/src/.libs
Used commands are:
lost my problem "g++: error: ./.libs/libmesos.so: No such file or directory"
44
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
