Unable to upload jar via livy client to livy session

Question

Code :

LivyClient client = null;
try
{
  String livyUrl = "http://dummy16814.zycus.net:8998";
  client = new LivyClientBuilder().setURI(new URI(livyUrl)).build();

  String piJar = "E:\\livy_old.jar";
  System.err.printf("Uploading %s to the Spark context...\n", piJar);
  client.uploadJar(new File(piJar)).get();

  int samples = 10;
  System.err.printf("Running PiJob with %d samples...\n", samples);

  double pi = client.submit(new PiJob()).get();
  System.out.println("Pi is roughly: " + pi);
}
finally
{
  client.stop(true);
}

Getting below exception while uploading jar

Uploading E:\livy_old.jar to the Spark context... Exception in thread "main" java.util.concurrent.ExecutionException: java.io.IOException: Bad Request: "requirement failed: Local path /root/.livy-sessions/61a5e39c-d199-4bb8-967b-960b4e3e9ee3/livy_old.jar cannot be added to user sessions." at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.zycus.spark.PiJob.main(PiJob.java:64) Caused by: java.io.IOException: Bad Request: "requirement failed: Local path /root/.livy-sessions/61a5e39c-d199-4bb8-967b-960b4e3e9ee3/livy_old.jar cannot be added to user sessions." at com.cloudera.livy.client.http.LivyConnection.sendRequest(LivyConnection.java:197) at com.cloudera.livy.client.http.LivyConnection.post(LivyConnection.java:162) at com.cloudera.livy.client.http.HttpClient$2.call(HttpClient.java:155) at com.cloudera.livy.client.http.HttpClient$2.call(HttpClient.java:152) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Answer 1

By default Livy does not allow local file to be attached to user session: https://groups.google.com/a/cloudera.org/forum/#!topic/livy-user/mm-XEhANDHU I resolved this problem by specifying the local path in livy-server-0.3.0/conf/livy.conf:

livy.file.local-dir-whitelist =/root/.livy-sessions/

This should have been mentioned in their documentation

Answer 2

Add spark.master yarn-cluster to the spark configuration file in my case it is spark-defaults.conf.

Add this entry on all nodes in cluster.

Answer 3

You have to modify two properties in livy.conf if you are running locally.

livy.spark.master = local,

livy.file.local-dir-whitelist=[/path/tothe/file]

I was able to get it working after changing these two params