Getting a jar does not exist and java.lang.ClassNotFoundException while running a simple twitter sentiment analysis code

Question

I have been struggling with this for some time now. I have been trying to run a simple twitter sentiment analysis code which seemed to work fine earlier but doesn't work anymore. I am using spark 1.3.1 with scala 2.10.4. I read somewhere that TwitterUtils doesn't work with spark 1.0+ so I tried a workaround. Everything seems to be in place according to the books..the right directory structure for scala, a fat jar using sbt assembly, the right paths but somehow spark is unable to pickup the jar file and I also get a ClassNotFoundException with it.

What could possibly be going wrong and how can I fix it?

EDIT:

The command line

../bin/spark-submit --class Sentimenter --master local[4] /home/ubuntu/spark/spark_examples/target/scala-2.10/twitter-sentiment-assembly-1.0.jar

The Error:

Warning: Local jar /home/ubuntu/spark/spark_examples/target/scala-2.10/twitter-sentiment-assembly-1.0.jar does not exist, skipping.
java.lang.ClassNotFoundException: Sentimenter
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:274)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:538)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

The build.sbt file:

lazy val root = (project in file(".")).
  settings(
    name := "twitter-sentiment",
    version := "1.0",
    scalaVersion := "2.10.4",
    mainClass in Compile := Some("Sentimenter")        
  )

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.3.1" % "provided",
  "org.apache.spark" %% "spark-streaming" % "1.3.1" % "provided",
  "org.apache.spark" % "spark-streaming-twitter_2.11" % "1.3.1"
)

// META-INF discarding
val meta = """META.INF(.)*""".r
assemblyMergeStrategy in assembly := {
  case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first
  case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
  case n if n.startsWith("reference.conf") => MergeStrategy.concat
  case n if n.endsWith(".conf") => MergeStrategy.concat
  case meta(_) => MergeStrategy.discard
  case x => MergeStrategy.first`

And here is the code I got from another forum on twitter sentiment

import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.SparkContext
import org.apache.spark.streaming.twitter._
import org.apache.spark.SparkConf

object Sentimenter {
  def main(args: Array[String]) {
    System.setProperty("twitter4j.oauth.consumerKey","xxxxxxxxxxxxx");
    System.setProperty("twitter4j.oauth.consumerSecret","xxxxxxxxxxxx");
    System.setProperty("twitter4j.oauth.accessToken","xxxxxxxxxxxx");
    System.setProperty("twitter4j.oauth.accessTokenSecret","xxxxxxxxxx");

    val filters = new Array[String](2)
    filters(0) = "Big Data"
    filters(1) = "geofencing"
    val sparkConf = new SparkConf().setAppName("TweetSentiment").setMaster("local[4]").set("spark.driver.allowMultipleContexts", "true")
    val sc = new SparkContext(sparkConf)
    // get the list of positive words
    val pos_list =  sc.textFile("/home/ubuntu/spark/src/main/scala/Positive_Words.txt") //Random
      .filter(line => !line.isEmpty())
      .collect()
      .toSet
    // get the list of negative words
    val neg_list =  sc.textFile("/home/ubuntu/spark/src/main/scala/Negative_Words.txt") //Random
      .filter(line => !line.isEmpty())
      .collect()
      .toSet
    // create twitter stream
    val ssc = new StreamingContext(sparkConf, Seconds(5))
    val stream = TwitterUtils.createStream(ssc, None, filters)
    val tweets = stream.map(r => r.getText)
    tweets.print() // print tweet text
    ssc.start()
    ssc.awaitTermination()
  }
}

Answer 1

I think that if you write

../bin/spark-submit --class Sentimenter --master local[4] --jars /home/ubuntu/spark/spark_examples/target/scala-2.10/twitter-sentiment-assembly-1.0.jar

or try to rearrange the flags in the spark-submit e.g.:

../bin/spark-submit --jars /home/ubuntu/spark/spark_examples/target/scala-2.10/twitter-sentiment-assembly-1.0.jar --master local[4] --class Sentimenter

you could have it work.