import Spark source code into intellij, build Error: not found: type SparkFlumeProtocol and EventBatch

Question

IntelliJ: 14.1.4

Spark: 1.5 release source code

I'm importing Spark source code into IntelliJ, and following steps on Spark website.

I'm getting errors below when building and compiling the project. I've googled around and tried what is suggested here in spark user list to "Generate Sources and Update Folders" in Maven tool bar for "Spark Project External Flume Sink", but still with same errors.

I'm pretty sure it's a resolve issue since all other classes are successfully resolved. Maybe I'm not using IntelliJ correctly? Any suggestions please?

Error:(45, 66) not found: type SparkFlumeProtocol
  val transactionTimeout: Int, val backOffInterval: Int) extends SparkFlumeProtocol with Logging {
                                                                 ^
Error:(70, 39) not found: type EventBatch
  override def getEventBatch(n: Int): EventBatch = {

Answer 1

I've solved the problem, and it turns out that the "Spark Project External Flume Sink" is excluded when importing Spark source code under default settings.

What I did:

1. File -> Project Structure -> Modules -> "spark-streaming-flume-sink_2.10" -> Sources
2. in the tree display of folders, initially "target" folder is excluded, but the "SparkFlumeProtocol" and "EventBatch" classes are compiled into this folder.
3. set "target" folder as "Sources", then leave all under "target" as "Excluded" except for "scala-2.10", see attached screenshot.

In this way, the compiles classes are included and the classes can be resolved correctly after a re-build project.

--- update June-8-2016 --------

or more specifically, the whole path of this module

please note the type and color, it will affect the package name

package org.apache.spark.streaming.flume.sink;