How Spark read file with underline the beginning of the file name?

Question

When I use Spark to parse log files, I notice that if the first character of filename is _ , the result will be empty. Here is my test code:

SparkSession spark = SparkSession
  .builder()
  .appName("TestLog")
  .master("local")
  .getOrCreate();
JavaRDD<String> input = spark.read().text("D:\\_event_2.log").javaRDD();
System.out.println("size : " + input.count());

If I modify the file name to event_2.log, the code will run it correctly. I found that the text function is defined as:

@scala.annotation.varargs
def text(paths: String*): Dataset[String] = {
  format("text").load(paths : _*).as[String](sparkSession.implicits.newStringEncoder)
}

I think it could be due to _ being scala's placeholder. How can I avoid this problem?

Answer 1

This has nothing to do with Scala. Spark uses Hadoop Input API to read file, which ignore every file that starts with underscore(_) or dot (.)

I don't know how to disable this in Spark though.