I am using Spark
on Windows
. I know in *nix
, accessing local file code likes this:
val textFile = sc.textFile("file:///usr/local/spark/README.md")
But how can I access a local file on Windows
? I have tried following methods:
val logFile = "C:\spark-1.3.1-bin-hadoop2.4\README.md"
val logFile = "file\\C:\spark-1.3.1-bin-hadoop2.4\README.md"
But all can't work.
To access the file in Spark jobs, use SparkFiles. get(fileName) to find its download location. A directory can be given if the recursive option is set to true. Currently directories are only supported for Hadoop-supported filesystems.
Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc. Spark supports text files, SequenceFiles, and any other Hadoop InputFormat.
Unfortunately in windows you have to escape "\"
.
Try:
"C:\\spark-1.3.1-bin-hadoop2.4\\README.md"
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With