Menu
can i save a file into the local system with the saveAsTextFile syntax ?
This is how i'm writing the syntax to save a file: insert_df.rdd.saveAsTextFile("<local path>")
when i'm trying to do this i'm getting error as no permissions, but i have all the permissions to that specific local path, looks like it is treating the file as HDFS file.
728
You can save the RDD using saveAsObjectFile and saveAsTextFile method. Whereas you can read the RDD using textFile and sequenceFile function from SparkContext.
Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc. Spark supports text files, SequenceFiles, and any other Hadoop InputFormat.
Saving the text files: Spark consists of a function called saveAsTextFile(), which saves the path of a file and writes the content of the RDD to that file. The path is considered as a directory, and multiple outputs will be produced in that directory. This is how Spark becomes able to write output from multiple codes.
Physically, RDD is stored as an object in the JVM driver and refers to data stored either in permanent storage (HDFS, Cassandra, HBase, etc.) or in a cache (memory, memory+disks, disk only, etc.), or on another RDD. RDD stores the following metadata: Partitions — a set of data splits associated with this RDD.
I think you should try "file:///local path" instead of "/local path".
149
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
