Writing a RDD to a csv

Question

I have a RDD which is of the form

org.apache.spark.rdd.RDD[(String, Array[String])]

I want to write this into a csv file. Please suggest me how this can be done.

Doing myrdd.saveAsTextFile on gives the output as below.

(875,[Ljava.lang.String;@53620618)
(875,[Ljava.lang.String;@487e3c6c)

Answer 1

You can try:

myrdd.map(a => a._1 + "," + a._2.mkString(",")).saveAsTextFile

Answer 2

The other answer doesn't cater for escaping. Perhaps this more general solution?

import au.com.bytecode.opencsv.CSVWriter
import java.io.StringWriter
import scala.collection.JavaConversions._
val toCsv = (a: Array[String]) => {
  val buf = new StringWriter
  val writer = new CSVWriter(buf)
  writer.writeAll(List(a))
  buf.toString.trim
}
rdd.map(t => Array(t._1) ++ t._2)
   .map(a => toCsv(a))
   .saveAsTextFile(dest)