Apache Spark reading UTF-16 CSV file

Question

I am trying to read a CSV file encoded in UTF-16.

val test = spark.read
.format("com.databricks.spark.csv")
.option("header", "true")
.option("inferSchema", "true")
.option("delimiter",";")
.option("dateFormat", "yyyy-MM-dd HH:mm:ss.SSS")
.option("encoding", "UTF-16")
.option("charset", "ISO-8859-1")
.load("...")

As a result I get extra lines:

Is it possible that Spark can only work with UTF-8 encoding? Or there is some other way to read UTF-16 CSV into a dataframe?

Answer 1

I was also facing a similar issue while trying to read a csv file having UTF-16 format.

I am using mac and I didn't knew what is the encoding of the csv file that I was reading. Initially I was reading the file without providing any option for encoding and it was giving me two '?' in front of the column name.

So tried to find out the encoding of the csv file that I was reading by using below command in my mac.

file -I yourFile

The output of this command showed me that the encoding of the file is charset=utf-16le

Now I am reading the file by providing this options and it works fine.

val df = spark.read.
            format("csv").
            option("quote", "\"").
            option("escape", "\\").
            option("multiLine", "true").
            option("inferSchema", "true").
            option("header","true").
            option("encoding", "UTF-16").
            load(sourceS3path)

There is no needing of providing the option of charset if it is just reading the UTF-16 file.