Reading multiple json files from Spark

Question

I have a list of json files which I would like load in parallel.

I can't use read.json("*") cause files are not in the same folder and there is no specific pattern I can implement.

I've tried sc.parallelize(fileList).select(hiveContext.read.json) but hive context, as expected, doesn't exists in executor.

Any ideas?

Answer 1

Looks like I found the solution:

val text sc.textFile("file1,file2....")
val df = sqlContext.read.json(text)

Answer 2

Also, you can specify directory as a parameter:

cat 1.json
{"x": 1.0, "y": 2.0}
{"x": 1.5, "y": 1.0}
sudo -u hdfs hdfs dfs -put 1.json /tmp/test

cat 2.json
{"x": 3.0, "y": 4.0}
{"x": 1.8, "y": 7.0}
sudo -u hdfs hdfs dfs -put 2.json /tmp/test

sqlContext.read.json("/tmp/test").show()
+---+---+
|  x|  y|
+---+---+
|1.0|2.0|
|1.5|1.0|
|3.0|4.0|
|1.8|7.0|
+---+---+    

Answer 3

Function json(paths:String*) takes variable arguments. (documentation)

So you can change your code like this:

sc.read.json(file1, file2, ...)