Spark - Adding JDBC Driver JAR to Google Dataproc

Question

I am trying to write via JDBC:

df.write.jdbc("jdbc:postgresql://123.123.123.123:5432/myDatabase", "myTable", props)

The Spark docs explain that the configuration option spark.driver.extraClassPath cannot be used to add JDBC Driver JARs if running in client mode (which is the mode Dataproc runs in) since the JVM has already been started.

I tried adding the JAR path in Dataproc's submit command:

gcloud beta dataproc jobs submit spark ... 
     --jars file:///home/bryan/org.postgresql.postgresql-9.4-1203-jdbc41.jar

I also added the command to load the driver:

  Class.forName("org.postgresql.Driver")

But I still get the error:

java.sql.SQLException: No suitable driver found for jdbc:postgresql://123.123.123.123:5432/myDatabase 

Answer 1

From my experience adding driver to the properties usually solves the problem:

props.put("driver", "org.postgresql.Driver")
db.write.jdbc(url, table, props)

Answer 2

You may want to try adding --driver-class-path to the very end of your command arguments:

gcloud beta dataproc jobs submit spark ... 
    --jars file:///home/bryan/org.postgresql.postgresql-9.4-1203-jdbc41.jar \
    --driver-class-path /home/bryan/org.postgresql.postgresql-9.4-1203-jdbc41.jar

Another approach if you're staging the jarfile onto the cluster before the job anyway is to dump the jarfile you need into /usr/lib/hadoop/lib/ where it should automatically be part of the driver classpath for both Hadoop and Spark jobs.