Using pyspark to connect to PostgreSQL

Question

I am trying to connect to a database with pyspark and I am using the following code:

sqlctx = SQLContext(sc) df = sqlctx.load(     url = "jdbc:postgresql://[hostname]/[database]",     dbtable = "(SELECT * FROM talent LIMIT 1000) as blah",     password = "MichaelJordan",     user =  "ScottyPippen",     source = "jdbc",     driver = "org.postgresql.Driver" ) 

and I am getting the following error:

Any idea why is this happening?

Edit: I am trying to run the code locally in my computer.

Answer 1

Download the PostgreSQL JDBC Driver from https://jdbc.postgresql.org/download.html

Then replace the database configuration values by yours.

from pyspark.sql import SparkSession  spark = SparkSession \     .builder \     .appName("Python Spark SQL basic example") \     .config("spark.jars", "/path_to_postgresDriver/postgresql-42.2.5.jar") \     .getOrCreate()  df = spark.read \     .format("jdbc") \     .option("url", "jdbc:postgresql://localhost:5432/databasename") \     .option("dbtable", "tablename") \     .option("user", "username") \     .option("password", "password") \     .option("driver", "org.postgresql.Driver") \     .load()  df.printSchema() 

More info: https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html