How to load Impala table directly to Spark using JDBC?

Question

I am trying to write a spark job with Python that would open a jdbc connection with Impala and load a VIEW directly from Impala into a Dataframe. This question is pretty close but in scala: Calling JDBC to impala/hive from within a spark job and creating a table

How do I do this? There are plenty of examples for other datasources such as MySQL, PostgreSQL, etc. but I haven't seen one for Impala + Python + Kerberos. An example would be of great help. Thank you!

Tried this with information from the web but it didn't work.

SPARK Notebook

#!/bin/bash
export PYSPARK_PYTHON=/home/anave/anaconda2/bin/python
export HADOOP_CONF_DIR=/etc/hive/conf
export PYSPARK_DRIVER_PYTHON=/home/anave/anaconda2/bin/ipython
export PYSPARK_DRIVER_PYTHON_OPTS='notebook --ip=* --no-browser'

# use Java8
export JAVA_HOME=/usr/java/latest
export PATH=$JAVA_HOME/bin:$PATH

# JDBC Drivers for Impala
export CLASSPATH=/home/anave/impala_jdbc_2.5.30.1049/Cloudera_ImpalaJDBC41_2.5.30/*.jar:$CLASSPATH
export JDBC_PATH=/home/anave/impala_jdbc_2.5.30.1049/Cloudera_ImpalaJDBC41_2.5.30

# --jars $SRCDIR/spark-csv-assembly-1.4.0-SNAPSHOT.jar \
# --conf spark.sql.parquet.binaryAsString=true \
# --conf spark.sql.hive.convertMetastoreParquet=false

pyspark --master yarn-client \
        --driver-memory 4G \
        --executor-memory 2G \
        # --num-executors 10 \
        --jars /home/anave/spark-csv_2.11-1.4.0.jar $JDBC_PATH/*.jar
        --driver-class-path $JDBC_PATH/*.jar

Python Code

properties = {
    "driver": "com.cloudera.impala.jdbc41.Driver",
    "AuthMech": "1",
#     "KrbRealm": "EXAMPLE.COM",
#     "KrbHostFQDN": "impala.example.com",
    "KrbServiceName": "impala"
}

# imp_env is the hostname of the db, works with other impala queries ran inside python
url = "jdbc:impala:imp_env;auth=noSasl"

db_df = sqlContext.read.jdbc(url=url, table='summary', properties=properties)

I received this error msg (Full Error Log):
Py4JJavaError: An error occurred while calling o42.jdbc. : java.lang.ClassNotFoundException: com.cloudera.impala.jdbc41.Driver

Answer 1

You can use

--jars $(echo /dir/of/jars/*.jar | tr ' ' ',') 

instead of

--jars /home/anave/spark-csv_2.11-1.4.0.jar $JDBC_PATH/*.jar

or for another approach please see my answer