Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Error running spark on databricks: constructor public XXX is not whitelisted

Tags:

apache-spark

pyspark

databricks

I was using azure databricks and trying to run some example python code from this page.

and got this exception:

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.classification.LogisticRegression(java.lang.String) is not whitelisted.

Thanks, Lidong

like image 438
lidong Avatar asked Mar 30 '19 03:03

lidong

People also ask

Does Databricks support Spark?

Databricks is a Unified Analytics Platform on top of Apache Spark that accelerates innovation by unifying data science, engineering and business. With our fully managed Spark clusters in the cloud, you can easily provision clusters with just a few clicks.

How does Spark work in Databricks?

Spark provides native bindings for the Java, Scala, Python, and R programming languages. In addition, it includes several libraries to support build applications for machine learning [MLlib], stream processing [Spark Streaming], and graph processing [GraphX]. Apache Spark consists of Spark Core and a set of libraries.

Is Databricks faster than Spark?

In conclusion, Databricks runs faster than AWS Spark in all the performance test. For data reading, aggregation and joining, Databricks is on average 30% faster than AWS and we observed significant runtime difference (Databricks being ~50% faster) in training machine learning models between the two platforms.

Does Databricks run on JVM?

Databricks supports multiple languages but you'll always get the best performance with JVM-based languages. Databricks has a few nice features that makes it ideal for parallelizing data science, unlike leading ETL tools.

1 Answers

This error shows up with some library methods when using High Concurrency cluster with credential pass through enabled. If that is your scenario a work around that may be an option is to use a different cluster mode.

py4j.security.Py4JSecurityException: ... is not whitelisted This exception is thrown when you have accessed a method that Azure Databricks has not explicitly marked as safe for Azure Data Lake Storage credential passthrough clusters. In most cases, this means that the method could allow a user on a Azure Data Lake Storage credential passthrough cluster to access another user’s credentials.

Reference: https://docs.azuredatabricks.net/spark/latest/data-sources/azure/adls-passthrough.html

like image 175
Dustin V Avatar answered Oct 05 '22 09:10

Dustin V
Sign in to Comment

Related questions

How to partition Spark RDD when importing Postgres using JDBC?
Using typesafe config with Spark on Yarn
How to avoid boxing bytes in array in custom datasource?
Spark: grouping rows in array by key
Converting mysql table to spark dataset is very slow compared to same from csv file
Pyspark: cast array with nested struct to string
Modify spark DataFrame column
Select columns that satisfy a condition
How to convert unix timestamp to the given timezone with Spark
Why does spark-ml ALS model returns NaN and negative numbers predictions?
Apply custom function to cells of selected columns of a data frame in PySpark
Spark SQL - reading csv with schema
Combine multiple raw files into single parquet file
Spark writing/reading to/from S3 - Partition Size and Compression
Authentication for Spark standalone cluster
split a Spark column of Array[String] into columns of String
Pickling monkey-patched Keras model for use in PySpark
Retain raw JSON as column in Spark DataFrame on read/load?
Why do I get so many empty partitions when repartionning a Spark Dataframe?
Apache Spark vs Spring Cloud data flow [closed]

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!