Issue while creating SparkSession object using SparkConf

Question

I am new to Spark and need some guidance on the below issue - Whenever I am trying to create a SparkSession object using a SparkConf object I am getting the below error -

AttributeError: 'SparkConf' object has no attribute '_get_object_id'

I am using Spark 2.3 with Python 3.7 in local mode .

sconf=SparkConf.setAppName("test")
ss=SparkSession.builder.config(conf=sconf).getOrCreate()

I have read some of the solutions available in internet but none of them has resolved my issue .

Even when I am trying to create the SparkSession object directly i.e. without explicit SparkConf object , then also I am getting the same error -

ss=SparkSession.builder.master("local").getOrCreate()

AttributeError: 'SparkConf' object has no attribute '_get_object_id'

Answer 1

Do not create a new SparkConf() object : it will be a python object, apparently not compatible with the non-python parts of Spark (it doesn't have the mandatory _get_object_id() method, as expressed by the error message).

The config() method from the builder is cumulative, so you should do :

ss = SparkSession \
    .builder \
    .appName("test") \
    .config("some.parameter", "some.value") \
    .config("some.other.parameter", "some.other.value") \ 
       etc...
    .getOrCreate()