AssertionError: SparkContext._active_spark_context is not None

Question

I create an object which when running the __init__ function creates a map from a dictionary. This is done outside of any function or classes. So it runs when the module gets loaded during imports.

It works fine when I run it, but when I run it using SparkStreaming, I get an assertion error shown below. It is thrown inside the __init__ function of the class.

Why am I facing this issue only when I use spark streaming and how do I fix it?

File "some_file.py", line 58, in __init__
    some_map = F.create_map(*[F.lit(x) for x in chain(*some_dict.items())])
  File "some_file.py", line 58, in <listcomp>
    some_map = F.create_map(*[F.lit(x) for x in chain(*some_dict.items())])
  File "/databricks/spark/python/pyspark/sql/functions.py", line 139, in lit
    return col if isinstance(col, Column) else _invoke_function("lit", col)
  File "/databricks/spark/python/pyspark/sql/functions.py", line 85, in _invoke_function
    assert SparkContext._active_spark_context is not None
AssertionError

Answer 1

Try to add spark context to your code:

spark_context = pyspark.SparkContext.getOrCreate()