Submit python dependencies into spark cluster

Question

Is there any proper way to submit complex dependencies into spark using python? I found following way while searching through the internet:

def import_pymystem3(x):
    import pymystem3
    return x

int_rdd = spark.sparkContext.parallelize([1,2,3,4])
int_rdd.map(lambda x: import_pymystem3(x))
int_rdd.collect()

However, this way of accessing imports is inconvenient due to map() usage: accessing external library from inside map() disables usage of this import on other RDDs.

Apache documentation suggests --py-files, and this is how I do it:

Create dependencies.txt, listing all of the dependencies I use in, then

sudo python36 -m pip install dependencies -r requirements.txt

sudo zip -r ../dependencies/zip .

and finally spark-submit --executor-memory 50g --driver-memory 50g --py-files [path to requirements.zip] [path to project.py]

And this is what I see: NotADirectoryError: [Errno 20] Not a directory: '/home/.../dependencies/dependencies.zip/sklearn/__check_build'

Moreover, other imports are not loaded as well: ModuleNotFoundError: No module named 'nltk'

Is there any working approach of submitting complex libraries into apache spark cluster using pyspark? All of the needed packages are installed on the worker nodes.

Answer 1

You are installing the dependencies in the system (or environment). If you want to make a zip, you should indicate the target path. in the pip command,

This is your coded adapted:

# Sudo should not be needed
python36 -m pip install -t ./dependencies -r requirements.txt
zip -r dependencies.zip ./dependencies