Menu
How can I ship C compiled modules (for example, python-Levenshtein) to each node in a Spark cluster?
I know that I can ship Python files in Spark using a standalone Python script (example code below):
from pyspark import SparkContext sc = SparkContext("local", "App Name", pyFiles=['MyFile.py', 'MyOtherFile.py']) But in situations where there is no '.py', how do I ship the module?
711
Since Python 3.3, a subset of its features has been integrated into Python as a standard library under the venv module. PySpark users can use virtualenv to manage Python dependencies in their clusters by using venv-pack in a similar way as conda-pack.
zip the contents of C:\python27 to an USB key. copy all python DLLS: copy C:\windows\system32\py*DLL K: (if K is your usb drive) unzip the contents of the archive somewhere on the second machine. add the DLLs directly in the python27 directory.
If you can package your module into a .egg or .zip file, you should be able to list it in pyFiles when constructing your SparkContext (or you can add it later through sc.addPyFile).
For Python libraries that use setuptools, you can run python setup.py bdist_egg to build an egg distribution.
Another option is to install the library cluster-wide, either by using pip/easy_install on each machine or by sharing a Python installation over a cluster-wide filesystem (like NFS).
176
There are two main options here:
.zip/.egg, pass it to SparkContext.addPyFile.pip install into a bootstrap code for the cluster's machines. People also suggest using python shell to test if the module is present on the cluster.
42
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
