Can I add arguments to python code when I submit spark job?

Question

I'm trying to use spark-submit to execute my python code in spark cluster.

Generally we run spark-submit with python code like below.

# Run a Python application on a cluster ./bin/spark-submit \   --master spark://207.184.161.138:7077 \   my_python_code.py \   1000 

But I wanna run my_python_code.pyby passing several arguments Is there smart way to pass arguments?

Answer 1

Even though sys.argv is a good solution, I still prefer this more proper way of handling line command args in my PySpark jobs:

import argparse  parser = argparse.ArgumentParser() parser.add_argument("--ngrams", help="some useful description.") args = parser.parse_args() if args.ngrams:     ngrams = args.ngrams 

This way, you can launch your job as follows:

spark-submit job.py --ngrams 3 

More information about argparse module can be found in Argparse Tutorial

Answer 2

Yes: Put this in a file called args.py

#import sys print sys.argv 

If you run

spark-submit args.py a b c d e  

You will see:

['/spark/args.py', 'a', 'b', 'c', 'd', 'e']