Multiple Answers on stackoverflow for AWS Glue say to set the --conf table parameter. However, sometimes in a job we'll need to set multiple --conf key value pairs in 1 job.
I've tried the following ways to have multiple --conf values set all resulting in error:
--conf
to be equal and overwrites the value in the 1st parameter with the 2nd's value.spark.yarn.executor.memoryOverhead=1024 spark.yarn.executor.memoryOverhead=7g spark.yarn.executor.memory=7g
. This results in a failure to start the job.spark.yarn.executor.memoryOverhead=1024, spark.yarn.executor.memoryOverhead=7g, spark.yarn.executor.memory=7g
. This results in a failure to start the job.--conf
string separate each key value. E.G. spark.yarn.executor.memoryOverhead=1024 --conf spark.yarn.executor.memoryOverhead=7g --conf spark.yarn.executor.memory=7g
. This results in the glue job hanging.How do I set multiple --conf table parameters in AWS Glue?
You can pass multiple parameters as below:
Key: --conf
value: spark.yarn.executor.memoryOverhead=7g --conf spark.yarn.executor.memory=7g
This has worked for me.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With