According to http://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html# the new EMR release, 4.0.0, allows for a configuration json file to configure components. Also looking at http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emr-configure-apps.html, I have the json structured:
[
{
"Classification": "mapred-site",
"Properties": {
"mapreduce.map.memory.mb": "4096",
"mapreduce.map.java.opts": "-Xmx4096M"
}
},
{
"Classification": "hadoop-env",
"Properties": {},
"Configurations": [
{
"Classification": "export",
"Properties": {
"HADOOP_CLIENT_OPTS": "-Xmx4096M"
}
}
]
}
]
The Hadoop env property works, but the mapred-site properties are not reflected when a create an EMR cluster. I can always set these properties from Hive, but any ideas how to make it work with configuration file?
I experienced a similar issue, but with the "hadoop-env" classification.
From this post: https://github.com/aws/aws-cli/issues/1502 I found that making the keywords (Classification, Properties, ...) lower case solved the problem.
Note: I am using AWS console, not CLI.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With