René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Using Boto3 to create EMR cluster.","articleBody":"Hi All,\n\nI am trying to automate the EMR cluster creation using Boto3. Which i am using to create the EMR cluster. I need a cluster created with Impala configured.\nHere is the parmas i passed to run_job_flow \nName='AutmateEMR',\n    ReleaseLabel='emr-4.6.0',\n    Instances={\n        'InstanceGroups': [{'InstanceCount':4,'InstanceRole':'CORE','InstanceType':'r3.8xlarge','Name':'slave'},{'InstanceCount':1,'InstanceRole':'MASTER','InstanceType':'r3.8xlarge','Name':'master'}],\n        'Ec2KeyName': 'MyKey',\n        'KeepJobFlowAliveWhenNoSteps': True,\n        'TerminationProtected': False,\n        'Ec2SubnetId': 'id',\n        'EmrManagedMasterSecurityGroup': 'value',\n        'EmrManagedSlaveSecurityGroup': 'value',\n        'ServiceAccessSecurityGroup': 'value',\n         },\n    BootstrapActions=[{'Name': 'Install Impala2','ScriptBootstrapAction': {'Path': 's3://coeus/bigtop/impala/impala-install'}}],\n    Applications=[{'Name':'Hadoop','Name':'Spark','Name':'Ganglia','Name':'Hive','Name':'Presto-Sandbox'}],\n    JobFlowRole='EMR_EC2_DefaultRole',\n    ServiceRole='EMR_DefaultRole',\n    VisibleToAllUsers=True|False,\n    Tags=[{\"Key\":\"owner\",\"Value\":\"myname\"}],\n    Configurations=[{\"Classification\":\"hadoop-env\",\"Properties\":{},\"Configurations\":[{\"Classification\":\"export\",\"Properties\":{\"JAVA_HOME\":\"/usr/lib/jvm/java-1.8.0\"},\"Configurations\":[]}]},{\"Classification\":\"spark-env\",\"Properties\":{},\"Configurations\":[{\"Classification\":\"export\",\"Properties\":{\"JAVA_HOME\":\"/usr/lib/jvm/java-1.8.0\"},\"Configurations\":[]}]}]\n\nThis code successfully creates the cluster  but when i try to run the MapR jobs like distcp on the cluster it throws this error\n\"Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster\"\n\nI created the cluster using the console and passing same parameters  the cluster gets created and I am able to run the MapR commands (Distcp) without having any issues. I am not sure why does EMR cluster created with Boto3 has the issues with hadoop config.\n\nHere is the cli export of the cluster i created using the  console.\n\naws emr create-cluster --applications Name=Hadoop Name=Spark Name=Ganglia Name=Presto-Sandbox Name=Hive --bootstrap-actions '[{\"Path\":\"s3://coeus/bigtop/impala/impala-install\",\"Name\":\"Custom action\"}]' --tags 'owner=myname' --ec2-attributes '{\"KeyName\":\"mykey\",\"InstanceProfile\":\"EMR_EC2_DefaultRole\",\"ServiceAccessSecurityGroup\":\"\",\"SubnetId\":\"\",\"EmrManagedSlaveSecurityGroup\":\"\",\"EmrManagedMasterSecurityGroup\":\"\"}' --service-role EMR_DefaultRole --release-label emr-4.6.0 --log-uri ' ' --name 'automate' --instance-groups '[{\"InstanceCount\":1,\"InstanceGroupType\":\"MASTER\",\"InstanceType\":\"r3.8xlarge\",\"Name\":\"master\"},{\"InstanceCount\":4,\"InstanceGroupType\":\"CORE\",\"InstanceType\":\"r3.8xlarge\",\"Name\":\"slave\"}]' --configurations '[{\"Classification\":\"hadoop-env\",\"Properties\":{},\"Configurations\":[{\"Classification\":\"export\",\"Properties\":{\"JAVA_HOME\":\"/usr/lib/jvm/java-1.8.0\"},\"Configurations\":[]}]},{\"Classification\":\"spark-env\",\"Properties\":{},\"Configurations\":[{\"Classification\":\"export\",\"Properties\":{\"JAVA_HOME\":\"/usr/lib/jvm/java-1.8.0\"},\"Configurations\":[]}]}]' --region\n\nI am out of ideas why it should be happening. any help is highly appreciated.\n","author":{"url":"https://github.com/rahul22022","@type":"Person","name":"rahul22022"},"datePublished":"2016-09-19T23:52:04.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/8/aws-python-sample/issues/8"}