View
24
Download
0
Category
Preview:
Citation preview
AmazonConfidential–SharedUnderNDA 1
AWSBatchPrivateBetaGettingStartedGuideNovember15,2016
AmazonConfidential–SharedunderNDA
ThankyouforparticipatingintheAWSBatchPrivateBetaProgram.Yourwillingnesstotestandprovidefeedbackonanunreleasedserviceisgreatlyappreciatedbytheteam.Afewcaveatsforyoutotakeintoconsiderationbeforeyoubeginusingthisservice.First,thisisaprivatebetaandtheserviceisnotyetreadyforproductionworkloads.Wewillmakereasonableattemptstoensurethattheservicefunctionsasexpectedduringthisperiod,butdonotbesurprisedifthingsbreakorbehavedifferentlythantheyshouldfromtimetotime.Second,youshouldassumethattheresourcescreatedaspartofthisbetawillnotpersistaftertheconclusionofthePrivateBeta.AnyJobQueues,ComputeEnvironments,JobDefinitions,andotherAWSBatchPrivateBetaresourcescanandlikelywillbedeletedattheendofthePrivateBetaperiod.Third,pleaserespecttheconfidentialityofthisPrivateBetaprogram.AllaspectsofthisservicearesharedwithyouunderthetermsofyouragreementswithAWS,includingthenon-disclosureagreement.
Questions&Issues:AsyouareworkingtestingAWSBatch,pleasedirectanyquestionsdilithium-beta-feedback@amazon.com.
TableofContents: Questions&Issues:................................................................................................................................................1
TableofContents:..................................................................................................................................................1
Introduction:..........................................................................................................................................................2
CurrentcapabilitiesandKnownIssues:..................................................................................................................3ComputeEnvironments(CEs):...........................................................................................................................................3JobQueues(JQs):...............................................................................................................................................................3JobDefinitionsandJobs:...................................................................................................................................................3AWSBatchWebConsole:..................................................................................................................................................3
Downloadfilesusedinthisexample:.....................................................................................................................3
AWSCommand-LineInterface(CLI)Configuration:...............................................................................................3
AccountSetupActivities:.......................................................................................................................................4CreateanECSInstanceRole:.............................................................................................................................................4CreateaServiceRole:........................................................................................................................................................7
Step1:CreateanIAMPolicy.........................................................................................................................................7Step2:CreateanIAMRole............................................................................................................................................8Step3:AttachtheIAMPolicytoyourIAMRole............................................................................................................9Step4:EnsurethatyourVPCsupportsexternalnetworkaccess..................................................................................9
TestingtheHappyPath:.........................................................................................................................................9CreateaComputeEnvironment:.....................................................................................................................................10DescribeYourComputeEnvironment:............................................................................................................................10
AmazonConfidential–SharedUnderNDA 2
CreateaJobQueue:.........................................................................................................................................................11DescribeYourJobQueue:................................................................................................................................................11RegisteraJobDefinition:.................................................................................................................................................12DescribeYourJobDefinition:...........................................................................................................................................12SubmitaJob:....................................................................................................................................................................13ListYourRunningJob:......................................................................................................................................................13DescribeYourRunningJob:.............................................................................................................................................14
CleanupYourHappyPathEnvironment:..............................................................................................................14DeregisterYourJobDefinition:........................................................................................................................................14UpdateYourJobQueue:..................................................................................................................................................14DeleteYourJobQueue:...................................................................................................................................................14UpdateYourComputeEnvironment:..............................................................................................................................15DeleteYourComputeEnvironment:................................................................................................................................15
Additionaltestsyoumaywanttoconduct:.........................................................................................................15AJobQueuewithMultipleComputeEnvironments:......................................................................................................15MultipleJobQueuesMappedtotheSameComputeEnvironment:...............................................................................15CreateanUnmanagedComputeEnvironment:...............................................................................................................15SubmitaCollectionofJobswithVaryingResourceRequirements:................................................................................16
Concepts:..............................................................................................................................................................17Jobs:.................................................................................................................................................................................17JobDefinitions:................................................................................................................................................................17JobQueues:.....................................................................................................................................................................17ComputeEnvironments:..................................................................................................................................................18
ManagedComputeEnvironments:..............................................................................................................................18UnmanagedComputeEnvironment:...........................................................................................................................18
Workflows,Pipelines,andJobDependencies:................................................................................................................18Scheduler:........................................................................................................................................................................18
Introduction:Batchcomputingisacommonwayfordevelopers,scientistsandengineerstoaccesslargeamountsofcomputeresources.AWSBatchremovestheundifferentiatedheavyliftingofconfiguringandmanagingtherequiredinfrastructure,makingiteasierthanevertorunbatchcomputingworkloadsontheAWScloud.Theservicealsomakesiteasyforadministrators,developers,andindependentsoftwarevendors(ISVs)toprovidevaluable,time-savingapplicationtemplatesfortheirusers.AWSBatchwillbefamiliartousersoftraditionalbatchcomputingsoftwareandcanefficientlyprovisionresourcesinresponsetojobssubmittedinordertoeliminatecapacityconstraints,reducecomputecosts,anddeliverresultsfasterthanever.Asafully-managedservice,AWSBatchautomaticallyprovisionscomputeresourcesandoptimizestheworkloaddistributionbasedonthequantityandscaleofthejobssubmitted.WithAWSBatch,thereisnoneedtoinstallormanagebatchcomputingsoftware,allowingyoutofocusonanalyzingresultsandsolvingproblems.AWSBatchplans,schedules,andexecutesyourbatchcomputingworkloadsacrossthefullrangeofAWScomputeservices,suchasAmazonEC2,AmazonEC2SpotwithLambdasupportarrivingshortlyafterlaunch.AWSBatchreducesoperationalcomplexities,savingtimeandreducingcosts,makingiteasyfordevelopers,scientists,andengineerstoruntheirbatchjobsintheAWScloud.
AmazonConfidential–SharedUnderNDA 3
CurrentcapabilitiesandKnownIssues:ComputeEnvironments(CEs):ManagedComputeEnvironmentsdonotyetscaleupordown.ThenumberofinstanceslaunchedisdeterminedbythedesiredvCpusspecifiedfortheCE. JobQueues(JQs):JobQueueoperationsfunctionasexpected. JobDefinitionsandJobs:WedonotyetsupportLambdaJobsorArrayJobs(“copies”>1).JobDefinitionscanonlybeusedtospecifyacontainerizedjob.Jobdependenciesarenotyetsupported.AWSBatchWebConsole:ThePrivateBetadoesnotyetincludeaccesstotheAWSBatchWebConsole.AlloperationsmustbesubmittedviatheAWSCLI.
Downloadfilesusedinthisexample:Youcandownloadthesamplecodeforthistutorialfrom:https://s3.amazonaws.com/batch-preview/docs/AWS-Batch-Private-Beta-GSG-commands.txt
AWSCommand-LineInterface(CLI)Configuration:TheAWSCLIisabotocorebasedpackagethatsupportsdynamicadditionofadditionalservices.Ifyouhaven’talreadyinstalledtheAWSCLI,pleasefollowtheinstructionspublishedat:https://aws.amazon.com/cli/InordertorunAWSBatchcommandsviaAWSCLI,youwillneedtoinstalltheC2Jservicemodel.Pleasedownloadthefollowingfiletoyourcomputer:https://s3.amazonaws.com/batch-preview/model/aws-batch-private-model.jsonToaddAWSBatchtoyourAWSCLI,usethefollowingcommand:aws configure add-model --service-name batch --service-model file://aws-batch-private-model.json
Iftherearenoerrors,therewillbenooutput.Youcantestthatthemodelhasbeenaddedwiththefollowingcommand:aws batch help
Theoutputshouldlooksimilartothefollowing:BATCH() BATCH() NAME
AmazonConfidential–SharedUnderNDA 4
batch - DESCRIPTION AWS Batch makes it easier than ever to run batch computing workloads on the AWS cloud. Batch computing is a common way for developers, scien- tists and engineers to access large amounts of compute resources, and AWS Batch removes the undifferentiated heavy lifting of configuring and managing the required infrastructure. It also makes it easy for admin- istrators, developers and independent software vendors to provide valu- able, time-saving application templates for their users. AWS Batch will be familiar to users of traditional batch computing software and can efficiently provision resources in response to jobs submitted in order to eliminate capacity constraints, reduce compute costs, and deliver results faster than ever. As a fully-managed service, AWS Batch enables developers, scientists, and engineers to easily and efficiently run batch computing workloads of any scale. AWS Batch automatically provisions compute resources and optimizes the workload distribution based on the quantity and scale of the workloads. With AWS Batch, there is no need to install or manage batch computing software, allowing you to focus on analyzing results and solving problems. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute ser- vices, such as Amazon EC2, Amazon EC2 Spot, and AWS Lambda. AWS Batch reduces operational complexities, saving time and reducing costs, mak- ing it easy for developers, scientists, and engineers to run their batch jobs in the AWS cloud. AVAILABLE COMMANDS o cancel-job o create-compute-environment o create-job-queue o create-resource
YoucanconfiguresecuritycredentialsandothersettingsfortheAWSCLIbyfollowingtheseinstructions:http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
AccountSetupActivities:BeforeyoucanworkwithAWSBatchduringtheprivatebeta,youwillneedtocreatesomeAWSIdentityandAccessManagement(IAM)roles.Specifically,youwillneedanECSInstanceRolethatthatenablesinstanceswithinyourComputeEnvironmentstointeractwithotherAWSServices.YouwillalsoneedaServiceRolethatdefineswhichservicesAWSBatchmayinvokeonyourbehalfinordertooperatetheserviceandcreate,modify,ordeleteresourcessuchasECSClusters,LaunchConfigurations,AutoScalingGroups,andCloudWatchLogGroups. CreateanECSInstanceRole:Inordertocreateyourinstancerole,logintotheAWSWebConsoleforIAMandnavigatetotheRolespage:https://console.aws.amazon.com/iam/home#roles
AmazonConfidential–SharedUnderNDA 5
Yourbrowserwindowshouldlooksomethinglikethis:
Clickonthe“CreateNewRole”button,enterthename“AWSBatchECSInstanceRole”andclick“NextStep”
AmazonConfidential–SharedUnderNDA 6
ScrolldownwithintheAWSServiceRolecategoryandselecttherolecalled“AmazonEC2RoleforEC2ContainerService”andclickthe“Select”button.
Choosethemanagedpolicythatispresentedinthelistandclick“NextStep”
AmazonConfidential–SharedUnderNDA 7
Thepageshouldnowlooksomethinglikethis.Reviewtheroleandclick“CreateRole”
Findyourroleontheresultingpageandconfirmthatitwassuccessfullycreated. CreateaServiceRole:NowyouneedtocreateaServiceRolethatwillgiveAWSBatchpermissiontocreateandmanageAWSresourcesonyourbehalfinordertooperatetheservice.Atlaunch,AWSBatchwillcreatethisServiceRoleforyouautomatically.DuringthePrivateBetayouwillneedtocreateitmanuallyusingathreestepprocess:
1. CreateanIAMPolicy2. CreateanIAMRole3. AttachtheIAMPolicytotheIAMRole
Note:Inordertocompletethisstep,youneedtohavetheAWSCLIproperlyinstalledandconfiguredtouseAWScredentialswithiam:*permissions.Step1:CreateanIAMPolicyFirstcreateafilecalledAWSBatchBetaServicePolicy.jsonwiththefollowingcontents:{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "autoscaling:*", "ecs:*", "ec2:*", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "iam:PassRole" ],
AmazonConfidential–SharedUnderNDA 8
"Resource": "*" } ] }
NextcreateanIAMPolicycalledAWSBatchBetaServicePolicywiththefollowingAWSCLIcommand: aws iam create-policy --policy-name AWSBatchBetaServicePolicy --policy-document file://AWSBatchBetaServicePolicy.json Theoutputshouldlooksimilartothefollowing:{ "Policy": { "PolicyName": "AWSBatchBetaServicePolicy", "CreateDate": "2016-11-06T20:17:26.627Z", "AttachmentCount": 0, "IsAttachable": true, "PolicyId": "AMPSIQB76QBFYGEZ5I2PM", "DefaultVersionId": "v1", "Path": "/", "Arn": "arn:aws:iam::012345678912:policy/AWSBatchBetaServicePolicy", "UpdateDate": "2016-11-06T20:17:26.627Z" } }
Step2:CreateanIAMRoleNowthatanIAMPolicyhasbeencreated,thenextstepistocreateanIAMRole.ThisrequiresapolicydocumentthatisspecifictotheAWSBatchPrivateBetaenvironment.FirstcreateafilecalledAWSBatchBetaServiceRole.jsonwiththefollowingcontents:{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::398747504704:root", "arn:aws:iam::398264837155:root" ] }, "Action": "sts:AssumeRole" } ] }
Next,createanIAMRolecalledAWSBatchBetaServiceRolewiththefollowingAWSCLIcommand: aws iam create-role --role-name AWSBatchBetaServiceRole --assume-role-policy-document file://AWSBatchBetaServiceRole.json
Theoutputshouldlooksimilartothefollowing:{ "Role": { "AssumeRolePolicyDocument": { "Version": "2012-10-17",
AmazonConfidential–SharedUnderNDA 9
"Statement": [ { "Action": "sts:AssumeRole", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::398747504704:root", "arn:aws:iam::398264837155:root" ] } } ] }, "RoleId": "AROAIZAGRNO45LX666252", "CreateDate": "2016-11-06T20:44:18.566Z", "RoleName": "AWSBatchBetaServiceRole", "Path": "/", "Arn": "arn:aws:iam::012345678912:role/AWSBatchBetaServiceRole" } }
PleasenotetheARNforyourAWSBatchBetaServiceRole,asyouwillneeditlater.TheARNlistedaboveissimplyanexample.Step3:AttachtheIAMPolicytoyourIAMRoleToattachtheAWSBatchBetaServicePolicyIAMPolicytotheAWSBatchBetaServiceRoleIAMRole,usethefollowingAWSCLIcommand:aws iam attach-role-policy --role-name AWSBatchBetaServiceRole --policy-arn arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSBatchBetaServicePolicy
Iftherearenoerrors,therewillbenooutput.Step4:EnsurethatyourVPCsupportsexternalnetworkaccessInstancesinyourcomputeenvironments(bothmanagedandunmanaged)requireexternalnetworkaccessinorderfortheECSAgenttocommunicatewiththeAmazonECSserviceendpoint.ifyourcontainerinstancesdonothavepublicIPaddresses,thentheymustusenetworkaddresstranslation(NAT)oranHTTPproxytoprovidethisaccess.Formoreinformation,seeNATInstancesintheAmazonVPCUserGuideandHTTPProxyConfigurationinthisguide.
TestingtheHappyPath:Congratulations!YouarenowreadytobeginusingAWSBatch.BycompletingtheAWSBatchHappyPath,youwillbecomefamiliarwiththeprimarycapabilitiesoftheservicebycreatingandtestingasimple,yetfully-capableAWSBatchconfiguration.Uponcompletionofthisexercise,youwillhaveperformedthefollowing:
1. CreateanddescribeaComputeEnvironment2. CreateanddescribeaJobQueue3. RegisteraJobDefinition4. SubmityourfirstAWSBatchjob
AmazonConfidential–SharedUnderNDA 10
CreateaComputeEnvironment:AComputeEnvironmentistheresourcewithinAWSBatchthatdefinestheComputeResourcesthatwillbeavailableforcontainerjobs.Forthehappypath,theComputeEnvironmentwillbeMANAGED,usingEC2on-demandinstances.IMPORTANT!YouwillneedtousethefollowingattributesfromyourownAWSaccountinordertosuccessfullycallCreateComputeEnvironment:
§ AmazonVPCSubnetId(s)§ AmazonEC2SecurityGroupId(s)§ AmazonEC2KeypairName§ AmazonECSContainerInstanceIAMRoleName§ AWSBatchServiceRoleARN
Usingyourownvaluesfortheseattributes,thefollowingAWSCLIcommand,shouldcreateaVALIDComputeEnvironment:aws batch create-compute-environment --compute-environment-name demo1 --type MANAGED --state ENABLED --compute-resources type=EC2,minvCpus=0,maxvCpus=1024,desiredvCpus=16,instanceTypes=optimal,subnets=subnet-367ab30a,subnet-db3c9ff6,subnet-ff3691a4,subnet-3173e178,securityGroupIds=sg-8ec710f3,ec2KeyPair=aws-batch-beta-keypair,instanceRole=AWSBatchECSInstanceRole --service-role arn:aws:iam::123456789012:role/AWSBatchBetaServiceRole
Theoutputshouldlooksomethinglikethis: { "computeEnvironmentName": "demo1", "computeEnvironmentArn": "arn:aws:batch:us-east-1:0123456789012:compute-environment/demo1" }DescribeYourComputeEnvironment:WithinAWSBatch,theDescribeComputeEnvironmentAPIcanbeusedtolistComputeEnvironmentsortodescribeaspecificComputeEnvironment.Inaddition,youcanpasseithertheComputeEnvironmentNameorARN,withinalist.Assumingyoufollowedthestepabove,theAWSCLIcommandtodescribethefreshlycreatedComputeEnvironmentis:aws batch describe-compute-environments --compute-environments demo1
Theoutputshouldlooksimilartothefollowing:{ "computeEnvironments": [ { "status": "VALID", "serviceRole": "arn:aws:iam::123456789012:role/AWSBatchBetaServiceRole", "computeEnvironmentArn": "arn:aws:batch:us-east-1:123456789012:compute-environment/demo1", "computeResources": { "subnets": [ "subnet-367ab30a", "subnet-db3c9ff6",
AmazonConfidential–SharedUnderNDA 11
"subnet-ff3691a4", "subnet-3173e178" ], "tags": {}, "desiredvCpus": 16, "minvCpus": 0, "instanceTypes": [ "optimal" ], "securityGroupIds": [ "sg-8ec710f3" ], "instanceRole": "AWSBatchECSInstanceRole", "maxvCpus": 1024, "type": "EC2", "ec2KeyPair": "aws-batch-beta-keypair" }, "ecsClusterArn": "arn:aws:ecs:us-east-1:123456789012:cluster/demo1_Batch_c1f45197-ca4a-39a2-8ce9-a366bc38feb6", "state": "ENABLED", "computeEnvironmentName": "demo1", "type": "MANAGED", "resources": [] } ] }
Youcanalsotrywithoutspecifyingthe--compute-environmentsparameterandperformaliststylecall.OnceyourComputeEnvironmentisVALID,pleaseproceedtoJobQueuecreation.CreateaJobQueue:AWSBatchJobsaresubmittedtoaJobQueue.YoucanhavemultipleJobQueueswithinyourAWSaccount,eachlinkedtomultipleComputeEnvironments.WhencreatingaJobQueue,youassociateitwithaVALIDComputeEnvironment.Assumingyoufollowedthestepabove,theAWSCLIcommandtocreateaJobQueueis:aws batch create-job-queue --job-queue-name demo1 --state ENABLED --compute-environment-order order=0,computeEnvironment=demo1 --priority 500
Theoutputshouldlooksimilartothefollowing:{ "jobQueueArn": "arn:aws:batch:us-east-1:123456789012:job-queue/demo1", "jobQueueName": "demo1" }
DescribeYourJobQueue:WithinAWSBatch,theDescribeJobQueueAPIcanbeusedtolistJobQueuesordescribeaspecificJobQueue.Additionally,youcanpasseithertheJobQueueNameorARN,withinalist.Assumingyoufollowedthestepabove,theAWSCLIcommandtodescribethefreshlycreatedJobQueueis:aws batch describe-job-queues --job-queue demo1
Theoutputshouldlooksimilartothefollowing:
AmazonConfidential–SharedUnderNDA 12
{ "jobQueues": [ { "priority": 500, "state": "ENABLED", "jobQueueArn": "arn:aws:batch:us-east-1:123456789012:job-queue/demo1", "computeEnvironmentOrder": [ { "computeEnvironment": "arn:aws:batch:us-east-1:123456789012:compute-environment/demo1", "order": 0 } ], "jobQueueName": "demo1" } ] }
Youcanalsotrywithoutspecifyingthe--job-queuesparameterandperformaliststylecall.RegisteraJobDefinition:AnAWSBatchJobDefinitionenablesyoutoregisteralloftheattributesofaJobinadvance.TheJobDefinitioncanthenbeusestoeasilysubmitJobs.Thefollowingisasamplejob,thatwilluseabusyboxDockerimageandrunsleep120.TheAWSCLIcommandtoregistertheJobDefinitionis:aws batch register-job-definition --job-definition-name sleep --type container --container-properties '{ "image": "busybox", "vcpus": 1, "memory": 128, "command": [ "sleep", "120"]}'
Theoutputshouldlooksimilartothefollowing:{ "jobDefinitionArn": "arn:aws:batch:us-east-1:123456789012:job-definition/sleep", "jobDefinitionName": "sleep" }
DescribeYourJobDefinition:TheAWSBatchDescribeJobDefinitionsAPIcanbeusedtolistJobDefinitionsordescribeaspecificJobDefinition.YoucanoptionallypasseithertheJobDefinitionNameorARN,withinalist.Assumingyoufollowedthestepabove,theAWSCLIcommandtodescribeyourJobDefinitionis:aws batch describe-job-definitions --job-definitions sleep
Theoutputshouldlooksimilartothefollowing:{ "jobDefinitions": [ { "containerProperties": { "mountPoints": [], "parameters": {}, "image": "busybox",
AmazonConfidential–SharedUnderNDA 13
"environment": {}, "vcpus": 2, "command": [ "sleep", "120" ], "ulimits": [], "volumes": [], "memory": 2048, "resources": [] }, "jobDefinitionArn": "arn:aws:batch:us-east-1:123456789012:job-definition/sleep", "type": "container", "jobDefinitionName": "sleep" } ] }
SubmitaJob:YouarenowreadytosubmityourfirstAWSBatchjob.Todoso,runthefollowingcommand:aws batch submit-job --job-name MySleep --job-definition sleep --job-queue demo1
Theoutputshouldlooksimilartothefollowing:{ "jobName": "MySleep", "arraySize": 1, "jobId": "arn:aws:batch:us-east-1:123456789012:job/1eac6559-18a1-4fc6-b3aa-5d1b2e204923" }
ListYourRunningJob:Nowthatyouhavesubmittedyourfirstjob,youcanwatchit’sprogressasitisscheduledandexecuted.Todoso,runthefollowingcommand:aws batch list-jobs --job-queue demo1 --job-status RUNNING
Theoutputshouldlooksimilartothefollowing(assumingyouhavesubmittedmultiplejobs):{ "jobSummaryList": [ { "jobId": "7fb78800-1c37-4940-9ac3-983dbff76ea8" }, { "jobId": "7fb78800-1c37-4940-9ac3-983dbff76e00" }, { "jobId": "7fb78800-1c37-4940-9ac3-983dbff76e03" } ] }
Tryrunningthesamecommandusingdifferentstates(e.g.RUNNABLE,PENDING,andSUCCEEDED).
AmazonConfidential–SharedUnderNDA 14
DescribeYourRunningJob:Ifyouwouldliketoexamineoneofyourjobsinmoredetail,youcanusetheAWSBatchDescribeJobsAPI:aws batch describe-jobs --jobs 1eac6559-18a1-4fc6-b3aa-5d1b2e204923
Theoutputshouldlooksimilartothefollowing:{ "jobs": [ { "container": { "mountPoints": [], "image": "busybox", "environment": [], "vcpus": 2, "command": [ "sleep", "120" ], "volumes": [], "memory": 128, "ulimits": [] }, "parameters": [], "jobDefinition": "sleep", "jobQueue": "arn:aws:batch:us-east-1:123456789012:job-queue/demo1", "jobId": "1eac6559-18a1-4fc6-b3aa-5d1b2e204923", "dependsOn": [], "jobName": "MySleep" } ] }
CleanupYourHappyPathEnvironment:NowthatyouhavesuccessfullyrunyourfirstAWSBatchjob,let’scleanupeverythingthatwehavecreated.DeregisterYourJobDefinition:Usingthefollowingcommand,youcanderegistertheJobDefinitioncreatedearlier:aws batch deregister-job-definition --job-definition arn:aws:batch:us-east-1:560348900601:job-definition/sleep
UpdateYourJobQueue:BeforeyoucandeleteyourJobQueue,youfirstneedtochangeitfromENABLEDtoDISABLEDviathefollowingcommand:aws batch update-job-queue --job-queue demo1 --state DISABLED
DeleteYourJobQueue:NowthatyourJobQueueisDISABLED,youcandeleteit:aws batch delete-job-queue --job-queue demo1
AmazonConfidential–SharedUnderNDA 15
UpdateYourComputeEnvironment:AswithJobQueues,beforeyoucandeleteyourComputeEnvironment,youfirstneedtochangeitfromENABLEDtoDISABLEDviathefollowingcommand:aws batch update-compute-environment --compute-environment demo1 --state DISABLED
DeleteYourComputeEnvironment:NowthatyourComputeEnvironmentisDISABLED,youcandeleteit:aws batch delete-compute-environment --compute-environment demo1
AllAWSresourcescreatedinyouraccount,shouldnowbegone.
Additionaltestsyoumaywanttoconduct:AJobQueuewithMultipleComputeEnvironments:WhathappenswhenyoucreateaJobQueuewithmultipleComputeEnvironments?Wherearejobsexecuted?HowareresourceswithineachComputeEnvironmentutilized? MultipleJobQueuesMappedtotheSameComputeEnvironment:WhathappenswhenyoucreatemultipleJobQueuesmappedtothesameComputeEnvironment?WhichJobQueue’sjobsareexecutedfirst?CreateanUnmanagedComputeEnvironment:TrytocreateanUnmanagedcomputeenvironmentusingthefollowingcommand:aws batch create-compute-environment --type UNMANAGED --compute-environment-name UnmanagedDemo1 --service-role arn:aws:iam::123456789012:role/DilithiumBetaServiceRole
Theresponseshouldlooklikethis: { "computeEnvironmentName": "UnmanagedDemo1", "computeEnvironmentArn": "arn:aws:batch:us-east-1:123456789012:compute-environment/UnmanagedDemo1" }
NowdescribeyourUnmanagedComputeEnvironmentandretrievetheARNoftheECSclusterthatwascreatedforyou:aws batch describe-compute-environments --compute-environment UnmanagedDemo1
Theresponsewilllooklikethis: { "computeEnvironments": [ { "status": "VALID", "serviceRole": "arn:aws:iam::123456789012:role/DilithiumBetaServiceRole", "computeEnvironmentArn": "arn:aws:batch:us-east-1:123456789012:compute-environment/UnmanagedDemo1", "statusReason": "ComputeEnvironment Healthy",
AmazonConfidential–SharedUnderNDA 16
"ecsClusterArn": "arn:aws:ecs:us-east-1:123456789012:cluster/UnmanagedDemo1_Batch_c3b9955f-9f66-3c0a-9d0f-55ceb9e5edad", "state": "ENABLED", "computeEnvironmentName": "UnmanagedDemo1", "type": "UNMANAGED", "resources": [] } ] }
Inthisexample,yourECSclusterARNis: arn:aws:ecs:us-east-1:123456789012:cluster/UnmanagedDemo1_Batch_c3b9955f-9f66-3c0a-9d0f-55ceb9e5edad
andtheECSclusternameis:UnmanagedDemo1_Batch_c3b9955f-9f66-3c0a-9d0f-55ceb9e5edad
NowfollowtheseinstructionstolaunchinstancesoftheECS-optimizedAMIintoyourUnmanagedCE’sECSCluster.http://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_container_instance.htmlSubmitaCollectionofJobswithVaryingResourceRequirements:HowwelldoestheAWSBatchSchedulerwork?ArejobsscheduledinamannerthatefficientlyutilizesresourceswithinyourComputeEnvironments?ThefollowingPythonscriptcanbeusedtosubmit50jobstoaJobQueuecalleddemo1,eachjobrunningthesleepcommandforsomewherebetween60and360secondswithvaryingvCPUandmemoryrequirements.#!/usr/bin/python import botocore.session import botocore.exceptions import random region = 'us-east-1' endpoint_url = 'https://dilithium-beta.us-east-1.amazonaws.com' session = botocore.session.get_session() client = session.create_client('batch', region, endpoint_url=endpoint_url) y=50 for x in range(0, y): vcpus=random.randrange(1,32+1,2) memory=random.randrange(512,2048+512,512) sleep=str(random.randrange(60,360,60)) command=['sleep', sleep] request = client.submit_job(jobQueue='demo1', jobName='sleep', jobDefinition='sleep', arraySize=1, containerOverrides=dict(vcpus=vcpus,memory=memory*vcpus,command=command)) print(request['jobId'])
AmazonConfidential–SharedUnderNDA 17
Concepts: Jobs:JobsaretheunitofworkexecutedbyAWSBatch.JobscanbeexecutedasLambdafunctionsorcontainerizedapplicationsrunningonAmazonEC2.Containerizedjobscanreferenceacontainerimage,command,andparametersoruserscansimplyprovidea.ziporscript(bash,python,perl,etc…)thatrunswithinadefaultAmazonLinuxcontainer. Userscansubmitalargenumberofindependent“simplejobs”aswellas“arrayjobs”thatrunmanycopiesofanapplicationagainstanarrayofelements.Arrayjobsareanefficientwaytorun:
• Parametricsweeps• MonteCarlosimulations• Processingalargecollectionofobjects
Jobscanhavethefollowingstates:SUBMITTED:Ajobthathasbeensubmittedtothequeue,buthasnotyetbeenevaluatedbythescheduler.PENDING:Ajobthatresidesinthequeueandisnotyetabletorunduetoadependencyonanotherjoborresource.RUNNABLE:Ajobwhichresidesinthequeue,hasnooutstandingdependencies,andisthereforereadytobescheduledtoahost.JobsinthisstatearestartedassoonassufficientresourcesareavailableinoneoftheComputeEnvironment’smappedtotheJob’sQueue.STARTING:Thesejobshavebeenscheduledtoahostandtherelevantcontainerinitiationoperationsareunderway.RUNNING:ThejobisrunningeitherasaLambdafunctionorcontainerjobonacomputeresourcewithinaComputeEnvironment.SUCCEEDED:Thejobhassuccessfullycompleted.FAILED:Thejobhasfailed. JobDefinitions:SimilartoECSTaskDefinitions,JobDefinitionsspecifyhowjobsaretoberun.Whileeachjobmustreferenceajobdefinition,manyparameterscanbeoverridden.Someoftheattributesspecifiedinajobdefinitioninclude:
• IAMroleassociatedwiththejob• vCPUandmemoryrequirements• Mountpoints• ContainerorLambdaproperties• Environmentvariables
JobQueues:JobsaresubmittedtoaJobQueue,wheretheyresideuntiltheyareabletobescheduledtoacomputeresource.AnAWSaccountcanhavemultipleJobQueues.Forexample,youmightcreateaqueueforhighpriorityjobsandanotherqueueforlow-priorityjobs.JobshaveaprioritythatisusedbytheSchedulertodeterminewhichJobQueue’sJobsshouldbeevaluatedforexecutionfirst.
AmazonConfidential–SharedUnderNDA 18
ComputeEnvironments:JobqueuesaremappedtooneormoreComputeEnvironment.ComputeEnvironmentscontaintheEC2instancesusedtoruncontainerizedbatchjobs.AgivenComputeEnvironmentcanalsobemappedtooneormanyJobQueues.WithinaJobQueue,theassociatedComputeEnvironmentseachhaveanorderthatisusedbytheSchedulertodeterminewheretoplacejobsthatarereadytobeexecuted.IfthefirstComputeEnvironmenthasfreeresources,thejobwillbescheduledtoahostwithinthatComputeEnvironment.IftheComputeEnvironmentisunabletoprovideasuitablecomputeresource,theSchedulewillattempttorunthejobonthenextComputeEnvironment. ManagedComputeEnvironments:ManagedComputeEnvironmentsenableyoutodescribeyourbusinessrequirements(e.g.instancetypes,VPCSubnets,min/max/desiredaggregatevCPUsacrossallinstances,andspotbidasa%ofOn-Demand)andAWSBatchlaunchesandscalesresourcesonyourbehalf.UnmanagedComputeEnvironment:Alternatively,youcanlaunchandmanageyourownresourceswithinanUnmanagedComputeEnvironment.TheinstancethatyoulaunchneedtoincludetheECSagentandrunasupportedversionofLinuxandDocker.UnmanagedComputeEnvironmentsareusefulifyouneedtoconfigureyourcomputeresourcesinawaythatisdifferentfromthedefaultconfigurationprovidedbyManagedComputeEnvironments.SeethispagetolearnhowtoconfigureEC2instanceswithinanUnmanagedComputeEnvironment:http://docs.aws.amazon.com/AmazonECS/latest/developerguide/container_instance_AMIs.htmlWorkflows,Pipelines,andJobDependencies:Jobscanexpressadependencyonthesuccessfulcompletionofotherjobs,specificelementsofanarrayjob,orAWSBatchresources(countersorEFSfilesystems).AWSBatchdoesnothaveaSubmitWorkflowAPI.Instead,customersusetheirpreferredworkflowengineandlanguagetosubmitjobs.Flow-basedsystemssimplysubmitjobsinserialwhileDAG-basedsystemssubmitmanyjobsatonce,identifyinginter-jobdependenciesbyJobnameorID.Scheduler:TheSchedulerevaluateswhen,where,andhowtorunjobsthathavebeensubmittedtoajobqueue.OurinitialalgorithmisFIFOwithdependencyawareness,meaningthatwerunjobsinapproximatelytheorderinwhichtheyaresubmittedaslongasalldependenciesonotherjobsresourceshavebeenmet.
Recommended