AWS re:Invent 2016: The Effective AWS CLI User (DEV402)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Kyle Knapp, Amazon Web Services

December 1, 2016

DEV402

The Effective AWS CLI User

Important:

This talk will be advanced

Background Material

• AWS CLI User Guide

• AWS CLI Command Reference

• 2015 AWS CLI re:Invent talk



http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html

http://docs.aws.amazon.com/cli/latest/reference/

https://www.youtube.com/watch?v=TnfqJYPjD9I

https://www.youtube.com/watch?v=vP56l7qThNs

https://www.youtube.com/watch?v=qiPt1NoyZm0

Background Material

• AWS CLI User Guide

• AWS CLI Command Reference




Slides will be online!

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html

http://docs.aws.amazon.com/cli/latest/reference/

https://www.youtube.com/watch?v=TnfqJYPjD9I

https://www.youtube.com/watch?v=vP56l7qThNs

https://www.youtube.com/watch?v=qiPt1NoyZm0

Unified tool to manage AWS services

AWS Command Line Interface

The Effective AWS CLI User Tenets

The effective AWS CLI user:



Uses an iterative workflow




Troubleshoots well




Troubleshoots well

Is resourceful with tooling




Troubleshoots well


Understands performance implications

A Sample Application

Amazon VPC

Amazon VPC Amazon EC2

Amazon VPC Amazon EC2 IAM

Amazon VPC Amazon EC2 IAM Amazon S3

AWS Cloud

VPC

10.0.0.0/16

AWS Cloud

router

Destination Target

10.0.0.0/16 local

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

AWS Cloud

router

Destination Target

10.0.0.0/16 local

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

AWS Cloud

Internet

gateway

router

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

security group

AWS Cloud

Internet

gateway

router

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

security group

AWS Cloud

instance

Internet

gateway

router

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

role

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

security group

AWS Cloud

instance

Internet

gateway

router

Amazon

S3

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

role




$ aws ec2 create-vpc --cidr-block 10.0.0.0/16


{

"Vpc": {

"VpcId": "vpc-24de2d4d",

"InstanceTenancy": "default",

"State": "pending",

"DhcpOptionsId": "dopt-f8ca2291",

"CidrBlock": "10.0.0.0/16",

"IsDefault": false

}

}


{

"Vpc": {



"State": "pending",


"CidrBlock": "10.0.0.0/16",

"IsDefault": false

}

}


{

"Vpc": {



"State": "pending",


"CidrBlock": "10.0.0.0/16",

"IsDefault": false

}

}

$ vpc_id=vpc-24de2d4d


{

"Vpc": {



"State": "pending",


"CidrBlock": "10.0.0.0/16",

"IsDefault": false

}

}

$ vpc_id=vpc-24de2d4dWe can do better…

$ vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 \

--query Vpc.VpcId --output text)



{"Vpc": {

"VpcId": "vpc-24de2d4d", "InstanceTenancy": "default", "State": "pending", "DhcpOptionsId": "dopt-f8ca2291", "CidrBlock": "10.0.0.0/16", "IsDefault": false

}}



{"Vpc": {


}}



{"Vpc": {


}}



{"Vpc": {


}}



{"Vpc": {


}}



$ echo "$vpc_id”

vpc-24de2d4d



$ echo "$vpc_id"

vpc-24de2d4d

$ create_vpc_output=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16)



$ echo "$vpc_id"

vpc-24de2d4d


$ vpc_id=$(jp -u Vpc.VpcId <<< "$create_vpc_output“)



$ echo "$vpc_id"

vpc-24de2d4d


$ vpc_id=$(jp -u Vpc.VpcId <<< "$create_vpc_output")

$ echo "$vpc_id"

vpc-24de2d4d



$ echo "$vpc_id"

vpc-24de2d4d


$ vpc_id=$(jp -u Vpc.VpcId <<< "$create_vpc_output")

$ echo "$vpc_id"

vpc-24de2d4d

And we can make this even better!

Problem Statement

Demo

$

Let’s talk about how

$ aws ec2 create-vpc --cidr-block 10.0.0.0/16 --generate-cli-skeleton output

{

"Vpc": {

"VpcId": "VpcId",

"State": "State",

"CidrBlock": "CidrBlock",

"DhcpOptionsId": "DhcpOptionsId",

"Tags": [

{

"Key": "Key",

"Value": "Value"

}

],

"InstanceTenancy": "InstanceTenancy",

"IsDefault": true

}

}


{

"Vpc": {

"VpcId": "VpcId",

"State": "State",



"Tags": [

{

"Key": "Key",

"Value": "Value"

}

],


"IsDefault": true

}

}

"Vpc":{"type":"structure","members":{

"VpcId":{"shape":"String"

},"State":{"shape":"VpcState"

},"CidrBlock":{"shape":"String"

},"DhcpOptionsId":{"shape":"String"

},"Tags":{"shape":"TagList"

},"InstanceTenancy":{"shape":"Tenancy"

},"IsDefault":{"shape":"Boolean"

}}

} botocore/botocore/data/ec2/2016-09-15/service-2.json


{

"Vpc": {

"VpcId": "VpcId",

"State": "State",



"Tags": [

{

"Key": "Key",

"Value": "Value"

}

],


"IsDefault": true

}

}









}}



{

"Vpc": {

"VpcId": "VpcId",

"State": "State",



"Tags": [

{

"Key": "Key",

"Value": "Value"

}

],


"IsDefault": true

}

}









}}



{

"Vpc": {

"VpcId": "VpcId",

"State": "State",



"Tags": [

{

"Key": "Key",

"Value": "Value"

}

],


"IsDefault": true

}

}









}}


AWS Cloud

VPC

10.0.0.0/16

Destination Target

10.0.0.0/16 local

router

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

AWS Cloud

Internet

gateway

router

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

Demo

$

Important Takeaways

• --generate-cli-skeleton output

• Leverage UNIX commands (e.g.

echo, history)



Troubleshoots well

$ aws ec2 create-security-group --group-name re:Invent \

--vpc-id VpcId


--vpc-id VpcId

aws: error: argument --description is required


--vpc-id VpcId



--description 're:Invent demo' --vpc-id VpcId


--vpc-id VpcId



--description 're:Invent demo' --vpc-id VpcId

An error occurred (InvalidVpcID.NotFound) when calling the CreateSecurityGroup operation: The vpc ID 'VpcId' does not exist

--debug


--description 're:Invent demo' --vpc-id "$vpc_id” --debug


--description 're:Invent demo' --vpc-id "$vpc_id” --debug


--description 're:Invent demo' --vpc-id "$vpc_id" -–debug

2016-11-04 10:30:33,532 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.11.10 Python/2.7.10 Darwin/15.6.0 botocore/1.4.65

2016-11-04 10:30:33,533 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security-group', '--group-name', 're:Invent', '--description', 're:Invent demo', '--vpc-id', 'vpc-43cb382a', '--debug']

2016-11-04 10:30:33,533 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x10eefc488>

2016-11-04 10:30:33,533 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x10ecde578>

2016-11-04 10:30:33,533 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.

2016-11-04 10:30:33,537 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/ec2/2016-09-15/service-2.json

2016-11-04 10:30:33,622 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.ec2: calling handler <function register_retries_for_service at 0x10e904500>

2016-11-04 10:30:33,623 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: ec2

2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler <functools.partial object at 0x10ef02940>

2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG -Removing operation: import-instance

2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG -Removing operation: import-volume

2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler <function add_waiters at 0x10ef03578>

2016-11-04 10:30:33,631 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/ec2/2016-09-15/waiters-2.json

2016-11-04 10:30:33,635 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'dry-run', <awscli.arguments.BooleanArgument object at 0x10febe950>), (u'no-dry-run', <awscli.arguments.BooleanArgument object at 0x10febe990>), (u'group-name', <awscli.arguments.CLIArgument object at 0x10febe9d0>), (u'description', <awscli.arguments.CLIArgument object at 0x10febea10>), (u'vpc-id', <awscli.arguments.CLIArgument object at 0x10febea50>)])

2016-11-04 10:30:33,623 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: ec2

2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler <functools.partial object at 0x10ef02940>

2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG -Removing operation: import-instance

2016-11-04 10:30:33,628 - MainThread - awscli.customizations.removals - DEBUG -Removing operation: import-volume

2016-11-04 10:30:33,628 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler <function add_waiters at 0x10ef03578>

2016-11-04 10:30:33,631 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/ec2/2016-09-15/waiters-2.json

2016-11-04 10:30:33,635 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'dry-run', <awscli.arguments.BooleanArgument object at 0x10febe950>), (u'no-dry-run', <awscli.arguments.BooleanArgument object at 0x10febe990>), (u'group-name', <awscli.arguments.CLIArgument object at 0x10febe9d0>), (u'description', <awscli.arguments.CLIArgument object at 0x10febea10>), (u'vpc-id', <awscli.arguments.CLIArgument object at 0x10febea50>)])

And this continues on for a while…

Parse command$ aws ec2 create-security-group \

--group-name re:Invent \--description 're:Invent demo' \--vpc-id vpc-24de2d4d \--query GroupId --output text

Parse command

botocore client call

# Python coderesponse = ec2_client.create_security_group(

GroupName=‘re:Invent’,Description=‘re:Invent demo’,VpcId=‘vpc-24de2d4d’

)

$ aws ec2 create-security-group \--group-name re:Invent \--description 're:Invent demo' \--vpc-id vpc-24de2d4d \--query GroupId --output text

Parse command





)

Parse command





)

Parse command

Format response


sg-1d032e64




)

Parse command

Format response





)

sg-1d032e64

Parse command

Format response





)

sg-1d032e64

Parse command

Format response




)

Parse command

Format response


Validate/Serialize parameters

"CreateSecurityGroupRequest”{"type":"structure”,"required":["GroupName", "Description”],"members":{

"DryRun":{"shape":"Boolean”

},"GroupName":{

"shape":"String”},"Description":{

"shape":"String”},"VpcId":{

"shape":"String”}

}}

botocore/botocore/data/ec2/2016-09-15/service-2.json

Parse command

Format response

Make botocore client call


HTTP request/response

Parse command

Format response

Make botocore client call



Parse response

"CreateSecurityGroupResult”:{ "type":"structure”,"members":{

"GroupId":{"shape":"String”

}}

}botocore/botocore/data/ec2/2016-09-15/service-2.json

Parse command

Format response



Parse response

--debug

Parse command

Format response



Parse response

$ aws ec2 create-security-group \--group-name re:Invent \--description 're:Invent demo’ \--vpc-id "$vpc_id" -–debug

Parse command

Format response



Parse response

awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security-group', '--group-name', 're:Invent’,'--description', 're:Invent demo’,'--vpc-id', 'vpc-43cb382a', '--debug']

$ aws ec2 create-security-group \--group-name re:Invent \--description 're:Invent demo' \--vpc-id "$vpc_id" -–debug

Parse command

Format response



Parse response



Parse command

Format response



Parse response



Parse command

Format response



Parse response

$ aws ec2 create-security-group \--group-name re:Invent \--description file://description.txt \--vpc-id "$vpc_id" -–debug

Parse command

Format response



Parse response


Parse command

Format response



Parse response


awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'create-security-group', '--group-name', 're:Invent’,'--description', 'file://description.txt’,'--vpc-id', 'vpc-43cb382a', '--debug']

Parse command

Format response



Parse response



Parse command

Format response



Parse responseawscli.arguments - DEBUG - Unpacked value of u're:Invent demo' for parameter "description": u're:Invent demo'



Parse command

Format response



Parse responseawscli.arguments - DEBUG - Unpacked value of u're:Invent demo' for parameter "description": u're:Invent demo'



Parse command

Format response



Parse response


Parse command

Format response



Parse response


botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateSecurityGroup) (verify_ssl=True) with params: {

'body': {'Action': 'CreateSecurityGroup','GroupName': 're:Invent’,'Version': '2016-09-15’,'VpcId': 'vpc-43cb382a', ‘GroupDescription': 're:Invent demo’},

‘url':'https://ec2.us-east-2.amazonaws.com/’,...}

Parse command

Format response



Parse response





Parse command

Format response



Parse response





Parse command

Format response



Parse response





Parse command

Format response



Parse response


botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST / HTTP/1.1" 200 None

Parse command

Format response



Parse response


botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST / HTTP/1.1" 200 None

Parse command

Format response



Parse response


botocore.parsers - DEBUG - Response body:<?xml version="1.0" encoding="UTF-8"?><CreateSecurityGroupResponse>

<requestId>request-id</requestId><return>true</return><groupId>sg-5ec80f37</groupId>

</CreateSecurityGroupResponse>

Parse command

Format response



Parse response





Parse command

Format response



Parse response





Parse command

Format response



Parse response


botocore.retryhandler - DEBUG - No retry needed.




Parse command

Format response



Parse response


botocore.retryhandler - DEBUG - No retry needed.




VPC

10.0.0.0/16

Subnet

10.0.0.0/24

AWS Cloud

Internet

gateway

router

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

security group

AWS Cloud

Internet

gateway

router

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

Demo

$

Important Takeaways

• --debug

• Search for Traceback’s

• Leverage knowledge of AWS CLI

architecture




• BASH: variables, pipes, functions


• UNIX commands: xargs, sort, tail



• External tools: jp, jpterm, aws-shell



• External tools: jp, jpterm, aws-shell

• DIY (Do it yourself) tooling

alias

[alias]st = statusci = commitbr = branchco = checkout

~/.gitconfig


~/.gitconfig

$ git co some-branch


~/.gitconfig

$ git co some-branch


~/.gitconfig

$ git co some-branch $ git checkout some-branch

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

security group

AWS Cloud

Internet

gateway

router

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

security group

AWS Cloud

instance

Internet

gateway

router

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

role

Demo

$

Important Takeaways

• alias

• Leverage BASH and external

tooling




Performance Implications

Client-side vs. server-side filtering

Pagination

aws s3 cp


Client-side vs. server-side filtering

$ aws ec2 describe-images \

--query 'Images[?starts_with(to_string(Name),`amzn-ami-hvm-`)]'







$ aws ec2 describe-images \--filters Name=name,Values='amzn-ami-hvm-*' --query 'Images'








Pagination

$ aws ec2 describe-snapshots


{"Snapshots”: [{"SnapshotId": "snap-0",...

},...

]} }



},...

]} }


},...

]} }



},...

]} }


},...

]} }


},...

]} }



},...

]} }


},...

]} }


},...

]} }


},...{"SnapshotId": "snap-1000",...

},...{"SnapshotId": "snap-2000",...

},...

]} }

$ aws ec2 describe-snapshots --output text --query 'Snapshots[].SnapshotId'


},...

]} }


},...

]} }


},...

]} }



},...

]} }


},...

]} }


},...

]} }


},...

]} }


},...

]} }


},...

]} }


snap-0...

snap-1000...

snap-2000...}


aws s3 cp

upload

Thread pool

Thread

1

Thread

2

Thread

3

upload

Thread pool

6 5 4 3 2 1

Thread

1

Thread

2

Thread

3

Parts

upload

Thread pool

6 5 4

3

2

1Thread

1

Thread

2

Thread

3

Parts

upload

Thread pool

6 5

3

4

1Thread

1

Thread

2

Thread

3

Parts

upload

Thread pool

6

5

4

1Thread

1

Thread

2

Thread

3

Parts

upload

Thread pool

5

4

6Thread

1

Thread

2

Thread

3

upload

Thread pool

Thread

1

Thread

2

Thread

3

[default]region = us-east-1

~/.aws/config

[default]region = us-east-1s3 =

max_concurrent_requests = 20multipart_chunksize = 16MBmultipart_threshold = 64MBmax_queue_size = 10000

~/.aws/config

[default]region = us-east-1s3 =

max_concurrent_requests = 20multipart_chunksize = 16MBmultipart_threshold = 64MBmax_queue_size = 10000

~/.aws/config

max_concurrent_requests

Thread pool

6 5 4 3 2 1

Thread

1

Thread

2

Thread

3

Parts

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

security group

AWS Cloud

instance

Internet

gateway

router

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

role

VPC

10.0.0.0/16

Subnet

10.0.0.0/24

security group

AWS cloud

instance

Internet

gateway

router

Amazon

S3

Destination Target

10.0.0.0/16 local

0.0.0.0/0 igw-id

role

Demo

$

Important Takeaways

• Use server-side filtering when

possible

• Pagination with --output text

• s3 configuration options






Example: --generate-cli-skeleton output



Troubleshoots well

Example: --debug




Example: alias




Example: s3 configurations

Remember to complete

your evaluations!

Thank you!

Technology

AWS re:Invent 2016: The Effective AWS CLI User (DEV402)