Orchestrating Lambda with Step Functions

@technovangelist

@technovangelist

@technovangelist

@technovangelist

Who am I

Matt Williams

Evangelist at Datadog@technovangelistmattw@datadoghq.com

Organizer of DevOps Days Boston 2017 & 2018

@technovangelist

Who is Datadog

SaaS-based Monitoring & Analytics Infrastructure, APM, LogsOpen Source AgentTrillions of data

points per day

We are hiring!!

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

I l ike to talk about Serverless…

@technovangelist

What is AWS Lambda

ServerlessAs far as you are concernedDon’t pay for idleScale with ease

FaaSNode 8.10 (java, go, python, c#)

Trigger-based

@technovangelist

Trigger Based?

S3DynamoDBKinesisSNSSESCognitoCloudFormationCloudwatch Logs & Events

CodeCommitConfigAlexaLexAPI GatewayIoTCloudFrontLambda

@technovangelist

AWS Lambda

@technovangelist

@technovangelist

@technovangelist

That’s pretty cool right? But not perfect

You still have to configure everything elseS3DynamoDBAPI GWetc

SAMCloudformationServerless

@technovangelist

AWS Lambda

@technovangelist

…but serverless !== ec2

@technovangelist

…but serverless !== applications

@technovangelist

…applications are complicated

@technovangelist

…serverless functions should respect the single responsibility principle…

@technovangelist

@technovangelist

@technovangelist

we can create many serverless functions

@technovangelist

we can create many serverless functions

but how do we orchestrate them?

@technovangelist

Event/Message/Database as Traffic Cop

Amazon Kinesis Amazon SQS Amazon DynamoDB

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

https://github.com/Nordstrom/hello-retail

@technovangelist

Enter AWS Step Functions

What about Azure?

@technovangelist

a step function is

@technovangelist

a step function is

a state machine

@technovangelist

but what about SWF?(Amazon Simple Workflow Service)

@technovangelist

AWS Step Function Concepts

State MachineStates

TaskChoiceSucceed/FailPassWaitParallel

TransitionsExecutions

You define the State Machine with the Amazon States Language

A diagram will be generated based on that structure

@technovangelist

State Example: Task

@technovangelist

State Example: Choice

@technovangelist

State Example: Succeed / Fail

@technovangelist

State Example: Pass

@technovangelist

State Example: Wait

@technovangelist

State Example: Parallel

@technovangelist

Many ways to build Step Functions

I used Serverless Framework

@technovangelist

Benefits of Serverless Framework

Define everything in a single file:IAM Role StatementsLambda Function HandlersLambda Function TriggersStep Function StatesCloudFormation for other things

@technovangelist

Amazon ECRAWS

Lambda

AmazonS3

AmazonDynamoDB

Amazon CloudFront

AWS CodeBuild

AWS CodeCommit

AWSCloudFormation

IAM

Amazon API Gateway*

AWS Step Functions

AmazonSNS

@technovangelist

@technovangelist

1.Use ML to figure out which slide2.Is this slide different than the last time?3.If so, pull slide data from Dynamo4.If slide different then create document

in s35.Build the site6.Wait for it

1.Is it built7.Done

@technovangelist

Type: TaskResource: Lambda Function

@technovangelist

@technovangelist

Type: TaskResource: Lambda Function

@technovangelist

@technovangelist

Type: TaskResource: Lambda Function

@technovangelist

Type: TaskResource: Lambda Function

@technovangelist

How about monitoring?

We monitor Lambda via Logs

MONITORING|unix_epoch_timestamp|value|count|my.metric.name|#tag1:value,tag2

console.log(monitoringstring);

@technovangelist

What are the Lambda Metrics

aws.lambda.duration, min, max, sumaws.lambda.errorsaws.lambda.invocationsaws.lambda.throttlesaws.lambda.iterator_age

@technovangelist

What are the Step Function Metrics

aws.states.execution_time (.maximum, .minimum)aws.states.executions_aborted, failed, started, succeeded, timed_outaws.states.lambda_functions_scheduled, started, succeeded, timed_outaws.states.execution_throttledaws.states.lambda_function_time (.maximum, .minimum)aws.states.lambda_function_run_time (.maximum, .minimum)aws.states.lambda_function_schedule_time (.maximum, .minimum)aws.states.lambda_functions_started

@technovangelist

@technovangelist

@technovangelist

@technovangelist

@technovangelist

@technovangelist

@technovangelist

Lambda and Step Function Challenges

• Logging and debugging is a bit…painful• Don’t trust the SF when it claims a step is complete• Cold start is a thing but it may not be as bad as you think. • Cold start is a thing and it could be worse than you think• Every Lambda should do one thing…except if you can’t

afford it

@technovangelist

@technovangelist

Summary

Step Functions make AWS Lambda orchestration easyLambda, Serverless, and Step Functions can be an interesting part of your infrastructureMonitor your Lambda functions to ensure peak performance

Source can be found at:https://github.com/DataDog/mattw-reinvent2017-demo

@technovangelist

Matt Williams

Evangelist at Datadog@technovangelistmattw@datadoghq.com

Organizer of DevOps Days Boston 2017 & 2018