AWS

EC2 - Elastic Compute Cloud

Three instances types:1. On Demand - Run as per requirement2. Reserved - Reserved Upfront3. Spot Instances - Bid for unused instances

Instances are configured for variety of purposes and are grouped based upon their applucation like:

1. General Purpose Instances2. Memory Optimised Instances3. Computing Optimised Instances4. GPU Optimised Instances

and the above EC2 instances are sub grouped in their own categories to provide a variety of instances for the user’s based upon their needs.

Security Group - Security group defines a set of rules according to which the permission to access the instances is granted (like a firewall).

It is based upon port and ip-mapping. It can be easily configured by the user based upon his needs and wants to provide access to whom.

EBS - Elastic Block Storage

Three types:1. General Purpose SSD - Now Default2. Provisioned IOPS (PIOPS) - Provisioned by the user according to his needs3. Magnetic - Older can be used to store less intensive data.

The performance of EBS is based upon two primary factors:1. IOPS - Input/Output per second2. Throughput - Read/Write rate to storage

All the GPS volumes comes with a predefined set of IOPS and also can burst @ 3000 IOPS for an extended time. The IOPS are accumulated at 3 IOPS/GB. The max IOPS supported in this system is 10,000 IOPS and 160MB/s of throughput.

Baseline performance for 100GB SSD: 100*3 IOPS/GB = 300 IOPS (for 100GB GPS)When your volume requires more than the baseline performance I/O level, it simply uses I/O credits in the credit balance to burst to the required performance level, up to a maximum of 3,000 IOPS.

With Provisioned IOPS (SSD) volumes, you can provision a specific level of I/O performance. Provisioned IOPS (SSD) volumes support up to 20,000 IOPS and 320 MB/s of throughput. This allows you to predictably scale to tens of thousands of IOPS per EC2 instance. The ratio of IOPS provisioned to the volume size requested can be a maximum of 30; for example, a volume with 3,000 IOPS must be at least 100 GiB.

● EBS volumes behave like raw, unformatted block devices. You can create a file system

on top of these volumes, or use them in any other way you would use a block device

(like a hard drive).

● You can use encrypted EBS volumes to meet a wide range of data-at-rest encryption

requirements for regulated/audited data and applications.

● You can create point-in-time snapshots of EBS volumes, which are persisted to Amazon

S3. Snapshots protect data for long-term durability, and they can be used as the starting

point for new EBS volumes. The same snapshot can be used to instantiate as many

volumes as you wish. These snapshots can be copied across AWS regions.

● EBS volumes are created in a specific Availability Zone, and can then be attached to any

instances in that same Availability Zone. To make a volume available outside of the

Availability Zone, you can create a snapshot and restore that snapshot to a new volume

anywhere in that region. You can copy snapshots to other regions and then restore them

to new volumes there, making it easier to leverage multiple AWS regions for

geographical expansion, data center migration, and disaster recovery.

● Performance metrics, such as bandwidth, throughput, latency, and average queue

length, are available through the AWS Management Console. These metrics, provided

by Amazon CloudWatch, allow you to monitor the performance of your volumes to make

sure that you are providing enough performance for your applications without paying for

resources you don't need.

S3 - Simple Storage Service

Secure, durable, highly scalable storage service. It is based on write once read many approach.

Content storage and distribution, Disaster Recovery, Static website hosting etc.

Two components:

1. Buckets - Container Service for Objects, organises s3 at highest level of hierarchy,

provides access control, unit for usage reporting

2. Objects - Files, folders stored inside of the Buckets.

Objects consists of Data opaque to s3, MetaData which is set of name-value pair that describes

the object and Object is uniquely identified inside the bucket by a key (name) and a Version ID.

Keys: Used to uniquely identify objects inside buckets. Every Object inside a bucket has a

unique key.

A combination of bucket, key and version ID uniquely identify objects inside amazon s3.

Permissions can be set in 3 different ways in S3

1. User-Based Permissions/IAM (By creating/modifying user’s and attach roles to them

from AWS Management console from existing roles or creating new roles)

2. Bucket Policies (Permissions can be set on a bucket level so that the objects inside the

bucket will have those permissions, fine grained) [ex: grant all user’s read only

permission on buckets]

3. ACL’s (Grant permission to bucket or its objects at object or bucket level, coarse grained

approach, can apply basic read/write permissions to other AWS accounts, limits to grant

permissions by using ACL)

Storage Classes:

1. Standard: Designed for high redundancy 11 9’s percent. Designed to sustain loss of data

in two facilities (provides good data replication so high redundancy)

2. Reduced Redundant Storage: It’s less redundant than standard stoarge class also less

expensive. Provides 99.99% durability.

3. Glacier: Based upon write once never read approach. Used for archiving data where

data is infrequent and sereval hours of retrival time is acceptable.

Storage class in S3 storage can be modified as required or while objects are loaded.

Other features of S3:

Encryption: S3 by default does encryption (SSE-SE) of data using it’s standard encryption

library (256 bit symmetric key). Provides a good level of encryption using multiple keys.

a. Data is encrypted by a key.

b. The key used to encrypt the data is itself encrypted using a master key.

c. The master key is itself rotated on a regular basis for security.

User can also encrypt its data using his own key(SSE-C). This key is not stored by S3 and is

immediately discarded after encryption is done.

User can use AWS key management serivce for encryption of data (SSE-KMS). This approach

also provides auditing features where one can see who has used his encryption key and when.

Audit Logs: Provides logging facility for auditing purposes.

Multi-Factor Authentication Delete

Time limited access to objects

Versioning: Select the object and click on versioning. To have multiple versions of the same

object.

Cross region replication of Data: create new bucket in a different region. Activate replication in

the source bucket and point it to the destination bucket in the other region.

Lifecycle Rules (Archive to Glacier): Lifecycle of objects can be configured in which after the

defined duration the object will get archived in Glacier and after the defined “permanent delete”

duration the object will be deleted from the Glacier as well. Retrieving of objects from glacier

takes time.

Static Website Hosting: Can be used to host static website using buckets and one can also use

Route53 to manage DNS service. Scales automatically according to traffic demands. [ex: create

two buckets named site.com (with default index.html and error.html page) www.site.com bucket

which redirects to the site.com bucket. Set A record of website to point to site.com bucket and

CNAME WWW to www.site.com bucket.]

Regions and Availability Zones

Regions are the Geographical locations where one can configure different AWS services ex:

Sydney, Singapore, Tokyo etc.

A Region can have multiple Availability zones. AWS allows us to place resources in multiple

locations.

Each region is completely independent. Each Availability Zone is isolated, but the Availability

Zones in a region are connected through low-latency links.

Amazon EC2 resources are either global, tied to a region, or tied to an Availability Zone.

Regions: Each Amazon EC2 region is designed to be completely isolated from the other

Amazon EC2 regions. This achieves the greatest possible fault tolerance and stability.

Amazon EC2 provides multiple regions so that you can launch Amazon EC2 instances in

locations that meet your requirements.

When you view your resources, you'll only see the resources tied to the region you've specified.

This is because regions are isolated from each other, and we don't replicate resources across

regions automatically.

Availability Zones: When you launch an instance, you can select an Availability Zone or let us

choose one for you. If you distribute your instances across multiple Availability Zones and one

instance fails, you can design your application so that an instance in another Availability Zone

can handle requests.

Note: The various resources are tied to different levels in AWS. As mentioned:

Resource Type Description

AWS

Account

Global You can use the same AWS account in all regions.

Key Pairs Global or

Regional

You can use the key pairs that you create using Amazon EC2

only in the region where you created them. You can create

and upload an RSA key pair that you can use in all regions.

For more information, see Amazon EC2 Key Pairs.

Amazon

EC2

Resource

Identifiers

Regional Each resource identifier, such as an AMI ID, instance ID, EBS

volume ID, or EBS snapshot ID, is tied to its region and can be

used only in the region where you created the resource.

User-

Supplied

Resource

Regional Each resource name, such as a security group name or key

pair name, is tied to its region and can be used only in the

region where you created the resource. Although you can

Names create resources with the same name in multiple regions, they

aren't related to each other.

AMIs Regional An AMI is tied to the region where its files are located within

Amazon S3. You can copy an AMI from one region to another.

For more information, see Copying an AMI.

Elastic IP

Addresses

Regional An Elastic IP address is tied to a region and can be associated

only with an instance in the same region.

Security

Groups

Regional A security group is tied to a region and can be assigned only

to instances in the same region. You can't enable an instance

to communicate with an instance outside its region using

security group rules. Traffic from an instance in another region

is seen as WAN bandwidth.

EBS

Snapshots

Regional An EBS snapshot is tied to its region and can only be used to

create volumes in the same region. You can copy a snapshot

from one region to another. For more information, seeCopying

an Amazon EBS Snapshot.

EBS

Volumes

Availability

Zone

An Amazon EBS volume is tied to its Availability Zone and can

be attached only to instances in the same Availability Zone.

Instances Availability

Zone

An instance is tied to the Availability Zones in which you

launched it. However, note that its instance ID is tied to the

region.

VPC

Used to create own private cloud inside AWS and configure networking policies just like in local

data center using Network ACL’s, Security Groups, NAT and Network Gateway’s, VPN’s.

By default now AWS comes with a default VPC preconfigured which can be used or user can

create their own VPC.

It has two components:

1. VPC

2. Subnet’s

VPC belongs to a particular region ex: 10.0.0.0/16 is a VPC and it can have multiple Subnets

present in different availability zones. Here by basic networking we can have 256 subnets of the

type 10.0.1.0/24, 10.0.2.0/24,10.0.3.0/24 and so on…

These subnet’s can belong to a single az or can be present in multiple az. Instances can be

booted in multiple subnet’s present in different az to achieve high availability of resources.

By default All subnet’s within a VPC can communicate with each other using private ip

addresses.

Instances inside a Subnet can be connected using ssh in the following manner:

1. Using VPN tunnels

2. Assign gateway to VPC (only one gateway can be assigned to each VPC), Assign public

ip to the instance, configure route so that traffic can pass through the internet gateway.

then ssh is possible to public ip address.

3. The NAT is used in case only if instances wants do download packages, softwares and

updates from the internet. This cannot be used in connecting us with the instances.

The VPC in Amazon can be connected to our existing infrastructure securely using IPSec VPN

connection. We can also configure it for HA using Direct Connect and VPN for failover

conditions. For more higher HA each subnet can have a separate vpn connection using BGP (2

IpSec tunnels). A total of 4 Ipsec tunnels and multiple gateways on customer ends enhances

the availability further. Using Aws direct connect in the approach with VPN as backup eliminates

the possible chances of downtime.

In VPC we can have one Subnet publicly available through internet gateway and other subnet’s

as private.

We can configure software VPN for VPC to VPC connectivity.

Software firewall to internet can be done using NAT instances and configuring routes so that all

traffic to internet passes through these instances.

VPC Peering: It enables one VPC to communicate with other VPC’s. It is a really helpful

approach which can be architectured for clients based upon their requirements. ex: single VPC

in center communicating with other VPC’s to gather logs from multiple VPC’s or for code

deployment strategies.

VPC Peering can also be done across accounts.

Note:

(Transit connection is not possible ex: connecting VPC C from VPC A via VPC B).

Also, peer VPC address ranges cannot overlap.

Security groups not supported across VPC’s

Nat can also be configured for VPC peering in which traffic will flow to and fro between the VPC

through the NAT instances.

AWS Elastic Load Balancer

Used to balance incoming traffic load among the instances

In AWS VPC ELB is supported both on public ip address and on private ip addresses.

Full control over ELB Security group.

The ELB is created on a separate VPC from that of the servers and the interface is securely

connected with the server’s. If it’s a private ELB a private ip-address is assigned if public then a

public ip address is assigned.

Two types of listeners

1. TCP/SSL

Incoming client connection bound to server connection, no header is attached from elb,

round robin algo used.

2. HTTP/HTTPS

Incoming client connection terminates at ELB and pooled to the server, headers may be

modified, uses x-forwarded for header for client ip-address, least outstanding request

algo used for routing, sticky session support session available.

Health Checks

Two types of health checks:

1. TCP (not that reliable, ICMP ping response)

2. HTTP (more reliable, must return a 2xx response)

Idle Timeouts

Length of time for which client connection should be kept open. Default is 60 seconds can be

set between 1 to 3600 seconds.

It allows the ELB to close the client session when no longer in use.

We can have Multiple instances running the same application in different availability zones and

also multiple ELB’s in different availability zones pointing to the corresponding instances and

Route53 load balancing incoming traffic on the different load balancers.

We can also have a situation in which we have instance/s in a single availability zone and ELB’s

in multiple availability zones pointing to the same instance/s present in the single availability

zone.

By default now it is recommended to have the ELB present in multiple availability zones to

increase redundancy.

Cross Zone Load Balancing

Using cross zone load can cause traffic in balance between the instances to resolve this it is

better to enable cross zone load balancing among the elb’s.

SSL and HTTPS ciphers are used according to industry standards.

Queue of upto 1024 is supported by load balancer in case the instances are not able to scale to

such an extent after which a 503 error is shown to the client if connection exceeds.

elb also supports logging which records client ip, date , time, request path and server response.

elb can handle failure of single availability zone, Route53 re routes the traffic away from the

failed elb.

RDS

Database services provided by AWS. Consist of Mysql, SQL, Postgres and Oracle databases.

No need to manage database (backups/update/upgrade etc.) done by AWS. Just upload the

data to the database after spinning an RDS.

No need to install any software or DB software done by AWS itself.

We can replicate the database to multiple availability zones to ensure high availability in case of

any failures.

Also supports cross region replication.

Data File upload can be done in two ways

1. Standard Mysql: Processes sequentially, restart if failed, good for small databases.

2. Flat file: Load schema initially then the data, resume data restore in case of failure, best

for large databases.

Can also be configured for Replication. Data can be warehoused in AWS Redshift from RDS as

a source.

Also supports read replicas which only serve read requests of the client. Scale the RDS

instance by considering Read and Write Ratio’s and using multi read replica’s if application is

more read oriented and distribute the read traffic among the replica’s (note: this will not affect

the write operation as data will have to written on all replica’s equally).

InnoDB Buffer warming: Dump buffer pool, reload after reboot is complete. Cache warmimg can

be done by defining the time period (like 1 hour) so that if a failure occurs after half an hour than

we will get cache warmed up though it will be 30 minutes stale (performance improvement than

may not be high).

Changing schema: can be done in multiple ways

1. Using Read replica: promote the read replica to R/W and make the schema changes in it

than create a read replica from it and promote the R/W replica as master(Not easy, may

break replication).

2. Native 5.5 feature: Easy to use, less performance impact. Some performance

implication. Takes relatively less time.

3. PT Online schema change: run from an ec2 instance to perform schema changes.

Needs installation, could take longer, less performance impact.

Burst Mode: GP2 and T2

GP2 uses SSD and gives 3IOPS/GB base and 3000+ IOPS on burst

T2 gives credits per hour when below base performance, can store 24 hours of credits AWS

cloudwatch matrix to see credits.

CloudFront

Caching facility provided by AWS which transmits copy of our content to various edge locations

so the users are served the content with less latency and reduces load on base instances.

Content is divided into 3 categories:

1. Static Content (images, css, javascript etc., can be distrubted to more than one user,

doesn’t change for sec, min, hours)

2. Dynamic Content (php, html etc.)

3. Live Streaming (videos, live streaming etc.)

Static Content on S3

Transfer of static content from S3 to Cloudfront is free, decrease load on server, highly available

and scalable

Control access to content on S3 using OAI (Origin Access Identity), content can be accessed

only via cloudfront, ensures content is not leaking and S3 url is not being used anywhere else.

Control Access to content on Cloudfront using Signed Url (ex: marketing email) or Signed

Cookies (ex: streaming, whole site authentication).

Cache at every layer: browser caching. Set max-age or expiry date in headers, HTML5

application cache, helps eliminate network latency, browser cache is limited

Cache at every layer: edge caching. Set High TTL for intermediary caches, don’t forward query

strings or headers or cookies. Use cloudfront defaults.

Version Objects

Allows for easy updates and rollback, use version names so no additional api calls are needed,

set high TTL on objects that do not change frequently, each object is treated as a unique object

in the browser

Dynamic Content

Content unique to every request(/index.php), Content Changes frequently (seconds, minutes)

but not unique for every request (Weather updates, API etc.), content changes on end user

request (query strings, cookies, headers) (ex: mobile vs desktop user’s, search keyword in

query strings etc.)

Cloudfront supports TTL as low as 0 seconds. no-cache, no-store etc. most content can be

cached even if for few seconds, helps offload origin load, cloudfront will server stale content if

origin is unavailable, supports “if-modified-since”, “if-none-match” when object in cache has

expired.

Provides reporting for various objects.

Multiple cache behaviour. Only forward required headers. avoid forwarding user agent header,

avoid forwarding all cookies

Streaming Media

Live streaming, On demand streaming, audio streaming, generally involves transmit of manifest

file, media file and player. Manifest file (TTL: 2sec), Media file (60sec), Player (static store in S3,

high TTL 24 hrs)

Use HTTP based streaming protocols, use cloudfront with ec2 for media straming,use

fragmented streaming formats(native in cloudfront), HLS etc, dont forward header, query string,

cookies, use signed cookies.

We can set custom error pages in cloudfront.

Design for failure. Set Route 53 to point to two elb’s instead of one so in case of failure the

cache can get data from server’s in separate availability zones.

Supports SSL between browser and edge also between edge and origin, redirect HTTP to https

to use encryption in apache config.

Use AWS IAM (for user’s to permit access to API’s)< Use Cloudtrail to record API calls. Use

*.cloudfront.net SSL. Certificate cost per IAM certificate, Can use single UCC certificate for

multiple two level domains, can also use wildcard certificate for three level domains.

Use different price classes for various edge locations and optimize cost accordingly.

Domain Sharding. Many browser’s limit the number of parallel connections to a single domain,

shard assets over multiple domains, Use multiple CNAME aliases and wildcard Cert for SSL.

Also supports Access log for analysis.

Route 53

Route 53 not only can be used for DNS resolving but also for HA of our resources in case of any

failures at the DNS level itself.

Route 53 can be configured for automatic failover in which case if a failover occurs it reduces

the downtime with a considerable amount of time and takes actions automatically without

human intervention.

Route 53 checks for two parameters:

1. Health: Is the ELB or EC2 is available or not

2. Weight: How much weight is present on each ELB or EC2 in multiple subnets.

ex: If we have two elb’s serving the same content with a Weight of 50 on each and suppose one

of the elb goes down then the load will be shifted by route53 to the other elb ie. to 100 which

increases availability of application in case of any failure.

If the other elb also due to any reason goes down then the Route53 does health checks and if

found negative switches to the other failure process (S3 bucket content in this example)

automatically.

ex:

Record Type Health Check Value Weight

reinv.net ALIAS (Evaluate Target) prod.reinv.net 100

reinv.net ALIAS reinv-fail.s3-web 0

prod.reinv.net ALIAS (Evaluate Target) prod-1.elb 50

prod.reinv.net ALIAS (Evaluate Target) prod-2.elb 50

Glacier

Amazon Glacier is an extremely low-cost storage service that provides durable storage with

security features for data archiving and backup. With Amazon Glacier, customers can store their

data cost effectively for months, years, or even decades. Amazon Glacier enables customers to

offload the administrative burdens of operating and scaling storage to AWS, so they don't have

to worry about capacity planning, hardware provisioning, data replication, hardware failure

detection and recovery, or time-consuming hardware migrations.

Amazon Glacier is a great storage choice when low storage cost is paramount, your data is

rarely retrieved, and retrieval latency of several hours is acceptable.

Amazon Simple Storage Service (Amazon S3) supports lifecycle configuration on a bucket that

enables you to optionally transition objects to Amazon Glacier for archival purposes.

In Amazon Glacier, a vault is a container for storing archives, and an archive is any object, such

as a photo, video, or document that you store in a vault.

Any archive operation, such as upload, download, or deletion, requires that you use the AWS

Command Line Interface (CLI) or write code. There is no console support for archive operations.

For example, to upload data, such as photos, videos, and other documents, you must either use

the AWS CLI or write code to make requests, using either the REST API directly or by using the

AWS SDKs.

aws-cli to upload archive to glacier:

aws glacier upload-archive --account-id - --vault-name my-vault --body archive.zip

In general, retrieving your data from Amazon Glacier is a two-step process:

1. Initiate a retrieval job.

2. After the job completes, download the bytes of data.

The retrieval job executes asynchronously. When you initiate a retrieval job, Amazon Glacier

creates a job and returns a job ID in the response. When Amazon Glacier completes the job,

you can get the job output (archive data or inventory data).

Note:

1. Files uploaded to glacier vault can also be uploaded in chunks which enables better

performance and resume capability if the internet connection is not stable.

2. Glacier also provides granular security and compliance in which user’s can configure

strict policies ex: any archive stored in amazon vault must have to be retained for a

duration of 365 days and cannot be deleted even by super user account (admin).

Amazon allows to review these policies in a period of 24hrs because once a policy gets

applied it cannot be rolled back afterwards. These policies can be easily configured and

associated with vault’s using AWS console.

DynamoDB

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and

predictable performance with seamless scalability. No need for administrative tasks like scaling,

software updates, patching etc. done at AWS level itself.

With DynamoDB, you can create database tables that can store and retrieve any amount of

data, and serve any level of request traffic. You can scale up or scale down your tables'

throughput capacity without downtime or performance degradation, and use the AWS

Management Console to monitor resource utilization and performance metrics.

Tables

Similar to other database management systems, DynamoDB stores data in tables. A table is a

collection of data.

Items

Each table contains multiple items. An item is a group of attributes that is uniquely identifiable

among all of the other items.

Attributes

Each item is composed of one or more attributes. An attribute is a fundamental data element,

something that does not need to be broken down any further.

Primary Key

When you create a table, in addition to the table name, you must specify the primary key of the

table. As in other databases, a primary key in DynamoDB uniquely identifies each item in the

table, so that no two items can have the same key.

RedShift (MPP Database for high vol structured data)

Redshift is data warehousing service provided by amazon. It is used for databases that are

highly structured but also very large in size which cannot be handled by traditional relational

databases which are best suited for high structured data but able to manage low quantity of it.

Redshift can be easily scaled from a 2TB datastore to many PB (PetaBytes) of data storage.

Amazon Redshift comprises of a single master node and adjacent computing nodes. The

queries which comes up through master node is equally distributed among the computing nodes

which run the queries in a parallel fashion and returns the end result. This results in an overall

reduction in execution time of the query as compared to a single node executing it. The same

holds true for writing in these kind of databases. Works with existing analysis tools.

Features:

1. Standard SQL

2. Optimized for fast analysis

3. Very scalable

Redshift Features:

1. Uses column based approach to store data (row based storage causes additional I/O,

with column approach only read data that we need)

2. Uses Data Compression (COPY compresses automatically, can analyze and override,

more performance less cost)

3. Zone mapping (Track minimum, maximum value for each block, skip over blocks that

don’t contain relevant data)

Data can be loaded into Redshift using S3, DynamoDB, EMR, Linux box. If already have data

sources from which you want to upload data to Redshift either frequently or infrequently use a

intermediate layer like ETL(Execute, Translate, Load) service provided by many third parties

like: informatica etc.

EMR (Elastic Map Reduce) (Hadoop)

EMR is the service for Applications that require Hadoop infrastructure. This kind of databases

are best suited for data which is large in size and also highly unstructured. Hadoop uses a

clustering approach in computing. It includes:

1. Input file: This is the data on which computing has to be performed. Partitioned into

multiple files.

2. Functions: The operation that has to be performed on the data.

Putting Data into EMR

1. Put the data into S3

2. Choose hadoop distribution, no. of nodes, types of nodes, hadoop apps live hipe, pig

etc.

3. Launch the cluster using EMR console. CLI, SDK or API’s

4. Get the output from S3

By adding data into S3 it can then be used by many EMR clusters for analysis on the same set

of data running different functions and output the result back into S3.

The clusters can be easily resized and many parallel clusters can be executed in AWS.

Can also use spot instances for Hadoop computing nodes. The cluster can also be terminated

after the analysis is complete to reduce costs.

The data can also be stored in HDFS (local disk) if you don’t want to store it in S3.

EMR Cluster is divided into 3 parts:

1. Master instance group: It is the primary managing node in the hadoop cluster.

2. Core Instance group: These are the instances that have HDFS(local disk) attached with

them and performs processing on the data to provide output.

3. Task Instance group: This instance group doesn’t has any HDFS attached with them.

These are only used for computing on the data. This instance group can also be

provisioned with spot instances to reduce cost. Multiple Task instance groups can be

attached in a single EMR cluster and can be provisioned with spot instances for multiple

price ranges. By using multiple spot options in this group helps in managing cost and

also managing fluctuations in processing power if spot instances become over priced.

Lambda

AWS Lambda is a compute service where you can upload your code to AWS Lambda and the

service can run the code on your behalf using AWS infrastructure. After you upload your code

and create what we call a Lambda function, AWS Lambda takes care of provisioning and

managing the servers that you use to run the code.

AWS Lambda runs your code on a high-availability compute infrastructure and performs all of

the administration of the compute resources, including server and operating system

maintenance, capacity provisioning and automatic scaling, code monitoring and logging. All you

need to do is supply your code in one of the languages that AWS Lambda supports (currently

Node.js, Java, and Python).

When using AWS Lambda, you are responsible only for your code. AWS Lambda manages the

compute fleet that offers a balance of memory, CPU, network, and other resources. This is in

exchange for flexibility, which means you cannot log in to compute instances, or customize the

operating system or language runtime. These constraints enable AWS Lambda to perform

operational and administrative activities on your behalf, including provisioning capacity,

monitoring fleet health, applying security patches, deploying your code, running a web service

front end, and monitoring and logging your Lambda functions.

The lambda function can be configured as

1. event driven: If a particular event occurs the lambda function will be called.

2. HTTP request: The lambda function will be executed if an HTTP request is made using

the API’s.

Lambda can be used to create backends, event handlers and data processing systems.

Services supported by lambda includes: S3, DynamoDB, Kinesis, Cloudformation, SWF,

Cloudtrail, SES, Cognito, API Gateway, SNS, Cloudwatch logs.

(ex: lambda can be used for auditing cloudwatch logs, where cloudwatch calls the lambda

function which in turn can process the requests and store the results in DynamoDB, S3, or

Redshift. ex: Lambda can also be used to create thumbnail of images being uploaded to S3

bucket.)

Lambda can now run functions for up to 5 minutes.

It also now supports versioning capability where we can use different versions of the same

code, aliases can also be associated to these different versions of the code.

Resource sizing can be done in lambda to provision the resources that will be used while

running the lambda functions or it can be left to be done by lambda itself automatically.

Lambda also supports scheduled execution of lambda functions using standard cron syntax.

Lambda functions can also be provisioned to run inside our own private VPC.

IAM

Identity Access management provides a means by which we can control which users can

access what resources inside of AWS. It basically is divided into categories like:

1. User: Defines the user and its corresponding credentials (password, Multi-factor

authentication, access key)

2. Groups: This comprises of group of users with a predefined set of access permissions to

AWS resources.

3. IAM Policies: These policies can be attached with users or groups and defines the

resources and the level of permission on those resources and whether the access is

allowed or not.

Best practices:

1. Grant least privileges to users: Grant only the permissions that are required.

2. Is user’s will be accessing the resources only through code and api requests then allow

access using access keys if they require access through management console then only

grant them access through passwords.

3. Define users in groups and assign permissions to the group this helps in easy

management.

4. Restrict access by adding conditions.

5. Enable AWS Cloudtrail to get logs of all the API calls being made.

6. Rotate security credentials regularly.

7. Use multi factor authentication.

8. Use IAM roles to share access.

9. Use IAM roles on ec2 instances to access resources.

Code Commit

AWS CodeCommit is a fully-managed source control service that makes it easy for companies

to host secure and highly scalable private Git repositories. CodeCommit eliminates the need to

operate your own source control system or worry about scaling its infrastructure. You can use

CodeCommit to securely store anything from source code to binaries, and it works seamlessly

with your existing Git tools.

Access to these repositories can be given to users using IAM.

Code Deploy

Code Deploy automates the manual process of deploying code to a single or multiple instances.

It needs codedeploy agent running on ec2 instances to be able to use it. It can also be used to

deploy code at on-premises servers or at any other data center which is chargeable but not

chargeable if used for ec2 instances. Code Deploy can be configured with Git repositories and

adding Appspec YAML file with the application code. The YAML file consists of three important

details:

1. file source and destination: gives the source and destination location to which the files

will be copied.

2. File permissions: can be used to grant privileges to different files/directories in the code.

3. Hooks: These are different kind of hooks which can be executed at different levels. Ex:

Setup of apache server, starting apache server, restarting apache server. These hooks

can execute different kinds of scripts like sh, python etc. and perform different kind of

operations we configure in those scripts.

CodeDeploy helps in completely automating the process of deployment based upon different

approaches:

1. One at a time: In this approach one server is updated at a time with rest other’s serving

the users.

2. Half at once: Half of the instances are updated once leaving the rest for delivering

content.

3. All at once: All the server’s are updated at once.

4. Custom: Define our own number of instances for deployment at a time.

This helps in updating code running on multiple servers easily and effectively. If code deploy

finds any abnormality in the updation process it automatically rolls back to previous edition of

the code. It detects updation process as unsuccessful if the user defined scripts generates error

during their execution or flags as successful if all the scripts defined by the user executes

successfully. Code Deploy can also be configured with integration tools like Jenkins etc.

Code Pipeline

AWS CodePipeline is a continuous delivery service for fast and reliable application updates.

CodePipeline builds, tests, and deploys your code every time there is a code change, based on

the release process models you define. This enables you to rapidly and reliably deliver features

and updates. You can easily build out an end-to-end solution by using our pre-built plugins for

popular third-party services like GitHub or integrating your own custom plugins into any stage of

your release process. With AWS CodePipeline, you only pay for what you use. There are no

upfront fees or long-term commitments.

You will be able to use the CodePipeline user interface to construct a graphical model of your

release process using a combination of serial and parallel actions. The workflow can include

time-based or manual approval gates between each stage. For example, you could choose to

deploy new changes to staging servers only during weekday working hours in your own time

zone.

CodePipeline watches your source code repo for changes and triggers the appropriate

workflow. A release workflow could build the code in a production build tree, run test cases, and

deploy tested code to a staging server. Upon final approval (a manual gate), the code can be

promoted to production and widely deployed.

Code Pipeline has variety of stages in configuration phase:

1. Name of the pipeline: Give any name as required.

2. Source: Code Source (Amazon S3, Github)

3. Build Provider: Choose if project requires creation of build (ex: for java based projects).

Two options are present: a. No Build, b. Jenkins (give jenkins url, name of project in

jenkins)

4. Beta: This is the beta or testing stage in pipeline (This acts as the staging phase in the

pipeline, production stage will be added later on after the pipeline is built using the initial

UI). We can define how we want to deploy the code in the Beta phase either by

CodeDeploy or by Elastic Beanstalk. Ex: Selecting CodeDeploy define the Application

Name present in CodeDeploy as well Deployment Group present in CodeDeploy which

will handle the Beta release.

5. Service Role: This will redirect to IAM roles. Define the role which will be used by

CodePipeline and give it permission of Allow so that code pipeline gets permission to

deploy the code.

6. Review: This stage gives an overview of all the settings that the user has defined in his

pipeline.

After the above steps have been completed we are provided with a graphical representation of

our pipeline with stages such as: a. Source (Representing Github) and Beta (Defining

CodeDeploy for Code Deployment). Now we can add more stages in this pipeline using the UI

and define the actions which will take place inside of that stage. In this manner we can create

our own custom pipeline which defines our complete release process with multiple stages for

staging, testing and production phases as per our requirement.

CloudFormation

AWS CloudFormation gives developers and systems administrators an easy way to create and

manage a collection of related AWS resources, provisioning and updating them in an orderly

and predictable fashion. You can use AWS CloudFormation sample templates or create your

own templates to describe the AWS resources, and any associated dependencies or runtime

parameters, required to run your application.

You can also visualize your templates as diagrams and edit them using a drag-and-drop

interface with the AWS CloudFormation Designer.

The CloudFormation has following components.

1. Template: Templates consists of a group of resources with their properties which will

build up your application infrastructure. Templates can be configured using GUI tool

known as CloudFormation Builder. The format of template is json. Each resource is

listed separately and specifies the properties that are necessary for creating that

particular resource.

2. Stack: Before you create a stack from a template, you must ensure that all dependent

resources that the template requires are available. A template can use or refer to both

existing AWS resources and resources declared in the template itself. AWS

CloudFormation takes care of checking references to resources in the template and also

checks references to existing resources to ensure that they exist in the region where you

are creating the stack. If your template refers to a dependent resource that does not

exist, stack creation fails.

3. Builder: Graphical Interface provided by AWS which can be used to create Templates by

visualising different resources with their relations and dependencies.

Cloudformation also supports capability to check the progress of stack creation the different

resources created during stack formation are also labelled accordingly to make recognition

better.

The resources created during the creation of stack can be cleaned up easily by deleting the

stack.

Elastic BeanStalk

Aws service by which we can deploy different kinds of applications (ex: python, node.js, php,

java etc.) from simple clicks of a button and define what infrastructure resources are needed for

the application with their respective properties without the need for us to configure those

resources. ex: suppose we have a compressed python application in the form of zip file, By

using Elastic BeanStalk wizard we can:

1. Define the name of the beanstalk with a short description

2. Now set the environment tier i.e. Web Server, Select the platform i.e. Python for our

example, environment type i.e. Autoscaled or Single Instance.

3. upload the compressed form of the application or upload it into s3 and provide its url we

can also use sample application.

4. Environment information: Name your environment or go by default. Also consists of the

environment url and provide description for the environment if required.

5. Configure RDS properties if your application requires database connectivity, we can also

configure vpc in this wizard.

6. In the Configuration details page select the type of instance you want for your

application, key pair to be used, email address, root volume type etc.

7. Environment tags can be configured to tag our environment resources.

8. Review the configured settings and Launch if the settings meets up our requirements.

It also does health check of our application and in case of any error the beanstalk process rolls

back. we can now access our application with the elastic beanstalk url provided to us by the

interface.

Application updates can also be performed using eb cli to using “eb deploy” after configuring

some setting parameters for elastic beanstalk cli.

It has 3 components:

1. Environment: Infrastructure resources like EC2, elb, autoscaling, runs single application

version at a time, an application can have multiple environments like testing and

production.

2. Application versions: Application code, Stored in S3, An application can have many

application versions (easy to rollback to previous versions)

3. Saved configurations: Configuration that defines how an environment and its resources

behave, can be used to launch new environment quickly or roll back configuration, an

application can have multiple saved configurations

Beanstalk also provides us with a temporary CNAME that can be used for application testing

purposes.

Application Versions: An application can have multiple versions stored in S3, code can also be

deployed from github

Application Environments: An application can have multiple environments like testing and

production and a particular version of the application can be deployed in the environment

Saved Configurations: Save these for easy duplication for A/B testing or non-desruptive

deployments.

Deployment can be done either by using aws management console, cli or eclipse or VS add ons

Initial App Workflow using cli:

1. Initialize git repo: git init .

2. Create elastic beanstalk app: eb init, and follow prompt to configure environment.

3. Add your code: git add .

4. Commit: git commit -m “message”

5. launch application using beanstalk: eb create

Update your app:

1. Update code.

2. Push updated code: git add . , git commit -m “v2”, eb deploy

3. Monitor Deployment status: eb status

Add extensions to your environment using ebextensions like sqs, cloudfront.

Zero Downtime Deployment: Using rolling deployments to achieve zero downtime deployments

using beanstalk.

OpsWorks

OpsWorks enables us to create our infrastructure stack using the capabilities of chef and its

recipes. Opsworks provides us with more control over our infrastructure than provided by

Beanstalk (which is more suitable to deploy a simple web application, and all the configuration

part is handled by amazon itself). Opsworks is based upon layered architecture where different

resources like servers, databases etc. are added as different layers inside the stack. By using

opswork we can not only define our the layers (instances, database etc.) which will be part of

our infrastructure stack but we can also customize these layers as per our needs. We can also

execute sample chef recipes provided by amazon or can also create our own recipes which will

be executed using opswork. Opsworks can also be used for on premises infrastructure. It also

easily can be integrated with third party CGI tools for application deployment using tools like

Jenkins, Travis etc.

1. Create a Stack. A stack contains the set of Amazon EC2 instances and instance

blueprints, called layers, used to launch and manage these instances (e.g., all the PHP

servers and the MySQL database used for your production web application). Apps, user

permissions, and resources are scoped and controlled in the context of the stack.

2. Define the layers of your stack. A layer defines how to set up and configure a set of

instances and related resources such as volumes and Elastic IP addresses. AWS

OpsWorks includes layers for common technologies such as Ruby, PHP, HAProxy,

Memcached, and MySQL (Linux only), and makes it easy to extend existing layers or

create custom layers. Lifecycle events can be used to trigger Chef recipes on each

instance to perform specific configuration tasks. For example, the deploy event could

trigger a script to create a database table for a new app.

3. Assign instances to your layers. Create instances in configurations you choose,

including instance size, Availability Zone, volume creation and RAID configuration, EIP,

security group, and operating system. Start your instances, or apply them to auto scaling

groups.

4. Define and deploy your apps. To define an app, simply tell AWS OpsWorks where it can

find your code and specify any additional deployment tasks, such as database

configuration. AWS OpsWorks supports a variety of repositories such as Git, SVN,

HTTP, and Amazon S3. When you deploy the app, AWS OpsWorks will pull code from

your repository, place it on the instances, and run the specified deployment tasks so that

your application is properly configured. Afterwards, you can view your app’s deployment

logs to review the deployment steps, verify its functionality, and debug any issues.

The recipes can be executed at different time levels or can also be executed manually:

● Setup occurs on a new instance after it successfully boots.

● Configure occurs on all of the stack's instances when an instance enters or leaves the

online state.

● Deploy occurs when you deploy an app.

● Undeploy occurs when you delete an app.

● Shutdown occurs when you stop an instance.

CloudWatch

Monitoring service provided by AWS which helps in gathering different data from our resources

and represent it in the form of graphs and enables us to set alarms for different kinds of

situations using SNS. Cloudwatch has a throughput to the limit to the data being sent which is

approximately 1 Mbps. Sometimes there may be a delay in the data arriving at Cloudwatch.

Logs in cloudwatch can be set for deletion after a particular interval. Filter types for logs

supported are Literal Terms, Common Log Format, JSON.

Data is gathered from different sources:

1. Default: By Default cloudwatch gathers data like CPU Utilization, Memory Utilization,

Network in/out, Read/Write to disks etc, these logs are collected by default represented

in the form of graphs and can be used in making relevant decisions in the planning

process, alarms can also be set on these based upon user defined matrices.

2. EC2 internal logs (User Defined): Various logs files generated inside of ec2 instances

(ex: /var/log/secure) or custom logs for our application (ex: /var/log/apache2/app.log) can

also be transmitted to cloudwatch. An agent has to be installed on the instance/s for this

purpose and configured to send user defined logs to cloudwatch. In the cloudwatch

panel we can easily access these logs data as raw, graphs or can also set alarms based

upon our own custom parameters (ex: error, debug, custom strings, HTTP responses

etc.) to send us notifications based upon SNS topics we define.

3. S3 logs (bucket logs): S3 buckets flag for logging can also be set and a cron job can be

set in an ec2 instance to read the logs from the s3 bucket and then send it to cloudwatch

(make sure the instance as sufficient permissions to do so), as the data is sent to

cloudwatch we can then access it using cli or by console as well as set alarms and view

GUI graphs for the same.

4. CloudTrail logging: AWS service CloudTrail now also comes with the the support for

cloudwatch by which we can provision CloudTrail can send logs to cloudwatch.

These logs collected are used by different services:

1. Used in generating graphical UI representation in the form of graphs.

2. Used in setting up alarms for notifications.

3. Used by AWS service known as Advisor for giving recommendation to improve cost,

performance and security of our cloud infrastructure.

4. Cloudwatch data can also be gathered using cli, api and stored in files like xlsx or can be

used by applications for processing.

5. Integration with other (third party) monitoring tools.

CloudTrail

While using different services offered by amazon like S3, Database, Cloudfront etc. we access

these resources by making API calls. CloudTrail provides us the feature to get the details of all

these activities in the form of log files which can be used for troubleshooting, compliance,

security, analysis and monitoring. These logs are stored in S3 bucket defined by the user.

These logs can also be sent to Cloudwatch for analysis and setting up notifications.

It answers the questions like:

Who made the API call, When made the API call, What was the API call, Which resources acted

upon in the API call, Where the API call made from and made to.

During Creation process:

1. Mention the name of the Cloudtrail and define whether to create a new bucket in S3 or to

use and existing bucket for storing logs. This process can also be done using cli.

The logs stored by CloudTrail in S3 can also be configured to get moved to Glacier after a

certain time period for compliance.

CloudTrail can also be configured to send logs from multiple aws accounts to a single bucket.

Files will be arranged per account and region in S3 bucket for easier future access.

By Default CloudTrail encrypts logs stored at S3 using S3 server side encryption. We can also

define a user based key (KMS) that will be used for encryption and decryption.

CloudTrail also enables us to validate the log files stored in S3 are not changed. Detect whether

a log file was deleted or modified or changed. Cloudtrail will send digest file per hour for

validating the authenticity the digest file, digest file contains the hash value for the log files.

being sent will be signed by cloudtrail.

Cloudtrail also supports the integration of third party softwares for accessing these log files and

generating reports and performing analysis.

Simple Notification Service

This is the notification service provided by amazon by which the users subscribed can get

notification for our custom defined topics (events) through email, sms, etc. Can be used through

api to engage customer’s (ex: mobile apps using sns to mobile). It is a public subscribed

messaging service. The way it works is:

1. Create Topic: Create a new topic give any name to it.

2. Choose subscribers: Click subscribers and the format in which they will be notified i.e.

email, sms, http etc.

3. Once the user subscribes to the topic we can then add content to our sns topic which will

then get pushed to the subscriber by calling the topic through api or if an event occurs

which triggers the topic to send notification.

Types of publish:

1. Direct Publish(available to mobile devices only at this point): Pushes topic to a wide

variety of mobile devices whether it is iOS, Android, Kindle etc. without the concern of

using their respective push notification apis. (ex: create an application that pushes

notification to the user using sns)

2. Topic Publish: Publish topic to a wide variety of devices like sqs, email, http, sms etc

using filtering mechanisms provided by sns. (ex: users subscribe to sms notification of a

topic, now we can publish notification to those set of devices like mobiles)

SNS also accompanies well with S3 buckets and can be used to send push notifications based

upon different events for various S3 buckets like a notification system. SNS can also be

accompanied with cloudwatch which will notify user if a particular threshold exceeds and

accompanying SNS additionally with lambda function for an output functionality we wish. This

helps in additionally provisioning resources like ec2, storage etc. in case any event occurs.

Simple Email Service

Email service provided by amazon which can be used to send large number of emails using

email infrastructure provided by amazon with zero maintenance and troubleshooting. This can

be used to run email campaigns, newsletters etc. This service is used to send and receive

emails. This service can also be used by either Management console, cli or apis. The ses

service can be merged with your existing email id as the SMTP server (email sending server) by

which the emails send from your mail id will be labelled FROM address as your legitimate mail

id but the email will be sent from amazon ses servers.

The mail that gets bounced or are flagged by users as Spam reaches to their corresponding

ISP’s. The ISP’s then returns the notification back to SES which then returns the notification

back to us through email and can be configured to use SNS as well.

The email can also be configured to trigger an event in which case as any such email is

received it will execute our predefined set of codes which can be done using Lambda.

The Bounced or marked as Spam emails should be taken with high priority to make sure that

the reputation for your emails sent can be maintained and our emails reaching to customers

doesn't gets counted as Spam by Spam filters. Best practise is to stop sending email further to

such email id’s.

To use SES:

1. Make sure to verify your sender email address (through verification email) or domain

name (done by adding TXT record to domain name provided by AWS)

2. Request for production access so that you can send email to large number of users

otherwise it will be in sandbox mode in which we can only send email to authorised

emails only.

3. Get the unique Url, Username and Password for SES and use these credentials to send

large number of emails either by using mail clients like Thunderbird or Outlook, or these

credentials can also be used in application code to send out emails through code using

ses apis.

Authenticate email sent by sender

1. DKIM: adds encrypted signature to your email which is used by ISP to authenticate your

email being sent. Proves to recipient that sender of email is legitimate, also proves email

content has not been tampered with, help ISP to associate reputation with your domain.

Setup using EasyDKIM and add TXT record to DNS records, enables automatic key

rotation.

2. SPF: Tells recipient which ip address are authorised to send email from your domain

name. Done by adding DNS TXT record consisting of authorised ip addresses.

Please note daily sending quota and maximum size for sending email through ses. Request for

additional limit increase if required.

Use mailbox simulator for load testing. Also test email sizes. Also test your feedback system for

increased load.

Use HTTP instead of SMTP to send email to SES to reduce number of connections required to

and fro to send an email also use persistent connection i.e. sending all the email through a

single connection instead of opening a new connection for every email being sent. Send in

parallel to send email to a large number of users using multiple processes/threads.

Simple Queue Service

Distributed queue system which allows applications to queue messages that one component in

the application generates to be consumed by another component. With SQS we can decouple

components of our applications and make them work independently. It helps in situation where

producer is creating more messages than being processed by the consumer. SQS also allows

multiple resources (producers/consumers) to read and write to the queue. No need for these

components (resources) to coordinate with each other.

General SQS operations:

1. Create Queue: Creates Queue for storing messages

2. Create Message: Creates message inside the queue for processing

3. Read Message: Reads message from the queue.

4. Delete Message: Deletes message from the queue.

5. Purge: Purges all the messages from the queue.

6. Delete Queue: Deletes the queue.

SQS also supports batch processing by which instead of retrieving one message at a time it

reads multiple messages at once from the queue. Visibility timeout is defined by the user as the

time between producer adding message to the queue, consumer reading the message and

processing it and deleting it from the queue between the visibility timeout period.

1. A system that needs to send a message will find an Amazon SQS queue, and use

SendMessage to add a new message to it.

2. A different system that processes messages needs more messages to process, so it

calls ReceiveMessage, and this message is returned.

3. Once a message has been returned by ReceiveMessage, it will not be returned by any

other ReceiveMessage until the visibility timeout has passed. This keeps multiple

computers from processing the same message at once.

4. If the system that processes messages successfully finishes working with this message,

it calls DeleteMessage, which removes the message from the queue so no one else will

ever process it. If this system fails to process the message, then it will be read by

another ReceiveMessage call as soon as the visibility timeout passes.

5. If you have associated a Dead Letter Queue with a source queue, messages will be

moved to the Dead Letter Queue after the maximum number of delivery attempts you

specify have been reached.

ElastiCache

AWS managed service that lets you easily create, use and resize cache clusters in the cloud.

The service improves the performance of web applications by allowing you to retrieve

information from fast, managed, in-memory caches, instead of relying entirely on slower disk-

based databases. ElastiCache supports two open-source in-memory caching engines:

1. Memcached - a widely adopted memory object caching system. ElastiCache is protocol

compliant with Memcached, so popular tools that you use today with existing

Memcached environments will work seamlessly with the service. Patterns for sharding,

no persistence, multi threaded, supports strings, objects, slab allocator.

2. Redis – a popular open-source in-memory key-value store that supports data structures

such as sorted sets and lists. ElastiCache supports Master / Slave replication and Multi-

AZ which can be used to achieve cross AZ redundancy. Single threaded, read replicas,

supports data types, more like Nosql db, pub/sub functionality, persistence, atomic

operations.

1. Select Engine (Memcached/ Redis)

2. Choose (version, port, parameter group, cluster name, node type) + redis options (multi AZ & replication, S3 backup location)

3. Choose (subnet group, AZ, Security group, maintenance windows, SNS) + redis option (Enable backups)

We obtain are: Endpoints

Endpoint: Refer to particular node in your cluster

Primary Endpoint: For redis replication group, a consistent DNS alias that refers to node where

writes should be sent

Configuration Endpoint: For memcached, a consistent DNS alias to query for the list of current

nodes participating in the cluster

Multi-AZ for Redis: Using this approach a redis node can be made primary and all other redis

nodes in different az as its read replicas, all the write queries goes to the primary node and

replicated to the read replicas, provides a highly available architecture.

Connecting to redis and memcached: either telnet or using client libraries different for different

programming languages, aws also provides its own libraries to connect to caching servers.

Sharding:

1. Modulo based upon number of servers: not a good approach as alot keys get reassigned

when we add a server to the cluster

2. Consistent hashing: a better approach. Consider a ring, divide it into slots, servers are

mapped to slots.

Memcache divide the memory in the form of slots with each slot having a variable amount of

chunk size, during the write operation the data is written to the slot which is the most compatible

by its chunk size, this does waste some memory but provides us with a caching ability, In case

no free memory is available to store more data than the least recently used (LRU) object is

removed from cache and new data is written.

Monitoring can be done using phpMemcache UI or by using CloudWatch Matrices

Caching can be architectured based upon different requirements and scenarios. Basic scenario

uses cache in congestion with application server in a simple manner.

Starting with lazy caching which is highly beneficial for read intensive patterns. Writes and

updates are performed in real time using database. Handles user update to their profile.

Elastic Transcoder

Transcoding service which enables us to transcode videos. It has three main components:

1. Input Bucket: This is the bucket from which Transcoder accesses the file.

2. Pipeline: This pipeline is used to transcode the supplied video file based upon user

defined preset.

3. Output Bucket: This bucket is used to store transcoded videos.

During the time of creation of the pipeline we define the pipeline name, input bucket and the

output bucket with the IAM role for the transcoder. After creating the pipeline we create a new

job and define the input video file from the input bucket, also mention the preset we want to

convert it to with the output file name and execute this job. After the job is complete we can get

the transcoded file from the output bucket.

We can also customize the pre existing presets (aspect ratio, frame rate etc.) and create our

own custom preset to be used by transcoder.

Transcoder also support parallel transcoding and multiple format transcoding of a single file at

the same time in parallel(4 at most from console). It also supports visual watermarks, thumbnail

generation, notification using sns, encryption. Elastic transcoder also supports creation of

playlist by which we can add different presets of transcoded file and view the playlist later, can

also be used for adaptive bitrate streaming of videos. It does not support live media transcoding

it only supports file based transcoding. To stream the media we can either use AWS Cloudfront

for HTTP distribution of video or setup server like IIS which can also be used for video playback.

Video playback directly from S3 is not supported.

On-Demand Streaming

In the on-demand streaming case, your video content is stored in S3. Your viewers can choose

to watch it at any desired time, hence the name on-demand. A complete on-demand streaming

solution typically makes use of Amazon S3 for storage, the Amazon Elastic Transcoder for

video processing, and Amazon CloudFront for delivery.

Once uploaded, you may need to convert your video into the size, resolution, or format needed

by a particular video player or mobile device. The Amazon Elastic Transcoder will take care of

this for you. The Elastic Transcoder takes content from one S3 bucket, transcodes it per your

request, and stores the result in a second S3 bucket. Again, you can manage this process from

the AWS Management Console, the command line, or via the Elastic Transcoder APIs.

With your content safely stored and available in the formats required by your users, the next

step is global delivery. This is a job for Amazon CloudFront. This service gives you scalable

global delivery at a low price point. Because CloudFront caches content at the edges, your

customers will experience uninterrupted video playback with minimal delays due to buffering.

At this point our storyline forks, and you have two options. You can deliver the entire video file

to the device before playing it, or you can stream it to the device.

The first option is very easy to implement and is supported by just about every mobile device

and desktop. All you need to do is to put your content in an S3 bucket and create a CloudFront

distribution that points to the bucket. Your user’s video player will use CloudFront URLs

(accessible as part of the distribution) to request the video file. The request will be directed to

the best edge location, based on the user’s location. CloudFront will serve the video from its

cache, fetching it from the S3 bucket if it is not already cached. This option has a couple of

downsides. It makes inefficient use of your viewer’s bandwidth. If the user doesn’t bother to

watch the entire video, content would never be seen is still downloaded. Skipping ahead or fast-

forwarding also necessitates downloading of content that may never be seen.

The second option is almost always preferred. A newer family of video streaming protocols

including Apple’s HTTP Live Streaming (HLS), Microsoft’s Smooth Streaming, (SS) and

Adobe’sHTTP Dynamic Streaming (HDS) improve the user experience by delivering video as it

is being watched, generally fetching content a few seconds ahead of when it will be needed.

Playback starts more quickly, fast-forwarding is more efficient, and the overall user experience

is smoother. Mobile users appreciate this option because it doesn’t waste their bandwidth and

they get to see the desired content more quickly.

You will need to do a little more work in order to implement the second option. Use the Elastic

Transcoder to convert your video files to HLS format (the most widely supported streaming

protocol). This will split the video into short segments, and will also create a manifest file. The

player uses the manifest file to fetch and play the segments as needed.

EC2 Container Service

Amazon EC2 Container Service (Amazon ECS) is a highly scalable, fast, container

management service that makes it easy to run, stop, and manage Docker containers on a

cluster of Amazon EC2 instances. Amazon ECS lets you launch and stop container-enabled

applications with simple API calls, allows you to get the state of your cluster from a centralized

service, and gives you access to many familiar Amazon EC2 features.

You can use Amazon ECS to schedule the placement of containers across your cluster based

on your resource needs, isolation policies, and availability requirements. Amazon ECS

eliminates the need for you to operate your own cluster management and configuration

management systems or worry about scaling your management infrastructure.

Amazon ECS contains the following components:

1. Cluster: A logical grouping of container instances that you can place tasks on.

2. Container instance: An Amazon EC2 instance that is running the Amazon ECS agent

and has been registered into a cluster.

3. Task definition: A description of an application that contains one or more container

definitions. A task definition is required to run Docker containers in Amazon ECS.

4. Scheduler: The method used for placing tasks on container instances.

5. Service: An Amazon ECS service allows you to run and maintain a specified number of

instances of a task definition simultaneously.

6. Task: An instantiation of a task definition that is running on a container instance.

7. Container: A Linux container that was created as part of a task.

In summary, ECS enables us to run our application which is made up of different docker

containers each having its own set of properties and set of tasks that it can execute. Task

Definition defines these different containers and their properties which are used in our

application. We can execute as many tasks inside each container as we require. We can also

schedule the tasks to either run first time during execution or to run indefinitely in which case if

the task fails inside a container ECS will kill the failing task and re run the task again or will try to

re run until the threshold value. We can define the resources to be associated with each

container like cpu and memory during while defining the task definition. ECS has API for long

running applications and services In the create service option just reference the task definition

and the number of tasks you want to run.

Data Pipeline

AWS Data Pipeline is a web service that helps you reliably process and move data between

different AWS compute and storage services, as well as on-premise data sources, at specified

intervals. With AWS Data Pipeline, you can regularly access your data where it’s stored,

transform and process it at scale, and efficiently transfer the results to AWS services such as

Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon Elastic MapReduce (EMR).

AWS Data Pipeline helps you easily create complex data processing workloads that are fault

tolerant, repeatable, and highly available. You don’t have to worry about ensuring resource

availability, managing inter-task dependencies, retrying transient failures or timeouts in

individual tasks, or creating a failure notification system.

AWS Data Pipeline allows you to take advantage of a variety of features such as scheduling,

dependency tracking, and error handling. You can use activities and preconditions that AWS

provides and/or write your own custom ones. This means that you can configure an AWS Data

Pipeline to take actions like run Amazon EMR jobs, execute SQL queries directly against

databases, or execute custom applications running on Amazon EC2 or in your own datacenter.

AWS Data Pipeline handles:

● Your jobs' scheduling, execution, and retry logic

● Tracking the dependencies between your business logic, data sources, and previous

processing steps to ensure that your logic does not run until all of its dependencies

are met

● Sending any necessary failure notifications

● Creating and managing any temporary compute resources your jobs may require

(ex: create data pipeline, which uses an EMR cluster, read apache web access logs, reformat

the log output to amazon S3)

By using Data Pipeline we can create a template (using GUI interface) or use an existing

template and can define the data flow between the resources. We can easily define the source

of the data (S3, DynamoDB etc.) and process that data (copy, transform etc. using EC2, EMR,

redshift and many more activities) and can load the transformed data into storage services like

S3, DynamoDB etc. We can easily schedule these data driven jobs using Data Pipeline(min 15

minutes interval). Supports SNS and logging of activities being carried out.

Elasticsearch Service

Amazon Elasticsearch Service is a managed service that makes it easy to deploy, operate, and

scale Elasticsearch in the AWS Cloud. Elasticsearch is a popular open-source search and

analytics engine for use cases such as log analytics, real-time application monitoring, and

clickstream analytics.

Elasticsearch can be easily combined with logstash for data ingestion and kibana for data

visualization. The elk (elasticsearch, logstash, kibana) stack provides tool for real time analytics

and data visualization.

Two types of nodes in elasticsearch cluster:

1. Data nodes: these nodes are responsible for handling the data and performing any

query on the data.

2. Master node: responsible for holding cluster stability and managing.

By enabling zone awareness feature the nodes will be distributed between multi az for high

availability.

The index is a collection of data which in turn comprises of many shards (instance of lucence

with a portion of an index) and with each shard having a number of documents associated with

unique id. Each document comprises of collection of fields.

The shards are distributed between the nodes of the cluster for analysis. Each shard also has a

backup of it running inside a different instance known as replica for in case if the primary shard

goes down the replica can be promoted as primary.

Data can be loaded either by using elasticsearch api or using logstash for log based or event

based data or lambda by inputting data from sources like DynamoDB, S3 and Kinesis and

loading it into lambda and then into elasticsearch for analysis. Cloudwatch can also be

configured to push its logs to elasticsearch for analysis and viewing.

ElasticSearch supports automated daily snapshots which are retained for a duration of 14 days.

Manual snapshots also supported that are stored in S3 and can be taken based upon our own

time duration.

Kinesis

For streaming data ingestion and processing.

The producers continually push data to Streams and the consumers process the data in real

time. Consumers can store their results using an AWS service such as Amazon DynamoDB,

Amazon Redshift, or Amazon S3. It has 4 use case scenarios.

The first scenario is around making the task of "Ingest Transform Load" a bit easier by handling

typical issues like scale, reliability, and durability of the underlying infrastructure. This is part of

being a managed service. Please bear in mind that as with all systems, our design choices can

lead to situations that can be construed to be limitations for other use cases. An example being:

a single data blob that is put into an Amazon Kinesis stream can be no larger than 50KB in size

in this initial release. So even though the aggregate stream can be arbitrarily large, what it

means is that if in your model, let's say the data record of interest is a few MB large - then it will

have to be chunked up into smaller pieces <=50KB before being put into the Amazon Kinesis

stream.

The second category is metrics extraction, and real-time KPI generation. Once data is ingested

continuously, there is always the desire to start running some useful application that can

generate some insights from the data. This data could then drive a dashboard or even be

emitted continuously into another store such as DynamoDB or Cassandra or anything else.

The third category is to run full on analytic applications that are operating on streams of data.

This could be a fraud application, network security/ intrusion detection algorithm, 'next-best-

click' evaluation, or others. Frequently this application will be part of a larger application stack.

The final category is to create more complex stream processing applications. An initial Amazon

Kinesis application can emit data - after say pre-processing and filtering the primary stream -

and put data into another stream that then drives other processing applications. In this was a

DAG of streams and processing stages could be created.

Simple Workflow Service

It is a task coordination and state management service for cloud applications. Allows us to

define a step by step workflow of the application. Each one of these steps can be broken as

tasks or components. Each components has workers that execute these instructions. It is a

distributed service which allows us to break our application on the basis of tasks to be carried

out and each of these tasks can be executed using simple workflow service. Each of these

components can be scaled up or down as per requirements.

Domain is used to determine scope of workflows

Multiple workflows can be present inside a single domain

Workflows cannot communicate with other workflows in different domain

Worker carries out the tasks assigned to it by the decider and returns the result back to the

decider

Activity Worker - Worker that performs an activity/task that is part of the workflow.

Activity worker poll SWF for new tasks it has to perform

After receiving the task the worker will execute the task and report back to swf

workers can consist of EC2 to carry out the task or can also use human to perform the task

Activity Task - Task assigned to worker that tells what activity to perform like encode video or

check inventory

Decision Task - Tells the decider that state of workflow execution has changed. Allows decider

to determine what next activity has to be performed.

Config

AWS Config is a fully managed service that provides you with an AWS resource inventory,

configuration history, and configuration change notifications to enable security and governance.

With AWS Config you can discover existing AWS resources, export a complete inventory of

your AWS resources with all configuration details, and determine how a resource was

configured at any point in time. These capabilities enable compliance auditing, security analysis,

resource change tracking, and troubleshooting.

DirectConnect

Service that enables to connect to our cloud infrastructure from existing physical infrastructure

using a direct link. This reduces removes the traffic going to internet and provides security and

stable high speed connectivity.

A single direct connection can be configured with multiple virtual interfaces with each

connecting to different VPC’s in our AWS infrastructure.