Jim Scharf

General Manager, DynamoDB

Time : 10:10 – 10:50

Getting Started with Amazon

DynamoDB

Getting Started with Amazon DynamoDB

AGENDA

• Brief history of data processing

• Relational (SQL) vs. Non-relational (NoSQL)

• DynamoDB tables & indexes

• Scaling

• Integration and Search Capabilities

• Pricing and Free Tier

• Customer Use Cases

Timeline of Database Technology

Data Volume Since 2010

• 90% of stored data generated in

last 2 years

• 1 Terabyte of data in 2010 equals

6.5 Petabytes today

• Linear correlation between data

pressure and technical innovation

• No reason these trends will not

continue over time

Technology Adoption and the Hype Curve

Relational (SQL) vs.

Non-relational (NoSQL)

Amazon’s Path to DynamoDB

RDBMSDynamoDB

Relational vs. Non-relational Databases

Traditional SQL NoSQL

DB

Primary Secondary

Scale Up

DB

DB

DBDB

DB DB

Scale Out

Why NoSQL?

Optimized for storage Optimized for compute

Normalized/relational Denormalized/hierarchical

Ad hoc queries Instantiated views

Scale vertically Scale horizontally

Good for OLAP Built for OLTP at scale

SQL NoSQL

SQL vs. NoSQL Schema Design

NoSQL design optimizes for

Compute instead of storage

NoSQL Opportunity

SQL NoSQL

Evolution of Databases

Amazon DynamoDB

Fully Managed

Low Cost

Predictable Performance

Massively Scalable

Highly Available

Consistently Low Latency At Scale

PREDICTABLE

PERFORMANCE!!!

High Availability and Durability

WRITESReplicated continuously to 3 AZ’s

Persisted to disk (custom SSD)

READSStrongly or eventually consistent

No latency trade-off

Designed to

support

99.99%of availability

Built for high

Durability

How DynamoDB Scales

partitions1 .. N

table

DynamoDB automatically partitions data

• Partition key spreads data (and workload) across

partitions

• Automatically partitions as data grows and throughput

needs increase

Large number of unique hash keys+

Uniform distribution of workloadacross hash keys

High-scale Apps

Flexibility and Low Cost

Reads per

second

Writes per

second

table

• Customers can configure a table

for just a few RPS or for

hundreds of thousands of RPS

• Customers only pay for how

much they provision

• Provides maximum flexibility to

adjust expenditure based on the

workload

Fully managed service = Automated Operations

DB hosted on premise DB hosted on Amazon EC2

Fully managed service = Automated Operations

DB hosted on premise DynamoDB

DynamoDB Tables & Indexes

DynamoDB Table StructureTable

Items

Attributes

PartitionKey

SortKey

Mandatory

Key-value access pattern

Determines data distribution Optional

Model 1:N relationships

Enables rich query capabilities

All items for key==, <, >, >=, <=“begins with”“between”“contains”“in”sorted resultscountstop/bottom N values

00 55 A954 FFAA

Partition Keys

Partition Key uniquely identifies an item

Partition Key is used for building an unordered hash index

Allows table to be partitioned for scale

Id = 1

Name = Jim

Hash (1) = 7B

Id = 2

Name = Andy

Dept = Eng

Hash (2) = 48

Id = 3

Name = Kim

Dept = Ops

Hash (3) = CD

Key Space

Partition:Sort KeyPartition:Sort Key uses two attributes together to uniquely identify an Item

Within unordered hash index, data is arranged by the sort key

No limit on the number of items (∞) per partition key• Except if you have local secondary indexes

00:0 FF:∞

Hash (2) = 48

Customer# = 2

Order# = 10

Item = Pen

Customer# = 2

Order# = 11

Item = Shoes

Customer# = 1

Order# = 10

Item = Toy

Customer# = 1

Order# = 11

Item = Boots

Hash (1) = 7B

Customer# = 3

Order# = 10

Item = Book

Customer# = 3

Order# = 11

Item = Paper

Hash (3) = CD

55 A9:∞54:∞ AA

Partition 1 Partition 2 Partition 3

Partitions are three-way replicated

Id = 2

Name = Andy

Dept = Engg

Id = 3

Name = Kim

Dept = Ops

Id = 1

Name = Jim

Id = 2

Name = Andy

Dept = Engg

Id = 3

Name = Kim

Dept = Ops

Id = 1

Name = Jim

Id = 2

Name = Andy

Dept = Engg

Id = 3

Name = Kim

Dept = Ops

Id = 1

Name = Jim

Replica 1

Replica 2

Replica 3

Partition 1 Partition 2 Partition N

Local secondary index (LSI)

Alternate sort key attribute

Index is local to a partition key

A1(partition)

A3(sort)

A2(item key)

A1(partition)

A2(sort)

A3 A4 A5

LSIs A1(partition)

A4(sort)

A2(item key)

A3(projected)

Table

KEYS_ONLY

INCLUDE A3

A1(partition)

A5(sort)

A2(item key)

A3(projected)

A4(projected)

ALL

10 GB max per partition

key, i.e. LSIs limit the # of

range keys!

Global secondary index (GSI)Alternate partition and/or sort key

Index is across all partition keys

A1(partition)

A2 A3 A4 A5

GSIs A5(partition)

A4(sort)

A1(item key)

A3(projected)

Table

INCLUDE A3

A4(partition)

A5(sort)

A1(item key)

A2(projected)

A3(projected) ALL

A2(partition)

A1(itemkey) KEYS_ONLY

RCUs/WCUs

provisioned separately

for GSIs

Online indexing

How do GSI updates work?

Table

Primary

tablePrimary

tablePrimary

tablePrimary

table

Global

Secondary

Index

Client

2. Asynchronous

update (in progress)

If GSIs don’t have enough write capacity, table writes will be throttled!

LSI or GSI?

LSI can be modeled as a GSI

If data size in an item collection > 10 GB, use GSI

If eventual consistency is okay for your scenario, use

GSI!

Scaling

Scaling

Throughput

• Provision any amount of throughput to a table

Size

• Add any number of items to a table

• Max item size is 400 KB

• LSIs limit the number of range keys due to 10 GB limit

Scaling is achieved through partitioning

Throughput

Provisioned at the table level

• Write capacity units (WCUs) are measured in 1 KB per second

• Read capacity units (RCUs) are measured in 4 KB per second

• RCUs measure strictly consistent reads

• Eventually consistent reads cost 1/2 of consistent reads

Read and write throughput limits are independent

WCURCU

Partitioning math

In the future, these details might change…

Number of Partitions

By Capacity (Total RCU / 3000) + (Total WCU / 1000)

By Size Total Size / 10 GB

Total Partitions CEILING(MAX (Capacity, Size))

Partitioning example Table size = 8 GB, RCUs = 5000, WCUs = 500

RCUs per partition = 5000/3 = 1666.67WCUs per partition = 500/3 = 166.67Data/partition = 10/3 = 3.33 GB

RCUs and WCUs are uniformly

spread across partitions

Number of Partitions

By Capacity (5000 / 3000) + (500 / 1000) = 2.17

By Size 8 / 10 = 0.8

Total Partitions CEILING(MAX (2.17, 0.8)) = 3

To learn more, please attend:

Deep Dive on DynamoDB Room E450a, 11:45am-12:45pm

Rick Houlihan, Principal Solutions Architect

Integration Capabilities

DynamoDB Triggers

Implemented as AWS

Lambda functions

Your code scales

automatically

Java, Node.js, and Python

DynamoDB Streams

Stream of table updates

Asynchronous

Exactly once

Strictly ordered

24-hr lifetime per item

Integration Capabilities (cont’d)

• Elasticsearch integration

• Full-text queries

Add search to mobile apps

Monitor IoT sensor status codes

App telemetry pattern discovery

using regular expressions

• Fine-grained access control

via AWS IAM

• Table-, Item-, and attribute-

level access control

Connect to other AWS Data Stores

Customer Use Cases

Over 200 million usersOver 4 billion items stored

Millions of ads per month

Cross-device ad solutions

130+ million new users in 1 year

150+ million messages per month

Process requests in milliseconds High-performance ads

Statcast uses burst scalability

for many games on a single day

Flexibility for fast growth

Web clickstream insights

Specialty online & retail stores

Over 5 billion items

processed daily

About 200 million messages

processed daily

Cognitive training

Job-matching platform

5+ million registered users

Mobile game analytics

10M global users

Home security

Wearable and IoT

solutions

170,000 concurrent players

The Climate Corporation (TCC) Scales with Amazon DynamoDB

The Climate Corporation is a San Francisco-based

company that examines weather data to help farmers

optimize their decision-making.

The elasticity of DynamoDB

read/write Ops made

DynamoDB the fastest and

most efficient solution to

achieve our high ingest rateMohamed Ahmed

Director of Engineering,

Site Reliability Engineering & Data Analytics

The Climate Corporation

”

“ • Climate is digitizing agriculture, helping

farmers increase their yields and productivity

using scientific and mathematical models on

top of massive amounts of data

• Weather and Satellite imagery is one large

source of data used in TCC’s calculations

• TCC uses DynamoDB to ingest a burst of

data and satellite images retrieved from 3rd

parties before processing them

• TCC goes from few Read/Write Ops to

thousands each day to keep up with the

bursts of data written and read from it main

DynamoDB tables

Thank you!

Agenda

• Brief history of data processing

• Relational (SQL) vs. Non-relational (NoSQL)

• DynamoDB tables & indexes

• Scaling

• Int and Search Capabilities

• Pricing and Free Tier

• Customer Use Cases

Timeline of Database Technology

Data Volume Since 2010

• 90% of stored data generated in

last 2 years

• 1 Terabyte of data in 2010 equals

6.5 Petabytes today

• Linear correlation between data

pressure and technical innovation

• No reason these trends will not

continue over time

Technology Adoption and the Hype Curve

Relational (SQL) vs.

Non-relational (NoSQL)

Amazon’s Path to DynamoDB

RDBMSDynamoDB

Relational vs. Non-relational Databases

Traditional SQL NoSQL

DB

Primary Secondary

Scale Up

DB

DB

DBDB

DB DB

Scale Out

Why NoSQL?

Optimized for storage Optimized for compute

Normalized/relational Denormalized/hierarchical

Ad hoc queries Instantiated views

Scale vertically Scale horizontally

Good for OLAP Built for OLTP at scale

SQL NoSQL

SQL vs. NoSQL Schema Design

NoSQL design optimizes for

Compute instead of storage

NoSQL Opportunity

SQL NoSQL

Evolution of Databases

The Year of the Monkey

DynamoDB!

Amazon DynamoDB

Fully Managed

Low Cost

Predictable Performance

Massively Scalable

Highly Available

Consistently Low Latency At Scale

PREDICTABLE

PERFORMANCE!!!

High Availability and Durability

WRITESReplicated continuously to 3 AZ’s

Persisted to disk (custom SSD)

READSStrongly or eventually consistent

No latency trade-off

Designed to

support

99.99%of availability

Built for high

Durability

How DynamoDB Scales

partitions1 .. N

table

DynamoDB automatically partitions data

• Partition key spreads data (and workload) across

partitions

• Automatically partitions as data grows and throughput

needs increase

Large number of unique hash keys+

Uniform distribution of workloadacross hash keys

High-scale Apps

Flexibility and Low Cost

Reads per

second

Writes per

second

table

• Customers can configure a table

for just a few RPS or for

hundreds of thousands of RPS

• Customers only pay for how

much they provision

• Provides maximum flexibility to

adjust expenditure based on the

workload

Fully managed service = Automated Operations

DB hosted on premise DB hosted on Amazon EC2

Fully managed service = Automated Operations

DB hosted on premise DynamoDB

DynamoDB Tables & Indexes

DynamoDB Table StructureTable

Items

Attributes

PartitionKey

SortKey

Mandatory

Key-value access pattern

Determines data distribution Optional

Model 1:N relationships

Enables rich query capabilities

All items for key==, <, >, >=, <=“begins with”“between”“contains”“in”sorted resultscountstop/bottom N values

00 55 A954 FFAA

Partition Keys

Partition Key uniquely identifies an item

Partition Key is used for building an unordered hash index

Allows table to be partitioned for scale

Id = 1

Name = Jim

Hash (1) = 7B

Id = 2

Name = Andy

Dept = Eng

Hash (2) = 48

Id = 3

Name = Kim

Dept = Ops

Hash (3) = CD

Key Space

Partition:Sort KeyPartition:Sort Key uses two attributes together to uniquely identify an Item

Within unordered hash index, data is arranged by the sort key

No limit on the number of items (∞) per partition key• Except if you have local secondary indexes

00:0 FF:∞

Hash (2) = 48

Customer# = 2

Order# = 10

Item = Pen

Customer# = 2

Order# = 11

Item = Shoes

Customer# = 1

Order# = 10

Item = Toy

Customer# = 1

Order# = 11

Item = Boots

Hash (1) = 7B

Customer# = 3

Order# = 10

Item = Book

Customer# = 3

Order# = 11

Item = Paper

Hash (3) = CD

55 A9:∞54:∞ AA

Partition 1 Partition 2 Partition 3

Partitions are three-way replicated

Id = 2

Name = Andy

Dept = Engg

Id = 3

Name = Kim

Dept = Ops

Id = 1

Name = Jim

Id = 2

Name = Andy

Dept = Engg

Id = 3

Name = Kim

Dept = Ops

Id = 1

Name = Jim

Id = 2

Name = Andy

Dept = Engg

Id = 3

Name = Kim

Dept = Ops

Id = 1

Name = Jim

Replica 1

Replica 2

Replica 3

Partition 1 Partition 2 Partition N

Local secondary index (LSI)

Alternate sort key attribute

Index is local to a partition key

A1(partition)

A3(sort)

A2(item key)

A1(partition)

A2(sort)

A3 A4 A5

LSIs A1(partition)

A4(sort)

A2(item key)

A3(projected)

Table

KEYS_ONLY

INCLUDE A3

A1(partition)

A5(sort)

A2(item key)

A3(projected)

A4(projected)

ALL

10 GB max per partition

key, i.e. LSIs limit the # of

range keys!

Global secondary index (GSI)Alternate partition and/or sort key

Index is across all partition keys

A1(partition)

A2 A3 A4 A5

GSIs A5(partition)

A4(sort)

A1(item key)

A3(projected)

Table

INCLUDE A3

A4(partition)

A5(sort)

A1(item key)

A2(projected)

A3(projected) ALL

A2(partition)

A1(itemkey) KEYS_ONLY

RCUs/WCUs

provisioned separately

for GSIs

Online indexing

How do GSI updates work?

Table

Primary

tablePrimary

tablePrimary

tablePrimary

table

Global

Secondary

Index

Client

2. Asynchronous

update (in progress)

If GSIs don’t have enough write capacity, table writes will be throttled!

LSI or GSI?

LSI can be modeled as a GSI

If data size in an item collection > 10 GB, use GSI

If eventual consistency is okay for your scenario, use

GSI!

Scaling

Scaling

Throughput

• Provision any amount of throughput to a table

Size

• Add any number of items to a table

• Max item size is 400 KB

• LSIs limit the number of range keys due to 10 GB limit

Scaling is achieved through partitioning

Throughput

Provisioned at the table level

• Write capacity units (WCUs) are measured in 1 KB per second

• Read capacity units (RCUs) are measured in 4 KB per second

• RCUs measure strictly consistent reads

• Eventually consistent reads cost 1/2 of consistent reads

Read and write throughput limits are independent

WCURCU

Partitioning math

In the future, these details might change…

Number of Partitions

By Capacity (Total RCU / 3000) + (Total WCU / 1000)

By Size Total Size / 10 GB

Total Partitions CEILING(MAX (Capacity, Size))

Partitioning example Table size = 8 GB, RCUs = 5000, WCUs = 500

RCUs per partition = 5000/3 = 1666.67WCUs per partition = 500/3 = 166.67Data/partition = 10/3 = 3.33 GB

RCUs and WCUs are uniformly

spread across partitions

Number of Partitions

By Capacity (5000 / 3000) + (500 / 1000) = 2.17

By Size 8 / 10 = 0.8

Total Partitions CEILING(MAX (2.17, 0.8)) = 3

To learn more, please attend:

Deep Dive on DynamoDB Room E450a, 11:45am-12:45pm

Rick Houlihan, Principal Solutions Architect

Integration Capabilities

DynamoDB Triggers

Implemented as AWS

Lambda functions

Your code scales

automatically

Java, Node.js, and Python

DynamoDB Streams

Stream of table updates

Asynchronous

Exactly once

Strictly ordered

24-hr lifetime per item

Integration Capabilities (cont’d)

• Elasticsearch integration

• Full-text queries

Add search to mobile apps

Monitor IoT sensor status codes

App telemetry pattern discovery

using regular expressions

• Fine-grained access control

via AWS IAM

• Table-, Item-, and attribute-

level access control

Connect to other AWS Data Stores

Customer Use Cases

Over 200 million usersOver 4 billion items stored

Millions of ads per month

Cross-device ad solutions

130+ million new users in 1 year

150+ million messages per month

Process requests in milliseconds High-performance ads

Statcast uses burst scalability

for many games on a single day

Flexibility for fast growth

Web clickstream insights

Specialty online & retail stores

Over 5 billion items

processed daily

About 200 million messages

processed daily

Cognitive training

Job-matching platform

5+ million registered users

Mobile game analytics

10M global users

Home security

Wearable and IoT

solutions

170,000 concurrent players

The Climate Corporation (TCC) Scales with Amazon DynamoDB

The Climate Corporation is a San Francisco-based

company that examines weather data to help farmers

optimize their decision-making.

The elasticity of DynamoDB

read/write Ops made

DynamoDB the fastest and

most efficient solution to

achieve our high ingest rateMohamed Ahmed

Director of Engineering,

Site Reliability Engineering & Data Analytics

The Climate Corporation

”

“ • Climate is digitizing agriculture, helping

farmers increase their yields and productivity

using scientific and mathematical models on

top of massive amounts of data

• Weather and Satellite imagery is one large

source of data used in TCC’s calculations

• TCC uses DynamoDB to ingest a burst of

data and satellite images retrieved from 3rd

parties before processing them

• TCC goes from few Read/Write Ops to

thousands each day to keep up with the

bursts of data written and read from it main

DynamoDB tables

Thank you!