May, 2019 - Percona...© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ©...

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

May, 2019

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Open source engines Commercial engines

RDS platform

• Automatic fail-over

• Backup & recovery

• X-region replication

• Isolation & security

• Industry compliance

• Automated patching

• Advanced monitoring

• Routine maintenance

• Push-button scaling

Image credit: By Mackphillips - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=55946550

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

CLIENTS

RDS

PostgreSQL

Aurora

PostgreSQL

EBSAurora

Storage

Postgres 9.6/10/11 — same extensions

Backup/Recovery - PITR

High Availability & Durability

Secure – IAM Auth

Read Replicas

Cross Region Snapshots

Scale Compute – Online Scale Storage

Outbound Logical Replication

Cross Region Replication

11 - Soon

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Log-Based Storage

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Queued Work

Log Buffer

PostgreSQL Aurora PostgreSQL

Storage

A Queued Work

Storage

B C D E

0 0 0 0 0

A B C D E

2 2 1 0 1

A B C D E

4 3 4 2 4

A B C D E

6 5 6 3 5

A B C D E

Durability

Tracking

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Aurora

update t set y = 6;

Block in

Memory

t-v1t-v2t-v3

Aurora

Storage

t-v1t-v2t-v3

no

checkpoint

=

no FPW

Block in

Memory

PostgreSQL

t-v1t-v2t-v3

checkpoint

datafile

t-v1t-v2

Full

Block

t-v3

WAL

archive

4K

4K8K

update t set y = 6;

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Aurora

RW

Peer

Storage

Nodes

Coalesce

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

0

5,000

10,000

15,000

20,000

25,000

30,000

1 31 61 91 121 151 181 211 241 271

Inse

rts

Pe

r S

eco

nd

Minutes

Insert Workload—PostgreSQL 9.6

BASE 16GB Max WAL Aurora PostgreSQL

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

3,729

4,871

17,158

-

5,000

10,000

15,000

20,000T

PS

(2

Up

da

tes

pe

r T

ran

sacti

on

)

Update Workload—PostgreSQL 9.6

BASE 16GB Max WAL Aurora PostgreSQL

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

3 GiB Redo

Recovered in 19

seconds

10 GiB Redo

Recovered in 50

seconds

30 GiB Redo

Recovered in 123

seconds

0

20

40

60

80

100

120

140

160

0 20,000 40,000 60,000 80,000 100,000 120,000 140,000

Re

co

ve

ry T

ime

in

Se

co

nd

s (l

ess

is

be

tte

r)

Writes / Second (more is better)

R EC O VERY T IME FR O M C R A SH UNDER LO A D

Bubble size represents redo log, which must be recovered

As PostgreSQL

throughput goes up, so

does log size and crash

recovery time

Amazon Aurora has no redo.

Recovered in 3 seconds while

maintaining significantly greater

throughput.

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Base Architecture

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

RO

Application

RW

Application

RO

Application

Async

Invalidation

& Update

Async

Invalidation

& UpdateWrite log

recordsRead

blocks

RW

Automatic

Scalable

to 64TB

Availability zone 1 Availability zone 3Availability zone 2

Au

rora

Sto

rag

e

RORORORO

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Durability—4 of 6 Quorum

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

EBS EBS

Availability zone 1

Availability zone 2 Availability zone 3

EBS

COMMIT

Region

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

610

21

31

712

28

123

0

20

40

60

80

100

120

140

50 90 99.9 99.99

La

ten

cy (

ms)

Percentile

High Concurrency Sync Write Test

2 Node (4 copy) 3 Node (6 Copy)

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Aurora Storage

Availability zone 1

Availability zone 2 Availability zone 3

COMMIT

Region

Aurora Storage

Aurora Storage

Aurora Storage

Aurora Storage

Aurora Storage

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

0.00

100.00

200.00

300.00

400.00

500.00

600.00

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

resp

on

se t

ime

, m

s

minute

sysbench response time (p95), 30 GiB, 1024 clients

PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup)

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Replicas and Clones

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

PostgreSQL

RW

EBS Snapshot

PostgreSQL

RO

EBS

update

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Aurora

RW

Aurora

RO

update

Aurora Storage

update in

memory

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Async Replication

and applyPostgreSQL

Aurora

PostgreSQL

RW ROaccounts

tellers

branches

history

accounts

tellers

branches

history

Async Replication

and memory update

RW ROaccounts

tellers

branches

history

accounts

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

pgbench RW 8K tps on Primary – RO 200k tps on replica

backfill

pgbench_history

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

pgbench RW 8K tps on Primary – RO 200k tps on replica

backfill

pgbench_history

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

RO

Application

RW

Application

RW

Reporting

Application

Write log

recordsRead

blocks

Availability zone 1 Availability zone 3Availability zone 2

Au

rora

Sto

rag

e

Primary Storage

Clone Storage

Clone

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

0

5000

10000

15000

20000

25000

0 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57 60 63 66 69 72 75 78

Tra

nsa

ctio

ns

Pe

r S

eco

nd

(T

PS

)

Minutes

PGBench RW Scale 10K - Target Rate 20K TPS

Main Database

Clone Database

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Replication

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

PostgreSQL

instance

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Caching

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

488GB RAMPG + OS

processes

shared_buffer

s

25%

Linux

Pagecache

50+%

select of data –

check for block

in

shared_buffers

if not in

shared_buffers

load from

pagecache/disk

EBS

duplicate

buffers

PG + OS

processes

shared_buffer

s

75%

PostgreSQL Aurora PostgreSQL

Aurora

Storage

select of data –

check for block

in

shared_buffers

or load from

Aurora storage

Survivable

Cache

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

689,068

417,496

334,691

682,931

-

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

Tran

sact

ion

s P

er

Seco

nd

(tp

s)

pgbench read only - scale 22,000 - r4.16xlarge

Aurora 75% Cache PostgreSQL 25% Cache

1.6x 2.0x

18K read iops

no reads

heavy double

buffering

no double

buffering

no survivable

cache

Approx 350GB working

set

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

0 60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960 1020 1080 1140 1200

Tra

nsa

cti

on

s p

er

Se

co

nd

(T

PS

)

Seconds

PGBench 20X RO / 1X RW 160GB Cached - Failover at 600 Seconds

Baseline

340 seconds

32 seconds

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

RO

Application

RW

Application

RO

Application

Async

Invalidation

& Update

Availability zone 1 Availability zone 3Availability zone 2

Au

rora

Sto

rag

e

RORORORO

apg_ccm_enabled=on

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

0 60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960 1020 1080 1140 1200

Tra

nsa

cti

on

s p

er

Se

co

nd

(T

PS

)

Seconds

PGBench 20X RO / 1X RW 160GB Cached - Failover at 600 Seconds

Baseline

CCM Enabled

32 seconds

340 seconds

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Performance

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

• enable_bitmapscan=off

• enable_indexscan=off

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Use Baseline

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Vacuuming

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

4,000

5,000

6,000

7,000

8,000

9,000

10,000

1 61 121 181 241 301 361 421 481 541 601 661 721 781 841 901 961 1021 1081 1141 1201 1261

TP

S

Minutes

Updates—No Vacuum Running

transaction id wrap

around

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

PostgreSQL

Aurora PostgreSQLSubmit

Batch I/O

up to

256 blocks

402 seconds

163 seconds

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Thank you!