Embed Size (px)
Citation preview
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
28 June 2017
Deep Dive on Amazon Relational Database Service
Toby Knight - Manager, Solutions Architecture, AWS
What to expect
• Amazon RDS overview• Security• Customer story• Migrating to RDS• Metrics and monitoring• Scaling on RDS• Backups and snapshots• High availability
No infrastructure management
Scale up/downCost-effective
Instant provisioning
Application compatibility
Amazon Relational Database Service (Amazon RDS)
Amazon RDS engines
Commercial Open source Amazon Aurora
Amazon Aurora vs. MySQLFeature RDS Aurora RDS MySQL
Number of replicas Up to 15 Up to 5
Replication type Asynchronous (milliseconds) Asynchronous (seconds)Replication performance impact on primary
Low High
Replica can act as failover target Yes (no data loss) Yes (potentially minutes of loss)
Storage Up to 64 TB, auto growth Up to 6 TB, specify storage limit
Automated failover Yes, to replica Yes, to standby
User-defined replication delay No YesReplica support for different data or schema vs. primary
No Yes
Cross-region replication No Yes
Data cache survives Yes No
Trade-offs with a managed service
Fully managed host and OS• No access to the database host operating system• Limited ability to modify configuration that is managed on the
host operating system• No functions that rely on configuration from the host OS
Fully managed storage• Max storage limits
• SQL Server—4 TB• MySQL, MariaDB, PostgreSQL, Oracle—6 TB• Aurora—64 TB
• Growing your database is a process
Selected Amazon RDS customers
Security
Amazon Virtual Private Cloud (Amazon VPC)Securely control network configuration
Availability Zone
AWS Region
10.1.0.0/16
10.1.1.0/24Manage connectivity
AWS Direct Connect
VPN Connection
VPC Peering
Internet Gateway
Routing Rules
Security groupsDatabase IP firewall protection
Protocol Port Range SourceTCP 3306 172.31.0.0/16
TCP 3306 “Applicationsecurity group”
Corporate address admins
Application tier
Compliance
Singapore MTCS
27001/900127017/27018
MySQL, Oracle, PostgreSQL
• SOC 1, 2, and 3• ISO 27001/9001• ISO 27017/27018• PCI DSS• FedRamp• HIPAA BAA• UK government
programs• Singapore MTCS• Germany C5
Compliance
Aurora
• SOC 1, 2, and 3• ISO 27001/9001• ISO 27017/27018• PCI DSS• Germany C5
SQL Server
• SOC 1, 2, and 3• ISO 27001/9001• ISO
27017/27018• PCI DSS• UK government
programs• Singapore MTCS• Germany C5
SSL
Available for all six engines
Using SSL to encrypt a connection to a DB instance
mysql -h myinstance.c123xyz.rds-eu-west-1.amazonaws \--ssl-ca=rds-combined-ca-bundle.pem --ssl-verify-server-cert
At-rest encryption
• DB instance storage• Automated backups• Read Replicas• Snapshots
• Available for all six engines• No additional cost• Support compliance requirements
AWS KMS — RDS standard encryption
Two-tiered key hierarchy using envelope encryption• Unique data key encrypts customer data• AWS KMS master keys encrypt data keys
Benefits:• Limits risk of compromised data key• Better performance for encrypting large data• Easier to manage small number of master keys
than millions of data keys• Centralized access and audit of key activity
Data Key 1
Amazon S3 Object
Amazon EBS
Volume
Amazon Redshift Cluster
Data Key 2 Data Key 3 Data Key 4
CustomApplication
Customer MasterKey(s)
Enabling encryptionAWS Command Line Interface (AWS CLI)
aws rds create-db-instance --region eu-west-1 --db-instance-identifier sg-cli-test \--allocated-storage 20 --storage-encrypted \--db-instance-class db.m4.large --engine mysql \--master-username myawsuser --master-user-password myawsuser
aws rds create-db-instance --region eu-west-1 --db-instance-identifier sg-cli-test1 \--allocated-storage 20 \--storage-encrypted \--kms-key-id xxxxxxxxxxxxxxxxxx \--db-instance-class db.m4.large --engine mysql \--master-username myawsuser --master-user-password myawsuser
Amazon RDS + AWS KMS useful hints
• You can only encrypt on new database creation• Encryption cannot be removed• Master and Read Replica must be encrypted• Unencrypted snapshots cannot be restored to encrypted DB• Cannot restore MySQL to Aurora or Aurora to MySQL• You can now copy encrypted or unencrypted snapshots
across regions
IAM managed accessYou can use AWS Identity and Access Management (IAM) to control who can perform actions on RDS
Users and DBAApplications DBA and Ops
Your database RDS
Controlled with IAMControlled with database grants
Applications
IAM policies for RDSPolicies
"Action": ["rds:Describe*","rds:ListTagsForResource",
"ec2:DescribeAccountAttributes","ec2:DescribeAvailabilityZones","ec2:DescribeSecurityGroups","ec2:DescribeVpcs”,"cloudwatch:GetMetricStatistics",
"logs:DescribeLogStreams", "logs:GetLogEvents"
],"Effect": "Allow","Resource": "*"
"Action": ["rds:*","cloudwatch:DescribeAlarms",
"cloudwatch:GetMetricStatistics","ec2:DescribeAccountAttributes","ec2:DescribeAvailabilityZones","ec2:DescribeSecurityGroups","ec2:DescribeSubnets",
"ec2:DescribeVpcs","sns:ListSubscriptions","sns:ListTopics", "logs:DescribeLogStreams", "logs:GetLogEvents"
],"Effect": "Allow","Resource": "*"
Read Only
Full Access
NEW: IAM DB auth for MySQL and Aurora
You can now also use AWS Identity and Access Management (IAM) to control access to the database
Users and DBAApplications DBA and Ops
Your database RDS
Controlled with IAM
IAM DB Auth for MySQL and Amazon Aurora
1. Create RDS DB instance with IAM DB auth enabled2. Create the user in the DB3. Attach an IAM policy to the IAM user or role4. Get an authentication token5. Connect to DB using IAM DB auth
For more details:http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html
SportPursuit business overview
Founded in 2011Flash sale business – Unbeatable deals on sports and outdoor gear from the world’s leading sports brandsMission – Give access to and inspire sports enthusiasts to discover gear that they’ll fall in love with
3.5m+ Members – the UK’s largest private shopping club for sports & outdoor enthusiasts1000+ of the world’s best sports brands / 40% of which are on-UK7 Languages / 8 Currencies / Shipping to 40+ countries40% YoY growth
2-way proposition: brands & customers
- Marketing to huge audience of sports enthusiasts- Channel for clearing excess stock, without compromising brand identity
- Access to the best discounts on sports and outdoor gear from the world’s leading sports brands
Technical architecture 2011
e-commerce Platform
MySQL Database
EC2 EC2
Technical architecture 2017
ELB
ELB
Varnishe-commerce
Platform
API
Redis
Memcache / Redis
ERP Platform
RDS - PostgreSQL
RDS - MySQL
Application Server
Cloudfront
Analytics Platform
RDSMySQL
RedshiftElasticsearch
RDS estate statistics
~ 25 RDS instances (incl. replicas)Volume of RDS data
• Production environment: ~2TB (incl. replicas)• Entire estate: ~7TB
Throughput of data• e-commerce platform
• Peak: ~100 MB/S• Average: ~30 MB/S
Challenges before RDS
Manual database administration• Creating slave replicas• Solving replication errors• Version upgrades
Backups• Time consuming as amount of data increase
Refreshing staging environments• Time consuming: export -> anonymise -> create -> import
RDS positives
Reduction in manual database administration• Create many read replicas with ease• Painless version upgrades – Multi-AZ for production
Daily automated backups out of the box
Weekly refresh of staging environments• Automated snapshot -> anonymise -> restore• Feature request: anonymisation as a service - serverless
Lessons learned
Fix replication errors on slaves/replicas ASAP• Binary log disk usage
Don’t use MyISAM tables
Restrictions in sub-accounts• Can’t use automated snapshots • Database creation: automated -> manual -> restore
Sticking points
Adjusting to cloud philosophy• Destroy & re-create failing component: Pets / Cattle• Mindset change
• Health of the overall platform, not on a single element
Time taken to make changes / upgrades
Reboot required for option group changes
Upcoming projects
Aurora• Plan to benchmark against Aurora vs RDS MySQL• Test to ensure compatibility with e-commerce platform
BI / reporting• RedShift for enterprise data warehouse• Data Pipeline for data ingestion
BI / Reporting
Data warehouse• Redshift• Board BI tool• Python Django
Data Pipeline• RDS > Redshift
Python Django• API integration to 3rd
parties
Migrating to RDS
Historically, Migration = Cost, Time
Commercial data migration and replication software
Complex to setup and manage
Legacy schema objects, PL/SQL or T-SQL code
Application downtime
Database Migration – 2 Steps
Step 1: Schema Conversion Tool Overview
ü Move data to the same or different database engine
ü Keep your apps running during the migration
ü Start your first migration in 10 minutes or less
ü Replicate within, to, or from Amazon EC2 or RDS
AWS Database Migration Service
Customerpremises
Application Users
AWS
Internet
VPN
Start a replication instanceConnect to source and target databaseSelect tables, schemas, or databases
Let the AWS Database Migration Service create tables, load data, and keep them in syncSwitch applications over to the target at your convenience
Keep your apps running during the migration
Flexible migration approach
Replication instance
Source Target
Target
Target
Multiple targets
Replication instance
Source Target
Source
Source
Multiple sources
Source
L
Target
Replication instance instanceSelective
Metrics and monitoring
Accessing Amazon RDS metrics
Amazon RDS standard metrics45 MetricsChange Time Period
Dive Deeper
Create Alarms
Amazon RDS Enhanced Monitoring
Access to over 50 metrics in 7 categories:
• Memory, • I/O, • CPU, • File system, • Load, • Swap• Processes
Amazon RDS Event Notifications
• Get notified when events occur on your database instances
• 17 different event categories (availability, backup, configuration change and so on)
• Uses Amazon Simple Notification Service (Amazon SNS)
Scaling on RDS
Scale out with Read Replicas
Relieve pressure on your master node for supporting reads and writes.
Bring data close to your customer’s applications in different regions
Promote a Read Replica to a master for faster recovery in the event of disaster
Replicas within and cross-region• MySQL, MariaDB,
PostgreSQL• Aurora
Engines Needing Other Tools• Oracle • Microsoft SQL Server
Creating and promoting Read ReplicaRead Replica creation and promotion are accessed from the Instance Actions button in the RDS console
Creating and promoting Read Replicas with CLI create-db-instance-read-replica--db-instance-identifier <value>
--source-db-instance-identifier <value>
Creating and promoting Read Replicas With CLI create-db-instance-read-replica--db-instance-identifier <value>
--source-db-instance-identifier <value>[--db-instance-class <value>][--availability-zone <value>][--port <value>][--auto-minor-version-upgrade | --no-auto-minor-version-upgrade][--iops <value>][--option-group-name <value>][--publicly-accessible | --no-publicly-accessible][--tags <value>][--db-subnet-group-name <value>][--storage-type <value>][--copy-tags-to-snapshot | --no-copy-tags-to-snapshot][--monitoring-interval <value>][--monitoring-role-arn <value>][--kms-key-id <value>][--pre-signed-url <value>][--enable-iam-database-authentication | --no-enable-iam-database-authentication][--source-region <value>][--cli-input-json <value>][--generate-cli-skeleton <value>]
Scaling up and down
• Handle higher load or lower usage
• Control costs
Scaling Up and DownConsole
NEWStop & Start DB Instances
Stop your RDS database instance
aws rds stop-db-instance \--db-instance-identifier myDbInstance
…and start it again
aws rds start-db-instance \--db-instance-identifier myDbInstance
Backups and snapshots
RDS backups
MySQL, PostgreSQL, MariaDB, Oracle, SQL Server• Scheduled daily backup of entire instance• Archive database change logs• Up to 35 day retention for backups• I/O suspension as backup is initiated (but not with multi-AZ deployment)• Multiple copies in each AZ where you have instances for a deployment
Aurora• Automatic, continuous, incremental backups• Point-in-time restore• No impact on database performance• 35 day retention
RDS restore
• Restoring creates an entire new database instance• You define all the instance configuration just like a new
instance
Snapshots
• Full copies of your Amazon RDS database that are different from your scheduled backups
• Backed by Amazon S3• Typical use cases
• Resolve production issues• Nonproduction environments• Point-in-time restore• Final copy before terminating a database• Disaster recovery• Cross-region copy• Copy between accounts
High availability
Minimal deployment: Single AZ
Availability Zone
AWS Region
10.1.0.0/16
10.1.1.0/24
Amazon Elastic Block Store Volume
High availability: Multi-AZ
Availability Zone A
AWS Region
10.1.0.0/16
10.1.1.0/24
Availability Zone B
10.1.2.0/24
Replicated storage
Same instance type as master
High availability—Multi-AZ to DNS
dbinstancename.1234567890.us-west-2.rds.amazonaws.com:3006
Aurora high availability• Aurora cluster contains primary
node and up to 15 secondary nodes
• Failing database nodes are automatically detected and replaced
• Failing database processes are automatically detected and recycled
• Secondary nodes automatically promoted on persistent outage, no single point of failure
• Customer application can scale out read traffic across secondary nodes
AZ 1 AZ 3AZ 2
PrimaryNodePrimaryNodePrimaryNode
PrimaryNodePrimaryNode
SecondaryNode
PrimaryNodePrimaryNode
SecondaryNode
Amazon S3
Aurora-DNS Failover
AppRunningFailure Detection DNS Propagation
Recovery Recovery
DBFailure
MYSQL
AppRunning
Failure Detection DNS Propagation
Recovery
DBFailure
AURORA WITH MARIADB DRIVER
1 5 - 3 0 s e c
5 - 2 0 s e c
1 5 - 3 0 s e c
Driver benefits
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!Toby Knight
Manager, Solutions ArchitectureAmazon Web [email protected]
@tobywknight
