AWS re:Invent 2016: Continuous Compliance in the AWS Cloud for Regulated Life Sciences Applications within Merck (LFS302)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Scott Paddock – AWS Security Solutions Architect

Jeff Feist – Merck Cloud Architect

November 28, 2016

Continuous Compliance in the AWS Cloud

for Regulated Life Sciences Applications

LFS302

Agenda

• Overview of Continuous Compliance in Life Sciences

• How can you get to Continuous Compliance on AWS?

• Architectural Considerations for Continuous Compliance

• Tools for success

• Leveraging and expanding

• Compliance of Regulated Workloads @ Merck

• The Past

• The Present

• The Future

Objective: Continuous Compliance in the Cloud

• Regulations and guidelines for pharma, med devices, and medical

applications

• 3 Critical Domains

• Laboratory

• Clinical

• Manufacturing

• FDA Electronic Records and Electronic Signatures

• Consumer Safety through Data Integrity, Reproducibility, Traceability

If it isn’t documented, it didn’t happen.

Automation and Continuous Compliance

• Design Controls

• Computer System Validation

• Production Environment Controls

• Records and Reports

• Auditing

What sort of things does this include?

Continuous Compliance Questions

How do I architect for compliance in AWS?

How can I make architecting for compliance repeatable?

How can I validate that my architecture is compliant before

deployment?

How can I ensure continuous compliance in production?

Compliance – Architecting

• Dedicated Hosts/Instances (if ePHI)

• Eligible Services (if ePHI)

• Encryption at rest and in flight

• Logging & Monitoring

AWS BAA Eligible Services (as of 11/28/2016)

Wait, I don’t see VPC or KMS or…Amazon

EC2

Amazon

EMR

Amazon

GlacierAmazon

S3

Amazon

Redshift

Amazon

EBS

Elastic Load

Balancing

AWS HIPAA Eligible Services (prior to re:Invent)

Amazon

Snowball

Amazon

DynamoDB

Amazon

RDS (MySQL,

PostgreSQL,

Aurora, and

Oracle)

Transmit, Store, or Process PHI

AWS BAA Considerations

Amazon

EC2

Amazon

EMR

Amazon

GlacierAmazon

S3

Amazon

Redshift

Amazon

EBS

Elastic Load

Balancing

AWS HIPAA Eligible Services (prior to re:Invent)

Amazon

Snowball

Amazon

DynamoDB

Amazon

RDS (MySQL,

PostgreSQL,

Aurora, and

Oracle)

Amazon ECS Amazon

CloudWatch

AWS

CodeCommit

AWS

CodeDeploy

AWS

CodePipeline

SQS SNS

AWS Config

AWS

Device Farm

Other AWS Services

Not Applicable

or

Decouple

Transmit, Store, or Process PHI

Computer System Validation

Hardware Era Cloud EraVirtualization Era

Protocol-driven

manual activities

Procedure-driven

manual activities

Code-driven

automated activities

Validation of systems to ensure accuracy, reliability, consistent intended

performance, and the ability to discern invalid or altered records.

Compliance – Ongoing Evaluation

• Control mapping to a given framework

• Automate evidence collection

• Logs

• Configuration values

• Notification, roll-backs, and alarms

Building Blocks – Tools for success

Flow Logs

• Agentless

• Enable per ENI, per subnet, or per VPC

• Save data to S3

• Alarm on those metrics (more in a moment)


CloudWatch

• Monitor AWS resources

• View and graph stats

• Set alarms

• React to changes


CloudTrail

• Logs API calls

• Enable globally for all AWS Regions

• Includes KMS key requests

• Archive & Forward


Config

• Record configuration changes continuously

• Timeline view of resource changes

• Archive & Compare

• With Config Rules:

• Enforce Best Practices

• Automatically roll-back unwanted changes

Putting them together

• Enable, collect, and secure log data

• Feed into your favorite aggregator/SIEM

• Splunk, Sumologic, etc.

• ElasticSearch/Logstache/Kibana (“ELK”)

• CloudWatch Logs / CloudWatch Alarms

• Establish baselines and create alerts

Logs→ Metrics→ Alerts/Actions

AWS

Config

CloudWatch /

CloudWatch LogsCloudWatch

alarms

AWS

CloudTrail

Amazon EC2

OS logs

Amazon

Flow Logs

Amazon SNS

email notification

HTTP/S

notification

SMS notifications

Mobile push

notificationsAnd more…

Or your preferred SIEM / Log

aggregator

Config – Where’s the Evidence?

• Many compliance audits require access

to the state of your systems at arbitrary

times

• A complete inventory of all resources

and their configuration attributes at the

AWS API level is available for any time

Config – Timeline View

Expanding the Scope

We’ve seen tools aligned to deployed applications.

What about Development?

Development Environment Controls

• Automate deployment to

production.

• Establish and monitor

control parameters

programmatically.

• Record and justify

deviations from automated

processes.

• Let security run incident

response program.

• Feed end user requests

into the design controls.

CC end usersCC engineers

Records and Reports

• Logs in CloudTrail and

CloudWatch

• CloudFormation Templates and

custom code

• Application validation records

• Virtual infrastructure

qualification records

• Life Sciences end user account

info & training records

• Life Sciences engineer account

info & training records

• AWS technical support cases

• Automated Logging Generate

• Review

• Analyze

• Act, Present, or SubmitUse

• Keep originals or true copies

• Define retention period & locations

• Ensure protection & retrievabilityRetain

• Record destruction authorizationDispose

Auditing

Review your…

• AWS account credentials

• IAM users

• IAM groups

• IAM roles

• IAM providers for SAML and

OpenID Connect

• Mobile apps

• Amazon EC2 security

configurations

• Resource-based policies in

other services like S3

• Monitor activity in your AWS

account

• Training records

In Summary

• Infrastructure as Code is fundamentally transforming IT

compliance in Life Sciences systems

• Automation and shorter change cycles require rethinking

the traditional manual Life Sciences compliance activities

• Cloud skills are the new prerequisites for hiring and

development of Life Sciences system engineers and

QA/RA

• Life Sciences organizations are achieving more control

with less effort than ever before

An Example of an Organization

that has done this for Real!

Jeff Feist – Merck Cloud Architect

We are a global

healthcare company

with a 125-year history

of working to make a

difference in global

health.

HEADQUARTERS

Kenilworth, NJ, U.S.A.

operating in more than

60 countries

Merck & Co., Inc.

is our legal name and is listed on the New

York Stock Exchange under the

symbol "MRK."

EMPLOYEES

approximately 68,000

worldwide (as of 5/5/16)2015 REVENUES

$39.5 billion; 56% of sales come

from outside the United States

2015 R&D EXPENSE

$6.7 billion; 19 drug candidates in

late-stage development

BUSINESSES

Pharmaceuticals, Vaccines,

Biologics and Animal Health

Agenda

• Compliance of Regulated Workloads @ Merck• The Past

• The Present

• The Future

The Past

• Focus on security

• Manual checks

• Best guess at standard configuration

• Environment drift

• Limited monitoring / alerting for authorized access or

activities

• Usage was growing faster than our team

Merck AWS Account Models

• Proof of Concept

• Short term / temporary workloads

• Self service support

• Traditional

• “Data center” like hosting

• Federated support model

• Modern

• Cloud native / cloud designed architectures

• Self service support

Goals – Where we wanted to go

• Historical auditability of resource lifecycle

• Near real-time alerting for events that warrant additional investigation

• Monitor environment health

• Tight control over API permissions

• Traceability back to user for actions

• Automated validation that the environment is configured as intended

• Scheduled & on demand

• Archived for historical auditing

Goals – Where we wanted to go

• Historical auditability of resource lifecycle

• Near real-time alerting for events that warrant additional investigation

• Monitor environment health

• Tight control over API permissions

• Traceability back to user for actions

• Automated validation that the environment is configured as intended



The Present – No Dev Solutions

• Configuration only. Zero development needed.

Historical auditability of resource lifecycle

• Leverage AWS Config for EC2, RDS, IAM

• S3 Versioning (MFA Delete)


Near real-time alerting

• Privileged account usage

• Events that warrant additional investigation

• Lets us know when a specific event happened (or tried to happen) in the

environment

AWS

CloudTrail

Amazon

CloudWatchAmazon

SNS

AWS IAM Alarm

Metric

Filter


Monitor environment health

• Review weekly Trusted Advisor emails

• For critical checks leverage Support API => email

• Response depends on specific alert

• Yellow isn’t necessarily a bad thing!

The Present – Minor Dev Solutions

Tight control over API permissions

AWS

Lambda

AWS IAM –

Account 1

Scheduled

EventPolicy

Restrictions AWS STS

AWS IAM –

Account N

The Present – Minor Dev Solutions

Traceability back to user for actions

• Custom Federation Broker

The Continuous Compliance Checker

Automated validation that the environment is

configured according to our approved requirements



Scope

• Account Settings

• Monitoring Configurations

• Administrative Permissions


Runs daily to ensure each Account matches our standard

configuration

AWS cloud

Account 1

AWS cloud

Account NExpected

Results

AWS API Calls

Actual

Results

1

23

4

Expected

+ Actual

+ Eval

5

AWS cloud

Account 1

AWS cloud

Account NExpected

Results

AWS API Calls

Actual

Results

1

23

4

Expected

+ Actual

+ Eval

5

How it Runs

Assumes read-only role in

each target account

Executes multiple API calls

against each in scope service

Actual response is compared

to expected for success or

failure

Results are archived in S3

If any failures, are alerted to

our team for remediation

1

2

3

4

5


Results Comparison

• Expected results file contain expected API response

• Actual results are returned from API call

• Direct comparison of actual to expected to determine if they

match

Results Comparison

• Expected results file contain expected API response

• Actual results are returned from API call

• Direct comparison of actual to expected to determine if they

match

Demonstrating Control

• Requirements specification to match template

• Change request with approvals for updating template

• Pre/post approved test plan verifying script behaved as expected

Output

Alert Response

• Alerts are no good if they are ignored. How do we respond when an alert is received?

• Investigation, Documentation, Close out

• The response required for each alert is documented in our job aids

• Each alert is reviewed by one or more team members and an action is taken:

• Confirmation of non-issue

• Troubleshooting and resolution

• After action is taken, the alert is closed out including the details of the non-action or action

• Closeout includes documenting and archiving historical details & actions taken into team

repository or enterprise ticketing system

• Need to show traceability of action for lifecycle for alert

AlertClose

outDocument

Investigation

Where We’re At Today

• Can comfortably build new AWS Accounts and know they will stay in sync

with our standards

• Eliminated Account drift in our accounts

• Confidently show tight controls on the environment

• Reduced security risks in the environment

• More visibility into Account usage

• Allows us to focus on growing instead of maintaining

The Future

• Eliminate servers. Serverless for everything!

• Expand compliance checks to detect based on

CloudTrail usage

• Trigger a compliance check after an event has occurred

• Correct issues instead of just alerting on them

• Need to carefully evaluate this approach

Thank you!

Feel free to email with any questions:

Scott Paddock: [email protected]

Jeff Feist: [email protected]

mailto:[email protected]

mailto:[email protected]

Elasticsearch, Kibana and CloudWatch Logs integration

• Push CloudTrail to CloudWatch Logs:

http://amzn.to/2cacyC3

• Push CloudWatch Logs to Elasticsearch:

http://amzn.to/2efmWdN

• Put a Kibana front-end on it:

• http://amzn.to/2dIZjIz

Resources – ELK Tutorials/Documentation

http://amzn.to/2cacyC3

http://amzn.to/2efmWdN

http://amzn.to/2dIZjIz

Technology

AWS re:Invent 2016: Continuous Compliance in the AWS Cloud for Regulated Life Sciences Applications within Merck (LFS302)