Building a DevSecOps Culture - ETDA...1. Test security as early as possible to accelerate feedback. 2. Prioritize preventive security controls to stop bad things from happening. 3

© 2019, Amazon Web Services, Inc. or its Affiliates.

Tim Anderson, Sr. Technical Industry Specialist, AWS Security

Building a DevSecOps Culture


Outcomes

• Blueprint for building DevSecOps operating model in your organization

• Understand the security practitioners’ point of view and embrace it to drive innovation

• Identifying your current operating characteristics in your organization and using that to drive a strategy for DevSecOps


What Is Culture?

• Culture is the “software of the mind.” It is the core logic that organizes

people’s behavior

• The culture reflects the lessons learned that are important enough to pass

on to the next generation

• Values, beliefs, and practices that have been developed and reinforced

over time

Culture is “the Way We Do Things Around Here.”


Competing Forces

Business

Development Operations

Build it faster Keep it stable

Security

Make it secure


Corporate Culture& Values

OperatingModel

OrganizationalStructure

Decision Making& Governance

LeadershipDevelopment

Roles, Skills, and Career Paths Compensation &

Incentives

70% of customer challenges are non-technical.


The Current State of Affairs

Lack of business agility

•Slow to onboard new customers

•Outpaced by disruptors

•Rogue dev projects

Security lacks agility

•Slow threat assessments

•Can’t patch fast enough

•Reactive security posture

•Rogue dev projects

Risk of change to the business

•Regulatory/compliance constraints

• Impact to the workforce

•Cost overruns

•Scaling security

• Impact to governance strategy and processes

© 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates.

Building the business case for DevSecOps transformation


What is DevSecOps?

DevSecOps is the combination of cultural philosophies, practices, and tools that exploits the advances made in IT automation to achieve a state of production immutability, frequent delivery of business value, and automated enforcement of security policy.

DevSecOps is achieved by integrating and automating the enforcement of preventive, detective, and responsive security controls into the pipeline.

Security

OperationsDevelopment


Tenets of DevSecOps

1. Test security as early as possible to accelerate feedback.

2. Prioritize preventive security controls to stop bad things from happening.

3. When deploying a detective security control, ensure it has a complementary responsive security control to do something about it.

4. Automate, automate, automate.


The Benefits

Fast time to market or time to value for internal

use products

Less waste from producing unneeded

capabilities

Less waste from producing capabilities that do not accomplish

objectives

Less waste in processes (both inside and outside

of IT)

Reduced risk Increased innovationBetter operational controls through

automation


Traditional IT vs. Full Stack Engineering

Traditional ITactivity-based teams

Full-Stack Engineeringoutcome-based teams

Ap

plicatio

n D

eve

lop

me

nt

Ap

plicatio

n Q

A &

Testing

Ap

plicatio

n R

un

time

Mid

dlew

are & In

tegratio

n

Datab

ase

Netw

ork

Storage

Co

mp

ute

Ap

plicatio

n D

eve

lop

me

nt

Ap

plicatio

n Q

A &

Testing

Ap

plicatio

n R

un

time

Mid

dlew

are & In

tegratio

n

Datab

ase

Netw

ork

Storage

Co

mp

ute

Full-Stack Engineering Team

Infrastructure

Operations

Security

Finance

Handoffs

Wait Time

CompetingPriorities

ResourceConstraints

Challenges New Burdens


Cost is optimized by distributing accountability across pools of resources

Biz Case & Reqs

Business

Creative & Functional

Design

Finance & PMO

Prioritization

Software Development

Engineering

Integration & Perf.

QA & Testing

Deploy & Manage

Infra & Ops

Policy & Compliance

Security

Idea

Value

Defects

Defects

Defects

Defects

Defects

Wait

Wait

Wait

Wait

Wait

Wait

Wait

Wait

Defects

Each Step Delays Time-to-ValueDefects passed downstream are often discovered late in the delivery cycle and have to be revisited.

In the process, pervasive handoffs, bottlenecks, and defects are created.


Identify desired attitudes

and behaviors for

successful

cloud adoption

Communicate attitudes

and behaviors

Align explicit and

implicit reward systems

Align hiring,

training, and

incentive practices

How to Influence Cultural Change?


Be aware of top 5 pitfalls

2. Poor Communications

3. Insufficient Resource Allocation

4. Undefined KPI’s and Outcomes

5. Timing

1. Lack of Executive Sponsorship


Building a security focused CCOE to drive cultural change - DevSecOps 1.0 Team


Commission a CCOE

• Cross-functional / Hands-on

• Product focused

• Co-located and dedicated

• Empowered

• A change agent

• Creates roadmap

• Establishes standards

• Partners with early-adopters

A two-pizza, empowered, and accountable team that owns the cloud strategy, establishing

the cloud service, and helping the business / dev teams migrate their first few applications


Driving Change with The CCOE

Building reusable patterns / Product focused

Ingraining security with every team member

Visibility of team operations

Continuous improvement – feedback cycle and actions

Look to simplify


Structure of Product Teams

How do we structure these product teams?

ProductManagement

ProductDesign

ProductEng. & Test

ProductOperations

Viability

Desirability

Feasibility

Operability


Cloud Center of Excellence - Roles

IT | Product Owners

Singularly accountable for platform vision and its

viability from a business perspective

Workforce preparedness, communications, training,

resource and career management plans

IT | Engineering Manager

Career Development

Accountable for team management, team-member development, and overall

HR responsibilities

IT | Engineers

Feasibility & Operability

Accountable for product technical feasibility and delivery across platform, operations, and security; engineering,

testing and operations

Viability

Fin | Financial Analysts HR | OCM Specialists

IT | Scrum Master

Productivity

Facilitates Agile process and ensures forward progress towards business

outcomes by the product team

Financial budgeting, tracking and reporting; show-

back/charge-backs and cost optimization

Translates business objectives and governance requirements

to platform architecture

IT | Cloud Architects

Desirability

ProductManagement

ProductDesign

ProductEng. & Test

ProductOperations

Viability

Desirability

Feasibility

Operability


Infrastructure Operations Security

Shared Infrastructure

Deployment Assets

Operate & Manage

Build, Test, & Deploy

Detect & Respond

Define & Enforce

PrimitivesAMIs, Container Images, CloudFormation

Configuration ManagementCookbooks, Playbooks, Manifests

Enterprise “Stacks”CloudFormation / Terraform

WAN | LAN | DNSVPN / DirectConnect | VPC

Accounts, IAM, & SSOAWS Organizations, Accounts, & IAM

3rd Party PaaSMesosphere, Cloud Foundry, Kubernetes

Continuous Integration & DeliveryJenkins, Bamboo, AWS CodePipeline

Configuration ManagementChef Server, Ansible Tower, Puppet Master

Source Code & Artifact RepositoriesGitHub, TFS, Artifactory, Nexus

TelemetryMetric & Log Collection

InsightsAlerts, Escalations, and Dashboards

Inventory | Capacity | FinanceReporting & Management

IAM & Policy ManagementAWS IAM, Cloud Custodian, AWS Config

Network SecurityWAF, IDS, DLP, Outbound Proxy

Secrets & EncryptionAWS Cert Manager & KMS | Vault

Threat & Vulnerability Management

Security Information & Event Mgmt.

Incident Response & Forensics

Service Presentation & OrchestrationAWS Service Catalog

Products are continuously “dogfooded” internally by the CCOE as part of providing overall service to the Enterprise

What products does the team provide?


● Adaptive Home Page

Experiences

Two kinds of products:

Services

• Search• Cart• Account• Item• Advertising• Promotions• Digital Asset• Others...

Navigation

Promotions

Customer Profile

Promotion Content Cartridge

Recommendations

Ad

apti

ve H

om

e P

age

SearchCart

Digital Asset Digital Asset Digital Asset Digital Asset Digital Asset

Cart

Account

Account

Search

Promotions

AdvertisingItem Item Item Item Item Item

Promo Promo Promo Promo Promo

Item Item Item Item Item

Digital Asset

Digital Asset

Digital AssetDigital Asset

Digital Asset

Digital Asset

Digital Asset

How do you re-envision the world as products?


Navigation

Promotions

Customer Profile

Promotion Content Cartridge

Recommendations

Ad

apti

ve H

om

e P

age

SearchCart

Digital Asset Digital Asset Digital Asset Digital Asset Digital Asset

Cart

Account

Account

Search

Promotions

AdvertisingItem Item Item Item Item Item

Promo Promo Promo Promo Promo

Item Item Item Item Item

Digital Asset

Digital Asset

Digital AssetDigital Asset

Digital Asset

Digital Asset

Digital Asset

Adaptive Home Page

Search

Account Cart

Item Digital Asset

Advertising Promotions

Products are delivered by stable “product teams”


Transforming security to be a business enabler


Security Blind Spots

Can’t scaleLack of rigorDisparate sources

</>

#

@

+=28.25

If(


Get Humans Away from Your Data


Instead of “No”, security should say “how can we do this?”

• Change from gating to guardrails

• Establish norms for security hygiene and set high quality standards

• Craft policy enabling teams to operate freely within the determined constraints – stepping towards continuous authorization.

• Consistently communicate the connection between security and business objectives


Giving security confidence – Proving Assurance

Threat modeling

Feed security cases to the Dev team - work it like high priority defects

Address separation of duties concerns

Adopting zero known defect approach

• Rigorous testing in each environment

• Peer review - Each technologist should be thinking about possible defects and possible security vulnerabilities. Code should always be reviewed by a peer, who should also be looking for vulnerabilities

Continuously vet/audit security in dev and prod


Source Build Test Production

• Version Control

• Branching

• Code Review

• Compilation

• Unit Tests

• Static Analysis

• Packaging

• Integration Tests

• Load Tests

• Security Tests

• Acceptance Tests

• Deployment

• Monitoring

• Measuring

• Validation


Continuous Delivery of Cloud Services

Pull Approved Platform Artifacts

from Shared Repos

2Extend Approved Artifacts for App

Stacks

3Build & Test

Cloud Services

3

Integrate & Deploy Cloud

Services

4

PopulateRevise

PrioritizeBacklog

2

Publish Artifacts & Documentation for

Cloud Services

6Operate Cloud

Services

5

Lifecycle Management of

Application Stacks

6Operate

Application Stacks

5Build, Test, and

Deploy App Stacks

4

1

Continuous Delivery of Application Services

1

Service Catalog or Shared

Repositories

Team Interaction and Workflow

On-Boarding, Coaching, &

Product Feedback

Threat Modelling

Secure/Hardened Environments

Security Focused Code Review

Security Focused Code Review

Automated Security Testing

Automated Security Testing


432

What does it look like?

Developers

AMI

Lambda Function

1. Scan for creds

2. Static analysis

3. Logic / Library scan

4. Smoke test

5. Deploy into repo5

Logs Logs Logs Logs

Logs

AWS Cloud

1


CI/CD is a MUST!

•Commit frequently

•Builds on every commit

•Build once in a given execution flow

•Deploy to a running environment for further testing

Clean room

Everything that is code goes into a repository

•If its not in a repository, it doesn’t go into production environments!

Start with continuous delivery (“gated” promotion) and build up to continuous deployment once evidence of a high-level of excellence in testing is clear

Deploy to canaries, test, deploy to an AZ, test, deploy to a Region, test

General best practices used by Amazon developers

Code Reviews are one of the best mechanisms for “good” code:

•Does this code look clean and can someone else understand it?

•Is the design of it meeting the expectations of its needs?

•Are there better/easier ways to do this same thing?

Style checkers

•Will someone else in the company be able to update/fix/maintain this code?

Auto-rollbacks can be the quickest recovery mechanism after failure

•Rollback first, then debug what went wrong with logs/graphs/etc.

Thorough dashboards

•What is happening now?

•What ”normal” look like?

•What do I do if this graph looks wrong/an alarm has been triggered?

•What events can I correlate with a move in a graph?


Consistency Breeds Trust

CI/CD

Normalize processes and

tech stack

normal vs. abnormal behavior

Maintain disciplined ITSM

use

Configuration management

Release management


A deeper look at enabling people

Everybody is a Security Engineer

Pair Programming Works

Tooled Correctly for Continually Learning

Certification Rules

Recruit for Alignment with our Tenets

Recognize what Motivates Engineers and Developers


Using CI/CD to Drive Cultural Security Milestones

Deeply understand your SDLC

• Sit security team members with a development team for as many days as you can.

Catalog the controls

• Gain visibility into CI/CD pipelines. That’s where change management and control happens now.

Document every instance of human interaction

• With systems that process data. Let engineering & operations teams drive this goal.

Reduce human access

• Set and achieve a goal to reduce human access to systems that process sensitive data by 80%

Set a goal to deploy workloads from source.

• Catalog the % of workloads that are built on automation vs. those built with manual steps


Mechanisms for building a security culture

1. Consistently communicate the connection between security and mission objectives.

2. Set up practices to “build security in,” and fast feedback mechanisms to correct mistakes.

3. Establish norms for security hygiene and set high quality standards.

4. Adopt a zero-known-defect approach.

5. Continuously vet security, both in development and production.



Thank [email protected]

Documents

Building a DevSecOps Culture - ETDA...1. Test security as early as possible to accelerate feedback. 2. Prioritize preventive security controls to stop bad things from happening. 3