Cutting Through The Red Tape

Cutting through the Red Tape – FINRA’s Ongoing Move to Continuous Delivery

Michael Dillon, VP of Development Services

Kym Weiland, Director of Enterprise Release

July 2015

2

1 Who We Are

Copyright 2015 FINRA

Who We Are - FINRA

Financial Industry Regulatory Authority Largest independent regulator for all securities

firms doing business in the U.S. ~4,000 brokerage firms ~163,500 branch offices ~637,000 registered securities representatives

3

Arial Body Copy

Providing independent, vigorous

regulation

Educating & informing investors

Inviting active industry involvement &

input

Actively supportingfirms’ compliance efforts

Our Mission:Investor Protection. Market Integrity.


What We Do—Regulate the Markets

Every day, we watch over nearly 6 billion shares traded in U.S. listed equities markets—using technology powerful enough to detect potential abuses.

In fact, FINRA handles more “big data” on a daily basis than the Library of Congress or Visa®—processing approximately 6 terabytes of data and up to 50 billion transactions every day to build a complete, holistic picture of market trading in the U.S.

4

5

2 Current State


Rate of Change

175 Applications

Continuous Build in Development >49K builds YTD (100K Projected)

Deployments YTD Lower Environment = >38K QA = >12K Production = >2K

Highly diverse technology platforms FINRA Data Centers

– Solaris, Linux, Windows, Data Appliances, Oracle, SQL SVR, Hadoop Amazon Web Services

– Linux, EC2, S3, RDS, Redshift, EMR

6


AutomationJenkins

XLDeploy

ControlJIRA

StashSubversion

FINRA’s Delivery Ecosystem

7

DEVDEVCheck In/OutCheck In/Out

Subversion, Stash, TFS,IDEs

Extract/BuildExtract/Build

Ant, XML,PackageMake,ArtifactoryPro,

Maven

LabelLabel

Stash, Subversion,TFS

Unit TestUnit Test

JUnit/Clover,MSBuild/MSCover

App DeployApp Deploy

XLDeploy

SmokeTest

SmokeTest

JTAF/MSL

INTINT

QCQC

PRODPROD

5

Datamart

Traceability Per Drop- Build Records

- Labels, Build Time, KLOC, Test Results/Coverage

- Deploy Records- Build Record + Target Environment

App MonitoringApp Monitoring

Nagios

ComplianceCompliance

Blackduck, ArtifactoryPro

46

1

2

8

3

FINRA Data Centers

7


Journey to get there – internal reflection

What are the largest technology roadblocks today for delivery and what are solutions to streamline them?

What innovations are required to make a technology truly services oriented?

What should the end-to-end delivery stream look like and how do we automate it into reality?

How do we measure and monitor success along the way?

Is there a balance that can be achieved between fully integrated Devops (a.k.a developers with pagers) and Production Centralization?

How to define Devops with your corporate environment?

8


Hurdles for Adoption

Complexity of Organizational Structure Multiple teams and roles with disparate ownership in the process adds to longer

than necessary time to solve problems Multiple handoffs hinder a holistic view on streamlining automation and support Impacts desired communication and collaboration

Risk Sensitive Culture High assurance demands to avoid business impact, reputation risk, etc. High process and orchestration resulting in tight weekend change windows

Self-Enabled vs. Centralized Operational Support Differences in application architectures, level of automation and maturity of the

teams make it difficult to migrate from centralized orchestration

9


Opportunities

Cloud Migration Decision to migrate to AWS forced a re-think of strategy, roles and approach Natural enabler of automation across the delivery lifecycle in addition to promoting

self-enablement Moving towards a loosely coupled architecture

Mature Support Automation Considerable and sustained experience in automated build, test and deployment Continual focus on streamlining and improving the automation eco-system

Strong Executive and Organizational Support Our CIO has made this as a priority effort The development organization has a lot of energy and excitement directed towards

meaningful change A continuing investment towards an agile and collaborative culture

10


Current State Workflow for AWS Deployments

11

AW

SO

n P

rem

ises

Configure Build Env

MergeTrigger Build

Execute Build

Pre-Deploy Tests

Multiple Iterations

Development & QC

Provision Environment

Deploy

Promote

Orchestrate

Approval Decision

Post-Deploy Tests

Push to

XLDStage

Deploy

Smoke Test

Automated

Manual

Production


Impact on Roles

Release Merging of “Release Implementation” and “Application Operations” roles Shifting from project-based releases across Dev/QA/Prod to orchestrating

organizational dependencies in Production

Test Hiring focus on developers vs non-technical business testers Continual focus on streamlining and improving our automation abilities

CM/Build Enabling distributed CM/Build Engineers with devops privileges and responsibilities

Traditional Infrastructure/Ops Gradually being re-positioned by automation gains to a provider of core services

12


AutomationJenkins

XLDeployPuppet

ControlJIRA

StashSubversion

FINRA’s Delivery Ecosystem - Revised

13

DEVDEVCheck In/OutCheck In/Out

Subversion, Stash, TFS,IDEs

Extract/BuildExtract/Build

Ant, XML,PackageMake,ArtifactoryPro,

Maven

LabelLabel

Stash, Subversion,TFS

Unit TestUnit Test

JUnit/Clover,MSBuild/MSCover

App DeployApp Deploy

XLDeploy

SmokeTest

SmokeTest

JTAF/MSL

INTINT

QCQC

PRODPROD

5

Datamart

Traceability Per Drop- Build Records

- Labels, Build Time, KLOC, Test Results/Coverage

- Deploy Records- Build Record + Target Environment

App MonitoringApp Monitoring

NagiosAppDynamicsCloudwatch

ComplianceCompliance

Blackduck, ArtifactoryPro

46

1

2

9

8

3

FINRA Data Centers

Provision/Configure(AWS)

Provision/Configure(AWS)

Puppet, XLDeploy,CloudFormation

7


Evolution of Delivery Systems

Manual Deployments (Sets of Instructions with teams of people)

Automation Based (Base-lined, Managed scripts, teams of people)

Orchestrated (Automated systems, with Automated Delivery Dependencies; Some Manually Managed)

Continuous Delivery (Automated Orchestrated Delivery, Notification, Automated Assurances)

14

We Are Here

15

3 Moving To A New Reality


Future State Goals

Self-enabled, one button deployment Teams are appropriately enabled to deploy at will throughout the week with

minimal support interaction and orchestration Information to support release aggregated in a scorecard manner:

– Automated test results (e.g., coverage, execution history, etc.) which are transparent and sufficient to support push decision

– Operational readiness (e.g, security, infrastructure changes, etc.)– Orchestration (e.g., upstream/downstream dependencies, SLAs)

Automated environment promotion and quality checks throughout the delivery flow All changes are traceable through the entire delivery pipeline and across

environments

16


The Delivery Pipeline

An automated, self-enabled means for teams to define and visualize their development/build/deploy pipeline Any implements (e.g., XLDeploy) of the delivery “engine” are abstracted via the

pipeline which removes multiple tickets, requests and support teams to do common development functions

17


Future State Workflow for AWS Deployments

18

AW

SO

n P

rem

MergeConfigure Build Env

Trigger Build

Multiple Iterations

Development & QC

Promote

Production

Execute Build

Pre-Deploy Tests

StagePush

to XLD

Provision Environment

DeployPost-

Deploy Tests

Automated

Manual

New Scorecard

Scorecard(Approval)


Release Scorecard

19


Anticipated Benefits

Weekend intensive deployments shift to one-button deployments anytime during the week Responsive, timely changes flow faster to the user instead of monolithic releases

Multi-team deployment dependencies shift to self-service deployments by the application team Less external team bottlenecks and dependencies due to automation

Deployments are more reliable since they are “cheap” to perform over and over Builds reliability iteratively by constantly revealing issues as changes are integrated

into the application and deployed

High risk changes are more transparent and exceptional High impact cross-project dependencies are more easily assessed and orchestrated

20


Continuing Challenges

Security Code classification, encryption and data protection concerns as new technologies

and services emerge to meet the cloud impetus

Application Architecture Architecture implements are typically customized and difficult to automate and

orchestrate consistently within and across projects

Test Automation Need to shift from traditional roles (Dev vs QA) to multifunctional teams so

automation assets are a team commitment Unit & API tests need to be the majority in contrast to today’s end-to-end test

focus

Infrastructure as code Technology rapidly maturing but constantly changing which makes consistent

implementation difficult

21

Questions?

23

4 Backup Slides


FINRA Open Source – https//github.com/FINRAOS

24