Case Study: SunTrust’s Next Gen QA and Release Services Transformation Journey

Case Study: SunTrust’s Next Gen QA andRelease Services Transformation Journey

Beth Johnson

DevOps: Continuous Delivery

SunTrust

SVP, Release and Test Manager

DO4X131S

#CAWorld

2 © 2015 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD

© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.

The content provided in this CA World 2015 presentation is intended for informational purposes only and does not form any type

of warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.

For Informational Purposes Only

Terms of this Presentation


Abstract

Sun Trust’s journey from challenge identification, charter definition, execution practices and key metrics and results in their transformation of traditional QA and Release functions into a more cohesive, collaborative and “continuous” model.

Beth Johnson

SunTrust

Senior Vice President, Release and Test Manager


Agenda

HISTORICAL CHALLENGES

CHARTER: WHAT WE SET OUT TO DO

EXECUTION: APPROACH, CULTURE, ESSENTIAL TECHNOLOGIES

KEY METRICS & RESULTS TO DATE

WHAT'S NEXT

1

2

3

4

5


About SunTrust

At SunTrust, success is defined by the impact

we make on the lives of our clients,

our communities and our teammates.

As one of the nation’s largest and strongest financial services companies, Suntrust

offers a full suite of products and services designed to meet the needs of our

consumer, business, corporate and institutional clients. Lighting the Way to

Financial Well Being is our commitment to helping instill confidence in the

financial circumstances of everyone we interact with.

Sub-head


Historical Challenges

Complexity

LOTS of legacy systems and platforms

LOTS of dependencies across differenct systems

LOTS of manual effort required to adequately test and release

LOTS of time required between releases

Collaboration

Dev and QA teams not familiar with each others jobs

App teams unaware of how their apps/ervices impacted other teams

Agile going well in some dev teams, but not being scaled to QA and Ops teams

Culture

Historical success rate with “doing things differently”

Mix of GSI teams (Cognizant, Infosys), consultants and W2 team members

Perception from LOBs as being slow


Our FocusSUNTRUST OBJECTIVES

AGILITY SCALABILITYCOST SIMPLICITY

ACCELERATE RENOVATE INNOVATE

Change Management

Tools Adoption

Cost Savings

Automation

Process to adopt Managed services

Shift Left

Domain

Pricing Model

Agile Model

PA

APPROACH to Next Gen QA Model


Transforming Testing Services

Definition & Communication of a Managed Services Vision

Transition Change Management from T&M (project based) to a managed

service

Lunch & Learns and EXPO (internal SunTrust tech conference)

Domain centric approach

Simplify and streamline demand management

Re-introduce Static Testing by partnering with BA team

New approaches & testing technology (Service Virtualization, App

Dependency Analysis, Cross Browser testing)

Execution highlights over the last 12 months…


Impact to Date in Testing Services…YTD savings of $6.5M with projected savings of $9.7M USD by November 2015

2 of 9 Can’t Fail Initiatives

implemented successfully

12 out of 25 SoW commitments

implemented and 10 In-Progress

DELIVERY

70% automatable automation and

64% Automated execution with a

cum. effort savings of 63,388 hrs.,

translating to a savings of $140K

AUTOMATION

Requirement Volatility Index (RVI) has

dropped from 82% to <50%

VALUE ADD

Significantly lower Defect Leakage

<1%, compared to SLA targets of 4%

LOW DEFECT LEAKAGE

82% of projects tested by Infosys were

successfully implemented with ZEROproduction defects

IMPROVED QUALITY

126 new projects are in Managed

Service model post Oct

MANAGED SERVICE


Transforming Release Services

Built a Performance Center of Excellence (PCoE)

Formed a Data Management Office (DMO)

Expand Service Virtualization (400+ services virtualized today) further left

in SDLC (functional test and early dev and integration test)

Deployed Test Environment Application Management (TEAM) and

automated much more of the environment staging and setup tasks

Execution highlights over the last 12 months…


Impact to Date in Release Services…

Test Environment Application Management (TEAM)

Data Management Office (DMO)

Performance Center of Excellence (PCoE) Service Virtualization (SV)

Fulfilled over 4412 data requests.

Data Reservation process introduced DMO assumed additional responsibilities for

OLBM related mining/ conditioning

4% of Data Requests addressed via

Automation Procedures

85% more data requests fulfilled successfully

Completed engineered & regression performance testing

for over 40 projects Enabled early performance engineering for ALL critical

projects Introduced new types of non functional testing such as

Endurance, Stress, Scalability, Fail-Over

1500+ hours of production performance triage reduced

by finding issues before PROD

Reduced the system / third party dependencies in performance testing

Over 400 app services are available for use

Significant speed increase and better quality with SV being use prior to QA in Dev

Supported 34 test environment cycles Achieved faster cycle run time after reducing known defects &

automating manual environment preparation activities

18% reduction in defects by automating environment

deployment


Leap Frogging Innovation and Testing Trends

• Defined the process, built the SharePoint site for idea gathering and tracking

• Formed the Innovation council to review and decision ideas

• Participated in 36 hr. digital challenge and provided about 56 ideas from the team

Innovation Tuesday

• Custom built tool for managing MS workflow

• Inbuilt exhaustive reporting

• Project Early Warning or ALERT system to automatically monitor and report project status

Service Delivery Platform and Alerts

• Analysis of Defects data of 4 major releases

• Utilized QA Analytics tool to analyze Application Risk and Defect patterns

• Predicted risk applications and vendors for upcoming releases

Predictive Analytics

• Introduced Domain Advisors in Testing practice to work with BA’s

• Implemented Static Testing to measure Requirement Quality

• Defined RQI and RVI to measure and Influence upstream quality

Domain Approach to Testing

• Virtualization Journey with CA partnership

• EXPO to create awareness of MS• Lunch n Learns to spread

message

Show and Tell

• Improved Daily and Weekly Status reports

• Enhanced SLA/Metrics report• Touch Screen dashboard for

Release and Test Reports (POC)

QA Command Center


Progress to Date vs. 5 Year Metrics…In the last 5 years…

Reduced the production defect leakage at to < 1% at enterprise level

Testing cost per project reduced by more than 27%

Requirement coverage increased from <70% to 98%

Extended testing reach from manual testing to SOA, IVR, Mobile, Mainframe, Package based testing

Extensive usage on advanced tools for effective testing – CA SV, Test Harness, HP Service Test, SSIS

Proactive Proof of Concept (POC) on latest tools - CrossBrowserTesting, Sprinter, TOSCA, Turnkey

6x increase in number of projects served over the past 4 years


SunTrust Approach to Innovation…Incremental but with a success at end of each step


What’s next?2016 plans…

Industrialization of Testing on Digital Platforms

Develop comprehensive End to End methodology to support agile

initiatives across SunTrust

Platform agnostic Automation solutions

Explore Robotics and Artificial Intelligence in Testing

Automation of the broader release process


Q & A


Recommended Sessions

SESSION # TITLE DATE/TIME

DO3X118S The why, where and how of Service Virtualization 11/19/2015 at 02:00 pm

DO3X116SLloyds Banking Group Leverages Service Virtualization to

Deliver Quality Technology Services11/19/2015 at 03:00 pm

DO3X114S

Freeing the World from Slow: How Service Virtualization

and the Concept of S.P.E.E.D. Will Steer You Away from

Committing Continuous Delivery Suicide

11/19/2015 at 03:45 pm


Must See Demos

Service Virtualization

DevOps

Theater 3

DevOps Simulation

DevOps

Theater 3

SV Innovations

DevOps

Theater 3

SV Use Cases

DevOps

Theater 3


Follow On Conversations At…

Smart Bar

DevOps

Theater 3 & 4

Tech Talks

DevOps

Theater 3 & 4


For More Information

To learn more, please visit:

http://cainc.to/Nv2VOe

CA World ’15

http://cainc.to/Nv2VOe

http://bit.ly/1wbjjqX