| ©2012, Cognizant

©2012, Cognizant

CBC Practices Collaboration: Elizabeth Lerner, Partner – Program Management Consulting Susan Schanta, Associate Director - Program & Quality Consulting September 18, 2012

New Techniques to Elevate Quality Assurance to Program Assurance

| ©2012, Cognizant 1

Agenda

2 Introduction

3 Program Management

7 Program Assurance - Extending QA Principles

10 Program Schedule Standards

15 Managing Environmental Limitations

11 BRD Ambiguity Reviews

14 Architectural Reviews

17 UAT Strategy

18 Quality Measures

20 Balanced Scorecard

| ©2012, Cognizant

Introduction

• An IT-enabled Transformation should recognize that these are typically business-driven, not technology-led.

• A Program drives the conversion of

strategic intent into reality through the co-ordination of multiple projects focused on common objectives.

• Program Management stretches the project management model through a heightened level of cross-team coordination, cost management and quality controls to deliver a predictable outcome.

• Controls are needed to deliver

strategic, mission critical, and high risk programs of transformation successfully.

• Introducing a quality management

model into the program early reduces risk and increases flexibility to respond to challenges, and create efficiencies, preventing of do-over’s.

Program Management

Program Assurance

Trends Analysis

Value Management

Planning & Strategy

Architecture Project

Management

Implementation Management

Process Modeling

Applications Information

Infrastructure Planning

Gap Analysis

Project Monitoring

Resource Management

Risk Management

Strategy Formulation

2

| ©2012, Cognizant

Program Management

GOVERN

Build and implement best-practice project governance and controls while minimizing bureaucracy

SUPPORT

Help project teams by providing hands-on PM expertise. Coach and mentor project teams and become a center for project knowledge and learning

ASSURE

Provide independent challenge to projects. Assist

in the identification and management of cross-project

dependencies to ensure successful delivery

INFORM

Establish regular and accurate project reporting to provide ‘early warning’ management information

Program Management

Office

Deliver Business Solutions Successfully On Time, to Cost, and Quality

Enhance Organizational Capability Resource Management, Delivery, and On Demand Reporting

Driving the Strategic Agenda

A Project is an endeavor funded by finite resources with a finite duration and a specific outcome. A Program is a group of projects aligned by a single objective or outcome that can be better managed collectively than individually.

3

| ©2012, Cognizant

Program Management Dimensions

Review & Reporting

Risk & Issue Management

Program Assurance

Resource Management

Cost Management

Benefits Management

Communication & Business Input

Program Standards

Vendor Management

Governance & Organization Ensure projects are properly set up for success and business need is clearly articulated. Implement the overall, pragmatic, and consistent framework for managing the portfolio of client initiatives.

Consolidated view of projects. Maintain key management information from CFO level down. Report key actions and decisions required

Planning & Tracking Implement consistent approach for planning and tracking projects. Provide consolidated view of the portfolio roadmap

Implement portfolio-wide risk planning and management process to ensure the mechanism for dealing with issues is speedy / robust

Implement an independent challenge and assurance function that helps identify and resolve key issues of common concern and threats to benefits across the program

Build and maintain an overall resource plan to align the supply of resources and skills to the demand of the project portfolio

Program finances are managed, tracked, and reported across the Project Portfolio to ensure the cost of work remains in budget

Track and report the delivery of benefits arising from the portfolio and the value add to the organization

Implement the discipline of engaging and managing business input to projects. Develop and implement a communication plan

Set and maintain standard approach, tools, and techniques to support program delivery. Ensure standards adherence

Ensure rigor in the procurement and management of subcontracted third party services and products.

Program Management provides real management value by striking a balance between challenging and supporting projects.

Program scope clearly articulated. Implement consistent, pragmatic framework to management a portfolio of projects

4

| ©2012, Cognizant

Program Assurance

Plan > Assure > Control Purpose: Define quality standards in advance; review deliverables against standards; continuously question whether delivery is in line with the overall goals.

Benefit: Ensures that what the program delivers will achieve the overall aims

Agree

Agree to expectations up from to ensure that program deliverables are fit for purpose.

Record the agreement in a deliverable description record including what will be delivered, target date, format / template, and to what quality standard.

Program Quality Director helps the PM define the deliverable, apply knowledge of best practices, and ensure the end deliverable will meet the program’s requirements.

Question

Throughout the lifecycle and at all levels of the program, ask:

- Does this align with the program’s vision?

- How will this help achieve the program’s goals?

Test

When a deliverable is completed, it must be assessed against the quality standards set forth in the deliverable description.

The Program Quality Director will facilitate the process to ensure an appropriate independent reviewer has the opportunity to ‘test’ the deliverable and sign-off.

5

| ©2012, Cognizant

Polling Question

Program Assurance includes audits for

Project Schedules, Resource Management, Cost Management

Requirements Traceability, Environment Definition, Construction Readiness

Code Turnover Requirements, System Test Process

All of the Above

6

| ©2012, Cognizant

Program Assurance - Extending QA Principles

Program Assurance provides quality governance within the Program Management framework by introducing independent measurements to track and report against deliverables. Measurements are captured through frequent audits and risk assessments, making it possible to improve program execution.

Quality Assurance

• Requirements Traceability

• Test Coverage

• Test Execution Pass / Fail Rates

• Defect Management

Program Assurance

• Program Standards

• Planning and Tracking

• Cost Management

• Resource Management

• Environmental Limitations

• Risk / Issue Management

• Quality Gates

BRD Ambiguity Reviews

Construction Readiness

Architectural Reviews

• UAT Strategy

• Zero Uninspected Defects at Release

7

| ©2012, Cognizant

Techniques to Introduce and Maintain Controls

• Establish and maintain controls

• Ensure program management information is up-to-date and available

• Identify and manage business stakeholders

• Consolidate and analyze data across projects

• Produce regular and ad-hoc reports

• Track and realize program benefits

• Ensure all project plans are properly integrated

• Ensure plans remained aligned and dependencies and cross-project issues and risks are effectively managed

• Ensure project plans are complete, feasible, and all options are considered

• Ensure current status and outlook is realistic

• Ensure all issues are surfaced and addressed

• Advising and supporting individuals (especially PMs) in difficult or

unfamiliar circumstances

• Support the business through effective change management

• Conduct reviews (e.g. risk or reviews, reviews)

• Facilitating planning workshops

• Leading / participating in taskforces teams to resolve issues

Process and data

• Hard techniques; the most visible PMO deliverables

• Earns the right to do other things - necessary but not sufficient

• Set up right and get running smoothly

Content and people

• How experience is leveraged to add major value

• Earns the right to advise at a senior level

• The key to personal development through the project office

• Less obvious, need to search out

Controlling

Reporting

Integrating

Challenging

Coaching

Supporting

Controls are needed to deliver strategic, mission critical, and high risk programs of transformation successfully. Effective control is required to minimize disruptions and ensure that the program will meet its aims within time and cost constraints.

A set of controls handle the different challenges of program management, underpinned by a robust governance framework.

8

| ©2012, Cognizant

Project Lifecycle - Fundamental Building Block

Approval to proceed

Project Management Stages

Project Management * Deliverables

Program Related Processes

• Project Brief • Statement of Requirements

• Business Case

• Project Charter

• Management Plan

• Cost Plan

• Detailed Schedules

• Risk Assessment

• Quality Plan

• Create Balanced Scorecard

• Progress Reports

• Cost Reports

• Change Requests

• Risk / Issue Registers

• Implement Plan

• Quality Assurance

• Assurance Audits

• Post Implementation Review (PIR)

• Benefits Dashboard

Pre-feasibility Feasibility Management Planning

Execution and Delivery

Review and Closure

Start-Up

Work prioritization and approvals process (portfolio management)

Resource scheduling and allocation process

Annual budgeting / forecasting process

* Technical deliverables excluded from diagram

Approval to proceed

Approval to proceed

A consistent project lifecycle is the fundamental building block for control

Monitoring and Controlling

9

| ©2012, Cognizant

Program Schedule Standards

How do you measure? Consistent structure in project schedules results in consistent work effort measurement in the Integrated Project Plan. • Consistent Structure for Tasks

Across Phases - Requirements, Design, etc. By Team - Sub teams within departments By Artifact - SDLC documents

• Options

Tracking artifacts in separate matrix Separate stakeholder detailed project plans for

day by day tracking of low level tasks

Make Program Level Decisions Ensure a consistent measurement of program progress

10

| ©2012, Cognizant

BRD Ambiguity Reviews

Ambiguity Type

Search for What to Look for Comment Feedback

Term Appropriate Not quantifiable Ambiguous - Define appropriate

Term As required Not quantifiable Ambiguous - Define as required

Term Difficult Not quantifiable Ambiguous - Define difficult

Term e.g. Versus i.e.

Misuse of i.e. versus e.g. i.e. means "that is" I work most effectively at small coffee shops i.e. Starbucks… e.g. means "for example" I like melons e.g. honeydew, watermelon, cantaloupe, etc.

Should be i.e. which means for that is Should be e.g. which means for example

Term Error Error message or error reason not stated… What is the error message?

Term Error Error condition stated but not what causes the error condition to occur?

What is the error condition that causes this scenario?

Term Flexible Not quantified Ambiguous - Define flexible

Term If Applicable

Not quantified Ambiguous - What determines applicability?

Term ly Search for "ly" to find words that end in "ly" where the words are not quantifiable because each person can apply their own meaning. Examples: frequently, accordingly, efficiently, normally, similarly, commonly, etc.

Define _______ Examples: Ambiguous - Define frequently Ambiguous - Define efficiently

Term Maximum Not quantifiable Ambiguous - Define maximum

Term Minimum Not quantifiable Ambiguous - Define minimum

• Business Requirements define what will be delivered in the release.

• Ambiguity Reviews provide a measurement of whether scope documents are quantitative, clear, correct, and complete. Ambiguity reviews help eliminate defects in later phases by building quality into the scoping effort.

11

| ©2012, Cognizant

Sample Ambiguity Review

12

| ©2012, Cognizant

BRD Ambiguity Reviews

275

47 454

7

78

624

3

14

136

0

200

400

600

800

AmbiguousTerm

ClarificationNeeded

Field/Naming

Consistency

Glossary

Grammarand Wording

IncompleteRequirement

PossibleDuplicate

PossibleMissing

Requirement

UnclearRequirement

New Accounts BRD Ambiguity Review by Category

141

240

54

27 64

274 79 45 67

50

0 100 200 300 400 500

Ambiguous Term

Field/NamingConsistency

IncompleteRequirement

New Accounts Most Frequent Ambiguous Findings

(Ambiguous Adjective) All

Date/Time?

Define specific

Should Title be included?

SSN/TIN Field consistency

What are the valid values? Is this a limited list?

What are the valid values? Is this a limited list? What about Provincefor Canada?

• Can a non-SME understand the requirements?

• Are the requirements quantifiable?

• Can the design be measured for requirements fulfillment - Yes or No?

• Does the reader need to have all requirements available to understand the context of the requirement?

13

| ©2012, Cognizant

Architectural Reviews

Traditional Approach

• Architecture Documents Created

• Architectural Review Board during Design Phase High Level Design Functional Specifications Detailed Technical Design

Enhanced Approach - Construction Phase Reviews

• Selective Criteria Highest Impact on Failure Highest Volume Most Complex Algorithm

• Format White board walkthrough Q&A Session

14

| ©2012, Cognizant

Managing Environmental Limitations

• Establish working list of environmental limitations based on existing systems

• Meet with key stakeholders to document limitations

• Facilitate meetings to address solutions to limitations

• Escalate budget requirements for licenses, resources, etc. to Project Manager

15

| ©2012, Cognizant

Polling Question

When does UAT planning occur in large programs?

Incrementally throughout the lifecycle of individual projects

Only at the end of the program

Across all project lifecycle phases and at the end of the program

After Construction but before text execution

16

| ©2012, Cognizant

UAT Strategy

Shift User Acceptance Testing Earlier

Project Level: Early validation in System Test phase to confirm requirements fulfillment for deliverables such as reports, screens, calculations and business rules. This also offers an opportunity for the business to refine their workflow processes in response to the new or updated application.

Program Level UAT: Traditional E2E validation to validate the life of a transaction.

Event Calendar

A predefined schedule to help define, manage, and report on UAT test execution. The calendar supports the operational process of test execution by providing a shared understanding of the schedule and expectation for business user time commitments, scheduled system activities such as environmental updates and running batch jobs for electronic file processing. Run Books

Compilation of transactions that will be executed during a specific UAT cycle in the Event Calendar. Run Books assist the business users in organizing their test execution content and help control the order of processing. Run Books also provide a reference for the UATC and business to ensure everyone understands the goal of what will be executed in a given time period.

17

| ©2012, Cognizant

Quality Measures

18

| ©2012, Cognizant

Quality Measures

67% 33%

Project 1 Pass Fail

45% 55%

Project 2 Pass Fail

82%

18%

Project 3 Pass Fail

19%

81%

Project 4 Pass Fail

34%

66%

Project 5 Pass Fail

89%

11%

Project 6 Pass Fail

19

| ©2012, Cognizant

Balanced Scorecard

The Balanced Scorecard is a strategic performance management tool to track progress against plan.

• Point in time view • Mixture of financial and non-

financial measures • Tracks execution • Monitors quality • Highlights risks and issues

Getting Started

• Set expectations low • Define pain points and

milestones tracked across the program and / or project

• Determine available data inputs • Create initial scorecard at the

project level and build into a program level scorecard

• Use Excel to model appearance • Automate when stable

20

| ©2012, Cognizant

Balanced Scorecard

Program Level

• % Work Complete • Budget - Estimate vs. Actual • Program Tollgates - Final Dates • Project - Artifact Deliverables • Top 3 Risks • Top 3 Issues

Requirements

• Ambiguity Review Results • BRD to FSD Requirements Mapping • Requirements Assigned by Team • Test Case Coverage to Requirements • Program Completion of FSDs

Testing

• Test Execution % Complete • Test Case Pass / Fail Rates • Functional Modules Impact • Automation % Coverage • Performance Benchmark Results

Defect Management

• Defect Fix Rate Trend to Complete • Defect Root Cause • Defects by State • Defect Trend by Severity • Regression Defects • % Regression Defects - Automation

21

| ©2012, Cognizant | ©2012, Cognizant

Questions

22

| ©2012, Cognizant 23

Speakers Bios

Elizabeth Lerner, Partner - Program Management Consulting, Cognizant

Elizabeth (Liz) has spent 30 years managing large-scale change programs as a Strategic Global Program Director. In this capacity, she has been leading, managing, and delivering mission-critical results to top-tier global corporations. Liz brings with her a vast experience while working in various industries, successfully leading and delivering strategic change programs, overseeing initiatives of large-scale scope and complexity resulting in significant business value, top and bottom line growth, global industry leadership, business expansion, acquisitions and divestitures, operational efficiencies, and organizational restructuring. Liz has proven experience in strategy consulting for Fortune 100 clients. She is recognized for delivering large-scale programs on time and within budget and scope, thereby consistently meeting and exceeding customer expectations.

Liz can be contacted at elizabeth.lerner@cognizant.com.

Susan Schanta, Associate Director - Program & Quality Consulting, Cognizant

Susan has spent 20 years managing large-scale Program and Quality Assurance programs in new ventures for global Fortune 500 companies. She brings with her vast experience spanning the financial, healthcare, and retail domains. She has led several corporate initiatives in business optimization, onshore-to-offshore transitions, and QA enterprises including SDLC management, automation, and performance. Susan’s expertise in the industry-best practices and project execution has helped companies achieve their goals. Her expertise goes into niche skills such as implementing lifecycle disciplines through change management, sharply reducing budget variances, improving performance of tiered applications, introducing new disciplines for UAT execution to achieve improved quality and business workflow processes, and devising calibration methods to measure these achievements.

Susan can be contacted at susan.schanta@cognizant.com.

| ©2012, Cognizant 24

References

Project Management Institute

http://www.pmi.org/

Project Management: 6 Best Practices for Ensuring Software Implementation Success

http://www.cio.com/article/683570/Project_Management_6_Best_Practices_for_Ensuring_Software_Implementation_Success

The Ambiguity Review Process

http://benderrbt.com/Ambiguityprocess.pdf

Eliminating Ambiguity from Your Requirements

http://www.modernanalyst.com/Resources/Articles/tabid/115/Default.aspx?ArticleType=ArticleView&ArticleID=989

Building & Implementing a Balanced Scorecard: Nine Steps to Success

http://www.balancedscorecard.org/BSCResources/TheNineStepstoSuccess/tabid/58/Default.aspx

Balanced Scorecard & The Project Manager

http://www.ittoday.info/Articles/Balanced_Scorecard_and_Project_Manager.pdf