CHICAGO - Quality Engineered Software and Testing Expo · CHICAGO Hyatt Regency McCormick Place ... performance test plan, scenarios, and executes ... Template Manual Upload Outstanding

CHICAGO

Hyatt Regency McCormick Place 2233 South Doctor Martin Luther King Junior Drive

Chicago, IL 60616

Presenter: Company:

Shaun Bradshaw Zenergy Technologies, Inc

Thursday 4/18/2013 3:00 PM - 4:00 PM

QA Process Implementation on

Large Scale Projects

Title

Slide 1 Software Peace of Mind in Action

QA Process Implementation on Large-Scale Projects

QUEST 2013Presented By:

Shaun Bradshaw


???

Agenda

• What constitutes a large-scale project?• QA Approach• Data Strategy• Environments / Infrastructure• Team Structure• Communications• Integration Testing• Performance Testing• Key Take Aways


???

Large-Scale Projects

• What constitutes a “large-scale project”?– Number of resources?– Duration?– Number of integrated components?– Cost?– Other?


???

Large-Scale Projects

• What constitutes a “large-scale project”?– Number of resources? 100+– Duration? 2+ years– Number of integrated components? 6 (major)– Cost? $70 million– Other? Multiple external vendors operating in strict

waterfall methodology


???

Terminology

• System Testing– Performed at the component level– Includes functional, data transformation/validation,

and security tests

• Integration Testing– Performed across multiple components of the

solution– Includes integration, data quality validation, and

cross-component security tests


???

QA Approach

• Align with overall development methodology– Don’t create “culture shock” by following completely

different test methodology• If the project is waterfall, utilize the V-model of QA or

something similar• If the project is agile, help establish iterations that QA can

work with and ensure open collaboration across the project

– Ensure the QA approach is thoroughly communicated across the project team


???

QA Approach – Phased Testing Approach

• Here was our approach


???

QA Approach – High Level Timeline


???

Determine the Data Strategy

• Data Acquisition– Create test data

• From component systems• Based on test criteria

– Pull data from production• Ensure proper data quality

– Data meets the needs of the testing– Data is in “proper” form

• Data masking?

– Data Volume?


???

Environments / Infrastructure

• Determine, as early as possible, what environments are available & necessary– Component-level system test environment– Integration test environment– Performance test environment (possibly production if

this is the first instance of the application)– UAT environment


???

Team Structure

• Key Roles– Test Architect – Creates overall test strategy

communicates/socializes it to project team– Test Manager – Manages day-to-day test activities– Test Leads – Component-based/Technology-based– Test Analysts – Develop and execute tests– Test Project Manager – Ensures test activities are

properly scheduled– Integration Lead – Pulls together integration test

effort


???

Team Structure

• Key Roles (continued)

– Performance Lead/Engineers – Develops performance test plan, scenarios, and executes

– Test Data Modeler – Determines data needs & acquires test data

– Business Leads– provide user-type input/review of test scenarios and test cases

– Technical Leads – Assist in creation of QA environment and data needs

– Release Manager – Helps manage all configuration and component movement


???

Communications

• The BIGGEST Challenge for QA– Utilize 360° communication strategy

• QA approach and reasoning to upper management (early and often)

• Resource and schedule requirements to peers• Tactics, techniques, and goals to down line resources

– Beware of assumptions made due to lack of communication


???

Communications

• The BIGGEST Challenge for a Large-Scale Project– Identify the right resources to include when

discussing• QA Approach / Test Strategy• Data Requirements / Strategy• Resources & Schedules• Test case framework• Test phases


???

Integration Testing

• The 2nd BIGGEST Challenge for a Large-Scale Project– Logistical challenges– Political challenges

• Purpose – demonstrates that the IT processes and systems/components built to support business processes are mutually integrated correctly


???

Integration Testing

• Integration Test Approach– Map out critical data flows between components

(IDEF is a good model)– Determine “day in the life” scenarios ensuring that all

critical data flows are covered– Develop integration test cases using previously

created system test cases to minimize rework (use a modular test framework)

– Utilize a “bottom-up” approach where possible (start with C2C scenarios, then E2E scenarios)


???

Integration Testing

DRAFT

DI(Includes

COPS/RDM)

DW(ODS/ODM)

BI

NXG

Ess Base

PS Formatted Data to JGen(Batch Data)

Combo Edit

Explosions

SIF Formatted Files

Direct to E2 Formatted Files

Processing Schedule

Attachments

Journal Line Attachments

Hyperion Out: Actuals and Budget

Hyperion In: Budget DataGL91I0014

Webservice PS Chart of Accounts (partial or

complete) to be validated by DI (PCR 33)

Reports

PeopleSoft UI

Spreadsheet Uploads

Converted e2 to PS Data

Manual Data Entries

Data Dictionary / SIF

NXGBalance

Life Com

COPS Audit Info

Application and User Security

Internal ControlsBusiness Rules

Rejected File

Self Service Balance(Daily Balance File)

Config/Settings: Open Periods, Mapping Rules, Ledger Code, Chart Fields, Open Periods, Trees, Business Units

JE Lines/Hdr(Nightly Batch)

AutoSys

Excel Active Directory

PS Query Results

E2 Mapping Rules

Black Box e2 Formatted Files

E&Y

CIMS

Server/Mainframe Environment

Business Rules

Error Theshold

E2 to PS Chart field translation ‐ flat file

Input (PCR 15)

GLCDS

Converted Rejection Info Email

Fin DetailTable (Nightly Batch)

Compass Integration Model A1101711 1545.vsd

Date Dimension File (On Demand)

Via Data Analyst Package to RDM

Budget On Demand (ETL)

Financial Transactions File (FTF)

E2 Balances

Manual UploadOf Opening /

Outstanding Items

Match/Pass RulesRemitter Lookup Table

AccountStructureExtract

Extract Criteria Template

Manual Upload

Outstanding Template

Trigger for FTF

Trigger for Balance

Reconciliation and Reporting Data

Common View CSA(Webpage)

PS Universe(Read Only)

DI Staging Tables(Read Only)

ODS/ODM/RDM Universe (Read Only)

COPS Universe(Read Only)

Non Chart Field Table (Excel File) Via Data

Analyst Package to RDM

Stop File

VPD Security

Oracle

Webservice NDM

Informatica

Self Service Business Object Reports

Self Service EssBase Reports

80 CompassDeveloped Reports

BO XIR3 Oracle DBAs

BO Portal (Internet)

Financial ControllerHyperion EssBase 9.3.3 (Arc Cube) Consumers

InfoView

Oracle

Excel Add‐in

Accurate Webclient

Informatica

Business Objects Commonview/CSA

Prod Application Admin

InformaticaDA Tool

Email Server

WLK Day Transactions

Outstanding Items Conversion Data

Remitter Number Table

Excel (min 2007)

Perl and Korn Shell, Acs Scripts

NXG BO Universe Reports

NXG AdminNXG Reconciler

BO User

WebDriver

Generator

Ariba

JE Lines/HdrLedger (Trickle Feed)

PeopleSoft Autosys Adapter

SFTP

JGen Mappings

Users

RDM Universe

(Read Only)

Webservice Validation Data (PCR 33)

ETL Controls for ODS data movement: All but ODM data

EIS Trees (ARC)

EAS Ledger

Table Data (ARC)

Date Dimension

CDS

GL

Data Analyst

Admins

Admins

PS Users

Manual Open/Close Process

Processing Schedule

Processing Schedule

Excel Upload TemplateDate/Time

Manual Schedule

Processing Schedule

Date/Time

Internal Controls

Internal Controls

Internal Controls

Internal ControlsProcessing Schedule

Date/Time

Application and User SecurityApplication and

User SecurityApplication and User Security

Active Directory

System Connectivity

Oracle InformaticaOracle Active Directory

AutoSys

System Connectivity

RDM

01

02

03040506

01

02

03

0405

30 31 1011

13

1514

12

1514

01

02

03

010203

040506

10

11

30 31 32 10

30 31 32

1617

10

11

12

30 31

03

01

02

04

10111213

1415

01

03

Application and User Security


System Connectivity


System Connectivity

AutoSys

InformaticaOracle Active Directory


System Connectivity

AutoSys

Webservice NDM


System Connectivity

AutoSys Server/Mainframe Environment

System Connectivity

Active Directory

Consumers

Shared File System

Validation Error Report Shared File System

Translated PS File

Admins

Cost Center Validation

Cost Center BU Account

Detail Extract via SFTP

PS

Upstream Sources

Upstream Sources

Upstream Sources

e2

e2

NXG Team

e2

Annuity Team

BI Team

Journal Entry Preparers

Accurate WebClient

NXG Data Base View

Author: Alan Smith

Trigger for FTF

Self Service Balance(Daily Balance File)

Tree Files

Balance Files

BO Reports

SQL Data Load

CDS

ICAS

Cost Center Validation Data

Modified ICAS Report Excel File Manual Load

Policy Level Claims Reports

Processing Schedule

Hourly JE extract for spreadsheet upload

Policy and Company Number

Business Objects

Treasury

Tax

Invest Accntg

Trigger for Balance

01

0203

30 31

01

30 31

30 3101020304


System Connectivity

Webservice

Controls for Trickle Feed and Batch for JE Lines etc. data

movement



12

04

32

Send controlled by DI but does not go through DI

13

BI Team

Notes

1617

40

111213

1840

33

40

30

40

32

40

33

02

05

10

40

10

40

32

40

32

40

1718

How to read the model

ACTIVITY A

ACTIVITY B

A Sends to B

B Pulls from A

SourceInput

DestinationOutputPrimary Flow

Controls, Contraints, Rules, Timings

Mechanisms, Systems, Users

Mechanisms and non‐data generated Controls grouped to remove clutter of lines

Red lines/text indicate High RiskGreen line is for Reports onlyDotted line is for PCR 33 only

Labels, Descriptions

10ICOM Number

Input 01‐09Output 10‐19Control 30‐39Mech 40‐49


???

Integration Testing

• Validation Approach– Focus on positive scenarios during integration (use

system testing for most of the negative test scenarios)

– Determine the best approach given time, resources, and risk• Validation of Execution• Validation of Results


???

Integration Testing

• Validation of Execution– Assumes detailed validation of expected results was

completed during system testing and system tesstscripts are being re-used as part of integration

– Generally, the most efficient method of validating results for integration scripts, but increases the risk of missing potential defects

– Best used when a component has been thoroughly system tested - it is acceptable to simply verify the process executed


???

Integration Testing

• Validation of Results– Doesn’t assume the correctness of results from

previous testing phases– Necessary to fully validate the process outputs at a

detailed level via external calculations (i.e. “tool”, spreadsheet, SQL query, manual calculations, etc.)

– Although dependable, low risk this method time/resource intensive


???

Integration Testing

• Execution– Its all about COMMUNICATION– Because an individual tester may not have requisite

knowledge to validate every step of an integration scenario “hand-offs” are necessary• Assign a resource to manage the communication of the

hand offs (we don’t want tests sitting)• Hold daily triage and schedule meetings• Be flexible


???

Performance Testing

• Application Performance– Verify each component is properly tuned as early as

feasible– Align with system test phase

• Infrastructure Performance– Utilize an integrated environment to determine

issues in the infrastructure of the solution– Align with integration test phase


???

Key Take Aways

• Develop an overall test strategy that aligns with the development methodology and culture of the organization

• Ensure key roles are properly filled– QA Architect– QA PM– Business Owners– Data Modeler

• Determine where/how you will get your data (the data strategy)


???

Key Take Aways

• Don’t rush to integration testing• Identify key business and technical resources to

assist in developing the integration scenarios• Utilize a modular testing framework so creation of

integration tests requires little additional work• Performance test key components as early as

possible and ensure the entire solution is performance tested

• Communicate, communicate, communicate

Questions?

Thank you!


Documents

CHICAGO - Quality Engineered Software and Testing Expo · CHICAGO Hyatt Regency McCormick Place ... performance test plan, scenarios, and executes ... Template Manual Upload Outstanding