Process Software Validation Virtual - LORE -- Homepagelore.ua.ac.be/Teaching/STestenMaster/Industrial2011/05_LMS.pdf · Software validation – LMS Virtual.Lab. ... LMS Test.Lab Test-based

May 2011 – LMS Confidential

Software validation – LMS Virtual.Lab

2 copyright LMS International - 2011

Outline

2

1 Company profile

Introduction to QA

Virtual.Lab Product Development Process

Software validation in the development phase

Software validation in the QC phase

Customer responsiveness in the deployment phase

3

4

5

6


Company Profile


LMS, 30 years of Engineering Innovation

� Servicing more than 100.000 R&D engineers … in 5.000 manufacturing companies

� Top talent in 30+ offices Worldwide… almost 1.000 professionals

� Visionary, industry-leading innovator

… 25% of budget in R&D

� Strong financial track record of profitable growth

Leading Partner in

Test & Mechatronic Simulation


LMS, serving more than 5000 leading manufacturers

in the auto, aerospace, fabrication and assembly industries

Partnering with automotive manufacturers Partnering with automotive suppliers

Partnering with aerospace companies Partnering with manufacturing companies


LMS, Global Talent for Global Customers

R&D Centers

Brasov

Breda ChennaiCoralville Gottingen Kaiserslautern

Leuven LyonPlymouthRoanne

Torino

Sales Offices

Baltimore Bangalore Beijing

Coventry Detroit LeonbergLeuven Los Angeles

Lyon Moscow Munich Novara Paris

Sao Paulo Seoul Shanghai Toulouse

Yokohama

Representatives

Bangkok Bangalore BeckumBucharest Cairo

Kfar Saba Krakow Kuala Lumpur Ljublijana

Madrid Moscow Nilufer BURSA PsárySão Paulo

SarovSt. Petersburg Sydney Veliky Novgorod

Engineering Services

Beijing

Brasov ChennaiCoralville DetroitKaiserslautern

Leuven LyonTorinoYokohama

Europe

40%

Americas

20%

Japan/Korea

20%

RIC

20%

33 15


LMS, a unique portfolio of engineering innovation solutions

LMS Tec.ManagerTest data management

LMS Imagine.Lab1D simulation solution

LMS Virtual.Lab3D simulation solutions

LMS Engineering and

Deployment Services

LMS SCADAS

Mobile Recorder Lab

LMS Test.LabTest-based engineering solutions

LMS Test.XpressAnalyzer testing solutions

LMS SCADAS

Mobile Recorder Lab


LMS Virtual.Lab, the integrated platformfor 3-D functional performance simulation

Virtual.Lab Rev.9 “For automotive and aerospace design simulation, LMS Virtual.Lab remains one of the most comprehensive, capable, and versatile packages we have seen”

Jeffrey Rowe, Editor

Integrated CAD – Simulation Unified Modeling Supported by Test

Acoustics NVH Motion Durability


Introduction QA


Introduction to Software Quality Assurance Concept

�The perfect quality of a software is reached when the tests of all the possible combinations between all functionalities provides no error.

�Practically, this is a very expensive process and can’t be applied in software development industry.

�Software Quality Assurance Concept is a set of activities that assures for a certain software the standard of quality expected by customers.

� Fit-for-purpose� Accurate, correct� Performant

� User-friendly


Cost of correcting bugs across development lifecycle

Software Project Survival Guide

Steven McConnell, 1998


Virtual.Lab Product Creation Process & supporting Software Tools


Virtual.Lab Product Creation ProcessDifferent phases and stages – 1 major release per year

�Business scoping

�Planning

�Development cycle with sprints (Agile)

�Quality Control (QC) – internal & external launch

�Deployment and maintenance

+/- 1 year


Frontload quality into the sprints of the development cycle

� Improve quality and predictability of release

� Improve development efficiency and workload balance� Developer fixes issues close to when he/she has implemented the functionality� Early start of development of QC scenarios � Facilitates trade-off discussions on improving quality / scope of current content versus adding

more content

� More efficient execution of QC� Reduce backlog at start of QC phase� Reduced number of blocked scenarios in QC period� QC more dominated by regression tests on new and existing functionality� Should allow reducing duration of QC period, automating scenarios and/or preparing for next

release better

Sprint 1 Sprint N-1 Sprint NQC – phase 1

Scenario testing & fixing

QC – phase 2Regression &

stress testing

Sprint exit criteria

Out of Development/start QC

Out of Quality ControlExit criteria

Pre-production

Front load quality

Development of new functionality


Software tools supporting the development process

� Defect tracking system Mantis for internal and customer problem reports

� Team responsible

� Priority (escalated, major, …)

� Goes through different stages

• New

• Assigned

Assigned to developer

• Resolved

Fixed by developer and ready for Cross-testing

• Closed

� Web-based planning tool and time logging (internal tool)

� Document management (Lotus Notes)

� Metrics in excel through querying databases

� Source code management system (e.g. CVS, Mercurial)

� Central Software build environment


Software validation in development phase




stress testing


OOD/start QC

OOQCExit criteria

Pre-production

Front load quality


Requirements and stories – Approval

� Requirements are provided by product management, application export or customer services

� Development provides feedback and approves the requirement

� Requirement owner and product development have iterations to refine requirement

and break requirement into smaller pieces that have value

each to end-user, so-called stories. Stories describe how the software will actually work for the user in the words of development and typically provide use case.

� Stored in content management system

� Assigned to a development sprint

� Approved by owner of requirement

� Expected to be ready before start of sprint


Milestone tests

� Done developments are validated by the product (line) managers, application experts

and/or Customer Services through milestone tests. The milestone test aims at the

following:

� Check before OOD that the functionality is implemented according to approved

specification or story, and that it is correct. This could be done through testing on a

small model.

� Review ease-of-use and the fit-for-purposeness by testing on realistic models

� Review the performance of the implementation by testing on realistic models

� Outcome

� Approval or re-test needed

� Logged Problem Reports (PRs)


Break down in stories + effort estimate

Validation of developed story in sprint (development cycle)

Test

ob

jects

Requirement docincluding scenario & data Approval by Product

Development

MS test session (PM-PD)

Fix MS feedback

Work Breakdown Structure -Analysis & Design

Coding

Code review

Prepare for MS test

Re-test/Verify Problem Reports (PRs)

Out of Milestone Test Sprint

Out of Consolidation Sprint

MS approved/story done

Consolidation testing and fixing

Exit criteria

Sprint x –

Develo

pm

ent of sto

ries

Sprint x-1

Demo

Approval by Product Management


QC phase




stress testing


OOD/start QC

OOQCExit criteria

Pre-production

Front load quality


Quality Control (QC) Process Description

�QC process is build on Scenario Based Concept : the tester follows a certain suite of steps, usual involving pre-processing, solver and post-processing, and check the validity of the test with a clear and measurable set of exit criteria.

�Different types of tests�Manual tests execution

�It follows a static test plan with milestones – predefined at the beginning of the QC cycle

�Automated tests�Are a collection of automated QC scenarios�Usual procedure is to transform a manual scenario test into a automated scenario test�The automated tests are executed with each QC build, along QC cycle

�Regression Tests�They follow a dynamic test plan – it is defined based on the quality evolution during the tests execution

�Early Visibility Tests� Test session in presence of the customer before software release

�Stress Tests by application experts (out-of-the-box)

�Pre-Production and Installation test


QC Phase – Manual tests

� Information about Manual Tests� The number of manual scenarios is increasing each release, to cover the new functionalities.

� Details regarding the Manual Execution� Starting point of QC scenario is a data set to import or database saved with previous version� Also, it is necessary to have the exit criteria data:

� One XML file and one related Excel file per each analysis case� The Virtual.Lab model and solution that generated the exit criteria

� A prototype of the QC report (describing the steps to follow, the exit criteria and other details) is needed to execute the QC scenario

� Deliverables after Manual execution of the QC scenario:� An updated QC report with the conclusions of the tests� The Virtual.Lab model that generated the exit criteria results

�Follow-up progress of test plan in weekly QC meeting

Important :Test enough side paths

in the scenario

Integrate representative

data/scenarios of customers


QC Phase - Automated tests – General overview

�Why we need to have Automated Tests part of the QC process?

� AT are testing the whole scenario: pre processing,

solver and post processing.

� AT provide a fast and reliable feedback regarding

the quality of a certain build.� AT don’t experience the human errors, especially

during the execution of some complex scenarios.

�How the AT are running?� The core part of the AT is the Visual Basic

Application, driving Virtual.Lab to execute designed

steps

� Most of the scenarios need a set of initial files.

� The QC report is web based and includes a quantitative report (at the suite level) and qualitative

report (at the model level)

�What are cons for AT?� AT don’t test the GUI� AT take significant effort to develop in Visual Basic

Application

�Challenge� Design of test for keeping cost of maintenance under control (align with evolutions of the software)


QC Phase - Automated tests – Individual Exit Criteria

�Individual Exit Criteria from automated tests� Each tested variable is compared against a target set of data

�A graph is automatically generated for

each tested variable containing computed

data and target data

�The graphs are stored as XML files for future reference


QC Phase - Early Visibility Tests (customer involved)

� Three types of EVTs (Early Visibility Tests) with customer

� Alpha version

• Scoped in time (e.g. 6 weeks testing period)

• Agree with customer on how training, supporting, reporting and responding to

feedback is handled

• Regular follow-up meetings between PD and customer

� Hands-on (1-2 days on-site)

• On-site feedback - customer uses product

• Participates return to LMS with customer models for QC

� Demo (from few hours to 1 day)

• On-site or through WebEx


QC Phase - Manual Regression Tests

QC Regression Manual Tests

� Repeat of manual tests done before to manually verify all source code

changes did not introduce regressions

� The selection of tests is generated based on the statistics of PRs generated

during development and QC execution

� QC regression tests are executed by QC team




stress testing


OOD/start QC

OOQCExit criteria

Pre-production

Front load quality


QC Phase – Stress tests

� Stress tests� Are tests covering a certain functionality, used in a customer specific context

� Tests are done by Customer Service / Product

Management / Experts in the field

� They have not a written a QC scenario or a QC

report; testers are logging PRs and report at the end when the tests are done


QC Coordination and Follow-up Progress

� QC Meetings� They have as goal to keep track of the progress of the QC process� At the QC meetings are participating the Product Development Director, Product Development Managers for each

Product Line and QC manager

� QC manager is providing statistics regarding QC execution, bugs fixing and verification, unit tests and automated

tests.

� Using this statistics and also Product Development Managers viewpoints, we conclude on the QC current status for each product line and, if necessary, the actions needed to correct the status.

� The statistics and the conclusions of the QC meetings are compiled in a weekly QC report

�Metrics� Progress scenario execution� Open PRs per severity

� PRs to cross-test

� Success/failure of ODTs

and automated QC tests

� Exit criteria

� GO/NO GO Meeting – Review delayed issues with

� Product Management�Customer Services

� Product development

PRs target Rev 10

1 2 3 4 5 6 7 8 9

QC week number

Am

ou

nt

of

PR

s a

t e

nd

of

we

ek

Incoming S0+S1

Fixed S0+S1

Open S0+S1

Fixed S2+S3

Fixed features


Pre-production - Installation And Light Tests

� Installation and Light Tests� These are tests executed during pre-production and using the final

release build

� Light tests (e.g. Tutorials) are executed for each product line

� Various installation tests on all supported platforms are validated

� Tests from master DVD before production

� Tests conclude in short time span (few days only)




stress testing


OOD/start QC

OOQCExit criteria

Pre-production

Front load quality

Release


ODTs

Coverage of ODTs and Scenario Based Testing

Fu

nctio

nality

1

Fu

nctio

nality

x

Full Coverage

Fu

nctio

nality

2

Fu

nctio

nality

n-1

Fu

nctio

nality

n

QC Scenarios Based Testing – Automated and Manual Tests

ODTs - focus on testing each functionality independently

Scenarios Based Testing – focus on testing of more functionalities, organized in a similar way as the customers

may use

. . .. . .


QC Coverage Concept

�What is the Coverage Concept?� Identify what test models and scenarios cover what functionality and identify the missing pieces� Reflects also how many functionalities are tested in a certain interval of time or for a certain goal

� Requires tools that track what the tester/using is doing, logging the details of the user actions and link it to the test models/scenarios

� Allows to develop impact analyzer, i.e. a fix for a software issue can be linked to test models/scenarios and re-run those tests.

VL Name Models Items

Accumulator 1 1

Aero Aero 0 0

Aero Hydro 2 2

Aero Vehicle 3 4

Amplifier 27 114

Beam 6 6

Body 426 2317

BodyFromNewPart 61 1660

BodyFromExistingPart 65 608

Bracket Joint 111 1235

Bushing 0 0

Bushing Link 2 2

Radial Bushing 3 3

Standard Bushing 137 1236

Spherical Bushing 0 0

Bump Stop 2 2

Frequency Bushing 0 0

Contact 0 0

Legacy CAD Contact 0 0

CAD Contact 3 12

Check Valve 1 2

Extruded-Surface-to-Revolved-Surface Contact 4 25

Pipe Roller Contact 0 0

Point Ground Contact 0 0

Point-to-Point Contact 7 28

Sphere-to-Rail Contact 0 0

Sphere-to-Extruded-Surface Contact 11 108

Sphere-to-Revolved-Surface Contact 5 80

Sphere-to-Ground Contact 0 0

Control Function 10 69

Csys 0 0

CV Joint 50 127

Cylindrical Joint 151 348

Dead Zone 3 6


Deployment and Maintenance


Virtual.Lab Product Development – Software in shipment

• Continous improvement (lessons learned)

• Measure success of quality improvement program (frontloading quality into sprints) through comparison with previous release

� Listen to feedback of customers, regions, PM, ES, …

� Responsiveness to customer bugs

• Hot fix procedure for escalated issues

• Metrics on major customer bugs fixed in 5 months

• Metrics on major and minor bugs fixed in in 12 months

� Responsiveness to customer features

� Customer Satisfaction survey

1 2 3 4 5 6 7 8 9 10 11 12 13

Weeks after release (master version)

Am

ou

nt

of

bu

gs

10 Blocking bug

9 Blocking bug

10 Blocking + Major

9 Blocking + Major


Documents

Process Software Validation Virtual - LORE -- Homepagelore.ua.ac.be/Teaching/STestenMaster/Industrial2011/05_LMS.pdf · Software validation – LMS Virtual.Lab. ... LMS Test.Lab Test-based