Improving the Management of the Software Acquisition ... · Process: a Methodological Approach in Automotive ... Cruise control Antilock brakes ... Lancia and Alfa Romeo vehicles

January 26th 2004 ASIS Intl. Conference

Improving the Management ofthe Software Acquisition

Process: a MethodologicalApproach in Automotive

Fabrizio FabbriniFabrizio FabbriniFabrizio FabbriniFabrizio Fabbrini, Mario, Mario, Mario, Mario Fusani Fusani Fusani Fusani,,,, Giuseppe Lami Giuseppe Lami Giuseppe Lami Giuseppe LamiSystem & Software EvaluationSystem & Software EvaluationSystem & Software EvaluationSystem & Software Evaluation Centre Centre Centre Centre – I.S.T.I. / C.N.R. – I.S.T.I. / C.N.R. – I.S.T.I. / C.N.R. – I.S.T.I. / C.N.R. Pisa Pisa Pisa Pisa (Italy) (Italy) (Italy) (Italy)

Edoardo SiveraEdoardo SiveraEdoardo SiveraEdoardo SiveraFiat Auto S.p.A. - DT - SIEE – PDE - R.D.E. Software Torino (Fiat Auto S.p.A. - DT - SIEE – PDE - R.D.E. Software Torino (Fiat Auto S.p.A. - DT - SIEE – PDE - R.D.E. Software Torino (Fiat Auto S.p.A. - DT - SIEE – PDE - R.D.E. Software Torino (italyitalyitalyitaly))))


Issues in AutomotiveIssues in AutomotiveIssues in AutomotiveIssues in Automotive§ The number of software-based components in

automotive systems is increasing:§ Comfort electronics

Seat and window movement§ Real-time critical functions

Cruise controlAntilock brakesEngine management

§ Multimedia applicationsGPS, DVDInternet

§ Wireless applicationsWireless applicationsWireless applicationsWireless applicationsTyre pressure controlAccess control

Ł Need to control the development cost of software-basedcomponent.

Ł Need to manage the development of the software-basedcomponent, to increase the quality of the final products.

Ł Need to define new relationships with the suppliers, to bettermanage sw-based component.


System & SoftwareSystem & SoftwareSystem & SoftwareSystem & SoftwareEvaluation Evaluation Evaluation Evaluation CentreCentreCentreCentre

§ The System & Software Evaluation Center (SSEC)performs independent certification activity in the area ofInformation Technology since 1984. In particular:

■ Software Product Evaluation [ISO/IEC 9126 & ISO/IEC 14598]Software Product Evaluation [ISO/IEC 9126 & ISO/IEC 14598]Software Product Evaluation [ISO/IEC 9126 & ISO/IEC 14598]Software Product Evaluation [ISO/IEC 9126 & ISO/IEC 14598]■ Software Product Evaluation and Certification according to definedSoftware Product Evaluation and Certification according to definedSoftware Product Evaluation and Certification according to definedSoftware Product Evaluation and Certification according to defined

requirements and standardsrequirements and standardsrequirements and standardsrequirements and standards■ Software Process Assessment (Software Process Assessment (Software Process Assessment (Software Process Assessment (process improvementprocess improvementprocess improvementprocess improvement, , , , capabilitycapabilitycapabilitycapability

determinationdeterminationdeterminationdetermination) [ISO/IEC 15504, SPICE]) [ISO/IEC 15504, SPICE]) [ISO/IEC 15504, SPICE]) [ISO/IEC 15504, SPICE]

§ SSEC is part of ISTI, an Institute of the Italian NationalResearch Council that performs research in ComputerScience, Information Technology and related applicationareas, mostly within the framework of national andinternational research projects. SSEC staff membersparticipate in national and international Working Groupsfor ISO standard definition in the field of SoftwareEngineering


Fiat AutoFiat AutoFiat AutoFiat AutoProduct & Process EngineeringElectronic & System Engineering

Tasks§ To define requirements

(HW, SW, communication,reliability, etc.) for theelectronic systems andcomponents used in allFiat, Lancia and AlfaRomeo vehicles.

§ To define requirementsfor the vehicle “body”electronic systems– Lock-unlock doors, passive

entry system, anti-thiefsystem, seat movement,windows, mirrors, etc.

Objectives§ To monitor the Software

technologies used in theembedded electronicsystems

§ To use standardmethodologies in order toevaluate the “processcapability” of the suppliers

§ To control the software life-cycle of the embeddedsystems

§ To define and applymethodologies to validatefunctional requirements ofthe embedded systems.

§ To define the SWarchitecture used inembedded systems


The ESCAPE ProjectThe ESCAPE ProjectThe ESCAPE ProjectThe ESCAPE ProjectGoalsGoalsGoalsGoals

§ To set up a methodology supporting themanagement of software projects andsuppliers

§ To improve FIAT process to select suppliers§ To improve the software development

process of suppliers§ To provide FIAT with methods to determine

the risks associated to software suppliers§ To give FIAT a better control on the software

development project and on the quality ofthe resulting product


The ESCAPE ProjectThe ESCAPE ProjectThe ESCAPE ProjectThe ESCAPE ProjectAction PlanAction PlanAction PlanAction Plan

■ Phase 1: FIAT Suppliers’ CapabilityDetermination

■ Phase 2: Monitoring Plan Definition■ Phase 3: Methodology Validation


Reasons for SPAReasons for SPAReasons for SPAReasons for SPA§ Software Process Assessment is a way to

better understand (and manage) thesoftware process development of asupplier. The assessment is done to:

define a “capability” level and a “risk” levelfor each supplierhave a criterium to choose suppliers basedon their “capability”understand weak and strong areas of thedevelopment processdefine better functional requirementsdefine better system verification andvalidation procedures

PhasePhasePhasePhase 1 1 1 1: Supplier’s Capability Determination


Assessment PreparationAssessment PreparationAssessment PreparationAssessment Preparation§ Planning the AssessmentPlanning the AssessmentPlanning the AssessmentPlanning the Assessment

On-site visitTime/Cost constraintsTechnical constraintsAssessment risk identification

§ Defining the Assessment PurposeDefining the Assessment PurposeDefining the Assessment PurposeDefining the Assessment PurposeCapability Determination[Process Improvement]

§ Defining the Assessment ScopeDefining the Assessment ScopeDefining the Assessment ScopeDefining the Assessment ScopeRequirements elicitation process (Requirements elicitation process (Requirements elicitation process (Requirements elicitation process (CUSCUSCUSCUS.3).3).3).3)System requirements analysis and design processSystem requirements analysis and design processSystem requirements analysis and design processSystem requirements analysis and design process((((ENGENGENGENG.1.1).1.1).1.1).1.1)Software design process (Software design process (Software design process (Software design process (ENGENGENGENG.1.3).1.3).1.3).1.3)System integration and testing process (System integration and testing process (System integration and testing process (System integration and testing process (ENGENGENGENG.1.7).1.7).1.7).1.7)Project management process (MAN.2)Project management process (MAN.2)Project management process (MAN.2)Project management process (MAN.2)



Project implementationProject implementationProject implementationProject implementationpre-assessment activities

§Introductory meetingIntroductory meetingIntroductory meetingIntroductory meetingTo introduce the SPICE(ISO15504) approachTo review the assessmentpurpose, scope and constraintsTo introduce the assessmentactivities and the provisionalassessment plan

§Pre-assessmentPre-assessmentPre-assessmentPre-assessmentquestionnairequestionnairequestionnairequestionnaire

To gather preliminaryinformation on the projects to beused as process instances

• sw life cycle• swrequirements• test reports• test plan• qualityrequirements



Project implementationProject implementationProject implementationProject implementationon-site activities

§ BriefingAssessment purpose,scope, constraints andmodelConfidentiality policyAssessment schedule

§ Data Acquisition &Validation

PresentationsDocument analysisInterviews

§ Process rating(provisional)

§ Debriefing

} Checklist-based



The Rating DilemmaThe Rating DilemmaThe Rating DilemmaThe Rating Dilemma

§ Different rating methods can beapplied

§ ranging from the mereprocessing of measuredindicators up to the unaidedassessor’s judgement

§ Need to establish therequirements to be satisfied fora rating method to be valid

§ Trade-off: assessor’s judgementdriven by checklists



Project implementationProject implementationProject implementationProject implementationpost-assessment activities

§ Process rating (final)For each process assessed,assign a rating to each processattributeRecord the set of processattribute ratings as the processprofile and calculate thecapability level rating

§ Reporting the resultsPrepare the assessment reportPresent the assessment resultsFinalize and distribute theassessment report



Phase 1 Outcomes Phase 1 Outcomes Phase 1 Outcomes Phase 1 Outcomes (I)(I)(I)(I)CUS 3

0

1

2

3

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11

project

cap

abili

ty le

vel

ENG 1.1

0

1

2

3

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11

project

cap

abili

ty le

vel

ENG 1.3

0

1

2

3

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11

project

cap

abili

ty le

vel

ENG 1.7

0

1

2

3

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11

project

cap

abili

ty le

vel

MAN 2

0

1

2

3

4

1 2 3 4 5 6 7 8 9 10 11

project

capa

bilit

y le

vel

Synthetic Results

0

1

2

3

4

CUS 3 ENG 1.1 ENG 1.3 ENG 1.7 MAN 2

CUS 3

0

1

2

3

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11

project

cap

abili

ty le

vel

ENG 1.1

0

1

2

3

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11

project

cap

abili

ty le

vel

ENG 1.3

0

1

2

3

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11

project

cap

abili

ty le

vel

ENG 1.7

0

1

2

3

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11

project

cap

abili

ty le

vel

MAN 2

0

1

2

3

4

1 2 3 4 5 6 7 8 9 10 11

project

capa

bilit

y le

vel

Synthetic Results

0

1

2

3

4

CUS 3 ENG 1.1 ENG 1.3 ENG 1.7 MAN 2



Phase 1 Outcomes Phase 1 Outcomes Phase 1 Outcomes Phase 1 Outcomes (II)(II)(II)(II)


§ Trends§Requirement analysis as a key issue§Awareness of the customer role in the acquisition

process§Need for new SW development models

§ Trade-offs§Platform-oriented vs customer-oriented§Resource (memory size, processor

performance, design complexity) saving vsmaintainability and reliability

§ Open issues§ Interoperability at subsystem level (ECU)§Safety and security implications


Phase 2: ObjectivesPhase 2: ObjectivesPhase 2: ObjectivesPhase 2: Objectives§ This phase aims at defining a plan for monitoring a software project at

different development phases to obtain quantitative measurements of thequality of the related work products and perform predictive evaluations ofthe quality of the final product.

§ For this purpose, a sample project is selected and used to identify keyprocesses to be assessed in order to derive their capability and define anevaluation plan for achieving quantitative data.

§ The plan provides structured joint reviews and a set of characteristics andmetrics along with the correspondant expected values.

PhasePhasePhasePhase 2 2 2 2: Monitoring Plan DefinitionMonitoring Plan DefinitionMonitoring Plan DefinitionMonitoring Plan Definition


Identification of KeyIdentification of KeyIdentification of KeyIdentification of KeyProcesses for MonitoringProcesses for MonitoringProcesses for MonitoringProcesses for Monitoring

§ Requirements Elicitation and Analysis§ Software Design§ Coding§ Software Testing

Motivation§ Common milestones of the software process of most FIAT Auto’s

suppliers§ Correspondence with the Assessment Scope of the Phase 1. A

relevant amount of information has been already collected



Selection of the Pilot SupplierSelection of the Pilot SupplierSelection of the Pilot SupplierSelection of the Pilot SupplierAdopted Criteria for the selection of a pilotproject:

§ High capability levels resulting from Phase 1 in orderto get confidence in suitable work products

§ Medium-small dimension of the project (100-150 Kloc)§ Development process at an appropriate stage§ Belonging to the Body Computer/Comfort Electronics

business area to maintain the same applicationdomain as Phase 1



Quality Evaluation MethodologyQuality Evaluation MethodologyQuality Evaluation MethodologyQuality Evaluation Methodology


ISO 9126

EfficiencyReliabilityUsability PortabilityMaintainabilityFunctionality

suitability maturity coexistencetestability

changeability

analyzability

Interoperability

accuracy

1.Requirements Analysis tools

2. Software Design analysis

tools

3.3.3.3. Coding verification

tools

4. Software Testing

Analysis tools

ISO 9126

EfficiencyReliabilityUsability PortabilityMaintainabilityFunctionality

suitability maturity coexistencetestability

changeability

analyzability

Interoperability

accuracy

1.Requirements Analysis tools

2. Software Design analysis

tools

3.3.3.3. Coding verification

tools

4. Software Testing

Analysis tools


Monitoring Techniques andMonitoring Techniques andMonitoring Techniques andMonitoring Techniques andToolsToolsToolsTools

§ Requirements analysis => Ambiguity and Expressivenessanalysis made by means of the QuARS tool

§ Software Design analysis => Checklists + QA-C tool§ Coding Style => SPLINT / QAC MISRA tools§ Software Testing analysis => Checklists



RequirementsRequirementsRequirementsRequirementsAnalysis Tool:Analysis Tool:Analysis Tool:Analysis Tool:

QuARSQuARSQuARSQuARS


Indicator-related dictionaries

Syntax Parser

Parsed.txt

sentences.txt

LexicalAnalysis

SyntacticAnalysis

multipleimplicitunderspec

subjectiveoptional

vagueweak

metrics

LogsViews derivation

0

1

2

3

4

5

6

7

8

1 2 3 4 5 6

GraphicsDomain dictionaries

Indicator-related dictionaries

Syntax Parser

Parsed.txt

sentences.txt

LexicalAnalysis

SyntacticAnalysis

multipleimplicitunderspec

subjectiveoptional

vagueweak

metrics

LogsViews derivation

0

1

2

3

4

5

6

7

8

1 2 3 4 5 6

GraphicsDomain dictionaries


Requirements AnalysisRequirements AnalysisRequirements AnalysisRequirements Analysis


§ Suitability evaluation: the presence of potentialambiguity and vagueness defect in the requirememts ismeasured

§ Testability evaluation: the requirements testability isevaluated by identifying requirements expressed in anon-imperative way, being too much complex or difficultto be understood.


Software Design analysisSoftware Design analysisSoftware Design analysisSoftware Design analysistoolstoolstoolstools

§ QA-C MISRA§ Checklists aiming at verifying:

§ The completeness and precision of the software architecturedesign and interfaces

§ The documentation of the software units and iterfaces§ The definition of test requirements§ The satisfaction of software design requirements

§ Interoperability evaluation: the accuracy and precision of thecomponent’s interfaces description is evaluated.

§ Changeability evaluation: the degree of modularity of thesoftware.

§ Co-existence evaluation: the completeness of the component’sinterfaces description is evaluated.



Software Coding analysisSoftware Coding analysisSoftware Coding analysisSoftware Coding analysistoolstoolstoolstools

§ SPLINT§ QA-C MISRA§ Checklists aiming at verifying:

§ The conformace to the best practices in software coding

§ Accuracy evaluation: the data type definition is checked out.§ Maturity evaluation: programming errors are measured§ Analyzability evaluation: the readability of the code and the degree

of explanatory information provided in it is measured.§ Changeability evaluation: the readability of the code is measured.



Software Testing analysisSoftware Testing analysisSoftware Testing analysisSoftware Testing analysistoolstoolstoolstools

§ Checklists aiming at verifying:§ The development of test cases covering the software

requirements§ The existence and the application of a regression test

strategy§ The existence and application of an integration testing

strategy§ The understandability and completeness of test logs

§ Maturity evaluation: test adequacy and fault detection metricsare calculated.

§ Analyzability evaluation: the accuracy and the completenessof test logs is evaluated.



Definition of the measurementDefinition of the measurementDefinition of the measurementDefinition of the measurementscale and profilesscale and profilesscale and profilesscale and profiles

§ The measurement scale is common for all the sub-characterisitcsto be evaluated:

§ T: top (the sub-characteristic has to reach the top score in all therelated measurements)

§ H: high (the sub-characteristic has to reach at least 75% of the topscore)

§ M: medium (the sub-characteristic has to reach at least 25% of thetop score)

§ N: not relevant


suitabilityinteroperabilityaccuracym

aturityanalyzabilitychangeabilitytestabilityco-existence

expected profile

resulting profile

§ The collection of the evaluationresults of the sub-characteristics(resulting profile) is compared withthe expected profile.


Phase 3: ObjectivesPhase 3: ObjectivesPhase 3: ObjectivesPhase 3: Objectives

§ The third phase of the ESCAPE project aims at validating theevaluation plan defined in the previous phase, by providing a generalmethodology to allow FIAT Auto to monitor in a quantitative way asupply during the development process and ask for possible correctiveactions earlier.

§ Possible critical factors in Phase 3:§ it takes a long time, especially in the verification of quality

characteristics such as reliability or maitainability;§ it needs a comparison between products having a different metrics

profiles in order to determine if and which correlation existsbetween metrics and quality;

§ it needs a large set of product quality reports to achieve a soundstatistic evaluation

PhasePhasePhasePhase 3 3 3 3: Validation of the MethodologyValidation of the MethodologyValidation of the MethodologyValidation of the Methodology


Validation mechanismValidation mechanismValidation mechanismValidation mechanismPhasePhasePhasePhase 3 3 3 3: Validation of the MethodologyValidation of the MethodologyValidation of the MethodologyValidation of the Methodology

0

1

2

3

S I A M A C T C

0

1

2

3

S I A M A C T C

COMPLIANCECOMPARISON

FINALPRODUCT

EX-POSTEVALUATION

PRODUCT’SQUALITY

EXPECTATIONS

PRODUCT’SQUALITYRECORDS

COMPLIANCECOMPARISON

NOK

OK•Inadequateexpectedprofile•Inadequatemetricationplan

ResultingProfile

ExpectedProfile

SW Supplier’sDevelopment

Process


Thank you.Thank you.Thank you.Thank you.Questions?Questions?Questions?Questions?

[email protected]@[email protected]@fiat.com

Documents

Improving the Management of the Software Acquisition ... · Process: a Methodological Approach in Automotive ... Cruise control Antilock brakes ... Lancia and Alfa Romeo vehicles