Towards automatic Model-In-the-Loop testing of ElectronicVehicle Information Centers

Domenico AmalfitanoAnna Rita FasolinoPorfirio Tramontana

Stefano Scala

TESTBEDS 2014 – Västerås, Sweden – September 16

ContextGUI Testing automation of Automotive

Embedded Systems.Automotive Embedded Systems are

composed by Electronic Control Units (ECUs).

TESTBEDS 2014 – Västerås, Sweden – September 16

RequirementsRequirements Analysis

ArchitectureFunctionality Sub-Systems Definition Interface Specifications

Model Based DesignSimulink/Stateflow Model Implementation

Auto CodingC Code Automatic Generation

Code ValidationSoftware in the Loop (SIL) test (Automatic)

Model ValidationModel in the Loop (MIL) test (Manual/Automatic)

HILHardware in the Loop (HIL) test

Model-In-the-Loop (MIL) testing of Electronic Vehicle Information Centers (EVICs)

FIAT ECU’s Model Based Development Life Cycle

EVIC

TESTBEDS 2014 – Västerås, Sweden – September 16

Excerpt of the Instrument Panel Cluster (IPC)

EVIC

It features a driver interactive display and optionally a trip computer;It assists the driver providing a set of information about the engine behavior and the state of mechanical and electrical components of the vehicle;It is placed within the IPC that provides the human-machine-interaction of the vehicle;

EVIC Model Based Design An EVIC is actually an event driven system (EDS)

◦ It receives inputs from the vehicle, processes them and returns output through its display.

The EVIC display can be considered as a UI showing textual messages, simple and more complex graphical icons.

TESTBEDS 2014 – Västerås, Sweden – September 16

Architecture of the EVIC Model

Example of EVIC model

MIL TestingMIL testing is used as a validation

technique. ◦It is based on the execution of

repeatable test sequences in a real time model simulation environment.

TESTBEDS 2014 – Västerås, Sweden – September 16

The current MIL testing approachTest Engineer specifies the tests manually.Each test consists of a Time History of input

signals to be sent to the EVIC's model and of a timed sequence of expected EVIC's display screen-shots.

Test Engineer manually runs the tests in the simulation environment and compares the rendered EVIC against the expected screen-shot.

This activity is time consuming and error-prone.

TESTBEDS 2014 – Västerås, Sweden – September 16

The addressed problemEVIC models are actually

executable files.◦When launched they provide a GUI

emulating the EVIC behavior.

It’s difficult to apply traditional GUI testing techniques.◦I.e. capture and replay.

How is it possible to evaluate the GUI oracles? TESTBEDS 2014 – Västerås, Sweden –

September 16

The proposed solutionA new testing process based on visual

testing techniques allowing the automatic verification of the testing results was proposed.

It relies on a different specification of the test cases.

The new test case format includes the Time Histories of input signals and a set of EVIC layout descriptions that are expected as output at specific times.

TESTBEDS 2014 – Västerås, Sweden – September 16

EVIC DescriptionEVIC is descried in terms of

zones, items, and related features.◦A zone is a well-defined area of the

display.A zone can contain only one

graphical object, or item.An item has a set of attributes

named features, having expected values. TESTBEDS 2014 – Västerås, Sweden –

September 16

The considered itemsAn EVIC Zone can render

three types of item:◦Text Item: rendering a text

in a given position of the zone.

◦ Icon Item: showing a graphical icon in a given position of the zone.

◦Complex Item: showing graphical object composed both by dynamic text and icons and is able to change configuration at run-time

TESTBEDS 2014 – Västerås, Sweden – September 16

Test CasesA Test Case links together a

Matlab Time History and a set of EVIC layout descriptions.

EVIC layout descriptions are serialized and stored in one or more Excel files.

Test cases are automatically translated into executable Matlab scripts named EVIC test scripts.

TESTBEDS 2014 – Västerås, Sweden – September 16

Example of EVIC Description

TESTBEDS 2014 – Västerås, Sweden – September 16

Automatic Testing Results EvaluationEVIC test script are executed in the

Matlab simulation environment and solicit the EVIC by the signals defined in the time histories.

During the script execution, at the required sampling time ti a snapshot of the EVIC is saved in graphical format.

At run-time, the snapshot is automatically analyzed in order to verify if it satisfies or not the expected layout representation reported in the Excel file.

TESTBEDS 2014 – Västerås, Sweden – September 16

How to test The Items rendered by the EVIC snapshot

are automatically isolated and selected.◦By exploiting the layout structural information

(i.e., position of the zones, dimension of the zones, position of the item in the zone, dimension of the item).

Selected Items must be evaluated. ◦verify if the they have the expected features

Three image processing techniques, one for each type of item, were proposed.

TESTBEDS 2014 – Västerås, Sweden – September 16

Icon Item TestingIcon Items are considered as Truecolor

images.The coefficients of correlation between

the three colors bands of the item under test and reference pictures were evaluated.

Two icons are considered equivalent if all the three coefficients are greater than 0.95.

The test passes if the selected item is considered equivalent to the expected picture. TESTBEDS 2014 – Västerås, Sweden –

September 16

Icon Item Testing

TESTBEDS 2014 – Västerås, Sweden – September 16

Text Item TestingIt is verified if the item shows the

expected text. ◦Optical Character Recognition (OCR)

feature provided by the toolbox.

The test passes if the rendered text is the expected one.

TESTBEDS 2014 – Västerås, Sweden – September 16

Complex Item TestingTo test a complex item the extraction

of its dynamic characteristics is needed

A specific feature extractor technique for each complex item was proposed. ◦Feature extractor identify specific

graphical properties of the complex graphical objects.

The test passes if the evaluated value matches with the expected one.

TESTBEDS 2014 – Västerås, Sweden – September 16

The value expressed by the indicator is extracted. ◦The extractor module analyzes the

acquired image and returns a value belonging to the range [0; 1].

TESTBEDS 2014 – Västerås, Sweden – September 16

Complex Item Testing

Examples of test execution

EvaluationA preliminary evaluation of the proposed testing

approach was carried out.The behavior of an EVIC, already manually tested,

was verified by exploiting the new process. Time Histories were reused to produce the new

test cases.◦ Test cases were enriched with the descriptions of the

expected EVICs.191 new test cases were produced and

automatically translated and executed. All the test cases passed the testing process

worked as expected on a real industrial case study.

TESTBEDS 2014 – Västerås, Sweden – September 16

ConclusionsWe presented a novel MIL testing

process aimed at automatizing the verification activity, performed in the automotive industry, of the EVIC component.

A preliminary evaluation showed us that it is possible to apply in an industrial context the Visual Testing approaches already exploited in other different fields.

TESTBEDS 2014 – Västerås, Sweden – September 16

Future WorksIncrease the experimentation.Apply the proposed approach

during the EVIC development process.◦To evaluate the possible benefits that

an industrial MIL process may have by adopting it.

Propose aided editing approaches to support the tester for the definition of test cases.

TESTBEDS 2014 – Västerås, Sweden – September 16

Thank you

TESTBEDS 2014 – Västerås, Sweden – September 16