01

cally, the diagnostic services implemented in the ECU must be known. The CANdelaStudio tool from Vector Informatik GmbH is globally established for specifying diagnostics. Among other things, it supports the international Open Di-agnostic data eXchange standard (ODX, ISO 22901) [3], that was released in 2008 and is now used worldwide by most vehicle manufacturers.Thus the technical prerequisites for automatic generation of diagnostic protocols have been met for a long time al-ready. Therefore, it is no wonder that there have been solu-tions for automatic diagnostic test generation and execu-tion for over 10 years already, such as the CANoe.DiVa tool from Vector.

Very comprehensive tests are generated automatically on the basis of diagnostic information in CANdela or ODX for-mat and the diagnostic protocol (e.g. UDS, KWP2000, OBD, WWH-OBD, …). These tests cover both valid and in-

The release of the Unified Diagnostic Services Standard (ISO14229) [1] in 2006 represented a significant step for the automation of diagnostic tests: with the harmoniza-tion of the diagnostic services, it became possible for the first time to implement cross-manufacturer diagnostic protocol tests in greater depth. The previous protocol KWP2000 (ISO14230) [2] still allowed a great deal of free-dom and thus must be made more concrete in an OEM-spe-cific way, that makes generally valid test implementation difficult. A revised version of the UDS was published in 2013, which makes it possible to formulate even more de-tailed protocol tests.Besides the protocol definition, formally described diag-nostic data represents the second important prerequisite for test generation. The functional scope of a power win-dow differs essentially from that of an engine ECU – and the implemented diagnostic functionality also differs ac-cordingly. In order to be able to generate a test automati-

Technical articles on diagnostic validation often begin as follows: Along the complexity of the ECUs in vehicles increases, the scope of diagnostics is also increasing, and with it the time and effort for diagnostic validation. However, thanks to current testing tools, the time and effort is no longer increasing linearly – fortunately. In particular, it has actually been possible for several years to generate and execute diagnostic protocol tests fully automatically. Despite a further in-crease in the functional scope, the time and effort for creating and executing tests remains fairly constant. With the established standards for unified diagnostic communication (UDS) and diagnostic description formats (ODX), it is pos-sible to achieve a comparatively high degree of test automation, and high diagnostic quality as a result. However, testing of the diagnostic application and the update software can also be automated. These days there are few areas in the development of automotive electronics with a similarly high potential for test automation.

... Just a Matter of Consequent Exploitation of Existing Possibilities

Automatic Diagnostic Validation is not Rocket Science

02

Technical Article / 03/2017

valid requests that put the error handling in the ECU to the test. The tests are loaded into the test environment (CA-Noe) and executed automatically. The test results are doc-umented in a detailed report [Figure 1]. For larger ECUs over 10,000 test cases are generated quickly without re-dundant testing.

Insert figure

Figure 1: Function blocks of the CANoe.DiVa automated test environment

In practice there is now a formal diagnostic description in the CANdela and/or ODX format for nearly every ECU. Therefore, it is a small step from the diagnostic specifica-tion to a comprehensive, automatic protocol test. Protocol errors can be detected for a long time simply with little ini-tial effort.

AUTOSAR Increases Diagnostic Protocol QualityWhen AUTOSAR software components are used for diag-nostics, a series of protocol violations by the tester (e.g., format violations) are handled by the basic software al-ready. Since the basic software is usually parameterized directly with the available diagnostic data, the diagnostic responses of an ECU are usually also protocol-compliant. The proportion of simple protocol errors falls noticeably when AUTOSAR components are used. The time and effort for test evaluation and troubleshooting fall accordingly. Nonetheless, the protocol test remains important for the following reasons among others:

> Errors can occur during configuration of the AUTOSAR components, integration, and application development > If the diagnostic implementation in the ECU does not match with the diagnostic data used by the diagnostic tester, correct diagnostics in the ECU is not possible despite “correct” implementation > Gaps in the error handling of the application implemen-tation can potentially be used to install malicious code

Since, in times of DoIP and OTA (“over the air”), no direct physical connection with the diagnostic tester is required, a protocol error may have a negative effect on these services for an entire fleet of vehicles, for example, due to limited

options during remote maintenance, because the required diagnostic functions do not work as desired, so time in the workshop is necessary (after all). In the worst case, a pro-tocol error could permit a safety-critical attack on the vehi-cle. Due to continuous availability of the vehicle, new use cases that the customer can experience will develop based on diagnostic functionality, which will presumably lead to more frequent and broader use of the diagnostic functions. It is thus foreseeable that the quality requirements on the diagnostic implementation will continue to increase.

Software Update ValidationDespite all the diagnostic standardization measures, the processes for updating ECU software (flashing) remain OEM-specific. Although the services used are standard-ized, functions such as incremental update or mechanisms for protecting the transmitted data (identification data, checksums, signatures, etc.) lead to quite different flash procedures in practice. Furthermore, different flash con-tainer formats are used in order to meet the process re-quirements of the respective manufacturers. In practice that means that there is also a specific software update tool for nearly every OEM. Therefore a supplier must not only work with various tools but often also develop and maintain a specific test environment for each OEM. A functioning flash procedure has always been indispens-able in production and service. Taking future software up-dates “over the air” (SOTA) into consideration, a reliable software update function in vehicles is more important than ever. With the new possibilities, new software will presumably be loaded more frequently – possibly at a fre-quency similar to what we are already familiar with for mobile devices (e.g., to fix security vulnerabilities). Situa-tions in which an ECU no longer functions after a failed update should definitely be avoided, of course. But even for established smartphone manufacturers, software up-dates do not always run smoothly – despite sophisticated protective measures.Cross-manufacturer software updates have been per-formed for several years already with the reprogramming tool vFlash from Vector, currently for over 90 different boot loader specifications of all relevant vehicle manufacturers. The test automation tool CANoe.DiVa uses the vFlash au-tomation interface and thus additionally offers automated software update tests for all boot loaders supported by vFlash. Besides the variables (timing, format, ...) in the val-id flash procedure, classical error situations are also tested automatically, such as undervoltage and interruptions at various points in the flash procedure. This allows the ro-bustness of the ECU software to be tested easily and com-prehensively. The tests sketched are essentially black box tests. Tests be-yond these, e.g. plausibility checks for process-relevant

03

Technical Article / 03/2017

data such as identification data or signatures, require manufacturer-specific detailed knowledge. There are al-ready specific extensions for well-known manufacturer for this type of test. The software update tests ensure the essential update features such as reliable data transfer and robust behavior in the event of errors. The convenience of automation for them is similar to that for the diagnostic protocol tests.

Validation of the Diagnostic ApplicationThe diagnostic functions in the ECU must not only satisfy formal aspects but also have correct content, of course. This is the only way to ensure sensible vehicle diagnostics. The current diagnostic formats (e.g., ODX) focus especially on the definition of the protocol aspects. Nonetheless, ap-plication-relevant information can be extracted from them for automatic test generation: for example, an automatic test of the plausibility of the transferred values can be de-rived from the description of the specified value ranges. Specific error causes can be derived from informal error codes or diagnostic parameters with heuristic methods and/or manufacturer-specific knowledge. In order to be able to put the insights obtained in specific automatic test runs into practice, the options for access to the (test) envi-ronment, and thus the options for simulation of the ECU, must be known. However, this information is usually avail-able during ECU development in any case:

> The network architecture is described in AUTOSAR (.arxml) or CANdb (.dbc). Signals can be read from the bus easily and manipulated through a remaining bus simulation. > The memory layout of an ECU is described in a2l files. Parameters can be read and manipulated easily via XCP. > The electrical inputs and outputs and the test configura-tion are usually described in machine-readable formats. These can be measured and triggered, e.g. through a Hardware-in-the-Loop (HiL) test in combination with the Vector VT system [Figure 2].

Furthermore, CANoe.DiVa supports the coupling of propri-etary parameters and DTC descriptions through an Excel exchange format. In an integrated test environment like Vector CANoe, all these sources can be combined and used in one test. When a virtualized ECU is used in development (e.g., Vector vVIRTUALtarget), the test setup for the application test is comparatively simple, since hardware I/Os can be simulat-ed easily in software.The degree of possible automation during application tests certainly does not reach that of protocol tests; how-ever, numerous options arise for semiautomatic or even fully automatic test generation, which it pays to take ad-vantage of.

Test Scope, Test Evaluation, and Further ProcessingThe high degree of automation for diagnostic protocol validation achieved through corresponding tools permits a greater test depth through further automated tests or additional tests of the user’s own – with constant time and effort: more and more OEMs are taking advantage of the automation options in order to systematically safe-guard the diagnostic application cases, e.g. for production and service, that are especially important to them. Corre-spondingly, the diagnostic function can be demonstrably safeguarded in early development phases already by the supplier. Many of the largest OEMs around the world now rely on the test support of CANoe.DiVa. In the process they also bene-fit from the comprehensive support for further processing of the test results, which in practice saves a great deal of time [Figure 3]:

Insert figure

Figure 3: Practical experience: testing time and effort for manual validation compared to automated validation [5]

> Sorting and filtering, for a targeted and needs-based view of the test resultsThe memory layout of an ECU is described in a2l files. Parameters can be read and manipulated easily via XCP. > Linking errors with the same cause

Insert figure

Figure 2: CANoe.DiVa: Software and hardware in-the-Loop

04

Technical Article / 03/2017

Christph Rätz Christoph Rätz is the head of the Diagnostics product line at Vector Informatik GmbH in Stuttgart.

Simon Müller Simon Müller studied software engineering at the University of Stuttgart. He is the product manager responsible for CANoe.DiVa at Vector Informatik GmbH in Stuttgart in the Diagnostics product line.

Translation of a German publication in Hanser Automotive issue 3-4/2017

Image rights: Vector Informatik GmbH

Literature References:[1] - [3] International Organization for Standardization: ISO 14229 (UDS), ISO 14230 (KWP2000), ISO 22901 (ODX)[4] Metzger E.; 2016: Presentation: The Vector Security Solution, Vector Cyber Security Symposium 2016[5] Peti P., Timmerberg A., Pfeffer T., Müller S., Rätz C.; 2008: Automatic validation of diagnostic services, ATZ elektronik 06I2008

> Adding comments to individual test results in order to classify the error and its cause, documenting notes on corrective measures, and controlling troubleshooting. > Transferring the error analysis results from one test run to another > Connecting the test solution to existing test data management or requirement systems for integration into existing processes

Conclusion/outlookThe diagnostic scope of ECUs and the quality requirements will continue to increase. However, automated solutions al-ready exist today that significantly reduce the time and ef-fort for diagnostic validation and, at the same time, signifi-cantly increase the test breadth and depth. The formally described data required for this is usually available in any case. The reuse of this data for test purposes as well is sim-ply logical.With the increase in networking beyond the boundaries of the vehicle, the visibility of the topic of automotive cyber security is also increasing rapidly. “Testing security” and “testing despite security” [4] have now become topics of focus in many areas of development. Expanded vehicle ac-cess “over the air” opens up new application domains for vehicle diagnostics – which must be safeguarded in turn. Many innovations can only be brought into serial produc-tion if they are supported in software and the central tools from early on in an user-friendly way. This is not rock-et science either – just an exciting challenge for software vendors.