Testing Strategies for Asymmetric Environments · Use techniques such as testing stubs, using...

V1.0 | 2019-05-29

Jeffrey Fortin, Product Manager, Code Testing

Testing Strategies for Asymmetric Environments

2

1. Asymetric Environments

2. Testing in an Asymetric Environment

3. Summary

4. Questions

Agenda

3

Software Based Systems are Evolving RapidlyAsymetric Environments

• Hundreds of files• Thousands of Lines of Code• Lots of “connections”• Lots of safety concerns

SOFTWAREEMBEDDED SOFTWARE

+SIZE

ENTERPRISESOFTWARE

+Connections

• Thousands of files• Millions of Lines of Code• Few “connections”• Few safety concerns

+Safety

4

Today’s challenge: Meet quality objectives while controlling costAsymetric Environments

5

Today’s challenge: Meet quality objectives while controlling costAsymetric Environments

Code size expandingSame regulatory requirementsSame or higher quality expectationsSame budgets and schedules

A strategy is needed …

6

Environments and DevOpsAsymetric Environments

Development• New Code• Unit Tests• Static Analysis• Code Coverage criteria met

• Development Host

Integration

• Latest code base• Updated continuously

QA• Integration Tests• Regression Tests• Updated Usually Overnight

Staging

• Simulate the production environment

• Perform tests that can’t be done sooner

Production• End user is now your tester

• Bug reports• External monitoring

7

1. Asymetric Environments

2. Testing in an Asymetric Environment

3. Summary

4. Questions

Agenda

8

Multiple ConfigurationsTesting in an Asymetric Environment

Development• Windows Host• Microsoft Visual Studio

Integration• Linux Build Server• GCC Compiler

QA• Docker• Simulator• Cross Compiler

Staging• Sim/Stim• HIL• Signal Generators• Network Simulation• Reference Hardware

Production• Device Management

• Embedded “Target”

9

Typical Embedded Code ExampleTesting in an Asymetric Environment

typedef unsigned long uint32_t;/* Memory Mapped Hardware Registers with *special* properties */uint32_t volatile REG1;static inline uint32_t getREG1();

/******************************************************************************************//* take action based on the value in REG1 and REG2 */int avoid_collision() {

if ((REG1 & 0xFF00) & 0xFE00) {return 1;

}return 0;

}

Can the code logic be tested early?

10

Embedded Code Example Refactored for TestingTesting in an Asymetric Environment

/******************************************************************************************//* Better implementation that allows for stubbing of the special registers */inline uint32_t getREG1() {

/* Using *inline* should give same performance as macros or inline register access */return REG1;

}int avoid_collision_api() {

/* Notice calls through stub-able accessor functions */if ((getREG1() & 0xFF00) & 0xFE00) {

return 1;}return 0;

}

typedef unsigned long uint32_t;/* Memory Mapped Hardware Registers with *special* properties */uint32_t volatile REG1;static inline uint32_t getREG1();

11

Simulating Hardware StatesTesting in an Asymetric Environment

Temperature is a 2’s compliment binary

Using a stub routine, we can easily test for any temperature setting

We can also test for “impossible” bit combinations

12

Injecting FaultsTesting in an Asymetric Environment

Bits 26-30 represent fault conditions

Testing with stubs we can stimulate the code to respond to the fault

We can even set multiple fault conditions

13

Call to getHW_TEMP_STATUS()

Testing ExampleTesting in an Asymetric Environment

TestDriver

ModuleUnder

Test

DummyFunctions

(stubs)

RealFunctions

14

Using Test Automation in the Development EnvironmentTesting in an Asymetric Environment

Development Environment is resource richThe automation system can cycle through

hundreds of test parameters and re-run tests on any code that has changed

Tests can be setup to run on command by anyone

15

Collecting Testing Metrics in the Development EnvironmentTesting in an Asymetric Environment

Get early indication of quality risksTesting coverage is a key metricOther metrics help to make better decisions

16

1. Asymetric Environments

2. Testing in an Asymetric Environment

3. Summary

4. Questions

Agenda

17

Leverage the Development Environment as much as possible

Use techniques such as testing stubs, using multiple compiler configurations and test automation

By doing so, embedded applications can be developed more efficiently and at lower risk than by only testing in the final Production Environment.

ConclusionSummary

18

1. Asymetric Environments

2. Testing in an Asymetric Environment

3. Summary

4. Questions

Agenda

19

Testing Strategies for Asymmetric EnvironmentsQuestions

!?

20 © 2019. Vector Informatik GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.0 | 2019-05-29

Author:Fortin, JeffreyVector

For more information about Vectorand our products please visit

www.vector.com