Intelligent Interleaving of Scenarios:A Novel Approach to System Level Test Generation

Shady Copty, Itai Jaeger(*), Shady Copty, Itai Jaeger(*), Yoav Katz, Michael VinovYoav Katz, Michael Vinov

IBM Research Laboratory in IBM Research Laboratory in HaifaHaifa

2

OutlineOutline

System-level stimuli generation challengesSystem-level stimuli generation challenges Interaction based testcase generationInteraction based testcase generation X-Gen: a system-level testcase generatorX-Gen: a system-level testcase generator Scenario interleaving in X-GenScenario interleaving in X-Gen Real life experience examplesReal life experience examples

– eServer, Cell BE, Xbox CPU chipeServer, Cell BE, Xbox CPU chip

SummarySummary

3

System verificationSystem verification

System verification is System verification is aimed at validating the aimed at validating the integrationintegration of several of several

previously previously verified verified corescores in a in a relatively relatively

short timeshort time

4

Challenges in Stimuli Generation forSystem Level Functional Verification

Challenges in Stimuli Generation forSystem Level Functional Verification Specifying system level scenarios in an abstract Specifying system level scenarios in an abstract

formform–While generating the required low level stimuliWhile generating the required low level stimuli

Generating coordinated system-level stimuliGenerating coordinated system-level stimuli–Projected to each and every core in the systemProjected to each and every core in the system

Effectively handling configuration changesEffectively handling configuration changes–2-way system vs. 8-way system2-way system vs. 8-way system

Adapting to core modifications and new coresAdapting to core modifications and new cores–PCI PCI PCIe PCIe

5

The concept of interactionThe concept of interaction Capturing the essence of the system-level Capturing the essence of the system-level

functionalityfunctionality– CPU-to-memory read/writeCPU-to-memory read/write– CPU-to-CPU inter processor interruptCPU-to-CPU inter processor interrupt– CPU initiated MMIOCPU initiated MMIO– IO initiated DMAIO initiated DMA– ……

Interaction: acts, actorsInteraction: acts, actors Independent of the system’s configurationIndependent of the system’s configuration Agnostic to specific core detailsAgnostic to specific core details

6

A DMA Interaction

The concept of interaction - exampleThe concept of interaction - example

A CPU stores to the doorbell register of the DMA engine

PLB

CPU#4CPU#3CPU#2CPU#1

DMAEngine

BridgeInterrupt Controller

Memory

PHB

IO BFM#2

IO BFM#1

The data is moved from the IO port to memory

The DMA engine interrupts the initiating CPU

7

Interaction based scenariosInteraction based scenarios

Stress the system bus through address contention on requests from multiple devices

Verification goal

Generate multiple IO reads, DMAs, and processor accesses to the same address

Test plan definition

Generate contention on the system busInteraction based scenario

8

X-Gen: a model-based system-level stimuli generator

X-Gen: a model-based system-level stimuli generator

Test Template

*.rqst

X-Gen Engine

Interactions

(Transactions) Configuration

(Topology)

Components

(Cores)

Abstract System Model

Test Case

CSP Solver

Specifying an interaction-based

scenarios

9

Interaction as the basic Interaction as the basic building blockbuilding block

Control over:Control over:– Participating coresParticipating cores– Interaction propertiesInteraction properties

A hierarchy if higher A hierarchy if higher level statementslevel statements– One-of: weighted One-of: weighted

random choicerandom choice– RepeatRepeat– All-ofAll-of

Request file

All of

Repeat x10

Load / store DMA interrupt

Target: mem2 Addr: 0x12?? Priority: 12-14

One of

Weight: 70 Weight: 30

A system-level test templateA system-level test template

10

A system-level test caseA system-level test case

Processors

• Instructions

• Registers initializations

• Translation tables

Memory

• Data initializations

Behaviorals (BFMs)

• Internal memory initializations

• Commands, such as:

•Send an interrupt

•Initiate a DMA

Hubs, bridges, adaptors

• Translation tables

• Data descriptors for send / receive operations

• Internal registers initializations

PLB

CPU#4CPU#3CPU#2CPU#1

DMAEngine

BridgeInterrupt Controller

Memory

PHB

IO BFM#2

IO BFM#1

11

Combining scenariosCombining scenarios Stressing the system through parallel Stressing the system through parallel

execution of several system level execution of several system level scenariosscenarios

For example:For example:– Read/write cache coherency from multiple Read/write cache coherency from multiple

processorsprocessors– MMIO / DMA to IO portMMIO / DMA to IO port– Inter processor interruptsInter processor interrupts

– Create address contention between DMAs Create address contention between DMAs and processor accessesand processor accesses

– Send interrupts to processors contending Send interrupts to processors contending on the same cache lineon the same cache line

12

CPU_to_memory_data_tarnsfer

IO_to_memory_DMA

CPU_to_memory_data_tarnsfer

CPU_to_memory_data_tarnsfer

IO_to_memory_DMA

IO_to_memory_DMA…

Scenario interleaving in X-GenScenario interleaving in X-Gen

CPU.rqst

- Repeat 80- CPU_to_memory_data_tarnsfer

IO.rqst

- Repeat 70 - IO_to_memory_DMA

X-GenTotal: 150

interactionsAccessing the same address

13

eServer system verification: Partitioned system-level testtemplate library

eServer system verification: Partitioned system-level testtemplate library

Cache coherency scenarios

PCI bridge stress scenarios

InfiniBand message passing

scenarios

Intervention scenarios

X-Gen

Maintenance saving:

avoiding exponential test template growth

14

Xbox 360 chip verification :Reusing scenariosXbox 360 chip verification :Reusing scenarios

eServer MP/MT scenarios

Xbox specific IO scenarios

X-Gen

Heavy vIP reuse:generating complex

system-level testcases within weeks from verification launch

15

General cell scenarios

Power management

function

Performancemanagement

function

X-GenMain scenario

Main scenario

Power down unit XXX

Main scenario

Main scenario

Restart unit XXX

Main scenario…

Cell BE verification:Verifying pervasive functionsCell BE verification:Verifying pervasive functions

Main scenario

Slow down unit YYY

Main scenario

Power down unit XXX

Main scenario

Main scenario

Resume frequency on unit YYY

Restart unit XXX

Main scenario…

16

SummarySummary Interleaving transaction-level scenarios is crucial Interleaving transaction-level scenarios is crucial

for system-level verificationfor system-level verification Partitioning the test plan according to different Partitioning the test plan according to different

system functions system functions – Avoids costly maintenance of test templatesAvoids costly maintenance of test templates– Promotes reuse of verification IPPromotes reuse of verification IP– Fast and effective verification of pervasive Fast and effective verification of pervasive

functions in conjunction with “mainstream” functions in conjunction with “mainstream” scenariosscenarios

Implemented in X-GenImplemented in X-Gen– IBM’s system-level testcase generatorIBM’s system-level testcase generator– Used in the verification of eServer, Xbox & Cell Used in the verification of eServer, Xbox & Cell

systemssystems

Thank You !!!

18

Traditional Approach #1:Combining Lower Level DriversTraditional Approach #1:Combining Lower Level Drivers

AdvantagesAdvantages– SimpleSimple– Quickly adapts to new Quickly adapts to new

corescores– Reuse of core VIPReuse of core VIP

DisadvantagesDisadvantages– Not system level Not system level

verificationverification• No coordinated No coordinated

stimulistimuli• No system level No system level

scenariosscenarios

IP1

Driver

IP2

Driver

IP2

Driver

Driver Driver Driver BUS

19

Traditional approach #2:Transaction Based VerificationTraditional approach #2:Transaction Based Verification

AdvantagesAdvantages– System level System level

abstractionabstraction– Allows complex Allows complex

coordinated coordinated stimuli stimuli

– Allows reuse of test Allows reuse of test specificationspecification

DisadvantagesDisadvantages– Transactor code is Transactor code is

monolithicmonolithic– Difficult to adapt to Difficult to adapt to

configurationconfigurationchangeschanges

– Difficult to adapt to Difficult to adapt to newnewcomponents / components / changedchangedfunctionality functionality

IP1

Transactor

BUS

IP2 IP2

Driver

Transactor

20

Request files: a dual effort methodologyRequest files: a dual effort methodology

Specifying a scenarioSpecifying a scenario Interactions as building blocksInteractions as building blocks Restrict actors, propertiesRestrict actors, properties Inter-interaction relationsInter-interaction relations

Request file

All of

Repeat x10One of

DMA

transfer

CPU load /

storeInterrupt

Read: 80

Write: 20

Address

Collision: 65%

The Bug

Intelligent background noiseIntelligent background noise Built-in testing knowledgeBuilt-in testing knowledge User directionUser direction