MicroTESK : Automation of Test Program Generation for Microprocessors

MicroTESK: Automation ofTest Program Generation for Microprocessors

Alexander [email protected]

Institute for System Programming of the Russian Academy of Sciences (ISPRAS)http://hardware.ispras.ru

IEEE East-West Design and Test Symposium - September 18-21, 2009 - Moscow, Russia 2 of 23

What does “MicroTESK” stand for?

Micro = Microprocessor Microprocessor = programmable device

Programmable = program-controlled device

TESK = TEsting and Specification ToolKit Testing = simulation-based verification Specification = specification-driven approach Toolkit = computer-aided automation

MicroTESK is a specification-driven (model-based) approach that automates simulation-based verification of programmable devices by means of test programs


Levels of microprocessor verification

lui s1, 0xdead

ori s1, s1, 0x0

lui s3, 0xbeef

ori s3, s3, 0xf

add v0, a0, a2

sub t1, t3, t5

add t7, s1, s3

Core-level verificationUnit-level verification

Model of a microprocessor is tested as a whole

with the help of test programs

Model of a particular unit is tested

via inputs and outputs signals


Test programs-based verification

Target Design(RTL, FPGA)

lui s1, 0xdeadori s1, s1, 0x0lui s3, 0xbeefori s3, s3, 0xfadd v0, a0, a2sub t1, t3, t5add t7, s1, s3

Test programs(assembler, C)

Reference simulator (C/C++, ADL)

Execution traces(formatted text)

0x2000: lui ...0x2004: ori ...0x2008: ori ...0x200c: lui ...0x2010: add ...0x2014: sub ...0x2018: add ...

Trace comparator (Perl, C/C++)

0x2000: lui ...0x2004: ori ...0x2008: ori ...0x200c: lui ...0x2010: add ...0x2014: sub ...0x2018: add ...


Approaches to test program construction

IF ID EX WB

IF ID EX WB

Requirements

Design

Unformalized knowledge

Formalized knowledge

Test programsManual Development

Random Generation

Template-Based Generation

Model-Based Generation

?

lui s1, 0x2779ori s1, s1, 0x0lui s3, 0x4eeori s3, s3, 0xfadd v0, a0, a2sub t1, t3, t5add t7, s1, s3


Advantages of model-based testing

Formalization helps to find problems in a design Deeper comprehension of requirements Formal checking of model consistency

Models automate verification of a design Correctness estimation Test sequence generation Testing adequacy criterion

Models simplify tests maintenance One model, many tests Reuse of models and testing knowledge


Approaches to design modeling

ADD x, y, z @ z ≥ 0LD u, x, z @ Cache_HitDIV y, u, x @ Division_By_Zero...DIV.S f, g, h @ Hard_To_RoundADD.S i, f, h @ InexactMUL.S g, i, f @ Invalid

Instruction-level modelsCycle-accurate models

Test templates are used for verificationTo generate tests, FSM models are used

ABS.fmt (fd:FPR, fs:FPR) {...}

ADD (rd:GPR, rs:GPR, rt:GPR) {...}

ADD.fmt (fd:FPR, fs:FPR, ft:FPR) {...}

...XOR (rd:GPR, rs:GPR, rt:GPR) {...}

XORI (rd:GPR, rs:GPR, rt:IMM) {...}


Combinatorial test program generation

Tests are constructed by combining Sequences of instructions Test situations for instructions Dependencies between instructions Execution traces, if branches are used

Volume of tests is reduced by heuristics Equivalence of instructions Length of sequences Depth of dependencies Length of execution traces


Structure of test programs

initialization

test case

…

test action

…

load a, (b)add c, a, dload e, (f)

move c, 1

store t, (b)move t, 0move b, 0x4

move f, 0x4

store t, (f)move t, 2

move g, 0xd

...bne c, g, error test oracle

address dependencyregister dependency

test situation


Instruction specification

Instruction interface Instruction operands Instruction precondition

Execution function Calculates outputs Updates a model state

Assembler format Textual representation

temp (GPR[rs]31||GPR[rs]31..0) + (GPR[rt]31||GPR[rt]31..0)if temp32 temp31 then SignalException(IntegerOverflow)else GPR[rd] sign_extend(temp31..0)endif

if NotWordValue(GPR[rs]) or NotWordValue(GPR[rt]) then UNPREDICTABLEendif

output rd: GPR, Wordinput rs: GPR, Wordinput rt: GPR, Word

ADD rd, rs, rt

instruction ADD(rd, rs, rt)


Test situations and dependencies

Parameters Specifies family of test situations / dependencies

Constructor / constraint Constructs / constrains values of the operands

Preparator Creates initialization program of the instruction

if NotWordValue(GPR[rs]) or NotWordValue(GPR[rt]) then UNPREDICTABLEendiftemp (GPR[rs]31||GPR[rs]31..0) + (GPR[rt]31||GPR[rt]31..0)if temp32 temp31 then SignalException(IntegerOverflow)else GPR[rd] sign_extend(temp31..0)endif

0x0af1036f0x1

0xf47298b90x3

0xffff0a310x00xdeadbeef0xd

datatag

0x19d02c340xa

0x7420ba9c0x2......

Cache (L1/L2) TLB

0xa0000x0000 0xb0200x0001 ...... 0xa0100x003e 0xb0000x003f

PFNVPN


load ?, (?)

add ?, ?, ?load ?, (?)

Test program generation scheme

load a, (b)

add c, a, dload e, (f)

cache miss

cache hit

normal

move c, 1

store t, (b)move t, 0

move b, 0x4

move f, 0x4

store t, (b)move t, 2

Test programadd

...

...

...

sub

storeload

cache hitcache miss

normaloverflow

register dependencyaddress dependency

address dependencyregister dependency

generation parameters

lui s1, 0xdeadori s1, s1, 0x0lui s3, 0xbeefori s3, s3, 0xfadd v0, a0, a2sub t1, t3, t5add t7, s1, s3


MicroTESK tool support

The approach is supported by the MicroTESK test program generator

Test programs are generated according to the specified goals

Technology and tool provide high level of automation

Test programs might include checks of microprocessor state

Specifications are developed in Java language

There are ready-to-use test data generators

Generator has graphical user interface


Generator development process

// Test program N // Dependencies: div[0]-add[1] div.s $1, $5, $3 // Underflow add.d $2, $1, $8 // Overflow mul.s $7, $4, $9 // Inexact

// Test program 1 // Dependencies: sw[0]-lw[1] sw v0, (t0) // L1 Miss lw v1, (t1) // L1 Hit dw v2, (t2) // TLB Miss

// Test program 0 // Dependencies: add[0]-sub[1] add v0, a0, a2 // Overflow sub t1, v0, t5 // Normal div t7, s1 // Div by Zero

...

Set of Test Programs:Delivered to Microprocessor Designers

Architecture Modeler(s) /Test Developer(s)

MicroTESKTest Program

Generator

Documentation(Instruction Set Manual,

Standards, Implementation Features, etc.)

Verification Analyst(s)

Verification Engineer(s)

Verification Plan (Generalized Goals and Tasks of Test Program

Generation)

Documentation: Provided byMicroprocessor Designers

Formal Specification of MicroprocessorInstruction Set

Instruction-Level Test Coverage (Test Situations

and Dependencies)

Configuration / Parameters for MicroTESK Test Program Generator


How a MicroTESK generator looks like


Case studies

Memory subsystem MIPS64 microprocessor DSP co-processor Floating-point co-processors DMA controller Branch prediction unit


Some statistics

Characteristic MMU MIPS64 DSP CP1 CP2

Number of instructions 4 221 109 114 75

Efforts, man-months 2 9 4 1 1

Volume of specifications and tests, lines of code 5560 47190 20610 6450 11550

Number of found bugs in a reference simulator 0 6 > 10 >10 >10

Number of found bugs in an RTL model 1 9 > 10 >10 >10

Many critical bugs were found in the designs, which had been verified by hand-written, random-generated, and template-based test programs


Future directions

Automatic extraction of test coverage

Advanced test template generation

Integration with ADL (ISE, nML, LISA, etc.)


Extraction of test situations

L1

TLB

Hit, Valid Hit, Invalid

Exception: TLBInvalid

Hit Miss

Exception: TLBRefill

Miss

Normal1 Normal2


Extraction of dependencies

associative buffer TLB<TLB_ENTRY, 64> { VIRTUAL_PAGE_NUMBER tag( VIRTUAL_ADDRESS va) { ... } ... INTEGER index(VIRTUAL_ADDRESS va) { ... } ...}

structure TLB_ENTRY { VIRTUAL_PAGE_NUMBER vpn; PHYSICAL_PAGE_NUMBER pfn; ...}


MicroTESK project

Download the MicroTESK demohttp://hardware.ispras.ru


Contacts

Institute for System Programming of RAS (ISPRAS)http://www.ispras.ru

Hardware Verification R&D @ ISPRAShttp://hardware.ispras.ru

Alexander [email protected]


Thank You!Questions?

Documents

MicroTESK : Automation of Test Program Generation for Microprocessors