EE694v-Verification-Lect5-1- Lecture 5 - Verification Tools Automation improves the efficiency and reliability of the verification process Some tools,

EE694v-Verification-Lect5 -1-

Lecture 5 - Verification Tools

• Automation improves the efficiency and reliability of the verification process

• Some tools, such as a simulator, are essential. Others automate tedious tasks and increase confidence in the outcome.

• It is not necessary to use all the tools.


Lecture Overview

• Will look at the other tools used in verification– Third party models

– Hardware modelers

– Waveform viewers

– Code coverage tools

– Verification languages

– Revision control

– Configuration management

– Issue tracking

– Metrics


Third Party Models

• Many designs use off the shelf parts• To verify such a design, must obtain a model to

these parts• Often must get the model from a 3rd party• Most 3rd party models are provided as compiled

binary models• Why buy 3rd party models?

– Engineering resources

– Quality (especially in the area of system timing)


Hardware Modelers

• Are for modeling new hardware. Some hardware may be too new for models to available– Example: In 2000 still could not get a model of the Pentium

III

• Sometimes cannot simulate enough of a model in an acceptable period of time


Hardware Modelers (cont)

• Hardware modeler features– Small box that connects to network that contains a real copy

of the physical chip

– Rest of HDL model provides inputs to the chip and obtains the chips output to return to your model


Waveform Viewers

• Lets you view transitions on multiple signals over time

• The most common of verification tools• Waveform can be saved in a trace file• In verification

– need to know expected output and whenever the simulated output

is not as expected • both the signal value and the signal timing

– use the testbench to compare the model output with the expected


Code Coverage

• A technique that has been used in software engineering for years.

• By covering all statements adequately the chances of a false positive (a bad design tests good) are reduced.

• Never 100% certain that design under verification is indeed correct. Code coverage increases confidence.

• Some tools may use file I/O aspect of language and others have special features built into the simulator to report coverage statistics.


Adding Code Coverage

• If built into simulator - code is automatically instrumented.

• If not built in - must add code to testbench to do the checking


Code Coverage

• Objective is to determine if you have overlooked exercising some code in the model– If you answer yes then must also ask why the code is present

• Coverage metrics can be generated after running a testbench

• Metrics measure coverage of– statements

– possible paths through code

– expressions


Statements and Blocks

• Statement coverage can also be called block coverage

• The Model Sim simulator can show how many times a statement was executed

• Also need to insure that executed statements are simulated with different values

• And there is code that was not meant to be simulated (code specifically for synthesis for example)


Path Coverage

• Measures all possible ways you can execute a sequence of statements

• Example has four possible paths


Path Coverage Goal

• Desire is to take all possible paths through code• It is possible to have 100% statement coverage but

less than 100% path coverage• Number of possible paths can be very, very large

=> keep number of paths as small as possible• Obtaining 100% path coverage for a model of

even moderate complexity is very difficult


Expression Coverage

• A measure of the various ways paths through code are taken

• Example has 100% statement coverage but only 50% expression coverage


100% Code Coverage

• What do 100% path and 100% expression coverage mean?– Not much!! Just indicates how thoroughly verification suite

exercises code. Does not indicate the quality of the verification suite.

– Does not provide an indication about correctness of code

• Results from coverage can help identify corner cases not exercised

• Is an additional indicator for completeness of job– Code coverage value can indicate if job is not complete


Verification Languages

• Verilog and VHDL were designed as design languages• Verification languages are designed for verification

– e/Specman from Verisity– VERA from Synopsys– RAVE from Cronology

• Even with a verification language still– need to plan verification– design verification strategy and design verification architecture– create stimulus– determine expected response– compare actual response versus expected response


Revision Control

• Need to insure that model verified is model used for implementation

• Managing a HDL-based hardware project is similar to managing a software project

• Require a source control management system• Such systems keep last version of a file and a

history of previous versions along with what changes are present in each version


Configuration Management

• Wish to tag (identify) certain versions of a file so multiple users can keep working

• Different users have different views of project


File Tags

• Each file tag has a specific meaning


Issue Tracking

• It is normal and expected to find functional irregularities in complex systems

• Worry if you don’t!!! Bugs will be found!!!

• An issue is anything that can affect the functionality of the design– Bugs during execution of the testbench

– Ambiguities or incompleteness of specifications

– A new and relevant testcase

– Errors found at any stage

• Must track all issues if a bad design could be manufactured were the issue not tracked


Issue Tracking Systems

• The Grapevine– Casual conversation between members of a design team in which

issues are discussed

– No-one has clear responsibility for solution

– System does not maintain a history

• The Post-it System– The yellow stickies are used to post issues

– Ownership of issues is tenuous at best

– No ability to prioritize issues

– System does not maintain a history


Issue Tracking Systems (cont.)

• The Procedural System– Issues are formally reported

– Outstanding issues are reviewed and resolved during team meetings

– This system consumes a lot of meeting time

• Computerized Systems– Issues seen through to resolution

– Can send periodic reminders until resolved

– History of action(s) to resolve is archived

– Problem is that these systems can require a significant effort to use


Code Related Metrics

• Code Coverage Metrics - how thoroughly does verification suite exercise code

• Number of Lines of Code Needed for Verification Suite - a measure of the level of effort needed

• Ratio of Lines of Verification Code to Lines of Code in the Model - measure of design complexity

• Number of source code changes over time


Quality Related Metrics

• Quality is subjective• Examples of quality metrics

– Number of known outstanding issues

– Number of bugs found during service life

• Must be very careful to interpret and use any metric correctly!!!

Documents

EE694v-Verification-Lect5-1- Lecture 5 - Verification Tools Automation improves the efficiency and reliability of the verification process Some tools,