Functionality Functionality Testing Testing

TechniqueTechniqueLiang Gao

AgendaAgenda• Test case development• Test case execution

What makes a What makes a functionality testerfunctionality tester

• Functionality testing is to test the “new” code. Input

TesterOutput

Software Function Spec

Marketing Requirement Spec

Developer's code

Test Plan/Test Cases

Defects

Scripts

Test Case Is Very Test Case Is Very ImportantImportant

• Development is science, Testing is an ART.

• Test case is the foundation of a QA department.

• Test case is the Crown Jewel of the testing.

• What is “Garbage in, garbage out”?

ExerciseExercise• A program called “sum” is developed in Linux, it

accepts two 32-bit signed integers, add them and display the result in the next line.

• The result is also 32-bit signed integer, if error is detected, error message will be displayed to stderr and error code will be returned.

SolutionsSolutions• What is the range of 32-bit signed integer?• What are the boundaries conditions?

o -231 <= X + Y <= 231-1o -231 <= X or Y <= 231-1

• What are the negative conditions?o Out of rangeo Invalid inputs

• External factorso System resources are lowo Simultaneous executiono Execution from another program (GUI as front-end), exit status code

• Other testingo Performance (speed of the execution)o Spell check on error messages (Localization?)o Size of the programo Versioningo Help

InputInput• Input types:

o +<#> (acceptable?)o 600Ao 600.60o <#> <100 space> <#>o <#> <tab> <#>o 0x<hex>?o “<#> <#>”o 100-digit numbero 0<#>?o 100 “0”?o 231 + 30 negative

number?o 100 “-” follow by valid

numbero %do &, $, #, ^G

• Arguments:o 0, 1, 3, 10 argumentso <#> <alphabet>o “ “ <#> <#>o 2 arguments with 200-

character each• Error messages:

o Ensure all error condition generates appropriate error messages in current localeo Ensure all error the exit

status code is non-zeroo Ensure error messages are

printed out in stderro Ensure no mis-spell and

grammar errors o Ensure consistency in error

messages (error format, error number)

External FactorExternal Factor• OS related factor:

o Run in different varieties of Linux distroo Run program in low-

memory condition (1MB of memory)o Develop shell scripts to

invoke 20 “sum” in background “&” and ensure the results are correcto Run “sum” from crono Use “ldd” to find out the

shared library “sum” is using, install different versions of shared library to ensure the program works

• Otherso Use “time” to measure

the performance of sum program. The execution elapse time should not exceed 0.2 second; the CPU usage should be less than 20% during executiono Find out the size of the

program, it should not be more than 2MB of disk spaceo Ensure the version is

displayed in help stringo Ensure the help program

is availableo Develop a script to run

10,000 sum program overnightoinstallation

-231

-231

231-1

231-1

X

X

XX

X

X

X

X

X

X

XX

X

X

X

Boundary TestingBoundary Testing

5 Basic Testing Case 5 Basic Testing Case Design TechniquesDesign Techniques

• Equivalent class • Boundary values• Decision tables• State-transition • All pair

Equivalence class (Equivalence class ( 等等价类价类 ))

• 等价列划分设计方法是把所有可能的输入数据，即程序的输入域划分成若干部分（子集），然后从每一个子集中选取少量具有代表性的数据作为测试用例。

Equivalence class Example

Equivalence class (Equivalence class ( 等等价类价类 ))

• Case #1: domain of data is an intervalo – Class(es) of valid valueso – Class of invalid values (out of boundary, inf. and sup.)o – Class of non member (for numerical values, characters are non

members)

• Case #2: Domain of data is a number of valueso –N classes for valid valueso –One class: absence of valueso –One class: too much values

Equivalence class (Equivalence class ( 等等价类价类 ))

• Case #3: data is a set of values that are processed in different mannero – One class for each valid valueo – One class of all other invalid values

• Case #4: data is a constraint (form, syntax, mean)o –One class for valid constrainto –One class for constraint violation

Boundary value Boundary value testingtesting

• Often combined with Equivalence class testing• Boundary value testing ensures proper

functionality at the boundary (or edges) or allowable data input. Boundary values include maximum, minimum, just inside/outside boundary, typical values, and error (malformed values).

Boundary value Boundary value testingtesting

• Boundary value testing depends greatly on experience

• Don’t come to Developer to ask for boundary value advise often

• We have an practical example for boundary testing

Decision tablesDecision tables

Decision Table Based Testing

State-transition State-transition diagramsdiagrams

• Often used on protocol conformance test

State Transition Testing Explained

All Pair TestingAll Pair Testing

NY L 1 fam A+ Pri 80% Yes Yes Yes Cust Yes Yes

NJ M 2 fam A Vac 90% No No No Bank

No No

FL H 3 fam A- Inv 100%

TX H+1 4 fam B

CA H+2 Coop <B

DC H+3 Condo

Other

Reg

ion

tier

Pro

pert

y

Cre

dit

Res

iden

ce

LT

V

NIV

NA

V

Ref

inan

ce

Clo

sing

Cos

t

Intr

o R

ate

Ban

k E

mp

All Pair TestingAll Pair TestingTwelve variables, with varying numbers of values, have

7 x 6 x 6 x 5 x 3 x 3 x 2 x 2 x 2 x 2 x 2 x 2 = 725,760

combinations of values.

“All Pairs does it in 50.” (Bernie Berger, STAREast2003)

All Pair TestingAll Pair Testing• Reduce the number of tests  Clean• Multiple test cases will find same defects (all

combination is an overkill)• Tests that cover all pairwise

o For any two parametersp1 and p2 and any valid values v1 for p1 and v2 for p2, there is a test in which p1 has the value v1 and p2 has the value v2.

In Our ExampleIn Our Example• All pair test cases: C2

12 = 12!/(2! x (12 - 2)!) = 66

• Triple pair test case: C312 = 12!/(3! x (12 -

3)!)

• Worst case:

C1

12

2 1 4095p

12

P=1

12

Can it Work?Can it Work?• 97% of failure are caused by 2 feature

interaction (NIST Medical software failures 2000)

What about coverageWhat about coverage• We measured the coverage of combinatorial design test sets for 10 Unix

commands: basename, cb, comm, crypt, sleep, sort, touch, tty, uniq, and wc. […] The pairwise tests gave over 90 percent block coverage.[D. M. Cohen et al., 1996][…]

• a set of 29 pair-wise AETG tests gave 90% block coverage for the UNIX sort command. We also compared pair-wise testing with random input testing and found that pair-wise testing gave better coverage.[D. M. Cohen et al., 1997]

• The block coverage obtained for [pairwise] was comparable with that achieved by exhaustively testing all factor combinations […][I. S. Dunietz et al., 1997]

• Our initial trial of this was on a subset Nortel's internal e-mail system where we able cover 97% of branches with less than 100 valid and invalid testcases, as opposed to 27 trillion exhaustive testcases.[K. Burr and W. Young, 1998]

• Compared to a traditional company that would use the quasiexhaustive strategy, an innovative company using the [Combinatorial] strategy would reduce its system level test schedule by sixty-eight percent (68%) and save sixty-seven percent (67%) in labor costs associated with the testing.[J. Huller, 2000]

• [Evaluating FDA recall class failures in medical devices we established that] [...] out of the 109 reports that [were] detailed [enough], 98% showed that the problem could have been detected by testing the device with all pairs of parameter settings.[D. R. Wallace and D. R. Kuhn, 2001]

• More than 70% of bugs were detected with two or fewer conditions (75% for browser and 70% for server) and approximately 90% of the bugs reported were detected with three or fewer conditions (95% for browser and 89% for server). [...] It is interesting that a small number of conditions (n<=6) are sufficient to detect all reported errors for the browser and server software.[R. Kuhn and M. J. Reilly, 2002]

Even have a website to Even have a website to promotepromote

Test Case QualityTest Case Quality• Single Test Case Quality

o Cleano Exacto Repeatable

• Test Case Coverage Qualityo How do you know you are not missing critical test cases?o After a test, if no bugs found, how confident are you to say the

software is in good quality.

Individual test case Individual test case qualityquality

• Chances are: functionality tester might be junior.

• Use standard testing procedures whenever possible

• Interact with programmer. • Rome is not built in one day, so is your test

plan / test cases o Review modify and Review modify and Review modify….o Bring developers, technical marketing engineers, product managers

into the review meeting

Bad Test CasesBad Test Cases• Create extended acl• Match group• Associate match group to E-ACL• Create action group with next interface as

tunnel• Create PBR policy• Associate pbr policy to the action and match

group• Apply pbr policy on the ingress interface

Create What ACL? need a detailed CLI Here

Match What Group? What is the group name, what is the cli?

Next Interface? Need to explain that.

Ingress Interface? Which one is the ingress interface?

Functionality Testing Functionality Testing (2 input, 2 talk) (2 input, 2 talk)

• Understand features based on software function specification and Marketing Requirement Documentation.

• Talk to the one who propose this new functionality (Why we need this feature).

• Talk to the one who wrote the function specification

Developer Checklist

6 “Understand” 6 “Understand” • Try to understand on “Functional Points”

• Try to understand more “Use Case/Scenarios”

• Try to understand more Why we need this feature (revenue impact)

• Try to understand more the modules will be affected by this feature.

• Try to understand more the performance impact.

• Try to understand more the way it is going to be implemented by the developersOSPF Hello Protocol

Test Case Attributes Test Case Attributes • Test ID

• Description

• Pre-Setup

• Platform

• Topology

• Priority

• Complexity

• Steps/Expected results

Sample Test Plan

Functionality Test Functionality Test Case Coverage Case Coverage

• Need to be sure all functionality points are covered

• Need to be sure all related modules combine test are covered. (IPv6, HA?)

• Need to be sure to have a platform coverage list.

• Enough use cases (experience and communication matters) (Build your Customer Scenario Library: various sample topologies, typical device configurations, typical applications)

Code Coverage Code Coverage • A quantum way to measure testing coverage.

• A special compiler patch that when build a software image, can place mark on the new code

• After you execute your black box test cases, you can know which code line is being executed.

• Very mature in Java (just Google “java code coverage)

• C has open source and commercialized tools (hard to integrate into a build environment if code base is huge) http://gcc.gnu.org/onlinedocs/gcc/Gcov-Intro.html

CLI Standard Testing Procedure Example - CLI Standard Testing Procedure Example -

11

• enable CLI, verify all options and parameters are implemented as defined in the function spec.

• Enable CLI, do “show running-config”, verify it is in the memoryDisable CLI, do “show running-config” , verify it is not in the memory. Do a “show xxx” to verify the CLI is taking effect.

• Repeat 1, but write the configure into flash, verify it is written in the flash memoryReload the box; verify the configuration is still there.

• Enable then disable CLI for at least 20 times, see if we have any memory leak, or other instability. If it is a CLI that can be repeated, like firewall rules. Enable many rules, then disable many rules.

• Verify there is no typos in the CLIEspecially when there are some errors prompt messages. Verity those messages have no typos.

• Conflict CLI on different features will report error• CLI response time

CLI Standard Testing Procedure Example - CLI Standard Testing Procedure Example -

22

• Boundary testing on maximum and minimum CLI parameters values that it can take, check with function spec to see if it is as designed.

• Negative testing on CLI parameters on o Numbers:

• Out of boundary numbers. 负数？ Maximum number + 1 if possible.

• empty• Put a string in to a number field.

o strings: • Strange characters. %(#*$*@#&#&^&<>:”:{}. • long characters ( as long as the system can take it, as

long as you can type on a terminal)• empty

GUI Testing ChecklistGUI Testing Checklist

GUI Testing Checklist

A 3 Level Test Case A 3 Level Test Case Review ProcessReview Process

Team Internal Review

Peer review from other team that has domain

expert knowledge

Testing Tiger Team Review

Internal peer review

External peer review

Tiger Testing Team is a elite group of team lead and managers that has rich testing experience

Testing Coverage Testing Coverage Quality ControlQuality Control

• 1st to see what developer wants to say• Test area coverage

o identify all functionality points (areas) on the software function specification

o Can do code coverage if image building environment support it

• Test type coverage (required for all new feature testing)o CLI or GUI testingo Functionality testingo Negative testing/Boundary testingo Use Case Testingo Performance testing

Developer Checklist

Test case execution Test case execution quality controlquality control

• Define the platform coverage• For both manual execution and auto execution

(running scripts)• Detailed step by step execution logs may need

to be saved and reviews for junior engineers • Regular rotation to keep engineer excited

about different testing areas better execution quality

• Cross testing to discover new bugs• Stringent schedule control

What If you can’t get What If you can’t get the Spec?the Spec?

• Whatever specs exist• Software change memos that come with each new

internal version of the program• User manual draft (and previous version’s manual)• Product literature• Published style guide and UI standards• Published standards (C language or RFC)• 3rd party product compatibility testing suites• Published regulations• Internal memos (e.g. project mgr.to engineers,

describing the feature definitions)

What If you can’t get What If you can’t get the Spec?the Spec?

• Marketing presentations, selling the concept of the product to management

• Bug reports (responses to them)• Reverse engineer the program• Interview people

o Developer leado Tech writero Customer serviceo Subject matter expertso Project manager

• Look at header files, source code, database table definitions• Specs and bug lists for all 3rd party tools that you use• Prototypes or lab notes on the prototypes.

What If you can’t get What If you can’t get the Spec?the Spec?

• •Look at compatible products• Look at customer call records