Software TestingTesting typesTesting strategyTesting principles
Testing ObjectivesTesting is a process of executing a program with the intent of finding an errorA good test case is one that has a high probability of finding an as-yet undiscovered errorA successful test is one that uncovers an as-yet undiscovered error
RemindersTesting is a process for finding semantic (logical) errors, but not syntactic (symbolical) errors, in implemented programs.Testing can reveal the presence of errors, NOT their absence.Can you tell the differences between testing, debugging and compilation debugging?
Testing TechniquesWhite-box testing (WBT)When knowing the internal workings of a productSome techniquesBasis path testingCondition testingData flow testing
Testing TechniquesBlack-box testing (BBT)When knowing the specified function that a product has been designed to performSome techniquesEquivalence partitioning (Input domain testing)Boundary value analysisComparison testing (back-to-back testing)
Basis Path Testing (WBT)StepsConstruct a flow chart based on a programCompute (cyclomatic) complexity of the flow chart, which also indicates the number of independent paths of the programDetermine basis pathsDesign test cases to go through these basis paths
Cyclomatic ComplexityThe complexity of a programThree ways for calculating the complexityThe number of regions of the flow graph(total edges) (total nodes) + 2Number of predicate edges + 1
Cyclomatic Complexity4 basis paths1-2-3-4-5-7-101-2-3-4-5-8-101-2-3-4-6-9-101-2-10(May have more than one option for definingbasis paths)
Condition Testing (WBT)Focus on testing each condition in the programBranch testing may be the simplest method for doing condition testingStepsDefine combinations of truth values for variables in predicate statementsDesign test cases to test all these combinations
Data Flow Testing (WBT)Select test paths of a program according to the locations of definitions and uses of variables in the program, then test these define-use chains.StepsDetermine define-use chains for each variablesDesign test cases to pass through these define-use chains (may have different testing criteria, ex. all-defines, all-uses, )
Equivalence Testing (BBT)Divide input domain of a program into classes of data from which test cases can be derivedEx. If an input condition specifies a range or a value, one valid and two invalid equivalence classes are defined.Ex. Boolean: one valid and one invalid classes can be defined.
Boundary Value Analysis (BBT)Lead to a selection of test cases that exercise boundary values of the input domainEx. A range v1 and v2, test cases should be designed with values v1 and v2, just above and just below v1 and v2, respectively.
Comparison Testing (BBT)Appropriate when multiple implementations of same specification have been producedUse same inputs to test whether they generate same outputs
Verification & Validation (V&V)Verification: "Are we building the product right"The software should conform to its specificationValidation: "Are we building the right product"The software should do what the user really requires
The V & V ProcessIs a whole life-cycle process - V & V must be applied at each stage in the software process.Has two principal objectivesThe discovery of defects in a systemThe assessment of whether or not the system is usable in an operational situation.
Static and Dynamic VerificationSoftware inspections Concerned with analysis of the static system representation to discover problems (static verification)May be supplement by tool-based document and code analysisSoftware testing Concerned with exercising and observing product behaviour (dynamic verification)The system is executed with test data and its operational behaviour is observed
Inspections and TestingInspections and testing are complementary and not opposing verification techniquesBoth should be used during the V & V processInspections can check conformance with a specification but not conformance with the customers real requirementsInspections cannot check non-functional characteristics such as performance, usability, etc.
Software InspectionsInvolve people examining the source representation with the aim of discovering anomalies and defectsDo not require execution of a system so may be used before implementationMay be applied to any representation of the system (requirements, design, test data, etc.)Very effective technique for discovering errors
Program InspectionsFormalised approach to document reviewsIntended explicitly for defect DETECTION (not correction)Defects may be logical errors, anomalies in the code that might indicate an erroneous condition (e.g. an uninitialised variable) or non-compliance with standards
Inspection Pre-conditionsA precise specification must be availableTeam members must be familiar with the organisation standardsSyntactically correct code must be availableAn error checklist should be preparedManagement must accept that inspection will increase costs early in the software processManagement must not use inspections for staff appraisal
Inspection ProcedureSystem overview presented to inspection teamCode and associated documents are distributed to inspection team in advanceInspection takes place and discovered errors are notedModifications are made to repair discovered errorsRe-inspection may or may not be required
Testing PrinciplesAll tests should be traceable to customersTests should be planned long begin testing beginsThe Pareto principle applies to software testingTesting should begin in the small and progress toward to testing in the largeExhaustive testing is impossibleTo be more effective, testing should be conduced by an independent third party
Software Testing StrategyUnit test -> Integration test -> Validation Test -> System Test
Unit TestUse white-box testing techniquesTest for each module, including interface, local data structures, boundary conditions, independent paths, error handling paths, and so on.
Integration TestTest the correctness of integrations of modules.Usually use an incremental integration strategy to integrate and test modules.Integration strategiesTop-down integrationBottom-up integrationCombination of both methods
Validation TestingTest whether the software functions in a manner that can be reasonably expected by the customer.The reasonable expectations are defined in the software requirements.MethodsAlpha testing-developer observes users operation and records all errors from the operationBeta testing-customer operates system in real environment without developer, but reports all errors to developer after his operation
System TestingFocus on not only software, but also the integration of software, hardware and environmentDifferent testingRecovery testingSecurity testingStress testingPerformance testing
Debugging ApproachesBrute force-insert write statement to indicate the position of the system when an error occurs, and find the causes of the error before the write statement.Backtracking-begin at the site where a symptom has been uncovered, and trace backward in the source until the causes have been located.Cause elimination-list all the hypotheses about the causes of an error, check and eliminate these hypotheses one by one, until the real cause has been found.
PS.Real-time systems/applications are harder to be tested than other systems-time/temporal considerations (hard real-time systems/soft real-time systems)For testing object-oriented programs, please refer to Dr. Kungs web page.