Deriving State-Based Test Deriving State-Based Test Oracles for Conformance Oracles for Conformance
TestingTesting
Jamie AndrewsJamie AndrewsAssociate ProfessorAssociate Professor
Department of Computer ScienceDepartment of Computer ScienceUniversity of Western OntarioUniversity of Western Ontario
London, OntarioLondon, Ontario
Plan for TalkPlan for Talk
Testing backgroundTesting background Log file analysis (LFA)Log file analysis (LFA) Process for developing artifactsProcess for developing artifacts Refinement of processRefinement of process
Testing Background
• 3 main tasks to testing:– Selecting test inputs– Running tests– Checking test outputs
• Checking test output not always trivial
SoftwareUnder Test
Test Input
Test Output
Checking Test Output
• May be too complex to check visually
• May be legitimately different from output of previous version
SoftwareUnder Test
Test Input
Test Output
Test Oracles
• Programs that check the output of other programs
SoftwareUnder Test
Test Input
Test Output Test Oracle
Pass/Fail
Test Oracles
• Input and output may be difficult to capture
• Oracle may have to parse complex I/O
SoftwareUnder Test
Test Input
Test Output Test Oracle
Pass/Fail
Log File Analysis (LFA)
• Log file: simple text output
• LFA: dynamic analysis for conformance checking
SoftwareUnder Test
Test Input
Test OutputLog FileAnalyzer
Pass/Fail
Log File
LFA ChallengesLFA Challenges
Make sure logging doesn’t slow down Make sure logging doesn’t slow down system too muchsystem too much– Can send logging data to another machineCan send logging data to another machine
Decide on logging policy for softwareDecide on logging policy for software Write log file analyzer programWrite log file analyzer program
– Special-purpose state machine-based language Special-purpose state machine-based language to help with thisto help with this
Log File Analysis Language Log File Analysis Language (LFAL)(LFAL)
Analyzer = collection of state machinesAnalyzer = collection of state machines Each machine notices some log file lines, Each machine notices some log file lines,
ignores othersignores others Log file lines trigger transitionsLog file lines trigger transitions Machine reports error if it:Machine reports error if it:
– Notices a lineNotices a line– Does not have a legal transition on itDoes not have a legal transition on it
ProcessesProcesses
Need a process for gettingNeed a process for getting– From requirementsFrom requirements– To logging instrumentation, analyzerTo logging instrumentation, analyzer
““Big-step” process:Big-step” process:– Used and taught to studentsUsed and taught to students
““Small-step” process:Small-step” process:– Suggested refinementSuggested refinement
Example: Elevator SystemExample: Elevator System
Requirement to check:Requirement to check:
““The doors are never The doors are never open when the open when the elevator is in motion.”elevator is in motion.”
Big-Step ProcessBig-Step Process
Requirements
SPFEs
Log File AnalyzerProgram
Logging Policy
Situations withPermitted andForbiddenEvents
Example: SPFEsExample: SPFEs
SPFE1:SPFE1:– Situation: Elevator door is openSituation: Elevator door is open– Permitted event: Door closesPermitted event: Door closes– Forbidden event: Elevator starts movingForbidden event: Elevator starts moving
SPFE2:SPFE2:– Situation: Elevator is movingSituation: Elevator is moving– Permitted event: Elevator stops movingPermitted event: Elevator stops moving– Forbidden event: Door opensForbidden event: Door opens
RelationshipsRelationships
SPFEs should re-express requirements to SPFEs should re-express requirements to be checkedbe checked
Logging policy should specify that we log all Logging policy should specify that we log all events that allow us to determine:events that allow us to determine:– Whether we are in each SituationWhether we are in each Situation– Whether each Permitted/Forbidden Event has Whether each Permitted/Forbidden Event has
happenedhappened
Example: Logging PolicyExample: Logging Policy
Log all door open / close events in the form Log all door open / close events in the form door_opendoor_open, , door_closedoor_close
Log all elevator move / stop events in the Log all elevator move / stop events in the form form start_movestart_move, , stopstop
Can instrument code based on thisCan instrument code based on this
From SPFEs to AnalyzerFrom SPFEs to Analyzer
Situations correspond to statesSituations correspond to states Permitted events correspond to transitionsPermitted events correspond to transitions Forbidden events should not have any Forbidden events should not have any
corresponding transitioncorresponding transition
Example: LFAL AnalyzerExample: LFAL Analyzer
machine door_safety;
initial_state closed_stopped;
from closed_stopped, on start_move, to moving;
from moving, on stop, to closed_stopped;
from closed_stopped, on door_open, to open;
from open, on door_close, to closed_stopped;
final_state Any.
closed_stopped
open
moving
stop
start_move
door_close
door_open
Some Past ProjectsSome Past Projects
Steam boiler simulator and analyzer (X. An)Steam boiler simulator and analyzer (X. An) WAP client development and testing (V. Liu)WAP client development and testing (V. Liu)
– 5 KLOC; 3 protocol layers verified5 KLOC; 3 protocol layers verified
Test case generation from oracles (R. Fu)Test case generation from oracles (R. Fu)
Problems with Big-Step ProcessProblems with Big-Step Process
Not always explicit:Not always explicit:– Which requirements are to be checkedWhich requirements are to be checked– Under what assumptions/conditionsUnder what assumptions/conditions
Sometimes “abstract” events mentioned in Sometimes “abstract” events mentioned in SPFEs cannot be directly loggedSPFEs cannot be directly logged– e.g. “door open” event may actually correspond e.g. “door open” event may actually correspond
to “send release command to door lock to “send release command to door lock actuator”actuator”
– Need more concrete, loggable eventsNeed more concrete, loggable events
Small-Step ProcessSmall-Step Process
CheckingAssumptions
Abstract Events
Log File AnalyzerProgram
Logging Policy
Concrete EventsSPFEs
Requirementsto be Checked
Future WorkFuture Work
Case studies of small-step processCase studies of small-step process Teaching small-step processTeaching small-step process ExperimentsExperiments
Potential Benefits and ProblemsPotential Benefits and Problems
Benefits: improved reliability, flexibility, Benefits: improved reliability, flexibility, scalability, traceabilityscalability, traceability
Problems:Problems:– False negatives and positivesFalse negatives and positives– Instrumentation maintenanceInstrumentation maintenance– Process weightProcess weight
SummarySummary
LFA is a method for test result checkingLFA is a method for test result checking We have used and taught a process for We have used and taught a process for
applying itapplying it We propose a refined process for future We propose a refined process for future
workwork
Frequently Asked Questions
• Do we have to discard regression testing?– No; can use as a complement
• How do we know what to put in log file?– Recommend identify reqs to check with LFA first– Develop logging policy, log file analyzer from those
• Efficiency? Scalability?– A few years ago processed 1000 lines/sec– Biggest analyzer 1200 NLOC, from 10-page spec– Recommend starting small, seeing if works for you