Effective Testing and Debugging Methods and Its Supporting System with Program Deltas

Effective Testing and Debugging Methods

and Its Supporting System with Program Deltas

Makoto Matsushita, Masayoshi Teraguchi,

and Katsuro InoueOsaka University

ISFST2000 at Kanazawa Nov/02/2000

Page 2

Software Maintenance

Software maintenance procedures

– Code modification to apply changes of specs

– Regression testing and debugging if any bugs

Supporting tools for software maintenance• Version management system (VMS)• Regression testing tool


Page 3

Debugging with Program Deltas

It is hard to debug features which are not changed during maintenance

Debugging with a code differences (deltas)• Using test results of each version

• The deltas may contain some bugs


Page 4

Ness and Ngo: Regression Containment

• Using VMS and testing tool• Correct output (○) and wrong output (×)

doesn’t appear by turns• Only a single “wrong” case• Only removes codes; no debugging


Page 5

Zeller: Delta Debugging

• Using test tool (no VMS is required)• Assumes “core” (△) output• Can’t consider development sequences• Only two “wrong” cases (△ and ×)• Only analysis; no debugging


Page 6

Our Research Objectives

An improved method with program deltas– Consider more “wrong” cases– Support actual debugging activities

Debugging methods “DMET”, andits supporting environment “DSUS”


Page 7

Today’s topic

• Our debugging method “DMET”• “DSUS” supporting environment• DMET evaluation with DSUS

prototype• Conclusion


Page 8

An approach of DMET

DMET is composed of three phases:– Detection: a program delta (may contains

some bugs) to be detected– Indication: show a delta with latest code– Reflection: apply modification to old

versions


Page 9

DMET: Detection Phase

• Using VMS and testing tools• Assumes wrong output ( － ) which is

different from the one of latest version• Detect the last correct output and the

oldest wrong output


Page 10

DMET: Indication Phase

• Program deltas are arranged as “removed” and “added” program codes

• Detected deltas (maybe in older version) are mapped to the latest version


Page 11

DMET: Reflection Phase

Modification to the latest version are also applied to older versions

– if application causes some errors, remove error version from test sequence

– new sequence are used for further testing


Page 12

Supporting System DSUS

• A GUI-based SDE for debugging– Program editors– Compile, Execute a program

• Cooperates other tools– RCS for version management system– DejaGnu for testing tools

• Auto-execution support– check-in/check-out, regression testing– three DMET procedures


Page 13

System Configuration

• GUI: user interaction

• DSUSmain: integration of other components

• RCS: wrapper for RCS commands

• DejaGnu: wrapper for DejaGnu commands


Page 14

DSUS Screenshot


Page 15

DSUS Evaluation

• Two experimentations– Experimentation A: correcting

programs which contain bugs in unchanged potion of program

– Experimentation B: debug programs corrected by experimentation A

• Preparation– Program specification and its program– Requirements for the new program– Test data


Page 16

Experimentation Procedures

• Experimentation A– Apply new requirement to a (prepared)

program– Correct programs which has a bug in

unchanged portion of the program

• Experimentation B– Debug program collected in exp. A– Subjects are divided into two groups

• debugging without DMET• debugging with DMET

– Compare Elapsed time of debugging


Page 17

Results of experiment A

• Three programs are collected• DMET detects bugs in two

programs; however, detected deltas of program Z doesn’t contain a bug

Program

# of vers

# of test

deltas detected?

X 28 25 1.3-1.4 yes

Y 91 10 1.81-1.88 yes

Z 72 70 1.1-1.46 no


Page 18

Results of experiment B

Debugging time is reduced in X and Y, which bug is detected correctly.

X Y Z

w/o DMET

60.0 103.0 10.6

w/ DMET

39.8 27.2 39.4

0

20

40

60

80

100

120

X Y Zプログラム

平均所要時間（

分）

DMET利用なしDMET利用ありw/ DMETw/o DMET

Program

ela

pse

d t

ime (

min

)


Page 19

Results of experiment B (cont.)

According to Welch’s test (5 percents differences), a significant difference of X+Y between both groups is detected

elapsed time

X+Y+Z X+Y

w/o DMET 60.0 103.0w/ DMET 39.8 27.2

DMET support effective debuggingif the detection is correct


Page 20

Discussion

In program Z, elapsed time of w/ DMET takes longer time than w/o DMET

– A bug in program Z is the case that DMET doesn’t consider; it is caused since some modification should be needed

– DMET generates wrong deltas– Testees are confused with deltas


Page 21

Discussion (cont.)

However, we may estimate that the deltas truly contains a bug since it’s doubtful that:

– the distance of deltas is longer, compared with total numbers of version

– the delta contains the “base version (1.1)”

program

# of vers

# of test

deltas detected?

Z 72 70 1.1-1.46 no


Page 22

Conclusion

• DMET contributes to reduce debugging costs of the engineers, if the detected program delta contains bugs

• DMET doesn’t consider all cases, it may generates wrong delta– If wrong delta, debugging takes

longer time– However, it can be estimated


Page 23

Further Research Topics

• Some estimation methods for checking DMET deltasIf the delta seems wrong, DSUS shows to the engineers that it maybe wrong.

• DMET extension for new programsCurrent DMET assumes that “base version” exists and it is correct. New program have no “base version”.

Documents

Effective Testing and Debugging Methods and Its Supporting System with Program Deltas