Prioritizing Test Cases for Regression Testing

  • View
    30

  • Download
    0

Embed Size (px)

DESCRIPTION

Sebastian Elbaum University of Nebraska, Lincoln Alexey Malishevsky Oregon State University Gregg Rothermel Oregon State University. Prioritizing Test Cases for Regression Testing. ISSTA 2000. Defining Prioritization. Test scheduling During regression testing stage - PowerPoint PPT Presentation

Transcript

  • Prioritizing Test Cases for Regression TestingSebastian Elbaum University of Nebraska, LincolnAlexey Malishevsky Oregon State UniversityGregg Rothermel Oregon State UniversityISSTA 2000

  • Defining PrioritizationTest scheduling During regression testing stageGoal: maximize a criterion/criteriaIncrease rate of fault detectionIncrease rate of coverageIncrease rate of fault likelihood exposure

  • Prioritization RequirementsDefinition of goalIncrease rate of fault detectionMeasurement criterion% Of faults detected over life of test suitePrioritization techniqueRandomlyTotal statements coverageProbability of exposing faults

  • Previous WorkGoalIncrease rate of fault detectionMeasurementAPFD: weighted average of the percentage of faults detected over life of test suiteScale: 0 - 100 (higher means faster detection)

  • Previous Work (2)A-B-C-D-EC-E-B-A-DE-D-C-B-AMeasuring Rate of Fault Detection

  • Previous Work (3)Prioritization Techniques

  • Summary Previous WorkPerformed empirical evaluation of general prioritization techniquesEven simple techniques generated gainsUsed statement level techniquesStill room to improve

  • Research QuestionsCan version specific TCP improve the rate of fault detection?

    How does fine technique granularity compare with coarse level granularity?

    Can the use of fault proneness improve the rate of fault detection?

  • Addressing RQNew family of prioritization techniquesNew series of experimentsVersion specific prioritizationStatementFunctionGranularityContribution of fault pronenessPractical implications

  • Additional Techniques

  • Family of Experiments8 programs 29 versions50 test suites per programBranch coverage adequate14 techniques2 control techniques optimal & random4 statement level8 function level

  • Generic Factorial DesignTechniquesPrograms50 Test Suites29 VersionsIndependence of codeIndependenceof suite compositionIndependence of changes

  • Experiment 1a Version SpecificRQ1: Prioritization works on version specific at stat. level. ANOVA: Different average APFD among stat. level techniquesBonferroni: St-fep-addtl significantly better

    GroupTechniqueValueASt-fep-addtl78.88BSt-fep-total76.99BSt-total76.30CSt-addtl74.44Random59.73

  • Experiment 1b Version SpecificRQ1: Prioritization works on version specific at function level.ANOVA: Different average APFD among function level techniquesBonferroni: Fn-fep not significantly different than Fn-total

    GroupTechniqueValueAFn-fep-addtl75.59AFn-fep-total75.48AFn-total75.09BFn-addtl71.66

  • Experiment 2: GranularityRQ2: Fine granularity has greater prioritization potentialTechniques at the stat. level are significantly better than functional levelHowever, best functional level are better than worse statement level

    Chart3

    76.375.09

    74.4471.66

    76.9969.32

    78.8869.8

    60.9660.96

    Statement

    Function

    Sheet1

    TECHS

    T01rand

    T02optimal

    T03st-total

    T04st-addtl

    T05st-fep-total

    T06st-fep-addtl

    T07fn-total

    T08fn-addtl

    T09fn-fep-total

    T10fn-fep-addtl

    T11fn-fi-total

    T12fn-fi-addtl

    T13fn-fi-fep-total

    T14fn-fi-fep-addtl

    TECHSStatementFunction

    T03st-total76.3T07fn-total75.09

    T04st-addtl74.44T08fn-addtl71.66

    T05st-fep-total76.99T09fn-fep-total69.32

    T06st-fep-addtl78.88T10fn-fep-addtl69.8

    TechniquesStatementFunction

    total76.375.09

    addtl74.4471.66

    fep-total76.9969.32

    fep-addtl78.8869.8

    random60.9660.96

    Sheet1

    Statement

    Function

    Sheet2

    Sheet3

  • Experiment 3: Fault PronenessRQ3: Incorporating fault likelihood did not significantly increased APFD. ANOVA: Significant differences in average APFD values among all functional level techniquesBonferroni: Surprise. Techniques using fault likelihood did not rank significantly betterReasons:For small changes fault likelihood does not seem to be worth it.We believe it will be worthwhile for larger changes. Further exploration required.

    GroupTechniqueValueAFn-fi-fep-addtl76.34A BFn-fi-fep-total75.92A BFn-fi-total75.63A BFn-fep-addtl75.59A BFn-fep-total75.48 BFn-total75.09CFn-fi-addtl72.62CFn-addtl71.66

  • Practical ImplicationsAPFD:Optimal = 99%Fn-fi-fep-addtl= 98%Fn-total = 93%Random = 84%Time:Optimal = 1.3Fn-fi-fep-addtl = 2.0 (+.7)Fn-total = 11.9(+10.6)Random = 16.5(+15.2)

  • ConclusionsVersion specific techniques can significantly improve rate of fault detection during regression testingTechnique granularity is noticeableIn general, statement level is more powerful but,Advanced functional level techniques are better than simple statement level techniquesFault likelihood may not be helpful

  • Working on Controlling the threats More subjectsExtending modelDiscovery of additional factors Development of guidelines to choose best technique

  • Backup Slides

  • ThreatsRepresentativenessProgramChanges Tests and process

    APFD as a test efficiency measureTools correctness

  • Experiment Subjects

    Program

    LOC

    Test Suite Avg. Size

    replace

    516

    19

    printtok1

    402

    16

    totinfo

    346

    7

    printtok2

    483

    12

    schedule1

    299

    8

    schedule2

    297

    8

    tcas

    138

    6

    space

    6218

    155

  • FEP ComputationProbability that a fault causes a failureWorks with mutation analysisInsert mutantsDetermine how many mutant are exposed by a test case

    FEP(t,s) =# of mutants of s exposed by t# of mutants of s

  • FI ComputationFault likelihoodAssociated with measurable software attributesComplexity metricsSize, Control Flow, and CouplingGenerated fault index principal component analysis

  • Overall

    GroupTechniqueValueAOptimal94.24BSt-fep-addtl78.88 CSt-fep-total76.99D CFn-fi-fep-addtl76.34D CSt-total76.30D EFn-fi-fep-total75.92D EFn-fi-total75.63D EFn-fep-addtl75.59D EFn-fep-total75.48F EFn-total75.09FSt-addtl74.44GFn-fi-addtl72.62GFn-addtl71.66HRandom59.73