Software Architecture-based Regression Testing

Embed Size (px)

Text of Software Architecture-based Regression Testing

  • 1.Software Architecture-based Regression TestingHenry Muccini Software Engineering and Architecture Group Universit degli Studi dell'AquilaI-67100 L'Aquila, Italy Marcio DiasDebra J. Richardson Department of Computer Science and Donald Bren School of Informatione-Science Research Institute,and Computer Sciences, University of Durham, UK University of California Irvine, USA

2. My main research areas: Analysis SEA Group- Software Engineering and Architecture Group Software Architecture Analysis:- Model-Checking SAs (the CHARMY framework)- SA-based Testing- SA-based Regression Testing (the SARTE project)- Model-checking driven Testing (the ModTest approach) Product Line:- Modeling Product Line ArchitectureSEA Group- Testing and Model Checking of Product Line 2 2005 by H. Muccini, WADS 2005 3. Our Experience on SA-based analysis SA conformance to requirements through MC Cidentify validate the SA model h Requirements M econformance o cwith respect to selected ddrive k SA model functional properties e ifunctionalCharmy Project l n Software M.C.propertiesArchitecture[NOK] grefine SA OK [test case selection] []SoftwareExtract SA-level Architecture Test cases T e provide confidence on the s SATC implementation tfulfillment to its i SystemMap SATC nImplementationinto realarchitectural spec g Test CasesSA-based Testing[NOK]TestTest Fault removal Exec Procedures SEA Group 3 2005 by H. Muccini, WADS conformance to SA through Testing Implementation 2005 4. ModTest: Model-Checking driven TestingSA conformance to requirements through MCa1 CRequirements identifySpecificationhRequirements a2M e ArchitecturalSA Revisiono c Specificationd drivek SA model a3e ifunctional Charmyl Software Analysisna4 Architecture[NOK]Criticalgrefine SA OK [test case selection] NOK sub-systems identificationOK a5 a7TeStor SystemTa6Implementatione Test spec Test Cases SystemImplementationt Implementationia8n [NOK] Test Test Case TestgProceduresExecutionImplementation Fault removal ExecRevision NOKOK SEA GroupImplementation conformance to SA through Testing System Release 4 2005 by H. Muccini, WADS 2005 5. Our Recent experience in SA-based Analysis Industrial Experience- PSTDA Italy [ICSE00,ICSE01] - Telcordia - Marconi [FME 03] - Siemens [ITM 04] Academic Experience- [FASE 04][IEEE TSE04] - [CBSE 05][COMPSAC05] SEA Group 5 2005 by H. Muccini, WADS 2005 6. Considerations What happens if the (architectural) modelchanges? - Usually, we need to remake analysis from scratchSoftware Architecture SA2Software Architecture SA1(version 2)(version 1)Component BComponent BComponent C'Component A Component C Component AComponent Y SEA Group 6 2005 by H. Muccini, WADS 2005 7. How changes affect ModTestSA conformance to requirements through MCC identifyhRequirements Me oc SA spec ddrivek SA model eifunctional l M.C.propertiesn Software Architecture[NOK] Test Casegrefine SA OK [test case selection]selectionT e Test Test spec s System implementation t Implementation i n[NOK]Test TestTest g Fault removal ExecProcedures execution SEA GroupImplementation conformance to SA through Testing 7 2005 by H. Muccini, WADS 2005 8. SA-based Regression Testing [WADS05] [COMPSAC05]Software Architecture SA2 Software Architecture SA1 (version 2) (version 1) Component BComponent B SA evolutionComponent C'Component AComponent CComponent AComponent Y Test Reuse CodeCode1evolution Code2Code (version 1)(version 2) SEA GroupTest Reuse Test Reuse 8 2005 by H. Muccini, WADS 2005 9. Traditional Regression Testing Test modified software to provide a certain confidence thatno new errors are introduced into previously tested code. Two key phases: i) testing the program P with respect to a specified test suite T, andii) when a new version P' is released, regression testing of themodified version P' versus a test suite T' Selective RT: - Goal: selecting T' as a relevant subset of T> t1 in T is included in T ' if there is the potential that it could produce different results on P' than it did on P SEA Group 9 2005 by H. Muccini, WADS 2005 10. First phase: SA-based Code Testing The code conformance to the SA has been already tested Software Architecture SCode P XClientComponent Class -------ConnectorClientA ClientBComponent ------------ ClientC -------Component Identify relevant Refine SA-level tests Run TC over P andSA-level Test Casesinto Code-level testsEvaluate ResultsSEA Group Step 4Steps 0 - 2 Step 3 10 2005 by H. Muccini, WADS 2005 11. SArTe Goal 1(code evolution)Software Architecture SA1 Test Conformance of a Modified(version 1) Implementation P' to the initial SAComponent B - Context: Component C Component A> P correctly implements the SA S> P' modifies P: some objects aremodified, and/or some new objectsare introduced.- Goal: Test the conformance of P'with respect to S, > while reusing previous test Code1information for selective regressionCode2 testing, thereby reducing the test cases(version 1)(version 2)that must be retested.Test Reuse- To handle Architectural Drift SEA Group 11 2005 by H. Muccini, WADS 2005 12. SArTe Goal 2 (SA evolution) Test Conformance of an Evolved Software Architecture SA2 Software Architecture Software Architecture SA1(version 2)(version 1)Component B - Context: Component BComponent C'> P correctly implements the SA S Component A Component CComponent A Component Y > S modifies S, adding or removing components Test > A modified implementation P' may have Reuse been also developed.- Goal: Test the conformance of Pwith respect to S', Code2 Code (version 2) > while reusing previous test information Testfor selective RT, thereby reducing the Reuse test cases that must be retested. SEA Group12 2005 by H. Muccini, WADS 2005 13. Goal 1: P changesSA-based Code Testing Step 0 SA-spec for S Step 1SA-basedTesting CriterionRegression Testing Step 2- Goal 1 - Extract SA-levelStep A Test Cases (ATCs)Build the GP Build the GPgraph for Pgraph for P Step 3Map ATCs into Code-level Test CasesStep Bas a way to select T for PCompare GP with GPStep 4Step C Run T over P Create a Test History and Evaluate Results Step DSelect T' for P' SEA Group 13 2005 by H. Muccini, WADS 2005 14. Considerations Differences with respect to traditional code-based selective RTtechniques:- code-level test cases are always selected starting from a well formalized architectural specification.- the oracle in SA-based RT is the software architecture specification itself. Advantages:- as in traditional RT, we reduce the size of the test suite for P', eliminating all those tests which do not need to be reapplied to P', and- when conformance faults are detected, we can gather information on how to adjust the initial architecture. SEA Group 14 2005 by H. Muccini, WADS 2005 15. Goal 2: SA changesSA-basedRegression TestingSA-based Code Testing- Goal 2 - Step a Step 0 SA-spec for S SA-spec for SStep b Step 1Testing CriterionTesting CriterionStep 2Extract SA-level Step cTest Cases (ATCs) Compare S and SStep 3 Map ATCs into Code-level Step d Test Cases as a way to select T for P Select ATCs from S that needto be retested in SStep 4Step eRun T over PMap ATCs into Code-level Test and Evaluate Results Cases as a way to select T for PStep f SEA GroupRun T over P and Evaluate Results 15 2005 by H. Muccini, WADS 2005 16. Goal 2 Idea Compare the two architectural specifications to identifychanged/unchanged portions of the SA.- Both structural and behavioral changes are taken into account> We compare the topology changes (if the SA structure changed) > We compare the behavioral changes (if the SA behavior changed)- and, in a fashion similar to traditional code-based RT,> ATC needs to be re-run in S', if it traverses a path modified when moving from S to S''SEA Group 16 2005 by H. Muccini, WADS 2005 17. Experiment 1: the Elevator SA Elevator System Elevator SystemVersion 1 Version 2 ElevatorADT1 ElevatorADT1ElevatorADT2ElevatorPanel1ElevatorPanel2ElevatorPanel1 SchedulerBuildingPanelBuildingPanel SEA Group 17 2005 by H. Muccini, WADS 2005 18. Experiment 2: the Cargo Router SystemCargo Router, Version 2 Cargo Router, Version 1 Note: In Version2, the connection (*)Port Vehicle Warehouse Clockis replaced with the connections (**) (P)(V)(W)(C) Bus1Planner (P) NextShipment (NS) (**) (*) Bus2Bus2A (**)Bus2B PortVehicleWarehouse PortVehicle Warehouse Artist (PA) Artist (VA) Artist (WA) Artist2 (PA2) Artist2 (VA2)Artist2 (WA2) Bus3Bus3ACargoRouter CargoRouter2(CR) Bus3C(CR2)Planner Artist (PlA) Translator (T)Bus4 Graphics Binding (GB)SEA Groupa) b) 18 2005 by H. Muccini, WADS 2005 19. Future Work To reconstruct the actual architecture when the first goaldetermines that the code no longer conforms to the initialSA Regression Testing of Component-based SoftwareArchitectures Regression-based Analysis of ModTest Apply/refine this approach into real systems (SiemensC.N.X.,Terma GmbH)SEA Group 19 2005 by H. Muccini, WADS 2005 20. Contact Information Henry Muccini - Dipartimento di Informatica, Universita' dell'Aquila, L'Aquila, Italy - Marcio Dias - Department of Computer Science and e-Science Research Institute, University of Durham, UK - Debra J. Richardson - Donald Bren School of Information and Computer Sciences, University of California Irvine, USA - SEA Group 20 2005 by H. Muccini, WADS 2005 21. SA diff e1 e1 e1e1 e1 e1 in(x) in(x)in(x)S0S1S0S0S1S1in(a) in(a)in(a) e10e1 e10 e10e1e1 e4e4 e4out(z)out(z)out(z) e4e4 e4S2in(y) S2S2in(y)in(y)S4S4S4 S5S5 S5 e8in(z) e5e8 e8in(z) in(z) e5out(w)out(w)e7 e6 S3e7e7S3S3 S5' S5'in(b) in(b)in(b) e6 e6S7e0S7S7e0e0 S6out(c)S6 S6out(c) out(c)e9e9 e9e6e6e6 ATC1 in TATC2 in TATC1 in T SEA Group 21 2005 by H. Muccini, WADS 2005