Engineering Self-Testable Autonomic Software

NORTH DAKOTA STATE UNIVERSITY DEPARTMENT OF COMPUTER SCIENCE

© NDSU SOFTWARE TESTING RESEARCH GROUP

Engineering Self-Testable Autonomic Software

Presenter: Tariq M. King

EASe 2011April 27-29, Las Vegas, NV, USA

A COMPARATIVE CASE STUDY

© NDSU SOFTWARE TESTING RESEARCH GROUP 2

Outline Motivation Background Applications Experiments Lessons Learned Related Work Conclusion

© NDSU SOFTWARE TESTING RESEARCH GROUP

Motivation Self-* features in autonomic computing

systems may incorporate dynamic software adaptation System adds, removes, replaces its own

components @ runtime

Raises reliability concerns asnew faults may be introduced due to dynamic adaptation

Most research neglects that self-test should be an integral part of these types of systems

3

A NEGLECTED SELF-* CHARACTERISTIC

Self-Configure

Self-Optimize

Self-Heal

Self-Protect

Self-Test

FAULTS? FAILURES?


King (2009) describes an approach that seeks to seamlessly integrate runtime testing into the workflow of Autonomic Managers (AMs) Idea: Instantiate new

AMs that are tailoredfor testing activities

Test Managers (TMs)monitor, intercept & validate the change requests generated byAMs to realize AST

Managed Resource

Background AUTONOMIC SELF-TESTING (AST)

Monitor

Analyze Plan

ExecuteKnowledge

EffectorSensorAutonomic Manager

Monitor

Analyze Plan

ExecuteKnowledge

Autonomic ManagerTest Manager


Under AST, test managers can operate according to two different validation strategies: Replication with Validation (RV) – tests

changes using separate copies of managed resources

Safe Adaptation with Validation (SAV) – tests changes in-place, directly on managed resources

Supported by an OO framework for implementing prototypes of self-testable autonomic software Java, Reflection, Threads, XML-Based Policies Interfaces with Automated Software Testing

Tools

Background AST STRATEGIES & TOOL SUPPORT


During preliminary investigations, we sought to locate freely available autonomic software that could be used to demonstrate our ideas on AST

When no such applications could be found, focus shifted to development through our own avenues: REU on Autonomic Computing

http://www.cis.fiu.edu/reu/

CVM Research Projecthttp://cvm.cs.fiu.edu/

Applications PRELUDE


Stevens (2007) developed a self-configuring data structure to demonstrate ideas on AST

Applications AUTONOMIC CONTAINER

ACTLightweight

StructuralAdaptations

RV Strategy


Ramirez (2008) extended the ACT prototype by applying it in the context of short-term scheduling A pool of agents service jobs according to a

high-level scheduling algorithm, and self-configure at runtime, e.g., InteractiveFCFS, Batch SJN

Job requests stored in an autonomic container

Self-Optimization is achieved by adding or removing agents in proportion to the workload

A subsequent point release introduced Self-Protection by monitoring exception throwing

Applications AUTONOMIC JOB SCHEDULER


Applications AUTONOMIC JOB SCHEDULER

Structural &Behavioral Adaptations

AJSLightweight(More Complex

than ACT)

RV Strategy


CVM is a model-driven platform for realizing communication services Collaboration: FIU & Miami Children’s

Hospital

Separates concerns of modeling, synthesis, coordination, and delivery of user communication Users specify their communication

requirements, which are represented as schemas

Schemas are then transformed into executable control scripts, which are used to configure the underlying network devices and protocols

Applications COMMUNICATION VIRTUAL MACHINE


Allen (2009) leverage AC and open-platform APIs for integrating communication services into CVM

Applications COMMUNICATION VIRTUAL MACHINE

CVMReal-World

BehavioralAdaptations

SAV Strategy

NCB LAYER


Conducted three sets of experiments as part of the case study: Development Experiments – compared

the relative effort required to build the autonomic and self-test infrastructures, and various prototypes

Performance Experiments – compared the runtime performance of different versions and variants of the prototypes

Test Quality Experiments – compared the effectiveness of test sets in revealing faults and exercising the program code of the prototypes

Experiments OVERVIEW


Experiments were performed on windows-based Intel® Core 2 Duo 2.4GHz PCs with 2–4 GB RAM

Several tools were used to support automatically collecting results: Eclipse Metrics – source code size and

other complexity metrics for development experiments

Eclipse TPTP – thread and memory analysis for performance experiments

JUnit, Cobertura, MuJava – test failures, code coverage, and mutation analysis for test experiments

Experiments SETUP ENVIRONMENT


Experiments DEVELOPMENT

COMPARISON OF AUTONOMIC & TEST INFRASTRUCTURES


Experiments DEVELOPMENT (CONT.)

COMPARISON OF PROTOTYPE DEVELOPMENT


Non-Self-Test vs. High Transparency Self-Test Used Java RMI to develop a self-test version

of AJS with a distributed RV self-testing process

Performed identical sets of user actions on both systems and compared memory and thread usage

Results: Less than 1% difference

Naive vs. “Enhanced” Interleaving Self-Test Introduced timeout into TM threads of CVM Purpose: Seek to achieve better interleaving

in favor of core system over self-test during SAV self-testing

Experiments PERFORMANCE


Experiments PERFORMANCE (CONT.)

SKYPE TWO-WAY

TOSMACK

THREE-WAY

COMPARISON OF SELF-TEST VARIANTS CVM


Experiments TEST QUALITY

MUTATIONSETUP


Experiments TEST QUALITY (CONT.)

MUTATION RESULTS

CODE COVERAGE RESULTS


RV can be used to provide a transparent self-testing process, while SAV requires optimizations.

Using TMs as a test harness was convenient but problematic: Complexity, Synchronization Safety mechanisms (locks) were instrumental

Selecting and tailoring open-source testing tools presented unnecessary difficulties

Benefits: Prevention of Runtime Faults Testing Aligned with Development

Lessons Learned


“Testing and assurance are probably the least focused phases in engineering self-adaptive software…” – Salehie (2009), ACM Trans… King et al. (2007), and Zhang et al. (2007)

Da Costa et al. (2010) JAAF+T: Java Self-Adaptive Framework +

Test Modify MAPE structure to include a self-test

activity

Munoz & Baudry (2009) Artificial Table Testing of Adaptive Software

Related Work


Presented a case study on the engineering of autonomic software with self-testing features

Reached a level of “enablement”, still many open research issues: Dynamic regression test case selection and

scheduling based on runtime testing goals Runtime test case maintenance Reducing overhead of runtime testing

Current Directions: Cloud, Virtualized RV.

Conclusion


Acknowledgements Dr. Masoud Milani, FIU

Dr. S. M. Sadjadi, FIU

Participants of the REU Summer 2006 and 2007 Programs on Autonomic Computing at FIU

Students and collaborators on CVM project

This work has been supported in part by the NSF under grants IIS-0552555, and HRD-0833093


Thank You!Questions?

¿Preguntas?問題Sawwalвопросы質問domandeερωτήσεις

Documents

Engineering Self-Testable Autonomic Software