SmarTest: Proposal for accelerating the detection of …segura/files/presentations/sanchez15...SmarTest: Proposal for accelerating the detection of faults in Drupal Ana B. Sánchez,

SmarTest: Proposal for accelerating the detection of faults in Drupal

Ana B. Sánchez, Sergio Segura and Antonio Ruiz-Cortés

Ana Belén Sánchez #DrupalcampSpain2015

Ana B. Sánchez #DrupalcampSpain2015

¿Who am I? Where do I come from?

Ana Belén Sánchez [email protected]

Research Group of Applied Software Engineering

(www.isa.us.es)

Escuela Técnica Superior de Ingeniería Informática

University of Seville

I am a researcher and a Teacher


¿Why am I here, in a Drupal Conference?

Ph.D. in automated testing on variability-intensive systems


Testing problems

Too many configura-ons!

Time-‐consuming


Testing variability-intensive systems

Test case selection techniques reduce the test space by selecting a portion of the configurations to be tested.



Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal.



Artificial models to test

How to evaluate these testing techniques?


Towards real highly-configurable tools

Studied Tools


Testing of real highly-configurable tools

1.  Available code

2.  Detailed fault reports

3.  Automated test cases

4.  Extensive documentation

5.  > 630,000 users and developers

6.  > 30,000 modules


Our approach

Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing in the Wild: The Drupal Case Study Published in Software and Systems Modeling Journal, 2015

(http://www.isa.us.es/publications/type/article-journal/9998/variability-testing-wild-drupal-case-study)


Our approach: The Drupal case study

Module .info files

Drupal documentation

Module JIT


The Drupal variability model Drupal 7.23

Node Filter CtoolsFile

Options

Path

Blog Forum Ctools access ruleset

Ctools custom content

Views content

Field Field UI

Field SQL storage

ImageSystem UserViews

Views UITaxonomy

Comment

Text

Date

Token

Forum Requires Taxonomy Views content Requires Views Panels IPE Requires Ctools Rules Requires Entity tokens Forum Requires Options Taxonomy Requires Options Panel Nodes Requires Ctools Rules scheduler Requires Entity APIForum Requires Comment Date views Requires Ctools Pathauto Requires Token Rules scheduler Requires Entity tokensImage Requires File Date views Requires Views Pathauto Requires Path Rules UI Requires Entity API Views Requires Ctools Panels Requires Ctools Rules Requires Entity API Rules UI Requires Entity tokens Views UI Requires Ctools

PathautoIMCE

Date views

Date popup

Date API

WebForm

Link EntityAPI

Entitytokens

CKEditor Captcha

ImageCaptcha

Features Panels

Panel nodes

Panels IPE

LibrariesAPI

JqueryUpdate

GoogleAnalytics

Rules

Rules UIRulesscheduler

Backupmigrate

MandatoryOptional

48 modules 21 dependencies


Drupal information

LoC

Module Size

Changes Commits

Tests 352 test cases 24152 assertions


Drupal information

CC

Cyclomatic Complexity

Drupal Installations

Drupal Contributors


Drupal faults

Bug tracking System

3.232 Faults in v7.23 160 integration faults:

•  132 caused by 2 modules




Statistical study

Module Size Faults Module

Changes Faults

Faults v7.22

Faults v7.23

Module CC Faults


Statistical study

Module Developers Faults

Tests ? Faults ?

Optional Modules Faults

91 %

Core Mandatory Modules Faults

41 %


Test case prioritization criteria

Size–driven Criterion

Fault–driven Criterion

CC-driven Criterion

Change–driven Criterion

Priori-za-on criteria based on previous sta-s-cal study


Evaluation

94.2 % Test case selection

Fault-driven Prioritization

2.

Could we accelerate the detection of faults in Drupal using previous information?

86.2 % Test case selection

1.


Applying our proposal to Drupal

SimpleTest


Applying our proposal to Drupal

Analyzing the SimpleTest module

Main view SimpleTest


Our proposal: SmarTest

Main view SmarTest

Customizable dashboard







Test Prioritization SmarTest



Continuos feedback in real time SmartTest



¿What improvements do we add to SimpleTest?

SimpleTest ü  Designed to make testing easier

ü  A powerful automated testing

ü  Selector to select tests to execute

ü  Detailed inform. about test execution

ü  Possibility to follow the trace of errors

SmarTest ü  Customizable Dashboard with run-time extracted data to guide the testing

ü  Test prioritization fo detect faults faster

ü  Automated testing with feedback in real time

ü  Time-out option to automatically stop the test execution



Things that we are working on…

•  Improve the dashboard with more information and new widgets

•  Include new test prioritization criteria and multi-prioritization criteria

•  Study the integration of SmarTest with other test systems: Behat, phpUnit…

•  Improve the SmartTest UI

•  Add the SmarTest to Drupal sandbox

•  ¿Any suggestions?


[email protected] anabsj

Documents

SmarTest: Proposal for accelerating the detection of …segura/files/presentations/sanchez15...SmarTest: Proposal for accelerating the detection of faults in Drupal Ana B. Sánchez,