Understanding the Test Automation Culture of App Developers

  • View
    202

  • Download
    0

Embed Size (px)

Text of Understanding the Test Automation Culture of App Developers

Slide 1

Understanding the Test Automation Cultureof App DevelopersPavneet Singh Kochhar1, Ferdian Thung1, Nachiappan Nagappan2, Thomas Zimmermann2, David Lo1

1Singapore Management University2Microsoft Research

{kochharps.2012,ferdiant.2013,davidlo}@smu.edu.sg, {nachin,tzimmer}@Microsoft.comInternational Conference on Software Testing, Verification and Validation (ICST15)

1

SmartphonesSmartphones have become pervasiveWorldwide sales increased by 42.3% from 2012 to 20132

Android has 78.4% of the smartphone market1Millions of apps available[1] Gartner, Gartner says annual smartphone sales surpassed sales of feature phones for the first time in 2013

The increasing reliance on open source projects implies that it is important to investigate potential weaknesses in its development process.2

Study GoalsUnderstand the test automation culture of app developersUnderstand the pain points of developers3Are mobile apps adequately tested?

Understanding Test Automation CultureAnalyse over 600 Android appsExecute test cases Assess test adequacySurvey Android & Microsoft developersUnderstand common testing tools usedChallenges faced by them

4

Focus: open source + Java

4

OutlineMotivation and GoalsTest Adequacy Data Collection & Basic StatisticsEmpirical ResultsRecommendationsConclusion and Future Work

5

Test AdequacyTest Adequacy CriterionProperty that must be satisfied for a test suite to be thorough. Often measured by code coverage.

Code CoveragePercentage of the code executed by test casesLine coverageBlock coverage

6

OutlineMotivation and GoalsTest Adequacy Data Collection & Basic StatisticsEmpirical ResultsRecommendationsConclusion and Future Work

7

Data CollectionRepository for Free and Open Source Android applicationsWe collect apps hosted onIn total, we have 627 appsOur dataset has large and popular apps8

First Survey3905 distinct email addresses of developers of 627 appsAsk questions about testing tools used and challenges facedWe receive 83 responses (Response rate 2.13%)Unit of analysis is individual developer

9

First SurveyQuestions:How do you test your app code?Do you use any test automation tools? What are the challenges you face during testing either manually or using automated tools?10

Second SurveyWe improve our survey questions based on the responses of first survey Email 678 developers in MicrosoftWe receive 127 responses (Response rate 18.73%)Unit of analysis is individual developer11

Second SurveyQuestions:How do you test your app code? - Manually, Automated tools, Dont testWhat type of testing do you do? - Unit testing, integration testing, system testing, regression testing etc.What automated testing tools you use?12

Second SurveyQuestions:Why do you use testing tools? - Generating test cases, executing test cases etc.Do you face these challenges? - Time constraints, compatibility issues, lack of exposure to tools, poor documentation etc.What are the top 2 things you look for/need/would like to see?13

Basic Statistics14ToolsNumber of Apps% of AppsWithout Test Cases53885.81%With Test Cases8914.19%

Distribution of Apps w.r.t Test CasesTotal 627 apps

Basic Statistics15Distribution of Apps w.r.t LOC

Basic Statistics16Distribution of Apps w.r.t Developers

OutlineMotivation and GoalsTest Adequacy and Code MetricsData Collection & Basic StatisticsEmpirical ResultsRecommendationsConclusion and Future Work

17

Research Questions18RQ1: Are Android applications well tested? RQ2: Do Android developers use automated testing tools to test their applications? What are the tools commonly used and the challenges faced by them while testing their applications?

RQ3: Do Microsoft developers use automated testing tools to test their applications? What are the tools commonly used and the challenges faced by them while testing their applications?

Research Questions19RQ1:Current State of Testing in Android Applications

RQ1: Current State of Testing20Distribution of Test Suites (89 apps)

RQ1: Current State of Testing21Line Coverage (41 apps)

RQ1: Current State of Testing22Block Coverage (41 apps)

RQ1: Current State of Testing23Only 14.19% (89 out of 627) of the apps contain test cases

Only 0.64% (9.75% of 14.19%) have line coverage above 40%Results

Research Questions24RQ2:Survey of Android Developers

RQ2: Survey of Android Developers25Android Automated Testing Tools UsageToolsNumber of RespondentsJUnit18MonkeyRunner8Robotium7Robolectric6Android unit testing framework6Monkey1No tool35

RQ2: Survey of Android Developers26Challenges faced by developersToolsNumber of RespondentsTime constraints20Compatibility issues16Lack of exposure11Cumbersome tool9Emphasis on development6Lack of organization support5Unclear benefits4

RQ2: Survey of Android Developers27Challenges faced by developersTime Designing and implementing test cases takes some extra time, which makes it difficult to finish the project in timeCompatibility I tried robolectric, but ran into several issues, that were probably also related to the fact that I am using Scala on AndroidUsability I think Monkey runner is kinda cumbersome, and breaks easily when changing layout options

RQ2: Survey of Android Developers28Challenges faced by developersLack of Support The advice I was given was ... not bother with trying to use the Android testing tools/frameworksPoor Documentation Testing is documented there, but not very well and there should be far more informationSteep Learning Curve I fear it would represent a strong learning curve

RQ2: Survey of Android Developers29Developers use tools such as JUnit, Robolectric, Robotium etc.

However, large number of developers prefer manual testing

Developer face challenges such as time constraints, compatibility issues, lack of exposure, cumbersome tools, unclear benefits, poor documentation etc.Results

Research Questions30RQ3:Survey of Microsoft Developers

RQ3: Survey of Microsoft Developers31Types of Testing

RQ3: Survey of Microsoft Developers32Automated Testing Tools UsageToolsNumber of RespondentsVisual Studio35Internal tool8Selenium7Microsoft Test Manager5Others (Qunit,Robotium etc.)27

RQ3: Survey of Microsoft Developers33Automated Testing Tools Usage

RQ3: Survey of Microsoft Developers34Challenges faced by developers

34

RQ3: Survey of Microsoft Developers35Top 3 reasons for using automated testing tools are executing test cases, finding potential bugs, and analyzing code coverage.

Top 3 challenges faced by Microsoft developers are time constraints, compatibility issues, and lack of exposure to tools.

Developers prefer well documented and easy to use toolsResults

RecommendationsNeed to promote new tools that can be easily used by developersNeed tools which can help test difficult casesNeed tools which have good documentationNeed to spread information about tools through online channels36

Many existing test case generation tools have only been evaluated on very small, small, and medium scale projects.36

Conclusion37Only 14% of the apps contain test cases & 0.64% (9% of 14%) of the apps have coverage above 40%.Android developers prefer using JUnit whereas Microsoft developers use Visual Studio, Internal tool, Selenium.Developers face challenges such as time constraints, compatibility issues, lack of exposure, cumbersome tools, unclear benefits, poor documentation etc.

Future WorkExpand the study to include more appsAddress the threats to external validitySurvey more developers to get more responses We plan to develop tools to address pain points.38

Effort of attaining a particular coverage level.Benefit of attaining higher level of coverage, measured for example by number of post-release defects.

38

Thank you!Questions? Comments? Advice?{kochharps.2012,ferdiant.2013}@smu.edu.sgdavidlo@smu.edu.sg{nachin, tzimmer} @microsoft.com

39

SMU-IHiSSoftware Analytics