Software Testing in the Cloud •System Migration STITC (Software Testing in the Cloud) •Testing in

  • View

  • Download

Embed Size (px)

Text of Software Testing in the Cloud •System Migration STITC (Software Testing in the Cloud)...

  • Software Testing in the Cloud

    Tauhida Parveen, PhD

  • About Me

    • University Department Chair, Software

    Engineering, Keiser University

    • PhD, CS, Florida Institute of Technology

    • MBA, University of Central Florida

    • ISTQB Certified Tester (Foundation, Manager, Agile)

    • Independent consultant and trainer

    • Software Quality Engineer/ Tester

    • Yahoo!, Sabre, Wikimedia, …

  • About Me…

    Vice President & COO

  • Motivation

    • Cloud computing is here to stay

    • Changing the way we engineer software

    • Leverage the power of cloud in testing

  • Objectives

    What is the cloud about?

    How is it different from traditional computing?

    What is testing in the cloud?

    What does it mean for testers?

    Where to start?

  • What is Cloud Computing?

    An old idea whose time has finally come

    If computers of the kind I have advocated become the computers of the future, then computing may someday be organized as a public utility just as the telephone system is a public utility. The computer

    utility could become the basis of a new and important industry.

    John McCarthy, 1961

  • What is Cloud Computing?

    Virtualized hardware

    SOA, On-demand, utility model

    Rapid elasticity

    Resource abstraction techniques



    Not in your control (mostly) / things are taken care of

  • Cloud Services

  • Cloud Services

  • Cloud Services

  • Cloud Delivery Models

    Public • Services offered by a provider that are publicly available, can be used with a

    contractual agreement

    Private • Virtualized resources available as a service owned by individual organization

    • Client managed, customization, high efficiency, privacy and security policies


    • Mix of public and private

    • Extend private cloud via VPN

    • Good candidate for incremental migration

  • Cloud Benefits

    Faster, cheaper, better


    • Multiple test beds for multiple release testing

    • Instant setup, tear down


    • Virtualized environments

    • Scalable resources


    • Pay-per-use

  • What Does it Mean for Testers?

    How would you test this?

  • Software Testing in the Cloud

    Lies at the intersection of three key areas

    • Software Testing

    • Cloud Computing

    • System Migration

    STITC (Software Testing in the Cloud)

    • Testing in the cloud

    • Testing of the cloud

    • Migrating testing to the cloud

  • Traditional Testing

    Low asset utilization

    Scalability: Long time to increase capacity

    Long time to build datacenters

    Difficult to manage

    Duplicate test systems

    Creates unnecessary waste

    Testing in the Cloud

    Improved asset utilization

    Less time (instantaneous) increase and reduction in capacity

    Purchased as a service from cloud providers

    Better management and increased productivity

    Aggregated system

    Cleaner, greener testing

  • Case Studies

    Case Study #1: Concurrent execution of JUnit test cases

    Case Study #2: Concurrent execution of GUI testing of iOS applications

    • It can take a large number of test cases to cover all the variations that users could experience

    • Different configurations

    • OS versions (iOS ?)

    • Device Orientations (Portrait, Landscape)

    • Screen sizes (3.5", 4")

    • Form Factors (iPhone, iPad, Universal)

  • Case Study #2

  • Software Testing in the Cloud

  • Software Testing in the Cloud


    • Process

    • Technology

    • People

    Cloud testing strategy

    Roles and responsibilities

    Can’t be achieved as an execution-only


  • Management: Process






    •Network connectivity





    •Project management


    •Faster execution

    •Faster feedback

    •Efficient testing process

    •Value proposition




    •Vendor lock

    •Tool chain




  • Management: Process

    Migration Strategy

    • Where are you now?

    • Where do you go next?

    SMART-T: A decision framework to migrate testing to the cloud


    Faster Cheaper Better

    Business Drivers




    Types of


    Technical Factors

    Types of











    Workshop Pilot Study



    Trial Migration








    Actual Migration




    Operational Results

  • Management: Process

    Evaluate current overlapping activities

    • Agile development and testing

    • Test early, test often

    • TDD, Unit testing

    • JUnit in the cloud

    • Google Test in the cloud

    • HadoopUnit

    • Faster feedback

    • Automation

    • Regression testing

  • Management: Process

    Cost model

    • How much does it cost you to test now?

    • Do you have a test lab? Machines for every project?

    • How often do you configure the machines?

    • How often do you upgrade them?

    ABC: Activity Based Cost model – Identify and put $$ to all the big and small activities

  • Management: Technology

  • Management: Technology

    Testing as a Service

    Distributed processing technologies

    Test execution environments

  • Management: Technology

    Amazon Web Services (AWS)

  • Management: People

    Will your testers be able to adapt to cloud testing?

    • Testing at many levels – User Interface, Services, Governance, Process, Security, Integration


    • Beyond unit testing

    System administrators?

    • Cloud service providers


    Continuing education

    Testing skills, programming skills

    Communication skills

  • Testing in the Cloud: Strategy

    Does testing in the change the foundation of testing?

    Does it involve new technique, technology, and domain?

    How is it going to effect your current tasks? (good/bad)

    Do you need new skills?

    Are there different types of testing?

    Are there tools to help you?

    Risks associated with cloud testing

    Assumptions, constraints

  • Testing in the Cloud: Strategy

    Critical Success Factors (CSF)

    What are the CSF for testing in the cloud?

    • Automation?

    • Skilled testers?

    • Test lab/environment?

    How do your measure them?

    • Post release evaluation?

    • Number of bugs for the release

    • Test lab setup and tear down?

  • Testing in the Cloud: Strategy

  • Roles and Responsibilities

    Developers: Unit testing, Integration testing

    Test Analyst: Implement testing in the cloud strategy

    Technical Test Analyst: Testers

    Customers: Acceptance testing

    IT department: POC to cloud provider, monitoring, availability

    Cloud providers: SLA, services testing

  • Lessons Learned

  • Lessons Learned

    Testing Environment is a legacy system

    Acceptability of partial migrations

    Selecting/Preparing test execution environment

    Configuration in the cloud is challenging

    The nature of the test cases

    False negatives

  • What Does it Mean for Testers?

    How would you test this?

  • Key Points

    Software testing is challenging activity

    SOA, virtualization, computational grid, and cloud computing bring efficiency in development process/consumer need

    Leverage these technologies in software testing

    Plan, start early, know your goals

    Understand the holistic system architecture

    Understand the migrated test data

    Establish communications with the vendor

  • Key Points

    Cloud computing is not hype!!!

    Provides the infrastructure to process Big Data

    It’s changing the testing landscape

    It’s changing the testing profession

    We have had similar shifts in the past