Software Testing Process & Trend

  • Published on
    13-Sep-2014

  • View
    3.436

  • Download
    3

DESCRIPTION

This is the slide deck that KMS's Director of Delivery presented at Can Tho University on Saturday, September 28th, 2013

Transcript

  • SOFTWARE TESTING METHODOLOGY & TREND

    September 2013 KMS Technology - http://kms-technology.com

    Vu Pham Delivery Director vupham@kms-technology.com

  • AGENDA

    Software Testing Process & Trends 20

    Automation Testing & Tools 20

    Future of Software Testing 20

    Q&A 30

    2

  • 2013 KMS Technology

    SOFTWARE TESTING PROCESS

  • AGENDA

    Testing Process Evolution

    Components of Testing Process Framework

    4

  • DEVELOPMENT PROCESS EVOLUTION

    60s: Waterfall 80s: RUP 00s: Agile 70s: V-Model

    5

  • DEVELOPMENT PROCESS EVOLUTION (CONT.)

    6

    Client Advantages Disadvantages

    Simple model and easy to manage Applicable for small software

    Big Design Up Front Defect detected at late phases High amounts of risk and uncertain

    Early testing involvement Clear relationship between test phases

    development phases

    Still possess limitation of sequential model Require high amount of documentation Duplication of testing effort

    Risk and uncertain are managed Testing activities and process are

    managed

    Heavy documentation Late customer involvement only at UAT

    Adaptable to changes Early client involvement - Avoid

    unrealistic requirements Avoid spending time on useless

    activities

    Require high-capable people Need representative from client Problem scaling up the architecture

  • SO HOW TESTING IS CHANGED?

    Black-box testing System testing Functional testing Part-time tester

    Grey-box testing System/Integration

    testing

    Functional testing Full-time tester

    White-box testing System-system Non-functional testing Fit-for-Use Professional tester

    7

    60 80: Nice To Have ~ 90: Should Have 00 : Must Have

  • AGENDA

    Testing Process Evolution

    Components of Testing Process Framework

    8

  • TESTING CENTER OF EXCELLENCE

    Test Solutions

    Automation Testing

    Performance Testing

    Mobile Testing

    Specialty Testing

    Best Practices

    Process Assessment

    Testing Estimation

    Continuous Process

    Improvement

    Exploratory/Risk-

    based Testing

    Quality

    Policy

    Guidelines &

    Templates

    Fundamental

    Testing Process

    Quality Metrics &

    Standards

    Plan Test

    Design Test

    Execute Test

    Close Test

    9

    TCoE = Processes + Practices + Solutions

  • WHY TEST SOLUTIONS?

    10

    About the Client Clearleap was the first company providing data streaming solution to offer a complete platform that allows TV everywhere possible

    Business Challenges

    Simulate high volume of concurrent

    users 100,000+ Complete within a tight schedule Limited budget for tool

    KMSs Solutions

    Tool Evaluation: Execute a proof of concept to evaluate both commercial and open source tools

    Planning: Determine a test strategy, approaches

    Test Design and Development: Design and develop scalable load testing architecture

    Execution and Reporting: Perform load testing and analyzing/reporting test results

    Achievements

    Developed a scalable solution based on Jmeter

    Extremely reduced the cost of testing and tremendously increased ROI

    Found critical performance issues

  • WHY TEST SOLUTIONS? (CONT.)

    It takes months to build up solution from beginning

    Cost of commercial tools v.s open source tools

    Effective solutions differentiates us from other vendors

    Typical Testing Solutions:

    Automation testing (web, desktop, mobile)

    Performance/Load Testing

    Security Testing

    Database/ETL Testing

    11

  • WHY BEST PRACTICES?

    12

    About the Client Global company supporting clinical trials in 67 countries. The Client offers services which include behavioral science, information technology, and clinical research

    Business Challenges 100% on time delivery with zero critical

    bugs Complicated paper process following

    FDA regulations Various testing platforms for both mobile

    devices and desktop

    KMSs Solution Process Establishment: Identify gaps in

    current process; Leverage start-of the-art practices

    Process Improvement: Define and measure performance /quality metrics

    Lifecycle Testing: Perform all lifecycle testing activities

    Test Automation: Develop an automation framework to shorten test cycle

    Achievements New process helps reducing 60% testing

    effort No critical defects identified during 1 year

    of engagement Moved paper work process to test

    management system open new trend in clinical trial industrial

  • WHY BEST PRACTICES? (CONT.)

    13

    Best practice improves outcome of activities

    Best practice has been proved of it effectiveness

    The more practices we use the higher maturity we are

    Typical Testing Best Practice:

    Review and Lesson-Learnt

    Root Cause Analysis

    Risk-based/Exploratory Testing

    Estimation Method, ROI Model

    Quality Metric Dashboard

  • Definition: CPI is an ongoing effort to improve quality of products, services, or processes

    In software testing CPI is seeking for improvement of: Quality

    Productivity

    Cost of Quality

    Time to Market

    CONTINUOUS PROCESS IMPROVEMENT

    14

    Assess

    Plan Implement

    Evaluate

  • Three metric categories in practice:

    Product Quality Metrics How good the overall quality of the product

    Process Effectiveness Metrics How the processes of delivery are performed

    Testing and Test Automation Metrics Detail status of testing activities, test outcome

    Metrics are standards of measurement by which efficiency, performance, progress, or quality of a plan, process, project or product can be assessed with the aim to support continuous improvement

    Wikipedia

    QUALITY METRICS

    15

  • Defects by Status Open Defects by Severity Open Defects by Severity & Functional Area Open Defects by Severity & Release Open Defect Aging

    Product Quality Metrics

    Defect Identification in Pre-Prod / Prod Weekly Defect Rates per Environment Defect Escape Ratio

    Defects by Phase Found / Functional Area Defects by Origin / Functional Area

    Process Effectiveness Metrics

    Test Coverage Planning Execution Status / Execution Rate by

    Functional Area/Cycle Defect Rejection Ratio Test Productivity

    Testing Metrics

    Percent Automatable Automation Progress Percent of Automated Testing Coverage

    Test Automation Metrics

    QUALITY METRICS (CONT.)

    16

  • Definition: Risk-based testing is testing method that base on identified risks to

    determine the right level of quality

    prioritize the tests and testing effort

    focus on most important testing areas first

    with the aim to be clear of current quality status and to get the best return by the time completing testing

    RISK-BASED TESTING

    17

  • EXPLORATORY TESTING

    18

    A style of testing in which you explore the software while

    simultaneously designing and executing tests, using feedbacks from the last test to inform the

    next. Elisabeth Hendrickson

    This type of testing helps: Discovering unknown and un-detected

    bugs

    Testers in learning new methods, test strategies, think out of the box

  • 2013 KMS Technology

    AUTOMATION TESTING & TOOLS

  • AGENDA

    Software Test Automation

    Software Performance Testing

    Tools Support Testing

    20

  • THINKING OF AUTOMATION

    Test Automation is

    Business values of Automation

    Greater Coverage More time for QA doing manual exploratory/risk-based

    testing.

    Improved Testing Productivity Test suites can be run earlier and nightly

    Reduced Testing Cycle Help shorten time-to-market

    Doing what manual testing cannot Load testing

    Using Testing Effectively Automation testing reduces tediousness,

    improve team morale

    Increased Reusability Tests can be ran across different platforms and

    environments

    The use of software and tools to perform the testing

    Code-Driven Testing at source code level with a variety of input arguments.

    GUI-Driven Testing at GUI level via keystrokes, mouse clicks, etc.

    21

  • THINKING OF RETURN ON INVESTMENT

    Tool, Implementation, Maintenance, Training,

    etc. Save Time, Early

    Response, Reliable, Repeatable, etc.

    ROI: The most important measurement for test automation ROI (effort): planning, development, maintenance, training, etc.

    ROI (cost): tool license, environment, management, automation resources, etc.

    ROI (quality): found defect, test coverage, etc.

    22

  • END-TO-END TEST AUTOMATION PROCESS

    1

    Assessment

    Evaluation

    2

    Pilot

    Planning

    3

    Design

    Implementation

    4

    Execution

    Report

    5 Maintenance

    23

    Plan Test

    Design Test

    Execute Test

    Close Test

  • ASSESSMENT & EVALUATION

    Assessment Understand organization vision,

    priorities, process & methodology

    Understand Application & Technology

    Identify the Test requirements

    Evaluation: Vendor discussion (optional)

    Tool evaluation

    Recommendations

    Finalize Testing tools

    24

  • PILOT & PLANNING

    Pilot Do Proof of Concept

    Define Test process

    Finalize Test Approach & Methodology

    Define Entry & Exit criteria

    Planning: Identity test requirements, test

    cases for Automation

    Set up test environment

    Define Automation framework

    Finalize Resources and Test schedule

    25

  • DESIGN & IMPLEMENTATION

    Design Define standards, guidelines, Pre

    & Post test procedures

    Design input, output data

    Monitoring tools and report metrics

    Design Automation framework

    Implementation:

    Build driver script, actions, keywords, data driven

    Build scripts

    Validate and run under application test

    26

  • EXECUTION & MAINTENANCE

    Execution & Report Setup environment

    Run and schedule tests

    Provide detailed and summary report

    Provide automation handbook & training

    Maintenance: Implement new change request

    Define new enhancement

    Keep up-to-date with new function of application under test.

    27

  • AUTOMATION CHALLENGES

    High up-front investment cost

    Demanding of skilled resource

    Selection of the best testing tools and approach

    Ineffective collaboration process

    Persuade stakeholders to say Yes

    28

  • AGENDA

    Software Test Automation

    Software Performance Testing

    Tools Support Testing

    29

  • PERFORMANCE TESTING

    Determines

    User expectations

    System constrains

    Costs

    Focuses on

    To answer How many?

    How much?

    What happens if?

    Speed

    Scalability

    Stability

    30

  • CROWD SPEED AVAILABITY

    How many users before crashing?

    Do we have enough hardware?

    Where are the bottlenecks in the system?

    Is the system fast enough to make customers happy?

    Will it slow down or will it crash?

    Did I purchase enough bandwidth from my ISP?

    How reliable is our system

    Will our system cope with the unexpected?

    What will happen if our business grows?

    The failure of an application can be costly Locate potential problems before our customer do Assume performance and functionality under real-work conditions Reduce infrastructure cost

    31

    PERFORMANCE TESTING OVERVIEW

  • THE FUTURE CHALLENGES OF AUTOMATION

    32

  • AGENDA

    Software Test Automation

    Software Performance Testing

    Tools Support Testing

    33

  • TESTING TOOLS LANDSCAPE

    34

    ALM Application Life-cycle Management

    Purpose: communicates across multiple project teams

    Typical Tools: Rally, VersionOne, HP ALM

    TMS Test Management System

    Purpose: manages requirement test matrix

    Typical Tools: HP QC, Test Link, QAComplete, qTest

    DTS: Defect Tracking System

    Purpose: manage defect

    Typical Tools: BugZilla, Jira, Mantis

    ATT: Automation Testing Tools

    Purpose: Regression and specific tests

    Typical Tools: QTP, TestComplete, Selenium, Watir, JMeter, LoadRunner

  • NEW TREND IN TESTING TOOLS

    35

    Auto-sync requirements, test cases & defects

    Import/export, integrate with other systems

    Capture tools integrate into defect tracking tool

    Save Time & Less Work

    View, mark result, update test cases and defects without leaving the target test application

    Create defect quickly Faster & Easy to Use

    Easy to customize new features

    Integrate into many specified tools

    Customization & Integration

    Control and keep track of changes, assignments

    Track status across lifecycles

    View the real-time status, statistical data, associated trends

    More Control, Visibility

    Flexible and low cost Cloud Deployment

  • 2013 KMS Technology

    FUTURE OF SOFTWARE TESTING

  • WHERE WE ARE?

    Ho Chi Minh City and Hanoi are continuously in the top 10 emerging IT outsourcing cities (07 Today) http://www.tholons.com/Top50_article.pdf

    37 Confidential

    What is typical ratio of Testers in VN IT company?

  • WHERE WE ARE? (CONT.)

    38 Confidential

    Ho Chi Minh city is destination of global outsourcing in testing

  • WHAT ARE OUR OPPORTUNITIES?

    Facts: Testing outsourcing

    market value triple increased for every 4 year

    Many VN outsourcing companies are testing focus: Logigear, TMA, Global CyberSoft, KMS, FSOFT

    39 Confidential

  • FUTURE OF SOFTWARE TESTING

    1. Faster Higher Stronger Faster release

    Need value from every hour spent on testing

    Higher quality

    Greater test coverage of specified and implied requirements

    Stronger capability

    Not only functionality but also performance, security, usability

    Ability to develop test solutions

    2. Complicated technology/application platform Cloud Computing, Mobile, Enterprise System

    40

  • FUTURE OF SOFTWARE TESTING (CONT.)

    3. Global testing team global competition Communication, Crowd-source Testing ...

    4. Automation testing is must More effective solutions are needed

    5. Less on processes, more on practices Agile, Exploratory, Rapid testing

    41

  • SUMMARY

    1. Testing is crucial for today business

    2. It becomes professional of choice

    3. Vietnam is destination of testing outsourcing

    4. Automation testing is must in future

    5. Requires intellectually, analytically and creatively mindset

    6. It takes years to become good

    7. Cant be good if just learn from daily works

    8. Is fast-paced career advancement

    42

  • 2013 KMS Technology

    ABOUT KMS TESTING SERVICE

  • KMS QA SERVICES FRAMEWORK

    Testing Tools

    Proprietary Tools

    Commercial Tools

    Open source Tools

    Automation

    Frameworks

    Test Processes

    Process Assessment

    Best Practice

    Implementation

    Continuous Process

    Improvement

    Quality and Project

    Management Metrics

    Test Management

    Scrum QA

    Services

    Regression QA

    Services

    Automation QA

    Services

    Performance & Load

    Testing Services

    Code Analysis

    Services

    Flexible Staffing Option

    Streamlined Processes & Frameworks

    Tools & Automation Strategic Solution/ Best Practices

    Test Planning & Estimation

    Test Design & Implementation

    Test Execution QA Metrics

    Driven Monitoring

    QA Metrics Driven Process Improvements

    44 KMS Technology Confidential

  • Sprint Planning &

    Communication

    Plan tasks

    Estimate tasks

    Coordinate tasks

    Participate in Scrum

    Leverage qEstimate

    Test Scenario & Test Case Creation

    Create Scenarios

    Create Test Cases

    Cross-Team review

    Report Progress

    Test Scenario/Case mapping with Mind Mapping Tool

    Test Execution & Defect

    Identification

    Execute Test Cycles

    Log Defects

    Leverage qTrace for documenting defects

    Defect Management

    Verify defect fixes

    Follow up on failures

    Monitor Aging Defects

    Root cause analysis on defects

    SCRUM QA SERVICES

    45 KMS Technology Confidential

    Scrum QA Services

    Regression QA Services

    Automation QA Service...

Recommended

View more >