SOFTWARE TESTING METRICS

“A METRIC IS A QUANTITATIVE MEASURE OF THE DEGREE TO WHICH A SYSTEM, SYSTEM COMPONENT, OR PROCESS POSSESSES A GIVEN ATTRIBUTE”

DAVID TZEMACHWWW.MACHTESTED.COM

MAY 12 2017

AGENDA

• OVERVIEW

• WHAT WE CAN MEASURE USING METRICS

• COMMON METRICS TO EVALUATE TEST PROCESS

• WHY DO WE NEED TO USE METRICS

• TEST METRICS LIFE CYCLE (TMLC)

• TYPE OF METRICS

• FUNDAMENTAL TESTING METRICS

METRICS AND WHAT IT ACTUALLY MEANS?

OVERVIEWONE OF THE MOST IMPORTANT THINGS IN ANY PROJECT IS TO DETERMINE THE FACTORS OF SUCCESS, THESE FACTORS CAN BE

BASED ON QUALITY, COSTS AND TIME.

IN SOFTWARE TESTING PROJECTS, WE WILL NEED TO MEASURE THE QUALITY THAT CAN BE BASED ON MANY CRITERIA INCLUDING

THE 3 DESCRIBED ABOVE.

EXAMPLES

• SPRINT QUALITY IN AGILE PROJECTS (A VELOCITY IS A BUILT-IN TOOL THAT WE CAN USE TO DETERMINE IT).

• THE PROGRESS OF A PROJECT (% OF THE WORK THAT ALREADY COMPLETED/REMAINED).

• THE QUALITY OF TESTING ARTIFACTS (DEFECTS AND TEST DOCUMENTATION).

• SYNCHRONIZATION AMONG TEST AND DEVELOPMENT TEAMS.

• THE QUALITY OF THE PROCESS USED DURING THE PROJECT

• DEVIATIONS FROM THE ORIGINAL PLAN.

COMMON TEST METRICS

WHAT ARE THE MOST COMMON TEST METRICS • HOW LONG WOULD IT TAKE TO COMPLETE THE TESTING PROCESS?

• HOW MANY DEFECTS ARE OPENED FOR THE SAME ROOT CAUSE?

• HOW MUCH EFFORT WENT INTO THE TESTING OF THE PRODUCT?

• HOW MANY TEST CASES ARE DESIGNED PER REQUIREMENT?

• HOW MANY TEST CASES ARE DESIGNED PER RISK (RBT)?

• TEST COVERAGE COMPARED TO THE ORIGINAL ESTIMATIONS.

• THE MEMBER OF TEST CASES THAT WERE EXECUTED.

• DEFECTS DIVIDED BY SEVERITY.

• DEFECTS DIVIDED BY PRIORITY.

• ANY TESTING DELAYS.

• TEST PRODUCTIVITY.

WHY DO WE NEED TO USE METRICS?

SO WHY SHOULD WE THINK ABOUT TEST METRICS?TESTING METRICS ARE THE MOST EFFICIENT AND IMPORTANT WAY TO MEASURE THE QUALITY OF A TESTING PROJECT, INADDITION, WE CAN SPECIFY A FEW MORE REASONS SUCH AS:

• WITHOUT USING METRICS, YOU CANNOT MEASURE THE QUALITY OF THE IMPLEMENTED PROCESS.

• USING TEST METRICS, MANAGERS CAN TAKE DECISIONS ON DIFFERENT PHASES OF THE TESTING PROJECTS AND

CONTINUALLY IMPROVE THE PROCESS (NO MATTER WHAT IS THE TESTING METHODOLOGY).

• MANAGERS CAN JUSTIFY HARD DECISIONS THAT THEY NEED TO TAKE DURING THE PROJECT.

• MANAGERS CAN EVALUATE THE RISKS AND DETERMINE FUTURE PLANS TO REMOVE THEM.

• TEST METRICS WILL HELP TO MONITOR THE SUCCESS FACTORS OF THE TESTING PROJECT (I ALREADY REVIEWED SOME

OF THEM IN THE PREVIOUS PARAGRAPH).

TEST METRICS LIFE CYCLE

PHASES OF TMLC

ID Phase name

1 Analysis The analysis phase is the first phase of the TMLC were we determine the metrics that will use to measure the project/process/testing success.

2 Communicate The first thing that we will need to perform during this phase, is to explain the metrics (And which data should be captured to support them) to the resources in the testing projects (Product managers, team leaders and off course the testing teams).

3 Evaluation We will analysis the data captured by the test team and once it’s approved we will use to as an input to the identified metrics.

4 Report During the last phase of the TMLC, we will generate a report that summarizes the results and conclusions of the test metrics, publish it to all relevant resources and take feedback that we will use to improve the process.

TYPES OF METRICS

BASE METRICS (DIRECT MEASURE)

BASE METRICS ARE THE METRICS WHICH ARE DETERMINED FROM THE RAW DATA

GENERATED BE THE TESTERS THROUGHOUT THE TEST LIFE CYCLE (TLC), THESE DATA

INCLUDE INFORMATION ABOUT DEFECTS STATUSES, TEST COVERAGE AND MORE.

IN ADDITION, USING THIS RAW DATA WE CAN PROVIDE INPUTS TO THE FORMULAS

USED TO DERIVE CALCULATED METRICS.

CALCULATED METRICS (INDIRECT MEASURE)

CALCULATED METRICS ARE BASED ON THE DATA GENERATED IN BASE METRICS

(CONVERSION OF BASE METRICS DATA INTO MORE USEFUL INFORMATION).

USING THIS TYPE OF METRIC, THE TEST LEAD CAN TRACK AND MEASURE THE TESTING

PROGRESS AT DIFFERENT PHASES OF THE TESTING PROCESS.

TYPES OF METRICS

OVERVIEW

LET’S REVIEW SOME OF THE MOST COMMON METRICS THAT WE CAN USE TO DETERMINE

DIFFERENT ASPECTS OF THE TESTING QUALITY, AS YOU WILL SEE, EACH METRIC IS DESCRIBED AS A

SIMPLE MATHEMATICAL FUNCTION.

PRIOR TO USING THIS MATHEMATICAL FUNCTIONS, WE NEED TO UNDERSTAND THE IMPORTANCE OF

THE ABSOLUT NUMBERS DERIVED FROM THE ACTUAL TESTING PROCESS, THIS NUMBERS ARE BASED

ON A FEW BASIC CRITERIA (TOTAL NUMBER OF TEST, NUMBER OF FAILED/PASSED TESTS ETC.) THAT

WE WILL USE AS THE METRIC INPUT, SO THERE IS A HUGH IMPORTANT TO USE THE REAL NUMBERS.

GROUP A: TEST EXECUTION BASED METRICS

METRIC 1: MEASURING THE % OF PASSED TESTS

(PASSED TESTS) / (TOTAL NUMBER OF TEST EXECUTED) *100 = % OF THE PASSED TEST CASES

METRIC 2: MEASURING THE % OF FAILED TESTS

(FAILED TESTS) / (TOTAL NUMBER OF TEST EXECUTED) *100 = % OF THE FAILED TEST CASES

METRIC 3: MEASURING THE % OF THE REMAINING TESTS

(TESTS THAT ARE NOT EXECUTED) / (TOTAL NUMBER OF TESTS) *100 = % OF THE REMAINING TEST CASES

METRIC 4: MEASURING THE % OF THE EXECUTED TESTS

(TESTS THAT ARE EXECUTED) / (TOTAL NUMBER OF TESTS) *100 = % OF THE EXECUTED TEST CASES

GROUP B: METRICS USED TO MEASURE DEFECTS STATUSES

METRIC 5: MEASURING THE % OF FIXED DEFECTS

(NUMBER OF FIXED DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE FIXED DEFECTS

METRIC 6: MEASURING THE % OF INVALID DEFECTS

(NUMBER OF INVALID DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE INVALID DEFECTS

METRIC 7: MEASURING THE % OF WONTFIX DEFECTS

(NUMBER OF WONTFIX DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE WONTFIX DEFECTS

METRIC 8: MEASURING THE % OF CANNOT REPRODUCE DEFECTS

(NUMBER OF CANNOT REPRODUCE DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE CANNOT

REPRODUCE DEFECTS

METRIC 9: MEASURING THE % OF DEFERRED DEFECTS

(NUMBER OF DEFERRED DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE DEFERRED DEFECTS

GROUP C: METRICS USED TO MEASURE DEFECTS SEVERITY

METRIC 11: MEASURING THE % OF BLOCKER DEFECTS

(BLOCKER DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE BLOCKERDEFECTS

METRIC 12: MEASURING THE % OF CRITICAL DEFECTS

(CRITICAL DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE CRITICALDEFECTS

METRIC 13: MEASURING THE % OF MAJOR DEFECTS

(MAJOR DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE MAJORDEFECTS

GROUP D: METRICS USED FOR TIME MEASUREMENT

METRIC 14: AVERAGE TIME FOR DEVELOPMENT TO FIX DEFECTS

(TOTAL TIME USED FOR BUG FIXES) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE)

METRIC 15: NUMBER OF TEST EXECUTION FOR A SPECIFIC TIME PERIOD

(NUMBER OF TEST RUN) / (TOTAL TIME)

FOR ADDITIONAL KB’S PLEASE VISIT MY BLOG

WWW.MACHTESTED.COM