Software Development Metrics You Can Count On

Software Development Metrics You

Can Count On

Arthur Hicken

Evangelist, Parasoft

21 Apr 2015 International Conference on Software Quality - ICSQ 2015

Takeaways

Basic understanding of available SDLC metrics

Learn which metrics are dangerous and how to recognize them

Learn how to mine the data effectively.

9-11 Mar 2015 International Conference on Software Quality - ICSQ 2015 2

What Are Metrics

“A software metric is a quantitative measure of a

degree to which a software system or process

possesses some property. Since quantitative

measurements are essential in all sciences, there

is a continuous effort by computer science

practitioners and theoreticians to bring similar

approaches to software development.”

- Software metric - Wikipedia


Why Should I care about

Metrics?

What is my quality

What is my risk

Where is my risk

Is it good enough

Is it done


The Problem

“The software industry lacks standard metric and

measurement practices. Almost every software

metric has multiple definitions and ambiguous

counting rules… The result of metrics problems is

a lack of solid empirical data on software costs,

effort, schedules, quality, and other tangible

matters.”

- Strengths and Weaknesses of Software Metrics, 2006


Common Metrics

• Balanced scorecard

• Bugs per line of code

• Code coverage

• Cohesion

• Comment density

• Connascent software

components

• Coupling

• Cyclomatic complexity

(McCabe's complexity)

• DSQI (design structure

quality index)

• Halstead Complexity

• Instruction path length

• Maintainability index

• Number of classes and

interfaces

• Number of lines of code

• Program execution time

• Program load time

• Program size (binary)

• Function Points and

Automated Function Points


Metrics with a Capital M

Some metrics have taken on a life of their own

• Complexity

• Cohesion

• Coupling

• Maintainability

• KLOC

There are no silver bullets, no single metric that

defines “good” vs. “bad” software


Averages Aren’t Normal

• Averages help us

understand a trend

• More nuanced

analysis is more

productive

• Frequently the

average represents no

real data point

• Real risks hide behind

averages – IE slow

components9-11 Mar 2015 International Conference on Software Quality - ICSQ 2015 8

MEASURING


How Do You Measure Quality

• No – there is no way to quantify quality

• Yes – measurement is fundamental to quality

Can it be done?

• Complexity isn’t complicated

• Which developer is most productive?

No single metric


What Can You Measure

• Code churn

• Field bugs

• Static analysis findings

• Test failures

• Coverage

• Performance

• Counts (lines, files, …)

• Bug Arrival Rates


Varied Tool Chain

• Different teams use

different tools

• Difficult to unify

• Getting the most from

open-source

• How to get uniform

outcome


Getting Consistent Results

Map the tools

• Tricky but necessary

Get the results in one place

• data driven

Proper reporting

• Combined results

• By tool

Back-fill capabilities


Some Types of Graphics


• The instantaneous value of something

Gauges

• An incrementing and decrementing value

Counters

• Display multivariate data

Radar / Spider

• The statistical distribution of values in a stream of data

Histograms

• Show how one variable is correlated with another

Scatter plot

Sample Report


Checking Your Work

• Did you get the right numbers?

– Are they going in the right direction?

• Are you measuring enough?

– Are unexpected things happening?

• Are the measurements automatic?

– Manual estimates are inconsistent

– Multiple layers of manual collection yield compound

rounding errors


Getting Value

• Consistency is key

• Automation

• Tweak based on feedback loop

– Perform post-mortem on problems

– Verify that measurements are meaningful

• Doing the same thing will yield the same result


MINING


The Data Era


Keys to Success


Central Quality Hub

Control and Visibility

Real-time feedback & compliance

Aggregation into

actionable information

Bridge technical

finding with business impact

Harnessing “Big” Data

• Aggregate data

• Correlate data

• Mine data

• Create

– Reports

– Dashboards

– Tasks

– Alerts

• Continuous testing/delivery/release


Why Analyze

What’s Working / Not Working

What’s important

What’s critical

What’s more/less important than I thought


TIPS & TRAPS


Names Can Be Misleading

• Complexity isn’t complex – ask a developer

• Cyclomatic is roughly 1 + (number of loops) +

(number of if statements)

– Clean but long if-then-else gets flagged

– Short complicated code is “OK”

• a=b+c; can be unbelievably complex using function pointers

• In other words

– Branches alone doesn’t make the code complicated

as regards maintainability or probability of bugs

– Good indicator of how many tests are needed


Simple


Complex


KLOC

• Ups and downs

• How to count

– White space

– Comments

– Multiple statements on one line

– CONSISTENCY

• Meaningful?

• IEEE study – correlation to bugs

– A Critique of Software Defect Prediction Models


Problem: Who’s the Best

• Lines of code written

– Is it difficult

– Is it good

– Is it maintainable

– Is less/more better

– Comments?

• Field bugs

• Tasks completed


Analyzing Code Maintainability

Complicated

• Difficult to understand

Complexity

• The interactions between a number of entities

Coupling

• The manner and degree of interdependence between software modules

Cohesion

• The degree to which the elements of a module belong together


Problem: Where’s the Risk

• Which component / code is vulnerable

• Multiple data points

– Static code analysis (CWE)

– Unit test results

– Source control data

– QA and manual testing


Problem: Am I on Track?

• Have to know what “DONE” is first

• Need connection to requirements

• Need connection to source control

• Need connection to development management

data (scrums etc)


Problem: Where Are the Bugs

• No single metric can predict bugs

• Enhanced data is helpful

• Testability itself shows bugs

– Can you use auto test generation?

• Link to field bug-tracking system is valuable

• Correlate to Q&A activities


Coverage Metrics

• Some Types

– Functional

– Statement

– Branch

– Condition

– Modified Condition / Decision Coverage (MCDC)

• Meaning

• Danger of chasing coverage

• Reasonable answer – How Much is Enough?


A Word on Debt

• Debt is a real thing

– We incur debt consciously and accidentally

• Real debt isn’t trivial to measure

– Wrong frameworks, models, algorithms

• Anyone claiming they can fix 10,000 hours of

debt in a day is selling snake oil

• Static analysis alone cannot fully calculate debt


Conclusions

• Avoid simple metrics for decision making

• Be sure the metric means what you think it does

• Be sure that the practice you’re planning will

affect your metric


Q&A

• Web

– http://www.parasoft.com

• Blog

– http://alm.parasoft.com http://codecurmudgeon.com

• Social

– Facebook: https://www.facebook.com/parasoftcorporation

– Twitter: @Parasoft @CodeCurmudgeon

– LinkedIn: http://www.linkedin.com/company/parasoft

– Google+: +Parasoft +ArthurHickenCodeCurmudgeon


Technology

Software Development Metrics You Can Count On