Secure Systems Research Group - FAU Development Process Metrics A look at development process metrics with view to managing the development of secure software

Secure Systems Research Group - FAU

Development Process Metrics

A look at development process metrics with view to managing the development of secure software.

Dr. Michael VanHilst

22 March 2007


Development Process Metrics

A look at development process metrics with view to managing the development

of secure software.

Dr. Michael VanHilst

22 March 2007


ISO-15504 (level 2), CMM (level 4)

Measure how well the process is being followed (using GQM).1. Identify goals (i.e. reliable software products)2. Analyze causal chains in the process3. Identify process steps that contribute to the goals4. Define “measurements” that assess if identified steps are followed.5. Measurement can use indicators (presence of documents) or survey

questions (“How committed was management in the review meeting?”)6. Measure and hold assessment meetings every 2 months

W. S. Humphrey. A Discipline for Software Engineering. Addison-Wesley, January 1995.

Jarvinen, J., Hamann, D., and van Solingen, R., “On integrating assessment and measurement: towards continuous assessment of software engineering processes”, IEEE Software Metrics Symposium, 1999.

Birk, A., Derks, P., Hamann, D., Hirvensalo, J., Oivo, M., Rodenbach, E., van Solingen, R., and Taramaa, J., “Applications of measurement in product-focused process improvement: a comparative industrial case study”, IEEE Software Metrics Symposium, 1998.


Personal Software Process

W. S. Humphrey. A Discipline for Software Engineering. Addison-Wesley, January 1995.

Anne M. Disney and Philip M. Johnson, “Investigating Data Quality Problems in the PSP (Experience Paper)”, ACM SIGSOFT 1998

Philip M. Johnson, Hongbing Kou, Joy Agustin, Christopher Chan, Carleton Moore, Jitender Miglani, Shenyan Zhen, and William E.J. Doane, “Beyond the Personal Software Process: Metrics collection and analysis for the differently disciplined.”, IEEE ICSE 2003

• Requires developers to record lots of notes on how they spend time.

• Developed by Watts Humphrey at SEI, promoted as part of Team Software Process within CMMI.

• Experience shows dedication slacks off over time.

• Studies by Johnson, et al., find self reports inaccurate, easily skewed.


Causal Analysis (CMM level 5)

Dumke, R.R.; Blazey, M.; Hegewald, H.; Reitz, D.; Richter, K.: Causalities in Software Process Measurement and Improvement. International Conference on Software Process and Product Measurement (MENSURA 2006), November, 6-8, 2006

Several other papers on process discovery, inference, or compliance.


Low Level Non-Invasive Metrics

Johnson, P.M., Kou, H., Paulding, M., Zhang, Q., Kagawa, A., and Yamashita, T, Improving software development management through software project telemetry, IEEE Software, July 2005

Andrea Janes, Marco Scotto, Alberto Silliti, and Giancarlo Succi, “A Perspectrive on Non Invasive Software Management”, IEEE Instrumentation and Measurement Technology Conference (IMTC), 2006

Instrument IDE, CM, Office Tools, OS

Reports which app/file in foreground

Used to look for indicators of lowered productivity.

Johnson example looked at contributing factors in build failures.


Siemens Measurement System

• Four measurement categories: Time / Size / Quality / Cost. – Size metric undecided (IFPUG 4.2, COSMIC FFP V2.2)

• Four selected indicators are – Schedule / Budget compliance– Cost of Defect Correction– Feature changes– Defect Detection Profile

• An annual “Measurement Excellence Day” helps to keep participants in the SMS loyal to the program.


Bias in the Data

• Measures conformance to plans/predictions– Time spent– Code written (LOC,FP)– Defects found / fixed

• Analysis always blames the developer– Developer productivity (slow, error prone)– Process conformance (followed process?)

• Won’t drive process change or innovation


Shell Information Technology

• Five metrics are collected: (Tom Dekker)– Project Delivery Rate, – Speed of Delivery, – Defect Density, – Time Schedule, – Effort Cost.

• The target is: 90% of the projects should be within schedule, budget, and within planned scope.

• Revised estimation based on industry benchmarks proved much better than the original values based on Shell’s own history-based data. They were less biased than their own.

• (Less biased, closer to business goals)


Typical Effort vs. Calendar Day

alpha release beta releasestart

develop test … fix


Waste & Delay in Typical Effort

alpha release beta releaserequirements freeze

develop … test … fix

production waste

fault caused

fault foundfault fixedlag


Toyota Product Development System

releasestart

test & develop

Design defects out of the process

A product with many defects at first release will still have many defects at final release


IS Security Design Methods

Richard Baskerville, Information systems security design methods: implications for information systems development, ACM Computing Surveys, 25, 4 (December 1993) Pages: 375 – 414

Chung, L., Nixon, B. Dealing with non-functional requirements: three experimental studies of a process-oriented approach. Proceedings of ICSE’95, pp. 25–37

Eloff, MM, von Solms, SH, ‘Information Security Management : An Approach to combine Process Certification, and product Certification’, Computers and Security, Vol 19, pp 698 – 709

T. Tryfonas, E. Kiountouzis, A. Poulymenakou, Embedding security practices in contemporary information systems development approaches, Information Management & Computer Security, Oct 2001 9,4 183-197

• Much literature on requirements for security

• Very little on the development process


A Toyota Security Method?

• A product with many security flaws at first release will still have many flaws at final release

• We get what we measure. If we don’t measure security we won’t get security. So how do we measure security?

• Is there an idiot-proof or test-first equivalent for driving security flaws out in the development process?

• Suggestion from Nelly: use automated code analysis for security anti-patterns, unsafe library calls, …

Mylopolous has paper on process for transforming insecure code to secure cope.


Documents

Secure Systems Research Group - FAU Development Process Metrics A look at development process metrics with view to managing the development of secure software