How much testing is enough for software that can condemn a man to death? Traceability in an Agile criminal justice context

1

Traceability in an Agile Federal Agency Context

Craeg Strong CTO, Ariel Partners

October 21, 2014 Washington, DC

How Much Tes,ng Is Enough… When it Comes to Ma8ers of Life and Death?

2

So&ware Development since 1988 Large Commercial & Government Projects Turned Around Projects With Agile Apache Ant Open-‐Source Contributor New York & Washington DC Area

CTO, Ariel Partners CSM, CSP, CSD, CSPO, PSM,

PMI-‐ACP, PMP

www.arielpartners.com [email protected]

@ckstrong1

Craeg Strong

© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394

3

Agenda

1.  What Requirements Traceability Is 2.  Why & When It Is Important 3.  Some Typical Agile PracZces 4.  Why They Are Insufficient 5.  AlternaZve Approach: BDD/ATDD With SpecFlow 6.  How/When/Who/Why 7.  Solving The Problem With Agility 8.  How To Get There: TransiZonal Strategies


4

Context

Examples Use Microsoft technology Equivalents Exist in Other Language Ecosystems (Java, LAMP)

q  Air traffic control q  Loss reserve calculation for long-tail insurance q  Threat evaluation for detainee at border crossing q  Nuclear power plant control q  Digitized forensic evidence match at murder trial q  Trading risk circuit breaker for high frequency or options

trading q  Resource allocation for disaster relief

Heavily Regulated Industries Mission-‐CriZcal Systems Life-‐CriZcal Systems Financial Systems Legacy Systems Without Automated Tests


5

What Is Requirements Traceability?

•  Inventory of all requirements •  …traced to features implemented in the software, •  …traced to test cases that cover the functionality •  …with indicator for most recent pass/fail test status

Typical Artifact: Requirements Traceability Matrix (RTM)

System shall…

Module XYZ

Test Case 123


6

Why Is Requirements Traceability Important?

Ø Thousands of complex features Ø …developed over many years Ø …by people who are no longer on the

program Ø …few automated tests Ø …a brittle code base

How can we know we have tested everything?


7

Agile Approach: Automated Unit Tests

q  One test generally tests one class q  Written by developers q  Intended to be run after every change q  “Classic” unit tests: code only q  xUnit extended family

q Database testing q Graphical user interface (GUI)

testing q Smoke testing q Performance testing


8

Measuring Unit Test Coverage

q Measures how much of the codebase is covered by at least one unit test

q Typical target number: 80% © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394

9

LimitaZons of Unit TesZng Difficulty Creating Unit Tests q  Legacy projects may not have good automated test coverage q  Code may not be built for testability Only A Partial Solution q  Non developers cannot understand unit tests q  Coverage of code does not guarantee a feature works as needed

Build The Thing Right Build The Right Thing

GUI

UZliZes

Security / Logging Service

Business Logic

GUI Module Database Server

Persistence

Security / Logging

UZliZes

Service Business Logic

Feature


10

What about Test Driven Development (TDD)?

Test Driven Development q  A Design technique q  Tests are written first in order to drive the design q  Additional tests may be added later

Why Does It Work? q  There are generally multiple valid ways to design a piece of code q  Writing a test first tends to produce more testable, flexible and

modular design

More Typical Alternative q  Tests written during/after code q  May require more refactoring to achieve desired results TDD: Good Practice; But Does Not Solve The Problem


11

SoluZon: Behavior-‐Driven Development (BDD)

q Different Process q Different Participants q Different Focus: Build The Right Thing

User Story #53

Acceptance Criterion 1



Automated Acceptance Test #53.1

Given a claim has been reported… When the claim is entered… Then the system should return…

Also Known as Acceptance Test Driven Development (ATDD)


12

Feature File BDD With SpecFlow

1. Write Feature

Step DefiniZons 2. Generate Template

3. Fill In Code


13

BDD With SpecFlow: Roles and Tools

Developer

Tester

Analyst

Product Owner

Syntax HighlighZng Support Gherkin Plugin

Syntax HighlighZng Support Gherkin Plugin

PicklesDoc

Document GeneraZon

CodedUIGUI-‐Level AutomaZon

Test AutomaZon Base Fwk

(or Selenium)

(or MSTest)

Three Amigos

Project Portal/Wiki

Given… When…

Then… Given… When… Then…

Whiteboard


14

BDD With SpecFlow: Process 3. Tester Completes

“Happy Path” Scenario

Tester

Given…

When…

Then…

5. Developer Implements Code And Test Steps

Developer SpecFlow Pair

2. Create Dra& SpecificaZon

During MeeZng

User Story 240 -‐ Acceptance Criteria 1) XYZ Field should be read-‐only 2) Total should match 999.99

Scenario: Ensure XYZ Field is

read-‐only when…

Scenario: Ensure Total is…

Analyst

1. Three Amigos Discuss User Story or Bug

Developer Tester Analyst

PO @ignore

Scenario XYZ

Given…

When…

Then…

4. Developer Adds Dra& SpecificaZon

To TFS

Developer

Big? Risky?

Small?

Fix Now

Technical

Product Backlog

User Story 999 Cleanup XYZ….

8. Rework, Refactoring, or

Cleanup IdenZfied?

7. Review Completed

Test

Developer Tester Analyst

6. Tester, Analyst Add

Addl Scenarios

Given…

When…

Then…

Given…

When…

Then…

Analyst Tester


15

GeneraZng DocumentaZon From SpecificaZons

q Pickles Report Generator q Features organized In tree structure q More sophisticated commercial options available

(SpecFlow+) © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394

16

Organizing Automated Tests: By Type

q Named after class being tested

q Test project structure mirrors source code product structure

q Named after feature q Test project structure

mirrors business domain

Unit Tests Feature Tests


17

Organizing Automated Tests: Filter By Category

Category ExecuAon Speed Dependencies ExecuAon Context

@CI Very Fast None: Code Only Developer, before every check-‐in

@DB Slower Database structure, may use pre-‐set test data

CI Build

@smoke Slowest Full Running System Nightly Build

@CI Scenario: Ensure abc…

@DB

Scenario: Ensure xyz…

@smoke

Scenario: Ensure 123…

q  Each individual test case is categorized according to its dependencies

q  “Run All Tests” automatically runs the correct set of tests depending on execution context


18

Organizing Automated Tests: Test Frameworks

SpecFlow tests can be wri8en at different levels – even within the same feature file

Feature File

Unit!

!

CodedUI CodedUIStep DefiniZons – ImplementaZon AlternaZves

Gherkin


19

Organizing Automated Tests: Examples

Acceptance Criteria 1.  Confirm that a trade that would trigger margin call will require

secondary authorization protocol 2.  Confirm that a trader is not allowed to enter a quantity of

more than 50,000 shares for an options trade 3.  Confirm that an out of the money trade will be flagged

User Story #13628 Enforce Trade Limits For Option Trades

Possible Strategy Why?

CodedUI Test using Moq Specific GUI is important and needs to be tested

NUnit test, no DB •  CalculaZon is most important asset to test, GUI may change

•  Legacy system is being rewripen and replacement system must support same logic

NUnit Test with DB •  Significant base data setup required, mocking is not pracZcal

•  Significant logic to be tested in the persistence layer

CodedUI Test with DB Full-‐stack smoke test is desired (requires more maintenance; there should be relaZvely few of these)


20

Extending BDD For Traceability q Epics Linked to User Stories in TFS q SpecFlow Tests Map To User Stories

User Story 1098 Feature: 1098 Scenario: XYZ Scenario: ABC

Epic 1608 User Story 2689

User Story 3104

User Story 7510 Epic 2387

User Story 7513

Feature: 2689 Scenario: XYZ Scenario: ABC




Pros 1.  Simple 2.  Easy to track

Cons 1.  User story abstraction does not really

match feature 2.  Requires maintenance


21

Challenges for Maintaining Traceability

Issues 1.  A user story may describe only part of a true business

feature 2.  Later user stories may modify, contradict, or replace earlier

user stories

Mitigation: Let the tests help 1.  A failing SpecFlow test may indicate an older specification

needs maintenance 2.  If the earlier SpecFlow test does not fail, consider whether it

is really important to update an earlier user story with new information

User Stories are designed to be ephemeral – only ac,ve within their Sprint or Release


22

The Agile RTM

q  All system functions expressed as user stories q  All user stories have acceptance criteria q  Each criterion translated to an automated test using structured

English (Gherkin) q  Customized report matches epics and user stories to automated

acceptance tests q  Test fails unless software is implemented correctly

Links to Theme Record in TFS

Links to Epic Record in TFS

Links to User Story in TFS

Links to Feature DocumentaZon

Generated via Pickles


23

GeneraZng the Agile RTM

TFS Build

PowerShell Script

TFS Themes, Epics, User Stories

Console Runner Results of Automated Tests

PicklesDoc

Document GeneraZon SpecificaZons

Agile RTM Project Portal / Wiki Combine & Transform XML Files

Publish


24

Building A Test AutomaZon Capability BUILD TESTING CAPABILITY

q  Technical Frameworks

q  NUnit q  SpecFlow q  Moq

q  Reporting Capability q  Test Results q  Code Coverage q  Functional Coverage

q  Encouraging Adoption q  Examples Everywhere q  Mentoring q  Pair Programming

MAKE THE SYSTEM TESTABLE

q  Upgrade dot net framework

q  Drop support for old

operating systems

q  Upgrade third party libraries

q  Refactor code to isolate

dependencies

q  Simplify project structure

q  Different components may

need different strategies

Both Pieces Are Required For Success © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394

25

TransiZonal Strategy: StarZng with Manual Test Cases

Epic

Manual TestCase

Manual TestCase

Manual TestCase User

Story

Manual TestCase

Legacy Req

Legacy Req

q  Import legacy requirements into TFS

q Convert high-level requirements into epics

q Capture manual test cases in MTM

q  Link MTM test cases to their requirements and/or epics

q Capture all new work using user stories

q  Link legacy system requirements to epics alongside new user stories

Epic

Legacy Req

Legacy Req

Gradually migrate legacy “Shall” requirements to user

stories and acceptance criteria

Plan


26

ConverZng From Requirements to User Stories

Why do this? q System has long projected lifespan q Planned re-‐plarorm, migraZon, upgrade, or significant redevelopment

User Story

Legacy Req 1 to 1

Legacy Req Acceptance Criterion 1 to 1

User Story

Legacy Req 1 to many

User Story

User Story

Legacy Req

User Story many to 1

Legacy Req

Legacy Req


27

Legacy System Replacement

Benefits of SpecFlow for re-‐development q SpecificaZons confirm new system meets requirements q SpecificaZons represent up to half of overall effort q SpecificaZons provide documentaZon for the new system q Highlights areas where new system differs from old system q Significantly reduces risk

Legacy System Replacement System

Significant Reuse

Gherkin

SpecificaZons

Step DefiniZons


28

Summary

1.  Traceability is important for mission criZcal and life criZcal so&ware

2.  Automated test frameworks can provide the assurance we need ü Did we build the right thing ü Did we build it right ü Are all the required funcZons working correctly

3.  SpecFlow has been proven effecZve


29

QuesZons?


We are available for consulZng or Agile coaching. Feel free to contact us!

Ariel Partners (646) 467-‐7394 [email protected] www.arielpartners.com @arielpartners

Thank You!

Technology

How much testing is enough for software that can condemn a man to death? Traceability in an Agile criminal justice context