Better by Design - bcs.org · system – Originally for Fire Brigade ... Mode-ARGH! Desk ... Source: Tom Gilb, Principles of Software Engineering Management

1

www.nandbtesting.com © Newell & Budge, 2004

Better by Design

www.newellandbudge.com

Graham FreeburnNewell & Budge Testing Solutions

[email protected]

Tel: +44(0)1698 464 239Fax: +44(0)1698 464 250


Main Course

• Learning from Engineering• Working with developers

Desert

The Menu

• Summary• Some caveats

Starter

• Bad Design, No Design…• A light bulb..

Aperitif • Introduction

2


A Light Bulb..

• Testers only find failure..• Testing is (still) often too little, too late• How real are requirements?• Tester’s aren’t the only ones with

problems• A tale of two alarm clocks….

– Is good design (with tester involvement) a way forward?


Bad design.. No Design..

• Why people don’t design• Some examples

– Software– Tale of Two Alarm Clocks

“You can use an eraser on the drafting table or a sledge hammer on the construction site.”

Frank Lloyd Wright

3


Why people don’t design

• Old Days…– Get the &*!%*? Coded!– Fixed Architectures– Try it and see

• New World– Time Pressure– Development Tools– Requirements Code– Re-use– Packages / COTS


Software

So many… so little time….

4


• August 2004– Generic command and control

system– Originally for Fire Brigade– Modified for use by Coastguard– Crashed and reverted to orignal

design– “Turn left into High Street..”

• In a helicopter!


Software

5


Last month!

• UK Government Department

• Desktop upgrade• Caused 80,000 machines

(80% of total) to fail• No e-mail so officials had

to communicate by handwritten faxes


Tale of Two Alarm Clocks – the sublime

AlarmOn/Off

Hour

Time set

Alarm set

Drowse

Minute

Source: The Power of Modern Testing,Les Hatton, Scottish Testing Group (STG), Oct 2003

6


Tale of Two Alarm Clocks – the ridiculous

RandomNumbers

MoreButtons

Mode- ARGH!

Desk



Alarm Clock 2 – some examples

• Time setting– Press MODE button THREE times. Then press SELECT and SET

repeatedly• Alarm Setting

– Press MODE button TWO times. Then follow time setting.• Hourly chime on/off

– Press SELECT and MODE button simultaneously• Alarm on

– Press DATE and SELECT button simultaneously• Stop watch/lap counter

– Press MODE button ONCE and then beg for mercy as clock gibbers with entertaining series of random beeps

• Turning alarm off– Hurl out of window after standing at clock screaming.


7


Working with Developers

• Make the case for design• Get involved

– Designing for Testability– Testing the Design

• UML • Design Mapping• Quality Planning

– Early Test Design• Exploiting Bad Design


Working with developers - designing for Testability

As applications become more complex, spreading over multi-tiers, across varying networks, and including different client configurations, application architecture testing will become increasingly mandatory

» GIGA GROUP

Transistors (1960s)

Design for Testabilitystandards (late 80s)

IntegratedCircuits (late 60s)

Large scaleICs (early 70s)

Assembly language (1960s)

UML (late 90s)

3GLs, 4GLs ( 70s & 80s )

Object Languages (early 90s)

Software

Design for Testabilitystandards (00s)

Source: Achieving eBusiness Quality by Design,Sam Guckenhiemer, Rational, Software Test Automation, 2001

8


Working with developers - testing the design

UML

Use Case Diagrams

Class Diagrams

Packages & Object Diagrams

Component & Deployment Diagrams

Collaboration Diagrams

Statechart Diagrams Activity Diagrams

Sequence Diagrams



Design Mapping - “4C” compliance

Actor HierarchyDiagram

Sequence Diagram

Class/Type Diagram

Activity Diagram State Diagram

Use Case TextualDescription

Use Case Diagram Actors are identified by the same name

Actors are identified by the same name




States should correspond State events should correspond with Sequence messages

1. Use Case steps should correspond with Sequence Diagram message2. Sequence begins in a state that satisfies the Use Case pre conditions3. Sequence ends in a state that satisfies the Use Case post conditions

Use Case Name is identicalDiagram Associations are described in DescriptionExtension PointscorrespondIncludes correspond

Use Cases correspond with Activities

Objects are identified by the same name

State trigger events areClass methods

Objects should correspond

Sequence messages are Class object methods


1. States are identified by the same name2. Events, guard conditions and activities are identified the same

Actor HierarchyDiagram

Sequence Diagram

Class/Type Diagram

Activity Diagram State Diagram

Use Case TextualDescription

Use Case Diagram Actors are identified by the same name





States should correspond State events should correspond with Sequence messages

1. Use Case steps should correspond with Sequence Diagram message2. Sequence begins in a state that satisfies the Use Case pre conditions3. Sequence ends in a state that satisfies the Use Case post conditions

Use Case Name is identicalDiagram Associations are described in DescriptionExtension PointscorrespondIncludes correspond

Use Cases correspond with Activities

Objects are identified by the same name

State trigger events areClass methods

Objects should correspond

Sequence messages are Class object methods


1. States are identified by the same name2. Events, guard conditions and activities are identified the same

Correctness

Consistency Clarity

Completeness

Source: The Road to UML is paved with Good Intentions,Dion Johnson, StarEast 2002

9



Quality Planning

Opt A Opt B Opt C Opt D Opt E etc Attribute Impact Sum

Attribute 1

Attribute 2

Attribute 3, etc

60%+20%c=0.6

40%+ 10%c=0.2

Option Impact Sum

Costs

Benefit to cost ratio

Source: Tom Gilb, Principles of Software Engineering ManagementAddison Wesley, 1988

Attributes Options / Solutions

75%+ 10%c=0.8


Working with developers - exploiting bad design

• Exploratory / Rapid Testing– Cem Kaner, James Bach– Simultaneous observation design

and execution of tests– Heuristics, Exploratory branches

• “Brain on, eyes open, test !”– 17 User Interface Attacks– 6 system Interface Attacks– 2 Software/OS Interface

Attack approaches

10


Learning from Engineering

• What is good design?– Boeing 777

• Paperless design allowed simulated assembly of 777• Worked so well that only a nose mock-up was built

before assembly first flight vehicle

• Building Bridges• Design of experiments (DOE)• Information Feedback• The Importance of Context


Learning from Engineering- what is good design?

• Useful– It works well and functions as promised. It does what

it is expected to and satisfies a minimum or appropriate level of performance

• Usable– It has appropriate ergonomics and user interface,

considering how, where, how often and who will be using it.

• Desirable• Producible• Profitable• Differentiated

http://www.betterproductdesign.net/index.htm

11


A bridge to nowhere..

Spiral ramp of the larger 'bridge to nowhere', with a stub end where it should have been extended to the other (there were two!).

This end of the bridge has no guardrails at all, and the base of the spiral is protected by a high spiked fence.

The other end does have guardrails, but finishes abruptly with a 60 foot drop.

- M8 Motorway, Charing Cross district of Glasgow


Learning from Engineering- design of experiments (DOE)

• DOE have been applied in Quality control in many engineering fields for several decades

• Aim : to maximise the amount of information gained in an experiment by optimising the combinations of independent variables

• In Software Testing Terms – Combination or Combinatorial Testing– E.g. “All Pairs”

Source: An Investigation of the Applicability of the Design of Experiments to Software Testing, D. Richard Kuhn & Michael J. Reilly, Proceedings, 27th NASA/IEEE Software Engineering Workshop, December 2002

Table 1.Number and percent of faults triggered by n-way Conditions

Conditions Browser Server Modules(values on n) (194 bugs) (171 bugs)

(percent) (cumulative (percent) (cumulativepercent) percent)

1 28.6 28.6 41.7 41.7

2 47.5 76.1 28.6 70.3

3 18.9 95.0 19.0 89.3

4 2.2 97.2 7.1 96.4

5 2.2 99.4 0.0 96.4

6 0.6 100.0 3.6 100.0

12


Learning from Engineering - Information Feedback (design to operation)

Source: Mapping Function to Failure Mode during Component Development, Irem Y Tmer and Robert B. StoneResearch in Engineering Design 14 (2003)


Learning from Engineering- context is everything

Source: James Bach, www.satisfice.com, © Satisfice Inc. 2003

Test TeamExpertiseLoadingCohesionMotivationLeadershipProject Integration

Test LabTest PlatformsTest ToolsTest LibraryProblem Tracking SystemOffice Facilities

Product MissionStakeholdersQuality CriteriaReference Material

RequirementsProductProject LifecycleProject ManagementConfiguration ManagementDefect PreventionDevelopment Team

Development

TestProcess

StrategyLogisticsWork-products

MissionFind Important ProblemsAssess QualityCertify to StandardFulfill Process MandatesSatisfy StakeholdersAssure Accountability

Advise about QAAdvise about TestingAdvise about QualityMaximize EfficiencyMinimize CostMinimize Time

13


Some caveats• The need for good

requirements analysis isn’t diminished

• Economics– Practicality of design

proving– Cost of Quality C

osts

Quality Level

Failure Costs

Total Quality Costs

Costs forprevention

and detection

Minimum

Source: Juran’s Quality Control Handbook, 1988ISBN 0-070-33176-6


Summary

• Get involved in design – For everyone’s benefit

• Consider using– Design Mapping– Use Cases, Scenario testing…

• They are powerful weapons in the war against faults

• Learn to exploit bad design– Rapid Testing / Exploratory Testing

• Get in early!

14


Resources

The Rational Unified Process – An Introduction, 2nd EditionPhillippe Kruchten, ISBN 0-201-70710-1

Design Secrets : ProductsIndustrial Designers Society of America (IDSA)ISBN 1-56496-476-0

Total Design, Stuart Pugh,ISBN No 0-201-41639-5

Documents

Better by Design - bcs.org · system – Originally for Fire Brigade ... Mode-ARGH! Desk ... Source: Tom Gilb, Principles of Software Engineering Management