Quality Management and Testing Quality throughout the Lifecycle
Testing as a Quality Practice Types of Testing Agile Testing
Practices
Slide 2
2 Review: Reliability Terminology Failure Incorrect or
unexpected output Symptom of a fault Fault Invalid execution state
Symptom of an error May or may not produce a failure Error Defect
or anomaly in source code Commonly referred to as a bug May or may
not produce a fault Defects may be injected at any time in the
lifecycle Recall Watts Humphrey (father of PSP): (paraphrase) A
defect is anything that necessitates a change in the code
Slide 3
3 Review: The Cost of Defects Cost Development Phases Diagram
below is the assumed curve (Capers Jones, 2000) Software problems
are 100 to 1000 times more costly to find and repair after
deployment Diagram below is the assumed curve (Capers Jones, 2000)
Software problems are 100 to 1000 times more costly to find and
repair after deployment Cost to Repair Software Costs to Re-test,
Re-deploy, Re-* Cost of Lost Opportunities Cost of Lost Customers
Cost to Repair Software Costs to Re-test, Re-deploy, Re-* Cost of
Lost Opportunities Cost of Lost Customers Courtesy IBM Academic
Initiative
Slide 4
4 Software Qualities Correctness Reliability Robustness
Performance User friendliness Verifiability Maintainability
Repairability Safety Evolvability Reusability Portability
Understandability Interoperability Productivity Size Timeliness
Visibility Notes courtesy Andre Van Der Hoek UC Irvine Copyright
2001 Quality is concerned about more than finding and fixing
traditional bugs testing is in essence a preventive process whereas
you build quality in; you create it deals with measurable and
verifiable ilities of software
Slide 5
5 Verification and Validation Verification Ensure software
meets specifications Internal consistency Are we building the
product right? Validation Ensure software meets customers intent
External consistency Are we building the right product? Formal
Technical Reviews Test Planning & Review Measurement Analysis
& Reporting Process Definition & Standards V & V is a
whole life-cycle process
Slide 6
6 Views of QA Design, in formal notation Executable machine
code Execution on verified hardware Code, in verifiable language
Complete formal specification of problem to be solved
Correctness-preserving transformation Notes courtesy Andre Van Der
Hoek UC Irvine Copyright 2001 Design, in mixed notation Pentium
machine code Execution on commercial hardware Code, in C++, Java,
Ada, Mixture of formal and informal specifications Manual
transformation Compilation by commercial compiler Commercial
firmware An Idealistic View of QA A Realistic View of QA
Slide 7
7 Quality Practices in the Lifecycle Quality Practices Pervade
Lifecycle Activities: Requirements 8 quality measures (remember
these? correct, unambiguous, complete, consistent, prioritized,
verifiable, modifiable, traceable) Design Does your design satisfy
all the requirements? What constitutes a quality design anyway?
Coding What practices can developers do to improve quality? Test
Seen as the principle quality practice - but does it create
quality? Deploy The ability to get the software into the hands of
users Maintain What happens to quality as the system evolves over
time?
Slide 8
8 Complicating the Process Four inherent complexities to
building quality software: 1.Requirements 2.Inherent problem
complexity 3.Accountability 4.Inertia Note that only #2 is a
function of the domain The others are principally
social/communicative
Slide 9
9 First Complication - Requirements No matter how sophisticated
the QA process, the problem of creating the initial specification
remains Goes all the way back to requirements! Real needs Actual
Specification Correct Specification Notes courtesy Andre Van Der
Hoek UC Irvine Copyright 2001
Slide 10
10 Second Complication - Complexity Complex data communications
Electronic fund transfer Distributed processing Web search engine
Stringent performance objectives Air traffic control system Complex
processing Medical diagnosis system Sometimes, the software system
is extremely complicated making it tremendously difficult to
perform QA Notes courtesy Andre Van Der Hoek UC Irvine Copyright
2001
Slide 11
11 Third Complication - Accountability It is difficult to
divide the particular responsibilities involved when performing
quality assurance Project Management Development Group Quality
Assurance Group Notes courtesy Andre Van Der Hoek UC Irvine
Copyright 2001
Slide 12
12 Fourth Complication - Inertia Quality assurance lays out the
rules You will check in your code every day You will comment your
code You will Quality assurance also uncovers the faults Taps
developers on their shoulders Creates image of competition Quality
assurance is viewed as cumbersome Just let me code You cant tell me
what to do Quality assurance has a negative connotation Notes
courtesy Andre Van Der Hoek UC Irvine Copyright 2001
Slide 13
13 Quality Practices in Testing FURPS Model - sets expectations
for testing Actually a general quality model, this figure shows how
it fits for testing Reliability Test that the application behaves
consistently and predictably. Performance Test online response
under average and peak loading Functionality Test the accurate
workings of each usage scenario Usability Test application from the
perspective of convenience to end-user. Supportability Test the
ability to maintain and support application under production
use
Acceptance Tests Purpose: Determine if the product is ready for
final release Different levels of acceptance based on the user
population Maps to the types of acceptance tests (next slide) May
be part of a contractual basis If software development under
contract specifications, then passing acceptance tests may be
considered a final milestone on your deliverable. Not dedicated
solely to finding implementation defects Also used to identify
requirements, deployment, maintenance defects, etc.
Slide 16
Acceptance tests Benchmark Test: real-world conditions Actual
or simulated real user base Commonly used for systems w/ special
requirements Evaluate multiple offerings against the benchmark The
benchmark represents ground truth Pilot test: install on
experimental basis Uses actual end users Focuses on mainstream
requirements Specialized training and education of the user base
not required Alpha test: in-house pilot test Or sometimes with most
trusted partners Beta test: customer pilot Parallel testing: new
system operates in parallel with old Useful for side-by-side
comparison Release Candidate: early access in the community
Community approval that the product is ready for final release
Slide 17
Installation / Regression Testing Installation Testing Does the
system work in the field as it does in the lab? How easy or hard is
it to deploy and maintain? Are migration procedures accurate? Does
the documentation match? This is a big deal nowadays! Regression
Test Once system has passed a system test, future system tests must
guarantee new functionality doesnt break working functionality When
you run the Nth test, you have to run the N-1 tests before it Fits
particularly well with iterative and incremental processes
Slide 18
Functional/Automated Tests Functional: Test of the system as a
whole Functionally-oriented (black-box) Can include non-functional
aspects Typically though, these are done as part of stress testing
Not concerned with location of defects or interface problems These
are tasks for integration testing Defect expensive to fix when
found at this step Automated Testing and Test Tools Simulations
Record-and-play tools Workflow support for test scripts Tracking
and reporting results Nightly build and test tools Example:
http://public.kitware.com/dashboard.php
Slide 19
Integration testing Tests complete systems or subsystems
composed of integrated components Integration testing should be
black-box testing with tests derived from the specification Main
difficulty is localising errors Incremental integration testing
reduces this problem Interface Testing Takes place when modules or
sub-systems are integrated to create larger systems Objectives are
to detect faults due to interface errors or invalid assumptions
about interfaces Particularly important for object-oriented
development as objects are defined by their interfaces
Slide 20
Interfaces types Procedural interfaces Sub-system encapsulates
set of procedures called by other sub-systems Message passing
interfaces Sub-systems request services from other sub-systems
Parameter interfaces Data passed from one procedure to another
Shared memory interfaces Block of memory is shared between
procedures Types of Interface Errors Interface misuse: A calling
component calls another component and makes an error in its use of
its interface e.g. parameters in the wrong order Interface
misunderstanding: A calling component embeds assumptions about the
behaviour of the called component which are incorrect Timing
errors: The called and the calling component operate at different
speeds and out-of-date information is accessed
Slide 21
Stress testing Exercises the system beyond its maximum design
load. Stressing the system often causes defects to come to light
Stressing the system test failure behaviour. Systems should not
fail catastrophically. Stress testing checks for unacceptable loss
of service or data Particularly relevant to distributed systems
which can exhibit severe degradation as a network becomes
overloaded
Slide 22
The Testing Process Test data: inputs devised to test the
system Test cases: Test Data plus the predicted outputs from these
inputs if the system operates according to its specification Test
suite: A collection of Test Cases organized around some principle
or objective.
Slide 23
Test documentation Test plan describes system and plan for
exercising all functions and characteristics Test specification and
evaluation details each test and defines criteria for evaluating
each feature Test description test data and procedures for each
test Test script The actual test procedures (process description)
Test analysis report results of each test
Slide 24
Example Test Script
Slide 25
Example Test Script (cont.)
Slide 26
Agile Practices Change Roles Traditional Testing role The Last
Defender of quality Enforce a strict change management policy w/
rigid entry/exit criteria Heavy documentation & upfront
planning ( but a dynamic project plan?! ) Agile Testing role A team
member primarily focused on quality I was Loved and Appreciated
overheard on a testers 1 st agile project Integrate quality
practices into development process Test provides value to help
reduce risk Agile Principles: Quality is everyones responsibility
Developers write tests too (test-driven) Testers work in same
bullpen as developers No separate, closed-door testing group Track
quality with rest of development process Coverage, metrics, etc. on
each build
Slide 27
Agile Testing: Continuous Testing Agile likes transparency,
adaptability to change Dashboards used to show constant quality
progress Include results of unit, system, load, etc. tests As well
as quality measures from your metrics Compatible with other
transparency measures like burndowns
Slide 28
28 Agile Testing Top 10 Best Practices 1.Write tests first:
Code is not delivered to build unless it passes tests 2.Write
acceptance tests: Every story defines its acceptance criteria
3.Embrace change: Agile development means frequent change 4.Reduce
Test Documentation: R ecord the test and track results, thats it!
Use a Wiki to capture essence, not details (must must be
repeatable) 5.There is no handoff activity: Testing integrated in
the dev process 6.There is separate no integration testing
activity: Integration is continuous, supported with (automated)
regression testing 7.Automate everything you can! Make test scripts
the documentation 8.Test early (or Test First!) Add tests to
development build process before feature is completed Test features
before they are complete Test features before they are accessible
from its external interface (GUI) 9.Testing is not a phase, its a
way of life. Agile teams are test infected. How will we test it? is
as important as How will we build it? 10.Testing and development
are not separate teams
Slide 29
Quality and Testing Conclusions Dr. Garys Things to Remember:
1.Tested software != Quality software 2.Quality is injected, not
something defined in absentia 3.But sure, testing is important, and
important to do well Have clear testing objectives Dont treat it as
a last step checklist 4.Respect your testers you are on the same
team Dont hate the playa, hate the game--Ice T 5.Include your
testers early in the process 6.Start testing early in the process
7.Verification is important, but Validation is everything 29
Program testing can be used to show the presence of bugs, but never
to show their absence! --E.W. Dijkstra, 1972