View
217
Download
1
Tags:
Embed Size (px)
Citation preview
Ronald Kohlman
THE VALUES OF TESTING TECHNOLOGY
© 2011
2
Contents
Introduction What is the Value of Testing to
Business Value of Testing to Technology
Where does Testing fit in? Further Questions Importance of Testing Enemies of Testing
Questions for the Test Manager What do you bring to the
table? What does the Organisation
bring to the table? Approaches
Purpose of Testing Long Term Goals Test Objectives What are you testing?
Testing Architecture Four Stages Test Phases Test Management Test Planning
Closing
3
Introduction
This Presentation This presentation has the aim to generate discussion and
participation by you, questions are encouraged The ideas presented here are focussed on some of the
attributes from a testing framework
About Presenter – Ronald Kohlman Brings a wealth of many years of experience in IT, Program
Management and Testing from numerous major programs and projects across a number of industry sectors.
Granite Consulting Granite Consulting is a specialist human capital solutions
provider in the areas of technology and business project services. Our experienced team has extensive global experience and we provide innovative solutions that work.
4
Where does Testing fit in? The testing process should begin early in the
systems development life cycle, not just at the traditional testing phases.
Testing activities should be integrated with systems development activities
Where does testing fit within your organisation?
5
Where does Testing fit in?
6
Importance of Testing
A poor testing program can: cause mission failure significantly impact operational performance and reliability require significant rework double or triple field support and maintenance costs
It is a significant project cost Testing activities can be more than half the total program effort –
averages of 25-40% is a common benchmark To be effective, time up front is necessary to plan and organize
it A good testing program forces you to face and deal with
problems as the work is done, and when the cost of rework and fixes are much lower
A good testing program can’t make up for poor software quality, but it does help prevent many ills and will at least let you know you are in trouble early
Testing is insurance
7
What is the Value of Testing to Business
Is it to save money or risk?
Is it a “degree” of comfort?
Is it insurance?
Is it because we are supposed to do it?
Is it to stop “bugs” getting into production?
Where does Testing add value and what is its value?
8
Value of Testing to Technology Is it to prove something to ourselves and others
about the quality of what we have built/delivered?
What do you think is the value for Technology?
Is testing valued at all?
9
Further Questions What is your definition and understanding of Testing?
What are the main testing issues in your organisation?
What is the driver for testing in your organisation or organisations you have worked within?
What do you think is your organisation’s SDLC methodology and where does testing fit in?
To plan for your Testing, what do you think you would need?
10
Enemies of Testing
Vague requirements
Undocumented design
Poor code, or configuration
Compressing the development schedule risks introducing problems
Compressing the test interval risks not finding all critical problems
11
Questions for the Test Manager
Have I defined, and do I understand, my overall strategy for software testing?
Are the test planning requirements clearly defined, consistent, assigned for development, and supportable by the staff responsible for their implementation?
Are the test methods and techniques defined, are they consistent with the strategy, and can they be implemented in the software environment?
Is each test activity traceable to a controlled set of requirements and/or design data?
Are the configuration management control, and quality disciplines adequate to support the test strategies, methods, and techniques, and are they really in place?
Do I know when to quit?
What do you bring to the table? What works for you?
What doesn’t work you?
What is the one thing that you would suggest as “a must
have” in your
tool kit?
Where do you go to source information to help you?
Own knowledge base? Colleagues? Training?
Organisation?
Internet? Books? Papers? Conferences?
What knowledge do you pass on (what do you leave behind)?
How do you approach the work?
What do you look for?
How do you know you have all the information you
need?
What does the Organisation bring to the table?
Their own processes, culture, ideas, practices
and templates
Their own problems, issues, and perceptions
Organisation Maturity Level
Ability to accept your inputs, or not
Inability to clarify scope
Inability to do it the “right way”
Inability to understand testing (“Education”)
Wanting to “cut corners” & “squeeze” time
14
Approaches
Testing can be manually executed, automated, or a combination of both
Can be done with resources who are on-shore, off-shore, or a combination of both
Test every conceivable option possible to be satisfied of complete thoroughness, or run testing as a Risk Assessment – “Risk Based Testing” where you test only the highest risk areas first and subsequent areas as time and resources permits
Test multiple time across “test phases” e.g. Unit, systems, integration, performance, user acceptance, etc. Or, minimise the number of test phases to a lesser degree
15
Purpose of Testing The purpose of testing is to verify technical
performance, operational effectiveness, operational suitability; and it provides essential information in support of decision making by the business.
MISSION FOR TESTING To validate that the technology applications and solutions
delivered meet the businesses requirements and needs.
VISION FOR TESTING Operational excellence dictates the need to complete
more testing that is efficient, cost effective, and targeted; proving the quality of solutions being delivered to Project's customers is superior; thereby reducing the overall risk to the business.
16
Long-Term Goals Transfer knowledge and ownership of deliverable’s
and solution(s) to appropriate business stakeholders. Ensure that the testing methods used can be
leveraged for future use beyond the completion of any program or project.
Ensure that automated testing tools, project documents, and other processes, procedures and systems introduced by testing are retained and useable beyond the implementation of an application.
Continuous improvement of testing effort, processes, and practices.
17
Test Objectives
Testing is focused on exercising the solution to obtain confidence in its readiness for use and to demonstrate that it is working as specified.
Testing is the means by which objective judgments are made regarding the extent to which the system meets, exceeds, or fails to meet stated objectives.
Early, structured reviews are to be used to help prevent problems from ever being produced by work streams. Demonstration, detection, and prevention are important objectives and benefits of testing.
18
What are you testing?
Software? Infrastructure? Data? Interfaces?
Useability? Suitability? Robustness? Performance?
Functional / Non-Functional attributes?
Requirements – Business and Technical?
Processes?
All of the above
19
Testing Architecture
20
Four Stages
Planning Preparation Execution
Reporting
Planning: produces test strategies, approach, objectives, schedules and plans
Preparation: produces detailed test plans and test cases / scripts based on the test objectives and conditions; and establishes the test environment
Execution: the execution of the cases / scripts against the application(s) and therecording of test results
Reporting: the reporting of test progress and results, defects found, metrics, and testassessment reports
21
Test Phases
The test phases to be conducted by any particular Project may vary; but the primary test phases for are generally: unit test, systems test, integration test, user acceptance test.
There are other test phases (e.g. performance, operational readiness, etc.) that may be added to these primary ones, which will be up to you and the project to determine and identify within the Test Strategy.
The next diagram graphically represents these test phases.
22
Test Phases – Who does What?
Performed by: Program/Project Test Teams
Systems, Integration & E2E Testing
Ap
plic
atio
n C
on
fig
ura
tio
n
Sys
tem
s In
teg
rati
on
Sec
uri
ty /
Pen
etra
tio
n
E2E
Fu
nct
ion
al
Inte
gra
tio
n
Dat
a C
on
vers
ion
Pro
du
ctio
n R
ead
y T
o U
se
Performed by: Vendor / Developer
Unit Testing
Systems Testing (Vendor)
Ap
plic
atio
n P
erfo
rman
ce, S
tres
s &
Vo
lum
e
Tec
hn
ical
& F
un
ctio
nal
Co
nfi
gu
rati
on
Ap
plic
atio
n F
un
ctio
nal
In
teg
rati
on
Regression Testing
Performed by: Business
User Acceptance
Performed by:
Operations Team
Pre-Production
(Operational Readiness)
Tests
Bu
sin
ess
Sce
nar
ios
Bu
sin
ess
Pro
cess
Hig
h R
isk
Ttr
ansa
ctio
ns
COTS Solution (including any Customisations)
Project A
Project B
Program X
Program Y
Program / Project n...
Dep
loym
ent
Reg
ress
ion
Dep
loym
ent
To
Pro
du
ctio
n (
Pre
-Pro
du
ctio
n)
Ap
plic
atio
n D
eliv
ered
to
Pro
gra
m/P
roje
ct f
or
Tes
tin
g
E2E
Fu
nct
ion
al P
erfo
rman
ce
Dis
aste
r R
eco
very
Regression Testing Regression Testing
23
Test Management
The aim is to deliver quality applications in less time.
The process begins with managing requirements and test planning.
The test management process includes the preparation of the plans for execution of the test processes.
The test management activity will monitor and evaluate all test results and is performed during all life cycle processes and activities.
24
What are the key benefits of test management?
Reduce defects by capturing issues earlier in the planning process
Understand the impact changes in requirements have on testing by creating relationships through traceability
Ensure full coverage of requirements by your test cases and test plans, so everything is properly tested
Keep your planning and testing teams in sync throughout the process with open and ongoing collaboration
25
Test Planning
Like many elements of project management; testing is a discipline in itself and embodies a project in its own right.
Testing must be planned to ensure it delivers on its expected outcomes.
26
Verification Methods
Verification of test results is generally carried out through visual checking, by a tester, of actual results against expected results.
Not all parts of a solution features or characteristics are testable in a technical sense, but must be verified using an alternate method, such as:
27
Acceptance Criteria
The purpose of setting acceptance criteria is to lay down a clear and concise set of goals for the “project” to enable all the stakeholders and participants to understand exactly what the final product and deliverables should be measured against.
This is a necessary step in the life cycle for testing; because it clearly establishes the goals of testing at each phase and avoids confusion between stakeholders and other parties.
28
Testing Completeness
Testing will be complete if 100% of test cases have been attempted during a test phase or cycle and have a pass rate of greater than 95%.
95% is a minimum pass rate, a higher % is better, e.g. 97%, 98%, 99%, and of course 100%.
This translates to mean that the solution is “fit for purpose” with minimal risk, assuming that the 5% of failed tests do not equate to any outstanding critical defects.
If for example 1,000 test cases were run, that would mean; 50 failed and are still outstanding or were not run.
(Survivable) Reality though is that testing completeness will be a trade off between time, resources, and level of comfort that what is outstanding is “manageable”
29
Closing comments
Bring attention to opportunities and risks in testing
Championing of quality in testing
Raising awareness of test issues and opportunities
Advocating more effective testing practices
Proactively tackling quality and testing issues
Should we do technology testing?
30
Any Questions