View
136
Download
0
Category
Preview:
Citation preview
Service Overview Business Assurance
Advancing Testing Using Axioms@paul_gerrard
Paul Gerrardpaul@gerrardconsulting.comgerrardconsulting.com
Intelligent Assurance and Testing25-Nov-161
Helping clients transform their testing throughINNOVATION, COACHING and LEADERSHIP
Our CLIENTSWant tobe agilerather than follow Agile dogmaHave apragmatic approachand arefocused on deliveryWant asolution that fits, not a badly fitting suit.
AgendaIt all started with a thought experimentThere are no agreed principles in testingWhat does an Axiom look like?The sixteen Test AxiomsTest process improvement is a waste of timeImproving TestingConducting a Test Assessment (if I have time)Close
It all started with a thought experiment
Are there a set of axioms that provide a framework for all test approaches?
What is an Axiom?
An uncontested principle
2 + 2 = ???Something soself-evidently true it requires no proof
Example Axiom:
The shortest distance between two points is a straight line
We believe Axioms are INFALLIBLE(but we are prepared to be proven wrong)We KNOW that heuristics are FALLIBLE(cf the Context-Driven School)
Testing as the conscience of the project
Testers tend to take principled stances
We all have our favourite testing sayings butThere is no agreed set of laws so our profession is based on weak foundations.
We need to start with a definition of test(American Heritage Dictionary)
Test: (noun)A procedure for critical evaluation;A means of determining the presence, quality, or truth of something;A trial.
OK thats the intro
Lets play:Hunt the Axiom
Diverse principles promoted by diverse set of experts
Amazon offers 3557 choices
Differing backgrounds and experience of experts
Each present their own test principles
When some of them were written
Early focus on BUGS
Testing is an intellectually difficult activity
Complete testing is impossible
Independence of mind has value
Testing builds confidenceand so on
Test thinking has evolved from simple beginnings
Testing as a lifecycle activity
Model-based testing
Test-obsessed
Testing as exploration
Digital AssuranceAgile testing
Test-first, test-driven
Risk-based testing
Testing as learning
What could theTest Axioms look like?
From any perspective, they are self-evidently true
They apply to any test approach
They are context insensitive
They are not practicesPractices are always context-sensitive
A non-Axiomatic testing approach is unsound'
Projects that ignore the axioms might well be eCcEntRicWorse, you might be working in a
Project fromHELL
Different approaches reflect a different emphasis on the same axioms
Good test approaches align with the Axioms orThe Axioms are WRONG
Test AxiomsFormulated as a context-neutral set of rules for testing systemsThey also represent some critical thinking processes required to test any systemThere are clear opportunities to advance the practice of testing using themTesters Pocketbook: testers-pocketbook.comTest Axioms Website testaxioms.com
The Proposed Axioms
Axiom NameEach Axiom has a name and a one-line summary that states the principle.
Stakeholder AxiomTesting needs stakeholders
Assurance, Re-Training, Mentoring25-Nov-16 Aqastra Limited 200841
Test Basis AxiomTesters need sources of knowledge to select things to test
Test Oracle AxiomTesters need sources of knowledge to evaluate actual outcomes or behaviours
Fallibility AxiomOur sources of knowledge are fallible and incomplete
Scope Management AxiomIf you dont manage scope, you may never meet stakeholder expectations
Design AxiomTest design is based on models
Coverage AxiomTesting requires a coverage model or models
Delivery AxiomThe usefulness of the intelligence produced by test determines the value of testing
Environment AxiomTest execution requires a known, controlled environment
Event AxiomTesting never goes as planned; evidence arrives in discrete quanta
Prioritisation AxiomTesting needs a mechanism for ordering tests by value
Execution Sequencing AxiomRun our most valuable tests first we may not have time to run them later
Repeat-Test AxiomSome repeated tests are inevitable
Good-Enough AxiomThe scope of testing and acceptance are always compromises
Never-Finished AxiomTesting never finishes; it stops
Value AxiomThe value of testing is measured by the confidence of stakeholder decision making
DesignCoverageValueScopePrioritisationFallibilityEventOracleNever-FinishedGood-EnoughEnvironmentBasisRepeat-TestDeliveryThe 16 AxiomsSequencingStakeholder
How to group the Axioms?Would it be helpful to group the Axioms in some way?To give them a structureTo relate them to the real worldHow do the Axioms relate to the test process?
Assurance with Intelligence25-Nov-16 Gerrard Consulting Limited 200858
DesignCoverageValueScopePrioritisationFallibilityEventOracleNever-FinishedGood-EnoughEnvironmentBasisRepeat-TestDeliveryGrouping the AxiomsSequencingStakeholder
The three axiom groupsStakeholderValueScopeFallibilityGood-EnoughDeliveryRepeat-TestSequenceEnvironmentEventNever-Finished
DesignBasisCoveragePrioritisationOracle
The three groups - summarisedStakeholderThese are our customers what do they want from testing?What intelligence? When? How? How often?DesignWhat do we plan to deliver?How do we plan to deliver it?DeliveryHow do we manage the process of delivery?How do we communicate intelligence?
How can we use Test Axioms?Test Axioms are not beginners guidesThey can help you to think critically about testingThey expose flaws in other peoples thinking and their arguments about testingThey generate some useful by-productsThey help you to separate context from valuesInteresting research areas!First Equation of Testing, Testing Uncertainty Principle, Quantum Theory, Relativity, Exclusion Principle...You can tell I like physics.
Test Process Improvement is a Waste of Time!
Assurance, Re-Training, Mentoring25-Nov-16 Aqastra Limited 200863
How to improveI want to improve my (insert any activity here)_______ people improvement_______ organisation improvement_______ process improvement
Changing people (like me) and organisation (like my company) is so hard lets not even think about it.
There are no practice Olympics to determine the bestThere is no consensus about which practices are best, unless consensus means people I respect also say they like itThere are practices that are more likely to be considered good and useful than others, within a certain community and assuming a certain contextGood practice is not a matter of popularity. Its a matter of skill and context. The delusion of best practiceDerived from No Best Practices, James Bach, www.satisfice.com
The delusion of process models(e.g. Capability Maturity Model)Google searchCMM 22,900,000CMM Training 15,200CMM improves quality 66
So lots of consultants making money out of training (and assessment)?Does CMM improve quality?
Coordinate measuring machineThis talkAnother of my talksAnother of my talksReducing defects by 50% improves quality 2x???
The delusion of process models(e.g. CMM)One of my clients CMM level 3 and proud of it (chaotic, hero culture)Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping)40+ recommendations, only 7 adopted they couldnt changeHow on earth did they get through the CMM 3 audit?
People like simple models:levels of maturity, stepping stones, checklists, roadmaps and outside support for credibilityBut life is much more complicated, unfortunately
Things should be made as simple aspossible, but no simpler - A. EinsteinBut process models make improvement simple dont they?
People need process?A big problem with process is it becomes all encompassingProcess folk sell process and cast all things in terms of it, forgetting that people who are smart, succeed in spite of process, not because of itIt could be argued, that less smart people need process(By less smart, we're talking about people who need so much structure and enforced discipline they can only operate in the military, or in prison probably)Is our industry really staffed by such people?Do we really want process drones?
I believe that a scientist looking at nonscientific problems is just as dumb as the next guy
It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong Richard P. FeynmanPhysics quotes
I believe that a process consultant looking at non-process problems is just as dumb as the next guy
It doesn't matter how beautiful your process model is, it doesn't matter how smart you are. If it doesn't agree with reality, it's wrong Me!Process quotesUsing process change to fix cultural or organisational problems is not going to workImproving test in isolation is not going to work either
Improving Testing
First Equation of TestingAxioms+ Context+ Values+ Thinking =Approach
Why you are where you are
Context+ Values+ Thinking=Approach
Recommended