Service Overview Business Assurance

Advancing Testing Using Axioms@paul_gerrard

Paul Gerrardpaul@gerrardconsulting.comgerrardconsulting.com

Intelligent Assurance and Testing25-Nov-161

Helping clients transform their testing throughINNOVATION, COACHING and LEADERSHIP

Our CLIENTSWant tobe agilerather than follow Agile dogmaHave apragmatic approachand arefocused on deliveryWant asolution that fits, not a badly fitting suit.

AgendaIt all started with a thought experimentThere are no agreed principles in testingWhat does an Axiom look like?The sixteen Test AxiomsTest process improvement is a waste of timeImproving TestingConducting a Test Assessment (if I have time)Close

It all started with a thought experiment

Are there a set of axioms that provide a framework for all test approaches?

What is an Axiom?

An uncontested principle

2 + 2 = ???Something soself-evidently true it requires no proof

Example Axiom:

The shortest distance between two points is a straight line

We believe Axioms are INFALLIBLE(but we are prepared to be proven wrong)We KNOW that heuristics are FALLIBLE(cf the Context-Driven School)

Testing as the conscience of the project

Testers tend to take principled stances

We all have our favourite testing sayings butThere is no agreed set of laws so our profession is based on weak foundations.

We need to start with a definition of test(American Heritage Dictionary)

Test: (noun)A procedure for critical evaluation;A means of determining the presence, quality, or truth of something;A trial.

OK thats the intro

Lets play:Hunt the Axiom

Diverse principles promoted by diverse set of experts

Amazon offers 3557 choices

Differing backgrounds and experience of experts

Each present their own test principles

When some of them were written

Early focus on BUGS

Testing is an intellectually difficult activity

Complete testing is impossible

Independence of mind has value

Testing builds confidenceand so on

Test thinking has evolved from simple beginnings

Testing as a lifecycle activity

Model-based testing

Test-obsessed

Testing as exploration

Digital AssuranceAgile testing

Test-first, test-driven

Risk-based testing

Testing as learning

What could theTest Axioms look like?

From any perspective, they are self-evidently true

They apply to any test approach

They are context insensitive

They are not practicesPractices are always context-sensitive

A non-Axiomatic testing approach is unsound'

Projects that ignore the axioms might well be eCcEntRicWorse, you might be working in a

Project fromHELL

Different approaches reflect a different emphasis on the same axioms

Good test approaches align with the Axioms orThe Axioms are WRONG

Test AxiomsFormulated as a context-neutral set of rules for testing systemsThey also represent some critical thinking processes required to test any systemThere are clear opportunities to advance the practice of testing using themTesters Pocketbook: testers-pocketbook.comTest Axioms Website testaxioms.com

The Proposed Axioms

Axiom NameEach Axiom has a name and a one-line summary that states the principle.

Stakeholder AxiomTesting needs stakeholders

Assurance, Re-Training, Mentoring25-Nov-16 Aqastra Limited 200841

Test Basis AxiomTesters need sources of knowledge to select things to test

Test Oracle AxiomTesters need sources of knowledge to evaluate actual outcomes or behaviours

Fallibility AxiomOur sources of knowledge are fallible and incomplete

Scope Management AxiomIf you dont manage scope, you may never meet stakeholder expectations

Design AxiomTest design is based on models

Coverage AxiomTesting requires a coverage model or models

Delivery AxiomThe usefulness of the intelligence produced by test determines the value of testing

Environment AxiomTest execution requires a known, controlled environment

Event AxiomTesting never goes as planned; evidence arrives in discrete quanta

Prioritisation AxiomTesting needs a mechanism for ordering tests by value

Execution Sequencing AxiomRun our most valuable tests first we may not have time to run them later

Repeat-Test AxiomSome repeated tests are inevitable

Good-Enough AxiomThe scope of testing and acceptance are always compromises

Never-Finished AxiomTesting never finishes; it stops

Value AxiomThe value of testing is measured by the confidence of stakeholder decision making

DesignCoverageValueScopePrioritisationFallibilityEventOracleNever-FinishedGood-EnoughEnvironmentBasisRepeat-TestDeliveryThe 16 AxiomsSequencingStakeholder

How to group the Axioms?Would it be helpful to group the Axioms in some way?To give them a structureTo relate them to the real worldHow do the Axioms relate to the test process?

Assurance with Intelligence25-Nov-16 Gerrard Consulting Limited 200858

DesignCoverageValueScopePrioritisationFallibilityEventOracleNever-FinishedGood-EnoughEnvironmentBasisRepeat-TestDeliveryGrouping the AxiomsSequencingStakeholder

The three axiom groupsStakeholderValueScopeFallibilityGood-EnoughDeliveryRepeat-TestSequenceEnvironmentEventNever-Finished

DesignBasisCoveragePrioritisationOracle

The three groups - summarisedStakeholderThese are our customers what do they want from testing?What intelligence? When? How? How often?DesignWhat do we plan to deliver?How do we plan to deliver it?DeliveryHow do we manage the process of delivery?How do we communicate intelligence?

How can we use Test Axioms?Test Axioms are not beginners guidesThey can help you to think critically about testingThey expose flaws in other peoples thinking and their arguments about testingThey generate some useful by-productsThey help you to separate context from valuesInteresting research areas!First Equation of Testing, Testing Uncertainty Principle, Quantum Theory, Relativity, Exclusion Principle...You can tell I like physics.

Test Process Improvement is a Waste of Time!

Assurance, Re-Training, Mentoring25-Nov-16 Aqastra Limited 200863

How to improveI want to improve my (insert any activity here)_______ people improvement_______ organisation improvement_______ process improvement

Changing people (like me) and organisation (like my company) is so hard lets not even think about it.

There are no practice Olympics to determine the bestThere is no consensus about which practices are best, unless consensus means people I respect also say they like itThere are practices that are more likely to be considered good and useful than others, within a certain community and assuming a certain contextGood practice is not a matter of popularity. Its a matter of skill and context. The delusion of best practiceDerived from No Best Practices, James Bach, www.satisfice.com

The delusion of process models(e.g. Capability Maturity Model)Google searchCMM 22,900,000CMM Training 15,200CMM improves quality 66

So lots of consultants making money out of training (and assessment)?Does CMM improve quality?

Coordinate measuring machineThis talkAnother of my talksAnother of my talksReducing defects by 50% improves quality 2x???

The delusion of process models(e.g. CMM)One of my clients CMM level 3 and proud of it (chaotic, hero culture)Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping)40+ recommendations, only 7 adopted they couldnt changeHow on earth did they get through the CMM 3 audit?

People like simple models:levels of maturity, stepping stones, checklists, roadmaps and outside support for credibilityBut life is much more complicated, unfortunately

Things should be made as simple aspossible, but no simpler - A. EinsteinBut process models make improvement simple dont they?

People need process?A big problem with process is it becomes all encompassingProcess folk sell process and cast all things in terms of it, forgetting that people who are smart, succeed in spite of process, not because of itIt could be argued, that less smart people need process(By less smart, we're talking about people who need so much structure and enforced discipline they can only operate in the military, or in prison probably)Is our industry really staffed by such people?Do we really want process drones?

I believe that a scientist looking at nonscientific problems is just as dumb as the next guy

It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong Richard P. FeynmanPhysics quotes

I believe that a process consultant looking at non-process problems is just as dumb as the next guy

It doesn't matter how beautiful your process model is, it doesn't matter how smart you are. If it doesn't agree with reality, it's wrong Me!Process quotesUsing process change to fix cultural or organisational problems is not going to workImproving test in isolation is not going to work either

Improving Testing

First Equation of TestingAxioms+ Context+ Values+ Thinking =Approach

Why you are where you are

Context+ Values+ Thinking=Approach