The Art of Skillful Investigation

The Art of Skillful InvestigationJoris Meerts, 18 March 2014

The Art of Skillful Investigation

Belgium Testing Days

Joris Meerts, 18 March 2014

v1.0


About me

• Joris Meerts

• Software tester since 2007

• Capgemini

• Context-driven school

• www.testingreferences.com

• Member of the Dutch Exploratory Workshop onTesting (DEWT)

• @testingref


I want something that explains

what testing IS

not what it SHOULD BE


Normative theories


Thou shalt have no other gods before me.Thou shalt not make unto thee any graven image, or any likeness of any thing that is in heaven above, or that is in the earth beneath, or that is in the water under the earth.Thou shalt not bow down thyself to them, nor serve them: for I the LORD thy God am a jealous God, visiting the iniquity of the fathers upon the children unto the third and fourth generation of them that hate me;And shewing mercy unto thousands of them that love me, and keep my commandments.

Thou shalt not take the name of the LORD thy God in vain; for the LORD will not hold him guiltless that taketh his name in vain.

Remember the sabbath day, to keep it holy.Six days shalt thou labour, and do all thy work:

But the seventh day is the sabbath of the Lord thy God: in it thou shalt not do any work, thou, nor thy son, nor thy daughter, thy manservant, nor thy maidservant, nor thy cattle, nor thy stranger that is within thy gates:

For in six days the Lord made heaven and earth, the sea, and all that in them is, and rested the seventh day: wherefore the Lord blessed the sabbath day, and hallowed it.

What are normative theories?


Normative theories in testing

• Development of normative theoriesstarted early.

• We have many of them.

• IEEE 829, IEEE 1012 (V&V), TMap, ISEB, ISQTB, TMMi, ISO 29119


• Normative theories are hiding behind a ‘toolbox of practices’.

• They are never truly seen as theories.

• They are not challenged as theories would bechallenged.

Why are normative theories still around?


“Whether this method is

successful or not, is not known.

There are no scientific

publications on the subject.”


• William Howden - Reliability of the Path Analysis Testing Strategy (1976)• Richard Hamlet - Theoretical Comparison of Testing Methods (1989)• Elaine Weyuker - The Evaluation of Program-Based Software Test Data Adequacy

Criteria (1988)• Lee J. White - A Domain Strategy for Computer Program Testing (1980)• Luc Bougé - Test sets generation from algebraic specifications using logic

programming (1986)• Jefferson Offut - Criteria for Generating Specification-based Tests (1999)• Richard A. Kemmerer - Testing Formal Specifications to Detect Design Errors (1985)• Robert M. Hierons - Testing from a Z specification (1997)• John Goodenough, Susan Gerhart - Toward a Theory of Test Data Selection (1975)• Thomas J. Ostrand, Elaine Weyuker - Theories of program testing and the

application to revealing subdomains (1980)• Jeremy Dick, Alain Faivre - Automating the Generation and Sequencing of test

cases from model-based specifications (1993)• Deepinder Sidhu, Ting-Kau Lung - Formal Methods in Protocol Testing: a Detailed

Study (1989)

Mind you, there is research in testing!


“A scientific discipline emerges

. . . with the discovery of

useful and helpful concepts.

Scientific thought comprises in

addition the conscious search

for the useful and helpful

concepts.”

Edsger Wybe DijkstraOn the role of scientific thought (1974)


Descriptive theories


“Descriptive theories are needed

when nothing or very little is

known about the phenomenon in

question.”

Jaqueline FawcettThe Relationship of Theory and Research (1986)


“Anyway, I think we should stop

teaching testers practices that

they might or might not need.

Instead we should focus more on

teaching testers how to evaluate

their situation”

Markus GaertnerSoftware Tester – From New to Good to Great (2014)


Testing as investigation


Testing is the art of skillful

investigation by experimentation


• Mind maps

• Drawings

• Relationships

• Interaction

How to investigate:mental models


How to investigate:comparisons (1)

• Product history

• Manuals

• Standards

• Conversations

• Other products (suchas ketchup)

• Analogies


Sequence, Concurrence, Confluence, Synchronization, Share, Interaction, Continuity,

Hierarchy, Priority, Dependency, Repetition, Loop, Parameter, Prerequisite, Configuration,

Rule, Customize, Constraint, Resource, Access, Lock, State, History, Roll back, Restore,

Refresh, Clone, Temporary, Trace, Batch, Void, Absent, Feedback, Saturate, Sort, Scale, Corrupt, Integrity, Invoke, Timing, Delay,

Velocity, Instance, Format, Precision, Duplicate, Transmit, Wait, Source, Trigger, Lost, Reality,

Classification

How to investigate:comparisons (2)


experiment


How to investigate:conducting an experiment

• Set up a theory

• Conduct the experiment

• Observe the result


How to investigate:observation from a frame of reference

• William Kahan

• Father of the floatingpoint


How to investigate:reasoning

• Testers must provide sound evidence

• On the basis of theory

• On the basis of observations


• Jonathan Baron – Thinking and Deciding• Norwood Russell Hanson – Patterns of Discovery• Joy Paul Guilford – The Nature of Human Intelligence• Karl E. Wieck - Theory Construction as Disciplined Imagination• Ron Ritchhart - Intellectual Character : What It Is, Why It Matters, and How to Get It• Malcolm Gladwell – The Ketchup Conundrum• Philip N. Johnson-Laird - The Psychology of Syllogisms• Carl Sagan - The Demon-Haunted World• Shirley Gregor - The Nature of Theory in Information Systems• Paul Feyerabend – Against Method• Peter Naur - Programming as Theory Building• Jacqueline Fawcett - The Relationship of Theory and Research• Richard P. Feynman - Surely You're Joking, Mr. Feynman!• Philip N. Johnson-Laird - Mental Models• Plato - Selected Dialogues of Plato: The Benjamin Jowett Translation• Herbert A. Simon - The Sciences of the Artificial• Russell L. Ackoff - Towards a System of Systems Concepts• Jerome S. Bruner - Beyond the Information Given: Studies in the Psychology of Knowing• National Center for History in the Schools - Historical Thinking Standards• William Kahan - Needed Remedies for the Undebuggability of Large-Scale Floating-Point Computations in Science

and Engineering• Rikard Edgren, Martin Jansson, Henrik Emilsson - 37 Sources for Test Ideas

References


[email protected]

@testingref

www.testingreferences.com

Thank you for attending my presentation!

Technology

The Art of Skillful Investigation