A New Model for Testing
@paul_g
erra
rd
Paul [email protected]
gerrardconsulting.com
Programme Chair 2014
Helping clients transform their testing through
INNOVATION, COACHING and LEADERSHIP
Our CLIENTS– Want to be agile rather than follow Agile
dogma– Have a pragmatic approach and are focused on
delivery– Want a solution that fits, not a badly fitting
suit.
Typical tester job spec? (May 2014)• Skills & Requirements
– Thorough C# knowledge including Linq– Good .Net experience including charting and winforms
applications.– Experience building and maintaining automated test
systems– Ability to interact with the developers to anticipate
future needs– Ability to influence to ensure good test procedures are
developed and implemented
• Desirable Skills– Some knowledge of Java for mobile testing– An interest in photography. Wearable Computing
(Internet o
f Eve
rything)
Agenda
• Pressure to Change• Three Innovations• Where is this Leading?• A New Model for Testing• Consequences• Shift-Left is More Important than
Agile
What's happening?
• Stampede to mobile computing• Big Data• Internet of (Every)Thing, Pervasive
Computing• Continuous Delivery, DevOps• (Test) Analytics, Data-Driven
Development• Shift-left, embedded testers, no test
team• Did you really think Agile was the last
word?
Three Innovations That Are Driving
Change
Being agileShift LeftAnalytics
#1Agile doesn’t work
but being agile might
Agile is no longer innovative.Challenging Agile and being
agile might beIs continuous delivery for
you?
Agile doesn’t work?
• It works for some• But many organisations have difficulties• Mostly a cultural problem– Requirements take months to elicit and agree– Complex stakeholder relationships– Stakeholders with ‘all the knowledge’ can’t be
spared to be the on-site customer/product owner– Lack of confidence => big system/acceptance
tests
• We are Agile but ...
Pragmatism rules (again)
• Pragmatic companies use the things that work and discard those that don’t
• Let's be honest: Most are doing some form of hybrid approach:– Chaotic interfaces between Agile teams
and structured businesses– TDD, BDD, ATDD, SBE today…– Continuous Delivery, DevOps… tomorrow?
#2Shift Left
Developers in test?Testers in development?
Testing (testers?) everywhere
TestingBDD/Test-Driven)Trusted
Requirements + Stories
Shift left (redistributed testing)
Functional Requirements
Build (+little testing) but lots of rework
Testing (lots of checking + bugs)
Trusted Requirements +
StoriesTesting
SAVEDTIME
BDD/Test-Driven
Who is going to do all this?BAs, Developers or Testers?
From traditional delivery…
… to Continuous Delivery
Deployment into Production is not
the Same as ‘Going Live’
Release to production and test environments on-demand
Go Live weekly/monthly/quarterly…
#3Test Analytics
Analytics from productionAnalytics from test
An opportunity for testing
Analysis Insight Decision Development and Testing
ProductionTesting
Introducing Test Analytics
“Introducing Test Analytics”, Paul Gerrard:http://gerrardconsulting.com/index.php?q=node/630Julian Harty, "Symbiotic Relationships between Testing and SMAC":http://blog.bettersoftwaretesting.com/2013/07/symbiotic-relationships-between-testing-and-social-mobile-analytics-cloud-computing/
“The capture, integration and analysis of test and production monitoring data to inform business and software development decision-making”.
Data silos for test analytics(an illustration)
Stakeholder Requirements Assurance (Production) ApplicationMonitoring
Production Environment Monitoring
Stakeholders
Business Goals and measures
Stakeholder Involvement/ Engagement
Risk
Requirements
Stories/Feature Descriptions
Glossary of terms and Term Usage
Processes
Process Paths (Workflows)
Manual Tests
Generated Test Code (unit, integration, system level)
Application Instrumentation
Automated Test Execution History
Test Status
Feature Code Change History
Application Process Flows
User Accesses/ Activities
Feature Calls, Response times
Feature use cases (data usage)
Application Alerts/Failures
Database Accesses
Database Content
Production failures
System assets
Resource Usage Log
Performance data
System Events
System Alerts/ Failures/Incidents
Outages
Where is this leading?
If I knew that, I wouldn't be here...
Three development patterns
Structured
Agile Continuous
Goal-B
ased
Hi-Process
Autonomous
There aren't three patterns
There are many
From high process to DIY process• In the past, high process was seen to be
the technical solution to the software problem
• But technical solutions cannot solve human problems
• Agile promised flexibility, but a lot of Agile is really 'high process-in-the-small'
• True agility means a team can adapt or select its process on-the-fly.
The old ways won't work in the future
We need a New Model of Testing (free from logistics)
A New Model for Testing
Don't throw stones. Yet.I'm working on a refined version (with
others).Want to help/review?
Forget Logistics(for the time being)
Document or not?Agile v waterfall?
Planned v exploratory?This technology v that
technology?
ALL Testing is Exploratory
We explore sources of knowledge ...
... to build test models ...... that inform our testing.
All Testing is Based on Models
Examples of models
Models are innate, essential, human
Judgement, exploring and testing
Testing(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring (sources) Judgement
Creates testmodels
Uses testmodels
We explore sources of knowledge to build test models that inform our testing
BTW – Do Developers explore the same way? I think so.
Exploration process
ExplorationDefinitions specs/stories
Oracles
Sources
DesignModels
TestModels
Enquiring
Challenging
Sources:People, documents,experience, system under test
Modelling
Test Models:Can be documented
or mental models
Predicting
System under test
Testing process
TestingSystem
Under Test
Refining
Informing Applying
Interpreting
TestModels
Revise the System
Logging Revi
sing
More exploring Reporting
New Model Testing
29 page paper: http://dev.sp.qa/download/newModel
Some Consequences
Do developers and testers think the
same way?
Similar processes, not thinking the same
Easier to embed testers in development
Developer testing will improve
Covers functional and non-functional
testing?
Ignoring logistics means we can have the same
foundational process for ALL testing(I think)
Relation toTDD and BDD?
TDD is not testing BDD is modelling using stories
Testing v Checking?
I submit that a Check is "a test that can be applied by a
human or tool"But only humans can interpret
Test automation from a different
perspective
Automation uses different test models
Is that why automation efforts fail so often?
Developer Testing
The model works (for me)
Capabilities
Enquiring, Modelling, Predicting, Challenging
Informing, Applying, Interpreting, Refining
Reporting and Logging
• Analysis, enquiry and elicitation• Modelling• Creation of custom models, using
heuristics, guesses, brainstorming, ideation, creative thinking
• Custom test design techniques• Comparison of models, value,
advantages, disadvantages, compromises
• Identification, validation and use of oracles
• Predicate logic and proof• Hypothesis and inference• Socratic method• Rapid Review and Inspection
techniques• Test case design• Test models and the meaning of
coverage• Testing as controlled experiment• Observation, Note taking,
recording
A very different skillset• Basic data analysis and statistics• Decision-making with
incomplete data• Computer forensics• Fault tree analysis• Failure diagnosis• Bug advocacy, triage processes
and negotiation• Meaningful software and test
metrics• Visual presentation of data• Reporting and presentation skills• Understanding stakeholders• Test analytics• Risk management, risk-based
testing and decision-making• Critical Thinking• Interpersonal skills• Dealing with
uncertainty/fallibility
Shift-Left is More Important than
Agile
Developers write code; testers test
because developers can’t
No they don’t
Goal: End Manual Feature Checking
Let the tools do it andfree people to do testing
please!
Testing doesn’t just apply to existing software, at the
endWe test systems, not
software.We test early, we test often but it might not be called
testing
Testers Need to Learn How to Code
Some, perhaps many will have to
Testing is about measuring
achievement, not quality
Align systems delivery with project goals and risks and you’ve got a seat on the
board
Testers don’t own testing anymore
Be a TestMaster"Assurance through the
lifecycle"
New Model Testing changes what
testers & developers need to
learnRoles of developers/testers become
blurredPerhaps they are not different disciplines
after all?Why have a tester certification?
The phase after development is REWORK, not
TestingTesting must align with
development, not compete with it or rescue it
A New Model for Testinghttp://dev.sp.qa/download/newModel
@paul_g
erra
rd
Paul [email protected]
gerrardconsulting.com
Programme Chair 2014
Recommended
