Demise of Test Scripts,Rise of Test Ideas

Richard Robinson8 April 2013

Who is Richard?

● 10 year old tester● Consultant / contractor

Blog www.hellotestworld.com

Twitter @richrichnz

Skype richrob79

● Began thinking since Dice Game in 2009

● Co-organiser of Sydney Testers

● Background in Psych and Computer Science

● Enjoys running, piano, photography, travel

Subject Matter

"Demise of Test Scripts, Rise of Test Ideas"

What do I mean by test scripts:● upfront analysis● templated● preconditions, steps, expected result,

actual result, pass/fail● not automated

One of these...

Or these....

Terminology

CHARTER - Learning and execution log that captures all related information of the feature analysis. Includes metacognitive comments, tests, questions, bug observations, modelling and anything else relevant to the test session.

FEATURE - Any chunk of a system or product. Logical slice of functionality. eg. an interface, a webpage screen, a workflow scenario.

SILO Scratchpad for holding ideas of a common nature such as questions, test ideas, or problems.

TEST IDEAS - Lets discuss test scripts first, then get on to test ideas...

Test scripts - hands up

How do you feel about test scripts?

What do you think about when you are reading a test script document?

frustrated!

Repetition

Repetition

Repetition

Repetition

Repetition

RepetitionOMG!

Cem Kaner on test scripts

● Myth that scripts teach testers the system● In fact, they do the opposite● Steps switch off learning, intuition,

questioning... focus on steps themselves● Testers learn faster when they discover

concepts, rather than by being told them

There are books on this stuff...

Adults benefit from activity based and discovery-based styles of learning, not scripted instruction-based.

The Adult LearnerKnowles, Holton, & Swanson

Scripts promote confirmation bias

● Humans have selective attention● A scripts expected result biases them to

find it, and miss everything else.● This is known as confirmation bias.

http://en.wikipedia.org/wiki/Confirmation_bias

Inattentional blindness...

Test scripting focussed the mind on a strict set of steps and expected outputs. So what happens when something goes wrong that is not related to the test?Answer: blindness

Keith Klain on test scripts...

● Testers ignore everthing off-script

● Testers cannot react, learn and change their approach

● Factory model outdated● Offshoring becoming costly● Focus moving to skilled

testing

Let's do something about this...

Test scripts paint a bad picture for software testing, quality and don't add much value to a product or system.

Is there a better way?

Introducing test ideas...

● Single words or phrases that represent a concept to be considered for testing

● A checklist used as a guide to inspire valuable tests relevant to the feature

● Exist in any intended granularity

● Unit of testing currency that help focus tester on what is important

Here are some test ideas...

Happy pathUsabilityMain flowAlternate flowNegative testsError testsBoundary conditionsDatabase dataConfiguration

User personasProduction recordsProduction systemsPrivacyStandardsSecurityReconciliation testsPlatform performanceMobile

Why use test ideas?

● Act as a guide, not a control● Antifragile and evolve with project● Does not assume complete specifications● Traceability and progress reporting● Reporting on what was NOT tested● More likely to achieve insight● Empowers and motivates tester● Simplified, natural way of testing

Compatible with how people learn

When faced with a new feature, if a testers initial understanding is engaged, then the tester is more likely to succeed in understanding new concepts and information.

But how do I use test ideas?

● Can have none, one or many tests ● Can be of any granularity that seems

relevant to the feature under test● Can be captured in a checklist at any time● Needs to be evaluated against the context

of the testing effort● Should be explored from simple to complex,

alongside the learning process● Need to be captured in the summary report

for the feature under test

Demo feature: login page

Your mission is to test the latest feature to be developed, the login page.

Go test!

Demo of test ideas in a method...

Start here:● Developer demo● Repeat simple path● Tour the feature● Early discovery

Demo of test ideas in practice..

Feature introductionShould behave like any other log in screen. Our data is very important to us, so we should not allow a robot to attack the log in to steal user accounts. We pride ourselves on appearance so usabililty might be important. Requires user accounts to be set up. These ones have already been set up: - dev:password123

● Begin charter document

Demo of test ideas in practice..

● Begin charter documentInfo sourcesMyee, the developerCompare with other log in functions such as facebook, ebay http://ebay.com/loginIs there a standard for log in screens? Yes - found one: http://wikipedia.com/loginstandardsRead the spec and data dictionary for detail around workflows, found here http://corporate.wiki.com/datadictionary

Demo of test ideas in practice..

● Begin charter documentInitial test ideasInput field boundaries and definitionFailed login attemptsLocked accountReset account functionRequest password and usernameStay signed inConfigurationCookiesEncryption of data

Sessions and public terminalsMultiple sessions on multiple devicesSecurityUsability - tabbing, enter to submit?NotificationsError messages - different for bad username and bad password?Help linkTimeoutRe-captcha on reattempts

Demo of test ideas in practice..

● Begin charter documentInitial questionsWhat browsers are important?Is mobile browsing supported?How much time do I have to test this?Where can a user log in from?Images didn't display, is this a bug?Is this functionality out of the box or custom built?What is the name of the function?What platform are we operating on?What version of html and ajax are we using?Is the page compliant with YSlow?

Model moving partsLog in attempts: first time, less than max, at maxAccount state: unlocked, lockedLog-in state: logged in, logged out, session expired logged outTime: recent session expired, first time login on this pc, 2nd time (cookies)

Demo of test ideas in practice..

● Begin charter document

And the rest of the method...

● Execute some standard test ideas● Gather more test ideas, seek more

information● Investigate deeper test ideas● Consider automation● Consider performance● Consider security● Consider compliance● Document tests● Tester demo

Finally the coverage and report

Questions?

James Bach offers his opinion...

Play in browser

Method Analysis

1. Engage3. Info sources

2. Dev demo

4. Tour & charter ideas

5. Model

6. Gather info

7. Order ideas

8. Investigate

9. Automate

10. Performance

11. Security

12. Compliance

13. Document14. Test demo

Step 1: Engage

● Take early notes on the feature● What is important?● Why test at all?● Start your charter

Step 2: Dev demo

● Best done on testers machine● Prove that simple path works● Prove that config ok● Builds relations● Early questions● Keep it brief

Step 3: Identify info sources

● People● Drivers● Project state● Contract?● Vendors● Relationship managers● Data dictionary● Database access● Standards

Step 4: Tour and charter ideas

● System tours at your will● Early tests● Early learning● Gather test ideas● List questions● Identify testability challenges● Get curious!

Step 5: Model moving parts

● List states of things● List transitions● List actor types● List groups of relevant partitions● List key scenarios around time● List anything that moves● List key business rules

You may also like to jot down test cases from simple, standard flow, negative, complex cases

Step 6: Gather more info

● Read and gather more information● Process more test ideas from it● Refer to accronym heuristics

○ FEWHICCUPS, SFDPOT

● Ask questions● Learn learn learn!

Step 7: Order your ideas

● Checkpoint● Consider wider context● Stop look listen

○ Time○ Business drivers○ Stakeholder messages○ Contractual requirements○ Trouble areas (developer, history, complexity)

● Order tests by complexity○ Main flows / alternate flows○ Error cases○ Complex interactions

Investigate test ideas strength-based

● Gradually spiral outwards, from simple tests to complex

Time

Dep

th

BA

Dev

Tester

Step 8: Begin test idea experiments!

● You can start this any time, but you should be launching into it by now

● You are feeling confident about the feature● Your learning is now being cashed in● Conduct regular checkpoints● Raise bugs● Draft test procedures that may be reused

Step 9: Consider automation

Support automation● What tests require code to execute?● How can you improve the efficiency?● What tools might assist the testing?

Publish to auto framework● Propose some tests for inclusion● Purpose is to check only, not test● Monitoring function

Step 10: Consider performance

● Performance is a major concern at go live● How does load impact your feature?● How can you increase your volume?● Is this consideration relevant?

Step 11: Consider security

● Security compliance is a major consideration for most systems

● Sensitive data● Encryption● Hacking● Basic penetration tests

Step 12: Consider compliance

● Non-compliance can halt a project dead● Compliance standards● Safety of data● Global standards

Step 13: Document tests

● List out tests into specification● Indicate which are simple, standard, and

complex tests● Offers evidence of process for auditors● Charters are usually messy● Useful for identifying tests for regression

Step 14: Tester demo

● Tester demonstrates feature findings● Chance to give back info discovered● Chance to demo skills and test cases

Wise words...

Final word...

Play in browser

ResourcesMetrics and MeasurementMetrics, Qualitative Measurement, and Stakeholder Valuewww.kaner.com/pdfs/metricsCASTkaner.pdfKaner, 2009

Software Engineering Metrics: What Do They Measure and How Do We Know?testingeducation.org/a/metrics2004.pdfKaner & Bond, 2004

Testing CoverageSoftware Negligence and Testing Coveragehttp://www.badsoftware.com/coverage.htmKaner, 1995

Exploratory TestingExploratory Testing Explainedhttp://www.satisfice.com/articles/et-article.pdfBach, 2003

Session Based Test Managementhttp://www.satisfice.com/articles/sbtm.pdfJon Bach, 2000

Test Heuristics Cheat Sheethttp://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdfQualitytree.com, 2006

Mnuemonic mind maphttp://www.xmind.net/embed/YvxE/Charrett

How to Adjust to the Changing Face of Software Testing?http://www.cio.com/article/730131/How_to_Adjust_to_the_Changing_Face_of_Software_Testing?page=1&taxonomyId=3040Heusser, March, 2013

Blog: FEW HICCUPPShttp://www.developsense.com/blog/2012/07/few-hiccupps/Bolton, July 2012

Appendix: Meetup responses

"What do you think of test scripts?"

Appendix: Charter templateCHARTER

HYP/MISSION:

TASKS:

MODEL DATA:

INITIAL QUESTIONS:

INFORMATION:

EVIDENCE LOCATION:

ACCESS:

TESTS:

SCOPE:

VERIFICATION METHODS

LEARNING TASKS

=============

DEV DEMO -

OPERATION LEVEL TESTS -

READ KEY DOCUMENTATION -

REVIEW APPROACH DESCRIPTION

MODEL MOVING PARTS

REVIEW SPECIFIC TEST IDEAS

DB Field Check -

Mandatory Fields -

Optional Fields -

Field Length Boundaries -

Field Types

Full/Partial Records

Duplicate Records

XSD Structure Check -

Validation Rules Enforced

Negative Pathway Tests

Multiple Sets of Data

Logical Flow Tests

Error Coverage -

Intent Based Tests -

Check Logging for Errors

Privacy & Encryption

Standards -

Risk Evaluation

Light Load Tests -

Race Conditions -

UNIT TESTS IN SOAPUI

SCENARIO IDENTIFICATION -

DEV QUESTION & DEMO

INTEGRATE FULLY WITH FRAMEWORK -

DOCUMENT TEST CASES

DEMO TO TEAM

SUMMARY OF KEY FINDINGS

============

REPORT TO PM

============

Appendix: Skills framework

● Self-organised modules● Written by testers for testers● Closely related to specific test ideas

eg. DB Sql statements, bash scripts for logging, manipulation of data in np++