Test automation workshop slideset

Test AutomationTest Automation

[email protected]

Pondering Workshop for FYI on Winter 2011

Towo Toivola Attribution (Tekijä mainittava) http://creativecommons.org/licenses/by/3.0/

Welcome to the Workshop

I hope we all have an appetite for arguing and learning about test

automation

The slides are in English just to be sure, but I will present in Finnish if the audience prefers it.

Many ThanksMany ThanksMuch of this material is heavily

indebted to Maaret Pyhäjärvi, Erkki Pöyhönen, Petri Kuikka, Risto Kumpulainen, Dean Leffingwell..

Thanks also go to my friends at work and in the software engineering community

Towo Toivola 2009

About MeAbout MeExperiences define viewpointsMSc (eng) from HUTIn F-Secure Corp for 11 years

◦COTS software, custom systems◦Many many projects, large and small

A year in Kosovo◦IT administration, system building

Seminars, workshopsInteractive speaker

Towo Toivola 2009

About YouAbout YouHow many are test engineers?How many managers?Any programmers?How many have some

experience on testing tools?What is your industry?What are your most important

requirements for this workshop?

Towo Toivola 2009

DisclaimerDisclaimerAs with everything, nothing is

absoluteI shall not discuss load-testing

much, or test process automationNot an expert of the particular toolsYour terminology may differAttending this course may not be

enoughIt may be that you should not do

test automationThere are two worlds, I will try to

balance in between themTowo Toivola 2009

Agenda for TodayAgenda for Today

Towo Toivola 2009

0900 Welcome (this)

0920 What kind of TA system would you like?

1000 What is the goal of your TA initiative?

1105 Where does all the testing time go?

1255 How would you choose your TA tool?

1345 What is the right level for doing TA in your organization?

1430 How to take TA into use?

1530 How will you change your products to ease TA?

1600 Resourcing TA in the long run?

Lunch

BreakBreak

About the ScheduleAbout the ScheduleThere is a lot of ground to coverThis is a new format, bear with

meWe can change things

You can check what I am planning to leave out..

There should be enough breaks, we should be able to concentrate well on work◦What rules should we have?

Towo Toivola 2009

Format for this Workshop

We will use this format to deal with the questions in the agenda:

Presenting a question (5min)Pondering in small groups

(15min)Discussion with all together

(10min)Presenting of prepared material

(10min)This should make

it into about 30-50 mins per problem if we don't

get carried away.

Why this Format?Traditional teaching (providing answers)

does not promote thoughtThought and discovery promotes

learningStruggling with a problem first and

receiving guidance after that promotes understanding and application (I hope)

Interaction promotes understanding and information dissipation

I am trying to give you the most valuable things well rather than a lot of things

Technical details are not as important as people think

Question #1

What kind of TA system would you like to have?

(Outline your vision or what you want to have in one years time)

0920

Please take 15 to think about thisFirst quickly by yourselfThen within your groupDocument ideas, keywords and

major agreements on provided office supplies

Be prepared to present your views and findings, even if you are not so sure

Results and Discussion for Question #1

What kind of TA system would you like to have?

(Outline your vision or what you want to have in one years time)

Why do You Want What You Want?Tools and technical solutions often

dominate unduly in the decisionsOrganizational boundaries are often

taken as granted and TA may lead to local optimization

You have probably made a lot of implicit assumptions before even starting to actively think about the problem

People tend to focus on fixing what they understand best instead of that which is most important.

Now that you have already discovered what should be the end

result of today, we can start working in light of that assumption ;)

Question #2

What is the goal of your TA initiative?(Why are you here, why do you want to

do TA, what do you expect to gain?)

1000

Group Work Time

First quickly by yourselfThen within your groupDocument Be prepared to present


What is the goal of your TA initiative?

What’s the Use of Test What’s the Use of Test AutomationAutomation

Cost savings◦Ability to run the same tests cheaper

Faster development◦Ability to implement more features in

the same amount of timeFaster delivery

◦Ability to do snap changesBetter quality

◦Ability to reach and maintain higher quality

Towo Toivola 2009

Differences to Manual Differences to Manual TestingTestingSome tests that are manually

possible are not possible to automate

Some tests that are possible to automate are not possible to run manually

Development takes much more time, execution much less

Much more attention to detail, much less common sense

Requires more maintenanceTowo Toivola 2009

Enabling the Fast Enabling the Fast Feedback LoopFeedback LoopA common and useful use of test

automationMinimize the time from

programming to quality information◦Code – test – fix – retest in hours◦If not in minutes!

Continuous IntegrationFully automatic tests and

reporting”The most important limiting

factor for the velocity of an agile team” -DL

Towo Toivola 2009

VISIBILITY!

Regression and Regression and AutomationAutomation

Is the software still OK?Manual regression will get rid of

your good test engineersAutomation does not get boredThe courage to do changesThe ability to do quick releases

Towo Toivola 2009

VISIBILITY!

The Machine is Superior to The Machine is Superior to ManManAutomation is fasterAutomation has more attention

to detailAutomation can run through

more dataAutomation can execute more

combinationsAutomation can measure more

exactlyAutomation does not get tiredAutomation works through the

nightAutomation provides exact logs

Towo Toivola 2009

Man is Superior to the Man is Superior to the MachineMachineWorks around problemsCommon senseCreativityUnderstanding of the userUnderstanding of prioritiesUnderstands changeLooking outside the boxThe most important bugs are

found in the requirements and assumptions

Towo Toivola 2009

What Do You Want?What Do You Want?You Need to Prioritize, seriouslyIf you don’t set clear goals, don’t

expect clear benefits, or much results of any kind

Test faster?Test more often?Test better?Test in new ways?Test cheaper?

Towo Toivola 2009

Common MistakesCommon Mistakes(that cost a lot of money)(that cost a lot of money)

Saving a projectAiming for X% automatedWorking with untestable

softwareRunning a lot of tests, producing

little valueOrganizational issuesSkills of people working with

automation#1: Unrealistic assumptions

◦Automation will find loads of new bugs

◦No real resourcing needed

Towo Toivola 2009

VISIBILITY!

Real Benefit of TA

TA is only useful when it provides information that..

Gets a bug fixed

Helps you make a decision

It does this..More if you start

early in the project

More if you run it constantly

Only if you react to the results

Only if you believe the results

These require (re)action!

Test automation, when done sensibly, is hugely beneficial. It can

give you decisive competitive advantage over your competitors both financially and in terms of

employee motivation and satisfaction.

It's hard, but it's worth it.

Break TimeBreak Time

Towo Toivola 2009

1050

Question #3

Where does all the testing time go?(Testing software is an expensive

business, why does it cost so much? What do all those people spend all their

time on?)

1105

Group Work Time



Where does all the testing time go?

The Nature of a TestThe Nature of a Test

Which part can the automation Which part can the automation handle?handle?

Towo Toivola 2009

Where Does Your Testing Where Does Your Testing Time Go?Time Go?Do you use too much time

testing?Not enough time?Test planning?Test environment building?Regression testing?Testing new features?Testing large data collections?Testing with numerous

environments? Towo Toivola 2009

Things You Should Things You Should AutomateAutomateThings that are run very often

◦Core regression testsThings that are easy to automate

and bring valueThings that are tough to test

manually◦Performance◦Race conditions◦Linear combinations◦Large amounts of data◦Reliability

Things that have to be run quicklyROI !!!!11!!!!

Towo Toivola 2009

Things You Should Things You Should Not Not AutomateAutomateTests that are not importantTests that are easy to do

manually and hard to automateTests that are run seldom

◦Unless they are critical or very hard to run manually

Does-it-feel-rightTests where automation results

are not reliable

Towo Toivola 2009

But that is all the Sensible StuffIt is good to think about how to

balance between explorative and regression testing, but I bet your testers also use time on..

Meetings, overhead, bureacracyWaiting (for fix, hardware, decision,

info..)Arguing, pleading, fuming..Testing wrong thingsTesting wrong builds..

In the Insensible Reality..How much will making execution

faster actually help?Should you look at the whole

process?Maybe you should fix something

else (as well)?Could you use TA implementation

as a driver for new development culture?

Lunchtime!Lunchtime!

I have reserved one hour for lunch.

Towo Toivola 2009

1155

Question #4

How would you choose your TA tool?(Since you want to do TA, you need to somehow decide which tooling to use. How do you decide? How do you make

the decision an informed one?)

1255

Group Work Time



How would you choose your TA tool?

Finding a ToolFinding a ToolRemember that a tool is not the

same as test automation“A fool with a tool is a more

dangerous fool”There are many vendorsThere are many free alternativesTool finding must be considered

real work◦With goals◦With resources Towo Toivola 2009

Cost, What is it?Cost, What is it?Many costs

◦Cost of the tool (in the receipt)◦Cost of taking the tool into use

(mainly hidden) Including maintenance

◦Cost of not doing something else with the time

◦Cost of maintenanceYou should consider all costsCost of tool is usually overrated

Towo Toivola 2009

Impact of Tool on WorkImpact of Tool on Work

Towo Toivola 2009

Time

Now, what is expensive?

Cheap unsuitable tool

Big tool project

Small tool project

Project end

My AdviceMy Advice

Always take the one that is most suitable for you, taking into account:

Cost of toolIntended usePersonnel skills and attitudesGoals of the tool projectFuture plans for test automationDon't standardize for the sake of

it Towo Toivola 2009

Plan the Tool FindingPlan the Tool FindingMake it a proper project

If you still do projects

Plan – set goalsResource explicitlyHave many people join inScope out the tools out thereSelect a couple for trials

◦Do real work with them◦Not important, busy projects!

Towo Toivola 2009

RememberRememberYou will need support in using

the toolYou will need changes to the toolYour software will changeThe platform your software will

run on will change as wellYou may need changes to your

softwareOr your development process..

Towo Toivola 2009

What Should it be Like?What Should it be Like?Relatively easy to take into useEnables automatic testing, not

just playing with automation◦Including reporting

Supports your technology wellDoes not seriously restrict your

testersCan be run by a single person

and as a central systemFits your organization

Towo Toivola 2009

Or do You Need a Tool?You could make one, afterall you

make software Just right for you Intergrates nicely with your product and

development process

You could just use common utilities to create a mash-up tool

Build server SSH, remote exec Common scripting languages (with TA

libraries..)

In a Modern, Efficient, Agile OrganizationIn a galaxy far, far away..

The best people to make the trade-off decisions between tools are seasoned software development engineers who work in agile teams that have holistic responsibility for developing the software

Trust your professionals to make the right choice and support them

Their knowledge, interest and motivation are priceless in the implementation work of the TA

Success will truly inspire others

Question #5

What is the right level for doing TA in your organization?

(Should the SWE:s do UT/TDD? Should there be paired test engineers

perfoming them? Or module tests? Should there be multi-tier integration

tests? Can you do acceptance-level TA? Should you?)

1345

Group Work Time



What is the right level for doing TA in your organization?

What is Module- and Unit What is Module- and Unit testing?testing?

Towo Toivola 2009

Graphical User

Interface (GUI)

Module testing environment

Unit Testing Framework

When significant part of functionality

available in user interface.

When functionality is available from

below GUI or from side (components / modules and their

interfaces)

When each unit can be used to

catch the problems

“tester territory” “developer territory”

GUI Testing Tool

Application Programming

Interface (API)

Testing LevelsTesting Levels

Towo Toivola 2009

Cost / found

bug

Bug finding potential

System

Integration

Module

Unit

0% 100%

Can you find the silver bullet?

Why Test Automation on Why Test Automation on Many LevelsMany LevelsFaster automationFaster feedback loop

◦Ability to do snap changesBetter quality

◦Ability to reach and maintain higher quality

Faster (= cheaper) implementationLess maintenance work

◦Again, = cheaper

End result: Cost efficiencyTowo Toivola 2009

Differences to GUI Differences to GUI AutomationAutomationAPI:s are more straightforward to use

◦ Implementation is faster and cheaper◦ Tools are often free of charge

API:s change much more seldom than a GUI◦ Requires less maintenance

Some tests that are possible through API:s are not possible to run using the GUI

Some tests that are possible through the GUI are not possible to run using the API◦ You will test a bit less of your application◦ You can, probably, also module-test your GUI

Running takes typically less time Code – test – fix – retest in minutes

Detected bugs are usually easier to fix

Towo Toivola 2009

Question #6

How to take TA into use?(Who should do it? Where do we get the resources? Do we have the skills? Which project? Which product? How to create the culture? How to ensure the long-

term commitment of the organization?)

1430

Group Work Time



How to take TA into use?

Slide of Truth About Everything

Seriously, did you expect that I would have answers to that

question for you?Some thoughts will follow, but remember that every case is

unique.

Test Automation Test Automation ISIS Software Software System DevelopmentSystem DevelopmentGood test automation requires

that you understand testing well,And software developmentAnd system

implementation/administrationAnd preferably test automation

too.

Towo Toivola 2009

Most people don't reeeally get this

What Does That Mean?What Does That Mean?Source control and versioningBug trackingRequirements definitionProject management

◦Iterative and incremental if you want to be successful

ResourcingTesting!Fast feedback loop!Long-term commitmentETC.

Towo Toivola 2009

Don’t Leave the 20% Don’t Leave the 20% UndoneUndone

◦Usability◦Reliability◦User base◦Significance◦Reporting◦Analysing problems◦Maintenance

Towo Toivola 2009

Make sure you generate value ( = $$$$ )not just run tests

My serious main recommendation:One test case. Full chain. Beginning of

project.

Remember: TA is only useful when it

provides information that..Gets a bug fixed

Helps you make a decision

VISIBILITY!

Ugly slide..

Question #7

How will you change your products to ease TA?

(Should you? How? Should you change your process? Why?)

1530

Group Work Time



How will you change your products to ease TA?

A Word of WarningA Word of WarningAutomating an untestable

software can be ruinously expensive

Prepare for organization struggles when you need the software changed

And the processPrepare for the organization to

ignore any bugs found by automation unless they are reproduced by hand

Towo Toivola 2009

FurthermoreDeveloping untestable software is

ruinous anyway, stop doing itThere are N people changing the

software, how will n TA people keep up?

Who investigates the problems found in TA?

Who fixes? The bugs in software? The changes needed for TA?

Question #8

Resourcing TA in the long run?(Who will make sure TA keep generating value next year and the one after? How? Changing software! Maintenance hell!)

1600

Group Work Time



Resourcing TA in the long run?

Maintenance Maintenance If you get your tests running fine,

don’t think that’s most of the work done

The rest of your R&D works constantly – to change your software

Maintenance is in top 2 killers of test automation◦The other is organizational issues

You must be able to maintain your test code, constantly and easily

Towo Toivola 2009

Which Bit Would You Like to Which Bit Would You Like to Maintain?Maintain?

Towo Toivola 2009

How the Math Goes..How the Math Goes..You invest 3 people for automationYou invest 2 months for finding the

toolYou invest 6 months for developing

a good suite of testsChanges in the application make

the tests unreliable, so they will not be run, the organization will not believe in bugs found by the system

You just lost more than 3x(2+6)x5000e = 120 000e worth of investment

I did not include the price of the tool.. Towo Toivola 2009

What Should We LearnWhat Should We LearnImplementing a test automation

system is implementing a constantly supported service system based on software

Good system building and programming practises must be followed◦Duplicate code is your arch enemy

Resources must be allocated to maintain the system at all times

Towo Toivola 2009

08/03/10 Towo Toivola 2009

Something I Have Learned Something I Have Learned Since the 90’sSince the 90’sModel-based test automation

techniqueImportance of visual controlsSocial mass is more important

than technical massJust do it

Things we (probably) didn't have time for..

Technical detailsData-driven vs keyword-driven vs

model-basedDiscussing individual toolsDiscussing how to make software

in generalMuch else..

Debrief of the DayDebrief of the DayWhat do we remember from today?Was this day useful?

◦ What needs to be improved?◦ What was best?

What will you do differently in the future?

The topic is free, let’s have a discussion!

Towo Toivola 2009

BRoA #53 If you learn something and change nothing, you didn't learn anything.

Thank You for Your TimeThank You for Your Time

Towo Toivola 2009

We made it to the finish!

Next is a bunch of reserve slides that may come in handy at some

point...

RobustnessRobustnessAutomation scripts must run

reliablyWhen not testing something you

should perform operations in the most robust way possible

Error handling logicDuplicate code causes

unreliability and maintenance hell

Common operations belong to libraries◦Maintained, high-quality

Towo Toivola 2009

LibrariesLibraries

Towo Toivola 2009

Testing value

Required quality

Amount maintenance Test case

scripts(dozens or hundreds)

Test drivers(handful)

Utility libraries(handful) Use-case

libraries(handful)

Robustness exampleRobustness example

Towo Toivola 2009

Data-DrivenData-DrivenIf automation can do something,

it makes sense to do it a lotLoop over data, run

combinations◦Example: Log in with 10 incorrect

accounts and many correct accounts

Towo Toivola 2009

Keyword-DrivenKeyword-DrivenBasically a new language, just

for your testingReading in commands and data

instructions from a simple, human readable format

Enables non-technical people to understand and generate test cases◦Bring in the business knowledge

Requires quite a bit of workTowo Toivola 2009

(a very simple)(a very simple) Example Example

Go: EM_mainscreenDo: Enter message: “Hello work”Go: LogoutGo: EM_mainscreenDo: Verify message: “Hello work”

Towo Toivola 2009

ATDDATDDAcceptance/automated test

driven developmentAgree on the test first, on a level

understandable for allKeywords implement the testSoftware is developed to meet

the testHighly advanced development /

automation method

Towo Toivola 2009

(a very simple)(a very simple) Example Example

Given at effort manager login screen

When login with incorrect account

Then incorrect login reported

Towo Toivola 2009

Model-Based TestingModel-Based TestingCreating a state-machine that

models (some parts) of your software

Verifying the software behaves according to the state-machine

Combining transitions and data in many ways◦Different algorithms exist◦Random is pretty effective

Challenging, very advancedTowo Toivola 2009

(a very simple)(a very simple) Example ExampleState: emanager-login-screenInput: login-incorrectResultstate: badlogin-complaintInput: login-correctResultstate: emanager-mainscreenState: emanager-mainscreenInput: logoutResultstate: emanager-login-screen

…

Towo Toivola 2009

Automatic TestingAutomatic TestingBig difference between computer

aided testing and automatic testing

Automatic is automatic, human intervention is not needed◦Starts automatically ◦Runs automatically◦Analyses results automatically◦Reports automatically

Could your tests do that?Towo Toivola 2009

Exercise: AutomaticExercise: Automatic

Make your tests run so that they can be executed automatically

• Cmdline usage of TestPartner

• Driver Script

We should have about a half an hour.

Towo Toivola 2009

ReportingReporting

How would you like to get the test results?

Opening the tool?Going to the DB?By email?On a web page?In a file?As SMS?In public view?

Towo Toivola 2009

Technology

Test automation workshop slideset