Hybrid Keyword Data Driven Automation Frameworks

8/13/2019 Hybrid Keyword Data Driven Automation Frameworks

1/15

ANZTB 2010 Conference

Hybrid Keyword Data Driven Automation

Frameworks by Jonathon Wright

Tuesday, 2nd March 2010

ANZTB 2010 1

Hybrid Keyword Data DrivenHybrid Keyword Data Driven

Automation FrameworksAutomation Frameworks

An Insight to Approaches, Techniques &An Insight to Approaches, Techniques &

Methodologies.Methodologies.Jonathon Lee WrightJonathon Lee Wright

[email protected]@iee.org

www.automation.org.ukwww.automation.org.uk

www.linkedin.com/in/automationwww.linkedin.com/in/automation

2nd March 20102nd March 2010


Automation FrameworksAutomation Frameworks

What does this mean to you now?What does this mean to you now?

Hybrid Keyword Data Driven AutomationHybrid Keyword Data Driven Automation Frameworks (Slide 1)Frameworks (Sl ide 1) 2nd March 20102nd March 2010

Introduction


2/15





ANZTB 2010 2

Presentation PlanPresentation Plan

BackgroundBackground


Introduction

Hybrid Keyword Data Driven AutomationHybrid Keyword Data Driven Automation Frameworks (Slide 2)Frameworks (Sl ide 2)

TraditionalTraditional

ApproachApproach

Automation:Timeline

Automation:Failure Rate

Reasons forPast Failure

UnderstandingFailure

Break EvenPoint

Where toStart

GAPAnalysis

InitialInvestment

Break EvenPoint

Framework:Advantages

WhichTest Tool

One DedicatedTeam

BusinessProcess Testing

KeywordDriven

DataDriven

ModularityDriven

Summary

Framework ApproachFramework Approach(Hybrid Keyword Data Driven Automation Frameworks)(Hybrid Keyword Data Driven Automation Frameworks)

Buzzword HYBRIDBuzzword HYBRIDIntroduction

2nd March 20102nd March 2010Hybrid Keyword Data Driven AutomationHybrid Keyword Data Driven Automation Frameworks (Slide 3)Frameworks (Sl ide 3)


3/15





ANZTB 2010 3

Moving swiftly past the hypeMoving swiftly past the hype

Historically Automation is perceived as aHistorically Automation is perceived as aSilver Bullet of the Testing world.Silver Bullet of the Testing world.

The term has been adopted into a general metaphor, where "silver bullet" refers toThe term has been adopted into a general metaphor, where "silver bullet" refers to

any straight forward solution perceived to have extreme effectivenessany straight forward solution perceived to have extreme effectiveness. The. The

phrase typically appears with an expectation that somephrase typically appears with an expectation that some new technology or practicenew technology or practice

will easily cure a major prevailing problemwill easily cure a major prevailing problem

Introduction


Timeline:Timeline:

Product Test AutomationProduct Test Automation

Historical trends in automation frameworks:Historical trends in automation frameworks:

Background



4/15





ANZTB 2010 4

Learning from past experienceLearning from past experience

2004 2010 (Estimated)

Industry: Test Automation (net worth) $1 billion $6.3 billion

Automation Projects (failure cost) $0.6 billion $3.8 billion

Background


92%92% FAIL TO MEET TARGET ROIFAIL TO MEET TARGET ROI

Hybrid Keyword Data Driven AutomationHybrid Keyword Data Driven Automation Frameworks (Slide 6)Frameworks (Sl ide 6)Source A:ROI on Test Automation - http://www.keane.com

Source B:HP pay 4.5billion to acquire mercury - http://www.nytimes.com/2006/07/ 26/technology/26 hewlett.html

Reasons for past failureReasons for past failure

Lack of defined automation methodologyLack of defined automation methodology

Traditional Approach


WHY DO AUTOMATIONWHY DO AUTOMATION

PROJECTS TYPICALLYPROJECTS TYPICALLY FAIL?FAIL?

ATLM Diagram:Automated Software Testing (Du stin Rashka Paul) - ISBN 0-201-43287- 0

Source:Maximizing ROI and Avoiding the Pitfalls of Test Automation (Bill Ha yduk - RTTS) - http://www.rttsweb.com/research

Automation is not treated as a legitimateAutomation is not treated as a legitimateproject with the necessary planning /project with the necessary planning /

resourcesresources



5/15





ANZTB 2010 5


Test Automation is typically performed at the end of the development cycleTest Automation is typically performed at the end of the development cycle

UserRequirements

SystemRequirements

Global Design

Detailed Design

Implementation

Testing

Traditional - Test Automation

Functional Test Tool Dependences: Application Under Test (AUT) Delivery- Learn Object Repository

Hybrid - Test Automation

Business Process Modelling (BPM)- Mapping Business Process

Business Process Testing (BPT)- Reusable Modules (Jigsaw Pieces)

Descriptive Programming (DP)- Enables Dynamic Object Repository




After the initial success the automation scripts are not maintained for futureAfter the initial success the automation scripts are not maintained for futurebuildsbuilds

(contributing to a high failure rate of the remaining 32% of working automation projects)(contributing to a high failure rate of the remaining 32% of working automation projects)

Testers are typically untrained in test tools and programming techniquesTesters are typically untrained in test tools and programming techniques

No modularization (No modularization (reusable functionsreusable functions) in automation scripts) in automation scripts

Automated tests cases are usually designed based on front end functionalityAutomated tests cases are usually designed based on front end functionality((black box testingblack box testing))




6/15





ANZTB 2010 6

Putting the pieces of the puzzle togetherPutting the pieces of the puzzle together

LoginLoginRegisterRegister

ResetResetPasswordPassword

Real World ExampleReal World Example Simple Login ScreenSimple Login Screen

LOGINLOGINLOGINLOGIN



Registered

Registration

ForgotPassword

Login

Reset Password

BlackBlack--Box Activity DiagramBox Activity Diagram


Understanding why automation failsUnderstanding why automation fails

The Mine Sweeper EffectThe Mine Sweeper Effect



Automated testing repeats the same route over andAutomated testing repeats the same route over andover. Initially any defects that are found are whilstover. Initially any defects that are found are whilst

developing the scripts (i.e. manually)developing the scripts (i.e. manually)

Subsequent executions will have little chance of everSubsequent executions will have little chance of everfinding further defects unless the flow changesfinding further defects unless the flow changes



7/15





ANZTB 2010 7

Record Replay is not your friendRecord Replay is not your friend

Linear process mapping (AUT)Linear process mapping (AUT)


2nd March 20102nd March 2010Hybrid Keyword Data Driven AutomationHybrid Keyword Data Driven Automation Frameworks (Slide 12)Frameworks (Sl ide 12) Video Source: YouTube -Bugatt i Veyron Lake Crash -http ://www.youtube.com/watch?v=4NJmB1F2mdE

Breaking Even PointBreaking Even Point

Preparation (Preparation (VV)) Execution (Execution (DD))ROIROI

Resource (Resource (RR) for (n) Automated Tests) for (n) Automated Tests

TestTest ManualManual AutomatedAutomated ManualManual AutomatedAutomated 11 55 1010 2020

Scenario 1Scenario 1 3030 6060 1111 1.11.1 33 149%149% 77%77% 50%50% 32%32%



Scenario 4Scenario 4 3030 6060 99 0.90.9 44 156%156% 86%86% 57%57% 37%37%Scenario 5Scenario 5 3030 6060 1010 11 44 153%153% 81%81% 54%54% 35%35%

Scenario 6Scenario 6 3030 6060 1010 11 44 153%153% 81%81% 54%54% 35%35%





TotalsTotals 55 hrshrs 1010 hrshrs 140140 minsmins 1414 minsmins 3.23.2 142%142% 72%72% 47%47% 31%31%

RRnn = A= Aaa/ A/ Amm = (V= (Vaa ++ nn*D*Daa) / (V) / (Vmm ++ nn*D*Dmm))


2nd March 20102nd March 2010Equation Source:Automated Software Testing (Dust in Rashka Paul) - ISBN 0-201-43287 -0Hybrid Keyword Data Driven AutomationHybrid Keyword Data Driven Automation Frameworks (Slide 13)Frameworks (Sl ide 13)


8/15





ANZTB 2010 8

Framework ApproachFramework Approach

Introducing

(Hybrid Keyword Data Driven Automation Frameworks)(Hybrid Keyword Data Driven Automation Frameworks)


Automation:Timeline

Automation:Failure Rate

Reasons forPast Failure

UnderstandingFailure

Break EvenPoint

Where toStart

GAPAnalysis

InitialInvestment

Break EvenPoint

Framework:Advantages

WhichTest Tool

One DedicatedTeam

BusinessProcess Testing

KeywordDriven

DataDriven

ModularityDriven

Summary


Where to start?Where to start?Framework Approach


Under promise,Under promise,Over deliver?Over deliver?

First find out Then you can work out

WhatWhat needsneedsto be tested?to be tested? WhatWhat needsneedsbe automated?be automated?WhatWhat cancanbe tested?be tested? WhatWhat cancanbe automated?be automated?

WhatWhat couldcouldbe tested?be tested? WhatWhat couldcouldbe automated?be automated?

Quick winsQuick winsshould beshould beavoidedavoided

StartStart SMALLSMALLthinkthink BIGBIG

NEVERNEVER expect toexpect toautomateautomate 100%100%

RememberRemember to think of each module as a individual Jigsaw piece butto think of each module as a individual Jigsaw piece butkeep in mind what is needed to complete the entire puzzlekeep in mind what is needed to complete the entire puzzle

Focus on keyFocus on keycriticalcritical businessbusiness

processesprocesses

Keep it simple,Keep it simple,wherever possiblewherever possible



9/15





ANZTB 2010 9

Understanding theUnderstanding the full extentfull extent of Application Under Test (AUT):of Application Under Test (AUT):

GAP AnalysisGAP AnalysisFramework Approach


NOTE :NOTE : The GAP analysis file can be used toThe GAP analysis file can be used to generategenerate the parameter file which in turn is used to generate the dynamic object repositorythe parameter file which in turn is used to generate the dynamic object repository..

BusinessBusinessRiskRisk

ComplexityComplexity

UsageUsage FeasibilityFeasibility

ReusabilityReusability

GAP analysis = complete application functionalityGAP analysis = complete application functionality actualactual


based on sample set of 10 scenariosbased on sample set of 10 scenarios

leverageleverage

leverageleverage

initialinitialinvestmentinvestment

Investing the timeInvesting the time

Allocate realistic time frames & resources to achieve the initial framework:Allocate realistic time frames & resources to achieve the initial framework:

Framework Approach


Framework = planning + designFramework = planning + design+ development + testing+ development + testing

Framework

20hrs (prep)

14mins(execution)

ROI = 2.8


Manual

5hrs (prep)

140mins(execution)

N/A

Traditional

10hrs (prep)

14mins(execution)

ROI = 3.2


10/15





ANZTB 2010 10

Breaking Even PointBreaking Even Point

Preparation (Preparation (VV)) Execution (Execution (DD))ROIROI

Resource (Resource (RR) for (n) Automated Tests) for (n) Automated Tests

TestTest ManualManual AutomatedAutomated ManualManual AutomatedAutomated 11 55 1010 2020











TotalsTotals 55 hrshrs 2020 hrshrs 140140 minsmins 1414 minsmins 2.82.8 281%281% 33%33% 16%16% 11%11%

RRnn = A= Aaa/ A/ Amm = (V= (Va / na / n ++ nn*D*Daa) / () / (VVmm ++ nn*D*Dmm))

Framework Approach


Key AdvantagesKey AdvantagesFramework Approach


MaintainabilityMaintainability significantly reduces the testsignificantly reduces the test maintenancemaintenanceefforteffort

ReusabilityReusability due todue to modularitymodularityof test cases and library functionsof test cases and library functions

ManageabilityManageability -- effective test design, execution, and traceabilityeffective test design, execution, and traceability

AccessibilityAccessibility to design, develop & modify teststo design, develop & modify tests whilst executingwhilst executing

AvailabilityAvailability scheduled execution can runscheduled execution can run unattendedunattendedon aon a 24/724/7 basisbasis

ReliabilityReliability due todue to advanced error handlingadvanced error handling andand scenario recoveryscenario recovery

FlexibilityFlexibility frameworkframework independentindependentof AUT or environmentof AUT or environment

MeasurabilityMeasurability customisablecustomisable reportingreportingof testof test resultsresultsensure qualityensure quality



11/15





ANZTB 2010 11

Which?Which?

WhichWhich Automation Tool SuitsAutomation Tool Suits You?You?

Approach BPT/BPM Keyword Data Modularity Example

Functional Test ToolFunctional Test Tool QuickTestQuickTestProfessionalProfessional

Test Management ToolTest Management Tool Quality CenterQuality Center(BPTEE,(BPTEE, BPM)BPM)

Standalone IDE ToolStandalone IDE Tool VBS (WindowsVBS (Windows

Scripting Host)Scripting Host)Combination of all the aboveCombination of all the above Hybrid Solution*Hybrid Solution*

Framework Approach


** optimised solution with preferred vendorsoptimised solution with preferred vendors

One dedicated TeamOne dedicated Team

REQUIREMENTSREQUIREMENTS TEST PLANNINGTEST PLANNING

TEST EXECUTIONTEST EXECUTION TEST REPORTINGTEST REPORTING

SOLUTIONSOLUTION

Framework Approach


Business Analysts(Functional Specs)(Change Requests)

Business Analysts(Functional Specs)(Change Requests)BABA

System Analysts(Business Process Modelling)(Requests for Change)

System Analysts(Business Process Modelling)(Requests for Change)SASA

Test Analysts Test Analysts TATA

Subject Matter Experts- Domain Experts (BA/SA)- Developers

Subject Matter Experts- Domain Experts (BA/SA)- Developers

SMESME

Automation Architects Framework Automation Experts


AAAA

Test Analysts Test AnalystsTATA

Subject Matter Experts- Domain Experts (BA/SA)-Developers

Subject Matter Experts- Domain Experts (BA/SA)-Developers

SMESME


Automation Architects Framework Automation ExpertsAAAA

Test Managers Test Managers TMTM

Project Managers(Dashboard)

Project Managers(Dashboard) PMPM



12/15





ANZTB 2010 12

Login

Subject Matter Experts (SME)Subject Matter Experts (SME)design tests using documentationdesign tests using documentation

to map logical data flow andto map logical data flow and

process modellingprocess modelling

Step 2Step 2 Design Test Case(s)Design Test Case(s)

Business Process Testing (BPT)Business Process Testing (BPT)

Collaborative

Solution

Input Field Values Screen

Format

Username 64 Characters TextPassword 12 Characters Hidden

Login Button Visible

Subject Matter Experts (SME)Subject Matter Experts (SME)

System Requirements Specification (SRS)System Requirements Specification (SRS)

Functional specifications document (FSD)Functional specifications document (FSD)Software design document (SDD)Software design document (SDD)

Detailed design document (DDD)Detailed design document (DDD)

Step 1Step 1 Export Core LogicExport Core Logic

Framework Automation ExpertsFramework Automation Experts

Step 3Step 3 Code GenerationCode Generation

Framework Automation Experts transformFramework Automation Experts transform

designed tests into automationdesigned tests into automationcomponents ready for keyword stagecomponents ready for keyword stage

Framework Approach


Login

Subject Matter Experts (SME)Subject Matter Experts (SME)

Step 4Step 4 Create Test Case(s)Create Test Case(s)

Subject Matter Experts (SME) create testSubject Matter Experts (SME) create test

cases using business level keywords storedcases using business level keywords storedwithin the MasterDriver file containing all thewithin the MasterDriver file containing all the

test scenarios relating to the AUTtest scenarios relating to the AUT

Action Object Event(Type)

Activate Login Window

Set Username Editbox

Set Password Editbox

Press Login Button

Keyword DrivenKeyword Driven

Generic Reusable Libraries

Actions

Object

Handler

Event

Handler

ObjectRepository

Actions

Example Runtime Process

The framework calls the necessaryThe framework calls the necessary

functions to process each keyword /functions to process each keyword /component and dynamically createscomponent and dynamically creates

the object repository at runthe object repository at run--timetime

Log.Framework ObjectRepository.CreateObject.Window ObjectRepository.CreateObject.Button ClearAlerts Log.Events Sync.WinWaitObject.Button Action.Preamble.Check.ButtonState Event.PushDynamic.Button Action.Postamble.Check.ButtonState Action.Reaction.Check

Popup.Handler Error.Recovery (retry/skip/baseline) Log.Results Log.Reports

Framework Approach


Driver & Data Repository

MasterDriver File Parameters File

The Hybrid Keyword Data DrivenThe Hybrid Keyword Data Driven

Automation Framework processesAutomation Framework processeseach action/object/eventeach action/object/event

keyword/component in turnkeyword/component in turn


13/15


14/15





ANZTB 2010 14

Guidelines behind working smarterGuidelines behind working smarter

CentralizedCentralized -- creation of the centralizedcreation of the centralized master driver filemaster driver file totoprovideprovide stablestable,, repeatablerepeatablekeyword / data driven test scenarioskeyword / data driven test scenarios

DynamicDynamic -- runtime generation of theruntime generation of the object repositoryobject repository from thefrom theparameter fileparameter file utilizingutilizing descriptive programmingdescriptive programmingtechniquestechniques

LegacyLegacy supportsupport -- forfor manualmanualexecution &execution & AUT roll backAUT roll back

VersionVersion controlledcontrolled -- codecode,, driver filesdriver files&& object repositoriesobject repositories

UnifiedUnified reportingreporting -- supportingsupporting realreal--timetimetest execution &test execution &

generic test results output (.xml) to analyze quality of buildsgeneric test results output (.xml) to analyze quality of builds GenericGeneric reusable modules / librariesreusable modules / libraries -- provide reliable processingprovide reliable processing

ofof preamblepreambleandand postamblepostamble actionsactions

Framework Approach


Modularity:Modularity:

Generic Framework ArchitectureGeneric Framework Architecture

Framework Approach



15/15





SSUMMARYUMMARYFramework Approach


HybridHybrid utilising the bestutilising the best technologiestechnologiesandand resourcesresources to do the jobto do the job

KeywordKeyword creating simple &creating simple & robustrobusttest cases written usingtest cases written using businessbusinesslevellevel keywordskeywords combined withcombined with natural languagenatural language

DataData effective use ofeffective use of business databusiness data to provide as anto provide as an input sourceinput source

DrivenDriven processes componentprocesses component actionsactions,, objectsobjects && eventsevents seamlesslyseamlessly

AutomationAutomation that isthat is collaborativecollaborative,, distributeddistributed andand scalablescalable

FrameworksFrameworks independent ofindependent of applicationapplication oror environmentenvironment under testunder test



Automation FrameworksAutomation FrameworksWhat does it mean to you now?What does it mean to you now?

Thanks for your attention!Thanks for your attention!Thanks for your attention!Thanks for your attention!Thanks for your attention!Thanks for your attention!Thanks for your attention!Thanks for your attention!

Documents

Hybrid Keyword Data Driven Automation Frameworks