Embed Size (px)
Citation preview
Naviga&ng(Test(Automa&on(Pi2alls(
Sail(Or(Fail?(
Who$Is$This$Guy?$
! Paul$Grizzaffi$! Automa6on$Manager$and$Program$Architect$–$MedAssets$! Quality$Accelerator$! Career$focused$on$automa6on$strategy,$tools,$frameworks$! Telecom,$stock$trading,$eDcommerce,$health$care$! STPCon,$ASEE,$DevOps$Live!,$DFW$QAA,$QA$Trailblazers$! Cer6fied$ScrumMaster$
Email$ [email protected]$
LinkedIn$ hSp://www.linkedin.com/in/paulgrizzaffi$
TwiSer$ @pgrizzaffi$
Overview$
! Not$an$indictment$of$your$automa6on$ini6a6ve$! Not$meant$to$discourage$or$overwhelm$! Not$“one$ring$to$rule$them$all”$! Not$a$“how$to”$$
Anti-
Our$Process$Isn’t$Working,$Automa6on$Will$Fix$That!$
Process$Rehabilita6on?$
! No$! Broken$process$==$Broken$automa6on$ini6a6ve$
! Must$know$your$process$and$your$job$! May$have$to$do$things$differently$
! Can$be$a$good$excuse$
Wrong$Coverage,$Wrong$Tool$
! Exis6ng$test$cases…$! Ques6onable$coverage$! Not$“automa6on$ready”$
! This$lead$to$automa6on$with$ques6onable$value$! Need$to$test$on$lowDend,$deployment$hardware$! Purchased$tool$required$too$much$horsepower$! Ini6a6ve$labeled$a$failure$
Test$Scripts$Are$Hard,$Record$and$Playback$$Is$The$Best$Way$To$Go!$
Not$So$Fast…$
Record$and$Playback$
! Has$its$place$! Be$mindful$of$the$cost$of$reDrecording$! Smaller$safety$net$! Valuable$start$point$! Good$for$“disposable”$automa6on$! Consider$tools$that$make$maintenance$easy$! Take$a$layered$approach$
System(Under(Test((SUT)(
Domain(Specific(Languages((DSLs)(
Tool(Abstrac&on(Layer(
Raw(Tools(
Test(Script(Framework(
What$is$this$layering$of$which$you$speak?$
GameStop(
eHCommerce(
Website(
PC(Download(
Website(
PC(Download(
Client(.(.(.(
Mobile(
Loyalty(App(
Domain(Specific(Languages((DSLs)(Game(Stop(
DSL(
PC(Download(
DSL(
Android(
DSL(.(.(.(
RSpec(
Raw(Tools(WebSpec(&(
Selenium(.(.(.(Robo&um(Proprietary(
Mechanize/(
Savon(
Tool(Abstrac&on(Layer(Finders( .(.(.(Auxiliary(APIs(
Test$Cases$ DSL$ Finder$
login_link:$element$=$$$$get_element.by_id(“foo”);$return$element;$$
So$What$
TC400:$login(“ace”,$“passwd”);$do_something();$.$.$.$
TC2:$login(“gene”,$“passwd”);$do_something();$
login(id,$pw):$login_link.click();$id_field$=$id;$password_field$=$pw;$ok_buSon.click();$logout_link.should_exist();$$
TC1:$login(“paul”,$“passwd”);$do_something();$do_something_else();$$
UH(
OH!(
Test$Cases$ DSL$ Finder$
login_link:$element$=$$$$get_element.by_id(“bar”);$return$element;$$
So$What$
TC400:$login(“ace”,$“passwd”);$do_something();$.$.$.$
TC2:$login(“gene”,$“passwd”);$do_something();$
login(id,$pw):$login_link.click();$id_field$=$id;$password_field$=$pw;$ok_buSon.click();$logout_link.should_exist();$$
TC1:$login(“paul”,$“passwd”);$do_something();$do_something_else();$$
UH(
OH!(
Test$Cases$ DSL$ Finder$
login_link:$element$=$$$$get_element.by_id(“bar”);$return$element;$$
So$What$
TC400:$login(“ace”,$“passwd”);$do_something();$.$.$.$
TC2:$login(“gene”,$“passwd”);$do_something();$
login(id,$pw):$login_link.click();$id_field$=$id;$password_field$=$pw;$ok_buSon.click();$logout_link.should_exist();$profile_link.should_exist();$$TC1:$
login(“paul”,$“passwd”);$do_something();$do_something_else();$$
Less$Maintenance!$
Do$I$REALLY$Need$To$Do$All$This?$
! No$! But…it$can$be$a$good$idea…IF…$
! You$don’t$know$your$technology$roadmap$! You$can’t/won’t$be$bound$to$a$single$tool$! No$tool$for$all$crossDtech$needs$! You$can$invest$
But,$without$an$appropriate$QA$and$Automa<on$strategy…$
My$Product$Code$is$in$C#$So$My$Test$Scripts$Must$be$in$C#$
Again,$Not$So$Fast…$
Test$Script$Languages$
! Consider$your$product$–$API?$Messages?$GUI?$! Consider$your$env$–$HW/OS/Accessibility?$! Consider$your$audience$! Same$language$can$leverage$SMEs$! Dynamic$language$can$provide$addi6onal$abstrac6on$for$nonDprogrammer$users$
! Scrip6ng/NonDCompiled$languages$can$have$faster$release$6me$
$Goodbye$$Hello$Ruby$Tuesday$
! Website$is$primarily$C#/.NET$! Test$scripts$wriSen$in$Ruby$! Ruby$has$“rather$friendly”$syntax$! Use$of$JRuby$provides$Ruby$+$Java$! Can$“hand$patch”$! Unit$test$for$tool$is$essen6al$for$corner$cases$! Also$C++$D>$Proprietary,$Java$D>$C++$
We$Must$Automate$Every$Aspect$Of$Our$Tes6ng!$
All$or$Nothing?$
! Any$benefit$is$usually$beSer$than$no$benefit$! Consider$automa6onDassist$! Consider$“helper$tools”$–$turn$the$crank$
! “Bang$for$the$buck”$! “It$demos$well”$
! “Tout$the$Tool”$! “Drug$dealer”$approach$
My$Lovely$Assistant$
! Changing$order$management$systems$(OMS)$! Need$to$pump$in$a$lot$of$orders$! Crea6ng$orders$in$the$browser$was$working$! No$programma6c$access$to$new$OMS$yet$! Pump$orders$through$browser,$verify$OMS$manually$
Aqer$Wri6ng$The$Automated$Test$Scripts,$$We’re$Finished$With$Automa6on,$Right?$
Care$and$Feeding$
! Con6nuing$investment$for$con6nuing$return$! Product$evolu6on$==$automa6on$evolu6on$! Consider$layering$your$framework$! Account$for$“downstream$maintenance”$
! Tool$upkeep$! Tool$upgrades$! License$and$support$fees$! Script$parity$with$product$
Feed$a$Cold,$Starve$an$OS$Upgrade$
! Belt$6ghtening$leads$to$automa6on$scru6ny$! No$approval$for$OS$upgrade$because$no$perceived$business$case$
! Major$product$funder$needs$3$new$tool$server$installa6ons$
! Current$version$of$the$OS$does$not$support$new$hardware$
! Automa6on$team$scrambles,$late$to$deliver$
This$Automa6on$Tool$Is$What$We$MUST$Use$For$All$Of$Our$Tes6ng$
Hammer$and$Nails$
! Consider$the$“hammer$and$nails”$analogy$! Right$tool$for$the$job,$consider$your$audience$! Hit$all$the$nails$! Use$nails$instead$of$screws$! Layers$for$tool$integra6on$and$crossDtech$tests$! Avoid$“3$of$clubs$mentality”$
Hisng$the$Nail$on$the$Head$
• Lucky$enough$to$start$with$greenDfield$• Ini6al$framework$release$focused$on$browser$• Using$browser$for$downstream$was$wasteful$• Chose$to$maximize$browserDbased$scenarios$• Integrated$downstream$later$
Automa6on$Is$A$QA$Job$
Whose$Job$is$this$Anyway?$
! Works$best$as$a$crossDfunc6onal$team$effort$! QA$+$Dev$+$Automa6on$+$OPs$+$RM$+$Security…$! Some$things$are$easier$to$automate$! Some$things$can’t/shouldn’t$be$automated$! Early$QA/Automa6on$par6cipa6on$! Feedback$loop$to$risk/reward$decisions$
A$Tale$of$Two$Ci6es$Teams$
! Ini6al$tool$release$for$eDCommerce$team$! Developed$automa6on$for$IE$and$Safari$! Safari$automa6on$can’t$click$all$clickables$! Not$used$to$“automa6on$as$a$team$effort”$! Next$website$release$had$new$nonDclickables$! Increase$in$teeth$gnashing$and$cloth$rending$
! Ini6al$tool$release$for$PC$download$client$! Mee6ngs$with$Dev,$QA,$and$Automa6on$! Early$access$to$prototype$technology$! Recep6ve$to$design$changes$for$automa6on$! And$a$good$6me$was$had$by$all$
Automa6on$Will$Fix$All$Of$Our$Quality$Problems!$
Magic$Bullet$
! No$magic$bullet$! BUT…automa6on$is$a$BIG$bullet$! Generally,$confirms$behavior,$doesn’t$“find$bugs”$! Automa6on$for$wellDdefined,$repeatable$tests$! Humans$for$“guerilla$tes6ng”$! Don’t$automate$if$no$value$
Let’s$Make$a$Deal$(of$the$Day)$
! PC$download$client$“Deal$of$the$Day”$feature$! Pop$up$window$alerts$user$of$the$deal$! Changes$daily$! Essen6ally$a$clickable$image$! Hard$to$know…$
! Which$product$! Which$image$! Which$des6na6on$
! Possible$heuris6cs,$but$likely$false$posi6ve/nega6ve$! S6ll$needs$to$verified$manually$! No$value$in$automa6ng$
Takeaways$
! Applies$to$build,$buy,$and$“steal”$! Beware$incorrect$expecta6ons$! Make$business$decisions$with$business$data$! Must$be$in$sync$with$longDterm$goals$! Something’s$beSer$than$nothing$! “Don’t$swat$a$fly$with$a$Buick”$! Think$solu6ons$and$strategies,$not$just$tools$! Don’t$forget$about$maintenance$
open source
Implementa6on$leq$to$the$reader$as$an$exercise$
Q&A$
Email$ [email protected]$LinkedIn$ hSp://www.linkedin.com/in/
paulgrizzaffi$TwiSer$ @pgrizzaffi$
Code$Comparison$
it "should login" do @browser.a.className(“ats-login”).click @browser.input.with("id.search(/(emailAddress|loginCtlUser)/)!= -1").value = “[email protected]” @browser.input.id("/loginControl_password/").value = “mypw" @browser.input.name("/loginButton/").click @browser.li.className("loginlink").find.a.should_exist @browser.a.id("/LogOut/").should_exist end
it "should login with DSL" do @browser.log_in_link.click @browser.log_in(“[email protected]”, “mypw”) end
it "should login with finders" do @browser.log_in_link.click @browser.email_field.value = “[email protected]” @browser.password_field.value = “mypw" @browser.log_in_button.click @browser.ensure_header_loaded @browser.wait_for_landing_page_load @browser.log_out_link.should_exist end
