Service Overview Business Assurance
Digital Transformation, Testing and Automation@paul_gerrard
Intelligent Definition and AssuranceSlide 1
Intelligent Assurance and Testing5-Feb-161
Helping clients transform their testing throughINNOVATION, COACHING and LEADERSHIP
Our CLIENTSWant tobe agilerather than follow Agile dogmaHave apragmatic approachand arefocused on deliveryWant asolution that fits, not a badly fitting suit.Intelligent Definition and AssuranceSlide 2
"Automation is the future!"Heard that before?What exactly is possible and impossible with automation, right here, right now?Where arethe Digital, DevOps and Continuous Deliverycrazes leading us?Where do testers fit?How do testers work in high-paced, automation-dominated environments?Let's look into the near future and discuss how we survive or thrive.
Intelligent Definition and AssuranceSlide 3
AgendaPAC Report on Digital TestingWhat is Digital Anyway?What Goes Wrong with Automation?A Modelling Problem, not a Tools ProblemModels of Testing and AutomationFuture TestingThe Tools Knowledge BaseTomorrow's Challenges.Intelligent Definition and AssuranceSlide 4
PAC Report:Digital Testing: Strategies, Challenges & Measuring Success
http://www.testplant.com/pac-reportSponsored by Accenture, HP, TestPlant, Applause and WorksoftIntelligent Definition and AssuranceSlide 5
Digital is real, important, but only beginning
Intelligent Definition and AssuranceSlide 6
Testing is moving closer to the business
Intelligent Definition and AssuranceSlide 7
Automation levels are still too low
Intelligent Definition and AssuranceSlide 8
What is Digital anyway?Digital is the buzz-phrase of the momentSocial, Mobile, Analytics, Cloud (SMAC)Consumerization of ITBuzzword bingo*Agile laggards (gov, large companies) playing catch-up?S/W development at the pace of marketing.
* https://zoomph.com/blog/top-20-digital-marketing-buzzwords/Intelligent Definition and AssuranceSlide 9
Why is IT moving closer to business?Marketers in charge?Customer Engagement (CE) focus:Real-Time benefits *during* CESocial media is a tool, not a platformCE simplicity is criticalData used proactively to manage CELeverage and embrace customer feedbackAnticipate and act on customer needsMuch more aggressive uses of ITMore 'business-focused' IT Product Owners.
Intelligent Definition and AssuranceSlide 10
Frequent/regular s/w delivery is criticalMobile users expect apps to change almost dailyNew features, offers, opportunities all the timeUsers use the best apps to get what they wantNot necessarily the best supplierBusinesses are in an APPS RACESpeed of delivery is...Partly about pro-actionPartly about survival.Intelligent Definition and AssuranceSlide 11
Assurance, Re-Training, Mentoring5-Feb-16 Aqastra Limited 200811
Automation (not just test) is criticalBusiness wants IT responsiveness (true agility)Not necessarily 100s of releases/dayRapid, regular turnaround from ideas to software deliveryWith continuous integration/deployment, DevOps, developers can now promise Continuous DeliveryTesters need to provide Continuous Assurance"Automation through the (shortened) lifecycle"Intelligent Definition and AssuranceSlide 12But effective test automation success is still hard to achieve.
A recent survey
? - 1997Intelligent Definition and AssuranceSlide 13
Test automation ambitionIntelligent Definition and AssuranceSlide 14
We'll have to make a remarkable improvement to make this happen in one year
What Goes Wrong withTest Execution Automation?
Tools are sensitive instruments
Intelligent Definition and AssuranceYou don't need tools and tools won't help, if your software is unstable.Slide 16
The anti-regression goalWhat are you trying to do with automation?Anti-regression is the primary goalDetect unwanted changeInsurance that developers can make changes safelyHow do you achieve this?The automation model must align with the developer modelYou have to collaborate early to achieve this.Intelligent Definition and AssuranceSlide 17
Testing and automation a modelling problem not a tools problem
Need to shift testing thinking to the left.We model to improve requirements,to eliminate mistakes in code and automation.
Brian Marick's Original Agile Testing Quadrant
Intelligent Definition and AssuranceSlide 19
Crispin & Gregory (Agile Testing version)
Intelligent Definition and AssuranceSlide 20
Assurance, Re-Training, Mentoring5-Feb-16 Aqastra Limited 200820
Intelligent Definition and AssuranceSlide 21
Assurance, Re-Training, Mentoring5-Feb-16 Aqastra Limited 200821
Intelligent Definition and AssuranceSlide 22
What does this tell you?
Are these models useful and still relevant?
Intelligent Definition and AssuranceSlide 26
Questions arisingAutomation is a challengeWe haven't been able to implement tools as effectively as we could in the pastWhat are the prospects of getting better at test automation?What are the alternatives?Is there a cultural challenge too?Will testing and testers be the barrier to success in digital?Intelligent Definition and AssuranceSlide 27
Testers and DevOpsHow do Testers fit into a DevOps regime?Intelligent Definition and AssuranceSlide 28
Tests, Automation and TrustCan we trust automation to do all our testing?Consider four areas:Checks that can be automated by developersChecks that can be automated (typically by system testers) to exercise API-level, end-to-endCompatibility checks across browsers, operating systems, platforms.Tests that can only be performed by a humanCan you 'let go' of late, manual checking?It requires proactive effort and trust?Intelligent Definition and AssuranceSlide 29
How to carve up testing?(AKA Test Strategy)Push testing to the developersMinimal automated end-to-end/UI testsManual-only tests during development?Story-boarding, BDD-style prototyping?UI checks performed on mock-ups/wire-frames?Checks that run once, manually, and never again?Which to retain for regression/automation?Intelligent Definition and AssuranceSlide 30
Easier to Trust if you're a visionaryIs your concern that you can't trust the developers?Shifting test thinking to the left reduces doubt?Testers as pathfinders: identify/assess risks, formulate tests; ensure they are included in automation etc.You are not the gatekeeper of quality, the last defence, the only person who caresThink more like a visionary, risk-identifier, risk manager, pathfinder, a facilitator and a coach/mentor.Intelligent Definition and AssuranceSlide 31
Monitoring and ImprovementA key discipline of DevOps is a deeper level of monitoring in productionAlerts on failure before users experience their impactAmbitious, but this is the ultimate goalWhen problems are encountered, use the analyticsTo trace the cause and resolve itTo refine the test process ("let's adapt so we don't let that bug through again")Automated tests in the process are 'monitoring' tooDevOps process monitors enlarge the scope of testing.Intelligent Definition and AssuranceSlide 32
New Model Testing
Intelligent Definition and AssuranceThe LeftThe RightSlide 33BTW, this is where the tools do some checkinghttp://dev.sp.qa/download/newModel
Left and Right
Automation wont work if the thinking to the left is not in placeThinking on the left contributes to the definition of features and how you test themIf you arent involved on the left, you arent in a good position to test (with or without tools)Models on the left are the blueprints for the tests & automation on the rightYour opportunity to flush out muddled, loose, incomplete, flaky requirements is on the leftApplying automation to muddled, loose, incomplete, flaky features is doomed before you startIntelligent Definition and AssuranceThe LeftThe RightSlide 34
Shift-Left (thinking, not people)If you are not contributing on the left, you are doomed to be a victim of muddled, incomplete thinkingChallenge requirements thru example and improve themShare your models!Usage models: (test through the UI) must be meaningful to usersService models: (services/APIs) meaningful to developers and align with business rulesIn case of test/automation difficulty go to 1.
Intelligent Definition and AssuranceSlide 35
Assurance, Re-Training, Mentoring5-Feb-16 Aqastra Limited 200835
Your homework for todayAre tools just drones that can only execute regression testing?Could tools find functional bugs in untested software?If I can think of a test, can I automate the application of it?Think of tools as extensionsof your mind, not robots.Intelligent Definition and AssuranceSlide 36This is where the tools play their part
TheTools Knowledge Basehttps://tkbase.com
How many tools do you use?NewRelic - Application monitoring - gives us the eyes on our app and how it's being used / performingPaperTrail - Log file collector - brings in log files from various servers to one single place - great for systems running across multiple serversOpsView - Monitoring and alerting tool which we use to bring together monitoring from various systemsNagios - Used for monitoring and alertingPagerDuty - Used to alert (SMS and Email and Phone) when a service craps outElastic Search, Log Stash and Kibana - Data analysis and monitoring and trending - super powerful for analyzing what our product is actually doingChef - Auto build and deploy technology to allow us to rapidly build and destroy environments (with Chef Kitchen and Knife)Vagrant - Create Virtual EnvironmentsReal Time Board - Virtual Whiteboard - amazingly usefulPivota