Digital Transformation, Testing and Automation

Digital Transformation, Testing and Automation

@paul_gerrard

Paul [email protected]

om

gerrardconsulting.com Intelligent Definition and Assurance Slide 1

http://testaxioms.com/?q=node/25

Helping clients transform their testing through

INNOVATION, COACHING and LEADERSHIP

Our CLIENTS– Want to be agile rather than follow Agile

dogma– Have a pragmatic approach and are focused on

delivery– Want a solution that fits, not a badly fitting

suit.

Intelligent Definition and Assurance Slide 2

"Automation is the future!"• Heard that before?• What exactly is possible and impossible with

automation, right here, right now?• Where are the Digital, DevOps and

Continuous Delivery crazes leading us?• Where do testers fit?• How do testers work in high-paced,

automation-dominated environments?• Let's look into the near future and discuss

how we survive or thrive.


Agenda• PAC Report on Digital Testing• What is Digital Anyway?• What Goes Wrong with Automation?• A Modelling Problem, not a Tools Problem• Models of Testing and Automation• Future Testing• The Tools Knowledge Base• Tomorrow's Challenges.


PAC Report:Digital Testing: Strategies, Challenges & Measuring Success

http://www.testplant.com/pac-reportSponsored by Accenture, HP, TestPlant, Applause and Worksoft


http://www.testplant.com/pac-report

http://www.testplant.com/pac-report

Digital is real, important, but only beginning


Testing is moving closer to the business


Automation levels are still too low


What is Digital anyway?• Digital is the buzz-phrase of the moment– Social, Mobile, Analytics, Cloud (SMAC)– Consumerization of IT– Buzzword bingo*

• Agile laggards (gov, large companies) playing catch-up?

• S/W development at the pace of marketing.

* https://zoomph.com/blog/top-20-digital-marketing-buzzwords/


Why is ‘IT moving closer to business’?• Marketers in charge?• Customer Engagement (CE) focus:– Real-Time benefits *during* CE– Social media is a tool, not a platform– CE simplicity is critical– Data used proactively to manage CE– Leverage and embrace customer feedback– Anticipate and act on customer needs

• Much more aggressive uses of IT• More 'business-focused' IT Product Owners.


Frequent/regular s/w delivery is critical• Mobile users expect apps to change almost

daily– New features, offers, opportunities all the time

• Users use the best apps to get what they want– Not necessarily the best supplier

• Businesses are in an APPS RACE• Speed of delivery is...– Partly about pro-action– Partly about survival.


Automation (not just test) is critical• Business wants IT responsiveness (true agility)– Not necessarily 100s of releases/day– Rapid, regular turnaround from ideas to software

delivery• With continuous integration/deployment,

DevOps, developers can now promise Continuous Delivery

• Testers need to provide Continuous Assurance• "Automation through the (shortened) lifecycle"


• But effective test automation success is still hard to achieve.

A recent survey? - 1997


Test automation ambition


We'll have to make a remarkable improvement to make this happen in one year

What Goes Wrong with

Test Execution Automation?

Tools are sensitive instruments

Intelligent Definition and Assurance

You don't need tools and tools won't help, if your software is unstable.

Slide 16

The anti-regression goal• What are you trying to do with automation?• Anti-regression is the primary goal– Detect unwanted change– Insurance that developers can make changes

safely• How do you achieve this?– The automation model must align with the

developer model– You have to collaborate early to achieve

this.


Testing and automation – a

modelling problem not a tools problemNeed to shift testing thinking to the

left.We model to improve requirements,to eliminate mistakes in code and

automation.

Brian Marick's Original Agile Testing Quadrant


Crispin & Gregory (Agile Testing version)




What does this tell you?

Are these models useful and still

relevant?


Questions arising• Automation is a challenge• We haven't been able to implement tools

as effectively as we could in the past• What are the prospects of getting better

at test automation?• What are the alternatives?• Is there a cultural challenge too?• Will testing and testers be the barrier to

success in digital?Intelligent Definition and Assurance Slide 27

Testers and DevOps

How do Testers fit into a DevOps regime?


Tests, Automation and Trust• Can we trust automation to do all our testing?• Consider four areas:

1. Checks that can be automated by developers2. Checks that can be automated (typically by

system testers) to exercise API-level, end-to-end3. Compatibility checks across browsers, operating

systems, platforms.4. Tests that can only be performed by a human

• Can you 'let go' of late, manual checking?• It requires proactive effort and trust?



How to carve up testing?(AKA Test Strategy)• Push testing to the developers• Minimal automated end-to-end/UI tests• Manual-only tests during development?• Story-boarding, BDD-style prototyping?• UI checks performed on mock-ups/wire-

frames?• Checks that run once, manually, and

never again?• Which to retain for regression/automation?

Slide 30


Easier to Trust if you're a visionary• Is your concern that you can't trust the

developers?• Shifting test thinking to the left reduces doubt?• Testers as pathfinders: identify/assess risks,

formulate tests; ensure they are included in automation etc.

• You are not the gatekeeper of quality, the last defence, the only person who cares

• Think more like a visionary, risk-identifier, risk manager, pathfinder, a facilitator and a coach/mentor.

Slide 31

Monitoring and Improvement• A key discipline of DevOps is a deeper level of

monitoring in production– Alerts on failure before users experience their impact– Ambitious, but this is the ultimate goal

• When problems are encountered, use the analytics– To trace the cause and resolve it– To refine the test process ("let's adapt so we don't let that

bug through again")• Automated tests in the process are 'monitoring' too• DevOps process monitors enlarge the scope of testing.


New Model Testing


The Left The Right

Slide 33

BTW, this is where the tools do some

checking

http://dev.sp.qa/download/newModel

Left and Right

Automation won’t work if the thinking to the left is not in place

Thinking on the left contributes to the definition of features and how you test them

If you aren’t involved on the left, you aren’t in a good position to test (with or without tools)

Models on the left are the blueprints for the

tests & automation on the right

Your opportunity to flush out muddled, loose, incomplete, flaky requirements is on the left

Applying automation to muddled, loose, incomplete, flaky features is doomed before you start


The Left The Right

Slide 34


Shift-Left (thinking, not people)1. If you are not contributing on the left, you are

doomed to be a victim of muddled, incomplete thinking

2. Challenge requirements thru example and improve them

3. Share your models!– Usage models: (test through the UI) must be

meaningful to users– Service models: (services/APIs) meaningful to

developers and align with business rules4. In case of test/automation ‘difficulty’ – go to 1.

Slide 35

Your homework for today• Are tools just drones that can only

execute regression testing?• Could tools find functional bugs in

untested software?• If I can think of a test, can I automate

the application of it?• Think of tools as extensions

of your mind, not robots.


This is where the tools play their part

TheTools Knowledge

Basehttps://tkbase.com

How many tools do you use?NewRelic - Application monitoring - gives us the eyes on our app and how it's being used / performingPaperTrail - Log file collector - brings in log files from various servers to one single place - great for systems running across multiple serversOpsView - Monitoring and alerting tool which we use to bring together monitoring from various systemsNagios - Used for monitoring and alertingPagerDuty - Used to alert (SMS and Email and Phone) when a service craps outElastic Search, Log Stash and Kibana - Data analysis and monitoring and trending - super powerful for analyzing what our product is actually doingChef - Auto build and deploy technology to allow us to rapidly build and destroy environments (with Chef Kitchen and Knife)Vagrant - Create Virtual EnvironmentsReal Time Board - Virtual Whiteboard - amazingly usefulPivotal Tracker - Agile tracking toolFiddler - Proxy web toolFirebug - Proxy web toolZed Attack Proxy - Security testing toolBurpsuite - Security Testing ToolHipChat - Real Time IM communication toolSlack - Real Time IM Communication toolJira and Confluence - bug tracking and wikiCloudFormations - Creates templates for Amazon instances

"No doubt we have some more hiding away but that's a pretty good list."

23tools


302 bloggers, 16,064 blogposts indexed and searchable (35-40 new posts daily)

2,336 tools, indexed and searchableWork in progress: tools tagged with

297 features

Challenges for the next few years• We have plenty tools that can run

thousands and millions of tests; what we need now are tools that support modelling and test design

• How do we acquire a coherent, integrated tool-kit with 15-30 tools?

• How do we fit testing into DevOps/CD?• Testing role absorbed shift-left or –right?• Where does 'manual' testing fit now?


Thank-You!

@paul_gerrard

Paul Gerrardemail: [email protected]

Twitter: @paul_gerrardLinkedIn: http://linkedin.com/in/paulgerrard

tkbase.com | testopera.com


mailto:[email protected]

http://linkedin.com/in/paulgerrard

http://linkedin.com/in/paulgerrard

Software

Digital Transformation, Testing and Automation