Upload
hitachi-consulting
View
4.002
Download
1
Embed Size (px)
DESCRIPTION
What is the future of test automation? The possibilities associated with hybrid cloud computing provide instant scalability, flexibility and availability for testing on demand with no upfront investment. This provides the industry with a perfect opportunity to utilise powerful test automation platform solutions. The global testing cloud marketplace will allow for the joint collaboration of leading test specialists following industry best practice. This enables firms of all sizes to access the latest test approaches and methodologies whilst providing a unified platform for domain experts to represent business processes and user story acceptance criteria in a natural language.
Citation preview
Jonathon Wright ([email protected])
http://www.linkedin.com/in/automation
BCS SIGIST Winter 2013 Conference (Thursday @ 2:45 PM, 5th December 2013)
Test Automation in the (Hybrid) Cloud
@jonathon_wright#BCS_SIGIST
www.sigist.org.uk
My starting point: Founder of the (Hybrid) Approach Pattern (2003)
@jonathon_wright#BCS_SIGIST
@jonathon_wright#BCS_SIGIST
The current economic climate is making companies
review their approach to IT even more closely
“We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant
scalability, flexibility, and availability for testing-on-demand. - But how well does this work in practice?”
• Why (Business Benefits & Value)
• What (Cloud Test Automation)
• When (Timeline)
• Agile Portfolio Management (Scope)
• Transition (AMMi)
• Dependencies (Cloud Type)
• Design (First Day Testing)
• Design (Manual’mation)
• Implementation (Agnostic)
• Cloud Platform (IaaS)
• High Volume Automation (HVAT)
• Continuous Build & Delivery
• Actionable Insight (Funnel Virtualisation)
• Dashboards (Real-time)
TESTINGAS A SERVICE
(TaaS)
INFRASTRUCTURE AS A SERVICE
(IaaS)
SOFTWARE AS A SERVICE(SaaS)
REAL-TIME VISIBILITYAND CONTROL
INSTANT SCALABILITYSCALE UP AND DOWN
HIGH AVAILABILITY
AMMI
PLATFORM AS A SERVICE(PaaS)
ISO 29119
PERFORMANCE TESTINGAS A SERVICE
(PTaaS)
PAY AS YOU USE(PAYU)
SLA’s
Background – The Journey to the Cloud
TEST AUTOMATIONAS A SERVICE
(TAaaS)
Background – Three Decades in Automation – Mid-Late 90s
@jonathon_wright#BCS_SIGIST Product Test Engineer
Gen Zero
(Linear)AUTSUT
Budget in pounds, company in thousands
MainframeLocations:2Teams:5Tool(s): 1
License x1
Background – Three Decades in Automation – Late 90s
@jonathon_wright#BCS_SIGIST Product Test Engineer
1st Gen
(Modular)AUTSUT
Locations:5Teams:10 Client ServerTool(s): 3
Licenses 10+
http://www.google.co.in/patents/US7526498
Budget in thousands, company in millions
Automation Framework ArchitectCore Hours 1am to 10am (EST)
Graduate TestersCore Hours 9am to 5pm (EST)
Automation EngineersCore Hours 9am to 5pm (GMT)
Background – Three Decades in Automation – Early 00s
@jonathon_wright#BCS_SIGIST Automation Consultant
2nd Gen
(Data)AUTSUT
Locations:5Teams:15 Client ServerTool(s): 5
Budget in hundreds of thousands
Licenses x25
Automation Consultants (UK)Core Hours 9am to 5pm (GMT)
Development (2 week sprints)Core Hours 9am to 5pm (GMT+1)
Background – Three Decades in Automation – Late 00s
@jonathon_wright#BCS_SIGIST
Automation Framework Architect
3rd Gen
(Keyword)SUT
Locations:20+Teams:100+Tool(s): 15+
Budget in millions, company in billions
Web
AUT
Automation Consultants (UK)Core Hours 9am to 5pm (GMT)
Automation Consultants (NY)Core Hours 9am to 5pm (EST)
Automation Consultants 9am to 5pm (UTC+8)
Development Capability (US)Core Hours 9am to 5pm
Development Capability (EUR)Core Hours 9am to 5pm
Licenses 100+
Background – Three Decades in Automation – Early 10s
@jonathon_wright#BCS_SIGIST Global Head of Automation
4th Gen
(Hybrid)
Locations:25+Teams:50+Tool(s): 20+
Budget in millions, company in billions
AUTSUT
Cloud
Automation Consultants (NY)Core Hours 9am to 5pm (EST)
Automation Consultants (UK)Core Hours 9am to 5pm (GMT)
Automation Consultants 9am to 5pm (UTC+8)
Development Capability (EUR)Core Hours 9am to 5pm
Licenses PAYU
Background – Three Decades in Automation – Today
@jonathon_wright#BCS_SIGIST Agile Portfolio Management
5th Gen
(TaaaS)
Locations:100sTeams:1k+Tool(s): 50+
Budget 78 million, company investment 4 billion
Cloud
SUT AUT
Automation Consultants (NY)Core Hours 9am to 5pm (EST)
Automation Consultants 9am to 5pm (UTC+8)
wA
gile
Team
s (A
sia)
Co
re H
ou
rs 9
am t
o 5
pm
(U
TC+8
)
wAgile Teams (US)Core Hours 9am to 5pm (EST)
wAgile Teams (EUR)Core Hours 9am to 5pm (CET)
Automation Consultants (UK)Core Hours 9am to 5pm (GMT)
Background: Where's testers? – BCS SIGIST Winter 2011 Conference
Time Zone 1 Time Zone 2 Time Zone 3
Community Tester - Platform: Hybrid Community Cloud (e.g. MyDb.com)
Beta Tester - Platform: Public Cloud (e.g. sdod.db.com)
Alpha Tester - Platform: Public/Private Cloud (e.g. dbras.db.com)
Cloud Tester - Platform: Private Cloud (e.g. vpn.internal.db.com)
Internal Tester - Platform: Private Cloud (e.g. dod.internal.db.com)
Testing Command Center (Domain Specific)
• Supporting Core Testing Activities (Environment Monitoring/Management)
•Defect Monitoring & Risk Assessment (Social Enterprise Activity Feeds)
•Testing Management (Test Assets & Resource Enablement)
Solution Command Center (Solution Specific)
• End Client Dashboard/Reporting (KPIs)
• Solution Management (Programme Level)
•Service Provider Management (SLAs/Who does what when)
Background – Testing as a Service – BCS SIGIST Winter 2011 Conference
@jonathon_wright#BCS_SIGIST
“Existing technologies of virtualisation, business process modelling, cloud based test automation tools and
rapid and easy internet access allow for the development of approaches that allow companies to order testing
as a service and pay only for what they use.”
Background – Test Automation as a Service
@jonathon_wright#BCS_SIGIST
“Over the three years we have been developing a azure cloud based solution to provide true
"Test Automaton as a Service" using a hybrid automation approach specifically designed with the Hybrid
Cloud (Private/Public) in mind to utilise the instant scalability, flexibility and availability of the Cloud.”
www.TaaaS.net
www.sigist.org.uk@jonathon_wright#BCS_SIGIST
Background – Test Automation as a Service
“The current implementation is a service-oriented architecture (SOA) based on utilising business process
modelling to create business level keywords written in natural language (domain specific languages)
validated against content sensitive validation to create reusable, jigsaw pieces representing the Solution
Under Test (SUT).”
www.slideshare.net/Jonathon_Wright
platform, technology, client, browser, version, language, test type agnostic
Executing Cloud Tests
feat. VM Dispenser Technology
Cloud Test Environmentsnatural language validated against
content sensitive validation
Creating Cloud Tests
Background – Test Automation as a Service
service-oriented architectureBorn in the Cloud
@jonathon_wright#BCS_SIGIST
Relevant Clear traceability of the business value of Automation through the visualisation
of the tests via Business Process Modelling (BPMNv2.2 compliant)
Effective Self-validating test assets achieved using natural language with context sensitive
validation against business and testing rules, workflows and data
Maintainable Self-maintaining test asset loader/scraper
Efficient Reports on SUT health including ratings such as percentage availability since
build/release, reported errors over time and traffic to error ratio
Manageable Unified platform which non-domain experts can use a natural language to
represent business processes and user story acceptance criteria
Portable
Technology agonistic - Platform, client, component, browser, version &
language
Test type agnostic – smoke, regression, integration & performance
Reliable Fault tolerance is built in to report and continue on different levels of fuzzy
matching combined with the non-technology specific test definition language
Diagnosable
Actionable defects provided by environment under test (EUT) live pause-
playback supported by dynamic data adapters (DDA) for accelerated defect
investigation and resolution
Source: The Big Picture of Test Automation: Test Trustworthiness – Alan Page, Microsoft (2012)
Summary - Test Automation as a Service (TaaaS.net)
@jonathon_wright#BCS_SIGIST
www.sigist.org.uk
RIP Traditional Automation
Frameworks (1993-2013)
Panel Discussion - Should software testers be able to code?
@jonathon_wright#BCS_SIGIST
“Here lies beloved architects, engineers and developers who invested a disproportional amount of time and
effort searching for the holy grail of test automation. Careers ended suddenly after a portfolio test manager
asked them to provide the actual business value of what they were doing.”
www.sigist.org.uk
Q1) How can we move to ‘Test Automation in the Cloud’?
@jonathon_wright#BCS_SIGIST
“Existing technologies of virtualisation, business process modelling, cloud based test automation solutions and
rapid and easy internet access allow for the development of approaches that allow companies
to order ‘Testing as a Service’ and pay only for what they use (PAYU).”
Automation Ready
Cloud Power
Test Automation in the Cloud
Where are we going next?@jonathon_wright#BCS_SIGIST
Example Automation Projects Automation Level Investment in Effort Business Benefit
Description AMMi Gen Implementation Planning Execution Accel. ROE
Client Specific Implementation - Hybrid Approach 4 5 24 63 17 6.9 112%
Client Specific Implementation - BDD Approach 3 4 30 83 22 5.2 85%
Client Specific Implementation - Traditional 1 1 120 250 67 1.7 26%
Testing As A Service Model – Manual’mation 3 1 28 5 25 4.6 200%
Testing As A Service Model - Hybrid Approach 4 5 12 31 19 6.1 186%
TaaaS - Cloud Based Implementation 5 5 5.6 25 7 16.0 306%
www.sigist.org.uk
Q2) Are we ready for ‘Test Automation in the Cloud’?
Automation Maturity Model index (AMMi®)Assessment to provide a current automation maturity level and the target maturity level:
* Source: AMMi®(http://www.AMMi.org.uk) + Test Automation Patterns (http://TestAutomationPatterns.wikispaces.com)
AMMi Phase Target Criteria
Level 0 Accelerating Automated Test Lifecycle Methodology (ATLM)
Level 1 Traditional Framework Driven (Gen 1-5)
Level 2 Managed Processes are planned, performed, measured, and controlled
Level 3 Sustaining Automation process defined and validated against international
standards (ISO-29119)
Level 4 Quantified Predictability of automation process performance
Level 5 Optimising Automation process variation and statistical predictability
@jonathon_wright#BCS_SIGIST
“Sometimes half the battle just starting companies on their automation journey by taking them one step
closer to becoming ready for automation.” AMMi Foundation 2003
Q3) – Which ‘Automation Tool Platform’
www.sigist.org.uk@jonathon_wright#BCS_SIGIST
“My current role at one of the worlds largest investment banks. Their landscape consisted of over 300+
solutions under test (SUT), representing 2,700+ applications under test (AUT),
across 50+ global locations with over 300 automation practitioners worldwide.”
Q3) – Which ‘Automation Platform’ Axe 4 (Beta2)
@jonathon_wright#BCS_SIGIST
A1
Login.Process
Request.VM
Access.VM
Manage.VM
Logout.Process
A3
A2
C1C3
D2C2D1
C4
E2D4 E1
E3 E4
D3
Business Process
Scenarios (BPS)
Business Process
Data (BPD)
Business Process
Modelling (BPM)
Business Process
Test (BPT)
Quick Overview - ‘First Day Automation’ Approach
@jonathon_wright#BCS_SIGIST www.sigist.org.uk
Moment in time
Start Cloud Tester
Modelling Tool (BizAgi or Visio)
Static Source
Dynamic Source
Cloud Source
Direct Source (xPDL or XAML)
Test Asset Loader(Test Asset Cube)
Business Process Modelling
BPT Tool(UFT or Specification)
Business Process Tests
Test Scheduler (WCF or REST)
TaaaS.net(Test Designer)
Q4) How do we become ‘First Day Automation’ Ready?
Agile Portfolio Management – BPM Export (Visio)
@jonathon_wright#BCS_SIGIST
Agile Portfolio Management – Business Process Modelling
@jonathon_wright#BCS_SIGIST
Solution Lifecycle Management - Business Process Scenarios
@jonathon_wright#BCS_SIGIST
BPS Tool
(BizAgi or ALM/QC)
Direct Source (Workflow or XML)
Business Process Scenarios
Hyper-Test.com(Test Asset Journals)
www.eurostarconferences.com
Solution Lifecycle Management – Business Process Scenarios
HP Quality Centre (10+) / Application Lifecycle Management (11.5)@jonathon_wright#BCS_SIGIST
Solution Lifecycle Management – Business Process Testing
@jonathon_wright#BCS_SIGIST
“Everyday business and consumer life creates 2.5 quintillion bytes of data per day and 90% of the data in the
world today has been created in the last two years?” with 1.5 petabytes of data across the current landscape.
Development SD-PM SD-TEST SD-AUTOMATIONVendors
Tier 3 - Sub-Domains – Application Level (x2,738)
Tier 2 - Domains – Solution Level (x330)
Domain Leads Domain StakeholdersDomain PMDomain Test Managers
Tier 0 – Programme Level (x4)
Heads of TestingHead of Automation Global Technology Office
Tier 1 - Workstream – Portfolio Level (x8)
Portfolio Test Managers Workstream StakeholdersWorkstream LeadsAPM Products Organisational
Agile
Portfolio
ReportingHybrid Cloud
Distributed AgileProject ArtefactsALM ProductsLocal Network
APM Products Global-widePublic Cloud Executive (Actionable Insight)
SLM Products Enterprise
AgilePrivate Cloud Enterprise
Reporting
Ch
art
er
Q5) What is ‘Agile Portfolio Management’?
Blu
ep
rin
tK
ey
Op
era
tin
g
Pro
ced
ure
s
Dyn
am
ic
Ag
ile
Pro
cess
es
Q6) How to provide ‘Actionable Insight / Automation Value’
Testing as a Service (TaaS) Dashboard (Realtime)
Risk Dashboard(Presentation Layer)
Benefit Analysis(Presentation Layer)
Story Board(Presentation Layer)
Business Risk
Project Risk
Technical Risk
Business Domain Language
Test Domain Language
Business Stories
Business Workflow
Business Rules
Business Process Design (BPD)(Visio Design Documentation)
Non-Functional Requirements (NFR)
(Dynamic Data Source (XML))
Detailed Session Definition (DSD)
(Dynamic Data Source (XAML))
Detailed Profile Definition (DPD)
(Dynamic Data Source (XAML))
Detailed Test Definition (DTD)
(Dynamic Data Source (XAML))
Business Components & Interfaces
(Dynamic Data Source (XAML))
Business Process Modelling (BPM)
(BPMNv2)
Business Process Scenarios (BPS)
(eXensible Process Definition Language)
Business Process Tests (BPT)
(Dynamic Data Source (XAML))
Business/Test Transactions Layer
Solution Under Test (SUT) Layer
Dynamic Data Adapters Dynamic Test Adapters
Business Questions
@jonathon_wright#BCS_SIGIST www.sigist.org.uk
Dashboards (Real-time) providing ‘Actionable Insight’
Theme: Travel@jonathon_wright#BCS_SIGIST
Automation Ready
Cloud Power
Test Automation in the Cloud
“To the cloud”@jonathon_wright#BCS_SIGIST
www.sigist.org.uk
Q7) Should we move to the cloud?
New Challenges & Opportunities ? and/or
A match made in heaven?
@jonathon_wright#BCS_SIGIST
@jonathon_wright#BCS_SIGIST
Co
nn
ecti
on
Systemic Failure
(Network or DNS)
Alternative Service Provider?
New Challenges
What happens when these services FAIL?
Unique Services
(Such as Banks)
"There's normally a role that technology plays in
the outage, but when I look at the root cause, by
far the greatest cause is people and processes."
Damian Saunders, a cloud networking director
“For decades, RBS failed to invest properly in its
systems” Ross McEwan, Chief Executive, RBS, December 3rd 2013
Diagram Source - The Federal Reserve Bank of New York, Shadow Banking System, 2012 (http://www.newyorkfed.org/research/staff_reports/sr458.pdf)
“Don’t assume the ecosystems (commercial or community) and
other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)
Consumer Freedom
www.sigist.org.uk@jonathon_wright#BCS_SIGIST
“Enable sophisticated modelling of the components of cloud deployments,
including servers, storage, databases and data.
For the consumption of cloud resources” - PlanForCloud.com (2012)
CompareTesting.com
Instant scalability
Flexible Test Execution
Flexible Test Assets Environments on-demand
Geo-based execution
Instant Accessibility
99.9% Availability (100% Web)
Instant Portability
Best of Breed (BoB) tools on demand
No upfront investment
Pay As You Use (PAYU)
New Opportunities
www.sigist.org.uk
@jonathon_wright#BCS_SIGIST
PrivateCombination
Public
SUT SUTSUT
Community
Clouds
Public
Clouds
Private
Clouds
TaaaS.net TaaaS.net
“... the prediction that 60% of the average enterprise will have 60% of its applications in the cloud.”
- Karl Flinders, Computer Weekly (2011)
“IBM are working with other cloud suppliers to establish the standards for cloud service provider compliance.”
- Murat Gunenc, IBM Cloud Representative @Rhodes W1 (2011)
Q8) Cloud Type
Automation Ready
Cloud Power
Test Automation in the Cloud
The Final Frontier@jonathon_wright#BCS_SIGIST
HybridCloud
PrivateCloud
Storage
Networking
Servers
Virtualization
Applications
Infrastructure(as a Service)
Runtimes
Databases
Continuous Integration,
Build & Deployment
PublicCloud
CommunityCloud
Q9) How can we move to ‘Test Automation in the Cloud’?
Lab Management
Test Controller(s)
Test Agent(s)
High Volume
Automated Testing (HVAT)
www.eurostarconferences.com
Test Agent(s) – Telerik Art of Testing Framework
@jonathon_wright#BCS_SIGIST
High Volume Mobile Automated Testing (HVMAT)
@jonathon_wright#BCS_SIGIST
High Volume Mobile
Automated Testing (HVMAT)
VMM Library
Data Protection
VM Factory
Data Masking
Data
Co
ntr
oller
–(C
om
plian
ce w
ith
DPA
)
Data
Pro
cess
or
–(W
ritt
en
leg
al co
ntr
act
)
Data
Pro
vid
er
–(E
uro
pean
Eco
no
mic
Are
a)
Feature(s) Work ID
Code
Team Foundation Service
TFS - Gated Check-in!
CancelOK
Build Server
Backlog
VS2013 x64 Cross Tools Command
Microsoft Windows [Version 8.1.9431]C:\Automation\>mstest /testlist:MasterDriver/testmetadata:”TestProject.vsmdi” /testsettings:”MobileDevices.runsettings”
Unit Tests
Feature-Driven Exploratory Test Case(s)
Issue(s) Action(s) Log ScreenCaptureIntelliTrace
Traffic Proxy
Continuous Deployment Test Controller(s) Test Agent(s) Browser(s)
Client(s)Load Agent(s)
Interface(s)
Solution Under Test
Application Under Test
ASP.net | DynoForms (Ajax/Json/Spoc)
Application Server(s)
Database Server(s) Shim/Stub(s)
Shim/Stub(s)
Business Process Scenarios (BPS)Business Process Tests (BPT) Business Process Model (BPM)
Business Process Data (BPD)Business Process Reporting (BPR)
Q10) How can we execute ‘Test Automation in the Cloud’?
Solution Lifecycle Management – Business Process Tests
“verbs and nouns in the test specific language are written in natural language which using context sensitive validation.”
Logi
n.P
roce
ss
Action Object Event
Enter Username Text
Enter Password Text
Press Login Button
Domain Specific Language
Act
ion
s
Ob
ject
s
Even
ts
Press | Login | Button
Nouns
Verbs
Test Specific Language (TSL)
@jonathon_wright#BCS_SIGIST
“The test editor is designed to be low waste, it will only offer nouns and verbs based on discovered artefacts not programming objects.”
Agile Portfolio Management – Business Process Scenarios
@jonathon_wright#BCS_SIGIST
Agile Portfolio Management – Business Process Modelling
@jonathon_wright#BCS_SIGIST
Agile Portfolio Management – Agile Feature(s)
Estimate Effort?@jonathon_wright#BCS_SIGIST
Agile Portfolio Management – Product Backlog
Microsoft Project 2013 (Connected to TFS)Team Foundation Service (Product Backlog)
@jonathon_wright#BCS_SIGIST
Agile Portfolio Management – Work Items > Tasks
Team Foundation Service TFS Workbench
Digital Analogue
@jonathon_wright#BCS_SIGIST
Agile Portfolio Management – Feature-Driven
@jonathon_wright#BCS_SIGIST www.eurostarconferences.com
Agile Portfolio Management – Exploratory Testing
@jonathon_wright#BCS_SIGIST www.eurostarconferences.com
@jonathon_wright#esconfs
Agile Portfolio Management – Test Case(s)
Application Lifecycle Management – Unit Test(s)
@jonathon_wright#BCS_SIGIST
Solution Lifecycle Management – Code
@jonathon_wright#BCS_SIGIST
www.eurostarconferences.com
Solution Lifecycle Management – Issue(s)
Test Case Migrator Plus (TFS)
@jonathon_wright#BCS_SIGIST
Application Lifecycle Management – IntelliTrace
@jonathon_wright#BCS_SIGIST
@jonathon_wright#esconfs
Application Lifecycle Management – Action(s) Log Application Lifecycle Management – Screen Capture(s)
@jonathon_wright#BCS_SIGIST
Solution Under Test – Traffic Proxy
@jonathon_wright#BCS_SIGIST
Solution Lifecycle Management - Interfaces
@jonathon_wright#BCS_SIGIST
Application Lifecycle Management – Client(s)
Real Mobile Devices (PerfectoMobile) Test Studio for iOS (Telerik)
@jonathon_wright#BCS_SIGIST
Solution Lifecycle Management – Build Server
@jonathon_wright#BCS_SIGIST
Solution Lifecycle Management – Continuous Deployment
@jonathon_wright#BCS_SIGIST
Solution Lifecycle Management – Test Controller(s)
www.eurostarconferences.com@jonathon_wright#BCS_SIGIST
Load Agent(s) – Visual Studio 2013 – Load Test Manager
@jonathon_wright#BCS_SIGIST
www.eurostarconferences.com
Solution Lifecycle Management – Test Agent(s)
@jonathon_wright#BCS_SIGIST
Cross Browser Testing – Telerik (Art of Testing - Framework)
@jonathon_wright#BCS_SIGIST
Agile Portfolio Management – Business Process Data
ALTOVA XML Spy 2013 (R2 SP1)
@jonathon_wright#BCS_SIGIST
www.eurostarconferences.com
Agile Portfolio Management – Reporting - Burndown
TFS Workbench
Monitor velocity & capacity
@jonathon_wright#BCS_SIGIST
Agile Portfolio Management – Reporting - Test Execution
Microsoft Test Manager 2012.R3@jonathon_wright#BCS_SIGIST
@jonathon_wright#BCS_SIGIST
“The best aspects of these proven test approaches
demonstrate how they have evolved over the past
decade, and this echoes some of the changes towards
more lean and agile business methodologies. They are in
a constant state of evolution – just as the underpinning
technology evolves over time.”
My section in the ‘Experiences in Test Automation’ book
(Dorothy Graham & Mark Fewster) 2012.
Final Thoughts - Test Automation in the Cloud
www.sigist.org.uk
Contact Details – Jonathon-Wright.com
Testing Blog (www.Right-Testing.com)
Automation Blog (www.TaaaS.net)
@jonathon_wright#BCS_SIGIST
LinkedIn + Email ([email protected])
www.linkedin.com/in/automation
Testing-Alliance.com