View
12
Download
0
Category
Preview:
Citation preview
© Copyright 2015 Vivit Worldwide
Testing Trends - Focusing on the Right Test Tools
and Industrializing Their Usage
March 11, 2015
© Copyright 2015 Vivit Worldwide
Brought to you by
Vivit Testing, Quality and Application Lifecycle
Management Special Interest Group (TQA-SIG)
and the South Africa Chapter
Leaders: Bernard P. Szymczak Jr, Andreas Birk, Robert Linton,
Christopher J. Scharer, Damian Versaci
www.vivit-worldwide.org
© Copyright 2015 Vivit Worldwide
Hosted by
Christopher J. Scharer
Vivit Director,
Vivit TQA-SIG Leader and
Iowa Chapter Leader
© Copyright 2015 Vivit Worldwide
Today’s Presenter
Mihai Grigorescu
Accenture Global HP Alliance Testing Lead
Vivit South Africa Leader
© Copyright 2015 Vivit Worldwide
Housekeeping
• This “LIVE” session is being recorded
• The recording will be available on BrightTALK
immediately after this session
• Q&A: Please type questions in the Questions
Box below the presentation screen
• Additional information available for you
behind the Attachment button and later on the
Vivit website
© Copyright 2015 Vivit Worldwide
AGENDA
Intro – Accenture testing
Testing trends – what the analysts say
Test Architecture and how TMMi fits into the picture
Testing ecosystem, choosing the right tools sets
ALM industrialization
Q & A
© Copyright 2015 Vivit Worldwide
Intro – Accenture testing
SAP Test tools implementation and support partner
HP SaaS SAP Sol Man to QC integration partner
© Copyright 2015 Vivit Worldwide
Intro – Accenture testing
© Copyright 2015 Vivit Worldwide
Testing trends – what the analysts say
Testing Spend
• Worldwide Discrete (Standalone) Enterprise Application Testing Services market was $16.7B in 2013 and growing 15% annually to
$33.7B in 2018 (Fig.1)
• U.S. Discrete (Standalone) Enterprise Application Testing Services market was $8B in 2013 and growing 13% annually to $15B in
2018 (see IDC report for graph)
• Budget spend % per projects is continuously increasing (Fig.2)
Fig.1
Source: IDC, Doc #249925 / July 2014 - Worldwide and U.S.
Discrete Testing Services 2014–2018 Forecast
Fig.2
Source: HP/SOGETI, World Quality Report 2014-2015
© Copyright 2015 Vivit Worldwide
Testing trends – what the analysts say
Drivers for using 3rd party services
• While drivers are multi-faceted, 31% of all respondents
indicate that the primary driver for using 3rd party testing
services centers on driving down costs.
• 37% of public sector organizations indicate the primary
driver for using 3rd party testing services is support for
complex infrastructure testing, with the second most
important being driving down costs.
• The top three characteristics buyers indicated were most
significant when selecting a provider for application testing
services were: reasonable costs (35%), high level of
responsiveness/rapid deployment (20%) and cloud testing
experience (20%)
0 5 10 15 20 25 30 35
Other
Support for vertical solution testing (e.g., banking, telecom)
Use turn-key test solutions for performance test or test automation
Support for localization – country specific readiness testing
Leverage fully managed specialized application test frameworks
Leverage fully managed application lifecycle testing services
As insurance against potential liability resulting from insufficiently tested products by in-…
Standardize internal QA environment across projects
Help with test data management services
Automate regression testing (to support complex business processes across multiple …
Leverage third-party tool expertise, talent and skills
Standardize testing tools and processes
Test staff augmentation
Drive productivity gains (e.g. more time for staff to pursue other initiatives)
Support for complex infrastructure testing (compatibility, interoperability)
Drive down costs
Source: IDC, Doc #249925 / July 2014 - Worldwide and U.S. Discrete Testing Services 2014–2018
Forecast
© Copyright 2015 Vivit Worldwide
Testing trends – what the analysts say
Fig.2
Source: HP/SOGETI, World Quality Report 2014-2015
Challenges and Outsourcing trends
• There is common trend of outsourcing testing services to specialized vendors, with a drop from 51% of companies usinginternal resources in 2012 to 30% in 2014 (Fig.1)
• In Agile environments, hybrids persist (i.e. Scrumfall), no real Agile environments, no specific methodologies nor adequatetools utilized, with difficulties in executing automation (Fig.2)
Fig.1
Source: HP/SOGETI, World Quality Report 2014-2015
© Copyright 2015 Vivit Worldwide
Testing trends – what the analysts say
• There is an increase in app
delivery which will reach
approximately 120 releases
per annum per app by 2020
• Companies are already
planning for upgrading to
HP ALM 12.x
Source: HP, March 2014
Source: VIVIT webcast poll, June 2014
© Copyright 2015 Vivit Worldwide
Testing trends – what the analysts say
• The main testing drivers for companies are: reduced time to
market, reduced costs, better resource management
(Fig. 1)
• Companies are investing in TaaS (Testing as a Service),
centralization, as well as managed services and outsourcing
their testing capabilities (Fig. 2)
Conclusion: we believe it is paramount to ensure
that test tools are optimized / industrialized within any test
organization in order to support the main testing drivers and
their respective processes adequately.
Source: HP, SOGETI - World Quality Report, 2012-2013
Source: HP, SOGETI - World Quality Report, 2013 -2014
© Copyright 2015 Vivit Worldwide
Testing trends – what the analysts say
Conclusions – key take away’s
• Digital transformation / business is here. One’s QA methodologies and tools need to caterdigital/TaaS/mobile testing. Virtualization (network, services etc.), testing in the cloud are allelements to consider and invest in.
• Applications are more fluid, delivered faster, which implies, leaner development & testing.Agile will take even more precedence than before, consequently develop competencies andcentralized capabilities.
• More automation (non-traditional) + DevOps are required for faster, organized release toproduction and for bridging the gab between dev teams and operations.
• With increase in overall testing spend, investing (more) in tools adoption is the next logicalinherent step.
• Security testing is still undermined (Sony knows for sure), companies need to invest in properenterprise IT security governance frameworks and conduct more testing in this space.
• Investments in skills and tools on one side, as well as change management on the other arethe crucial elements in winning the testing battle.
• With commoditization of testing, one needs to invest into more creative cost efficient TaaSmodels, more agile service based, output driven models, cross-shared and cross-skilledresources vs. legacy, traditional waterfall processes, dedicated resources and restrictivelicensing models.
© Copyright 2015 Vivit Worldwide
Test architecture and how TMMi fits into the picture
What a test architect IS What a test architect ISN’T
Testing expert: a test architect MUST come from a testing background, with experience in all
delivery models
Advisor: solution architecture, estimations, test planning, metrics definition (and
implementation), sales proposals
Fixer: get into challenged projects, produce short term/quick wins and long term improvement
plans
Testing evangelist: deliver papers, presentations, training, provide technical leadership and
strategic direction for their testing organization
Implementer: Setup of TCoE / Test Factory, execute Test Assessments (Internal, External &
TMMi), Enterprise Test Strategy (Definition/Setup), Test Optimization & Improvement, Non-
Functional Testing, Standard and Emerging Technologies / Test Tools
Test Manager
“Fire-fighter” for daily
operational issues
Delivery person (project
wise)
Great article on the topic here:
https://blogs.oracle.com/johnmorrison/entry/test_architect
What is a Test Architect?
© Copyright 2015 Vivit Worldwide
Test architecture and how TMMi fits into the picture
How TMMi fits in the picture of testing tools
Test Tools Maturity
Model
(based on TMMi)
Test Capability Models
(based on TMMi)
Test Maturity Model
integration
(TMMi)
© Copyright 2015 Vivit Worldwide
Test architecture and how TMMi fits into the picture
How TMMi fits in the picture of testing tools
• TMMi has 5 levels
• A TCoE will aim at a level 2 to 3, while test factories, between level 3 and 5
• Out of experience, we know that the perceived maturity level is higher than the actual one
© Copyright 2015 Vivit Worldwide
POLL 1
© Copyright 2015 Vivit Worldwide
POLL 2
© Copyright 2015 Vivit Worldwide
Test architecture and how TMMi fits into the picture
How TMMi fits in the picture of testing tools
• Accenture based
Test capabilities models
(based on TMMi)
© Copyright 2015 Vivit Worldwide
Test architecture and how TMMi fits into the picture
How TMMi fits in the picture of testing tools
© Copyright 2015 Vivit Worldwide
Test architecture and how TMMi fits into the picture
How TMMi fits in the picture of testing tools
Level 3
Level 4
Level 2
Level 5
Le
ve
l 1
Defect Management
Test Management
Requirement Management
Planning (release management)
Libraries and cross project asset sharing and reuse
Automation, Performance, Mobile, Security
Complete reporting capabilities
Executive Scorecards
Tool QA governance
Seamless integration to 3rd
party tools (CI)
Defect Management
Test Management
Requirement Management
Planning (release management)
Automation, Performance
Basic Mobile, Security
Advanced reporting (Dashboards across programs)
Scorecards
Tool QA governance
Basic CI
Defect Management
Test Management
Requirement Management
Advanced reporting (Project dashboards)
Project templates
KPI Scorecard
Some automation & performance
Defect Management
Test Management
Basic reportingAd-hoc usage
Basic defect management
Testing Tool Usage- Capability Maturity Level
Low High
© Copyright 2015 Vivit Worldwide
Testing ecosystem and focusing on the right tools
• Define (Test Architect) what tools to use in
the test organization based on needs and
long term requirements
• Ensure that all testing tools are integrated
for a streamlined testing process
• Maximize tools ROI via:
a) Tools industrialization
b) Strategic partnerships with tools
vendors, for band discounts
© Copyright 2015 Vivit Worldwide
POLL 3
© Copyright 2015 Vivit Worldwide
POLL 4
© Copyright 2015 Vivit Worldwide
POLL 5
© Copyright 2015 Vivit Worldwide
POLL 6
© Copyright 2015 Vivit Worldwide
ALM industrialization
HP ALM Industrialization
Test Tool Map
Tools Strategy
Tools Team
Upgrade versions
Upgrade editions
Accelerators
Integrations
Reporting
Quick Wins
Shift Left
© Copyright 2015 Vivit Worldwide
ALM industrialization
Test Tool Map
• Set up a test tools roadmap vs. your value chain / Software Development Life Cycle model
• Ensure that all testing areas are covered (Manual, Automation, Performance, Mobile, Sec., TDM, TEM
© Copyright 2015 Vivit Worldwide
ALM industrialization
Tools Strategy
Test tools strategy: The enterprise wide approach
which describes the solution architecture for the
deployment of all integrated test tools suites:
manual, automation, performance, mobile &
security, incl. additional tools such as requirements
accelerators, COTS and in-house grown (internal)
software utilized in the testing effort.
• Define and implement an overall test tool strategy for the next 3-5 years
• The test tool strategy needs to include capital and operating expenses (including software, hardware and
services costs) in order to provide the most efficient, cost effective solution for the organization
© Copyright 2015 Vivit Worldwide
ALM industrialization
Tools Team• Set up a dedicated test tools team to handle all test technologies in the organization
• Establish a test tool service catalogue & use the tools as bridging mechanisms into other business areas
© Copyright 2015 Vivit Worldwide
ALM industrialization
Upgrade versions
• Upgrade to the latest HP QC/ALM versions to make sure your organization gets the best value out of the
tool
• Make sure that your current QC version is still under support
What extended support entails:
Clients can still buy support beyond the standard EOL (End of Life)
product, e.g. 31 Jan 2015 in this case, however:
• It requires a CSP – Customer Support Plan, for max of 1 year @
150% of list price
• Availability of support resources: best effort (when available)
• Support type: no product fixes/enhancements requests etc.
ALM/QC 11.0 will be out of support in Jan 2015
(latest communication from HP):
Why consider upgrading editions:
• Avoidance of extended maintenance costs
• Continuation of vendor support after QC/ALM 11.0 end of support (Jan 2015)
• Decreased risks of troubleshooting issues and receiving product fixes/patches etc.
• License assignment per domain/business area (for all versions >11.5), enabling the possibility of producing a charge back model (to
business) for recuperating software license costs (best suitable for medium to large installations/test capabilities)
• Less disruption: no additional required upgrades (from older versions to more recent ones) in the next 3-4 years due to end of support.
• Access to newest ALM features & functionalities
HP ALM / QC Vendor Support Time-Table
Version 9.x 10.x 11.0 11.5 11.52 12.0 12.01
Support End Date Nov 2010 Aug 2013 Jan 2015 TBA TBA TBA TBA
© Copyright 2015 Vivit Worldwide
ALM industrialization
Upgrade editions
• If not already on Premier, consider the upgrade in Editions, in order to maximize the ROI on the test tools
© Copyright 2015 Vivit Worldwide
ALM industrialization
Accelerators• Utilize accelerators, either tools or other reusable assets in order to achieve maximum efficiency with the
organization test tools sets.
• SAP ERP QC Project Templates, Predictive Analytics, E-approve, Business Intelligence, Operations
accelerator, Requirements Management aiding tools, SAP/Automation accelerators etc.
© Copyright 2015 Vivit Worldwide
ALM industrialization
Integrations• Ensure that there are proper integration flows between test tools and that redundant / duplicate work is
not executed.
• Examples: AGM <-> ALM, Jira <-> ALM, Sol Man <-> ALM, ALM <-> PPM/XS, TFS <->ALM etc.
© Copyright 2015 Vivit Worldwide
ALM industrialization
Reporting• Define and implement an adequate test metrics framework
• Use ALM’s innate reporting capabilities (document generator, graphs and dashboards, KPI scorecards,
cross project reporting), Business Views, in conjunction with 3rd parties business intelligence apps;
publish QC/ALM graphs to Share Point Site (SPS)
© Copyright 2015 Vivit Worldwide
ALM industrialization
Quick Wins• Use the test capability to produce quick wins list that can be used with immediate effect in order to
improve the test processes (via the tools)
• Example below: defect management process, consistent project template
• Training: ART (Adoption Readiness Tool, other practical hands-on classes.
© Copyright 2015 Vivit Worldwide
ALM industrialization
Shift Left• Use ALM to drive Shift Left transformation into the organization
• Use simplified version of Shift Left (from a defect mgmt. perspective) – improve quality via defect root
cause analysis and improved management
Knowledge sharing – capability development (training, coaching)
Earlier defect detection, inspections, walkthroughs, JADs Methodologies + Tools
“Shift Left”Pushing quality upstream in the SDLC by enabling the testing capability with methodologies and tools
© Copyright 2015 Vivit Worldwide
Q & A
MIHAI.GRIGORESCU@ACCENTURE.COM
© Copyright 2015 Vivit Worldwide
Upcoming Vivit SIG Webinar
Best Practices in IT ArchitectureMarch 18, 2015 at 10:00 am EDT / 15:00 pm CET
Brought to you by the
Vivit Business Service Management Special Interest Group
For more information and registration go to:
www.vivit-worldwide.org
© Copyright 2015 Vivit Worldwide
Vivit Board of Directors election
Vote now on the 2015 Vivit Board of Directors election
and the proposed Bylaw changes:
http://www.vivit-worldwide.org/surveys/?id=459388
© Copyright 2014 Vivit Worldwide
HP Discover Las Vegas 2015
• June 2 – 4, 2015 at
The Venetian Resort
in Las Vegas.
• All members can
Register Now via the
Vivit link
www.hp.com/go/discover/vivit
to save $300.
© Copyright 2014 Vivit Worldwide
Vivit’s Deep Dive Sessions
• Deep Dive Sessions will be offered on Monday,
June 1st from 1:00 – 5:00 pm. Find more
information on website under the training section.
© Copyright 2015 Vivit Worldwide© Copyright 2015 Vivit Worldwide
Recommended