View
218
Download
0
Category
Preview:
Citation preview
8/8/2019 Transforming Mobile Handset Testing
1/12
Transforming MobileHandset Testing
How Accentures new ROSA Testing Framework deliversHigh Performance by reducing costs, enhancing qualityand protecting brand reputations.
8/8/2019 Transforming Mobile Handset Testing
2/12
2
Whats driving these challenges? With
mobile technology maturing and openingup to third-party developers, handset
platforms are incorporating a new
generation of applications. No longer
simply a talk and text device, this
intelligent companion is now enabling
fully-fledged entertainment capabilities,
financial services and enterprise
mobility.
Given the growing list of specifications
associated with these mobile devices,
the demands on product testing are
rapidly increasing as well. To achieve
high performance, manufacturers,
carriers and other stakeholders
are challenged to produce high-
quality products that meet the rising
expectations of their customers without
missing critical timelines or allowing
testing costs to explode. Accenture
research confirms this trend as
application testing costs are likely to rise
from 2 percent of overall testing costs
to as much as 30 percent under present
circumstances.
To smartly and strategically address
the challenges around cost, qualityand reputation, companies will need
to explore new ways of managing and
executing product testing. Existing
testing approaches tend to be
fragmented, costly and labor intensive.
Recognizing these challenges, Accenture
has developed a comprehensive new
testing framework and approach, which
is known as ROSA*. This holistic and
systematic strategy is designed to help
product development and validation
organizations cut overall testing costs
by 30 to 50 percent and improve time to
market by as much as 20 percent while
increasing the volume of release testing
by as much as 25 percent.
Such gains will soon be critical to
success. If companies are to generate
profitable and sustained growth in
todays hypercompetitive mobile arena,
theyll need to ensure their products
cost-effectively meet the increasing
rigors and expectations of thisdemanding global market.
Market Drivers: ThreeKey Trends in MobileCommunications
The communications world is rapidly
beginning to embrace fourth generation
(4G) technologies, while opening up
new mobile device innovations through
open source applications and platform
foundations such as Symbian and
Android. This trend is expected to
offer users a wider range of advanced
services while achieving greater network
capacity. What are the market drivers
encouraging handset manufacturers and
network operators to rethink how they
are approaching product testing?
Industry challenges in testing todays
devices are linked to three key trends:
unprecedented application growth,
global product launches and the
uncompromising demands of customers.
Unprecedented Application Growth.
Convergence leads to applications
that can no longer be tested in asimple and easily controlled network
environment. Applications, for instance,
Mobile handset manufacturers and network operators are under severe pressure
to rethink their approaches to product testing. As mobile communications
technology becomes increasingly sophisticated and complex, current testing
strategies leave them vulnerable to spiraling costs, diminishing product quality
and threats to their brand reputations.
*Testing by Remote access, Offshore testing, Simulatednetwork testing, Automation
8/8/2019 Transforming Mobile Handset Testing
3/12
3
are now enabled by connectivity to
financial services, content providers,points of sale and location based
services. They also impose new security
requirements. All these capabilities and
specifications add complexity to the test
environmentraising the bar associated
with assuring and validating necessary
performance levels.
The application could be a gaming
device using Java and 3D hardware
accelerators. It could be a health service
that integrates content from various
sources, provides useful tips on wellness,
analyzes the users health profile and
measures key health indicators via
Bluetooth. It could be a Web 2.0 service
enabling social networking and access
to enterprise-based applications. While
the possibilities are staggering, the
challenge of cost-effectively testing
these proliferating applications to ensure
they perform to spec are eclipsing
the existing test capabilities of most
companies.
Global Product Launches.
Whereas manufacturers may have had
the luxury of introducing new mobile
handsets region by region in the past,
they are now expected to launch them
worldwidemandating a global launchstrategy. Considering that managing the
economics and efficiencies of a global
product launch is very different from
what has been experienced in the past,
most manufacturers and operators view
this as a new challenge.
What does it mean for manufacturers?
Network configurations and behavior
are not necessarily similar across
geographies. Local operators launch
services that are relevant and unique to
the local market. This not only includes
language support features but also
services that are used by operators as
differentiators. They may not be able
to identify many performance failures
until after the launch has occurred and
the product hits the market. What does
it mean for operators? Their reputations
are on the line as they roll out services
that are expected to meet certain
performance levels.
Unfortunately, manufacturers andoperators will have far less time and
opportunity to test their complex
offeringsregion by regionin an era of
global product launches. If they remain
wedded to existing testing approaches,
their vulnerabilities will quickly becomeevident to their customers.
The Uncompromising Demands
of Customers.
Unsurprisingly, customer expectations
remain exceedingly high in terms of
their experience with these products.
Indeed, user experience has become
a key driver in the success of product
launches. As underlying technologies
and platforms are commoditized,
manufacturers begin to focus on user
experience and user interfaces to
differentiate their handsets.
The integration of user interfaces on
multiple platforms further increases
testing challenges. As handsets grow to
support business-to-consumer, business-
to-business and business-to-employee
applications, users expect performance
to match what theyve experienced with
their laptops and personal computers.
They expect typical applications like
Skype, YouTube, Adobe Flash to beavailable at similar performance levels
on their devices. They also want the
ability to download media content
and perform actions on content such
8/8/2019 Transforming Mobile Handset Testing
4/12
4
environment. However, their specific
testing challenges differ slightly.
Testing Challenges Facing
Manufacturers:
Handset manufactures spend millions
of dollars testing a device from
development to product launch. This
testing comes in two phases:
1. Primary testing (or hygiene
testing) involves taking a handset
through various certifications that
are typically performed or provided
by accredited test houses. This
includes tests related to Global
Certification Forum (GCF) and
Personal Communication Service Type
Certification Review Board (PTCRB),
Federal Communication Commission
(FCC) marking, electromagnetic
compatibility testing (EMC), RadioFrequency (RF) Performance,
Bluetooth and WiFi, and Microwave
Communications. While primary
testing is demanding, the cost, quality
and schedules for such testing are
relatively predictable. They bring in
low to moderate challenges with
growing device complexity.
as editing, sorting and sending. And,
finally, they expect their handhelds to
have an intuitive design that is highly
user friendly. Clearly, performance
expectations are immenseand growing.
As convergence, innovation and market
pressures continue to drive new device
specifications and higher customerexpectations, testing strategies must
inevitably be transformed.
In previous years, testing for mobile
communications devices was far more
deterministic. What needed to be
tested was a lot more well defined.
The complexity of testing was most
often attributed to communication
technologies that essentially were
standards driven.
In the sweep of a decade, the industry
has shifted from voice quality as a
differentiator to an era defined by
the introduction of exotic third-party
applications and content. In this new
environment, application testing is
becoming an enormous challenge and,
for some, a barrier to continued growth.
As open-source platforms invite a wider
set of application developers into the
market, standardization has become
extremely difficult. Device testing is no
longer a deterministic endeavor. Indeed,
application testing costs grew from 3
percent of overall testing costs in 2004
to ~20 percent of overall costs in 2008.
This number is expected to increase over
the next two years to more than 30
percent of overall costs.
Smart and strategic testing is
now essential if companies are to
reduce these costs, deliver expected
quality levels and defend their
brand reputations in an unforgiving
market. These are the issues that both
manufacturers and network operators
must now confront.
Business Challenges:What Manufacturers andOperators Now Face
Handset manufacturers and network
operators both face significant testing
hurdles in todays fast-moving market
8/8/2019 Transforming Mobile Handset Testing
5/12
8/8/2019 Transforming Mobile Handset Testing
6/12
6
Consider the value of the following four
elements in the ROSA framework:
Remote Testing.
Through this method, devices are tested
within a particular geography under a
communications carrier. However, tests
are conducted from a remote location
by accessing the keys of the device
over the Internet and monitoring the
responses on the device screen of the
testers computer. The advantages of
this approach are multifold:
Atestercantestalargenumberofdevice features in multiple locationssimultaneously without traveling tothe location. Virtual teams across theAccenture Global Delivery Networktest devices and report results,helping to reduce costs significantly.
Scriptsarecreatedeasilybyplaybackof a user scenario that is performedonce on the device. Quality andconsistency of testing can beconfirmed as a result.
Testexecutioncanbeautomatedby
a single command that runs a testlibrary at a predetermined time andlogs the results in a log file. Comparedto manual testing, the volume of teststhat can be executed in a given timeis far larger.
Withminimumeffort,testscriptscan
be ported from one device to another,which helps confirm overall advantageover time.
Offshore Testing.
A large part of a handset test
specification does not require a live
network. These are network independent
tests that can be run on the device
manually. This type of test can be moved
to locations where tester availability and
costs are low.
However, offshore solutions can also
be defined in terms of having access tofield testers within any given geographic
area. Service providers with a global
network of talent and testing experience
may have the ability to conduct tests in
various geographical locations, sparing
the manufacturer or carrier the costs of
flying experts to the field location and
setting up relevant field tests.
Simulated Network Testing.
This method of testing is used for
new feature testing when network
infrastructure is not yet ready fornetwork tests. Using a simulated
network, features can be tested. The
method provides a network condition
under which a reasonable number
of defects can be identified. The
advantages of such testing include:
Easydebuggingenvironmentfortheproduct development team to localize,analyze and fix defects as well asverify the defect resolution.
Mechanismtoreproduceadefectobserved in the field that canotherwise not be reproduced in thereal network environment.
Softwarestabilityensuredbybuildinga test library and continuouslyenhancing it with test cases that
verify defect resolutions.Automation.
It is essential to fully leverage
automation when designing a test
framework for handset testing.
Whatever the means of testing,
automation dramatically increases the
quality of testing results. It enables
testers to run thousands of test
cases unmanned. It enables testers to
execute a test library, allowing them
to conclude that a defect resolution
has taken care of all possible scenarios
in which a defect can potentially
occur. Such results are impossible to
achieve through manual testing. Testing
automation, in other words, is a key
factor in confirming product stability.
Figure 1: ROSA Framework Drives Significant Cost Reduction
15-25%Down
35-45%Down
40-50%Down
PresentCost
FirstYear
SecondYear
ThirdYear
Graduate Cost Reduction from 15-50%over 3 year period
Estimated Quality andProductivity Benefits
Volume Time
Test Cases
Estimated Cost Reduction
8/8/2019 Transforming Mobile Handset Testing
7/12
77
8/8/2019 Transforming Mobile Handset Testing
8/12
8
Process Transformation
In a world of proliferating features,
there is no sensible and cost-effective
way to manage the testing challenge
using conventional, feature-driven
testing approaches. Testing teams
would merely grow in a linear fashion,
adding significant costs while slowingdown the testing operation.
To confirm success with the ROSA
approach, companies must begin to
organize around skill sets and test
categories instead of product features.
In order to redesign the testing process
(see figure 2), it is first necessary to
classify a handset test specification into
the following categories:
Test from Remote (TfR): All tests that
can be performed by remote access.Feature or functionality tests can all betested by remote access while mobilitytests and tests that verify performancequality may not be possible by remote.
No Network Required (NoN/W): Alltests that can be performed standalonewithout needing to connect to anetwork. Most tests for the phonebook,
media player and similar features fallunder this category.
Test by Simulation (TbS): All teststhat can be performed in a simulatedenvironment. Most features that can betested over an ideal network conditionand do not require complex interfaces tovarious services fall under this category.
Tester on Field (ToF): All tests thatmust be performed in the field. Thisassumes the tests must be performed bya tester physically operating the devicein a real network condition.
All test cases and test plans can be
coordinated using these test categories
or methodsan option that radically
simplifies testing approaches for
manufacturers and operators.
Organizing for ChangeTypically, product-testing teams are
organized by function or feature,
working across multiple handset models.
The ongoing addition of features and
functions merely increases the number
of teams and resources necessary to
meet certain performance benchmarks.
Companies, however, can enable a
significant increase in efficiencies
and cost optimization by reorganizing
teams based on the ROSA framework.
By organizing around ROSA testing
categories, they gain the ability to
group all testing activities within testing
categories (see figure 3).
Reorganizing teams in this fashion helps
drive testing productivity by enabling
the focused training and professional
development of teams. For instance:
When no network is required,standalone test teams requirevery little knowledge of deviceinternals. Training can be focusedon relevant features, which enablesthe standardization of techniquesfor coverage, failure and concurrentbehavior.
Remote testing requires training oncapabilities in scripting, automation,defect identification and reporting.Under this approach, tool licensingcan be optimized and knowledge ofmultiple testing tools can be impartedefficiently, which may require higher-degree technical skills as a tester thatcan be developed in a conventionalmanner.
Figure 2: ROSA Framework Drives Savings in Key Testing Categories
TfRPossible to access and test device fromRemote location(Test from Remote)
NoN/WStandalone Testing
(No Network Required)
TbSBest Tested in controlled/Simulatedenvironment(Test by Simulation)
ToFTesting on Field inevitable(Test on Field)
20%
35%
20%
25%
Device Feature NoN/W ToF TfR TbSActive Idle Screen 20% 20% 50% 10%Analog TV 90% 10%Applications 10% 30% 40% 20%Browser 10% 10% 30% 50%Camera 50% 10% 10% 30%Bluetooth 30% 30% 30% 10%Content Download 10% 30% 40% 20%Data Synchronization 60% 10% 10% 20%Digital Rights Management 10% 20% 60% 10%Digital TV 10% 70% 20% 0%Dual Keyboard 40% 10% 10% 40%Dual Screen 60% 5% 25% 10%Dual SIM 0% 30% 60% 10%Email related functions 10% 30% 60%
File Formats 70% 20% 10%Flash RSS 30% 30% 40%HTTP 10% 10% 10% 70%Language 25% 10% 25% 40%Local Connectivity 70% 10% 10% 10%Location Based Service 10% 20% 60% 10%Mail 20% 20% 60%Media Player 70% 10% 10% 10%Messaging 10% 20% 30% 40%Navigate condition 25% 10% 15% 50%News Casting 10% 60% 30% 0%Provisioning 20% 30% 50%Radio 90% 10%Ringtones 50%RSS Reader 30% 40% 30%Settings 90% 10%Screaming 10% 40% 50%Subscreen 50% 20% 30%
Test Editors 60% 10% 10%Utilities 30% 30% 20% 20%
Video Telephony 20% 10% 70%
Figures are representative of the categorization as an average of the various device test specifications across organizations that
Accenture has worked with.
8/8/2019 Transforming Mobile Handset Testing
9/12
9
Simulated network testing wouldrequire a strong understanding ofnetwork behavior as well as theability to use low-level scriptinglanguages, work with USB and otherconnectivities on the device, andautomate testing. Team membersmust have the experience and skills to
debug logs on the network side andreproduce user scenarios observed inthe field to enhance defect resolutionsignificantly.
Field testing will require skills innetwork maps, managing devicesand simulations. Testers mustreport defects in an intelligent andinformative way to developers at thehome office and reproduce typicalscenarios that are critical but notrepeatable. Such skills can be builtthrough a structured approach, andknowledge transfer can be managedefficiently through appropriatetraining and project management.
While these principles of organizationaland process redesign can be appliedwithin a product development andvalidation organization, manufacturersand network operators are increasinglylooking to outside specialists for severalkey reasons.
First, specialist service providers can
rationalize their investments in tools,
knowledge and experience across a wide
customer base. Second, their focused
commitment to such solutions enables
them to continually develop new skills
and experienceremaining on the
leading edge of testing solutions. And,
finally, they can offer global testing
coverage by remotely testing products
and by deploying personnel in regions
worldwide.
Accentures Assets andExperience
Accenture offers an example of the
types of capabilities and expertise that
must now be deployed to deliver high
performance in terms of handset testing.Test Libraries. Accenture has devicetest libraries with more than 6,000test scenarios based on experiencewith carrier test specifications, GCFfield trials guidelines and in-house testdevelopment.
Simulator Labs. The firms ThirdGeneration (3G) Network Simulatorenables testing in a controlled networkenvironment. Early detection ofdefects, feature testing and release testautomation are all handled throughthese labs.
Global Remote Test Labs. Accenturehosts remote test labs across its GlobalDelivery Network and client locations,which enables benchmarking and helpsto optimize device testing in terms oftime and cost.
Solution Enablers. Accenture has madean array of strategic investments inits partnerships and alliances, whichprovides know-how on commercialtest infrastructures and knowledge ofstandards and test specifications.
Such capabilitieswhich manufacturers
and operators should expect from theirtesting solution providersrepresent
the foundations of an advanced testing
infrastructure. By deploying and
leveraging such assets, companies can
cost effectively master the complexities
and challenges posed by the new
generations of mobile devices.
Browser
Device Management
Feature n
Handsets
Functions
A B C D
Handsets
Model 1
Model 2
Model 3
Model 4
Test Teams
NoN/W ToF TfR TbS
Figure 3: Organizing Around testing Catergories
8/8/2019 Transforming Mobile Handset Testing
10/12
8/8/2019 Transforming Mobile Handset Testing
11/12
11
So it goes with the companies confronted with testing todays mobile handsets.Expectations are greater than ever. Costs threaten to spiral out of control.
Quality is in danger of diminishing. And should you lose your grip, consumers
will turn on you like an angry tiger.
8/8/2019 Transforming Mobile Handset Testing
12/12
Copyright 2010 Accenture
All rights reserved.
Accenture, its logo, andHigh Performance Deliveredare trademarks of Accenture.
About AccentureAccenture is a global managementconsulting, technology servicesand outsourcing company, withmore than 176,000 people servingclients in more than 120 countries.Combining unparalleled experience,comprehensive capabilities acrossall industries and business functions,and extensive research on the worldsmost successful companies, Accenturecollaborates with clients to help thembecome high-performance businessesand governments. The companygenerated net revenues of US$21.58billion for the fiscal year ended Aug.31, 2009. Its home page iswww.accenture.com.
Contact us
For more information on achievinghigh performance through innovationandtestinggotowww.accenture.com/productsoftware or contact your localrepresentative about how Accenturecan deliver high performance throughproduct software and testing solutions.
Please contact:Risto ValtakariExecutive DirectorProduct Software SolutionsTel: +358 205725 508
Contributed byVenugopal RamakrishnanAccenture Product Software Solutions
Recommended