Embed Size (px)
Citation preview
Mobile Testing Center of Excellence GroupPerfectomobile
Understanding the mobile environment
Risk management and Device governance
3 dilemma's
Emulators Vs. Real devices
Local vs. Cloud based device testing
How to reduce learning curve and TTM in mobile testing
Enterprise-grade mobile testing
Agenda
Mobile Environment
4,500,000
phones/ day
1,700,000
Smartphones
/ day
Mobile =
900,000
Androids
/ day
Source: Gartner/Google, 2012
800,000
PC’s /
day
300,000
Tablets /
day
~30 Billion$
revenues from
Mobile in 2012
US M-Commerce accounts for
11% U.S sales in 2012
Mobile =
Business
Critical
Mobile Pilots
Business critical
Mobile Revolutionizes Client Interaction
Mobile based shopping expected to grow by 24% in 2013 (U.S)
“Mobile is rewriting the
commerce playbook”
Mobile = Fragmented
Source: opensignalmaps, 2012
And the future is
landing…
Smartphones, Phablets, Tablets – All Part of The Matrix
9
Consumer Reaction – Less Tolerant than In Desktop
Mobile Introduces various challenges
Various OS
Types and Versions
Various Device/Tablets
OEM’s
Sample Screen Sizes
(Only Smartphone)
Application Types
On various browsers
AGILE ALM
Native Hybrid Web
2.3.3, 4.0.3
4.1.2, 4.2.2
4.3.3, 5.1
6.1, 6.1.1
QVGA: quarter
VGA (240×320
pixels)
HVGA: half VGA
(320×480 pixels)
WVGA: wide VGA
(480×800 pixels)
FWVGA: full wide
VGA (480×854
pixels)
nHD: one-ninth
high definition
(360×640 pixels)
qHD: one-quarter
high definition
(540×960 pixels)
Functional
Automation
Performance
Monitoring
New Mobile
OS’s
FireFox OS
Tizen
Samsung/Intel
Ubuntu
SailFish
By Jolla
RIM’s BlackBerry OS
10
Windows Phone
PEAK KEON
Z10 & Q10
Mobile OS market Trends
All of these OS’s still
accounts for ONLY 8% of the
overall market
Risk Management
Business Critical Requires Risk Management
WEB
• Browsers (IE, Chrome, Firefox & Safari)
• “Acceptable” not to support all browsers
• Assume users always have IE installed
• Enterprise can force IE and even a specific IE version
• Slow changes
• Stable technology
Risk Management
MOBILE
• Multiple technologies
• Unacceptable not to support a leading technology
• Can’t assume users always have something installed
• Enterprise can’t “force” devices anymore (BYOD)
• Rapid changes
• Screen sizes have more effect than in web
• Networks might have an effect
Risk Management Cont.
Single early failure = Non returning user (Crucial 30-60 seconds usage)
Risk Management Cont.
A clear process as part of the “mobile project” lifecycle
Define platforms and categories
Define coverage
Acceptable not to test - You can’t test everything
Define change frequency
Adapt to the market trends as they happen
Define SLA for changes
Define coverage as part of the ALM phase
Coverage visibility reports to project managers and executives
Risk Management - Device governance
Device Governance is a risk management for mobile projects
Risk Management Screen sizes matter
How many devices every quarter
What’s the process of replacement, what’s the SLA
Budget, don’t leave it to the last minute
You cannot afford not to have the latest trends
OS changes are a part of “device changes”
Some numbers:
Assume 30% change every quarter
10 devices – 50% coverage, 30 devices – 80% coverage
Tablets adds devices, does not change the mixture
Risk Management - Device Governance
Time to purchase
IT consideration
Security
Ready to work
You can’t cover everything
But you should define a target, which is the acceptable risk
iOS is not as simple as you think
6 Screen resolutions
At least 4 HW platforms (3GS, 4, 4S, 5,..)
(iPAD: Mini, 2, 3,4, iPODs)
At least 3 SW platforms
Perfecto’s 3M approach for choosing the right devices
http://help.perfectomobile.com/article/AA-01015/
Device Governance – Define coverage
Latest
55%
prev
minor
25%
prev
major
15%
Other
5%
The 3 Dillemas
They are a major part of the development phase
They have great development benefits
They can serve as good sanity for SOME test cases
The have limited testing benefits
They do NOT represent the real world
OEM are using these “versions” as a starting
point for their flavor of the device
It’s not “black” or “white”
Use them, but use them wisely
Emulators Vs. Device – 1st Dilemma
Intro to 2nd Dilemma – Local vs. Cloud
Risk management is not just device selection
How do you manage mobile ALM devices?
Purchase, inventory, lost, broken, SIM-cards, chargers…
How do you secure mobile ALM devices?
Secure access to your infrastructure (and “other leg” in the internet)
No Angry-Birds type applications allowed
Maintaining regulations such as GLBA and HIPAA
Don’t under-estimate mobile security concerns
How do you allow distributed work with those devices?
Multiple Rooms, buildings, campuses and continents
Device access, network access
Sharing of devices
Collaborations between teams
Device Governance in Enterprises
Back to this image…
One Chair
10 Devices
3 Extra Devices Many Cables
3 Workstations
One tester and 10-13 devices to test
3 of them are probably shared with … the kids
Limited sharing
Do the math with 20 testers…
Cannot really keep them up to date
Technical issues
Multiple workstations, drivers conflicts
More about this image
Importance of Mobile Cloud Solution
Source: CapGemini
The 3rd Dilemma – Extending Your Existing ALM
Risk to business outcomes due to
inability to meet mobile quality goals
Risk to business outcomes due to
inability to meet mobile quality goals
Inability to support fast,continuous delivery
Inability to support fast,continuous delivery
Inability to scale(test coverage, manual, offshore)
Inability to scale(test coverage, manual, offshore)
Operational nightmare(device mgmt. complexity, cost, risk)
Operational nightmare(device mgmt. complexity, cost, risk)
Fail to leverage existing ALM investment(Tools, process, policy, people)
Fail to leverage existing ALM investment(Tools, process, policy, people)
Mobile
Market
Diverse
Dynamic Localized
Mobile Strategy Must Change
31
Leverage your current ALM processes.
Integrate mobile-specific enhancements and new testing
approaches.
Preserve your investment and optimize mobile quality
The Perfecto Play
Mobile Application Quality
“Desktop ALM” solutions are a great starting place
• Testing frameworks
• Manual (i.e. Microsoft MTM, HP Sprinter)
• Automation (HP UFT, Rational)
• Performance Capabilities (i.e. HP LoadRunner)
• Production monitoring capabilities (i.e. HP BSM, IBM Tivoli)
• Continuous Delivery/Integration (i.e. Jenkins)
Mobile requires additional capabilities on top of the existing ALM
• Location based testing
• Network based testing
• Automation specific to mobile
• Performance and Monitoring specific to mobile
• Handset Validations
• Cross Platforms and Cross OS’s testing (Smartphones, Tablets, Emulators)
• Manual interoperability scenarios
• Continuous Delivery ready solution, shareable, collaborative
• Compliance with Regulation and Security
Close The Gap
33
