Upload
service2media
View
245
Download
0
Embed Size (px)
Citation preview
1
30 MINUTEN Welcome! How to build your Mobile App Competence Center? Webinar Series.. #5 App Testing and Distribution Remco de Vos, Project Manager Peter Broekroelofs, CTO
Agenda
2
• A Mobile Maturity Model for Your App Center - Summary • App Test Process and various elements and roles in testing • Automatic Testing • Our Approach to testing in various phases (definition, development, acceptance) • App Distribution • App Lifecycle Management
Companies struggle in transition to strategic mobility
3
Opportunistic Strategic Mobile-First
The majority of companies A small minority of companies Very few, if any, companies
A reactive IT department
Mobility Center of Excellence: C-level attention, self-empowered lines of business, a responsive IT department
Low
Organization-wide strategic focus
Medium High
Most of the time customers only. Some-times siloed employee classes, typically field and sales forces and applications addressing basic informational interactions
Limited extendibility of architecture
Addresses large subsets of both dedicatedly and occasional mobile workers and more sophisticated offerings to customers. Transactional driven
Affects all mobile workers and internal activities and sophisticated customer engagement
Common architecture for mobility
Common architecture for mobility and integrated into most IT business processes
Siloed point solutions
Sophisticated administration and management tools; voice, data and integrated communications services
Integrated platform capabilities and ubiquitous connectivity
Few formal policies with decisions heavily user-influenced
Policy-driven approach for management, security and compliance
Policy-driven and 'factory' approach to mobile innovation, re-casting business workflows
Proportion of companies
Mobile strategy center of gravity
Level of business model innovation
Users
Architecture
Technologies
Policies
The Opportunistic Market in Transition
Experimentation is giving way to a more thoughtful approach to mobility. Organizations are taking a step back and rethinking how best to maximize the value of mobility. 3
1 "The early days of mobile adoption were characterized by experimentation and unfettered departmental demand for mobile apps. These apps, funded by the business, tend to be natively developed, and are built quickly and cheaply and often without coordination with the rest of the organization or a view to long-term sustainment and value maximization.
2 Such experimentation and piloting are necessary for organizations to test and learn about mobility, with the lessons learned that enterprise mobility can show significant business value, and that not embracing enterprise mobility will put the organization at a competitive disadvantage.
Source: Gartner, 2013
Quick Introduction to our Mobile Maturity Model
5
Disclaimer J
How to grow your mobile maturity?
6
1. What is your Mobile Strategy Maturity goal? 2. How are you performing currently? 3. What are the gaps? 4. Define gaps that matter most 5. Prioritize and close the gaps
7
1. Strategy and Organization 2. Initiation and Design 3. Security and Management 4. Development and Maintenance 5. Test and Distribution 6. Backend and Integration
Mobile Maturity Model - Assessment Axes
8
Mobile Maturity Model - Level determination
OPPORTUNISTIC
Developing
Aware
STRATEGIC
Optimising
Practicing
MOBILE FIRST
Leading
Opportunistic Strategic Mobile First
Strategy & Organisation
Practice observed 1 ✔ x
Practice observed 2 ✔ x
Initiation & Design
Practice observed 1 ✔ x
Practice observed 2 ✔
Development & Maintenance
Practice observed 1 x
9
Mobile Maturity Model - Gap Identification
Chapter 5:
10
App testing & Distribution
Advanced app solutions | 11
Advanced app solutions | 12
Imagine.. Your customer calls within 5 minutes after delivering the Release Candidates…
Our Bloopers
Advanced app solutions | 13
‘Testing is not a phase in a project, it has focus in every phase!’
Advanced app solutions | 14
Preparation Definition Development Rework /Acceptance (post)-GoLive
Acceptance Criteria
Test plan Unit Testing Functional Acceptance Tests
Full sanity check
Reference Set Use Cases Integration Testing User Acceptance Tests
Graphical & interaction designs
End-to-End Functionality testing
Design Verification
Style Guides Usability Testing
Design Verification
System Testing
Performance Testing
Load testing & Stress testing
Testing in our projects.. Executing Preparation
Roles vs Testing
Advanced app solutions | 15
Designer Client Developer
Back-end Developer
Tester Customer
Unit Testing ✔ ✔
Integration Testing ✔ ✔ ✔
End-to-End Functionality testing ✔ ✔
✔
Usability Testing ✔ ✔
✔ ✔
Design Verification ✔ ✔
✔ ✔
System Testing ✔ ✔
✔
✔
Performance Testing ✔
✔
✔
✔
Load testing & Stress testing ✔ ✔ ✔ ✔
Functional Acceptance Tests ✔
User Acceptance Tests ✔
App Test and Distribution Axis
16
App Testing
Opportunistic Strategic Mobile First
• Testing is performed on emulators and a few mainstream devices.
• A test process focusses on requirements and features in happy flow.
• Testing is performed on a representative subset of devices.
• Client has a decicated testing discipline. Testers are involved from project start and part of the scrum team.
• More attention for exceptions.
• Testing is partly automated with (cross platform) unit - and functional test tools on a representative subset of devices.
• Client has implemented testing at various levels, including e.g. design/prototype usability testing, integration testing and performance testing.
Practices Observed
Test definition
Test tooling
Test set
Test process
Automated testing
Advanced app solutions | 17
• Unit testing • Functional testing
Unit testing Example function TestSuiteByteArray:UnitTest_createWithOddHexNumber()
local data = ByteArray.newFromHex("0xA20"); Assertions.assertEquals("0x0a20", data:toHex());
end
function TestSuiteByteArray:UnitTest_createWithCapitalAndNonCapitalLettersHex()
local data = ByteArray.newFromHex("0xA20f34");
Assertions.assertEquals("0xa20f34", data:toHex()); end
function TestSuiteByteArray:UnitTest_createByteArraynewFromBase64String()
local data = ByteArray.newFromBase64("og80");
Assertions.assertEquals("og80", data:toBase64());
end
function TestSuiteByteArray:UnitTest_bytesExtremeValuesWorkOk()
local data = ByteArray.newFromHex("0x0080FF"); -- 0, 128 , 255
Assertions.assertEquals(0, data:get(1));
Assertions.assertEquals(128, data:get(2));
Assertions.assertEquals(255, data:get(3)); End
Functional testing Example Scenario: Test the button listener
Given I am on the mainpage
And I go to the "Manual/UI/Button/Button listener" menu
Then I should see "M2AT.08.06.030"
Then I drag from "clickButton" to "pressButton"
And I should not see "Press OK to continue"
Then I drag from "pressButton" to "clickButton"
And I should see "Press OK to continue"
And I press the "OK" button
Manual versus automatic testing Automatic testing Pro's
• Fast verification on every code commit possible for large percentage of functionality
• Promotes test early and often policy, especially for unit testing
• Investing in testscripts gets stronger on multiple OS-s and form factors from the same source
Advanced app solu8ons | 20
Automatic testing Cons:
• Additional costs, continuously (every large change)
• Flexibility loss in GUI (costs) • Requires (regression) test setup
for multiple devices / OS’s • Covers likely only most
important flows • Hardly ‘waterproof’; manual
verification remains. • Break even point for apps with
short lifetime hard to reach
How dynamic to
setup which tests?
How do we perform our testing?
Advanced app solutions | 21
• Definition Phase • Development Phase • Acceptance Phase
When the app goes into beta and the first bug reports arrive…
Advanced app solutions | 22
Phases in a project – Definition Phase 1/2
Advanced app solutions | 23
Contract / Project plan • Definition of target devices /
operating systems • Definition of ‘Reference Set’ # • Severity Matrix • Definition of Severity
# A ‘reference set’ is a selected set of devices to meet the specifications and behavior of the most commonly used devices of a specific target group.
Phases in a project – Definition Phase 2/2
Advanced app solutions | 24
Testplan • Which tests? (integration, performance, method of acceptance, etc) • Process of reporting & tools • Test arrangements • Distribution during test phase Scrum Team • Defines ‘Definition of Ready for test’ • Defines ‘Definition of Done’ • Creates Use Stories with Acceptance criteria, Business Logic & Exceptions for validation in testing
Advanced app solutions | 25
Our bloopers (2)
Imagine.. Right Facebook key, wrong copy / paste…
Phases in a project – Development phase 1/3
Advanced app solutions | 26
• List of devices (‘reference set’) – testing on devices & compliancy. • Compliancy list & ticking your devices
• Builds are created and published on internal fast installation location
• Usage of the tool ‘M2Borrow’ • A library of all our devices with OS’s • Barcode based • Easy track & trace from borrowers
Advanced app solutions | 27
Phases in a project – Development phase 2/3
Advanced app solutions | 28
• Working in JIRA with tickets • Open > In Progress > Resolved > Closed
• Important: signalize defects as soon as possible. Example ‘Defini.on of Ready’
Phases in a project – Development phase 3/3
Advanced app solutions | 29
• ‘Resolved > Closed’
• Note: ‘Resolved > Closed’ is in some cases assigned to the Product Owner of the customer.
Example ‘Defini.on of Done’
Phases in a project – Acceptance phase
Advanced app solutions | 30
• Optional specific tests (performance, load/stress, etc) • Acceptance builds are distributed to the customer (published on fast installation location and/or
TestFlight and in near future on new S2M Appstore.)
• Customer verifies functionality on variety of users (‘Family & Friends’ testing)
• Submission preparation
Advanced app solutions | 31
Imagine.. Mismatching team logo for Steaua Bucharest
Blooper intermezzo (3)
App Test and Distribution Axis
32
App Distribution
Opportunistic Strategic Mobile First
Ad hoc distribution to the app stores. Internal distribution via mail or website.
Partial automation of the distribution, including monitoring. A process with measureable gate before 'going live' and in sync with multi channel approach and compelling events. Use of enterprise stores
App distribution policy fully integrated in the Customer Lifecyle Focus on customer engagement. B2E distribution via integrated enterprise store for all OS-s. Centrally managed with policy based device and application management
Practices Observed
B2C distribution: App stores
App Lifecycle Management
B2E distribution: Enterprise store
MAM integration
Advanced app solutions | 33
App Distribution
• Final verification testing for submission
• App Stores • Apple iTunes, Google Play, Windows Market place, Blackberry World
• MDM / MAM (Good, MobileIron, XenMobile) • Mobile Device/Application Management systems & enterprise app stores
• Lean, fast and easy app distribution mechanism • Via Service2Media enterprise app store with In House Certificates
Add-on: App Lifecycle Management
Advanced app solutions | 34
• Suggestion: use a version control mechanism apart from the stores, like our M2Version service.
• Verifies on every start-up with a back-end if a business rule applies for a specific audience (app version, OS (version), language/country, etc)
• Allows pop-up messages to specific group of users on start-up for i.e. • Voluntary updates with a link to a (different) app or URL • Mandatory updates with a link to a (different) app or URL • Maintenance or outage messages with optional URL • Retirements of an app • Etc.
Next Webinar:
June 12th
36
#6. Backend and integration