Upload
josiah-renaudin
View
165
Download
0
Embed Size (px)
Citation preview
T14TrackSession4/21/20162:00PM
"InnovationsinMobileTesting:ExpandingYourTestPlan"
Presented by:
Melissa Tondi
Denver Automation and Quality Engineering
Broughttoyouby:
340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·[email protected]·www.techwell.com
Melissa Tondi Denver Automation and Quality Engineering
Melissa Tondi has spent most of her career working within testing teams- concentrating on functional, performance, security, and mobile testing techniques. In her current roles, she is the founder of Denver Mobile and Quality (DMAQ) and, as Head of SQE at ShopatHome.com, she is assisting teams to continuously improve the design, build, test, and delivery of quality software. Throughout her career in the software test and quality engineering fields she has focused on organizing testing teams around three major tenets—efficiency, innovation, and culture. Melissa’s previous roles have included Vice President of Professional Services for a Mobile-focused startup, Director of Software Quality Engineering in the world’s leading education company; QA consultant for health care, finance, and software-as-a-service industries; and president of the Software Quality Association of Denver.
4/8/16
1
Mobile Dev+Test 2016 - Innovations in Mobile Testing: Expanding Your Test Plan
Melissa Tondi
Takeaways and Agenda
→ Level Setting: Your current (or new) mobile testing strategy – Understanding your Mobile Solutions
• Mobile Web: RWD and AWD • Apps: Native and Hybrid, Near Native • IoT (a brief definition)
→ Expanding your Test Plan - Test Types that Mobile has Introduced
→ Expanding your Test Plan - Accessibility → Expanding your Test Plan - Using Simulators and Emulators → Device proliferation and using the Device Matrix → Environments and Tools – Proxies, Performance, CI
4/8/16
2
Level Setting - Understanding your Mobile Solution: Mobile Web
What is “Mobile”? Mobile web sites designed to match every web-enabled device. They’re accessed through the mobile device’s web browser and do not need to be downloaded/installed. They re-direct to an M Site or similar.
– Responsive Web Design – an approach aimed at building sites to provide an appropriate viewing experience (reading and navigation with minimal resizing, panning, and scrolling) across a wide range of devices. It adapts the layout to the viewing environment.
• No matter how much you resize the screen, that same layout will automatically respond to that size
• A single codebase for a website can serve both mobile and desktop users. So instead of having two separate versions of your site (one desktop site and one mobile site), you will have a single responsive site that will accommodate simultaneously mobile and desktop users
Level Setting - Understanding your Mobile Solution: Mobile Web
What is “Mobile”? – Adaptive Web Design – uses the server to detect the device that’s
being used. A separate template is maintained for each device that is supported. There are several distinct layouts for multiple screen sizes and the layout used depends on the screen size used.
• The goal is a highly differentiated experience that is built for the specific intent of the mobile customer
• It allows you to specifically capture user intent more precisely on a mobile device
4/8/16
3
Level Setting - Understanding your Mobile Solution: Mobile Web
Responsive Web Design Pros and Cons – Harder to build initially, but is flexible as new devices and IoT
(Internet of Things) expand users’ experiences – Images are scaled down rather than resized or eliminated
altogether therefore, they generally take longer to load
Adaptive Web Design Pros and Cons – Faster to initially build, but requires maintenance as new devices
are supported – Because they are created for mobile users they have smaller and
optimized images. Images are loaded to meet the device resolution and screen size, making page load time more efficient.
Level Setting - Understanding your Mobile Solution: IoT
A Brief Definition… IoT (Internet of Things): a network of physical objects that feature an IP address for internet connectivity and the communication that occurs between these objects and other Internet-enabled devices and systems
→ Our current, common IP Addresses are based on IPv4. It uses 32-bit addresses and is limited to around 4.3 billion addresses
→ IPv6 is slated to replace IPv4 and uses 128-bit addresses with 3.4×1038 unique addresses of capability
4/8/16
4
Level Setting - Understanding your Mobile Solution: Things to Know
RWD and AWD → Use CSS (Cascading Style Sheets) to adjust layout → The spots where it changes are called “breakpoints” → Functionality may be hidden, or workflows may be adjusted based upon
size → Usually adjusted to vertically scroll Tools → BrowserStack
– https://www.browserstack.com/ → Mobile-Friendly Test
– https://www.google.com/webmasters/tools/mobile-friendly/ → responsive test
– http://responsivetest.net/#u=http://www.zootemplate.com|1024|768|1 → Responsinator
– https://www.responsinator.com/
Level Setting - Understanding your Mobile Solution: Things to Know
IoT → Gray Box Testing
– Understanding the Architecture, OS, Third Party Hardware, Apps, Firmware, Connectivity Protocols
→ Real Time Operating System (RTOS) – Operating Systems that serve real-time application process data as it
comes in, typically without buffering delays → Service Virtualization
– Emulates only the behavior of the specific dependent component. Instead of virtualizing an entire database, you monitor how the application interacts with the database and then replicate the related database behavior.
4/8/16
5
Level Setting - Understanding your Mobile Solution: Apps
Native Apps: Are coded with a specific programming language. They are fast, reliable, and powerful, and are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform. These apps are written in the language specified by the platform → Objective-C or Swift for iOS using the iOS SDK (Software
Developer Kit) → Java for Android using the Android SDK
Level Setting - Understanding your Mobile Solution: Apps
Hybrid Apps: These rely on development frameworks and are written with the same technology used for websites and mobile web implementations. They are hosted or executed inside a native container on a mobile device. They ensure cross-platform compatibility and can access the phone’s hardware (camera, GPS, user’s contacts, etc.) These are regarded as web applications wrapped inside of a mobile app. The mobile app wrapper itself uses a web viewer (a scaled down version of a browser) to display the web pages. Web pages are developed with the same technologies that are used for both traditional and mobile web like:
– CSS – HTML – JavaScript
4/8/16
6
Level Setting - Understanding your Mobile Solution: Apps
Near Native: These have business logic core written in a language other than what is provided by the platform. The UI and other aspects make full use of native components and APIs. The result is an app that can be targeted to multiple platforms but run natively on each. There is a somewhat high percentage of code reuse across platforms. Example: Apps written in Appcelerator's Titanium. They are generally written in Javascript, but make use of all native components. The app can be targeted to multiple platforms but it runs natively on each
Level Setting - Understanding your Mobile Solution: Things to Know
→ Apps require much greater hardware testing – 3D graphics – Device sensors
→ Apps need to work on all screen sizes – Avoid wasted space
→ Apps define XML layouts similar to HTML – Node Structure – Static Content (Images) – Dynamic Content (Color, text)
4/8/16
7
Level Setting - Understanding your Mobile Solution: Operating Systems
Android: → Custom version of Linux → Apps built in java → Chrome Browser
(WebKit) → Over 20,000 different
Android devices → Custom Manufacturer
code → Forked versions of
Android (like the Kindle)
iOS: → Proprietary Apple OS
built on BSD (Berkeley Software Distribution)
→ Only works on Apple hardware
→ Apps built in Objective C and Swift
→ Safari browser (WebKit) → Less than 20 different
devices → Low OS variance
Expanding your Test Plan – Test Types
Peripheral – Wireless Testing – NFC (Near Field Communication), Bluetooth/
Bluetooth LE Accessory, Stylus – Wired – Internal to Device – Headphone Jack, Keyboard – External to Phone - CC Readers, Bar Code Scanners
Connection Testing – USB Power/Data – 4G/LTE/CDMA+/Wimax – 3G/GSM/CDMA, 2G/Analog, WiFi, Hotspot Generation – Computer Tethering – Carrier – Throttle – Limiting bandwidth and measuring an App’s performance
4/8/16
8
Expanding your Test Plan – Test Types
Gestures – Swipe, tap, double tap, drag, pinch, spread expand, flick, shake,
orient, tilt, press, press and hold, press and tap, press and drag, swipe and hold, rotate.
Interruption Tests – Controlled - Plug in/out USB, power, and headphone, Home Button,
Power Button, Navigating to another App, Save State – Uncontrolled/”Elevator” test - SMS, Phone Call, Notifications
Expanding your Test Plan – Test Types
Syncing – Updating two or more locations to ensure applicable files are
equivalent. – App to Cloud, App to Computer, App to App, Device to Device
Internal Hardware Integration – Camera, GPS, Accelerometer, Battery Drain, SIM Card, Volume,
SMS, Microphone, Speakerphone
4/8/16
9
Expanding your Test Plan – Accessibility
Worldwide Web Consortium (W3C) – http://www.w3.org/TR/WCAG20/#guidelines - Web
https://www.w3.org/TR/wcag2ict/ - Mobile You’ll want to determine which Level is appropriate for your organization and users, but here’s the gist:
– Level A – A good place to start. Great for organizations that already have a product in use and who want to establish a baseline for accessibility conformance.
– Level AA – The next step. This level means that most people will be able to use your site/product in most situations. Many education and government agencies require this level.
– Level AAA – The most difficult to achieve and maintain. In rare situations, this may be required, but the W3C makes it clear that it is not possible to satisfy all Level AAA success criteria for some content.
Expanding your Test Plan – Accessibility
Exercise – Android
Settings > My Device >Accessibility > Vision > TalkBack > Turn TalkBack on
Exercise - iOS For iOS phones, go to Settings > General >Accessibility > VoiceOver > Turn VoiceOver on
4/8/16
10
Expanding your Test Plan – Accessibility: Things to Know (Android)
Android Gestures All gestures for TalkBack require only 1 finger. → Touch or move around the screen – speaks and select the item under
your finger → Swipe right or down – move to next item on screen → Swipe left or up – move to previous item on screen → Double-tap – select focused item → Swipe up then down – move to first item on screen → Swipe down then up – move to last item on screen → Swipe right then left – scroll forward on page or move slider → Swipe left then right – scroll back on page or move slider → Swipe up then left – Home button → Swipe down then left – Back button
Expanding your Test Plan – Accessibility: Things to Know (iOS)
iOS Gestures → 1 finger gestures:
– Touch or move around the screen – speaks and select the item under your finger
– Tap 2 times fast – activate the selected item – Swipe left or right – selects the previous or next item – Swipe up or down – move to the selected rotor option
→ 2 finger gestures: – Swipe up – read all from first object in selected area – Swipe down – read all from selected item – Scrub back and forth – go back, cancel, close pop-up – Pinch out/in – select/deselect – Rotate to open the web rotor and select rotor items (see the last point
for 1 finger gestures) → 3 finger gestures:
– Swipe left or right – move to previous or next horizontal screen/page – Swipe up or down – move/scroll down or up
4/8/16
11
Expanding your Test Plan– Emulators, Simulators
Choosing Emulators and Simulators
– A lab is not complete without a mixture of both
• A Simulator mimics only the software. iOS has one, but it does not simulate any of the iPhone hardware layer
• An Emulator mimics both software and hardware. Android has one, but only emulates CPU and Memory
Expanding your Test Plan– Emulators, Simulators
When to use Simulators/Emulators
→ Testing the Function • Broken buttons, missing images, correct formatting for vertical
and horizontal modes • RWD/AWD or mobile site elements display correctly
→ Previewing the Form
• Demo basic design and layout mockups before significant development investment is spent on polishing the UI
• See the application within the frame of a real device, compared to the window of a development program
4/8/16
12
Expanding your Test Plan – Emulators, Simulators
When to use Simulators/Emulators
→ Accelerating Development Time • For simple visual checks, opening an app in a device emulator
can be as simple as switching windows, compared to fully loading a physical device
→ Test more frequency with quick checks on an emulator, thus finding bugs faster and with greater context
Expanding your Test Plan – Emulators, Simulators: Things to Know
Simulators 16-Bit Display Operating System User Interface
Emulators Keyboard and Buttons WiFi Simulated SIM Card System on a Chip Sound Chip Flash Memory
4/8/16
13
Expanding your Test Plan – Device Testing Things to Know
24-Bit Display Touchscreen Multi-Touch Camera & Video Voice Recognition True Sound Quality USB Connections Bluetooth SD Cards Network Data Real Phone Calls Dual SIM Cards
Battery State Device Temperature Real GPS & Sensors Pre-Installed Apps App Store Access App Installation Email Sync Contacts Sync Calendar Sync App Log Encryption Network Security Fingerprint Scanner
Build Your Lab
4/8/16
14
Build Your Lab – Getting Started
Determining your Device Matrix → Operating System
– OS customizations, missing libraries, driver issues → Screen Size
– Rendering issues, usability, missing layouts → Pixel Density
– Density Independence, missing layouts → Aspect Ratio
– X,Y calculations, overlapping panels, display issues → System on a Chip (SoC)
– Hardware performance, Instruction set, battery, signal → Carrier
– Network protocol, speed, responsiveness, packet loss
Build Your Lab – Getting Started
Determining your Device Matrix – The Goal → Efficiency, not necessarily 100% coverage → Build your lab to cover both apps and mobile web → Define a list of categories of devices → Adjust the number of devices based on reasonable coverage
4/8/16
15
Build Your Lab – Getting Started
Determining your Device Matrix – Example Categories → Common: Matches the Most Common Display Configuration → Newest: Latest OS, Largest Screen, Highest Resolution → Oldest: Oldest, smallest, slowest device → Abnormal: Non-standard OS, Aspect Ratio, Orientation, and
Size → Popular: In Terms of Sales → Budget: Low-priced New Model → Flagship: By Manufacturer (for example)
– iPhone 7 (TBD) – Galaxy Note – Google Nexus – LG G5
→ Catch-All
Building your Lab – Android
Category DeviceName OS Size Density Resolu8on DPI AR Protocol SystemonChip
Newest Sam.GalaxyS 4.2.2 Normal xhdpi 1920x1080 441 16:9 LTE/GSM QualcommS4
Flagship LGNexus4 4.2 Normal xhdpi 768x1280 318 3:5 GSM QualcommS4
Oldest HTCTaNoo 1.6 Small ldpi 320x240 143 4:3 GSM QualcommS1
Popular Sam.GalaxyS3 4.1.2 Normal xhdpi 1280x720 326 16:9 CDMA SamsungExynos
Common Mot.Droid3 2.3.4 Normal hdpi 960x540 275 16:9 GSM CortexA9
Abnormal LGOpTmusVU 4.0 Large xhdpi 1024x468 256 4:3 GSM NvidiaTegra3
Budget DellVenue 2.2 Normal mdpi 480x800 228 3:5 GSM SnapdragonS2
Catch-all SonyXperiaP 2.3 Normal hdpi 960x540 275 16:9 GSM SonyNovaThor
4/8/16
16
Building your Lab - iOS
Device
Name
OS Display Aspect SoC Carrier
Newest iPhone6 8 5.5”1920x1080ppi 9:16 A8Chipwith*
MoTon
Coprocessor
T-Mobile
Oldest iPhone3g 6 3.5”320x480165ppi 2:3 AppleA3 AT&T
Common iPhone5s 7/8 4”1136x640326ppi 9:16 AppleA5 Verizon
Popular iPhone6+ 6 3.5”640x960330ppi 2:3 AppleA4 Sprint
iPad
(Re8na)
iPadAir 7 9.7”2048x1536264ppi 3:4 AppleA7 Verizon
iPod iPodTouch
(4thgen)
5 3.5”640x960326ppi 2:3 AppleA4 WiFi
Mini iPadMini 8 7”1024x768162ppi 3:4 AppleA5 AT&T
Build Your Lab – Summary
Determining your Device Matrix – End Result → Logically Select Devices → A Large Number of Permutations can be Covered with a
Handful of Devices → As New Devices are Announced, Run them through the DM
to Determine whether to Purchase
4/8/16
17
Environments
Test Environments – Browserstack.com or SauceLabs.com
• Online Simulators for any OS or mobile device • Useful for troubleshooting and regression testing • Defects aren’t always reproducible on real devices • Automation Cloud (supports WebDriver) • Quick rendering comparisons across multiple devices
– Hardware Rack • High Power USB Hub necessary to keep tablets charged • Stands to maintain device orientation
Tools – Things to Know
Mobile Automation Tools - Noteworthy → Mobile automation is still in its infancy → Mobile devices lack good automation hooks → Most tools can only control an app → Many automation tools require modification of the app
– Custom libraries, recompilation, or external application
→ Many automation tools can’t automate the stock web browsers – safari, chrome
→ Most tools can’t automate outside the app (OS, configuration, etc.)
4/8/16
18
Tools - Open Source vs “Closed”
→ Open Source is customizable, and maintainable by anyone trained
→ Open source tools typically require a custom framework
→ Anyone with a coding background can work on the tests
→ WebDriver is the W3C standard for web application testing
→ Closed tools are more usable for those without development backgrounds
→ Closed tools are expensive
→ Closed tools offer better reporting and integrations out of the box
→ Closed tools mean it may be hard to find people with experience in specific tools
Tools - Open Source Open Source Options → Selenium
– Multiple side projects support mobile testing – Selenium experience is very common
→ Appium, Android Driver, Robotium – Appium being merged as part of Selenium 3.0 – Acts as a remote webdriver node – Run tests on simulators or physical devices (USB) – Don’t have to change the app – Can automate default browsers (chrome / safari) – Can automate android / iOS apps – Same libraries used for desktop automation – Used in Browserstack and SauceLabs
→ MonkeyTalk – GUI based – Selenium based – Sort-of Free and Open source
4/8/16
19
Tools – “Closed”
→ EggPlant – Image based
automation tool – Can control full device
(not just apps) → HP UFT
– Large Market Share – GUI Based – Expensive
→ Ranorex – GUI Based – Supports android 2.2+
→ TestComplete – GUI or Code driven – Multiple language
choices – Also supports windows
applications
Other Tools
Proxies → A proxy intercepts all HTTP traffic from a computer or
mobile device → Traffic can be inspected, paused, re-executed, and
manipulated → Valuable for debugging purposes → Can be included with defect reports → Similar to chrome dev tools → Examples
– Fiddler (windows) – Charles (Mac) – BrowserMobProxy (Selenium)
4/8/16
20
Tools – Other Considerations
Performance Testing No such thing as client side performance testing → You can do memory, battery, CPU, RAM profiling during
normal testing activities → Generating Server load
– HTTP Traffic
→ Most load testing tools will support mobile – Configure it as a proxy – Record traffic – Parameterize – Generate Load
Tools – Other Considerations
Continuous Integration → A build server (Jenkins, TeamCity, Bamboo, CircleCI) kicks
off tests periodically → Can view pass/fail status, troubleshoot, etc. → Can be hooked up with most automation tools → Tests can be executed against physical devices, local
simulators, or cloud simulators → Speed up time through parallel execution → Setup a local GRID cluster for faster results → High power USB hub for tablets
4/8/16
21
Contact Information
→ email: [email protected]
→ Twitter: @melissatondi
→ Blog: MelissaTondi.blogspot.com
→ LinkedIn: Melissa Tondi