The Basics of Mobile App Testing & Test Automation

  • View
    1.808

  • Download
    1

Embed Size (px)

DESCRIPTION

Interested in a presentation video? Sign up at testdroid.com/webinars-archive The name of the game in today’s competitive mobile application landscape is a speed combined with robustness. How can you quickly deliver the best quality apps and how to ensure your app will stay on the top of download rankings at app markets? These questions will be answered as well as a comprehensive approach to improve your development process and app quality will be presented.

Text of The Basics of Mobile App Testing & Test Automation

  • 1.The Basics of Mobile App Testing & Test Automation 26 March 2014 Ville-Veikko Helppi Technical Product Manager ville-veikko.helppi@bitbar.com

2. Agenda App Quality and Mobile App Testing How To Get Started with Test Automation? Test Automation Frameworks 10 Tips for Successful App Testing Testdroid Update Demonstration Q&A Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 2 3. Agenda App Quality and Mobile App Testing How To Get Started with Test Automation? Test Automation Frameworks 10 Tips for Successful App Testing Testdroid Update Demonstration Q&A Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 3 4. Challenges in App Development Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 4 Fact: Most typical application fails when tested across different device variants used by consumers Failures: Both SW and HW cause failures Emulators do not offer realistic platform for dev Even app installation fails not tested at all Application crashes, leaks memory, consumes too much resources, behaves not as meant to behave UI scaling/display/resolution common problem Application performance is bad, draining battery 5. Why Apps Fail? Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 5 6. Testdroid Research Available Soon Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 6 7. What Improves the App Quality? Automate generic things as much as you can During the development your app changes the same must apply for testing! Carefully select (testing) technology & partner Use all 24 hours per day with test automation Cloud-based platform is the only solution to get you quickly covered in the global markets Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 7 8. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel Testdroid has helped thousands of app developers in these subverticals! What are the critical elements in each these verticals? Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 8 9. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel User Experience! Resource consumption & validation (CPU, Mem) Fully utilizing hardware e.g. Touch screen Mobile games require access to graphics APIs (e.g. OpenGL ES) Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 9 10. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel Usability! Functional and Behavioral testing Metrics analysis for all captured data (e.g. logs, screenshots, perf stats) Connection with Social Media + other apps Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 10 11. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel Performance! Connectivity, robustness and durability Screen orientation (portrait vs. landscape) Graphics quality, streaming capabilities User profiles Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 11 12. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel Security! Secure connectivity with back-end systems Top Quality extremely brand sensitive vertical Compliances and verification between real devices and infrastructure Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 12 13. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel Data! Connectivity and data connection with back-end systems Configurability of the app Bad quality can hurt the brand and leave you without customers Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 13 14. Agenda App Quality and Mobile App Testing How To Get Started with Test Automation? Test Automation Frameworks 10 Tips for Successful App Testing Testdroid Update Demonstration Q&A Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 14 15. Prerequisites for Mobile App Testing Testing environment as authentic as possible Real devices emulators/simulators Test coverage needs to be adequate Software is not identical in all Android/iOS devices Hardware is even more fragmented landscape Surrounding testing conditions (e.g. network) Try not to build in repetitive, manual processes Test automation framework that works for you Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 15 16. Trade-offs Manual vs. Automation Costs & Assets Time, People, Infrastructure, Tools, Training App Development & Testing Cycle Together Testing after Daily Builds (Continuous Integration) Tools & Technology Frameworks Myriad of frameworks available to help you Test Coverage & Reusability Global coverage is CRITICAL to SUCCESS Outcome Efficiency, Effectiveness and Results Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 16 17. Manual Automation Smaller coverage, More money burnt & time wasted, Error-prone Large coverage, quick ly completed, Less money & time wasted, Exact results. Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 17 18. What Framework Works You The Best? High-level & declarative test automation frameworks work well when app is under dev Easy to integrate with continuous integration Distance between use case and actual test What do you build: Native, Hybrid or Web app? The technical competence in your organization Desired outcome from test automation Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 18 19. Different Ways of Doing Test Automation Automatic test exercisersRecord and PlaybackHand written test scripts Benefits: Accurate, specific to your testing needs, plenty of options with frameworks, tools Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools availty Fastest & extremely automated, excellent for smoke testing/quick testing, availability Tradeoffs: Takes a lot of time, ties resources to write test cases/scripts, error- prone (humans) Compelling Recorder+Playback tools available for only few test automation frameworks Not accurate as real test cases Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 19 20. Developers Source code Repository Build failed Successful build QA & Testers Error reporting Test Cases Application Users / Consumers Commit Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 20 Test Automation During the Dev 21. Developers Source code Repository Build failed Successful build QA & Testers Error reporting Test Cases Application Users / Consumers Commit Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 21 Test Automation During the Dev 22. Why Real Devices are Must-to-Have? Emulators/simulators wont help you to test User Experience and Usability Hardware Software Infrastructure 0 % = the percentage of your app users that use emulator to run your app! Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 22 23. Test Automation Will Deliver You App Compatibility & Global coverage Reusability Test efficiency & effectiveness Faster time-to-market Test Automation during the Mobile App Development Process Testrequirements/Planning Smoke testing Regression testing Regression testing Regression testing Regression testing Functional & Interoperability testing Functional & Interoperability testing Functional & Interoperability testing Performance & Stress testing Performance & Stress testing Compatibility testing Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 23 24. Agenda App Quality and Mobile App Testing How To Get Started with Test Automation? Test Automation Frameworks 10 Tips for Successful App Testing Testdroid Update Demonstration Q&A Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 24 25. Android Instrumentation Framework InstrumentationTestRunner the primary plumbing for running tests on Android Android Instrumentation Framework is built on top of JUnit a standard test framework on for any Java development Mock objects methods for creating mock system objects such as content, service and intent Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 25 26. iOS Instruments (with Xcode) Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 26 27. Android Example: The Family Tree of Test Automation Frameworks JUnit Android Instrumentation Framework Robotium Espresso UI Automator Calabash Appium Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 27 28. Comparison of Test Automation Frameworks Supports Calabash Robotium uiautomator Espresso Appium Android Yes Yes Yes Yes Yes iOS Yes No No No Yes Mobile web Hybrid (webviews) Yes (Android) Limited to x.y clicks No Yes (Android & iOS) Scripting Language Ruby Java Java Java Almost any Test creation tools CLI Testdroid Recorder UI Automator viewer Hierarchy Viewer Appium.app Supported API levels All All 16 => 8, 10, 15-19 All Community Active Contributors Google Google Active Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 28 29. Agenda App Quality and Mobile App Testing How To Get Started with Test Automation? Test Automation Frameworks 10 Tips for Successful App Testing Testdroid Update Demonstration Q&A Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 29 30. Tip #1: Test early, Test often Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 30 31. Tip #2: Plan What to