Upload
anjan-dash
View
479
Download
0
Embed Size (px)
DESCRIPTION
Slides for Virtual Device Lab
Citation preview
Speed to Market in Mobile Development
Finding the right solution with continuous integration on real devices
Agenda Mobile trends & developers’ challenges Speed & quality through continuous integration
- Intuit Virtual Device Lab- Intuit mobile test automation
Takeaways
Aggressive mobile momentum
“2013 Internet Trends,” KPCB
Will surpass desktopby 2014
It should just work!
Proliferation of platforms, OS versions & device types
Developer challenges• Immense configuration matrix
– OS platforms (Android, iOS, Windows, phone vs. tablet, …)– Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)– Outputs (portrait/landscape, screen resolution, language)
• Application delivery through the markets– App signing, provision (app & device), submission
• Offline experience– Network, different network bandwidth/latency, no-network– HTML5 offline app
• Media challenges– Some codecs don’t work on a device– Streaming, sound
• Security & privacy
Intuit mobile app portfolio todayApple
Amazon
SMS
0 5 10 15 20 490
22
15
11
3
491 mobile apps for financial institutions
Ramping up Intuit’s innovation engine Pain: Takes days per build to test combinations of device/OS configurations physically & manually, one by one …QA can’t keep up with the changes
Goal: Reduce development iteration cycle from days to minutes on an ever-growing number of devices
Strategy: Speed & quality through mobile continuous integration
• iPhone 4• iPhone 4S• iPhone 5• iPad 2• iPad 3• iPad Mini
Devices
OS versions
• iOS 5.0• iOS 5.1 • iOS 6.0• iOS 6.1.x• iOS 7.0
Builds
X X
Daily builds
• Apple• Google• Amazon
Platforms
X
Continuous integration (CI)What is CI?
• Automate build, test & deployment process• Integrate & build early, often, daily, for every check-in• Best practice of Agile development• Jenkins is widely adopted, flexible & extensible
Benefits of CI in a mobile world
• Instant feedback on quality, functionality & system impact• Less complex, no integration hell• Detect and address device configuration issues early• Involve stakeholders early• Ultimately achieve shippable release any time
Test results
Speed and qualitythrough mobile CI
Key enablers:• Mobile device lab• Test automation
Test processorReal devices
Jenkins
SCM: source repository
Developers/IDE
Automatic test run
Submit the app to app store if passed
Check in
Check-in triggers
automated CI
Install the test app toreal devices in parallel
Run tests on real devices in parallel
Analyze test results
Trigger testing in mobile device lab
Process starts here
3rd-party vendor solutions didn’t work for us• Couldn’t handle E2E automation of testing pre-release apps
– Dependency on internal resources & infrastructure
• Vendor lock-in: customized CI & test automation solutions– No flexibility to use our existing tools & frameworks
• Disadvantages of vendors’ shared devices– Security concern on testing on shared devices– No access to corporate network– Cost-prohibitive to reserve devices
• Private cloud (on-site setup) with local devices not practical– With sensitive customer data, we need private cloud – Extensive use of devices for daily testing of hundreds of mobile apps is expensive
Why test on real devices?Most mobile features can be tested on simulators, but simulators have limitations:
• Device capabilities (GPS, camera, accelerometer, gyros), form factors & pixel density
• Real-world experience over the real connection(3G, 4G, crappy Wi-Fi, etc.)
• True performance, security tests• Testing on non-default Web browsers (e.g., Opera Mini)• Not all simulator/emulators have good quality
(don’t reflect real rendering/behavior, bugs)• Testing SMS & app integration
Intuit Virtual Device Lab
Intuit Virtual Device Lab (VDL)Gives access to real mobile devices using a browser
Intranet
VDL
User machine
User machine
User machine
3-click access
Always on
Integrated with dev infrastructure
Globally available
iPhoneiPad
GalaxyCaptivate
Optimus Nexus OneDesireIncredible
Manufacturers/devices:
DroidMilestoneMoto
VDL architecture – Android
Windows PC
Tomcat
Java image transferring application
Browser & Java applet
Internet/intranet
Device interaction app
Developer’s laptop
VDL
Image transfer
USB connection
VDL architecture – iOS
Mac with Xcode installed
Tomcat
Java VNC client
Port forwarding
Browser & Java applet
Internet/intranet
VNC server
USB connection
Developer’s laptop
VDL
Open source tools & technologies used in VDLKey capabilities Android iOS
Installing mobile applications into the real device
adb install <APK file> Fruitstraphttps://github.com/ghughes/fruitstrap
Transferring device screen images, controlling device
Screencast toolhttp://code.google.com/p/androidscreencast
Veency VNC app
Fetching device local files adb pull <filename> sftp command(jailbroken devices only)
VDL key features & functionality today Control the device through standard browser/RESTful APIs
Change orientation to landscape or portrait
Copy local file to the device from your PC
Get file from the device onto your local PC
Execute an ADB command on the device
See current running logs in your browser
Download history logs to your local PC
Continuous integration – how it works
Product builds
Trigger file launches test
VDL device inventoryResource
manager
Select a device
Test executionengine
Results
Monitor for new
builds
Mobile continuous integrationLeveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.
Submit, build, test … in minutes vs. hours
Select devices & tests1 Tests run automatically across selected devices with each build
2 See the results!3
Next step: over-the-air replaces USB connection
Fast onboarding
Scalable
Any device
Internet/intranet
App installation
Test execution
Results retrieval
VDL
Mobile Test Automation
Mobile test automation tools
Instrumentation-based
Non-instrumentation
AndroidiOS
MonkeyRunner
MOET
Sikuli
Android SDK
Robotium
TestDroid (BitBar)Calabash
(LessPainful)
Soasta
iOS SDK
UIAutomation
UISpec
Frank
MonkeyTalk
DA/PerfectoMob
ileeggPlantCross-platform solution
iOS Driver Appium
• Assessed as of 2012
Mobile test automation trends• Abstraction with domain-specific language (DSL)
– Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET)– Support behavior-driven development (BDD)
• Client/server-based approach (remotely driven tests via HTTP)• Instrumentation & non-instrumentation solutions complement
each other• Mobile cloud testing environment
– From hardware (cradle) to software approach to drive tests onreal devices
• Automated OTA app installation & tests
Intuit automation framework of choice/recommendation
• Native apps:
– Cucumber + Calabash for BDD on iOS & Android
• Mobile Web:
– Selenium/WebDriver
What is Cucumber?It lets stakeholders (customers, QE or developers) describe how software should behave in plain text.• BDD: Written in business domain-specific language (DSL)
– Good abstraction over different device types• One format for multiple purposes
– UI feature specification – Automated tests
• Cucumber itself is NOT a test automation tool
– Add-on to any test automation framework– Examples: Calabash, Sikuli & Frank mobile test
frameworks
What is Calabash?• Open source cross-platform mobile test framework
– iOS + Android, simulator + real device, native + hybrid• Supports Cucumber for BDD• Application needs instrumentation that embeds a Calabash HTTP server
– Tests run from a remote client (in a Jenkins slave)– Tests can be driven via Wi-Fi (no USB wire)
• Ruby clients in API level (Java clients in experiment)• Works great in touch simulation & real device testing
Intuit shared test library – rapid test development
Internal open source repository
Teams focus on creating
Use shared test lib
… and then contribute to common test library
Tests
Contribute
Tests
Common test library Product tests
TestsTests
Globalization test framework(I18N/L10N)
Performance/load test framework
Security test framework
Native/hybrid test framework
High-level test domain-specific language(Cucumber steps)
Mobileback-end
Mobilefront-end
Codebuild
Deploy
Test
Results
Connect device
Notifyemail
Getdevices
Consolidatereport
Uploadreport
From: manual
50*5= 250
15 30
5
10
20
15
30 10
= 6 hours
Codebuild
Trigger Execute tests in parallel
Consolidatedreport in archive
Deploy on devices
0 5 10 0
= 15 mins
To: automated CI
TIME SAVED: 95%Release quality increased
*
Stunning results: reduced iteration cycle from hours to minutes
Tested on 5 devices
VDL cost saving over vendors’ private clouds (today)
Annual cost ($)
# of devices
Intuit Virtual Device Lab
150
3rd-party vendor 1 local device in cradle
Assumptions: 50 native apps & daily check-in
Note: As vendor pricing changes, so do the savings
Savings
25
3rd-party vendor 2local device w/o cradle(Software approach)
Savings
Private cloud setup behind corp firewall
Key takeawaysBuilding custom solution vs. 3rd-party vendor solution
Understand your technical/platform/global spread Many vendor solutions provide a larger set of devices/OS versions
in the global market Quite a few offer mobile test services – might be tightly coupled with their
device access solution & automation framework Connecting to your corporate network is a challenge – some provide private
cloud (on-site setup within your network) but it’s cost-prohibitive
Invest in automation and CI Automate tests with robust automation framework Enable rapid test development through shared test library Adopt CI in mobile development for speed & quality
Thank you!!