Get Your Testing Process in Place

Senior iOS developernfrydenholm@ebay.com

Niels Frydenholm

Get Your Testing Process in Place

Technical Lead,, Xamarin Test Cloud@karlkrukow

Karl Krukow

“Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.”

Martin FowlerThoughtWorks

Continuous Delivery

Mobile Continuous Delivery Challenges

• App Store Distribution Model

• Maturity - No Industry Standards (yet)

• Complicated Tool Setup

• Realistic tests are challenging

• Tested on your users’ devices

• Vary Device Conditions - Device settings, Network…

Towards Continuous Delivery for Mobile

Tooling

People

Process

Aspects

Towards Continuous Delivery for Mobile

Tooling

People

Process

• Xamarin.UITest, Calabash

• Jenkins, TeamCity, TFS, …

• Xamarin Test Cloud

ExamplesAspects

Towards Continuous Delivery for Mobile

Tooling

People

Process

• Xamarin.UITest, Calabash

• Jenkins, TeamCity, TFS, …

• Xamarin Test Cloud

• Cross-functional Teams

• Behaviour Driven Development

• Shared responsibility

ExamplesAspects

Xamarin.UITest & Calabash

• Write Tests in C#

• Run with NUnit

• Xamrin/Visual Studio or CLI

Xamarin.UITest• Write Tests in Ruby

• Supports BDD, run with Cucumber

• RubyMine or CLI

Calabash

Xamarin.UITest & Calabash

Xamarin.UITest - Architecture

Your App

Test Cloud Agent

Calabash - Architecture

Your App

Test Cloud Agent

!

Scenario: Login

Given I am on the Login screen

When I login as "Nat"

Then I should go to the Assignments

Calabash - Architecture

Your App

Test Cloud Agent

Cucumber

!

Scenario: Login

Given I am on the Login screen

When I login as "Nat"

Then I should go to the Assignments

Calabash - Architecture

Your App

Test Cloud Agent

Cucumber

!

Scenario: Login

Given I am on the Login screen

When I login as "Nat"

Then I should go to the Assignments

Get Your Testing Process in Place

Senior iOS developernfrydenholm@ebay.com

Niels Frydenholm

Agenda

■ From story to running tests ■ Improvements in development and release cycle ■ Demo ■ Best practices (based on our learnings) ■ Final words ■ Questions?

eBay Classifieds, DenmarkTwo popular consumer apps and sites ■ DBA - DK’s #1 Classifieds site ■ BilBasen - DK’s #1 Online Vehicle Marketplace

A few numbers ■ Approx. 1/4 of the danish population is using the

DBA app ■ Mobile traffic share is 57% for DBA ■ Approx 1/8 of the danish population is using the

BilBasen app ■ Mobile traffic share is 54% for BilBasen

How we are organised

■ 4 iOS developers / (backend) ■ 1 backend developer ■ 1 QA expert ■ 1 Scrum master

The team

From Story to Running Tests

Implementing a storyBefore sprint ■ Product owner (PO) introduces story ■ Team and PO agrees on accept criteria’s

New feature idea

Lots of questions

Implementing a storyBefore sprint ■ Product owner (PO) introduces story ■ Team and PO agrees on accept criteria’s

New feature idea

Lots of questions

Implement featureWrite tests

ReviewTest

In Sprint ■ Developer(s) implements story and tests

■ Unit, Integration and UI tests ■ Done with code and tests are green

■ Review code (and tests) ■ Manuel test from other team member ■ Inspect code coverage to ensure test quality

Continuous Integration■ Jenkins on a couple of Mac Mini’s

- Unit -> Integration -> UI tests

- UI tests only in Simulator ■ A subset after commit ■ A subset in Xamarin Cloud ■ All UI tests nightly

!

■ Keep status visible and Green!

Focus on Automated Tests Has Improved ‘Everything’

Releases 2013

JAN

DBA versions

BilBasen versions

Refactored tests to Page objects

FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

2.0 2.0.1 2.1 2.2 2.3 2.3.1 2.4 3.0

4.1.2

4.1.3

4.2 5.0 5.0.1 5.0.2

5.2.1

3.53.3.1

3.2.3

3.2.23.1.1

Releases 2014

JAN

DBA versions

BilBasen versions

FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

3.0.1 3.1 3.2.1 3.4 3.4.2

5.0.3

3.2 3.3 3.4.1

5.0.4 5.0.5 5.0.6 5.1 5.2 5.2.2

Development improvements

• Limited refactoring - risk too high

• All changes = huge test effort

• Many bugs were not found until later…

• Lots of “legacy code”

• See first bullet

In early 2013

Development improvements

• Limited refactoring - risk too high

• All changes = huge test effort

• Many bugs were not found until later…

• Lots of “legacy code”

• See first bullet

• Planned refactoring of larger areas

• Boy scout clean up

• Adding tests to missing areas

• (Most) bugs are found immediately after commit

• QA can spend time on better things ■ Identify critical business areas ■ Find the crazy bugs

In early 2013 Now

Project example

■ Search criteria pages and search logic was totally re-written ■ From hardcoded options in the app ■ To meta driven from the server ■ Server side meta data was build as

part of a project ■ Saved searches now in sync with web

BilBasen - new search

BilBasen - new search

■ Most scenarios were already covered with UI tests

■ Manuel test effort ■ About one day of testing ■ Focused on network timings ■ Compare results with website

■ No serious bugs were found

Test results

• Entire team spent 2-5 days on test runs and bug fixes

• Serious bugs could be found in last minute

• Big releases to save the overhead

• Releases were often delayed

Release improvementsIn early 2013

• Entire team spent 2-5 days on test runs and bug fixes

• Serious bugs could be found in last minute

• Big releases to save the overhead

• Releases were often delayed

Release improvements

• Upload when all tests are green (all the time!)

• PO flexibility to change scope of release

• 1 hour test run for some releases ■ Bugs are very rarely found

In early 2013 Now

Demo

Best Practices (based on our learnings)

Best Practices

• Structure your test code well

• Page objects

• Replace “sleeps” with “wait_for”

Page objects and wait_for

Scenario: Check that change classification clears matrixdata with warning Given I am logged in as "Buyer" And I am on the SYI hub And I select classification "Hovedtelefoner" And I set "Type" to “In-ear” And I set price to "250" !

Before page objects (Ruby example)

Page objects and wait_for

Scenario: Check that change classification clears matrixdata with warning Given I am logged in as "Buyer" And I am on the SYI hub And I select classification "Hovedtelefoner" And I set "Type" to “In-ear” And I set price to "250" !

Before page objects (Ruby example)

And /^I set price to "(.*?)"$/ do |price| macro 'I swipe up' sleep(1) touch("view marked:'Price'") sleep(0.5) set_text("view marked:'Price'", price) sleep(0.5) touch("view marked:'OK'") sleep(1) end

Page object and wait_for

And /^I set price to "(.*?)"$/ do |price| @page.write_price(price) end

With page objects (Ruby example)

Page object and wait_for

And /^I set price to "(.*?)"$/ do |price| @page.write_price(price) end

With page objects (Ruby example)

def write_price(price) scroll_and_wait_for_row_with_mark("priceCell") touch("view marked:'Price'") keyboard_enter_text price close_keyboard end

SellYourItemPage class (page object)

def select_bundle(bundle_name) ..logic here.. end !def write_headline(headline) ..logic here.. end

Best Practices• Structure your test code well

• Page objects

• replace “sleeps” with “wait_for”

• Ensure stable test suite execution

• Give failed tests an extra run

Stability in your CI setup• UIAutomation + Simulator can often fail when running

a big test suite

• Re-run failed tests an extra time

• Cucumber rerun formatter

• NUnit-retry

• Red builds should be trusted, investigated and fixed

• Only use this if test suite starts to be unstable

Stability in your CI setup• UIAutomation + Simulator can often fail when running

a big test suite

• Re-run failed tests an extra time

• Cucumber rerun formatter

• NUnit-retry

• Red builds should be trusted, investigated and fixed

• Only use this if test suite starts to be unstable

Best Practices• Structure your test code well

• Page objects

• replace “sleeps” with “wait_for”

• Ensure stable test suite execution

• Give failed tests an extra

• Run tests on real devices

• Xamarin Test Cloud

• Screen sizes, OS versions

Best Practices• Structure your test code well

• Page objects

• replace “sleeps” with “wait_for”

• Ensure stable test suite execution

• Give failed tests an extra run

• Run tests on real devices

• Xamarin Test Cloud

• Screen sizes, OS versions

• Strive for fast test execution

• Use different test suites

• Avoid too much UI interaction for setup/teardown operations

Fast test execution

• Use the “Backdoor” to the app

• Xamarin.UITest.IApp.Invoke

• Invoke(“method”, param)

• Invoke logic - without using the UI

• Great for Setup / Teardown behaviour

• Backdoor logic should be tested with UI interactions in other tests

Fast test execution

• Use the “Backdoor” to the app

• Xamarin.UITest.IApp.Invoke

• Invoke(“method”, param)

• Invoke logic - without using the UI

• Great for Setup / Teardown behaviour

• Backdoor logic should be tested with UI interactions in other tests

• We use it for things like

• Create a user / Login

• Create listings (things for sale)

• Cleaning up

• Delete favorites

• Delete users listings

Fast test execution - backdoor optimised

Scenario: I can manage my listing and see the changes Given I am logged in as "UniqueSeller" using quick login And I have created a listing for "Hovedtelefoner" and is on the SYI VIP Then I see the VIP for "Her kommer en rimelig lang tekst" ! When I go back to my listings page And the created listing is in the list …more steps omitted…

Fast test execution - backdoor optimised

Scenario: I can manage my listing and see the changes Given I am logged in as "UniqueSeller" using quick login And I have created a listing for "Hovedtelefoner" and is on the SYI VIP Then I see the VIP for "Her kommer en rimelig lang tekst" ! When I go back to my listings page And the created listing is in the list …more steps omitted…

1 scenario (1 passed) 19 steps (19 passed)

0m34.054s1 scenario (1 passed) 19 steps (19 passed)

1m16.610s

Best Practices• Structure your test code well

• Page objects

• replace “sleeps” with “wait_for”

• Ensure stable test suite execution

• Give failed tests an extra run

• Run tests on real devices

• Xamarin Test Cloud

• Screen sizes, OS versions

• Strive for fast test execution

• Use different test suites

• Avoid too much UI interaction for setup/teardown operations

• Run tests for all active branches in CI

• Automate job creation

Feature builds■ All jobs required for a branch ■ Red builds are accepted for a period ■ Only merge to master with green builds

Final words

Start today…• QA is a Team effort / responsibility

• Bugs/Crashes will still live in the app

• Do not expect it to be effortless to get started

• It will quickly pay off

Start today…• QA is a Team effort / responsibility

• Bugs/Crashes will still live in the app

• Do not expect it to be effortless to get started

• It will quickly pay off

• Higher app quality (and code quality)

• Time to focus on new important features

• Faster time to market - with less bugs

• Happy customers - they like frequent updates of high quality

• Happy and calm QA

• Happy and brave developers

See it as an investment that gives you

Questions?

Niels Frydenholm@nfrydenholm