Advanced Testing and Continuous Integration

Developer Tools #WWDC16

Session 409

Advanced Testing andContinuous Integration

Zoltan Foley-Fisher Xcode EngineerEric Dudiak Xcode Engineer

Agenda

Testing Concepts

Agenda

Testing Concepts

Xcode• Crash Log Gathering

Agenda

Testing Concepts

Xcode Server• Advanced Triggers• Issue Tracking and Blame• Configurable Integration User

Agenda

Testing Concepts

xcodebuild

Overview

Testing frameworkXCTest

Sources

TestClassA.m

TestClassB.swift

TestClassC.swift

Testing frameworkXCTest

Compile

Build Products

UI.xctest

Unit.xctest

Sources

TestClassA.m

TestClassB.swift

TestClassC.swift

Test harnessXcode

Author tests

Test harnessXcode

Author testsExecute tests

Test harnessXcode

Author testsExecute testsReview reports

Continuous IntegrationXcode Server

Schedules bot runs

Schedules bot runsGenerates reports

Schedules bot runsGenerates reportsSends notifications

Command line toolxcodebuild

Used by Xcode Server

Used by Xcode ServerBuild and execute tests

Used by Xcode ServerBuild and execute testsFor custom CI systems

Previous Years’ Sessions

Testing with Xcode 6 WWDC 2014

Continuous Integration with Xcode 6 WWDC 2014

UI Testing in Xcode WWDC 2015

CI and Code Coverage in Xcode WWDC 2015

Testing Concepts

Behind the scenesHow Testing Works

MyTests.xctest MyApp.app

Fundamental conceptsTesting Timeline

Compilation Hosting Observation

How testing startsTest Hosting

Unit Tests UI Tests

MyApp.app

Unit Tests UI Tests

MyApp.app

Unit Tests UI Tests

MyApp.app

Unit Tests UI Tests

Host Application

MyApp.app UI Test Runner MyApp.app

Unit Tests UI Tests

Host Application

Unit Tests UI Tests

Host Application

Unit Tests UI Tests

Host Application Target Application

ComparisonTest Hosting

Unit Tests UI Tests

Direct access to host app data and APIs Uses Accessibility to access target app

All tests run in single app launch Tests launch app for every test case

ComparisonTest Hosting

Unit Tests UI Tests

Direct access to host app data and APIs Uses Accessibility to access target app

All tests run in single app launch Tests launch app for every test case

Compilation Hosting

Test case structureTest Observation

Test Case 1 Test Case 2

Test Suite A

Test Suite A Test Suite B

Test Bundle

Setup or tear down work

Test Bundle

Setup or tear down workCustom logging

Test Bundle

XCTestObservation ProtocolTest Observation

Test Bundle

optional public func testBundleWillStart(_ testBundle: AnyObject!)

BUNDLE WILL START

Test Bundle

optional public func testSuiteWillStart(_ testSuite: AnyObject!)

SUITE WILL START

Test Bundle

optional public func testCaseWillStart(_ testCase: AnyObject!)

CASE WILL START

Test Bundle

optional public func testCaseDidFinish(_ testCase: AnyObject!)

CASE DID FINISH

Test Bundle

optional public func testCase(_ testCase: AnyObject!, didFailWithDescription

description: AnyObject!, inFile filePath: AnyObject!, atLine lineNumber: AnyObject!)

CASE DID FAIL

Test Bundle

optional public func testSuiteDidFinish(_ testSuite: AnyObject!)

SUITE DID FINISH

Test Bundle

optional public func testBundleDidFinish(_ testBundle: AnyObject!)

BUNDLE DID FINISH

// Test Observation // Example LoggingObserver

class LoggingObserver: NSObject, XCTestObservation {

class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) }

class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) } internal func testBundleWillStart(_ testBundle: Bundle) { print("Lights down, show is about to start") }

class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) } internal func testBundleWillStart(_ testBundle: Bundle) { print("Lights down, show is about to start") } internal func testCase(_ testCase: AnyObject!, didFailWithDescription description: AnyObject!, inFile filePath: AnyObject!, atLine lineNumber: AnyObject!) print("That sounds off key…") }

class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) } internal func testBundleWillStart(_ testBundle: Bundle) { print("Lights down, show is about to start") } internal func testCase(_ testCase: AnyObject!, didFailWithDescription description: AnyObject!, inFile filePath: AnyObject!, atLine lineNumber: AnyObject!) print("That sounds off key…") } internal func testBundleDidFinish(_ testBundle: Bundle) { print("Lights up, don’t forget your coat!") } }

Use NSPrincipalClassTest Observation

Test Bundle Info.plist

Use NSPrincipalClassTest Observation

Test Bundle Info.plistTest specific instantiation

SummaryTesting Concepts

Crash Log Gathering

Crashes during testingCrash Log Gathering

Test host or target application may crash

Test host or target application may crashXcode completes test suite

Test host or target application may crashXcode completes test suiteBut how to resolve the crash?

Improved test reportsCrash Log Gathering NEW

Improved test reportsCrash Log Gathering

For Unit and UI tests

For Unit and UI testsLocal and Xcode Server crash logs

For Unit and UI testsLocal and Xcode Server crash logsLogs included in test report

For Unit and UI testsLocal and Xcode Server crash logsLogs included in test reportView crashes in Xcode’s Debug Navigator

DemoCrash Log Gathering

Crash logs gathered in test reports

Crash logs gathered in test reportsView crashes in Xcode’s Debug Navigator

Continuous Integration in XcodeXcode Server

Eric DudiakXcode Engineer

OverviewXcode Server

Custom environment variables

Custom environment variablesAdvanced trigger editing

Custom environment variablesAdvanced trigger editingIssue tracking and blame

Custom environment variablesAdvanced trigger editingIssue tracking and blameUpgrade integrations

Custom environment variablesAdvanced trigger editingIssue tracking and blameUpgrade integrationsConfigurable integration user

New in Xcode 7.3Custom Environment Variables

Integration scriptsAdvanced Triggers NEW

Email notificationsAdvanced Triggers NEW

We all have issuesIssue Tracking and Blame

Nobody is perfect

Nobody is perfect• Tests will fail

Nobody is perfect• Tests will fail• Errors will come up

Nobody is perfect• Tests will fail• Errors will come up• Xcode will blame you

Even if you are perfect

Even if you are perfect• SDKs change

Even if you are perfect• SDKs change• Language improvements

Even if you are perfect• SDKs change• Language improvements• Smarter tools

Identifying issue ownersIssue Tracking and Blame NEW

Identifying issue ownersIssue Tracking and Blame

Lets you know when it’s time to act

Lets you know when it’s time to actYou broke it

Lets you know when it’s time to actYou broke it• You introduced an issue

Lets you know when it’s time to actYou broke it• You introduced an issue• Issue is on or near line you modified

Lets you know when it’s time to actYou broke it• You introduced an issue• Issue is on or near line you modified• You are the only committer

You can help fix it

You can help fix it• You probably know how to fix an issue

You can help fix it• You probably know how to fix an issue• You seem to know about the area involved

You can help fix it• You probably know how to fix an issue• You seem to know about the area involved• Fuzzy matching

Bot configuration changesIssue Tracking and Blame NEW

Bot configuration changesIssue Tracking and Blame

Tracks changes to your bot configuration

Tracks changes to your bot configurationAttributes issues to changes where possible

Tracks changes to your bot configurationAttributes issues to changes where possibleIncluded in email reports

When it’s not your faultUpgrade Integrations NEW

When it’s not your faultUpgrade Integrations

Re-integrates your project

Re-integrates your projectSame revision as the last integration

Re-integrates your projectSame revision as the last integrationAny new issues are due to the upgrade

Re-integrates your projectSame revision as the last integrationAny new issues are due to the upgradePrevents blaming you for broken builds

Configurable Integration User

Improved visibility into your integrations

Improved visibility into your integrationsAllows customization

Improved visibility into your integrationsAllows customizationYou own and manage how your integrations run

Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the system

Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra

Custom Integration User

••••••••••••••••••••••••••••••

DemoXcode Server

Improved visibility

Improved visibilityCustomized simulators

Improved visibilityCustomized simulatorsNormal macOS user

Improved visibilityCustomized simulatorsNormal macOS userRuns as a menu extra

Best practicesConfigurable Integration User

Dedicate a new user

Dedicate a new userAvoid using administrator accounts

Dedicate a new userAvoid using administrator accountsStay logged in with Fast User Switching

Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lock

Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs

Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators

Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators• Networking (VPN and proxies)

Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators• Networking (VPN and proxies)• User data and settings

Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators• Networking (VPN and proxies)• User data and settings• Advanced provisioning

Building block for Continuous Integration Systemsxcodebuild

Test actionxcodebuild

xcodebuild test -workspace <path>

-scheme <name>

-destination <specifier>

Builds sources

-scheme <name>

Builds sourcesInstalls app on destination if needed

-scheme <name>

Builds sourcesInstalls app on destination if neededRuns tests

-scheme <name>

Builds sourcesInstalls app on destination if neededRuns testsReports results in console

-scheme <name>

Customizing testsTest Options NEW

-scheme <name>

-only-testing:TestBundleA/TestSuiteA/TestCaseA

-only-testing:TestBundleB/TestSuiteB

-only-testing:TestBundleC

Customizing testsTest Options NEW

-scheme <name>

-only-testing:TestBundleA/TestSuiteA/TestCaseA

-only-testing:TestBundleB/TestSuiteB

-only-testing:TestBundleC

xcodebuild test -scheme <name>

-skip-testing:TestBundleD/TestSuiteD/TestCaseD

Separating building from testingAdditional Test Actions NEW

Build for Testing Test without Building

Only buildingBuilding for Testing NEW

xcodebuild build-for-testing -workspace <path>

-scheme <name>

Builds sources for testing

-scheme <name>

Builds sources for testingOutputs products in Derived Data

-scheme <name>

Builds sources for testingOutputs products in Derived DataGenerates an xctestrun file

-scheme <name>

Only testingTest Without Building NEW

xcodebuild test-without-building -workspace <path>

-scheme <name>

Only testingTest Without Building

Finds binary products in Derived Data

-scheme <name>

Finds binary products in Derived DataInstalls if necessary

-scheme <name>

Finds binary products in Derived DataInstalls if necessaryRuns tests and report results as before

-scheme <name>

Only testingTest Without Building NEW

xcodebuild test-without-building -xctestrun <path>

Ingests the xctestrun file

Ingests the xctestrun fileFinds binary products relative to xctestrun file

Ingests the xctestrun fileFinds binary products relative to xctestrun fileRuns tests, reports results as before

Distributed testingxcodebuild

Build Machine Test Machine #2

Test Machine #1

xcodebuild build-for-testing

Test Machine #1

xcodebuild test-without-building

Test Machine #1

xcodebuild test-without-building

Test Machine #1

Test manifestxctestrun NEW

Test manifestxctestrun

Tests to run or skip

Tests to run or skipEnvironment variables

Tests to run or skipEnvironment variablesCommand-line arguments

Tests to run or skipEnvironment variablesCommand-line argumentsDocumentation in man pages

Tests to run or skipEnvironment variablesCommand-line argumentsDocumentation in man pagesman xcodebuild.xctestrun

Summary

Testing Concepts

xcodebuild

More Information

https://developer.apple.com/wwdc16/409

Xcode Open Hours Developer Tools Lab C Thursday 9:00AM

Xcode Open Hours Developer Tools Lab B Friday 9:00AM

Advanced Testing and Continuous Integration

Documents

Continuous Integration and Code Coverage in Xcode...UI Testing in Xcode Presidio Wednesday 11:00AM Continuous Integration with Xcode 6 WWDC 2014 Related Lab Testing and Continuous

TOP TESTING TOOLS - PractiTest · TOP TESTING TOOLS We all know that the days when all testing was done using a single ... CONTINUOUS INTEGRATION Continuous Integration (CI) ... BAMBOO

Continuous Delivery: automated testing, continuous integration and continuous deployment

Testing @Google The State of Continuous Integration...Continuous Integration at Google Scale" by Celal Ziftci and Jim Reardon (ICSE 2017) “Taming Google-Scale Continuous Testing,”

Load Testing using Continuous Integration tools

Continuous Integration Testing for - TMCnet · Continuous Integration Testing for IoT Endpoint Integrity Arthur “Code Curmudgeon” Hicken Parasoft . ... Effectiveness of end-to-end

CONTINUOUS PRACTICES€¦ · From continuous testing, continuous development, and continuous deployment to continuous integration and continuous improvement, our development cycles

Manoj kolhe - Continuous Integration Testing

Pragmatic Continuous Delivery - JAX DevOps · DevOps Kanban Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment What? A methodology for reducing the

Testing & Continuous Integration · 2009. 3. 27. · Testing & Continuous Integration Kenneth M. Anderson University of Colorado, Boulder CSCI 5828 — Lecture 19 — 03/17/2009 1

Continuous Integration testing based on Selenium and Hudson

Jenkins, Bhyve, and Webdriver: Continuous Integration ... · Jenkins, Bhyve, and Webdriver: Continuous Integration testing on FreeNAS Craig Rodrigues rodrigc@ixsystems.com

Automated Testing and Continuous Integration for the Deutsche

Automating Performance Testing over CLIF and ProActive ... CLIF with workflows.pdf · Performance testing in Continuous Integration Continuous testing of User Quality of Experience

ONTAP Continuous Integration/Testing

DEVELOPMENT, TESTING, AND CONTINUOUS INTEGRATION OF

Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

Continuous Integration, Deployment and Testing in DevOps ...1044691/FULLTEXT02.pdfDevOps teams during continuous integration, deployment and testing. i Results. 31 primary studies

Continuous Integration Testing - Dave Stanke › img › DaveStanke-Continuous... · Continuous Integration Testing davidstanke@ Yes we can (still) do integration testing There are

Continuous Integration Testing in Django