Advanced Testing and Continuous Integration

© 2016 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

Developer Tools #WWDC16

Session 409

Advanced Testing andContinuous Integration

Zoltan Foley-Fisher Xcode EngineerEric Dudiak Xcode Engineer

Agenda

Agenda

Testing Concepts

Agenda

Testing Concepts

Xcode• Crash Log Gathering

Agenda

Testing Concepts


Xcode Server• Advanced Triggers• Issue Tracking and Blame• Configurable Integration User

Agenda

Testing Concepts



xcodebuild

Overview

Overview

Testing frameworkXCTest


Sources

TestClassA.m

TestClassB.swift

TestClassC.swift


Compile

Build Products

UI.xctest

Unit.xctest

Sources

TestClassA.m

TestClassB.swift

TestClassC.swift

Test harnessXcode

Test harnessXcode

Author tests

Test harnessXcode

Author testsExecute tests

Test harnessXcode

Author testsExecute testsReview reports

Continuous IntegrationXcode Server


Schedules bot runs


Schedules bot runsGenerates reports


Schedules bot runsGenerates reportsSends notifications

Command line toolxcodebuild


Used by Xcode Server


Used by Xcode ServerBuild and execute tests


Used by Xcode ServerBuild and execute testsFor custom CI systems

Previous Years’ Sessions

Testing with Xcode 6 WWDC 2014

Continuous Integration with Xcode 6 WWDC 2014

UI Testing in Xcode WWDC 2015

CI and Code Coverage in Xcode WWDC 2015

Testing Concepts

Behind the scenesHow Testing Works

MyTests.xctest MyApp.app

Fundamental conceptsTesting Timeline

Compilation Hosting Observation

How testing startsTest Hosting

Unit Tests UI Tests


MyApp.app

Unit Tests UI Tests


MyApp.app

Unit Tests UI Tests


MyApp.app

Unit Tests UI Tests

Host Application


MyApp.app UI Test Runner MyApp.app

Unit Tests UI Tests

Host Application



Unit Tests UI Tests

Host Application



Unit Tests UI Tests

Host Application Target Application

ComparisonTest Hosting

Unit Tests UI Tests

Direct access to host app data and APIs Uses Accessibility to access target app

All tests run in single app launch Tests launch app for every test case

ComparisonTest Hosting

Unit Tests UI Tests

Direct access to host app data and APIs Uses Accessibility to access target app

All tests run in single app launch Tests launch app for every test case


Compilation Hosting



Test case structureTest Observation


Test Case 1 Test Case 2


Test Suite A


Test Suite A Test Suite B



Test Bundle


Setup or tear down work


Test Bundle


Setup or tear down workCustom logging


Test Bundle

XCTestObservation ProtocolTest Observation


Test Bundle

optional public func testBundleWillStart(_ testBundle: AnyObject!)

BUNDLE WILL START



Test Bundle

optional public func testSuiteWillStart(_ testSuite: AnyObject!)

SUITE WILL START



Test Bundle

optional public func testCaseWillStart(_ testCase: AnyObject!)

CASE WILL START



Test Bundle

optional public func testCaseDidFinish(_ testCase: AnyObject!)

CASE DID FINISH



Test Bundle

optional public func testCase(_ testCase: AnyObject!, didFailWithDescription

description: AnyObject!, inFile filePath: AnyObject!, atLine lineNumber: AnyObject!)

CASE DID FAIL



Test Bundle

optional public func testSuiteDidFinish(_ testSuite: AnyObject!)

SUITE DID FINISH



Test Bundle

optional public func testBundleDidFinish(_ testBundle: AnyObject!)

BUNDLE DID FINISH

// Test Observation // Example LoggingObserver

class LoggingObserver: NSObject, XCTestObservation {

}


class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) }

}


class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) } internal func testBundleWillStart(_ testBundle: Bundle) { print("Lights down, show is about to start") }

}


class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) } internal func testBundleWillStart(_ testBundle: Bundle) { print("Lights down, show is about to start") } internal func testCase(_ testCase: AnyObject!, didFailWithDescription description: AnyObject!, inFile filePath: AnyObject!, atLine lineNumber: AnyObject!) print("That sounds off key…") }

}


class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) } internal func testBundleWillStart(_ testBundle: Bundle) { print("Lights down, show is about to start") } internal func testCase(_ testCase: AnyObject!, didFailWithDescription description: AnyObject!, inFile filePath: AnyObject!, atLine lineNumber: AnyObject!) print("That sounds off key…") } internal func testBundleDidFinish(_ testBundle: Bundle) { print("Lights up, don’t forget your coat!") } }

Use NSPrincipalClassTest Observation


Test Bundle Info.plist


Test Bundle Info.plistTest specific instantiation

SummaryTesting Concepts


Crash Log Gathering

Crashes during testingCrash Log Gathering



Test host or target application may crash


Test host or target application may crashXcode completes test suite


Test host or target application may crashXcode completes test suiteBut how to resolve the crash?

Improved test reportsCrash Log Gathering NEW

Improved test reportsCrash Log Gathering NEW

Improved test reportsCrash Log Gathering

For Unit and UI tests

NEW


For Unit and UI testsLocal and Xcode Server crash logs

NEW


For Unit and UI testsLocal and Xcode Server crash logsLogs included in test report

NEW


For Unit and UI testsLocal and Xcode Server crash logsLogs included in test reportView crashes in Xcode’s Debug Navigator

NEW

DemoCrash Log Gathering


Crash logs gathered in test reports


Crash logs gathered in test reportsView crashes in Xcode’s Debug Navigator


Continuous Integration in XcodeXcode Server

Eric DudiakXcode Engineer

OverviewXcode Server


Custom environment variables


Custom environment variablesAdvanced trigger editing


Custom environment variablesAdvanced trigger editingIssue tracking and blame


Custom environment variablesAdvanced trigger editingIssue tracking and blameUpgrade integrations


Custom environment variablesAdvanced trigger editingIssue tracking and blameUpgrade integrationsConfigurable integration user

New in Xcode 7.3Custom Environment Variables

Integration scriptsAdvanced Triggers NEW

Email notificationsAdvanced Triggers NEW






We all have issuesIssue Tracking and Blame


Nobody is perfect


Nobody is perfect• Tests will fail


Nobody is perfect• Tests will fail• Errors will come up


Nobody is perfect• Tests will fail• Errors will come up• Xcode will blame you



Even if you are perfect



Even if you are perfect• SDKs change



Even if you are perfect• SDKs change• Language improvements



Even if you are perfect• SDKs change• Language improvements• Smarter tools

Identifying issue ownersIssue Tracking and Blame NEW

Identifying issue ownersIssue Tracking and Blame

Lets you know when it’s time to act

NEW


Lets you know when it’s time to actYou broke it

NEW


Lets you know when it’s time to actYou broke it• You introduced an issue

NEW


Lets you know when it’s time to actYou broke it• You introduced an issue• Issue is on or near line you modified

NEW


Lets you know when it’s time to actYou broke it• You introduced an issue• Issue is on or near line you modified• You are the only committer

NEW



You can help fix it

NEW



You can help fix it• You probably know how to fix an issue

NEW



You can help fix it• You probably know how to fix an issue• You seem to know about the area involved

NEW



You can help fix it• You probably know how to fix an issue• You seem to know about the area involved• Fuzzy matching

NEW

Bot configuration changesIssue Tracking and Blame NEW

Bot configuration changesIssue Tracking and Blame

Tracks changes to your bot configuration

NEW


Tracks changes to your bot configurationAttributes issues to changes where possible

NEW


Tracks changes to your bot configurationAttributes issues to changes where possibleIncluded in email reports

NEW

When it’s not your faultUpgrade Integrations NEW

When it’s not your faultUpgrade Integrations

Re-integrates your project

NEW


Re-integrates your projectSame revision as the last integration

NEW


Re-integrates your projectSame revision as the last integrationAny new issues are due to the upgrade

NEW


Re-integrates your projectSame revision as the last integrationAny new issues are due to the upgradePrevents blaming you for broken builds

NEW

Configurable Integration User


Improved visibility into your integrations


Improved visibility into your integrationsAllows customization


Improved visibility into your integrationsAllows customizationYou own and manage how your integrations run


Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the system


Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra

Custom Integration User




••••••••••••••••••••••••••••••















DemoXcode Server



Improved visibility


Improved visibilityCustomized simulators


Improved visibilityCustomized simulatorsNormal macOS user


Improved visibilityCustomized simulatorsNormal macOS userRuns as a menu extra

Best practicesConfigurable Integration User


Dedicate a new user


Dedicate a new userAvoid using administrator accounts


Dedicate a new userAvoid using administrator accountsStay logged in with Fast User Switching


Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lock


Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs


Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators


Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators• Networking (VPN and proxies)


Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators• Networking (VPN and proxies)• User data and settings


Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators• Networking (VPN and proxies)• User data and settings• Advanced provisioning

Building block for Continuous Integration Systemsxcodebuild

Test actionxcodebuild

xcodebuild test -workspace <path>

-scheme <name>

-destination <specifier>


Builds sources


-scheme <name>



Builds sourcesInstalls app on destination if needed


-scheme <name>



Builds sourcesInstalls app on destination if neededRuns tests


-scheme <name>



Builds sourcesInstalls app on destination if neededRuns testsReports results in console


-scheme <name>


Customizing testsTest Options NEW


-scheme <name>


-only-testing:TestBundleA/TestSuiteA/TestCaseA

-only-testing:TestBundleB/TestSuiteB

-only-testing:TestBundleC

Customizing testsTest Options NEW


-scheme <name>


-only-testing:TestBundleA/TestSuiteA/TestCaseA

-only-testing:TestBundleB/TestSuiteB

-only-testing:TestBundleC

xcodebuild test -scheme <name>

-skip-testing:TestBundleD/TestSuiteD/TestCaseD

Separating building from testingAdditional Test Actions NEW

Test

Separating building from testingAdditional Test Actions NEW

Build for Testing Test without Building

Only buildingBuilding for Testing NEW

xcodebuild build-for-testing -workspace <path>

-scheme <name>


Builds sources for testing



-scheme <name>


Builds sources for testingOutputs products in Derived Data



-scheme <name>


Builds sources for testingOutputs products in Derived DataGenerates an xctestrun file



-scheme <name>


Only testingTest Without Building NEW

xcodebuild test-without-building -workspace <path>

-scheme <name>


Only testingTest Without Building

Finds binary products in Derived Data

NEW


-scheme <name>



Finds binary products in Derived DataInstalls if necessary

NEW


-scheme <name>



Finds binary products in Derived DataInstalls if necessaryRuns tests and report results as before

NEW


-scheme <name>


Only testingTest Without Building NEW

xcodebuild test-without-building -xctestrun <path>



Ingests the xctestrun file

NEW




Ingests the xctestrun fileFinds binary products relative to xctestrun file

NEW




Ingests the xctestrun fileFinds binary products relative to xctestrun fileRuns tests, reports results as before

NEW



Distributed testingxcodebuild

Build Machine Test Machine #2

Test Machine #1



Test Machine #1



Test Machine #1

xcodebuild build-for-testing



Test Machine #1


xcodebuild test-without-building



Test Machine #1






Test Machine #1


Test manifestxctestrun NEW

Test manifestxctestrun NEW

Test manifestxctestrun

Tests to run or skip

NEW


Tests to run or skipEnvironment variables

NEW


Tests to run or skipEnvironment variablesCommand-line arguments

NEW


Tests to run or skipEnvironment variablesCommand-line argumentsDocumentation in man pages

NEW


Tests to run or skipEnvironment variablesCommand-line argumentsDocumentation in man pagesman xcodebuild.xctestrun

NEW

Summary

Summary

Testing Concepts



xcodebuild

More Information

https://developer.apple.com/wwdc16/409

Labs

Xcode Open Hours Developer Tools Lab C Thursday 9:00AM

Xcode Open Hours Developer Tools Lab B Friday 9:00AM

Documents

Advanced Testing and Continuous Integration