Ogres, onions and layers - TestCon Moscow...Solution: SwiftLint remarks use the same versions of...

1

Ogres, onions and layers story about tests at scale

Ewa Ludwiczak & Kamil PyćTestCon Moscow 2018

3

Ewa LudwiczakSenior Software Test Engineerewa.ludwiczak@allegro.pl

@ewabielskapoz

Kamil PyćSenior iOS Developerkamil.pyc@allegro.pl

@KamilPyc

4

Allegro application in Poland

5

Allegro application in Poland

6

Allegro application in Poland

7

Tests are like onions

8

1. Static analysis

2. Mutation testing

3. Snapshot tests

4. Functional tests

Tests are like onions

5. Continuous integration

6. Phased release

7. AB Tests

8. Application analytics

Web

9

instant updates

Backend Browsers

Why we bother?

Web

Mobile

10

instant updates

~2 days ~7 days

Browsers

App Store Review Devices

Backend

Why we bother?

11

1. Static analysis

2. Mutation testing

3. Snapshot tests

4. Functional tests

Tests are like onions

5. Continuous integration

6. Phased release

7. AB Tests

8. Application analytics

12

Problem 1:

13

Problem 1: Complex code

14

15

Problem 1: Complex codeSolution: SwiftLint

https://github.com/realm/SwiftLint

16

Problem 1: Complex codeSolution: SwiftLint

17

Problem 1: Complex codeSolution: SwiftLint

too short constant name

18

Problem 1: Complex codeSolution: SwiftLint

too long variable name

too short constant name

19

Problem 1: Complex codeSolution: SwiftLint

too long variable name

too short constant name

20

Problem 1: Complex codeSolution: SwiftLint

21

Problem 1: Complex codeSwiftFormat + SwiftLint

https://github.com/nicklockwood/SwiftFormat

22

Problem 1: Complex codeSwiftFormat + SwiftLint

23

Problem 1: Complex codeSolution: SwiftLint remarks

use the same versions of formatter & linter

run locally after each build

24

Problem 1: Complex codeSolution: SwiftLint remarks

use the same versions of formatter & linter

format obvious rules, lint less trivial

run locally after each build

25

Problem 1: Complex codeSolution: SwiftLint remarks

use the same versions of formatter & linter

format obvious rules, lint less trivial

run locally after each build

26

Problem 1: Complex codeSolution: SwiftLint remarks

apply to production code and tests

use the same versions of formatter & linter

27

1. Static analysis

2. Mutation testing

3. Snapshot tests

4. Functional tests

Tests are like onions

5. Continuous integration

6. Phased release

7. AB Tests

8. Application analytics

28

Problem 2:

29

Problem 2: Evergreen unit tests

30

Problem 2: Evergreen unit testsSolution: Mutation testing

function

31

Problem 2: Evergreen unit testsSolution: Mutation testing

function mutant

mutation

32

Problem 2: Evergreen unit testsSolution: Mutation testing

function mutant

mutation

33

Problem 2: Evergreen unit testsSolution: Mutation testing

change function logic

34

Problem 2: Evergreen unit testsSolution: Mutation testing

negate conditions

35

Problem 2: Evergreen unit testsSolution: Mutation testing

change mathematical operators

36

Problem 2: Evergreen unit testsSolution: Mutation testing remarks

code coverage says little about test quality

37

Problem 2: Evergreen unit testsSolution: Mutation testing remarks

code coverage says little about test quality

use manual mutation in non-trivial test cases

code coverage says little about test quality

not all mutations add value

use manual mutation in non-trivial test cases

38

Problem 2: Evergreen unit testsSolution: Mutation testing remarks

code coverage says little about test quality

not all mutations add value

use manual mutation in non-trivial test cases

39

Problem 2: Evergreen unit testsSolution: Mutation testing remarks

automated mutation testing

Problem 2: Evergreen unit testsSolution: Mutation testing

https://github.com/mull-project/mull

40

41

1. Static analysis

2. Mutation testing

3. Snapshot tests

4. Functional tests

Tests are like onions

5. Continuous integration

6. Phased release

7. AB Tests

8. Application analytics

42

Problem 3: Simple mistakesSolution: Code review

43

Problem 3: Simple mistakesSolution: Snapshot tests in code review

44

Problem 3: Simple mistakesSolution: Snapshot tests in code review

reference image recorded image

diff

45

Problem 3: Simple mistakesSolution: Snapshot tests in code review

value your time

involve testers and designers into review

spot visual mistakes on code review level

46

Problem 3: Simple mistakesSolution: Snapshot tests in code review

47

1. Static analysis

2. Mutation testing

3. Snapshot tests

4. Functional tests

Tests are like onions

5. Continuous integration

6. Phased release

7. AB Tests

8. Application analytics

48

Problem 4: Predicting consequences of change

49

Problem 4: Predicting consequences of change

50

Problem 4: Predicting consequences of change

51

Problem 4: Predicting consequences of changeSolution: End-to-end UI tests

https://github.com/kif-framework/KIF

Backend

52

Problem 4: Predicting consequences of changeSolution: End-to-end UI tests

separate test target

53

Problem 4: Predicting consequences of changeSolution: End-to-end UI tests

separate test target

flaky test

54

Problem 4: Predicting consequences of changeSolution: End-to-end UI tests

Backend

Problem 4: Predicting consequences of changeSolution: Functional UI tests with mocked API

55

JSON responses

Backend

Problem 4: Predicting consequences of changeSolution: Scheme validation

56

JSON files

JSON scheme

Problem 4: Predicting consequences of changeSolution: Scheme validation

57

JSON files

JSON scheme

Configuration file

Problem 4: Predicting consequences of changeSolution: Scheme validation

58

JSON files

JSON scheme

Configuration file

Backend request

Problem 4: Predicting consequences of changeSolution: Scheme validation

59

JSON files

JSON scheme

Configuration file

Backend request

Remote validation

Problem 4: Predicting consequences of changeSolution: Scheme validation

60

Configuration file

Remote validation

JSON scheme

API response

JSON file

Problem 4: Predicting consequences of changeSolution: Scheme validation

61

Configuration file

Remote validation

JSON scheme

API response

JSON file

Problem 4: Predicting consequences of changeSolution: Scheme validation

62

Configuration file

Remote validation

JSON file

JSON scheme

API response

63

Problem 4: Predicting consequences of changeSolution: Functional UI tests with mocked API

faster, more reliable UI tests

64

Problem 4: Predicting consequences of changeSolution: Functional UI tests with mocked API

faster, more reliable UI tests

relatively slow test coverage progress

65

Problem 4: Predicting consequences of changeSolution: Functional UI tests with mocked API

faster, more reliable UI tests

relatively slow test coverage progress

tests not critical to merge

66

Problem 4: Predicting consequences of changeSolution: UI tests improvements

More bugs detected10%comparison of 10 real API and mocked API UI test runs

Less flaky tests70%comparison of 10 real API and mocked API UI test runs

67

1. Static analysis

2. Mutation testing

3. Snapshot tests

4. Functional tests

Tests are like onions

5. Continuous integration

6. Phased release

7. AB Tests

8. Application analytics

68

Problem 5: Late feedbackSolution: Continuous integration

Problem 5: Late feedbackSolution: Continuous integration

69

Problem 5: Late feedbackSolution: Continuous integration

70

Get as many tester as possible

Automate your beta releases

Highlight what should be checked

71

Problem 5: Late feedbackSolution: Continuous integration

https://github.com/fastlane/fastlane

72

1. Static analysis

2. Mutation testing

3. Snapshot tests

4. Functional tests

Tests are like onions

5. Continuous integration

6. Phased release

7. AB Tests

8. Application analytics

73

Problem 6: Testing in laboratory condition

XXX

“Application doesn’t start after the update”

74

Problem 6: Testing in laboratory conditionSolution: Phased release

There are situations you cannot predict

Prepare for failure

75

Remote configuration

Problem 6: Testing in laboratory conditionSolution: Phased release

76

1. Static analysis

2. Mutation testing

3. Snapshot tests

4. Functional tests

Tests are like onions

5. Continuous integration

6. Phased release

7. AB Tests

8. Application analytics

77

Problem 7: Testing effectiveness

BUY NOW BUY NOW

78

Problem 7: Testing effectivenessSolution: AB Tests

79

Problem 7: Testing effectivenessSolution: AB Tests

80

https://motherboard.vice.com/en_us/article/8x8wp4/why-navigation-bar-looks-different-facebook

81

1. Static analysis

2. Mutation testing

3. Snapshot tests

4. Functional tests

Tests are like onions

5. Continuous integration

6. Phased release

7. AB Tests

8. Application analytics

82

Problem 8:

Application release

83

Problem 8:

https://fabric.io/kits/ios/crashlytics

Application release

Stable version

84

Problem 8:

Application release

Stable version

Comment on App Store

3

85

Problem 8:

“Where is categories button? I can’t find it!”

86

Problem 8:

“Where is categories button? I can’t find it!”

87

Problem 8: Decision making

“Where is categories button? I can’t find it!”

88

Problem 8: Decision makingSolution: Make use of app analytical data

89

Problem 8: Decision makingSolution: Make use of app analytical data

Data-based decisions

90

Problem 8: Decision makingSolution: Make use of app analytical data

Data-based decisions

Bug priority and severity

91

Problem 8: Decision makingSolution: Make use of app analytical data

Data-based decisions

Bug priority and severity

92

Analyze and plan your reaction

Problem 8: Decision makingSolution: Make use of app analytical data

93

1. Static analysis

2. Mutation testing

3. Snapshot tests

4. Functional tests

Tests are like onionshttps://bit.ly/2EWg91Y

5. Continuous integration

6. Phased release

7. AB Tests

8. Application analytics

94

Thank you for your attention!

Q&A

Ewa LudwiczakSenior Software Test Engineerewa.ludwiczak@allegro.pl

@ewabielskapoz

Kamil PyćSenior iOS Developerkamil.pyc@allegro.pl

@KamilPyc