Embed Size (px)
Citation preview
Ogres, onions and layers story about tests at scale
Ewa Ludwiczak & Kamil PyćTestCon Moscow 2018
3
Ewa LudwiczakSenior Software Test [email protected]
@ewabielskapoz
Kamil PyćSenior iOS [email protected]
@KamilPyc
4
Allegro application in Poland
5
Allegro application in Poland
6
Allegro application in Poland
7
Tests are like onions
8
1. Static analysis
2. Mutation testing
3. Snapshot tests
4. Functional tests
Tests are like onions
5. Continuous integration
6. Phased release
7. AB Tests
8. Application analytics
Web
9
instant updates
Backend Browsers
Why we bother?
Web
Mobile
10
instant updates
~2 days ~7 days
Browsers
App Store Review Devices
Backend
Why we bother?
11
1. Static analysis
2. Mutation testing
3. Snapshot tests
4. Functional tests
Tests are like onions
5. Continuous integration
6. Phased release
7. AB Tests
8. Application analytics
12
Problem 1:
13
Problem 1: Complex code
14
15
Problem 1: Complex codeSolution: SwiftLint
https://github.com/realm/SwiftLint
16
Problem 1: Complex codeSolution: SwiftLint
17
Problem 1: Complex codeSolution: SwiftLint
too short constant name
18
Problem 1: Complex codeSolution: SwiftLint
too long variable name
too short constant name
19
Problem 1: Complex codeSolution: SwiftLint
too long variable name
too short constant name
20
Problem 1: Complex codeSolution: SwiftLint
21
Problem 1: Complex codeSwiftFormat + SwiftLint
https://github.com/nicklockwood/SwiftFormat
22
Problem 1: Complex codeSwiftFormat + SwiftLint
23
Problem 1: Complex codeSolution: SwiftLint remarks
use the same versions of formatter & linter
run locally after each build
24
Problem 1: Complex codeSolution: SwiftLint remarks
use the same versions of formatter & linter
format obvious rules, lint less trivial
run locally after each build
25
Problem 1: Complex codeSolution: SwiftLint remarks
use the same versions of formatter & linter
format obvious rules, lint less trivial
run locally after each build
26
Problem 1: Complex codeSolution: SwiftLint remarks
apply to production code and tests
use the same versions of formatter & linter
27
1. Static analysis
2. Mutation testing
3. Snapshot tests
4. Functional tests
Tests are like onions
5. Continuous integration
6. Phased release
7. AB Tests
8. Application analytics
28
Problem 2:
29
Problem 2: Evergreen unit tests
30
Problem 2: Evergreen unit testsSolution: Mutation testing
function
31
Problem 2: Evergreen unit testsSolution: Mutation testing
function mutant
mutation
32
Problem 2: Evergreen unit testsSolution: Mutation testing
function mutant
mutation
33
Problem 2: Evergreen unit testsSolution: Mutation testing
change function logic
34
Problem 2: Evergreen unit testsSolution: Mutation testing
negate conditions
35
Problem 2: Evergreen unit testsSolution: Mutation testing
change mathematical operators
36
Problem 2: Evergreen unit testsSolution: Mutation testing remarks
code coverage says little about test quality
37
Problem 2: Evergreen unit testsSolution: Mutation testing remarks
code coverage says little about test quality
use manual mutation in non-trivial test cases
code coverage says little about test quality
not all mutations add value
use manual mutation in non-trivial test cases
38
Problem 2: Evergreen unit testsSolution: Mutation testing remarks
code coverage says little about test quality
not all mutations add value
use manual mutation in non-trivial test cases
39
Problem 2: Evergreen unit testsSolution: Mutation testing remarks
automated mutation testing
Problem 2: Evergreen unit testsSolution: Mutation testing
https://github.com/mull-project/mull
40
41
1. Static analysis
2. Mutation testing
3. Snapshot tests
4. Functional tests
Tests are like onions
5. Continuous integration
6. Phased release
7. AB Tests
8. Application analytics
42
Problem 3: Simple mistakesSolution: Code review
43
Problem 3: Simple mistakesSolution: Snapshot tests in code review
44
Problem 3: Simple mistakesSolution: Snapshot tests in code review
reference image recorded image
diff
45
Problem 3: Simple mistakesSolution: Snapshot tests in code review
value your time
involve testers and designers into review
spot visual mistakes on code review level
46
Problem 3: Simple mistakesSolution: Snapshot tests in code review
47
1. Static analysis
2. Mutation testing
3. Snapshot tests
4. Functional tests
Tests are like onions
5. Continuous integration
6. Phased release
7. AB Tests
8. Application analytics
48
Problem 4: Predicting consequences of change
49
Problem 4: Predicting consequences of change
50
Problem 4: Predicting consequences of change
51
Problem 4: Predicting consequences of changeSolution: End-to-end UI tests
https://github.com/kif-framework/KIF
Backend
52
Problem 4: Predicting consequences of changeSolution: End-to-end UI tests
separate test target
53
Problem 4: Predicting consequences of changeSolution: End-to-end UI tests
separate test target
flaky test
54
Problem 4: Predicting consequences of changeSolution: End-to-end UI tests
Backend
Problem 4: Predicting consequences of changeSolution: Functional UI tests with mocked API
55
JSON responses
Backend
Problem 4: Predicting consequences of changeSolution: Scheme validation
56
JSON files
JSON scheme
Problem 4: Predicting consequences of changeSolution: Scheme validation
57
JSON files
JSON scheme
Configuration file
Problem 4: Predicting consequences of changeSolution: Scheme validation
58
JSON files
JSON scheme
Configuration file
Backend request
Problem 4: Predicting consequences of changeSolution: Scheme validation
59
JSON files
JSON scheme
Configuration file
Backend request
Remote validation
Problem 4: Predicting consequences of changeSolution: Scheme validation
60
Configuration file
Remote validation
JSON scheme
API response
JSON file
Problem 4: Predicting consequences of changeSolution: Scheme validation
61
Configuration file
Remote validation
JSON scheme
API response
JSON file
Problem 4: Predicting consequences of changeSolution: Scheme validation
62
Configuration file
Remote validation
JSON file
JSON scheme
API response
63
Problem 4: Predicting consequences of changeSolution: Functional UI tests with mocked API
faster, more reliable UI tests
64
Problem 4: Predicting consequences of changeSolution: Functional UI tests with mocked API
faster, more reliable UI tests
relatively slow test coverage progress
65
Problem 4: Predicting consequences of changeSolution: Functional UI tests with mocked API
faster, more reliable UI tests
relatively slow test coverage progress
tests not critical to merge
66
Problem 4: Predicting consequences of changeSolution: UI tests improvements
More bugs detected10%comparison of 10 real API and mocked API UI test runs
Less flaky tests70%comparison of 10 real API and mocked API UI test runs
67
1. Static analysis
2. Mutation testing
3. Snapshot tests
4. Functional tests
Tests are like onions
5. Continuous integration
6. Phased release
7. AB Tests
8. Application analytics
68
Problem 5: Late feedbackSolution: Continuous integration
Problem 5: Late feedbackSolution: Continuous integration
69
Problem 5: Late feedbackSolution: Continuous integration
70
Get as many tester as possible
Automate your beta releases
Highlight what should be checked
71
Problem 5: Late feedbackSolution: Continuous integration
https://github.com/fastlane/fastlane
72
1. Static analysis
2. Mutation testing
3. Snapshot tests
4. Functional tests
Tests are like onions
5. Continuous integration
6. Phased release
7. AB Tests
8. Application analytics
73
Problem 6: Testing in laboratory condition
XXX
“Application doesn’t start after the update”
74
Problem 6: Testing in laboratory conditionSolution: Phased release
There are situations you cannot predict
Prepare for failure
75
Remote configuration
Problem 6: Testing in laboratory conditionSolution: Phased release
76
1. Static analysis
2. Mutation testing
3. Snapshot tests
4. Functional tests
Tests are like onions
5. Continuous integration
6. Phased release
7. AB Tests
8. Application analytics
77
Problem 7: Testing effectiveness
BUY NOW BUY NOW
78
Problem 7: Testing effectivenessSolution: AB Tests
79
Problem 7: Testing effectivenessSolution: AB Tests
80
https://motherboard.vice.com/en_us/article/8x8wp4/why-navigation-bar-looks-different-facebook
81
1. Static analysis
2. Mutation testing
3. Snapshot tests
4. Functional tests
Tests are like onions
5. Continuous integration
6. Phased release
7. AB Tests
8. Application analytics
82
Problem 8:
Application release
83
Problem 8:
https://fabric.io/kits/ios/crashlytics
Application release
Stable version
84
Problem 8:
Application release
Stable version
Comment on App Store
3
85
Problem 8:
“Where is categories button? I can’t find it!”
86
Problem 8:
“Where is categories button? I can’t find it!”
87
Problem 8: Decision making
“Where is categories button? I can’t find it!”
88
Problem 8: Decision makingSolution: Make use of app analytical data
89
Problem 8: Decision makingSolution: Make use of app analytical data
Data-based decisions
90
Problem 8: Decision makingSolution: Make use of app analytical data
Data-based decisions
Bug priority and severity
91
Problem 8: Decision makingSolution: Make use of app analytical data
Data-based decisions
Bug priority and severity
92
Analyze and plan your reaction
Problem 8: Decision makingSolution: Make use of app analytical data
93
1. Static analysis
2. Mutation testing
3. Snapshot tests
4. Functional tests
Tests are like onionshttps://bit.ly/2EWg91Y
5. Continuous integration
6. Phased release
7. AB Tests
8. Application analytics
94
Thank you for your attention!
Q&A
Ewa LudwiczakSenior Software Test [email protected]
@ewabielskapoz
Kamil PyćSenior iOS [email protected]
@KamilPyc
