Upload
eurostar-conference
View
38
Download
1
Embed Size (px)
DESCRIPTION
EuroSTAR Software Testing Conference 2012 presentation from Amsterdam, Netherlands.
Citation preview
Protecting the irreplaceable | f-secure.com
Olli-Pekka Puolitaival
• Lead quality engineer test automation
• 5 yrs test automation
• 4 yrs research
• Idea of model-based testing
• Story of OSMO tester
• How we test
• Experiences & tips
Index
MBT is new way to think
• Traditional way to think
• People do testing
• Regression automation reduce peoples work time
• New way to think
• Smart people want to do testing in smart way
• Computers are fast and hard worker, let them also try “stupid” things
5
6
7
8
9
10
11
12
13
Traditional MBT tools
• Hard to take in use
• Closed
• Limited extending
• Advanced algorithms
• Expensive
Osmo tester
• Easy start
• Flexible
• Easy to extend
• Simple algorithms
• Free (open source)
14
Why we made osmo?
Osmo tester idea
public class ExampleModel{
int i = 0;
@Guard(“increase") public guardfunction() { return true; //Return true if this transition is allowed to execute } @Transition(“example") public void transitionfunction() { System.out.println(i++); //This code is executed when transition is selected }
… public static void main(String[] args) { OSMOTester tester = new OSMOTester(new ExampleModel()); tester.generate(); }
}
15
“Model is pure java code, Which makes possible to use Any java library”
Many additional features
• Support online and offline MBT
• Requirements tagging
• Coverage calculations
• Many test design algorithms, also manual drive
• Coverage, time and length based stopping criteria
• Self written adapters to the SUT
• Custom test output formats
• Easy to integrate with Jenkins
• Model reuse and combining
• Probability numbers support
• Available: http://code.google.com/p/osmo/
16
Easy to start and custom for your needs
Tips
• Modelling
• Good model is high level, modular and cover core functionality
• Start from small and continue iteratively
• Adaptation
• Reuse code from test automation or development if possible
• Running
• Run tests always automatically
• Fixed run length for verifying and ”unlimited” for finding some new
• Reports
• Report what you need to know, visual is mostly better
• OSMO is not good if you need very sophisticated algorithms, but most people don’t need those
17
The domain is content cloud
Cli
ent
smo
ke
Our continuous integration
19
Backend components
Backend components
Clients
Backend components
Clients
Client Integration
test environment
Clients
Un
itte
st
Bu
ild
ing
Backend integration
Test Environment
Smo
ke
test
Reg
ress
ion
te
st
Pe
rfo
rman
ce
tes
t
Performance
Test Environment
Cli
ent
reg
ress
ion
Chain of Jenkins jobs runs everything
Jenkins
Android phone
Android client
CAN Solution Backend
Current test setup
Selenium webdriver
Firefox
Test status radiator
Web adapter
Nativedriver client
Nativedriver server
Model
OSMO
Android adapter
“run once per hour”
What kind of model?
• Idea of this testing
• Do actions and verify that other clients data is same
• Model includes actions which can be seen in other client like
• Upload a file
• Remove a file
• Rename a file
• Check quota
• Check that file exist
• Check that all files exist as supposed
21
Example test run
22
Osmo test engine Android Client Web page
Login
Upload file1.jpg
Login
Check file1.jpg
Upload file2.jpg
Remove file1.jpg Check that only file2.jpg exist
Logout
Logout
What are changing in test runs
• New file set each time
• New test accounts each time
• Random amount of uploaded files
• Randomly removed files
• Randomly renamed files
• Random checks
23
Experiences
• Useful testing because
• Tells well the status of core functionality
• Is very close the new user use case
• Test generation is easy to do with OSMO
• Modeling is easy and support well iterative development
• Test parameterization is easy
• The challenges
• Adapters are not easy to do and maintain
• I keep my testing enough simple, then I don’t need too much maintenance
• Automating test run was challenging in first time
• Developers was helping me
24
Next steps
• Look for new adapter for android
• Extend testing on other platforms
• Extend the power
• Cover many environments
• Many machines for testing
• Run continuously several kind of test sets:
• Short run for radiator
• “infinite run” for finding weakest point
25
My MBT cooking receipt
1. Find out the core functionality
2. Make a simple model of that
3. Make adaptation for test running
4. Automate test runs
5. Run model continuously against SUT
6. Put the results visible in your room for seeing the “temperature”
7. Keep the radiator GREEN
26