Upload
dashawn-skipwith
View
219
Download
1
Tags:
Embed Size (px)
Citation preview
Microsoft’s Protocol Documentation Program:
A Success Story for Model-Based TestingYiming Cao, Lead Software Design Engineer Windows Interoperability Engineering Team, Microsoft Corporation
Dagstuhl Seminar, October 2010, Leibniz-Zentrum für Informatik, Germany.
Part 1: Background
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 2
Microsoft Open Protocols
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
http://www.microsoft.com/protocols
9/4/2010 3
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
“BlueLine” Technical Document Testing Program of Windows (as of 03/09) 222 protocols/technical documents tested
22,847 pages studied and converted into requirements
36,875 testable requirements identified and converted into test assertions 69% tested using MBT 31% tested using traditional test automation
66,962 person days (250+ years) Hyderabad: 250 test engineers Beijing: 100 test engineers
9/4/2010 4
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Formalization
Technical Document Test Suite
EnglishStatements(Requirements)
AssertionsChecked against Windows(Accuracy)
• Validate consistency
• Simulate implementation process
Test-Driven Document Quality Assurance
9/4/2010 5
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Study
• Document study• Test strategy
(modeled or traditional)
• Requirements spec (initial)
• Estimates of Effort
Plan
• Requirements spec (final)
• Test plan• Architecture• Scenarios
Design
• Test design• Model, or• Test case
descriptions• Adapter
• Design
Final
• Test case code• Generated, or• Manual
• Adapter• Implementatio
n• Test logs and
report• Network
captures• Test setup
artifacts
Process: Managing a Project with 350 Test Engineers
Review after every phase• High-profile reviewer team consisting of industry experts and Microsoft
senior engineers, build along the model of a program committee• Review based on process report and test suite artifacts• Verdict accept, conditional accept, reject
9/4/2010 6
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 7
Requirements TrackingTechnical Document
Requirements Spec
Model or Test Design
Test Suite
Logs
Network Captures
Statements identified in document and tracked with requirement ID
IDs referenced from model or test assertion
IDs logged at test execution time
IDs send to network as beacon packages for network monitor capturing
manual
automated
9/4/2010
Typical Requirements Specification
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Verification Comment
Use this column to explain the reason why the requirement is non-verifiable or unverified (for Normative, Server requirements).
Req ID Doc Sect Description Po
sNeg
Derived
Scenarios Behavior ScopePriorit
y
Informative/
NormativeVerificatio
n
Verification
Comment
MS-CHAP_R14
3.2.5When a peer receives a Failure-Request message, if the embedded MSCHAPv2 packet indicates a password expiration error, the peer MAY reply with a Change-Password-Response message.
Protocol Client p2 Normative Unverified Client behavior not verified.
MS-CHAP_R15
3.3.1In addition, the authentication phase is terminated by sending an EAP-Success/EAP-Failure packet to the peer.
S1,S2,S3,S4,S5,S6
Protocol Server p2 Informative
Test Case
MS-CHAP_R16
3.3.5An EAP server MUST transmit a Challenge-Request message to initiate an Extensible Authentication Protocol Method for Microsoft CHAP authentication procedure.
S1,S2,S3,S4,S5,S6,S7,S8
Protocol Server p0 Normative Test Case
9/4/2010 8
Part 2: Model-Based Test Suite Development
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 9
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 10
Model-Based Document Analysis
Model
Implementation
Expected Outputs (Test
Oracle)
Inputs(Test
Sequences)
Control
Document
Generate
Observe
Feedback
Feedback
Author
Issue
Feedback
Test Suite
Verdict
Feedback
9/4/2010
Process Drilldown
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
PartiallyGenerated from Doc
Technical Document Issues (TDIs) filed in all phases
9/4/2010 11
Gathering Requirements
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
9/4/2010 12
Developing and Implementing Adapters
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
9/4/2010 13
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 14
Adapter Interface
TCTest Cases(traditional or MBT)
TC TC
TC TC TC
Adapter Implementation
Test Development
Adapter Development
SUT(System under test)
• Abstracts SUT functionality
• Contract between teams• Test case team• Adapter team
• Pluggable• Different server
setups• Different transports
Test Adapters
9/4/2010
Develop Model and Generate Tests
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
Potential dependencyon data types
9/4/2010 15
Spec Explorer 2010 Technology Breakdown Model programs
Guarded state update rules Rich object-oriented model state (collections, object
graphs) Language agnostic (Based on .Net intermediate
language interpretation)
Trace patterns Regular style language to represent scenarios Slicing of model program by composition
Symbolic state exploration and test generation Expands parameters using combinatorial interaction
testing Extracts a finite interface automaton (IA) from composed
model Traverses IA to generate standalone test code –or- Runs on-the-fly tests from IA
Integrated into Visual Studio 2010Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 16
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 17
Spec Explorer 2010 Look & Feel
Explore
Analyze
Generate
Execute
C# Model (or other .Net Language)
Model Graph
Test Suite
VSTT Result
9/4/2010
Spec Explorer Walkthrough: Chat Example
Users can Enter the session Exit the session List all session users Broadcast a
message Received by all session
users
18
Chat Room Server
User1
SUT
User2 User3
What happens if two users broadcast a message at virtually the same time?
Chat Requirements
19
R1: User MUST receive response for logon request
R2: User MUST receive response for logoff request
R3: User MUST receive response for list request
R4: List response MUST contain the list of logged-on users if successful
R5: All logged-on users MUST receive broadcast message
R6: Messages from one sender MUST be received in order
20
Chat traces
Is the trace correct?
(Assume two users (user1, user2) are logged on)
T1Broadcast(user1,”1”)Broadcast(user2,”2”)BroadcastAck(user2,”1”)BroadcastAck(user1,”2”) BroadcastAck(user1,”1”)BroadcastAck(user2,”2”)
T2Broadcast(user1,”1”)Broadcast(user2,”2”)BroadcastAck(user1,”2”)BroadcastAck(user1,”1”) BroadcastAck(user2,”1”)BroadcastAck(user2,”2”)
T3Broadcast(user1,”1”)Broadcast(user2,”2”)BroadcastAck(user1,”2”)BroadcastAck(user2,”2”) BroadcastAck(user1,”1”)BroadcastAck(user2,”1”)
If each user sends one message, any receiving order is correct!
21
Chat traces
Is the trace correct?
(Assume two users (user1, user2) are logged on)
T4Broadcast(user1,”1a”)Broadcast(user1,”1b”)BroadcastAck(user2,”1a”
)BroadcastAck(user1,”1a”) BroadcastAck(user1,”1b”)BroadcastAck(user2,”1b”)
T5Broadcast(user1,”1a”)Broadcast(user1,”1b”)BroadcastAck(user1,”1a”)BroadcastAck(user2,”1b”) BroadcastAck(user1,”1b”)BroadcastAck(user2,”1a”)
Local order consistency: messages sent by one user must be received in order
22
Chat Slice where local consistency does not matter
23
Chat Slice where local consistencydoes matter
24
Spec Explorer’s Conformance Notion
Alternating simulation: SUT must “simulate” all stimuli of model Model must “simulate” all responses of SUT (responses are
buffered)
Broadcast(1,”1”)
Broadcast(1,”2”)
Model SUT
Broadcast(1,”1”)
Broadcast(1,”2”)
BroadcastAck(1,”1”)
BroadcastAck(2,”1”)
BroadcastAck(1,”1”)
BroadcastAck(2,”1”)
BroadcastAck(2,”1”) BroadcastAck(1,”
2”)
Spec Explorer’s Approach to State ExplosionSlicing techniques:
Parameter selection (Constraints, Pairwise, etc.)
State filtering Trace patterns Requirement coverage
Slicing requires human intervention Smart slicing is an art
Spec Explorer tailored for slicing design25
26
Where a Trace can end: Accepting State Condition
Is the following a valid word?Micr
Does the following trace represents a useful test?Broadcast(1,”1”); Broadcast(1,”1”)
An Accepting state condition characterizes those states in which a trace can end Used to ensure that a trace does not stop at arbitrary
points Used to ensure that a test leaves the system in a good
state
Accepting state condition for Chat: All messages have been delivered to recipients
Demo: Chat Model
What the demo shows OO model state Using helper methods in models Modeling with events Slicing
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Demo: Server Message Block Protocol v2What the demo shows: Modeling a key Microsoft product Using C# to model state transition
systems Using trace patterns to slice test
purposes from models Generating and executing tests
9/4/2010 28
Part 3: Evaluation
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 29
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 30
Comparison MBT vs Traditional
Vendor 1 Vendor 20
10
20
30
40
50
60
70
80
90
100
5866
100 100
Model-BasedTraditional
• In % of total effort per requirement, normalizing individual vendor performance
• Vendor 2 modeled 85% of all test suites, performing relatively much better than Vendor 1
9/4/2010
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Remarks on Comparison
Numbers backed up by correlation analysis conducted by empirics expert
Engineers applying MBT were lay folks, most of them freshly hired from college
Few power modelers achieve much higher efficiency rates
Though model authoring takes only a smaller part of overall effort per requirement, modeling approach seems to determine efficiency
9/4/2010 31
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Factors for Success Tool support
Modeling in mainstream languages and environments
Visualization of exploration and actual generated test suite (no magic as in online testing)
Dealing with state explosion by scenario slicing Training and Guidance
One week class for every new hire (including both general methodology and model based testing)
Size of project generated critical mass for community help
Reviewing process added additional guidance
9/4/2010 32
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 33
Pain Points Observed
Delta Testing Support A new set of test cases generated based on a
new version of model: Which old cases can be re-used? Which old cases are not valid anymore? Which new cases need to be added?
By-passing assertion at test execution time Push-button technology for requirement
coverage with minimum transitions Good heuristics is needed when behavior is
infinite
9/4/2010
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 34
Pain Points Observed (continued) Ambiguity in Requirement Coverage
What does it mean to cover a requirement in model? Covering a transition once/multi-times? Or
even path coverage or pairwise coverage?
Rule Debugging Support C# provides rich programming
experience But rules can get so complex that a
debugger is needed to analyze what’s going wrong
9/4/2010
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Conclusions
Biggest application of behavioral model-based testing known so far
Probe big enough for relevant empiric results, proving efficiency gain of 42%
Will lead to more momentum on modeling in general inside of Microsoft
Model-based testing rocks!9/4/2010 35
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Advertisement…
Microsoft offers Spec Explorer 2010 as preview technology free of charge via Visual Studio Gallery:
http://visualstudiogallery.msdn.microsoft.com/en-us/271d0904-f178-4ce9-956b-d9bfa4902745 Licensing allows for commercial use Academic institutions can join MSDN
Academic Alliance for access to Visual Studio 20109/4/2010 36
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
9/4/2010 37