Upload
arnon-axelrod
View
92
Download
3
Embed Size (px)
Citation preview
3 Copyright © 2012 Retalix |
About you• Name• Company• Role• Number of people in the project• Average release cycle• % of man-months dedicated to bugs finding and
fixing• From 1 to 10 – how are you satisfied with the
quality of the software you deliver?
6 Copyright © 2012 Retalix |
The quality pyramid• Intuitive, sleek, “sexy”,
enjoyable…
• User friendly, HelpUseful
• Stable• Supportable• Secure
Reliable
• Useful features• Solves a user’s
problemUsable
Appealing
7 Copyright © 2012 Retalix |
The quality pyramid – “reality”• Intuitive, sleek, “sexy”,
enjoyable…
• User friendly, HelpUseful• Stable• Supportable• Secure
Reliable
• Useful features• Solves a user’s
problemUsable
Appealing
8 Copyright © 2012 Retalix |
The quality pyramid• Intuitive, sleek, “sexy”,
enjoyable…
• User friendly, HelpUseful
• Stable• Supportable• Secure
Reliable
• Useful features• Solves a user’s
problemUsable
Appealing
SbE / BDD / ATDD
11 Copyright © 2012 Retalix |
What is a bug?Wikipedia:“A software bug is an error, flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways”
12 Copyright © 2012 Retalix |
Why there are bugs?• Expectations
gaps• Miscommunicatio
n• Coding errors• Miscommunication
between teams or team members
• Misunderstanding the business needs
• Incomplete or incorrect
definition of the expected results
• Complexity• Misuse of tools• Bugs in tools
13 Copyright © 2012 Retalix |
Bugs - conclusion
By improving the communication and expectation coordination – we can prevent
many bugs
20 Copyright © 2012 Retalix |
The Agile ManifestoWe are uncovering better ways of developingsoftware by doing it and helping others do it.Through this work we have come to value:
Individuals and interactions over processes and toolsWorking software over comprehensive documentation
Customer collaboration over contract negotiationResponding to change over following a plan
That is, while there is value in the items onthe right, we value the items on the left more.
21 Copyright © 2012 Retalix |
Different names• Specification by Example• ATDD (Acceptance Test Driven Development)• Agile Acceptance Testing• Example Driven Development• Executable Specifications• BDD (Behavior Driven Development)
23 Copyright © 2012 Retalix |
User Stories“are a promise for a future conversation”
– Ron Jeffries As a <role/persona>I want <goal/desire> so that <benefit>.
In order to <receive benefit>as a <role/persona>I want <goal/desire>
As a user closing the appI want to be prompted to save my workso that I won’t loose all I’ve done so far
In order not to loose all I’ve done so faras a User closing the appI want to be prompted to save my work
26 Copyright © 2012 Retalix |
Specifying Collaboratively• Attendees: BA, Developer, Tester• Looking for simplest solution
– break the task to small pieces that add value – work incrementally
• Focus on Simple Examples of User Scenarios
27 Copyright © 2012 Retalix |
Specifying Collaboratively – DoD
How will the user use the solution? (examples!)
How can we demonstrate it?
How will we test it?
Definition of Done!
28 Copyright © 2012 Retalix |
Automating the examples• Test should be as close as possible to the DoD
– Certain tools (e.g. FitNesse) makes it easy!
• Test must fail before the code is developed…
29 Copyright © 2012 Retalix |
Developing the User Story Develop the fixtures first (follow the errors) Write the minimal code to pass the tests
Don’t add functionality that wasn’t requested! Apply TDD/Unit tests During and After development run all tests to make
sure nothing is broken Refactor as needed (remove duplication, give
meaningful names, etc.) Emergent design Make sure again that nothing has broken
30 Copyright © 2012 Retalix |
Running the tests Locally, before check-in
Automatically (CI, Nightly)
Provide transparency!
Should always be green!
31 Copyright © 2012 Retalix |
Living documentationAutomatic Specifications = Living Documentation!
Benefits:• A single, central source of truth• Provides visibility about the state of the project• Easy to identify contradicting requirements
32 Copyright © 2012 Retalix |
Living documentationAutomatic Specifications = Living Documentation!
How?• Maintain the scenarios (+some explanation text)
in a shared location• Organize in a logical manner
– Reorganize as needed
33 Copyright © 2012 Retalix |
Benefits of Specification by Example• Full automated test coverage
– Makes refactoring easy and safe!• Helps keeping the code clean all the time!
– Regressions are caught by the developer before check-in!• Virtually no bugs!
• Tests are easy to understand and maintain– Provide a comprehensive documentation which is always up to
date!– When conflicting requirements are caught (during
development), it’s easy to communicate it to the business people
34 Copyright © 2012 Retalix |
Benefits of Specification by Example• Trust is built among stakeholders• Forming an ubiquitous language• Less rework – eliminate waste• Facilitate change
– Short delivery cycles• Receive feedback earlier!• Respond to new requirements quickly
• Delivering the RIGHT Software!
37 Copyright © 2012 Retalix |
Existing application – specifications:• Purpose: Point-of-Sale application• Can sell products from catalog• 2 Screens:
– Selling screen– Products catalog
38 Copyright © 2012 Retalix |
Existing application – specifications:• Products catalog screen:
• Allows to create, delete and update products in the catalog• Each product has:
• Name• Is it sold it units or by weight?• Price per unit or Kg
• Validations:• Name is unique and non-empty• Price > 0
39 Copyright © 2012 Retalix |
Existing application – specifications:• Selling screen:
• Only one selling at a time (no history)• Can add, edit and remove lines• Each line contains:
• Product name• Quantity
• For unit products – must be an integer• For weighable products – can be decimal
• Price – calculated by Product’s price * quantity
40 Copyright © 2012 Retalix |
Existing application – specifications:• Selling screen (cont.)
• A product can appear in more than 1 line• The total is displayed at the bottom• “Next Customer” button:
• Enabled only there’s at least 1 line• Displays “Thank you for buying from us!”• Cleans the screen to prepare for next customer
• “Edit Products” button:• Enabled only when the selling is empty• Opens the Products Catalog screen
41 Copyright © 2012 Retalix |
New Requirement:• Support Promotions from the following type:
– Given Bamba costs 4 NIS– Promotion: “Buy 3 Bamba in 10 NIS”
42 Copyright © 2012 Retalix |
The Game Flow (phase 1)• We’ll split into 3 groups:
– Business Analysts– Developers– Testers
• Instructor represents the customer• Each team will have 5 minutes to complete their
role• The other groups are waiting outside the
classroom
43 Copyright © 2012 Retalix |
The Game Flow (phase 1)• Business Analysts role:
– Write a specification document for the requirement• according to their understanding of the customer’s
requirement• Can ask the Customer clarification questions
– Provide the specification document to the developers and testers
44 Copyright © 2012 Retalix |
The Game Flow (phase 1)• Developers role
– Read the Specifications written by the B.A.– Write a detailed flowchart or pseudo-code as needed– Specify all the changes that have to be done to the app
45 Copyright © 2012 Retalix |
The Game Flow (phase 1)• Testers role
– Write a test plan according to the Spec the B.A wrote– Each test should contain clear steps and expected
results– The goal of the test plan is to find as many bugs as
possible in the Developer’s “code”
46 Copyright © 2012 Retalix |
The Game Flow (phase 1)• Final stage:
– The instructor will “execute” the tests on the “Code”• Will write which tests passed and which failed
– The instructor (as the “Customer”) will perform additional tests and will indicate number of bugs and severity
– Note: In case we won’t find any serious bugs, we’ll repeat the game with more complex requirements
• Phase 2 of the game will be explained later…
48 Copyright © 2012 Retalix |
The Game Flow (phase 2)Simulation of Specification by Example:
• All groups work together• Specify using examples• Can ask the Customer for clarifications
51 Copyright © 2012 Retalix |
Manual Acceptance Tests• Tests that are conducted by the Customer or
Business Analyst to verify that a new requested feature or User Story is implemented as expected
• Usually done thought the UI
52 Copyright © 2012 Retalix |
Manual Regression Tests• Manual tests, (usually conducted by a tester) to
verify that old functionality still works
• Usually done though the UI
53 Copyright © 2012 Retalix |
Test Case (Scenario)• A test case represents a single scenario• A scenario is composed of steps that interact with
the system and verifies its behavior
55 Copyright © 2012 Retalix |
Test Case - Example• Create an Item called “Bamba” with price $3• Begin a selling transaction• Sell 2 “Bamba”• Ensure that the total is $6• Pay by cash• Ensure that the transaction is closed and its
balance is 0
56 Copyright © 2012 Retalix |
Automatic Acceptance Test• An automatic test that verifies that a new
requested feature or User Story is implemented as expected
• After the test passes once, it becomes an automatic Regression test
• Automatic UI Testing has high maintenance costs…
57 Copyright © 2012 Retalix |
Test Scope• Defintion1: The parts of the system and their
configurations that are tested by a particular test or group of tests
• Definition2: Everything that is not under the control of the test
58 Copyright © 2012 Retalix |
Test Scope
End-to-End
UIView ModelClient LogicServer Proxy
Service LayerBusiness
LogicDALORM
DB
59 Copyright © 2012 Retalix |
Test ScopeUI
View ModelClient LogicServer Proxy
Service LayerBusiness
LogicDALORM
DB
Usability, UX
60 Copyright © 2012 Retalix |
Types of TestsUI
View ModelClient LogicServer Proxy
Service LayerBusiness
LogicDALORM
DB
Integration
61 Copyright © 2012 Retalix |
Test ScopeUI
View ModelClient LogicServer Proxy
Service LayerBusiness
LogicDALORM
DBFunctional
Mock
Mock
Mock
62 Copyright © 2012 Retalix |
Test ScopeUI
View ModelClient LogicServer Proxy
Service Layer
DALORM
DB
Unit tests
Business Logic
63 Copyright © 2012 Retalix |
Test Scope• Smaller Scopes ensures that the tested
functionality is handled only inside this scope and nowhere else (loosely-coupled, lower maintenance costs). – It aids diagnosing the root cause of problems– Faster
• Larger scope verifies that the system behaves correctly as a whole
64 Copyright © 2012 Retalix |
Writing Good Automatic Acceptance Tests
• Create an abstraction between the test case and the Scope.– The Test Layer reflects the scenario in the most
readable way. • The Test Layer is agnostic to the Scope of the Test
– The Automation Layer is an adaptor between the Test Layer and the SUT• The Automation Layer is bound to the Scope of the
Test
65 Copyright © 2012 Retalix |
Automatic Tests Architecture
Automatic Test Case (Test Layer)Abstraction Layer
Automation Layer
UIView ModelClient LogicServer Proxy
Service LayerBusiness
LogicDALORM
DB
66 Copyright © 2012 Retalix |
Choosing the right scope• Some scopes are easier to automate than others• Some scopes are more likely to change than
others– Causing higher maintenance costs of test code
Choose the scope that covers most of the risk, while still easy to
maintain
67 Copyright © 2012 Retalix |
Complementary Testing Strategies• Manual Acceptance Tests• Exploratory tests• TDD / Unit tests• Usability tests• Performance tests• Load and scalability tests
69 Copyright © 2012 Retalix |
• “If I had asked my customers what they wanted they would have said a faster horse.”
– Henry Ford
71 Copyright © 2012 Retalix |
Deriving the Solution from the Goals• Collaboration empowers creativity and simplicity• Ask “Why?” and “Who?”
– Dig deep until you reach the “money” answer• Focus on the outputs first, only then on the inputs• If you’re given a solution, ask:
– How it would be useful?
– How will this solution help you achieve the goal better than what you’re doing now?
73 Copyright © 2012 Retalix |
Specifying Collaboratively• Benefits of Collaboration:
– Encourages Creativity– Prevents blind spots– Creates shared understanding– Promotes ubiquitous language– Produces specifications that are easy to understand – Produced tests that are easy to maintain
74 Copyright © 2012 Retalix |
Tips for effective collaboration• Consider having introductory meetings• Make sure that everyone have the necessary
background• Make sure that participants have the business
answers– Involve the relevant stakeholders– Product owner to get as much information as possible
before hand– Don’t over-prepare!
77 Copyright © 2012 Retalix |
Illustrating using examples• Examples is probably the best way to ensure
understanding• Examples should be precise – but not over-
detailed– Focus on the relevant domain concepts and Business
Rules– Avoid technical details like specific UI details
• Expected result should be specific and measurable as possible
78 Copyright © 2012 Retalix |
Illustrating using examples• Examples should be realistic
– But can be simplified to focus on the point at hand and be easy to understand
• Examples and expected results should be described using the user’s perspective– Avoid relying on database or logs (unless the logs are aimed for the
user)• Add a meaningful title to an example or a set of example
and a short description• Tip: start with basic examples and explore what’s missing• Given-When-Then!
79 Copyright © 2012 Retalix |
Illustrating Non-Functional Requirements
• Many “nonfunctional” requirements can be expressed in deterministic examples and measurements– E.g. performance…
• Qualitative non measurable requirements (e.g. usability) can be expressed manually using examples of “good” and “bad”
• Use UI sketches• Use check-lists for cross-cutting concerns
80 Copyright © 2012 Retalix |
Writing Good Executable Examples• Use Business Domain language• Describe from User’s perspective
– Think about different roles/personas• Focus on the relevant details!
– Hide irrelevant details in the automation layer– Focus on the Business Logic, not on the UI
81 Copyright © 2012 Retalix |
Writing Good Executable Examples• Start from the Happy-Path and explore• Avoid having dependencies on preconfigured,
predefined, shared data• Have automatic tests run in an isolated
environment
83 Copyright © 2012 Retalix |
Improving Executable Examples – SolutionName: Promotion Can Add Points To CustomerDescription: A promotion can add points to a customer (which he can use later in other promotions). The user can examine his points balance in the web-site.Steps:• Define item: bamba (price 3NIS)• Define promotion: For every 2 bamba – add 200 points to the customer account• Create new customer “Arnon”, with stating balance of 100• Begin selling for customer “Arnon”• Add: 5 bamba• Pay total• Check receipt shows added points: 400• Check balance for customer “Arnon” is 500 (though web-site)
85 Copyright © 2012 Retalix |
Automating the Examples to match the Specifications
• Choose the Tool appropriate for you– Features and ease-of-use– Price– Programming Language
• Choose the automation scope
92 Copyright © 2012 Retalix |
Validating Frequently• Run the tests automatically• Run Frequently
– Most Preferred: Continuous Integration– Nightly– Weekly– Iteration– Least Preferred: Release
93 Copyright © 2012 Retalix |
Validating Frequently• Separate short and long tests• Allow developers to run the tests locally before
check-in• Optimize tests• Run tests in parallel (on different
machines) to reduce time
94 Copyright © 2012 Retalix |
Reliable Automation• Same Code Quality as Production Code!• Fix test issues quickly!• Use isolated environment• Ban dependencies between tests• Avoid shared data• Avoid relying on time
– Rely on events rather than timeouts– Mock system time
95 Copyright © 2012 Retalix |
Provide transparency• Provide High-Level Report to management• Provide detailed report for failing tests for
Developers– Make it easy to find the root cause
• Mark failing tests for incomplete user stories differently from regression failures
96 Copyright © 2012 Retalix |
Provide Transparency• Manage failing tests
– Avoid marking tests as “Ignore”– Make tests that fail due to known bugs so that they’ll
appear in reports• Link to other ALM artifacts
98 Copyright © 2012 Retalix |
Living Documentation should be Easy to Understand• Executable Examples should be:
– Readable and clear for Business People– Focused
• Provide some textual/graphical explanations as needed
• Evolve the Language and “Refactor” to make it consistent
99 Copyright © 2012 Retalix |
Organizing the Executable Specifications• Specifications should be easy to find• Create links between tests/concepts (Wiki)• Have a primary hierarchical structured according
to functional areas– Evolve it as needed
• Use Tags for easy searching• Have secondary indexes for other categorizations
101
Copyright © 2012 Retalix |
General Advice…• Know the benefits of the new process• Know the benefits for each stakeholder
– But also their concerns• Listen! Listen! Listen!
– Get to know your organization and business• Have a clear vision!• Be enthusiastic and catch others!• Once you started – be determined!
102
Copyright © 2012 Retalix |
Benefits for Business Analysts• Developers will actually read and understand the
specifications– Developers won’t skip parts of the specifications
• You can track development progress easily• You can easily identify conflicts in business rules
103
Copyright © 2012 Retalix |
Challenges for Business Analysts• I don’t do tests – that’s not my job• It’s extra work• They will only look at the tests and won’t read the
requirements• What if I leave something out?• I can’t cover all the specifications with Examples
104
Copyright © 2012 Retalix |
Benefits for Testers• You’ll be involved from the start• You’ll build quality in from the start• You’ll have more influence of the developed
product• You will have much better understanding of the
domain• You’ll delegate a lot of dull work to developers• You’ll be more free for exploratory testing• Better relationships with the Developers
105
Copyright © 2012 Retalix |
Challenges for Testers• Will I loose my job because of automation?• Automated Tests won’t catch all problems• I don’t want to loose control over testing• It’s cheating – developers will know in advance
what I’m going to test…
106
Copyright © 2012 Retalix |
Benefits for Developers• Functional gaps will be flushed out before
development• Business Analysts will really understand special
cases• Automated tests will be your target for
development• You’ll have a clear Definition of Done• You’ll be able to give better estimations
107
Copyright © 2012 Retalix |
Challenges for Developers• I don’t have time to write fixtures or maintain
tests• Acceptance Tests turn out to be incomplete• Can’t we just write the tests ourselves?• We already write unit tests
108
Copyright © 2012 Retalix |
Tips for implementing SbE• Ride on a wider process change (if applicable)• Start small and show results• Focus on improving quality• Focus on reducing TTM• Engage management• Start from test automation and improve• Create a small “Sanity” automation suite to show
benefits quickly
109
Copyright © 2012 Retalix |
Tips for changing the process• Avoid testing terminology while talking to non-testers• Don’t focus on the tool
– Introduce a free tool instead of an expensive one…– Avoid fighting on the right tool
• Avoid buzzwords and “Agile” terminology• Leverage transparency in order to enforce • Conduct effective retrospectives to learn how to
improve!• Listen carefully to what people tell you!
111
Copyright © 2012 Retalix |
Conclusion
• What did you learn today?• What will you take with you from this workshop?
• Make a commitment!
• Verifying the Definition of Done
113
Copyright © 2012 Retalix |
Resources• Gojko Adzic, Bridging the Communications Gap:
Specification by Example and Agile Acceptance Testing (Neuri, 2009)
• My blog: • Gojko Adzic, Specification By Example: How
successful teams deliver the right software (Manning, 2011)
• My blog: http://blogs.microsoft.co.il/blogs/arnona