Black Box Software Testing

  • Published on
    03-Sep-2014

  • View
    3.283

  • Download
    4

Embed Size (px)

DESCRIPTION

 

Transcript

  • Black Box Software Testing Part 15. Testing User Documentation Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 527
  • Copyright Notice These slides are distributed under the Creative Commons License. In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one. For the rest of the details of the license, see http://creativecommons.org/licenses/by- sa/2.0/legalcode. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 528
  • Documentation is an Express Warranty A warranty is a statement of fact, either articulated or implied by law, respecting the quality or character of the goods to be sold. Under the Uniform Commercial Code an express warranty is: 2-313(a) Any affirmation of fact or promise made by the seller to the buyer which relates to the goods and becomes part of the basis of the bargain . . . 2-313(b) Any description of the goods which is made part of the basis of the bargain . . . 2-313(c) Any sample or model which is made part of the basis of the bargain. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 529
  • Documentation is an Express Warranty You cant disclaim an express warranty -- you are accountable for your claims. Uniform Commercial Code 2-316 (1): Words or conduct relevant to the creation of an express warranty and words or conduct tending to negate or limit warranty shall be construed whenever reasonable as consistent with each other; but . . . negation or limitation is inoperative to the extent that such construction is unreasonable. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 530
  • Black Box Testing: Testing Documentation Doc testing is important because: Errors in the manual increase risks of legal liability. Testing the documentation improves the reliability of the program. The documentation may be your mainstream test plan and your most up-to-date specification. Confusion in the manual reflects confusion in the programs design. Refer to Testing Computer Software, Chapter 10 Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 531
  • Testing Documentation: What to Test Verify every statement of fact and every reasonable implication Check the placement and accuracy of figures Audit the completeness of the manual (check that every feature is documented) Track errors in the documentation in a way that is normal for the Doc group. This probably doesnt involve the bug tracking system (but put code/doc mismatches there). If you give back marked up manuscripts, keep photocopies of your markups. Check your corrections against the next circulating draft of the manual. On average, you will cover 4 pages per hour in a reasonably stable program. Your second pass will go more quickly because the program is in better shape, but it will still take several minutes per page because you will actually test every page. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 532
  • Testing Documentation: Things to Say Your role is not editorial: you are not the authority on style and layout. Keep your tone non-judgmental. Point out upcoming changes (design changes, new error handling, data structures, etc.) that might affect the manual. Mark these in appropriate sections on the manuscript. Name experts or references to consult when the writer is confused. Suggest examples. Point to useful existing data files (for examples). Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 533
  • Testing Documentation: Things to Say When appropriate, you might do some writing. The writer might or might not use what you have written. You might write in two ways: words that you think belong in the manual as is (youre saying, Here, say it like this and it will be right.) explanations that are background material for the writer. Maybe a rough draft of what shell write. Note: the final review is a meeting just a few days before the book goes to the printer. If you have many small-detail late comments, offer the writer a chance to review them privately with you a few days before the review. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 534
  • Documentation Testing: On-Line Help Contents of the help Cross-reference jumps Glossary lookups Browse sequences Graphic hotspots Graphic display - color or resolution Window size, e.g. compile at 1024x768 and display at 640x480 Procedure sequences Balloon help / tool tips Index Search Context jumps Error messages Refer to Testing Computer Software, Chapter 10 Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 535
  • Publisher Liability for Content-Related Errors Winter v. G.P. Putnams Sons, 938 F.2d 1033, (9th Circuit) 1991. Winter became seriously ill from picking and eating mushrooms after relying on The Encyclopedia of Mushrooms, published by Putnam. Putnam did not verify the material in the book and did not intentionally include the error. Putnam was not liable for errors in the book. Noted that Jeppesen cases consistently held publisher liable for information error when information published was to be used as a tool. The court said that a software publisher might be liable for program that does not work as intended. ALM v. Van Nostrand Reinhold 480 NE.2d 1263, 1985. The Making of Tools. Plaintiff used it, and a tool shattered, causing injury. VNR not liable, but author of the book might be. Liability might also attach if the program provides professional services (such as tax preparation) and gives bad information. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 536
  • Warranties & Misrepresentations: What Must You Test? Advertisements Published specifications Interviews Box copy Fax-backs Manual Help system Warranty Web pages Readme Advice given to customers on Internet, CompuServe or AOL Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 537
  • Black Box Software Testing Part 16. Managing Automated Software Testing Several of these slides were developed by Doug Hoffman or in co-authorship with Doug Hoffman for a course that we co-taught on software test automation. Many of the ideas in this presentation were presented and refined in Los Altos Workshops on Software Testing. LAWST 5 focused on oracles. Participants were Chris Agruss, James Bach, Jack Falk, David Gelperin, Elisabeth Hendrickson, Doug Hoffman, Bob Johnson, Cem Kaner, Brian Lawrence, Noel Nyman, Jeff Payne, Johanna Rothman, Melora Svoboda, Loretta Suzuki, and Ned Young. LAWST 1-3 focused on several aspects of automated testing. Participants were Chris Agruss, Tom Arnold, Richard Bender, James Bach, Jim Brooks, Karla Fisher, Chip Groder, Elizabeth Hendrickson, Doug Hoffman, Keith W. Hooper, III, Bob Johnson, Cem Kaner, Brian Lawrence, Tom Lindemuth, Brian Marick, Thanga Meenakshi, Noel Nyman, Jeffery E. Payne, Bret Pettichord, Drew Pritsker, Johanna Rothman, Jane Stepak, Melora Svoboda, Jeremy White, and Rodney Wilson. Im indebted to James Whittaker, James Tierney, Harry Robinson, and Noel Nyman for additional explanations of stochastic testing. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 538
  • Overview About Automated Software Testing The regression testing paradigm 19 common mistakes 27 questions about requirements Planning for short-term ROI 6 sometimes successful architectures Conclusions from LAWST Breaking away from the regression paradigm and SQM, LLC. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 539
  • Overview In the mass-market software world, many efforts to automate testing have been expensive failures. Paradigms that dominate the Information Technology and DoD- driven worlds dont apply well to the mass-market paradigm. Our risks are different. Solutions that are highly efficient for those environments are not at all necessarily good for mass-market products. The point of an automated testing strategy is to save time and money. Or to find more bugs or harder-to-find bugs. The strategy works if it makes you more effective (you find better and more-reproducible bugs) and more efficient (you find the bugs faster and cheaper). I wrote a lot of test automation code back in the late 1970s and early 1980s. (Thats why WordStar hired me as its Testing Technology Team Leader when I came to California in 1983.) Since 1988, I havent written a line of code. Ive been managing other people, and then consulting, seeing talented folks succeed or fail when confronted with similar problems. Im not an expert in automated test implementation, but I have strengths in testing project management, and how that applies to test automation projects. That level of analysis--the managers view of a technology and its risks/benefits as an investment--is the point of this section. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 540
  • About Automated Testing Source of test cases Old Intentionally new Random new Size of test pool Small Large Exhaustive Serial dependence among tests Independent Sequence is relevant and SQM, LLC. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 541
  • About Automated Testing Evaluation strategy Comparison to saved result Comparison to an oracle Comparison to a computational or logical model Comparison to a heuristic prediction. (NOTE: All oracles are heuristic.) Crash Diagnostic State model and SQM, LLC. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 542
  • Issues Faced in A Typical Automated Test What is being tested? How is the test set up? Where are the inputs coming from? What is being checked? Where are the expected results? How do you know pass or fail? and SQM, LLC. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 543
  • The Complete Oracle Test Inputs Test Results Test Results Test Postcondition Data Precondition Data Oracle Postcondition Data Postcondition Precondition System Program State Program State Postcondition Under Program State Test Environmental Environmental Results Inputs Environmental Results Reprinted with permission of Doug Hoffman and SQM, LLC. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 544
  • Automated Software Test Functions Automated test case/data generation Test case design from requirements or code Selection of test cases Able to run two or more specified test cases Able to run a subset of all the automated test cases No intervention needed after launching tests Automatically sets-up and/or records relevant test environment Runs test cases Captures relevant results Compares actual with expected results Reports analysis of pass/fail and SQM, LLC. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 545
  • Characteristics of fully automated tests A set of tests is defined and will be run together. No intervention needed after launching tests. Automatically sets-up and/or records relevant test environment. Obtains input from existing data files, random generation, or another defined source. Runs test exercise. Captures relevant results. Evaluates actual against expected results. Reports analysis of pass/fail. Not all automation is full automation. Partial automation can be very useful. and SQM, LLC. Copyright 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 546
  • ...

Recommended

View more >