Test engineering foundation (v2.01)

Test Engineering FoundationEssential Knowledge for Test ProfessionalsRex BlackRBCS, Inc.31520 Beck Road Bulverde, TX 78163 USAPhone: +1 (830) 438-4830Fax: +1 (830) 438-4831www.rexblackconsulting.comrex_black@rexblackconsulting.com

Copyright (c) Rex Black 1996-2005

Introductions, Objectives,and OverviewTest Engineering Foundation Essential Knowledge for Test Professionals


Rex Black Consulting Services, Inc. Technical and Project Management Expertise for Quality

The CourseWho is this presenter?What is this course?What are the hours? breaks? Must I do exercises?Do I have homework?Can I get certified?May I use my cell phone or read e-mail during the course?Any others???



The MaterialsThese slides following the Foundation Syllabus 2005Terms come from the ISTQB Glossary (v1.1)In practice terminological variation is wideNot all terms in syllabus are defined in glossaryReference materials, glossary, and study aids for those pursuing certificationFor use in exercisesOmninet Marketing Requirements Document Omninet System Requirements Document Solutions for the exercisesA notepad on which to do the exercises



ISTQB Foundation Syllabus 2005Developed by a team of eight authors spanning seven countries: Thomas Mller (chair), Rex Black, Sigrid Eldh, Dorothy Graham, Klaus Olsen, Maaret Pyhjrvi, Geoff Thompson and Erik van VeendendaalForty primary reviewers spanning nine countriesFinal review and approval by 18 National BoardsDistills over 1,000 person-years of experienceThe ISTQB and the authors are the source of the syllabus (copyright 2005 by the authors and ISTQB) which is used by permission as the basis for this course



The Main Topics CoveredFundamentals of testingTesting throughout the software lifecycleStatic techniquesTest design techniquesTest managementTool support for testing



The ExercisesIn many of the exercises, you are working as a tester on the Omninet projectOmninet is a project to deploy a network of public access Internet kiosks in places like malls, theaters, and other public placesOn this realistic project, you will have a chance to apply many of the techniques we discussOther exercises have been added to illustrate specific points, too



Your Course ObjectivesThis material is designed to help you become a more effective and efficient test professional, and to help you obtain ISTQB certificationPlease spend the next few minutes writing down what youd like to get out of the courseUse the following page for your objectivesDont let the class end with an objective unfulfilled!



Course Objectives



Its Your CoursePlease join inAsk questionsMake commentsShare experiencesSecond opinions and disagreements welcomeWhy are you here?What do you want to learn, discuss, and teach?as well as this.The best sessions have a lot of this...


Chapter 1: Fundamentals of TestingTest Engineering Foundation Essential Knowledge for Test Professionals



1. Fundamentals of TestingWhy is testing necessary?What is testing?General testing principlesFundamental test processThe psychology of testing


Chapter 1: Fundamentals of TestingSection 1: Why is testing necessary?Test Engineering Foundation Essential Knowledge for Test Professionals



Why is Testing Necessary?Key conceptsHow bugs can cause harm Bugs and their effectsThe necessity of testing The role of testing in quality assuranceTerms to remember



The Threat of BugsCompanyDamaged reputation for qualityHigh or unpredictable maintenance costsUnexpected delays in release cyclesLack of confidence in systemLawsuitsEnvironmentPollutionWastePeople, societies, and statesLost jobsLost livesLost rightsLost missions Lost wars



Where Bugs Come From and What Bugs DoPeople put bugs (defects) into the systemRequirements and design specificationsCode (business logic and user interface)Documentation (electronic and hard copy)When the implementation of these bugs are executed, failures occurIf these failures are visible to customers, users, or other stakeholders, dissatisfaction with system quality results



From Whence the Bugs and Failures?Bugs occur due toprogrammer, analyst, and other individual contributor (including tester) fallibilitytime pressurecomplexity of the code, infrastructure, or problem to be solvedchanging and meshing technologiesmany system interactions.Failures occur due to bugs andenvironmental conditions misuse (deliberate and accidental)



Testing to Manage Quality RisksRisks and constraints for a software projectFeatures: Right setSchedule: Quickly enoughBudget: Acceptably cheapQuality: Ready for customers/release/next stepTesting provides the information to guide the project, reduce and manage the risks, and repair the important problemsTesting may also address compliance, contractual, and regulatory needs



What Does Quality Mean to You?Fitness for use vs. Conformance to requirementsTesting and qualityTests gives confidence where they find few bugsPassing tests reduce the level of quality risk Failing tests provide a chance to improve quality The test set gives an assessment of qualityWhat are the important quality characteristics for your system? Are you testing them (enough)?Testing, quality assurance, and quality improvementIdeally, testing is part of a larger quality assurance strategy for a projectFor future projects, analyze the root causes of defects found on current projects and take steps to reduce their incidence



Terms to RememberBug, defect, or faultError or mistakeFailureQualityRiskSoftwareTestTest case


Chapter 1: Fundamentals of TestingSection 1: A diversionTest Engineering Foundation Essential Knowledge for Test Professionals



What Happens When Quality is Lacking


Chapter 1: Fundamentals of TestingSection 2: What is testing?Test Engineering Foundation Essential Knowledge for Test Professionals



What is Testing?Key conceptsCommon objectives of testingThe purpose of testingin software development, maintenance, and operationsto find defects, provide confidence and information, and prevent defectsTerms to remember



Testing ObjectivesTypical general testing objectivesFinding bugs and providing programmers with the information they need to fix important bugsGaining confidence about the level of quality of the systemPreventing defects (through early involvement in reviews and advanced test design)Provide information about the most important aspects of the quality of the system under testHelp management understand system qualityCan you think of others for your projects?



Test Phases and ObjectivesUnit/Component TestFind bugs in the individual pieces of the system under test before the pieces are fully integrated into the system Integration/String TestFind bugs in the relationships and interfaces between pairs and groups of components in the system under test as the pieces come togetherSystem TestFind bugs in the overall and particular behaviors, functions, and responses of the system under test as a wholeAcceptance/Pilot TestDemonstrate that the product is ready for deployment/release or to assess quality and give information on the risk of deployment/releaseMaintenance TestCheck for errors introduced during development of the changes Operational TestAssess non-functional system characteristics such as reliability or availability



Effectiveness and EfficiencyEffective: Producing a decided, decisive, or desired result; impressiveTo be effective testers, we must select the appropriate objective and desired resultsEfficient: Productive of desired effect; especially productive without wasteTo be efficient testers, we must allocate resources (time and money) appropriatelyThese terms are most meaningful in the context of the entire development or maintenance process, not just the testing process.



Testing vs. DebuggingTesting finds failures that are caused by bugsDebuggingidentifies the root cause of a bugrepairs the codeand checks that the defect is fixed correctlyConfirmation testing ensures the fix resolves the observed failure Different responsibilities:Testers test Programmers debug



Find-Debug-Confirm



Beyond Test Execution Testing is not just running tests against a running systemOther test activities, before and after execution, includePlanning and controlChoosing test conditionsDesign test casesChecking test resultsEvaluating exit criteriaTest result reportingClosure/end-of-test tasksWe will revisit this later



Terms to RememberCodeDebuggingSoftware developmentReviewRequirementTest caseTest objectiveTestingTest basis


Chapter 1: Fundamentals of TestingSection 2: ExerciseTest Engineering Foundation Essential Knowledge for Test Professionals



Exercise: The Triangle TestsA program accepts three integers representing the lengths of a triangles sides. It outputs scalene (no equal sides), isosceles (two equal sides), or equilateral (three equal sides).Write an effective (finds common bugs) and efficient (as few tests as possible) set of test cases. Usually, a test case consists of tester action, data, and expected result, but here the action is generally input data.Discuss.


Chapter 1: Fundamentals of TestingSection 3: General testing principlesTest Engineering Foundation Essential Knowledge for Test Professionals



General Testing PrinciplesKey conceptsTesting reveals the presence of bugsImpossibility of exhaustive testingBenefits of early testingLumpiness of bugs: defect clusteringPesticide paradoxTesting should adapt to specific needsAbsence-of-errors fallacyTerms to remember



Testing Reveals Presence of Bugs: A ParableYou have a beautiful vegetable garden, but one day you see eaten leaves on the tomatoesOh no, you think, I have hornworms!You know you have bugs in your gardenIf you had not seen the symptoms, could you be sure you had no bugs? Some bugs are easy to spot, others arentTesting can reveal the presence of bugs, but cannot prove their absence



Mission Impossible: Exhaustive TestingJust make sure the software works before we ship itThis charter is demonstrably impossibleThe execution paths in non-trivial software are almost infiniteLarge dataflows separated across space (features) and time (static data)Slight changes can cause regressions which are not linear to the size of the changeMyriad usage profiles and field configurations, some unknown and some unknowable



Defusing Exhaustive Testing ExpectationsExhaustive testing as a way to prove the software works is a common (mis)expectationBad expectations create problems for test professionals and test teamsUnachievable high demands on test groupPerception of incompetence when these demands arent metTesters must be ready to communicate (in words the project stakeholders will understand) how testing can contribute



Benefits of Early QA and TestingThe cost of a bug tends to increase as the project continuesMost of the costs associated with pre-release bugs tend to be associated with the effort required to remove them, so the higher cost means longer schedulesThe more bugs enter a quality assurance or test activity, the more bugs will escape from that activity



Click here to see the answers



Defect ClusteringStudies have long shown and continue to show the unequal distribution of bugsMVS: 38% of field bugs in 4% of modulesIMS: 57% of field bugs in 7% of modulesCapers Jones reports that the excessive presence of error-prone modules causes a 50% reduction of productivity in software maintenance





Pesticide ParadoxReturn to your vegetable gardenYou spray pesticide on your garden, and the hornworms die, but the pesticide is not effective against all bugsJust as pesticides become less effective, so do testsFunctional tests cant find performance bugsTry new test techniques



These two pictures compare efficiencies of the top 11 bug-finding test suites (out of 27). The top graph shows the first time the set of test suites was run. The bottom graph shows the results for the fifth time that same set of test suites was run. The average test suite efficiency in the first pass, 0.4, is used as the axis crossing point in both graphs. With one exception, the same test suites are in the top 11 test suites, which shows bug clustering again. However, all test suites are less effective after five executions.



Absence-of-Errors FallacyFinding and fixing many bugs does not guarantee user, customer, and/or stakeholder satisfactionMany low-defect products have failed in the market placeSuccessful projects balance competing forces in terms of features, schedule, budget, and quality



Testing Should Adapt to NeedsDifferent projects, organizations, and products have different testing needsBest testing practices exist (and are discussed in this course) but you need to tailor them to your projectFailure to adapt the test team and its methods to these needs is a common result of dissolution of test teams



Terms to RememberExhaustive testing





Exercise: Test Principles Observed (and Not)Think back on a recent project.Note which of the principles that you can recall were observed.How about principles that you can recall were not observed (i.e., violated)?Discuss.


Chapter 1: Fundamentals of TestingSection 4: Fundamental test processTest Engineering Foundation Essential Knowledge for Test Professionals



Fundamental Test ProcessKey conceptsPlan, prepare, perform, perfectPlanning and controlAnalysis and designImplementation and executionEvaluating test exit criteria and reportingTest closure activitiesTerms to remember



Test Processes: Critical Testing ProcessesThe test engineers role focuses on some of these activities, not all, depending on how roles are defined. However, the effective and efficient test engineer must understand how the test process works and how it fits into the overall project from a big picture perspective.



ISTQB Fundamental Test ProcessAnother way to think of the test process is in terms of the following stepsPlanning and controlAnalysis and designImplementation and executionEvaluating test exit criteria and reportingTest closure activitiesThese steps may overlap or take place concurrently



Planning and ControlPlanningDetermine test scope, risks, objectives, strategiesDetermine required test resources Implement the test strategiesSchedule test analysis and designSchedule implementation, execution and evaluation of testsDetermine the test exit criteriaControlMeasure and analyze resultsMonitor and document progress, coverage and test exit criteriaInitiate corrective actions Make decisions



Analysis and DesignAnalysisReview the test basis (e.g., requirements or design specifications, network/ system architecture, quality risks)Identify test conditions, test requirements, or test objectives and required test data based on analysis of test items, its specification, behavior and structureDesignSelect specific combinations of test data, actions, and expected results to cover the test basisEvaluate testability of the requirements and systemDesign the test environmentIdentify any required infrastructure and tools



Implementation and ExecutionImplementationDevelop and prioritize test cases, create test data, write test proceduresPrepare test harnesses and write automated test scriptsOrganize test suites and sequences of test cases for efficient test executionVerify that the test environment has been set up correctlyExecutionExecute test cases (manual or automated) Log test results, and the versions of the software under test, test tools and the testwareCompare actual and expected resultsReport and analyze incidents Repeat corrected and/or updated tests Run confirmation and/or regression tests



Exit Criteria, Reporting, and ClosureExit and reportingCheck test logs against the test exit criteria specified in test planningAssess if more tests are needed or if the exit criteria specified should be changedWrite a test summary report for stakeholdersClosureConfirm test deliverables, final resolution or deferral of bug reports, and the acceptance of the system Finalize and archive testware, test environment and test infrastructureDeliver testware to the maintenance organizationPerform a retrospective to capture improvements for future releases, projects, and test processes



Terms to RememberTestwareTest planTest strategyTest executionTest basisTest log (Test) exit criteriaTest summary reportTest coverageTest condition



Terms to RememberTest dataTest inputConfirmation testingRegression testingIncident





Exercise: Test Steps and Tasks PerformedThink back on a recent project.Note which of the steps and tasks of the ISTQB test process were carried out.Note which of the steps and tasks of the ISTQB test process were not carried out.Note whether some steps overlapped or were completely parallel.Discuss.


Chapter 1: Fundamentals of TestingSection 5: The psychology of testingTest Engineering Foundation Essential Knowledge for Test Professionals



Psychology of TestingKey conceptsPsychological factors for testing successClear objectives Self-testing and independent testing Respectful communication Programmer and tester outlooks for successTerms to remember



Certainty Versus ProgressThorough research can lead to indisputable bug reports, unquestionably correct test cases, etc.Such certainty is intellectually satisfying and reduces rejected and irreproducible bug reportsBut certainty also can consume too much time and effort for the payoff and delay forward progressTest projects adapt to schedule pressures or failTesting computers is test engineeringNot a search for truththats science Engineering is making useful objects for customers



Test Result MisinterpretationOn the one hand we couldreport correct behavior as a bugassign excessively high severity or priorityotherwise overstate bug significanceOn the other hand we couldfail to detect or report incorrect behaviorassign excessively low severity or priority otherwise understate bug significanceSome tips to avoid these errors1. Have testers take breaks so they dont miss important events2. Automate where practical3. Define expected results as clearly as possible4. Assign the right testers to each test execution task5. Use peer reviews for test execution and bug reportsHoweverrecognize that perfect test execution takes too long. Define good enough for testing and live with that decision



Professional PessimismExplore depressing possibilities of failureAnticipate the worst possibilities in order to achieve best obtainable product qualityNot adversarial, but a different outlook than the programmersRemember: to assume nothing will fail during testing denies the entire history of computingCaveat: not a license to offendDont target programmers with reports or take glee in failureChallenge: to be positive, pleasant, and the bearer of bad news, all at oncePierre has the pessimism, but perhaps not professionalism?



Bad News == Bad Guy?Testers are sometimes on receiving end of emotions brought on by news of project problemsThe key is professional pessimismDont take on the role of Lone Champion of QualityProduct quality is a business decisionQuality risks are weighed against other risksDont second-guess developments bug-fix effortsNever gloat, even if you were right and everyone else was wrongJuan de Mariana: The greatest of follies is to exert oneself in vain, and to weary oneself without winning anything but hatred.Grim reaper or friendly guide?



Balanced CuriosityBalance need for thoroughness in any one area with need to cover many areas in a short timeEffective and efficient test engineers have a talent for spending time where the bugs areEffective and efficient test engineers can do thorough bug isolation quicklyIneffective and inefficient test engineersWrite tests to search for unlikely, low-impact bugsSpend hours researching trivial bugs



FocusTwo types of focus problemsPursuing issues narrow-mindedly, losing sight of more important prioritiesGetting distracted from key tasksBalance and re-evaluate priorities every so oftenStay focused on the goals of the test projectA seasoned test engineer can find his way towards test project goals, with clear signposts from his test manager



Defining Tester SkillsReadingSpecifications, e-mails, test cases, etc.WritingTest cases, bug reports, test documentation, etc.Not native language dependentStatistics and other mathematicsPertinent technology, project, and testing skillsTechnology: Programming languages and more, like operating systems, networking, HTML/Web, etc.Application domain: banking, human factors, office applications, etc.Testing: scripting, exploring and attacking the system, automation, performance modeling, etc.



Balancing the SkillsEffective and efficient tester have right mix of skills for tasks and activitiesApplication domain expertUnderstands intended behaviorSkilled testerKnows quality risks and test techniquesTechnical guruAware of technical issues and limitationsWhat is the right mix forInternet appliance testing?nuclear medicine testing?your project?The appropriate depth and length of each arrow in the figure depends on the project, process, and product



Terms to RememberIndependent testing





Exercise: Psychology in ActionReflect on the attitudes and behaviors of the most successful testers you know.To what extent do they display the psychological aspects discussed in this section?What other elements of their personalities and skillsets do you think lead to success for them?Discuss.


Chapter 2: Testing throughout the Software LifecycleTest Engineering Foundation Essential Knowledge for Test Professionals



2. Testing Throughout the Software LifecycleSoftware development modelsTest levels or phasesTest types or targetsMaintenance testing


Chapter 2: Testing throughout the Software LifecycleSection 1: Software development modelsTest Engineering Foundation Essential Knowledge for Test Professionals



Software Development ModelsKey conceptsThe relationship between development and test activitiesAdapting software development models to the context of the project and productReasons for different levels of testingTerms to remember



The V Model in Waterfall ProjectsUsually schedule- and budget-risk-drivenDo ever-deeper levels of design, then build, then testIntuitive and familiar modelBeats chaos!Its hard to plan that far in advance!When plans fail, test--at the end--suffers!



Evolutionary and Incremental ModelsSchedule-risk-driven to hit market window or delivery dateFeature set grown around core functionalityCan ship (something) any time once the core functionality is readyBecoming a popular approachRanges in formality from Extreme Programming to RAD and RUP

Still a temptation to ship a system with buggy featuresIn the agile world, the role of testing still evolving



System IntegrationMany projects involve integrating componentsRisk-mitigation optionsIntegrate, track, and manage vendor testing in distributed test effortTrust vendor component testingFix vendor testing/qualityDisregard and replace their testing (ouch!)Watch politics in last two optionsPlan on integration and system testing yourselfCoupling: Strong interaction or consequence of failure between component and systemIrreplaceability: Few similar components availableEssential: Key features in system unavailable if component does not work properlyVendor quality problems: Increased likelihood of a bad component



Verification & ValidationVerificationLook for bugs in phase deliverablesAre we building the system right?ValidationLooking for bugs in system, based on phase deliverablesAre we building the right system?



IEEE 12207 StandardScope Purpose, application, tailoring, compliance, limitationsNormative referencesDefinitionsApplication Lifecycle processes, tailoring, and fitting to organizationPrimary life cycle processes Acquisition, supply, development, operation, maintenanceSupporting processes Tech pubs, CM, QA, IV&V, audits, problem resolutionOrganizational life cycle processes Management, infrastructure, improvement, training



CMMI Process MaturityCapability Maturity Model Integration: a five-level model from the Software Engineering InstituteInitial: unpredictable, poorly controlled, reactiveManaged: process established at process level, often reactiveDefined: process established across organization, usually proactiveQuantitatively managed: process measured and controlled at organizationOptimizing: focus on continuous improvement, usually driven by dataTesting has been underemphasized in CMM, leading to test-specific model (e.g., Critical Testing Processes, Test Process Improvement, Testing Maturity Model)



Regardless of the ModelsGeneral characteristics of good testingTesting activity for each development activity (e.g., unit test and implementation)Test levels have focused objectives, with coordination to avoid gaps, overlapTest analysis, design begins early, prevents bugsTesters involved in any reviews they are qualified to attend, bringing their unique perspectiveYou can combine or reorganize test levels provided you keep these characteristics in mind



Terms to RememberTest levelVerificationValidationV-ModelIncremental development modelCommercial Off The Shelf (COTS)


Chapter 2: Testing throughout the Software LifecycleSection 1: Exercise 1Test Engineering Foundation Essential Knowledge for Test Professionals



Exercise: Models and RealityFamed quality expert W.E. Deming said, All models are wrong; some are useful. Indicate which lifecycle model in this section applied most closely to your past project (or, if there was no organizing model, indicate code-and-fix).To what extent do you think the model was useful? To what extent, if any, was it harmful?Discuss.


Chapter 2: Testing throughout the Software LifecycleSection 1: Exercise 2Test Engineering Foundation Essential Knowledge for Test Professionals



Exercise: Omninet Lifecycle ConsiderationsRead the Omninet Marketing Requirements Document.Is a V-model (sequential) or incremental model more appropriate for this project? Why?Are maintenance, integration, or verification and validation important for this project? Why?Discuss.


Chapter 2: Testing throughout the Software LifecycleSection 2: Test levels or phasesTest Engineering Foundation Essential Knowledge for Test Professionals



Test Phases or LevelsKey conceptsMajor objectives of testing for each levelTypical testing objects for each levelTypical targets of testing for each levelTesting work products for each levelTest participants for each levelTypes of defects and failures for each levelTerms to remember



Component (Unit) TestObjective: Find bugs in the individual pieces of the system under test prior to system integrationBasis: Code, database, reqs/design, quality risksTest types: Functionality, resource use, performance, structuralItem Under Test (IUT): Varies. Smallest independently testable item (function or class) or a distinct component providing services to othersHarnesses and tools: API level (drivers and stubs), freeware and commercial Responsible: Usually programmers, but the level of proficiency and degree of execution varies.



Unit/Component Test ProcessUnit/component testing typicallyinvolves access to the code is run in a development environmentrequires drivers, stubs, and/or harnessesis done by the programmer who wrote the codeOften, bugs are fixed upon being found without any reporting, which reduces the transparency of the development process with respect to qualityTest-first/test-driven developmentDevelop a set of unit testsBuild and integrate codeRun the tests and debug until the tests pass



Drivers and StubsDuring unit, component, and integration testing--and for testing APIs--its often necessary to simulate parts of call flow reachable from module(s) under testData setup sometimes necessary, alsoDriver: function(s) that call module(s) under testStub: function(s) called--directly or indirectly--by module(s) under test



Integration TestObjective: Find bugs in the relationships and interfaces between pairs and groups of components in the system under test as the pieces come togetherBasis: Design, architecture, schemas, dataflows, quality risksTest types: Functionality, resource use, performanceItem Under Test: Builds or backbonesHarnesses and tools: API and CLI level, freeware and commercialResponsible: Ideally both testers and programmers, but often no one



Integration TechniquesBig bangTake all tested modules; put them all together; testQuick, but wheres the bug?Why wait until all code is written to start integration?Bottom upStart with bottom layer modules; use appropriate drivers; testRepeat process, replacing drivers with modules, until doneGood bug isolation, but what if nasty problems are at the top?Top downLike bottom up, but start from top and use stubsGood bug isolation, but what if nasty problems are at the bottom?BackboneStart with critical modules; build initial backbone; use drivers and stubs; testRepeat process, replacing stubs and drivers with modules in risk orderGood bug isolation and finds integration bugs in risk order



Backbone Integration TechniqueBackbone 0 (BB0)Consider the following example (based on a real project)We start with a basic backbone Communication APIs Basic networking architecture Test basic functionality, error handling and recovery, reliability, and performanceQuality risk: Is the underlying system architecture untenable?Backbone 0: Testing the basic communication software and network architecture



Backbone Integration TechniqueBackbone 1 (BB1)Add the modules that implement the most critical core operations and services Again, test basic functionality, error handling and recovery, reliability, and performanceQuality risk: Do the core operations and functions integrate with the transport layer?Continue process with next level of quality risk. Backbone 1: Testing some core operations and services through the communication API and network



Backbone Integration TechniqueBackbone N (BBn)As the last step, we are testing (in this case, using automated test drivers) end-to-end through the GUIs on the host systemsQuality risk: Does the fully integrated system work?The final backbone for integration testing is also the first fully integrated build for system testBackbone N: Testing the entire, completely integrated system end-to-end. When this works, were ready for System Test



Integration Test LevelsThere may be more than one level of integration testing on a projectComponent integration testing: look for bugs in interactions between units or components following unit/component testSystem integration test: look for bugs in interaction between entire systems following system testSystem integration testing is complexMultiple organizations controlling the systems interfaces, making change dangerousBusiness processes may span systemsHardware/system compatibility issues can arise



System TestObjective: Find bugs in the overall and particular behaviors, functions, and responses of the system under test as a wholeBasis: Requirements, high-level design, use cases, quality risks, experience, checklists, environmentsTest types: Functionality, security, performance, reliability, usability, portability, etc.Item Under Test: Whole system, in as realistic-as-possible test environmentHarnesses and tools: API, CLI, or GUI, freeware and commercialResponsible: Typically independent testers



Acceptance TestObjective: Demonstrate that the product is ready for deployment/releaseBasis: Requirements, contracts, experienceTest types: Functional, portability, performanceItem Under Test: Whole system, sometimes in the production or customer environmentHarnesses and tools: GUI usually Responsible: Often users or customers, but also independent testers



Variations in Acceptance TestingUser acceptance testing: Business users verify fitness for functional purposes.Operational testing: Acceptance by system administrators (e.g., backup-restore, disaster recovery, user management, maintenance, security)Contract and regulation testing: Verification of conformance to contractually-agreed or legally mandated requirements, regulations, or standards.Alpha, Beta, and field testing: Testing and confidence-building by potential or existing customers. Beta testing and field testing are performed in the actual environment(s).



Pervasive Testing: Early, Cross-FunctionalTest execution activities for test phasesProject timeline



Why Pervasive Testing?Different participants can test different granularitiesDifferent skills for different granularitiesDifferent granularities emphasized in each phaseUnit testing: primarily structuralSystem testing: primarily behavioralAcceptance testing: primarily liveIts important to be flexible, thoughTest techniques of various granularities can be useful in all the test execution phasesPhase overlap a function of entry and exit criteriaNot all test phases occur on all projects



When Testing Pervades ProjectsTest tasks occur throughout the development effortTest execution is planned with multiple cycles to allow for fix timeFeatures will be dropped or slipped into later iterations rather than slipping test phase entry dates or entering before ready



Terms to RememberAlpha testingBeta or field testingBusiness process-based testingComponent testingContract acceptance testingDaily buildDriversFunctional requirementsIntegrationIntegration testing



Terms to RememberNon-functional requirementsOperational (acceptance) testingRegulation testingRequirements-based testingRobustness testingStubSystem testingTest-driven developmentTest environmentTestwareUser acceptance testing


Chapter 2: Testing throughout the Software LifecycleSection 2: ExerciseTest Engineering Foundation Essential Knowledge for Test Professionals



Exercise: Omninet Test LevelsRead the Omninet System Requirements Document.If you were managing all the testing for Omninet, which levels or phases of testing would you plan? Why?What would the major goals of each level or phase of testing be? What kind of acceptance test, if any, would you plan? Why?How do these test levels relate to and affect the lifecycle model you selected in the previous exercise?Discuss.


Chapter 2: Testing throughout the Software LifecycleSection 3: Test types or targetsTest Engineering Foundation Essential Knowledge for Test Professionals



Test Types or TargetsKey conceptsMajor software test types or targetsFunctional and non-functional testsStructural testsConfirmation and regression testsUse of functional, non-functional, and structural tests at various levelsTerms to remember



FunctionalityReasonable or required action not provided, inaccessible, or seriously impairedNo add function on a calculatorAdd function implemented, + key doesnt workCan only add integers, not real numbersRight action, wrong resultAdd function: 2+2=5?Right action, right result, wrong side-effectDivide function: 2/2=I (Roman numeral format)System, subsystem or component functionality is described in documents like requirements specification, use cases, or a functional specification (sometimes). However, often some functions remain undocumented, and testers must understand reasonable behavior.



SecuritySecurity threats includeViriiCracking into serversDenial of serviceCommon bad assumptionsEncryption (HTTPS) on Web server solves security problemsBuying a firewall solves problemsUnskilled network or system administrators can solve problemsSpecialized field of test expertiseThe determined--or bored--cracker might break into any of your servers. One exploit can lead the cracker to other vulnerabilities--and to valuable data.



Performance And ReliabilityPerformanceToo slow throughout performance curveUnacceptable knee in performance curveUnacceptable performance degradation over timeReliabilitySystem fails to complete normal functionsSystem functions normally, but randomly crashes or hangs



Stress, Capacity, and VolumeStress: extreme conditions cause failureCombinations of error, capacity, and volume testsCapacity: functionality, performance, or reliability problems due to resource depletionFill hard drive or memory to 80+%Volume: functionality, performance, or reliability problems due to rate of data flowsRun 80+% of rated transactions per minute, number of simultaneous users, etc.



Maintenance And MaintainabilityMaintenanceUpdate and patch install and deinstall processes dont workConfigurations cant be changed appropriately (e.g., plug-and-play, hot plugging, adding disk space, etc.)MaintainabilitySoftware itself (source code) not maintainableDatabases not upgradeableDatabases grow monotonicallySoftware not efficiently testable during maintenance; e.g., excessive regression



Usability and User InterfaceA system can function properly but be unusable by the intended customerCumbersome interfaces that do not follow workflowsInaccessible functionalityInappropriately difficult for the users to learnInstructional, help, and error messages that are misleading, confusing, or misspelled



Configuration, Compatibility, PortabilityA single platform may be configured in many different ways in softwareA family of platforms may support various hardware configurationsAre configuration changes handled? Add disk space or other storageAdd memoryUpgrade or add CPUInteroperability with programs, OS, databasePortability to various environments



Other Functional/Non-Functional TestsLocalization (user interface)Localization (operational)Standards and regulatory complianceError handling and recoveryDisaster recoveryNetworked/internet-worked or distributedTiming and coordinationData qualityData conversionOperationsInstallationDe-installationDate and time handlingDocumentationAnd many others



Structural TestsTests based on how the system is builtCodeDataDesignStructural (white box) coverage can be measured after functional and non-functional (black box) tests are run to check for omissionsThis topic will be covered in more depth later



Regression and ConfirmationRegression testing checks the effects of changes, since even small, localized, isolated changes, dont always have small, localized, or isolated effectsRegression strategies are covered in the next sectionConfirmation testing confirms thatChanges made to the system are presentBug fixes introduced in the system solve the observed symptomsRepeatability of tests helps with regression and confirmation testingAutomation, covered in depth later, is also helpful



Comparing Test TypesTools, data, and cases can be shared: encourage cross-pollination of techniques Well discuss techniques for each type of testing in upcoming sections



ISO 9126 Quality Characteristics/SubcharacteristicsFunctionality: suitability, accuracy, interoperability, security, complianceReliability: maturity (robustness), fault-tolerance, recoverability, complianceUsability: understandability, learnability, operability, attractiveness, complianceEfficiency: time behavior, resource utilization, complianceMaintainability: analyzability, changeability, stability, testability, compliancePortability: adaptability, installability, co-existence, replaceability, compliance



Terms to RememberAutomationBehavior-based testingBlack box or specification-based testingCode-based, white box, or structural testingCode coverageConfirmation testingFunctional testInteroperability testing Load testingMaintainability testingPerformance testing



Terms to RememberPortability testingRegression testReliability testingSecurity testingStress testingTest suiteUsability testingUse case testing





Exercise: Omninet Test TypesReferring back to the levels you selected in the previous exercise, list for each level the test targets or types you would include in each level.How does the lifecycle model you selected in a previous exercise affect the amount of regression testing? How does the lifecycle model you selected in a previous exercise affect the amount of confirmation testing? Discuss.


Chapter 2: Testing throughout the Software LifecycleSection 4: Maintenance testingTest Engineering Foundation Essential Knowledge for Test Professionals



Maintenance TestingKey conceptsReasons for maintenance testing Maintenance testing versus new application testingRole of regression testing and impact analysis Terms to remember



Reasons for MaintenanceThree typical triggers for maintenance and maintenance testingModification: enhancements, bug fixes, operational environment changes, patchesMigration: a new supported environmentRetirement: end-of-life of a subsystem or entire system triggers replacementMaintenance testing addresses the change itself--and what wasnt changed and shouldnt change



Testing Maintenance ReleasesRegression is the big risk for maintenance releasesSome organizations try to put a major release worth of features into a short maintenance releaseTime to develop new tests is scarceLarge project test estimation rules-of-thumb fail



RegressionRegression (misbehavior of a previously correct function, attribute, or feature due to a change) types: Local (fix creates new bug)Exposed (fix reveals existing bug)Remote (fix in one area breaks something in another area)Regression can affect new and existing features



Regression Strategy 1: Repeat All TestsIf our tests are aligned with quality, then repeating all tests should find most important regressionsAutomation is the only practical means for large complex systemWell cover automation more thoroughly in a later section



Regression Strategy 2: Repeat Some TestsOften, full automation is impossibleSelect some tests to repeatTraceabilityChange analysisRisk analysisUse cross-functional tests for accidental regression testingCan use code coverage to assess level of risk



Three Other Regression StrategiesRelease more slowlyRelease every six months rather than every monthPartial or even full repetition can increase coverage on bigger releasesCombine emergency patches with slower release process to allow for flexibility while keeping regression risk lowUse customer or user testingBeta for mass-market softwarePilot, staged or phase, parallel release for IT



Terms to RememberMaintenance testingImpact analysisModificationsMigrationRetirement





Exercise: Omninet MaintenanceAssume you can fully automate the testing of the Omninet kiosk and call center. If a post-release change is made to the call-center user interface that does not affect functionality, what would you retest?Assume you have not automated the testing of the Omninet kiosk and call center. If the same change is made, what would you retest? What if the change did affect functionality? Discuss.


Chapter 3: Static TechniquesTest Engineering Foundation Essential Knowledge for Test Professionals



3. Static TechniquesReviews and the test processReview processStatic analysis by tools


Chapter 3: Static TechniquesSection 1: Reviews and the test processTest Engineering Foundation Essential Knowledge for Test Professionals



Reviews and the Test ProcessKey conceptsSoftware work products and static techniquesThe importance and value of static techniquesThe difference between static and dynamic techniquesTerms to remember



Static TestingReviews and toolsReviews range from informal to very formalTools can perform some types of static testsStatic techniques can be used for requirements and designs, plus code, database schemas, documentation, testsModels and prototypesA diagram of a complex system can often reveal design problems that can hide in wordsAn ugly diagram means lots of bugsTest cases and dataTest analysis and design based on requirements and design specs is a form of structured reviewTest analysis and design often reveals problems



Static ToolsStatic analysisProblematic wording: Spell/grammar checkersDangerous programming: J-Test, Safer C, lintMeasurement: Complexity analysisSystem simulationsGeneral Purpose System SimulatorPerformance modeling/operations research tools Spreadsheets



Reviews: Costs and BenefitsCostsTime required to perform reviewsEffort required to gather and analyze metricsProcess improvementBenefitsShorter schedules (due to efficient bug removal)Shorter testing periods and lower testing costsDeveloper productivity Improved quality of product (which reduces downstream costs)Bottom line: Reviews of all kinds are proven, high-return techniques for improving quality



Relating Static and Dynamic TestingSimilaritiesSeek to identify defectsWork best when a broad cross-section of stakeholders are involvedSave the company money and timeDifferencesEach technique can find different types of defects more effectively and efficientlyStatic techniques find defects rather than failures



Terms to RememberReviewsDynamic testingStatic analysis


Chapter 3: Static TechniquesSection 1: ExerciseTest Engineering Foundation Essential Knowledge for Test Professionals



Exercise: Omninet Static TestingDo you see reviews and static analysis as useful for the Omninet project?If so, what kinds of problems do you think these reviews and static analyses would locate?What kinds of problems might they not locate?Discuss.


Chapter 3: Static TechniquesSection 2: Review processTest Engineering Foundation Essential Knowledge for Test Professionals



Review ProcessKey conceptsPhases, roles and responsibilities of a typical formal review The differences between different types of reviewThe factors for successful reviewsTerms to remember



Types of ReviewsInformal: no real process (hallway chats, buddy tests, pair programming), yet useful, cheap, popularPeer: documented and defined defect removal process, involving peers and technical experts but not managersWalkthroughs: author walks peers through the document or codeInspections: a trained moderator (other than the author) leads the inspection team (with defined roles) through a formal inspection process (rules, checklists, entry and exit criteria), which includes gathering defect removal metrics



Consensus and UnderstandingIncompleteness and ambiguity can hide the real meaning of the specificationsAgreement and uniform understanding of the specifications Long before any code exists, the specification must be handed to an outside testing group to be scrutinized for completeness and clarity. As [V.A.] Vyssotsky [of Bell Labs Safeguard Project] says, the developers themselves cannot do this: They wont tell you they dont understand it; they will happily invent their way through the gaps and obscurities. Fred Brooks The Mythical Man-Month 1975



A Generic Review ProcessPlanningKick-offPreparationReview meetingRework/repairFollow-up The details of the review process depend on the specific review type used on the projectIncludes estimating and planning, training participants, etc.Follow-up includes on individual items as well as overall process improvement analysis, evaluation of defect (bug) removal at phase exit reviews (exit meetings), etc.These steps of the process repeat per each item reviewed. Preparation is usually one to two hours alone. Meeting is one to two hours together. Rework/repair is fixing the bugs found.



Roles and ResponsibilitiesModerator: Lead the review meetingsScribe or secretary: Gather information on findingsAuthor: Describe, explain, answer questions on item Reviewer/inspector: Find defects (bugs) in itemManager: Plan, arrange resources and training, support, analyze process metricsIn some cases, one person may play multiple rolesAuthors sometimes act as moderatorsOne of the reviewers can act as the secretaryThe specifics are determined by the type of review



Suggestions for Successful ReviewsProvide trainingReview the product, not the producerSet and follow agenda and objectivesLimit debateFocusing on finding, not fixing, problemsTake written notesLimit and carefully select participantsInsist on preparation (e.g., by having people submit notes)Develop a checklist for each type of item that is reviewedReview the reviewsUse the right techniquesEnsure management supportLearn and get better!



Common Requirements and Design BugsAmbiguities: What exactly does that mean?E.g.: System shall allow user to read ISP e-mailWhat ISPs? What size e-mails? Attachments?Incompleteness: Okay, and then what?E.g.: Upon three invalid passwords, system shall lock users accountFor how long? How to unlock? Who can unlock? Untestability: How can I check this item?E.g.: System shall provide 100% availabilityNo known test technique to demonstrate perfect availabilityExcessive dependencies, coupling and complexityLook for ugly design diagrams and confusing requirements



IEEE 1028 Standard for Software ReviewsOverviewPurpose, scope, conformance, organization, applicationReferencesDefinitionsManagement reviewsResponsibilities, inputs/outputs, entry/exit criteria, proceduresTechnical reviewsResponsibilities, inputs/outputs, entry/exit criteria, procedures



IEEE 1028 Standard for Software ReviewsInspectionsResponsibilities, inputs/outputs, entry/exit criteria, procedures, data collection, process improvement WalkthroughsResponsibilities, inputs/outputs, entry/exit criteria, procedures, data collection, process improvementAuditsResponsibilities, inputs/outputs, entry/exit criteria, procedures



Terms to RememberReview processReview meetingModerator/ inspection leaderReviewerScribe (or secretary) InspectionEntry criteriaExit criteriaFormal review



Terms to RememberWalkthroughInformal reviewPeer or technical reviewMetricsKick-off





Exercise: Omninet Requirements ReviewBreak into teams of three to five people.Select a review technique from among those discussed. If your selected technique involves specific roles, assign the roles.Review the Omninet Marketing Requirements Document.Discuss your teams findings.


Chapter 3: Static TechniquesSection 3: Static analysis by toolsTest Engineering Foundation Essential Knowledge for Test Professionals



Static Analysis by ToolsKey conceptsThe objective of static analysis versus dynamic testing Typical defects and errors identified by static analysis Typical benefits of static analysis List typical code and design defects identified by static analysis tools Terms to remember



Static Analysis and Dynamic TestingLike dynamic testing, static analysis looks for defects in software source code and software modelsUnlike dynamic testing, static analysis is performed without actually executing the system Static analysis involves analysis of the system or its components by a tool, while dynamic testing does not always involve toolsStatic analysis can find defects that are hard to find or isolate in dynamic testingExamples include maintainability issues, unsafe pointer useIsolation is easier because you find the bug, not the symptom



What Can We Analyze? Program code (e.g. control flow and data flow)Models of the program (e.g., simulations)Generated output such as HTML and XMLRequirements and design documents



Benefits of Static AnalysisEarly and cheaper detection of bugs (before test execution starts)Warnings about where bug clusters might exist, due to dangerous programming, high complexity, etc. Location of bugs dynamic testing might missDetection of dependencies and inconsistencies in software models (e.g., such as link problems in Web pages)Improved maintainability of code and designPrevention of defects based on metrics gathered and lessons learned from analysis



Typical Static Analysis BugsReferencing a variable with an undefined valueInconsistent interface between modules and componentsVariables that are never usedUnreachable (dead) codeProgramming standards violationsSecurity vulnerabilitiesSyntax violations of code and software models



Using Static Analysis ToolsTypical users areProgrammers, often during component and integration testingDesigners and system architects during designDuring initial introduction against an existing system, static analysis tools may produce a large number of warning messages Compilers do some static analysis, but many sophisticated tools are available



Terms to RememberStatic analysisCompilerComplexityControl flowData flow





Exercise: Omninet Static AnalysisWhat kind of static analysis would you suggest for Omninet?Would you use static analysis in areas that would be subject to later testing?Discuss.


Chapter 4: Test Design TechniquesTest Engineering Foundation Essential Knowledge for Test Professionals



4. Test Design TechniquesIdentifying and designing test casesCategories of test design techniquesSpecification-based or black-box techniquesStructure-based or white-box techniquesExperience-based techniquesChoosing test techniques


Chapter 4: Test Design TechniquesSection 1: Identifying and designing test casesTest Engineering Foundation Essential Knowledge for Test Professionals



Identifying and Designing Test CasesKey conceptsBase testing on risk analysis Determine level of risk with likelihood and impact Specify test designs, cases, proceduresWrite test casesRelate test cases and test proceduresDevelop test execution schedule Terms to remember



Phases of Test DevelopmentTest development often proceeds in phases(Quality risk) analysis High-level test designLow-level test design (implementation)External inputs used to create internal deliverables (testware)



Analysis



High-Level DesignThese arrows show the relationships between the risk categories and the test suites. Well capture this relationship information in the form of whats called traceability.



Low-Level Design (Implementation)Each risk to be mitigated via testing will have one or more test cases associated with it. (Ive not shown all traces, to avoid clutter.)



Whats a Quality or Product Risk?RiskThe possibility of a negative or undesirable outcomeThe likelihood of a risk becoming an outcome is>0,

Chapter 4: Test Design TechniquesSection 1: Exercise 1Test Engineering Foundation Essential Knowledge for Test Professionals



Exercise: Omninet Risk AnalysisBased on your reading of the Omninet Marketing Requirements Document, the Omninet System Requirements Document, and your experience with testing and bugs, perform a risk analysis for OmninetDiscuss.



IEEE 829 Test Design SpecificationThe test design specification describes a collection of test cases at a high level, and includes the following sectionsTest design specification identifierFeatures to be tested (in this test suite)Approach refinements (specific techniques, tools, etc.)Test identification (tracing to test cases in suite)Feature pass/fail criteria (e.g., test oracle, test basis, legacy systems, etc.)This collection of test cases often called a test suiteSequencing (test suites and cases within suites) often driven by risk and business priority and affected by project constraints, resources, and progress



IEEE 829 Test Case SpecificationA test case specification describes the details of a test case, and includes the following sectionsTest case specification identifierTest items (what is to be delivered and tested)Input specifications (user inputs, files, etc.)Output specifications (expected results, including screens, files, timing, etc.)Environmental needs (hardware, software, people, props)Special procedural requirements (operator intervention, permissions, etc.)Intercase dependencies (if needed to set up preconditions)In practice, test cases vary significantly in effort, duration, and number of test conditions covered



IEEE 829 Test Procedure SpecificationA test procedure specification describes how to run one or more test cases, and includes the following sectionsTest procedure specification identifierPurpose (e.g., which tests are run)Special requirements (skills, permissions, environment, etc.)Procedure steps (logging, set up, start, proceed [steps themselves], measurement of results, shutdown/suspension, restart [if needed], stop, wrap up/tear down, contingencies)Test procedures are often embedded in test casesA test procedure is sometimes referred to as a test script, and may be manual or automated



Test Case Coverage (Traceability)Measure or correlate tests against areas of concernUsed as a way to measure and enhance the testsPractical typesRequirements specificationsDesign specificationsFunctional areasQuality risksConfigurationsThis is a commonly-used technique to ensure thorough test coverageOne techniqueUse spreadsheetList test cases and coverage area to measure0: none; 1: indirect; 2: direct


Sheet1

Test Case

1.11.21.31.41.51.61.71.82.12.22.32.42.5Total

Spec

1.1111126

1.2221128

1.30

1.4224

1.511114

1.622

1.721227

1.8213

1.91124

1.1011114

Total5425433422710


Terms to RememberPriorityRiskRisk identificationRisk analysisRisk controlRisk-based testingProduct riskTest caseTest conditionTest data



Terms to RememberTest case specificationTest design specificationTest procedure specificationTest scriptTraceability





Exercise: Omninet Test Design SpecificationBased on your quality risk analysis for Omninet, outline a set of test suites to address the important areas of risk.For each test suite, list briefly:What the test would coverHow you would recognize passing or failing testsEstablish the relationship between your test suites and the risks they will cover. Based on your risk analysis, how would you sequence the test suites? What other considerations would affect the sequencing of test suites?Discuss.


Chapter 4: Test Design TechniquesSection 2: Categories of test design techniquesTest Engineering Foundation Essential Knowledge for Test Professionals



Categories of Test Design TechniquesKey conceptsReasons for specification-based (black box), structure-based (white box), and experience-based testsCommon black box and white box techniques Characteristics and differences between specification-based testing, structure-based testing, and experience-based testingTerms to remember



Three Types of Test Design TechniquesSpecification-based (black box), functional and non-functionalCreate tests primarily by analysis of the test basisLook for bugs in the way the system behavesStructure-based (white box)Create tests primarily by analysis of the structure of the component or system Look for bugs in the way the system is builtExperience-based (attacks, checklists, exploratory)Create tests primarily based on understanding of the system, past experience, and educated guesses about bugsLook for bugs in the places other systems have bugs



Specification-Based or Black BoxCommon elements include:Formal or informal models used to specify the problem to be solved, the software or its componentsTest cases derived systematically from these modelsExamples include:Equivalence partitioning and boundary value analysisState transition diagramsDecision tables



Structure-Based or White BoxCommon elements include:System structure (e.g., code, design, etc.) used to derive the test cases, for example code and designThe extent of structural coverage can be measured for existing other test casesFurther test cases can be derived systematically to increase coverageExamples include:Statement coverageBranch coverage



Experience-BasedCommon elements include:The knowledge and experience used to derive test casesCan consider knowledge and experience of the software, its usage and its environment orknowledge about historical and likely defects and their distribution Examples include:AttacksChecklistsExploratory



Terms to RememberBlack box or specification-based techniquesExperience-based techniquesWhite box or structure-based techniques


Chapter 4: Test Design TechniquesSection 2: ExerciseTest Engineering Foundation Essential Knowledge for Test Professionals



Exercise: Omninet Test TechniquesRefer to your outline of test suites for Omninet.For each test suite, identify whether one, two, or all three of the following categories of test techniques would be useful in designing test cases:Specification-based (black box)Structure-based (white box)Experience-basedDiscuss.


Chapter 4: Test Design TechniquesSection 3: Specification-based or black box techniquesTest Engineering Foundation Essential Knowledge for Test Professionals



Specification-based TechniquesKey conceptsWriting test cases from given software models using equivalence partitioning, boundary value analysis, decision tables, and state transition diagramsThe main purpose of each technique and how coverage may be measuredUse case testingTerms to remember



Equivalence PartitioningDivide the inputs, outputs, behaviors, and environments into classes you think will be handled equivalentlyDefine at least one test case in each partition, or use boundary values in partitions that are rangesCan use marketing info to favor classesEx: Testing supported printersPhysical interface: Parallel, serial, USB 1.1, USB 2.0, infrared, Firewire, SCSI, others?Logical interface: Postscript, HPPL, ASCII, othersImage application: Laser jet, ink jets, bubble jets, dot matrix, line printers, letter quality, pen plotters, others



Boundary Value AnalysisA refinement of equivalence partitioning that selects the edges or end-points of each partition for testingEquivalence partitioning looks for bugs in the code that handles each equivalent classBoundary values also look for bugs in the definitions of the edgesCan only be used when the elements of the equivalence partition are orderedNon-functional boundaries (capacity, volume, etc.) can be used for non-functional testing, too



IntegerHow many items would you like to order?



Real-NumberWhat is the average temperature in Duluth?



Character and StringPassword (6-10 character alphanumeric)



DateEnter the departure date for your flight



TimeEnter the departure time for your flight



CurrencyEnter a bid price (under $1000):





Exercise: Classes and BoundariesYou are testing an e-commerce site that sells Omninet knick-knacks like baseball caps, jackets, etc.Create functional tests for accepting ordersThe system accepts a five-digit numeric item ID number from 00000 to 99999Item IDs are sorted by price, with the cheapest items having the lower (close to 00000) item ID numbers and the most expensive items having the higher (close to 99999) item ID numbersThe syst