Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Chapters18‐20
Highlights&Summary
Tes9ng,Reliability,&Quality• Tes9ngcannotproveaproducthasnoerrors
– Itcanonlyfinderrorsand/orshowtheyhavebeenfixed– …Orfailtofinderrors– Itcannotproveaproductiserrorfree
• Reliability– probabilityoffailurefreeopera9onofacomputerprograminaspecified
environmentforaspecified9meperiod– SoKwarereliabilityproblemscanusuallybetracedbacktoerrorsindesignor
implementa9on• Sta9s9calQualityAssurance
– Collect,categorizeandtraceeacherror– Isolatethemaincauses
• 80%errorsfrom20%source• Alsocalled“viralfewcauses”
– Focusonfixingthoseerrorsources– Result:80%reduc9oninerror
CS‐584/Emory2
SoKwareTes9ngObjec9ves
• Tes9ngistheprocessofexecu9ngaprogramwiththeintentoffindingerrors.
• Agoodtestcaseisonewithahighprobabilityoffindinganas‐yetundiscoverederror.
• Asuccessfultestisonethatdiscoversanas‐yet‐undiscoverederror.
CS‐584/Emory3
WhiteBoxvsBlackBoxTes9ng
• WhiteBox– Knowledgeofinnerworkingsoftheprogram– UnitTes9ngisanexample– EVERYstatement/condi9ontestedatleastonce Errorlikelihoodisinverselypropor9onaltopath’sexecu9onprobability
• BlackBox– Noimplicitknowledgeofinnerworkingsoftheprogram– Independenttestersrun“blackbox”tests(theydon’tknowthecode)
• Moderntes9nghasdroppedtheseterms– Func9onal:Doesitdowhatitsaysitwilldo?– Opera9onal:Doesitbehaveasexpected?– Performance:Doesitscale?– Generaluserinterfaceandsystemresponse– BoundaryValueAnalysis(ifnormallimits=a,bthentesta±1andb±1)
CS‐584/Emory 4
GoodTestAkributes
• Agoodtesthasahighprobabilityoffindinganerror.
• Agoodtestisnotredundant.• Agoodtestshouldbebestofbreed.• Agoodtestshouldnotbetoosimpleortoocomplex.
CS‐584/Emory 5
ObjectOrientedSoKwareTestStrategies
• the definition of testing must be broadened to include error discovery techniques applied to object-oriented analysis and design models
• the strategy for unit and integration testing must change significantly, and
• the design of test cases must account for the unique characteristics of OO software.
CS‐584/Emory 6
Tes9ngObject‐BasedSoKware
• Thread-based testing – integrates the set of classes required to respond to one
input or event for the system • Use-based testing
– begins the construction of the system by testing those classes (called independent classes) that use very few (if any) of server classes.
– After the independent classes are tested, the next layer of classes, called dependent classes
• Cluster testing [McG94] – defines a cluster of collaborating classes (determined
by examining the CRC and object-relationship model) is exercised by designing test cases that attempt to uncover errors in the collaborations.
CS‐584/Emory 7
Object‐OrientedTes9ngApproach
• Eachtestcaseshouldbeuniquelyiden9fiedandshouldbeexplicitlyassociatedwiththeclasstobetested,
• Thepurposeofthetestshouldbestated,• Alistoftes9ngstepsshouldbedevelopedforeachtestandshould
contain[BER93]:– alistofspecifiedstatesfortheobjectthatistobetested– alistofmessagesandopera9onsthatwillbeexercisedasa
consequenceofthetest– alistofexcep9onsthatmayoccurastheobjectistested– alistofexternalcondi9ons(i.e.,changesintheenvironmentexternal
tothesoKwarethatmustexistinordertoproperlyconductthetest)– supplementaryinforma9onthatwillaidinunderstandingor
implemen9ngthetest.
CS‐584/Emory 8
Object‐OrientedTestMethods
• Fault‐basedtes9ng– Thetesterlooksforplausiblefaults(i.e.,aspectsofthe
implementa9onofthesystemthatmayresultindefects).Todeterminewhetherthesefaultsexist,testcasesaredesignedtoexercisethedesignorcode.
• ClassTes9ngandtheClassHierarchy– Inheritancedoesnotobviatetheneedforthoroughtes9ngofall
derivedclasses.Infact,itcanactuallycomplicatethetes9ngprocess.
• Scenario‐BasedTestDesign– Scenario‐basedtes9ngconcentratesonwhattheuserdoes,notwhat
theproductdoes.Thismeanscapturingthetasks(viause‐cases)thattheuserhastoperform,thenapplyingthemandtheirvariantsastests.
CS‐584/Emory 9
OOTMethods:BehaviorTes9ng
CS‐584/Emory 10
emptyacctopen setup Accnt
set upacct
deposit(initial)
workingacct
withdrawal(final)
deadacct close
nonworkingacct
deposit
withdrawbalance
creditaccntInfo
Figure 14.3 State diagram for Account class (adapted from [KIR94])
WebAppErrors• Because many types of WebApp tests uncover problems that are first
evidenced on the client side, you often see a symptom of the error, not the error itself.
• Because a WebApp is implemented in a number of different configurations and within different environments, it may be difficult or impossible to reproduce an error outside the environment in which the error was originally encountered.
• Although some errors are the result of incorrect design or improper HTML (or other programming language) coding, many errors can be traced to the WebApp configuration.
• Because WebApps reside within a client/server architecture, errors can be difficult to trace across three architectural layers: the client, the server, or the network itself.
• Some errors are due to the static operating environment (i.e., the specific configuration in which testing is conducted), while others are attributable to the dynamic operating environment (i.e., instantaneous resource loading or time-related errors).
CS‐584/Emory 11
WebAppTes9ng‐1
• Contentisevaluatedatbothasyntac9candseman9clevel.– syntac9clevel—spelling,punctua9onandgrammarareassessed
fortext‐baseddocuments.– seman9clevel—correctness(ofinforma9onpresented),
consistency(acrosstheen9recontentobjectandrelatedobjects)andlackofambiguityareallassessed.
• Func)onistestedforcorrectness,instability,andgeneralconformancetoappropriateimplementa9onstandards(e.g.,JavaorXMLlanguagestandards).
• Structureisassessedtoensurethatit– properlydeliversWebAppcontentandfunc9on– isextensible– canbesupportedasnewcontentorfunc9onalityisadded.
CS‐584/Emory 12
WebAppTes9ng‐2
• Usabilityistestedtoensurethateachcategoryofuser– issupportedbytheinterface– canlearnandapplyallrequirednaviga9onsyntaxandseman9cs
• Navigabilityistestedtoensurethat– allnaviga9onsyntaxandseman9csareexercisedtouncoverany
naviga9onerrors(e.g.,deadlinks,improperlinks,erroneouslinks).
• Performanceistestedunderavarietyofopera9ngcondi9ons,configura9ons,andloadingtoensurethat– thesystemisresponsivetouserinterac9on– thesystemhandlesextremeloadingwithoutunacceptableopera9onal
degrada9on
CS‐584/Emory 13
WebAppTes9ng‐3• Compa)bility
– testedbyexecu9ngtheWebAppinavarietyofdifferenthostconfigura9onsonboththeclientandserversides.
– Theintentistofinderrorsthatarespecifictoauniquehostconfigura9on.
• Interoperability– testedtoensurethattheWebAppproperlyinterfaceswithotherapplica9onsand/ordatabases.
• Security– testedbyassessingpoten9alvulnerabili9esandakemp9ngtoexploiteach.
– Anysuccessfulpenetra9onakemptisdeemedasecurityfailure.
CS‐584/Emory 14
AssessingContentSeman9cs • Istheinforma9onfactuallyaccurate?• Istheinforma9onconciseandtothepoint?• Isthelayoutofthecontentobjecteasyfortheusertounderstand?• Caninforma9onembeddedwithinacontentobjectbefoundeasily?• Haveproperreferencesbeenprovidedforallinforma9onderivedfrom
othersources?• Istheinforma9onpresentedconsistentinternallyandconsistentwith
informa9onpresentedinothercontentobjects?• Isthecontentoffensive,misleading,ordoesitopenthedoortoli9ga9on?• Doesthecontentinfringeonexis9ngcopyrightsortrademarks?• Doesthecontentcontaininternallinksthatsupplementexis9ngcontent?
Arethelinkscorrect?• Doestheaesthe9cstyleofthecontentconflictwiththeaesthe9cstyleof
theinterface?
15CS‐584/Emory
DatabaseTes9ng
16CS‐584/Emory
client layer - user interface
server layer - WebApp
server layer - data transformation
database layer - data access
server layer - data management
database
HTML scripts
user data SQL
user data
SQLraw data
UserInterfaceTes9ng • Interfacefeaturesaretestedtoensurethatdesignrules,
aesthe9cs,andrelatedvisualcontentisavailablefortheuserwithouterror.
• Individualinterfacemechanismsaretestedinamannerthatisanalogoustounittes9ng.
• Eachinterfacemechanismistestedwithinthecontextofause‐caseorNSUforaspecificusercategory.
• Thecompleteinterfaceistestedagainstselecteduse‐casesandNSUstouncovererrorsintheseman9csoftheinterface.
• Theinterfaceistestedwithinavarietyofenvironments(e.g.,browsers)toensurethatitwillbecompa9ble.
17CS‐584/Emory
Tes9ngInterfaceMechanisms‐I • Links—naviga9onmechanismsthatlinktheusertosome
othercontentobjectorfunc9on.• Forms—astructureddocumentcontainingblankfieldsthat
arefilledinbytheuser.ThedatacontainedinthefieldsareusedasinputtooneormoreWebAppfunc9ons.
• Client‐sidescrip)ng—alistofprogrammedcommandsinascrip9nglanguage(e.g.,Javascript)thathandleinforma9oninputviaformsorotheruserinterac9ons
• DynamicHTML—leadstocontentobjectsthataremanipulatedontheclientsideusingscrip9ngorcascadingstylesheets(CSS).
• Client‐sidepop‐upwindows—smallwindowsthatpop‐upwithoutuserinterac9on.Thesewindowscanbecontent‐orientedandmayrequiresomeformofuserinterac9on.
18CS‐584/Emory
Tes9ngInterfaceMechanisms‐II• CGIscripts—acommongatewayinterface(CGI)scriptimplementsa
standardmethodthatallowsaWebservertointeractdynamicallywithusers(e.g.,aWebAppthatcontainsformsmayuseaCGIscripttoprocessthedatacontainedintheformonceitissubmikedbytheuser).
• Streamingcontent—ratherthanwai9ngforarequestfromtheclient‐side,contentobjectsaredownloadedautoma9callyfromtheserverside.Thisapproachissome9mescalled“push”technologybecausetheserverpushesdatatotheclient.
• Cookies—ablockofdatasentbytheserverandstoredbyabrowserasaconsequenceofaspecificuserinterac9on.ThecontentofthedataisWebApp‐specific(e.g.,useriden9fica9ondataoralistofitemsthathavebeenselectedforpurchasebytheuser).
• Applica)onspecificinterfacemechanisms—includeoneormore“macro”interfacemechanismssuchasashoppingcart,creditcardprocessing,orashippingcostcalculator.
19CS‐584/Emory
UsabilityTests • DesignbyWebEteam…executedbyend‐users• Tes9ngsequence…
– Defineasetofusabilitytes9ngcategoriesandiden9fygoalsforeach.– Designteststhatwillenableeachgoaltobeevaluated.– Selectpar9cipantswhowillconductthetests.– Instrumentpar9cipants’interac9onwiththeWebAppwhiletes9ngis
conducted.– DevelopamechanismforassessingtheusabilityoftheWebApp
• Differentlevelsofabstrac9on:– theusabilityofaspecificinterfacemechanism(e.g.,aform)canbe
assessed– theusabilityofacompleteWebpage(encompassinginterface
mechanisms,dataobjectsandrelatedfunc9ons)canbeevaluated– theusabilityofthecompleteWebAppcanbeconsidered.
20CS‐584/Emory
Compa9bilityTes9ng • Compa9bilitytes9ngistodefineasetof“commonlyencountered”client
sidecompu9ngconfigura9onsandtheirvariants• Createatreestructureiden9fying
– eachcompu9ngplaxorm– typicaldisplaydevices– theopera9ngsystemssupportedontheplaxorm– thebrowsersavailable– likelyInternetconnec9onspeeds– similarinforma9on.
• Deriveaseriesofcompa9bilityvalida9ontests– derivedfromexis9nginterfacetests,naviga9ontests,performancetests,and
securitytests.– intentofthesetestsistouncovererrorsorexecu9onproblemsthatcanbe
tracedtoconfigura9ondifferences.
21CS‐584/Emory
Component‐LevelTes9ng
• FocusesonasetofteststhatakempttouncovererrorsinWebAppfunc9ons
• Conven9onalblack‐boxandwhite‐boxtestcasedesignmethodscanbeused
• Databasetes9ngisoKenanintegralpartofthecomponent‐tes9ngregime
22CS‐584/Emory
Naviga9onTes9ng • Thefollowingnaviga9onmechanismsshouldbetested:
– Naviga)onlinks—thesemechanismsincludeinternallinkswithintheWebApp,externallinkstootherWebApps,andanchorswithinaspecificWebpage.
– Redirects—theselinkscomeintoplaywhenauserrequestsanon‐existentURLorselectsalinkwhosedes9na9onhasbeenremovedorwhosenamehaschanged.
– Bookmarks—althoughbookmarksareabrowserfunc9on,theWebAppshouldbetestedtoensurethatameaningfulpage9tlecanbeextractedasthebookmarkiscreated.
– Framesandframesets—testedforcorrectcontent,properlayoutandsizing,downloadperformance,andbrowsercompa9bility
– Sitemaps—Eachsitemapentryshouldbetestedtoensurethatthelinktakestheusertothepropercontentorfunc9onality.
– Internalsearchengines—Searchenginetes9ngvalidatestheaccuracyandcompletenessofthesearch,theerror‐handlingproper9esofthesearchengine,andadvancedsearchfeatures
23CS‐584/Emory
Tes9ngNaviga9onSeman9cs‐I • IstheNSUachievedinitsen9retywithouterror?• Iseverynaviga9onnode(definedforaNSU)reachablewithinthecontext
ofthenaviga9onpathsdefinedfortheNSU?
• IftheNSUcanbeachievedusingmorethanonenaviga9onpath,haseveryrelevantpathbeentested?
• Ifguidanceisprovidedbytheuserinterfacetoassistinnaviga9on,aredirec9onscorrectandunderstandableasnaviga9onproceeds?
• Isthereamechanism(otherthanthebrowser‘back’arrow)forreturningtotheprecedingnaviga9onnodeandtothebeginningofthenaviga9onpath.
• Domechanismsfornaviga9onwithinalargenaviga9onnode(i.e.,alongwebpage)workproperly?
• Ifafunc9onistobeexecutedatanodeandtheuserchoosesnottoprovideinput,cantheremainderoftheNSUbecompleted?
24CS‐584/Emory
Tes9ngNaviga9onSeman9cs‐II • Ifafunc9onisexecutedatanodeandanerrorinfunc9on
processingoccurs,cantheNSUbecompleted?
• Isthereawaytodiscon9nuethenaviga9onbeforeallnodeshavebeenreached,butthenreturntowherethenaviga9onwasdiscon9nuedandproceedfromthere?
• Iseverynodereachablefromthesitemap?Arenodenamesmeaningfultoend‐users?
• IfanodewithinanNSUisreachedfromsomeexternalsource,isitpossibletoprocesstothenextnodeonthenaviga9onpath.Isitpossibletoreturntothepreviousnodeonthenaviga9onpath?
• Doestheuserunderstandhisloca9onwithinthecontentarchitectureastheNSUisexecuted?
25CS‐584/Emory
Configura9onTes9ng • Server‐side
– IstheWebAppfullycompa9blewiththeserverOS?– Aresystemfiles,directories,andrelatedsystemdatacreatedcorrectly
whentheWebAppisopera9onal?– Dosystemsecuritymeasures(e.g.,firewallsorencryp9on)allowthe
WebApptoexecuteandserviceuserswithoutinterferenceorperformancedegrada9on?
– HastheWebAppbeentestedwiththedistributedserverconfigura9on(ifoneexists)thathasbeenchosen?
– IstheWebAppproperlyintegratedwithdatabasesoKware?IstheWebAppsensi9vetodifferentversionsofdatabasesoKware?
– Doserver‐sideWebAppscriptsexecuteproperly?– Havesystemadministratorerrorsbeenexaminedfortheiraffecton
WebAppopera9ons?– Ifproxyserversareused,havedifferencesintheirconfigura9onbeen
addressedwithon‐sitetes9ng?
26CS‐584/Emory
Configura9onTes9ng
• Client‐side– Hardware—CPU,memory,storageandprin9ngdevices– Opera)ngsystems—Linux,MacintoshOS,MicrosoKWindows,a
mobile‐basedOS– BrowsersoMware—InternetExplorer,Mozilla/Netscape,Opera,Safari,
andothers– Userinterfacecomponents—Ac9veX,Javaappletsandothers– Plug‐ins—QuickTime,RealPlayer,andmanyothers– Connec)vity—cable,DSL,regularmodem,T1
• Thenumberofconfigura9onvariablesmustbereducedtoamanageablenumber
27CS‐584/Emory
SecurityTes9ng
• Designedtoprobevulnerabili9esoftheclient‐sideenvironment,thenetworkcommunica9onsthatoccurasdataarepassedfromclienttoserverandbackagain,andtheserver‐sideenvironment
• Ontheclient‐side,vulnerabili9escanoKenbetracedtopre‐exis9ngbugsinbrowsers,e‐mailprograms,orcommunica9onsoKware.
• Ontheserver‐side,vulnerabili9esincludedenial‐of‐serviceakacksandmaliciousscriptsthatcanbepassedalongtotheclient‐sideorusedtodisableserveropera9ons
28CS‐584/Emory
PerformanceTes9ng• Doestheserverresponse9medegradetoapointwhereitisno9ceable
andunacceptable?• Atwhatpoint(intermsofusers,transac9onsordataloading)does
performancebecomeunacceptable?• Whatsystemcomponentsareresponsibleforperformancedegrada9on?• Whatistheaverageresponse9meforusersunderavarietyofloading
condi9ons?• Doesperformancedegrada9onhaveanimpactonsystemsecurity?• IsWebAppreliabilityoraccuracyaffectedastheloadonthesystem
grows?• Whathappenswhenloadsthataregreaterthanmaximumservercapacity
areapplied?
29CS‐584/Emory
LoadTes9ng• TheintentistodeterminehowtheWebAppanditsserver‐sideenvironmentwillrespondtovariousloadingcondi9ons– N,thenumberofconcurrentusers– T,thenumberofon‐linetransac9onsperunitof9me– D,thedataloadprocessedbytheserverpertransac9on
• Overallthroughput,P,iscomputedinthefollowingmanner:
• P=NxTxD
30CS‐584/Emory
StressTes9ng• Doesthesystemdegrade‘gently’ordoestheservershutdownascapacity
isexceeded?• DoesserversoKwaregenerate“servernotavailable”messages?More
generally,areusersawarethattheycannotreachtheserver?• Doestheserverqueuerequestsforresourcesandemptythequeueonce
capacitydemandsdiminish?• Aretransac9onslostascapacityisexceeded?• Isdataintegrityaffectedascapacityisexceeded?• WhatvaluesofN,T,andDforcetheserverenvironmenttofail?Howdoesfailure
manifestitself?Areautomatedno9fica9onssenttotechnicalsupportstaffattheserversite?
• Ifthesystemdoesfail,howlongwillittaketocomebackon‐line?• ArecertainWebAppfunc9ons(e.g.,computeintensivefunc9onality,data
streamingcapabili9es)discon9nuedascapacityreachesthe80or90percentlevel?
31CS‐584/Emory