1. Frontispiece2. Foreword3. Introduction4. TheOWASPTestingFramework5. WebApplicationSecurityTesting

i. IntroductionandObjectivesi. TestingChecklist

ii. InformationGatheringi. ConductSearchEngineDiscoveryandReconnaissanceforInformationLeakage(OTG-INFO-001)ii. FingerprintWebServer(OTG-INFO-002)iii. ReviewWebserverMetafilesforInformationLeakage(OTG-INFO-003)iv. EnumerateApplicationsonWebserver(OTG-INFO-004)v. ReviewWebpageCommentsandMetadataforInformationLeakage(OTG-INFO-005)vi. Identifyapplicationentrypoints(OTG-INFO-006)vii. Mapexecutionpathsthroughapplication(OTG-INFO-007)viii. FingerprintWebApplicationFramework(OTG-INFO-008)ix. FingerprintWebApplication(OTG-INFO-009)x. MapApplicationArchitecture(OTG-INFO-010)

iii. ConfigurationandDeploymentManagementTestingi. TestNetwork/InfrastructureConfiguration(OTG-CONFIG-001)ii. TestApplicationPlatformConfiguration(OTG-CONFIG-002)iii. TestFileExtensionsHandlingforSensitiveInformation(OTG-CONFIG-003)iv. ReviewOld,BackupandUnreferencedFilesforSensitiveInformation(OTG-CONFIG-004)v. EnumerateInfrastructureandApplicationAdminInterfaces(OTG-CONFIG-005)vi. TestHTTPMethods(OTG-CONFIG-006)vii. TestHTTPStrictTransportSecurity(OTG-CONFIG-007)viii. TestRIAcrossdomainpolicy(OTG-CONFIG-008)

iv. IdentityManagementTestingi. TestRoleDefinitions(OTG-IDENT-001)ii. TestUserRegistrationProcess(OTG-IDENT-002)iii. TestAccountProvisioningProcess(OTG-IDENT-003)iv. TestingforAccountEnumerationandGuessableUserAccount(OTG-IDENT-004)v. TestingforWeakorunenforcedusernamepolicy(OTG-IDENT-005)

v. AuthenticationTestingi. TestingforCredentialsTransportedoveranEncryptedChannel(OTG-AUTHN-001)ii. Testingfordefaultcredentials(OTG-AUTHN-002)iii. TestingforWeaklockoutmechanism(OTG-AUTHN-003)iv. Testingforbypassingauthenticationschema(OTG-AUTHN-004)v. Testrememberpasswordfunctionality(OTG-AUTHN-005)vi. TestingforBrowsercacheweakness(OTG-AUTHN-006)vii. TestingforWeakpasswordpolicy(OTG-AUTHN-007)viii. TestingforWeaksecurityquestion/answer(OTG-AUTHN-008)ix. Testingforweakpasswordchangeorresetfunctionalities(OTG-AUTHN-009)x. TestingforWeakerauthenticationinalternativechannel(OTG-AUTHN-010)

vi. AuthorizationTestingi. TestingDirectorytraversal/fileinclude(OTG-AUTHZ-001)ii. Testingforbypassingauthorizationschema(OTG-AUTHZ-002)iii. TestingforPrivilegeEscalation(OTG-AUTHZ-003)iv. TestingforInsecureDirectObjectReferences(OTG-AUTHZ-004)

TableofContents

OwaspTestingGuidev4

2

vii. SessionManagementTestingi. TestingforBypassingSessionManagementSchema(OTG-SESS-001)ii. TestingforCookiesattributes(OTG-SESS-002)iii. TestingforSessionFixation(OTG-SESS-003)iv. TestingforExposedSessionVariables(OTG-SESS-004)v. TestingforCrossSiteRequestForgery(CSRF)(OTG-SESS-005)vi. Testingforlogoutfunctionality(OTG-SESS-006)vii. TestSessionTimeout(OTG-SESS-007)viii. TestingforSessionpuzzling(OTG-SESS-008)

viii. InputValidationTestingi. TestingforReflectedCrossSiteScripting(OTG-INPVAL-001)ii. TestingforStoredCrossSiteScripting(OTG-INPVAL-002)iii. TestingforHTTPVerbTampering(OTG-INPVAL-003)iv. TestingforHTTPParameterpollution(OTG-INPVAL-004)v. TestingforSQLInjection(OTG-INPVAL-005)

i. OracleTestingii. MySQLTestingiii. SQLServerTestingiv. TestingPostgreSQL(fromOWASPBSP)v. MSAccessTestingvi. TestingforNoSQLinjection

vi. TestingforLDAPInjection(OTG-INPVAL-006)vii. TestingforORMInjection(OTG-INPVAL-007)viii. TestingforXMLInjection(OTG-INPVAL-008)ix. TestingforSSIInjection(OTG-INPVAL-009)x. TestingforXPathInjection(OTG-INPVAL-010)xi. IMAP/SMTPInjection(OTG-INPVAL-011)xii. TestingforCodeInjection(OTG-INPVAL-012)

i. TestingforLocalFileInclusionii. TestingforRemoteFileInclusion

xiii. TestingforCommandInjection(OTG-INPVAL-013)xiv. TestingforBufferoverflow(OTG-INPVAL-014)

i. TestingforHeapoverflowii. TestingforStackoverflowiii. TestingforFormatstring

xv. Testingforincubatedvulnerabilities(OTG-INPVAL-015)xvi. TestingforHTTPSplitting/Smuggling(OTG-INPVAL-016)

ix. TestingforErrorHandlingi. AnalysisofErrorCodes(OTG-ERR-001)ii. AnalysisofStackTraces(OTG-ERR-002)

x. TestingforweakCryptographyi. TestingforWeakSSL/TLSCiphers,InsufficientTransportLayerProtection(OTG-CRYPST-001)ii. TestingforPaddingOracle(OTG-CRYPST-002)iii. TestingforSensitiveinformationsentviaunencryptedchannels(OTG-CRYPST-003)

xi. BusinessLogicTestingi. TestBusinessLogicDataValidation(OTG-BUSLOGIC-001)ii. TestAbilitytoForgeRequests(OTG-BUSLOGIC-002)iii. TestIntegrityChecks(OTG-BUSLOGIC-003)iv. TestforProcessTiming(OTG-BUSLOGIC-004)v. TestNumberofTimesaFunctionCanbeUsedLimits(OTG-BUSLOGIC-005)vi. TestingfortheCircumventionofWorkFlows(OTG-BUSLOGIC-006)vii. TestDefensesAgainstApplicationMis-use(OTG-BUSLOGIC-007)viii. TestUploadofUnexpectedFileTypes(OTG-BUSLOGIC-008)

OwaspTestingGuidev4

3

ix. TestUploadofMaliciousFiles(OTG-BUSLOGIC-009)xii. ClientSideTesting

i. TestingforDOMbasedCrossSiteScripting(OTG-CLIENT-001)ii. TestingforJavaScriptExecution(OTG-CLIENT-002)iii. TestingforHTMLInjection(OTG-CLIENT-003)iv. TestingforClientSideURLRedirect(OTG-CLIENT-004)v. TestingforCSSInjection(OTG-CLIENT-005)vi. TestingforClientSideResourceManipulation(OTG-CLIENT-006)vii. TestCrossOriginResourceSharing(OTG-CLIENT-007)viii. TestingforCrossSiteFlashing(OTG-CLIENT-008)ix. TestingforClickjacking(OTG-CLIENT-009)x. TestingWebSockets(OTG-CLIENT-010)xi. TestWebMessaging(OTG-CLIENT-011)xii. TestLocalStorage(OTG-CLIENT-012)

6. Reporting7. Appendix

i. AppendixA:TestingToolsii. AppendixB:SuggestedReadingiii. AppendixC:FuzzVectorsiv. AppendixD:EncodedInjection

OwaspTestingGuidev4

4

“Openandcollaborativeknowledge:thatistheOWASPway.”WithV4werealizedanewguidethatwillbethestandardde-factoguidetoperformWebApplicationPenetrationTesting.--MatteoMeucci

OWASPthanksthemanyauthors,reviewers,andeditorsfortheirhardworkinbringingthisguidetowhereitistoday.IfyouhaveanycommentsorsuggestionsontheTestingGuide,pleasee-mailtheTestingGuidemaillist:

http://lists.owasp.org/mailman/listinfo/owasp-testing

Ordropane-mailtotheprojectleaders:AndrewMullerMatteoMeucci

TheOWASPTestingGuideversion4improvesonversion3inthreeways:

1. ThisversionoftheTestingGuideintegrateswiththetwootherflagshipOWASPdocumentationproducts:theDevelopersGuideandtheCodeReviewGuide.ToachievethiswealignedthetestingcategoriesandtestnumberingwiththoseinotherOWASPproducts.TheaimoftheTestingandCodeReviewGuidesistoevaluatethesecuritycontrolsdescribedbytheDevelopersGuide.

2. Allchaptershavebeenimprovedandtestcasesexpandedto87(64testcasesinv3)includingtheintroductionoffournewchaptersandcontrols:

IdentityManagementTestingErrorHandlingCryptographyClientSideTesting

3. ThisversionoftheTestingGuideencouragesthecommunitynottosimplyacceptthetestcasesoutlinedinthisguide.Weencouragesecuritytesterstointegratewithothersoftwaretestersanddevisetestcasesspecifictothetargetapplication.AswefindtestcasesthathavewiderapplicabilityweencouragethesecuritytestingcommunitytosharethemandcontributethemtotheTestingGuide.ThiswillcontinuetobuildtheapplicationsecuritybodyofknowledgeandallowthedevelopmentoftheTestingGuidetobeaniterativeratherthanmonolithicprocess.

Copyright(c)2014TheOWASPFoundation.

ThisdocumentisreleasedundertheCreativeCommons2.5License.Pleasereadandunderstandthelicenseandcopyrightconditions.

TheTestingGuidev4willbereleasedin2014.TheTestingguideoriginatedin2003withDanCuthbertasoneoftheoriginaleditors.ItwashandedovertoEoinKearyin2005andtransformedintoawiki.MatteoMeuccihastakenontheTestingguideandisnowtheleadoftheOWASPTestingGuideProject.From2012AndrewMullerco-leadershiptheprojectwithMatteoMeucci.

Frontispiece

WelcometotheOWASPTestingGuide4.0

Version4.0

CopyrightandLicense

RevisionHistory

OwaspTestingGuidev4

5Frontispiece

https://www.owasp.org/index.php/User:Mmeuccihttp://lists.owasp.org/mailman/listinfo/owasp-testingmailto:andrew.muller@owasp.orgmailto:matteo.meucci@owasp.orghttp://creativecommons.org/licenses/by-sa/2.5/

2014

"OWASPTestingGuide",Version4.0

15thSeptember,2008

"OWASPTestingGuide",Version3.0

December25,2006

"OWASPTestingGuide",Version2.0

July14,2004

"OWASPWebApplicationPenetrationChecklist",Version1.1

December2004

"TheOWASPTestingGuide",Version1.0

AndrewMuller:OWASPTestingGuideLeadsince2013.

MatteoMeucci:OWASPTestingGuideLeadsince2007.

EoinKeary:OWASPTestingGuide2005-2007Lead.

DanielCuthbert:OWASPTestingGuide2003-2005Lead.

MatteoMeucciPavolLuptakMarcoMoranaGiorgioFedonStefanoDiPaolaGianricoIngrossoGiuseppeBonfàAndrewMullerRobertWinkelRobertoSuggiLiveraniRobertSmithTripurariRaiThomasRyanTimBertelsCecilSuAungKhAntNorbertSzeteiMichaelBomanWagnerEliasKevinHorvat

Editors

v4Authors

OwaspTestingGuidev4

6Frontispiece

TomBrennanJuanGalianaLaraSumitSiddharthMikeHryekewiczSimonBennettsRaySchippersRaulSilesJayantaKarmakarBradCauseyVicenteAguileraIsmaelGonçalvesDavidFernTomEstonKevinHorvathRickMitchellEduardoCastellanosSimoneOnofriHarwordSheenAmroAlOlaqiSuhasDesaiRyanDewhurstZakiAkhmadDavideDanelonAlexanderAntukhThomasKalamarisAlexanderVavousisClerkendwellerChristianHeinrichBabuArokiadasRobBarnesBenWalther

DavideDanelonAndreaRosignoliIreneAbezgauzLodeVanstechelmanSebastienGioriaYiannisPavlosoglouAdityaBalapure

AnuragAgarwwalDanieleBellucciArielCoronelStefanoDiPaolaGiorgioFedonAdamGoodmanChristianHeinrichKevinHorvathGianricoIngrosso

v4Reviewers

v3Authors

OwaspTestingGuidev4

7Frontispiece

RobertoSuggiLiveraniKuza55PavolLuptakFerruhMavitunaMarcoMellaMatteoMeucciMarcoMoranaAntonioParataCecilSuHarishSkandaSureddyMarkRoxberryAndrewVanderStock

MarcoCovaKevinFullerMatteoMeucciNamNguyenRickMitchell

VicenteAguileraMauroBregolinTomBrennanGaryBurnsLucaCarettoniDanCornellMarkCurpheyDanielCuthbertSebastienDeleersnyderStephenDeVriesStefanoDiPaolaDavidEndlerGiorgioFedonJavierFernández-SanguinoGlynGeogheganStanGuzikMadhuraHalasgikarEoinKearyDavidLitchfieldAndreaLombardiniRalphM.LosClaudioMerloniMatteoMeucciMarcoMoranaLauraNunezGunterOllmannAntonioParataYiannisPavlosoglouCarloPelliccioniHarinathPudipeddi

v3Reviewers

v2Authors

OwaspTestingGuidev4

8Frontispiece

AlbertoRevelliMarkRoxberryTomRyanAnushShettyLarryShieldsDafyddStuddardAndrewvanderStockArielWaissbeinJeffWilliamsTusharVartak

VicenteAguileraMarcoBelottiMauroBregolinMarcoCovaDanielCuthbertPaulDaviesStefanoDiPaolaMatteoG.P.FloraSimonaFortiDarrellGroundyEoinKearyJamesKistKatieMcDowellMarcoMellaMatteoMeucciSyedMohamedA.AntonioParataAlbertoRevelliMarkRoxberryDaveWichers

Java,JavaWebServer,andJSPareregisteredtrademarksofSunMicrosystems,Inc.Merriam-WebsterisatrademarkofMerriam-Webster,Inc.MicrosoftisaregisteredtrademarkofMicrosoftCorporation.OctaveisaservicemarkofCarnegieMellonUniversity.VeriSignandThawteareregisteredtrademarksofVeriSign,Inc.VisaisaregisteredtrademarkofVISAUSA.OWASPisaregisteredtrademarkoftheOWASPFoundation

Allotherproductsandcompanynamesmaybetrademarksoftheirrespectiveowners.Useofaterminthisdocumentshouldnotberegardedasaffectingthevalidityofanytrademarkorservicemark.

v2Reviewers

Trademarks

OwaspTestingGuidev4

9Frontispiece

Theproblemofinsecuresoftwareisperhapsthemostimportanttechnicalchallengeofourtime.Thedramaticriseofwebapplicationsenablingbusiness,socialnetworkingetchasonlycompoundedtherequirementstoestablisharobustapproachtowritingandsecuringourInternet,WebApplicationsandData.

AtTheOpenWebApplicationSecurityProject(OWASP),we'retryingtomaketheworldaplacewhereinsecuresoftwareistheanomaly,notthenorm.TheOWASPTestingGuidehasanimportantroletoplayinsolvingthisseriousissue.Itisvitallyimportantthatourapproachtotestingsoftwareforsecurityissuesisbasedontheprinciplesofengineeringandscience.Weneedaconsistent,repeatableanddefinedapproachtotestingwebapplications.Aworldwithoutsomeminimalstandardsintermsofengineeringandtechnologyisaworldinchaos.

Itgoeswithoutsayingthatyoucan'tbuildasecureapplicationwithoutperformingsecuritytestingonit.Testingispartofawiderapproachtobuildingasecuresystem.Manysoftwaredevelopmentorganizationsdonotincludesecuritytestingaspartoftheirstandardsoftwaredevelopmentprocess.Whatisevenworseisthatmanysecurityvendorsdelivertestingwithvaryingdegreesofqualityandrigor.

Securitytesting,byitself,isn'taparticularlygoodstandalonemeasureofhowsecureanapplicationis,becausethereareaninfinitenumberofwaysthatanattackermightbeabletomakeanapplicationbreak,anditsimplyisn'tpossibletotestthemall.Wecan'thackourselvessecureandweonlyhavealimitedtimetotestanddefendwhereanattackerdoesnothavesuchconstraints.

InconjunctionwithotherOWASPprojectssuchastheCodereviewGuide,theDevelopmentGuideandtoolssuchasOWASPZAP,thisisagreatstarttowardsbuildingandmaintainingsecureapplications.TheDevelopmentGuidewillshowyourprojecthowtoarchitectandbuildasecureapplication,theCodeReviewGuidewilltellyouhowtoverifythesecurityofyourapplication'ssourcecode,andthisTestingGuidewillshowyouhowtoverifythesecurityofyourrunningapplication.Ihighlyrecommendusingtheseguidesaspartofyourapplicationsecurityinitiatives.

Creatingaguidelikethisisahugeundertaking,requiringtheexpertiseofhundredsofpeoplearoundtheworld.Therearemanydifferentwaystotestforsecurityflawsandthisguidecapturestheconsensusoftheleadingexpertsonhowtoperformthistestingquickly,accurately,andefficiently.OWASPgiveslikemindedsecurityfolkstheabilitytoworktogetherandformaleadingpracticeapproachtoasecurityproblem.

Theimportanceofhavingthisguideavailableinacompletelyfreeandopenwayisimportantforthefoundationsmission.Itgivesanyonetheabilitytounderstandthetechniquesusedtotestforcommonsecurityissues.Securityshouldnotbeablackartorclosedsecretthatonlyafewcanpractice.ItshouldbeopentoallandnotexclusivetosecuritypractitionersbutalsoQA,DevelopersandTechnicalManagers.Theprojecttobuildthisguidekeepsthisexpertiseinthehandsofthepeoplewhoneedit-you,meandanyonethatisinvolvedinbuildingsoftware.

Thisguidemustmakeitswayintothehandsofdevelopersandsoftwaretesters.Therearenotnearlyenoughapplicationsecurityexpertsintheworldtomakeanysignificantdentintheoverallproblem.Theinitialresponsibilityforapplicationsecuritymustfallontheshouldersofthedevelopers,theywritethecode.Itshouldn'tbeasurprisethatdevelopersaren'tproducingsecurecodeifthey'renottestingforitorconsiderthetypesofbugswhichintroducevulnerability.

Keepingthisinformationuptodateisacriticalaspectofthisguideproject.Byadoptingthewikiapproach,theOWASPcommunitycanevolveandexpandtheinformationinthisguidetokeeppacewiththefastmovingapplicationsecuritythreatlandscape.

ThisGuideisagreattestamenttothepassionandenergyourmembersandprojectvolunteershaveforthissubject.Itshallcertainlyhelpchangetheworldalineofcodeatatime.

ForewordbyEoinKeary,OWASPGlobalBoard

WhyOWASP?

OwaspTestingGuidev4

10Foreword

https://www.owasp.org/index.php/Eoin_Kearyhttps://www.owasp.org/index.php/Building_Guidehttps://www.owasp.org/index.php/Code_Review_Guidehttps://www.owasp.org/index.php/Testing_Guide

Youshouldadoptthisguideinyourorganization.Youmayneedtotailortheinformationtomatchyourorganization'stechnologies,processes,andorganizationalstructure.

Ingeneralthereareseveraldifferentroleswithinorganizationsthatmayusethisguide:

Developersshouldusethisguidetoensurethattheyareproducingsecurecode.Thesetestsshouldbeapartofnormalcodeandunittestingprocedures.

SoftwaretestersandQAshouldusethisguidetoexpandthesetoftestcasestheyapplytoapplications.Catchingthesevulnerabilitiesearlysavesconsiderabletimeandeffortlater.

Securityspecialistsshouldusethisguideincombinationwithothertechniquesasonewaytoverifythatnosecurityholeshavebeenmissedinanapplication.

ProjectManagersshouldconsiderthereasonthisguideexistsandthatsecurityissuesaremanifestedviabugsincodeanddesign.

Themostimportantthingtorememberwhenperformingsecuritytestingistocontinuouslyre-prioritize.Thereareaninfinitenumberofpossiblewaysthatanapplicationcouldfail,andorganizationsalwayshavelimitedtestingtimeandresources.Besuretimeandresourcesarespentwisely.Trytofocusonthesecurityholesthatarearealrisktoyourbusiness.Trytocontextualizeriskintermsoftheapplicationanditsusecases.

Thisguideisbestviewedasasetoftechniquesthatyoucanusetofinddifferenttypesofsecurityholes.Butnotallthetechniquesareequallyimportant.Trytoavoidusingtheguideasachecklist,newvulnerabilitiesarealwaysmanifestingandnoguidecanbeanexhaustivelistof"thingstotestfor",butratheragreatplacetostart.

Thereareanumberofcompaniessellingautomatedsecurityanalysisandtestingtools.Rememberthelimitationsofthesetoolssothatyoucanusethemforwhatthey'regoodat.AsMichaelHowardputitatthe2006OWASPAppSecConferenceinSeattle.

"Toolsdonotmakesoftwaresecure!Theyhelpscaletheprocessandhelpenforcepolicy."

Mostimportantly,thesetoolsaregeneric-meaningthattheyarenotdesignedforyourcustomcode,butforapplicationsingeneral.Thatmeansthatwhiletheycanfindsomegenericproblems,theydonothaveenoughknowledgeofyourapplicationtoallowthemtodetectmostflaws.Inmyexperience,themostserioussecurityissuesaretheonesthatarenotgeneric,butdeeplyintertwinedinyourbusinesslogicandcustomapplicationdesign.

Thesetoolscanalsobeseductive,sincetheydofindlotsofpotentialissues.Whilerunningthetoolsdoesn'ttakemuchtime,eachoneofthepotentialproblemstakestimetoinvestigateandverify.Ifthegoalistofindandeliminatethemostseriousflawsasquicklyaspossible,considerwhetheryourtimeisbestspentwithautomatedtoolsorwiththetechniquesdescribedinthisguide.Still,thesetoolsarecertainlypartofawell-balancedapplicationsecurityprogram.Usedwisely,theycansupportyouroverallprocessestoproducemoresecurecode.

Ifyou'rebuilding,designingortestingsoftware,Istronglyencourageyoutogetfamiliarwiththesecuritytestingguidanceinthisdocument.Itisagreatroadmapfortestingthemostcommonissuesfacingapplicationstoday,butitisnotexhaustive.Ifyoufinderrors,pleaseaddanotetothediscussionpageormakethechangeyourself.You'llbehelpingthousandsofotherswhousethisguide.

Pleaseconsiderjoiningusasanindividualorcorporatemembersothatwecancontinuetoproducematerialslikethis

TailoringandPrioritizing

TheRoleofAutomatedTools

CalltoAction

OwaspTestingGuidev4

11Foreword

https://www.owasp.org/index.php/OWASP_AppSec_Seattle_2006/Agendahttps://www.owasp.org/index.php/Membership

testingguideandalltheothergreatprojectsatOWASP.

Thankyoutoallthepastandfuturecontributorstothisguide,yourworkwillhelptomakeapplicationsworldwidemoresecure.

--EoinKeary,OWASPBoardMember,April19,2013

OwaspTestingGuidev4

12Foreword

https://www.owasp.org/index.php/Eoin_Keary

TheOWASPTestingProjecthasbeenindevelopmentformanyyears.Theaimoftheprojectistohelppeopleunderstandthewhat,why,when,where,andhowoftestingwebapplications.Theprojecthasdeliveredacompletetestingframework,notmerelyasimplechecklistorprescriptionofissuesthatshouldbeaddressed.Readerscanusethisframeworkasatemplatetobuildtheirowntestingprogramsortoqualifyotherpeople’sprocesses.TheTestingGuidedescribesindetailboththegeneraltestingframeworkandthetechniquesrequiredtoimplementtheframeworkinpractice.

WritingtheTestingGuidehasproventobeadifficulttask.Itwasachallengetoobtainconsensusanddevelopcontentthatallowedpeopletoapplytheconceptsdescribedintheguide,whilealsoenablingthemtoworkintheirownenvironmentandculture.Itwasalsoachallengetochangethefocusofwebapplicationtestingfrompenetrationtestingtotestingintegratedinthesoftwaredevelopmentlifecycle.

However,thegroupisverysatisfiedwiththeresultsoftheproject.Manyindustryexpertsandsecurityprofessionals,someofwhomareresponsibleforsoftwaresecurityatsomeofthelargestcompaniesintheworld,arevalidatingthetestingframework.Thisframeworkhelpsorganizationstesttheirwebapplicationsinordertobuildreliableandsecuresoftware.Theframeworkdoesnotsimplyhighlightingareasofweakness,althoughthelatteriscertainlyabyproductofmanyoftheOWASPguidesandchecklists.Assuch,harddecisionshadtopbemadeabouttheappropriatenessofcertaintestingtechniquesandtechnologies.Thegroupfullyunderstandsthatnoteveryonewillagreeuponallofthesedecisions.However,OWASPisabletotakethehighgroundandchangecultureovertimethroughawarenessandeducationbasedonconsensusandexperience.

Therestofthisguideisorganizedasfollows:Thisintroductioncoversthepre-requisitesoftestingwebapplicationsandthescopeoftesting.Italsocoverstheprinciplesofsuccessfultestingandtestingtechniques.Chapter3presentstheOWASPTestingFrameworkandexplainsitstechniquesandtasksinrelationtothevariousphasesofthesoftwaredevelopmentlifecycle.Chapter4covershowtotestforspecificvulnerabilities(e.g.,SQLInjection)bycodeinspectionandpenetrationtesting.

Abasictenetofsoftwareengineeringisthatyoucan'tcontrolwhatyoucan'tmeasure[1].Securitytestingisnodifferent.Unfortunately,measuringsecurityisanotoriouslydifficultprocess.Thistopicwillnotbecoveredindetailhere,asitwouldtakeaguideonitsown(foranintroduction,see[2]).

Oneaspectthatshouldbeemphasizedisthatsecuritymeasurementsareaboutboththespecifictechnicalissues(e.g.,howprevalentacertainvulnerabilityis)andhowtheseissuesaffecttheeconomicsofsoftware.Mosttechnicalpeoplewillatleastunderstandthebasicissues,ortheymayhaveadeeperunderstandingofthevulnerabilities.Sadly,fewareabletotranslatethattechnicalknowledgeintomonetarytermsandquantifythepotentialcostofvulnerabilitiestotheapplicationowner'sbusiness.Untilthishappens,CIOswillnotbeabletodevelopanaccuratereturnonsecurityinvestmentand,subsequently,assignappropriatebudgetsforsoftwaresecurity.

Whileestimatingthecostofinsecuresoftwaremayappearadauntingtask,therehasbeenasignificantamountofworkinthisdirection.Forexample,inJune2002,theUSNationalInstituteofStandards(NIST)publishedasurveyonthecostofinsecuresoftwaretotheUSeconomyduetoinadequatesoftwaretesting[3].Interestingly,theyestimatethatabettertestinginfrastructurewouldsavemorethanathirdofthesecosts,orabout$22billionayear.Morerecently,thelinksbetweeneconomicsandsecurityhavebeenstudiedbyacademicresearchers.See[4]formoreinformationaboutsomeoftheseefforts.

Theframeworkdescribedinthisdocumentencouragespeopletomeasuresecuritythroughouttheentiredevelopmentprocess.Theycanthenrelatethecostofinsecuresoftwaretotheimpactithasonthebusiness,andconsequentlydevelop

TestingGuideIntroduction

TheOWASPTestingProject

MeasuringSecurity:theEconomicsofInsecureSoftware

OwaspTestingGuidev4

13Introduction

appropriatebusinessprocessesandassignresourcestomanagetherisk.Rememberthatmeasuringandtestingwebapplicationsisevenmorecriticalthanforothersoftware,sincewebapplicationsareexposedtomillionsofusersthroughtheInternet.

Duringthedevelopmentlifecycleofawebapplicationmanythingsneedtobetested,butwhatdoestestingactuallymean?TheMerriam-WebsterDictionarydescribestestingas:

Toputtotestorproof.Toundergoatest.Tobeassignedastandingorevaluationbasedontests.

Forthepurposesofthisdocumenttestingisaprocessofcomparingthestateofasystemorapplicationagainstasetofcriteria.Inthesecurityindustrypeoplefrequentlytestagainstasetofmentalcriteriathatareneitherwelldefinednorcomplete.Asaresultofthis,manyoutsidersregardsecuritytestingasablackart.Theaimofthisdocumentistochangethatperceptionandtomakeiteasierforpeoplewithoutin-depthsecurityknowledgetomakeadifferenceintesting.

Thisdocumentisdesignedtohelporganizationsunderstandwhatcomprisesatestingprogram,andtohelpthemidentifythestepsthatneedtobeundertakentobuildandoperateatestingprogramonwebapplications.Theguidegivesabroadviewoftheelementsrequiredtomakeacomprehensivewebapplicationsecurityprogram.Thisguidecanbeusedasareferenceguideandasamethodologytohelpdeterminethegapbetweenexistingpracticesandindustrybestpractices.Thisguideallowsorganizationstocomparethemselvesagainstindustrypeers,tounderstandthemagnitudeofresourcesrequiredtotestandmaintainsoftware,ortoprepareforanaudit.Thischapterdoesnotgointothetechnicaldetailsofhowtotestanapplication,astheintentistoprovideatypicalsecurityorganizationalframework.Thetechnicaldetailsabouthowtotestanapplication,aspartofapenetrationtestorcodereview,willbecoveredintheremainingpartsofthisdocument.

Mostpeopletodaydon’ttestsoftwareuntilithasalreadybeencreatedandisinthedeploymentphaseofitslifecycle(i.e.,codehasbeencreatedandinstantiatedintoaworkingwebapplication).Thisisgenerallyaveryineffectiveandcost-prohibitivepractice.OneofthebestmethodstopreventsecuritybugsfromappearinginproductionapplicationsistoimprovetheSoftwareDevelopmentLifeCycle(SDLC)byincludingsecurityineachofitsphases.AnSDLCisastructureimposedonthedevelopmentofsoftwareartefacts.IfanSDLCisnotcurrentlybeingusedinyourenvironment,itistimetopickone!ThefollowingfigureshowsagenericSDLCmodelaswellasthe(estimated)increasingcostoffixingsecuritybugsinsuchamodel.

WhatisTesting?

WhyPerformTesting?

WhentoTest?

OwaspTestingGuidev4

14Introduction

Figure1:GenericSDLCModel

CompaniesshouldinspecttheiroverallSDLCtoensurethatsecurityisanintegralpartofthedevelopmentprocess.SDLCsshouldincludesecurityteststoensuresecurityisadequatelycoveredandcontrolsareeffectivethroughoutthedevelopmentprocess.

Itcanbehelpfultothinkofsoftwaredevelopmentasacombinationofpeople,process,andtechnology.Ifthesearethefactorsthat"create"software,thenitislogicalthatthesearethefactorsthatmustbetested.Todaymostpeoplegenerallytestthetechnologyorthesoftwareitself.

Aneffectivetestingprogramshouldhavecomponentsthattest:

People–toensurethatthereisadequateeducationandawareness;Process–toensurethatthereareadequatepoliciesandstandardsandthatpeopleknowhowtofollowthesepolicies;Technology–toensurethattheprocesshasbeeneffectiveinitsimplementation.

Unlessaholisticapproachisadopted,testingjustthetechnicalimplementationofanapplicationwillnotuncovermanagementoroperationalvulnerabilitiesthatcouldbepresent.Bytestingthepeople,policies,andprocesses,anorganizationcancatchissuesthatwouldlatermanifestthemselvesintodefectsinthetechnology,thuseradicatingbugsearlyandidentifyingtherootcausesofdefects.Likewise,testingonlysomeofthetechnicalissuesthatcanbepresentinasystemwillresultinanincompleteandinaccuratesecuritypostureassessment.

DenisVerdon,HeadofInformationSecurityatFidelityNationalFinancialpresentedanexcellentanalogyforthismisconceptionattheOWASPAppSec2004ConferenceinNewYork[5]:"Ifcarswerebuiltlikeapplications[...]safetytestswouldassumefrontalimpactonly.Carswouldnotberolltested,ortestedforstabilityinemergencymaneuvers,brakeeffectiveness,sideimpact,andresistancetotheft."

AswithallOWASPprojects,wewelcomecommentsandfeedback.Weespeciallyliketoknowthatourworkisbeingusedandthatitiseffectiveandaccurate.

Therearesomecommonmisconceptionswhendevelopingatestingmethodologytofindsecuritybugsinsoftware.Thischaptercoverssomeofthebasicprinciplesthatprofessionalsshouldtakeintoaccountwhenperformingsecuritytestsonsoftware.

Whileitistemptingtothinkthatasecurityscannerorapplicationfirewallwillprovidemanydefensesagainstattackoridentifyamultitudeofproblems,inrealitythereisnosilverbullettotheproblemofinsecuresoftware.Applicationsecurityassessmentsoftware,whileusefulasafirstpasstofindlow-hangingfruit,isgenerallyimmatureandineffectiveatin-depthassessmentsorprovidingadequatetestcoverage.Rememberthatsecurityisaprocessandnotaproduct.

Overthelastfewyears,securityprofessionalshavecometorealizethefallacyofthepatch-and-penetratemodelthatwaspervasiveininformationsecurityduringthe1990’s.Thepatch-and-penetratemodelinvolvesfixingareportedbug,butwithoutproperinvestigationoftherootcause.Thismodelisusuallyassociatedwiththewindowofvulnerabilityshowninthefigurebelow.Theevolutionofvulnerabilitiesincommonsoftwareusedworldwidehasshowntheineffectivenessofthismodel.Formoreinformationaboutthewindowofvulnerabilitypleasereferto[6].

WhattoTest?

FeedbackandComments

PrinciplesofTesting

ThereisNoSilverBullet

ThinkStrategically,NotTactically

OwaspTestingGuidev4

15Introduction

http://www.fnf.com

Vulnerabilitystudies[7]haveshownthatwiththereactiontimeofattackersworldwide,thetypicalwindowofvulnerabilitydoesnotprovideenoughtimeforpatchinstallation,sincethetimebetweenavulnerabilitybeinguncoveredandanautomatedattackagainstitbeingdevelopedandreleasedisdecreasingeveryyear.

Thereareseveralincorrectassumptionsinthepatch-and-penetratemodel.Manyusersbelievethatpatchesinterferewithnormaloperationsandmightbreakexistingapplications.Itisalsoincorrecttoassumethatallusersareawareofnewlyreleasedpatches.Consequentlynotallusersofaproductwillapplypatches,eitherbecausetheythinkpatchingmayinterferewithhowthesoftwareworksorbecausetheylackknowledgeabouttheexistenceofthepatch.

Figure2:WindowofVulnerability

ItisessentialtobuildsecurityintotheSoftwareDevelopmentLifeCycle(SDLC)topreventreoccurringsecurityproblemswithinanapplication.DeveloperscanbuildsecurityintotheSDLCbydevelopingstandards,policies,andguidelinesthatfitandworkwithinthedevelopmentmethodology.Threatmodelingandothertechniquesshouldbeusedtohelpassignappropriateresourcestothosepartsofasystemthataremostatrisk.

TheSDLCisaprocessthatiswell-knowntodevelopers.ByintegratingsecurityintoeachphaseoftheSDLC,itallowsforaholisticapproachtoapplicationsecuritythatleveragestheproceduresalreadyinplacewithintheorganization.BeawarethatwhilethenamesofthevariousphasesmaychangedependingontheSDLCmodelusedbyanorganization,eachconceptualphaseofthearchetypeSDLCwillbeusedtodeveloptheapplication(i.e.,define,design,develop,deploy,maintain).Eachphasehassecurityconsiderationsthatshouldbecomepartoftheexistingprocess,toensureacost-effectiveandcomprehensivesecurityprogram.

ThereareseveralsecureSDLCframeworksthatexistthatprovidebothdescriptiveandprescriptiveadvice.WhetherapersontakesdescriptiveorprescriptiveadvicedependsonthematurityoftheSDLCprocess.Essentially,prescriptiveadviceshowshowthesecureSDLCshouldwork,anddescriptiveadviceshowshowitsusedintherealworld.Bothhavetheirplace.Forexample,ifyoudon'tknowwheretostart,aprescriptiveframeworkcanprovideamenuofpotentialsecuritycontrolsthatcanbeappliedwithintheSDLC.Descriptiveadvicecanthenhelpdrivethedecisionprocessbypresentingwhathasworkedwellforotherorganizations.DescriptivesecureSDLCsincludeBSIMM-V;andtheprescriptivesecureSDLCsinculdeOWASP'sOpenSoftwareAssuranceMaturityModel(OpenSAMM)andISO/IEC27034Parts1-8,partsofwhicharestillindevelopment.

TheSDLCisKing

OwaspTestingGuidev4

16Introduction

WhenabugisdetectedearlywithintheSDLCitcanbeaddressedfasterandatalowercost.Asecuritybugisnodifferentfromafunctionalorperformance-basedbuginthisregard.AkeystepinmakingthispossibleistoeducatethedevelopmentandQAteamsaboutcommonsecurityissuesandthewaystodetectandpreventthem.Althoughnewlibraries,tools,orlanguagescanhelpdesignbetterprograms(withfewersecuritybugs),newthreatsariseconstantlyanddevelopersmustbeawareofthethreatsthataffectthesoftwaretheyaredeveloping.Educationinsecuritytestingalsohelpsdevelopersacquiretheappropriatemindsettotestanapplicationfromanattacker'sperspective.Thisallowseachorganizationtoconsidersecurityissuesaspartoftheirexistingresponsibilities.

Itisimportanttoknowhowmuchsecurityagivenprojectwillrequire.Theinformationandassetsthataretobeprotectedshouldbegivenaclassificationthatstateshowtheyaretobehandled(e.g.,confidential,secret,topsecret).Discussionsshouldoccurwithlegalcounciltoensurethatanyspecificsecurityrequirementswillbemet.IntheUSArequirementsmightcomefromfederalregulations,suchastheGramm-Leach-BlileyAct[8],orfromstatelaws,suchastheCaliforniaSB-1386[9].FororganizationsbasedinEUcountries,bothcountry-specificregulationandEUDirectivesmayapply.Forexample,Directive96/46/EC4[10]makesitmandatorytotreatpersonaldatainapplicationswithduecare,whatevertheapplication.

Successfullytestinganapplicationforsecurityvulnerabilitiesrequiresthinking"outsideofthebox."Normalusecaseswilltestthenormalbehavioroftheapplicationwhenauserisusingitinthemannerthatisexpected.Goodsecuritytestingrequiresgoingbeyondwhatisexpectedandthinkinglikeanattackerwhoistryingtobreaktheapplication.Creativethinkingcanhelptodeterminewhatunexpecteddatamaycauseanapplicationtofailinaninsecuremanner.Itcanalsohelpfindwhatassumptionsmadebywebdevelopersarenotalwaystrueandhowtheycanbesubverted.Oneofthereasonswhyautomatedtoolsareactuallybadatautomaticallytestingforvulnerabilitiesisthatthiscreativethinkingmustbedoneonacase-by-casebasisasmostwebapplicationsarebeingdevelopedinauniqueway(evenwhenusingcommonframeworks).

Oneofthefirstmajorinitiativesinanygoodsecurityprogramshouldbetorequireaccuratedocumentationoftheapplication.Thearchitecture,data-flowdiagrams,usecases,etc,shouldbewritteninformaldocumentsandmadeavailableforreview.Thetechnicalspecificationandapplicationdocumentsshouldincludeinformationthatlistsnotonlythedesiredusecases,butalsoanyspecificallydisallowedusecase.Finally,itisgoodtohaveatleastabasicsecurityinfrastructurethatallowsthemonitoringandtrendingofattacksagainstanorganization'sapplicationsandnetwork(e.g.,IDSsystems).

Whilewehavealreadystatedthatthereisnosilverbullettool,toolsdoplayacriticalroleintheoverallsecurityprogram.Thereisarangeofopensourceandcommercialtoolsthatcanautomatemanyroutinesecuritytasks.Thesetoolscansimplifyandspeedupthesecurityprocessbyassistingsecuritypersonnelintheirtasks.However,itisimportanttounderstandexactlywhatthesetoolscanandcannotdosothattheyarenotoversoldorusedincorrectly.

Itiscriticalnottoperformasuperficialsecurityreviewofanapplicationandconsideritcomplete.Thiswillinstillafalsesenseofconfidencethatcanbeasdangerousasnothavingdoneasecurityreviewinthefirstplace.Itisvitaltocarefullyreviewthefindingsandweedoutanyfalsepositivethatmayremaininthereport.Reportinganincorrectsecurityfindingcanoftenunderminethevalidmessageoftherestofasecurityreport.Careshouldbetakentoverifythateverypossiblesectionofapplicationlogichasbeentested,andthateveryusecasescenariowasexploredforpossiblevulnerabilities.

TestEarlyandTestOften

UnderstandtheScopeofSecurity

DeveloptheRightMindset

UnderstandtheSubject

UsetheRightTools

TheDevilisintheDetails

OwaspTestingGuidev4

17Introduction

Whileblackboxpenetrationtestresultscanbeimpressiveandusefultodemonstratehowvulnerabilitiesareexposedinaproductionenvironment,theyarenotthemosteffectiveorefficientwaytosecureanapplication.Itisdifficultfordynamictestingtotesttheentirecodebase,particularlyifmanynestedconditionalstatementsexist.Ifthesourcecodefortheapplicationisavailable,itshouldbegiventothesecuritystafftoassistthemwhileperformingtheirreview.Itispossibletodiscovervulnerabilitieswithintheapplicationsourcethatwouldbemissedduringablackboxengagement.

Animportantpartofagoodsecurityprogramistheabilitytodetermineifthingsaregettingbetter.Itisimportanttotracktheresultsoftestingengagements,anddevelopmetricsthatwillrevealtheapplicationsecuritytrendswithintheorganization.

Goodmetricswillshow:

Ifmoreeducationandtrainingarerequired;Ifthereisaparticularsecuritymechanismthatisnotclearlyunderstoodbythedevelopmentteam;Ifthetotalnumberofsecurityrelatedproblemsbeingfoundeachmonthisgoingdown.

Consistentmetricsthatcanbegeneratedinanautomatedwayfromavailablesourcecodewillalsohelptheorganizationinassessingtheeffectivenessofmechanismsintroducedtoreducesecuritybugsinsoftwaredevelopment.Metricsarenoteasilydeveloped,sousingstandardmetricslikethoseprovidedbytheOWASPMetricsprojectandotherorganizationsisagoodstartingpoint.

Toconcludethetestingprocess,itisimportanttoproduceaformalrecordofwhattestingactionsweretaken,bywhom,whentheywereperformed,anddetailsofthetestfindings.Itiswisetoagreeonanacceptableformatforthereportwhichisusefultoallconcernedparties,whichmayincludedevelopers,projectmanagement,businessowners,ITdepartment,audit,andcompliance.

Thereportshouldbecleartothebusinessownerinidentifyingwherematerialrisksexistandsufficienttogettheirbackingforsubsequentmitigationactions.Thereportshouldalsobecleartothedeveloperinpin-pointingtheexactfunctionthatisaffectedbythevulnerabilityandassociatedrecommendationsforresolvingissuesinalanguagethatthedeveloperwillunderstand.Thereportshouldalsoallowanothersecuritytestertoreproducetheresults.Writingthereportshouldnotbeoverlyburdensomeonthesecuritytesterthemselves.Securitytestersarenotgenerallyrenownedfortheircreativewritingskillsandagreeingonacomplexreportcanleadtoinstanceswheretestresultsdonotgetproperlydocumented.Usingasecuritytestreporttemplatecansavetimeandensurethatresultsaredocumentedaccuratelyandconsistently,andareinaformatthatissuitablefortheaudience.

Thissectionpresentsahigh-leveloverviewofvarioustestingtechniquesthatcanbeemployedwhenbuildingatestingprogram.ItdoesnotpresentspecificmethodologiesforthesetechniquesasthisinformationiscoveredinChapter3.Thissectionisincludedtoprovidecontextfortheframeworkpresentedinthenextchapterandtohighlighttheadvantagesanddisadvantagesofsomeofthetechniquesthatshouldbeconsidered.Inparticular,wewillcover:

ManualInspections&ReviewsThreatModelingCodeReviewPenetrationTesting

UseSourceCodeWhenAvailable

DevelopMetrics

DocumenttheTestResults

TestingTechniquesExplained

ManualInspections&Reviews

OwaspTestingGuidev4

18Introduction

Manualinspectionsarehumanreviewsthattypicallytestthesecurityimplicationsofpeople,policies,andprocesses.Manualinspectionscanalsoincludeinspectionoftechnologydecisionssuchasarchitecturaldesigns.Theyareusuallyconductedbyanalyzingdocumentationorperforminginterviewswiththedesignersorsystemowners.

Whiletheconceptofmanualinspectionsandhumanreviewsissimple,theycanbeamongthemostpowerfulandeffectivetechniquesavailable.Byaskingsomeonehowsomethingworksandwhyitwasimplementedinaspecificway,thetestercanquicklydetermineifanysecurityconcernsarelikelytobeevident.Manualinspectionsandreviewsareoneofthefewwaystotestthesoftwaredevelopmentlife-cycleprocessitselfandtoensurethatthereisanadequatepolicyorskillsetinplace.

Aswithmanythingsinlife,whenconductingmanualinspectionsandreviewsitisrecommendedthatatrust-but-verifymodelisadopted.Noteverythingthatthetesterisshownortoldwillbeaccurate.Manualreviewsareparticularlygoodfortestingwhetherpeopleunderstandthesecurityprocess,havebeenmadeawareofpolicy,andhavetheappropriateskillstodesignorimplementasecureapplication.

Otheractivities,includingmanuallyreviewingthedocumentation,securecodingpolicies,securityrequirements,andarchitecturaldesigns,shouldallbeaccomplishedusingmanualinspections.

RequiresnosupportingtechnologyCanbeappliedtoavarietyofsituationsFlexiblePromotesteamworkEarlyintheSDLC

CanbetimeconsumingSupportingmaterialnotalwaysavailableRequiressignificanthumanthoughtandskilltobeeffective

Threatmodelinghasbecomeapopulartechniquetohelpsystemdesignersthinkaboutthesecuritythreatsthattheirsystemsandapplicationsmightface.Therefore,threatmodelingcanbeseenasriskassessmentforapplications.Infact,itenablesthedesignertodevelopmitigationstrategiesforpotentialvulnerabilitiesandhelpsthemfocustheirinevitablylimitedresourcesandattentiononthepartsofthesystemthatmostrequireit.Itisrecommendedthatallapplicationshaveathreatmodeldevelopedanddocumented.ThreatmodelsshouldbecreatedasearlyaspossibleintheSDLC,andshouldberevisitedastheapplicationevolvesanddevelopmentprogresses.

Todevelopathreatmodel,werecommendtakingasimpleapproachthatfollowstheNIST800-30[11]standardforriskassessment.Thisapproachinvolves:

Decomposingtheapplication–useaprocessofmanualinspectiontounderstandhowtheapplicationworks,itsassets,functionality,andconnectivity.Definingandclassifyingtheassets–classifytheassetsintotangibleandintangibleassetsandrankthemaccordingtobusinessimportance.Exploringpotentialvulnerabilities-whethertechnical,operational,ormanagement.Exploringpotentialthreats–developarealisticviewofpotentialattackvectorsfromanattacker’sperspective,byusingthreatscenariosorattacktrees.

Overview

Advantages:

Disadvantages:

ThreatModeling

Overview

OwaspTestingGuidev4

19Introduction

Creatingmitigationstrategies–developmitigatingcontrolsforeachofthethreatsdeemedtoberealistic.

Theoutputfromathreatmodelitselfcanvarybutistypicallyacollectionoflistsanddiagrams.TheOWASPCodeReviewGuideoutlinesanApplicationThreatModelingmethodologythatcanbeusedasareferenceforthetestingapplicationsforpotentialsecurityflawsinthedesignoftheapplication.Thereisnorightorwrongwaytodevelopthreatmodelsandperforminformationriskassessmentsonapplications.[12].

Practicalattacker'sviewofthesystemFlexibleEarlyintheSDLC

RelativelynewtechniqueGoodthreatmodelsdon’tautomaticallymeangoodsoftware

Sourcecodereviewistheprocessofmanuallycheckingthesourcecodeofawebapplicationforsecurityissues.Manyserioussecurityvulnerabilitiescannotbedetectedwithanyotherformofanalysisortesting.Asthepopularsayinggoes“ifyouwanttoknowwhat’sreallygoingon,gostraighttothesource."Almostallsecurityexpertsagreethatthereisnosubstituteforactuallylookingatthecode.Alltheinformationforidentifyingsecurityproblemsisthereinthecodesomewhere.Unliketestingthirdpartyclosedsoftwaresuchasoperatingsystems,whentestingwebapplications(especiallyiftheyhavebeendevelopedin-house)thesourcecodeshouldbemadeavailablefortestingpurposes.

Manyunintentionalbutsignificantsecurityproblemsarealsoextremelydifficulttodiscoverwithotherformsofanalysisortesting,suchaspenetrationtesting,makingsourcecodeanalysisthetechniqueofchoicefortechnicaltesting.Withthesourcecode,atestercanaccuratelydeterminewhatishappening(orissupposedtobehappening)andremovetheguessworkofblackboxtesting.

Examplesofissuesthatareparticularlyconducivetobeingfoundthroughsourcecodereviewsincludeconcurrencyproblems,flawedbusinesslogic,accesscontrolproblems,andcryptographicweaknessesaswellasbackdoors,Trojans,Eastereggs,timebombs,logicbombs,andotherformsofmaliciouscode.Theseissuesoftenmanifestthemselvesasthemostharmfulvulnerabilitiesinwebsites.Sourcecodeanalysiscanalsobeextremelyefficienttofindimplementationissuessuchasplaceswhereinputvalidationwasnotperformedorwhenfailopencontrolproceduresmaybepresent.Butkeepinmindthatoperationalproceduresneedtobereviewedaswell,sincethesourcecodebeingdeployedmightnotbethesameastheonebeinganalyzedherein[13].

CompletenessandeffectivenessAccuracyFast(forcompetentreviewers)

RequireshighlyskilledsecuritydevelopersCanmississuesincompiledlibrariesCannotdetectrun-timeerrorseasilyThesourcecodeactuallydeployedmightdifferfromtheonebeinganalyzed

Formoreoncodereview,checkouttheOWASPcodereviewproject.

Advantages:

Disadvantages:

SourceCodeReview

Overview

Advantages:

Disadvantages:

OwaspTestingGuidev4

20Introduction

https://www.owasp.org/index.php/OWASP_Code_Review_Project

Penetrationtestinghasbeenacommontechniqueusedtotestnetworksecurityformanyyears.Itisalsocommonlyknownasblackboxtestingorethicalhacking.Penetrationtestingisessentiallythe“art”oftestingarunningapplicationremotelytofindsecurityvulnerabilities,withoutknowingtheinnerworkingsoftheapplicationitself.Typically,thepenetrationtestteamwouldhaveaccesstoanapplicationasiftheywereusers.Thetesteractslikeanattackerandattemptstofindandexploitvulnerabilities.Inmanycasesthetesterwillbegivenavalidaccountonthesystem.

Whilepenetrationtestinghasproventobeeffectiveinnetworksecurity,thetechniquedoesnotnaturallytranslatetoapplications.Whenpenetrationtestingisperformedonnetworksandoperatingsystems,themajorityoftheworkisinvolvedinfindingandthenexploitingknownvulnerabilitiesinspecifictechnologies.Aswebapplicationsarealmostexclusivelybespoke,penetrationtestinginthewebapplicationarenaismoreakintopureresearch.Penetrationtestingtoolshavebeendevelopedthatautomatetheprocess,butwiththenatureofwebapplicationstheireffectivenessisusuallypoor.

Manypeopletodayusewebapplicationpenetrationtestingastheirprimarysecuritytestingtechnique.Whilstitcertainlyhasitsplaceinatestingprogram,wedonotbelieveitshouldbeconsideredastheprimaryoronlytestingtechnique.GaryMcGrawin[14]summeduppenetrationtestingwellwhenhesaid,“Ifyoufailapenetrationtestyouknowyouhaveaverybadproblemindeed.Ifyoupassapenetrationtestyoudonotknowthatyoudon’thaveaverybadproblem”.However,focusedpenetrationtesting(i.e.,testingthatattemptstoexploitknownvulnerabilitiesdetectedinpreviousreviews)canbeusefulindetectingifsomespecificvulnerabilitiesareactuallyfixedinthesourcecodedeployedonthewebsite.

Canbefast(andthereforecheap)Requiresarelativelylowerskill-setthansourcecodereviewTeststhecodethatisactuallybeingexposed

ToolateintheSDLCFrontimpacttestingonly.

Withsomanytechniquesandapproachestotestingthesecurityofwebapplicationsitcanbedifficulttounderstandwhichtechniquestouseandwhentousethem.Experienceshowsthatthereisnorightorwronganswertothequestionofexactlywhattechniquesshouldbeusedtobuildatestingframework.Infactalltechniquesshouldprobablybeusedtotestalltheareasthatneedtobetested.

Althoughitisclearthatthereisnosingletechniquethatcanbeperformedtoeffectivelycoverallsecuritytestingandensurethatallissueshavebeenaddressed,manycompaniesadoptonlyoneapproach.Theapproachusedhashistoricallybeenpenetrationtesting.Penetrationtesting,whileuseful,cannoteffectivelyaddressmanyoftheissuesthatneedtobetested.Itissimply“toolittletoolate”inthesoftwaredevelopmentlifecycle(SDLC).

Thecorrectapproachisabalancedapproachthatincludesseveraltechniques,frommanualreviewstotechnicaltesting.AbalancedapproachshouldcovertestinginallphasesoftheSDLC.ThisapproachleveragesthemostappropriatetechniquesavailabledependingonthecurrentSDLCphase.

Ofcoursetherearetimesandcircumstanceswhereonlyonetechniqueispossible.Forexample,atestonawebapplicationthathasalreadybeencreated,butwherethetestingpartydoesnothaveaccesstothesourcecode.Inthis

PenetrationTesting

Overview

Advantages:

Disadvantages:

TheNeedforaBalancedApproach

OwaspTestingGuidev4

21Introduction

case,penetrationtestingisclearlybetterthannotestingatall.However,thetestingpartiesshouldbeencouragedtochallengeassumptions,suchasnoaccesstosourcecode,andtoexplorethepossibilityofmorecompletetesting.

Abalancedapproachvariesdependingonmanyfactors,suchasthematurityofthetestingprocessandcorporateculture.ItisrecommendedthatabalancedtestingframeworkshouldlooksomethingliketherepresentationsshowninFigure3andFigure4.Thefollowingfigureshowsatypicalproportionalrepresentationoverlaidontothesoftwaredevelopmentlifecycle.Inkeepingwithresearchandexperience,itisessentialthatcompaniesplaceahigheremphasisontheearlystagesofdevelopment.

Figure3:ProportionofTestEffortinSDLC

Thefollowingfigureshowsatypicalproportionalrepresentationoverlaidontotestingtechniques.

Figure4:ProportionofTestEffortAccordingtoTestTechnique

ANoteaboutWebApplicationScanners

OwaspTestingGuidev4

22Introduction

Manyorganizationshavestartedtouseautomatedwebapplicationscanners.Whiletheyundoubtedlyhaveaplaceinatestingprogram,somefundamentalissuesneedtobehighlightedaboutwhyitisbelievedthatautomatingblackboxtestingisnot(orwilleverbe)effective.However,highlightingtheseissuesshouldnotdiscouragetheuseofwebapplicationscanners.Rather,theaimistoensurethelimitationsareunderstoodandtestingframeworksareplannedappropriately.

Important:OWASPiscurrentlyworkingtodevelopawebapplicationscannerbenchmarkingplatform.Thefollowingexamplesshowwhyautomatedblackboxtestingisnoteffective.

Example1:MagicParametersImagineasimplewebapplicationthatacceptsaname-valuepairof“magic”andthenthevalue.Forsimplicity,theGETrequestmaybe:http://www.host/application?magic=valueTofurthersimplifytheexample,thevaluesinthiscasecanonlybeASCIIcharactersa–z(upperorlowercase)andintegers0–9.

Thedesignersofthisapplicationcreatedanadministrativebackdoorduringtesting,butobfuscatedittopreventthecasualobserverfromdiscoveringit.Bysubmittingthevaluesf8g7sfjdsurtsdieerwqredsgnfg8d(30characters),theuserwillthenbeloggedinandpresentedwithanadministrativescreenwithtotalcontroloftheapplication.TheHTTPrequestisnow:http://www.host/application?magic=sf8g7sfjdsurtsdieerwqredsgnfg8d

Giventhatalloftheotherparametersweresimpletwo-andthree-charactersfields,itisnotpossibletostartguessingcombinationsatapproximately28characters.Awebapplicationscannerwillneedtobruteforce(orguess)theentirekeyspaceof30characters.Thatisupto30^28permutations,ortrillionsofHTTPrequests.Thatisanelectroninadigitalhaystack.

ThecodeforthisexemplarMagicParametercheckmaylooklikethefollowing:

publicvoiddoPost(HttpServletRequestrequest,HttpServletResponseresponse){Stringmagic=“sf8g7sfjdsurtsdieerwqredsgnfg8d”;booleanadmin=magic.equals(request.getParameter(“magic”));if(admin)doAdmin(request,response);else….//normalprocessing}

Bylookinginthecode,thevulnerabilitypracticallyleapsoffthepageasapotentialproblem.Example2:BadCryptographyCryptographyiswidelyusedinwebapplications.ImaginethatadeveloperdecidedtowriteasimplecryptographyalgorithmtosignauserinfromsiteAtositeBautomatically.Inhis/herwisdom,thedeveloperdecidesthatifauserisloggedintositeA,thenhe/shewillgenerateakeyusinganMD5hashfunctionthatcomprises:Hash{username:date}WhenauserispassedtositeB,he/shewillsendthekeyonthequerystringtositeBinanHTTPre-direct.SiteBindependentlycomputesthehash,andcomparesittothehashpassedontherequest.Iftheymatch,siteBsignstheuserinastheusertheyclaimtobe.

Astheschemeisexplainedtheinadequaciescanbeworkedout.Anyonethatfiguresoutthescheme(oristoldhowitworks,ordownloadstheinformationfromBugtraq)canloginasanyuser.Manualinspection,suchasarevieworcodeinspection,wouldhaveuncoveredthissecurityissuequickly.Ablack-boxwebapplicationscannerwouldnothaveuncoveredthevulnerability.Itwouldhaveseena128-bithashthatchangedwitheachuser,andbythenatureofhashfunctions,didnotchangeinanypredictableway.

Manyorganizationshavestartedtousestaticsourcecodescanners.Whiletheyundoubtedlyhaveaplaceinacomprehensivetestingprogram,itisnecessarytohighlightsomefundamentalissuesaboutwhythisapproachisnoteffectivewhenusedalone.Staticsourcecodeanalysisalonecannotidentifyissuesduetoflawsinthedesign,sinceitcannotunderstandthecontextinwhichthecodeisconstructed.Sourcecodeanalysistoolsareusefulindetermining

ANoteaboutStaticSourceCodeReviewTools

OwaspTestingGuidev4

23Introduction

securityissuesduetocodingerrors,howeversignificantmanualeffortisrequiredtovalidatethefindings.

Tohaveasuccessfultestingprogram,onemustknowwhatthetestingobjectivesare.Theseobjectivesarespecifiedbythesecurityrequirements.Thissectiondiscussesindetailhowtodocumentrequirementsforsecuritytestingbyderivingthemfromapplicablestandardsandregulations,andfrompositiveandnegativeapplicationrequirements.ItalsodiscusseshowsecurityrequirementseffectivelydrivesecuritytestingduringtheSDLCandhowsecuritytestdatacanbeusedtoeffectivelymanagesoftwaresecurityrisks.

Oneoftheobjectivesofsecuritytestingistovalidatethatsecuritycontrolsoperateasexpected.Thisisdocumentedviasecurityrequirementsthatdescribethefunctionalityofthesecuritycontrol.Atahighlevel,thismeansprovingconfidentiality,integrity,andavailabilityofthedataaswellastheservice.Theotherobjectiveistovalidatethatsecuritycontrolsareimplementedwithfewornovulnerabilities.Thesearecommonvulnerabilities,suchastheOWASPTopTen,aswellasvulnerabilitiesthathavebeenpreviouslyidentifiedwithsecurityassessmentsduringtheSDLC,suchasthreatmodelling,sourcecodeanalysis,andpenetrationtest.

Thefirststepinthedocumentationofsecurityrequirementsistounderstandthebusinessrequirements.Abusinessrequirementdocumentcanprovideinitialhigh-levelinformationontheexpectedfunctionalityoftheapplication.Forexample,themainpurposeofanapplicationmaybetoprovidefinancialservicestocustomersortoallowgoodstobepurchasedfromanon-linecatalog.Asecuritysectionofthebusinessrequirementsshouldhighlighttheneedtoprotectthecustomerdataaswellastocomplywithapplicablesecuritydocumentationsuchasregulations,standards,andpolicies.

Ageneralchecklistoftheapplicableregulations,standards,andpoliciesisagoodpreliminarysecuritycomplianceanalysisforwebapplications.Forexample,complianceregulationscanbeidentifiedbycheckinginformationaboutthebusinesssectorandthecountryorstatewheretheapplicationwilloperate.Someofthesecomplianceguidelinesandregulationsmighttranslateintospecifictechnicalrequirementsforsecuritycontrols.Forexample,inthecaseoffinancialapplications,thecompliancewithFFIECguidelinesforauthentication[15]requiresthatfinancialinstitutionsimplementapplicationsthatmitigateweakauthenticationriskswithmulti-layeredsecuritycontrolandmulti-factorauthentication.

Applicableindustrystandardsforsecurityneedalsotobecapturedbythegeneralsecurityrequirementchecklist.Forexample,inthecaseofapplicationsthathandlecustomercreditcarddata,thecompliancewiththePCIDSS[16]standardforbidsthestorageofPINsandCVV2dataandrequiresthatthemerchantprotectmagneticstripdatainstorageandtransmissionwithencryptionandondisplaybymasking.SuchPCIDSSsecurityrequirementscouldbevalidatedviasourcecodeanalysis.

Anothersectionofthechecklistneedstoenforcegeneralrequirementsforcompliancewiththeorganization'sinformationsecuritystandardsandpolicies.Fromthefunctionalrequirementsperspective,requirementsforthesecuritycontrolneedtomaptoaspecificsectionoftheinformationsecuritystandards.Anexampleofsuchrequirementcanbe:"apasswordcomplexityofsixalphanumericcharactersmustbeenforcedbytheauthenticationcontrolsusedbytheapplication."Whensecurityrequirementsmaptocompliancerulesasecuritytestcanvalidatetheexposureofcompliancerisks.Ifviolationwithinformationsecuritystandardsandpoliciesarefound,thesewillresultinariskthatcanbedocumentedandthatthebusinesshastomanage.Sincethesesecuritycompliancerequirementsareenforceable,theyneedtobewelldocumentedandvalidatedwithsecuritytests.

Fromthefunctionalityperspective,thevalidationofsecurityrequirementsisthemainobjectiveofsecuritytesting.Fromtheriskmanagementperspective,thevalidationofsecurityrequirementsistheobjectiveofinformationsecurityassessments.Atahighlevel,themaingoalofinformationsecurityassessmentsistheidentificationofgapsinsecuritycontrols,suchas

DerivingSecurityTestRequirements

TestingObjectives

SecurityRequirementsDocumentation

SecurityRequirementsValidation

OwaspTestingGuidev4

24Introduction

https://www.owasp.org/index.php/OWASP_Top_Ten

lackofbasicauthentication,authorization,orencryptioncontrols.Moreindepth,thesecurityassessmentobjectiveisriskanalysis,suchastheidentificationofpotentialweaknessesinsecuritycontrolsthatensuretheconfidentiality,integrity,andavailabilityofthedata.Forexample,whentheapplicationdealswithpersonalidentifiableinformation(PII)andsensitivedata,thesecurityrequirementtobevalidatedisthecompliancewiththecompanyinformationsecuritypolicyrequiringencryptionofsuchdataintransitandinstorage.Assumingencryptionisusedtoprotectthedata,encryptionalgorithmsandkeylengthsneedtocomplywiththeorganizationencryptionstandards.Thesemightrequirethatonlycertainalgorithmsandkeylengthscouldbeused.Forexample,asecurityrequirementthatcanbesecuritytestedisverifyingthatonlyallowedciphersareused(e.g.,SHA-256,RSA,AES)withallowedminimumkeylengths(e.g.,morethan128bitforsymmetricandmorethan1024forasymmetricencryption).

Fromthesecurityassessmentperspective,securityrequirementscanbevalidatedatdifferentphasesoftheSDLCbyusingdifferentartifactsandtestingmethodologies.Forexample,threatmodelingfocusesonidentifyingsecurityflawsduringdesign,securecodeanalysisandreviewsfocusonidentifyingsecurityissuesinsourcecodeduringdevelopment,andpenetrationtestingfocusesonidentifyingvulnerabilitiesintheapplicationduringtestingorvalidation.

SecurityissuesthatareidentifiedearlyintheSDLCcanbedocumentedinatestplansotheycanbevalidatedlaterwithsecuritytests.Bycombiningtheresultsofdifferenttestingtechniques,itispossibletoderivebettersecuritytestcasesandincreasethelevelofassuranceofthesecurityrequirements.Forexample,distinguishingtruevulnerabilitiesfromtheun-exploitableonesispossiblewhentheresultsofpenetrationtestsandsourcecodeanalysisarecombined.ConsideringthesecuritytestforaSQLinjectionvulnerability,forexample,ablackboxtestmightfirstinvolveascanoftheapplicationtofingerprintthevulnerability.ThefirstevidenceofapotentialSQLinjectionvulnerabilitythatcanbevalidatedisthegenerationofaSQLexception.AfurthervalidationoftheSQLvulnerabilitymightinvolvemanuallyinjectingattackvectorstomodifythegrammaroftheSQLqueryforaninformationdisclosureexploit.Thismightinvolvealotoftrial-and-erroranalysisuntilthemaliciousqueryisexecuted.Assumingthetesterhasthesourcecode,shemightlearnfromthesourcecodeanalysisonhowtoconstructtheSQLattackvectorthatcanexploitthevulnerability(e.g.,executeamaliciousqueryreturningconfidentialdatatounauthorizeduser).

Athreatandcountermeasureclassification,whichtakesintoconsiderationrootcausesofvulnerabilities,isthecriticalfactorinverifyingthatsecuritycontrolsaredesigned,coded,andbuilttomitigatetheimpactoftheexposureofsuchvulnerabilities.Inthecaseofwebapplications,theexposureofsecuritycontrolstocommonvulnerabilities,suchastheOWASPTopTen,canbeagoodstartingpointtoderivegeneralsecurityrequirements.Morespecifically,thewebapplicationsecurityframe[17]providesaclassification(e.g.taxonomy)ofvulnerabilitiesthatcanbedocumentedindifferentguidelinesandstandardsandvalidatedwithsecuritytests.

Thefocusofathreatandcountermeasurecategorizationistodefinesecurityrequirementsintermsofthethreatsandtherootcauseofthevulnerability.AthreatcanbecategorizedbyusingSTRIDE[18]asSpoofing,Tampering,Repudiation,Informationdisclosure,Denialofservice,andElevationofprivilege.Therootcausecanbecategorizedassecurityflawindesign,asecuritybugincoding,oranissueduetoinsecureconfiguration.Forexample,therootcauseofweakauthenticationvulnerabilitymightbethelackofmutualauthenticationwhendatacrossesatrustboundarybetweentheclientandservertiersoftheapplication.Asecurityrequirementthatcapturesthethreatofnon-repudiationduringanarchitecturedesignreviewallowsforthedocumentationoftherequirementforthecountermeasure(e.g.,mutualauthentication)thatcanbevalidatedlateronwithsecuritytests.

Athreatandcountermeasurecategorizationforvulnerabilitiescanalsobeusedtodocumentsecurityrequirementsforsecurecodingsuchassecurecodingstandards.Anexampleofacommoncodingerrorinauthenticationcontrolsconsistsofapplyinganhashfunctiontoencryptapassword,withoutapplyingaseedtothevalue.Fromthesecurecodingperspective,thisisavulnerabilitythataffectstheencryptionusedforauthenticationwithavulnerabilityrootcauseinacodingerror.SincetherootcauseisinsecurecodingthesecurityrequirementcanbedocumentedinsecurecodingstandardsandvalidatedthroughsecurecodereviewsduringthedevelopmentphaseoftheSDLC.

ThreatsandCountermeasuresTaxonomies

SecurityTestingandRiskAnalysis

OwaspTestingGuidev4

25Introduction

Securityrequirementsneedtotakeintoconsiderationtheseverityofthevulnerabilitiestosupportariskmitigationstrategy.Assumingthattheorganizationmaintainsarepositoryofvulnerabilitiesfoundinapplications(i.e,avulnerabilityknowledgebase),thesecurityissuescanbereportedbytype,issue,mitigation,rootcause,andmappedtotheapplicationswheretheyarefound.SuchavulnerabilityknowledgebasecanalsobeusedtoestablishametricstoanalyzetheeffectivenessofthesecurityteststhroughouttheSDLC.

Forexample,consideraninputvalidationissue,suchasaSQLinjection,whichwasidentifiedviasourcecodeanalysisandreportedwithacodingerrorrootcauseandinputvalidationvulnerabilitytype.Theexposureofsuchvulnerabilitycanbeassessedviaapenetrationtest,byprobinginputfieldswithseveralSQLinjectionattackvectors.Thistestmightvalidatethatspecialcharactersarefilteredbeforehittingthedatabaseandmitigatethevulnerability.Bycombiningtheresultsofsourcecodeanalysisandpenetrationtestingitispossibletodeterminethelikelihoodandexposureofthevulnerabilityandcalculatetheriskratingofthevulnerability.Byreportingvulnerabilityriskratingsinthefindings(e.g.,testreport)itispossibletodecideonthemitigationstrategy.Forexample,highandmediumriskvulnerabilitiescanbeprioritizedforremediation,whilelowriskcanbefixedinfurtherreleases.

Byconsideringthethreatscenariosofexploitingcommonvulnerabilitiesitispossibletoidentifypotentialrisksthattheapplicationsecuritycontrolneedstobesecuritytestedfor.Forexample,theOWASPTopTenvulnerabilitiescanbemappedtoattackssuchasphishing,privacyviolations,identifytheft,systemcompromise,dataalterationordatadestruction,financialloss,andreputationloss.Suchissuesshouldbedocumentedaspartofthethreatscenarios.Bythinkingintermsofthreatsandvulnerabilities,itispossibletodeviseabatteryofteststhatsimulatesuchattackscenarios.Ideally,theorganizationvulnerabilityknowledgebasecanbeusedtoderivesecurityriskdriventestscasestovalidatethemostlikelyattackscenarios.Forexample,ifidentitytheftisconsideredhighrisk,negativetestscenariosshouldvalidatethemitigationofimpactsderivingfromtheexploitofvulnerabilitiesinauthentication,cryptographiccontrols,inputvalidation,andauthorizationcontrols.

Fromtheperspectiveoffunctionalsecurityrequirements,theapplicablestandards,policiesandregulationsdriveboththeneedforatypeofsecuritycontrolaswellasthecontrolfunctionality.Theserequirementsarealsoreferredtoas“positiverequirements”,sincetheystatetheexpectedfunctionalitythatcanbevalidatedthroughsecuritytests.Examplesofpositiverequirementsare:“theapplicationwilllockouttheuseraftersixfailedlogonattempts”or“passwordsneedtobeaminimumofsixalphanumericcharacters”.Thevalidationofpositiverequirementsconsistsofassertingtheexpectedfunctionalityandcanbetestedbyre-creatingthetestingconditionsandrunningthetestaccordingtopredefinedinputs.Theresultsarethenshownasasafailorpasscondition.

Inordertovalidatesecurityrequirementswithsecuritytests,securityrequirementsneedtobefunctiondrivenandtheyneedtohighlighttheexpectedfunctionality(thewhat)andimplicitlytheimplementation(thehow).Examplesofhigh-levelsecuritydesignrequirementsforauthenticationcanbe:

ProtectusercredentialsandsharedsecretsintransitandinstorageMaskanyconfidentialdataindisplay(e.g.,passwords,accounts)LocktheuseraccountafteracertainnumberoffailedloginattemptsDonotshowspecificvalidationerrorstotheuserasaresultofafailedlogonOnlyallowpasswordsthatarealphanumeric,includespecialcharactersandsixcharactersminimumlength,tolimittheattacksurfaceAllowforpasswordchangefunctionalityonlytoauthenticatedusersbyvalidatingtheoldpassword,thenewpassword,andtheuseranswertothechallengequestion,topreventbruteforcingofapasswordviapasswordchange.Thepasswordresetformshouldvalidatetheuser’susernameandtheuser’sregisteredemailbeforesendingthetemporarypasswordtotheuserviaemail.Thetemporarypasswordissuedshouldbeaonetimepassword.Alinktothepasswordresetwebpagewillbesenttotheuser.Thepasswordresetwebpageshouldvalidatetheusertemporarypassword,thenewpassword,aswellastheuseranswertothechallengequestion.

DerivingFunctionalandNonFunctionalTestRequirements

FunctionalSecurityRequirements

OwaspTestingGuidev4

26Introduction

Securitytestsneedalsotoberiskdriven,thatistheyneedtovalidatetheapplicationforunexpectedbehavior.Thesearealsocalled“negativerequirements”,sincetheyspecifywhattheapplicationshouldnotdo.

Examplesofnegativerequirementsare:

TheapplicationshouldnotallowforthedatatobealteredordestroyedTheapplicationshouldnotbecompromisedormisusedforunauthorizedfinancialtransactionsbyamalicioususer.

Negativerequirementsaremoredifficulttotest,becausethereisnoexpectedbehaviortolookfor.Thismightrequireathreatanalysttocomeupwithunforeseeableinputconditions,causes,andeffects.Thisiswheresecuritytestingneedstobedrivenbyriskanalysisandthreatmodeling.Thekeyistodocumentthethreatscenariosandthefunctionalityofthecountermeasureasafactortomitigateathreat.

Forexample,inthecaseofauthenticationcontrols,thefollowingsecurityrequirementscanbedocumentedfromthethreatsandcountermeasureperspective:

EncryptauthenticationdatainstorageandtransittomitigateriskofinformationdisclosureandauthenticationprotocolattacksEncryptpasswordsusingnonreversibleencryptionsuchasusingadigest(e.g.,HASH)andaseedtopreventdictionaryattacksLockoutaccountsafterreachingalogonfailurethresholdandenforcepasswordcomplexitytomitigateriskofbruteforcepasswordattacksDisplaygenericerrormessagesuponvalidationofcredentialstomitigateriskofaccountharvestingorenumerationMutuallyauthenticateclientandservertopreventnon-repudiationandManIntheMiddle(MiTM)attacks

Threatmodelingtoolssuchasthreattreesandattacklibrariescanbeusefultoderivethenegativetestscenarios.Athreattreewillassumearootattack(e.g.,attackermightbeabletoreadotherusers'messages)andidentifydifferentexploitsofsecuritycontrols(e.g.,datavalidationfailsbecauseofaSQLinjectionvulnerability)andnecessarycountermeasures(e.g.,implementdatavalidationandparametrizedqueries)thatcouldbevalidatedtobeeffectiveinmitigatingsuchattacks.

Aprerequisitetodescribingtheapplicationfunctionalityistounderstandwhattheapplicationissupposedtodoandhow.Thiscanbedonebydescribingusecases.Usecases,inthegraphicalformascommonlyusedinsoftwareengineering,showtheinteractionsofactorsandtheirrelations.Theyhelptoidentifytheactorsintheapplication,theirrelationships,theintendedsequenceofactionsforeachscenario,alternativeactions,specialrequirements,preconditionsandandpost-conditions.

Similartousecases,misuseandabusecases[19]describeunintendedandmalicioususescenariosoftheapplication.Thesemisusecasesprovideawaytodescribescenariosofhowanattackercouldmisuseandabusetheapplication.Bygoingthroughtheindividualstepsinausescenarioandthinkingabouthowitcanbemaliciouslyexploited,potentialflawsoraspectsoftheapplicationthatarenotwell-definedcanbediscovered.Thekeyistodescribeallpossibleor,atleast,themostcriticaluseandmisusescenarios.

Misusescenariosallowtheanalysisoftheapplicationfromtheattacker'spointofviewandcontributetoidentifyingpotentialvulnerabilitiesandthecountermeasuresthatneedtobeimplementedtomitigatetheimpactcausedbythepotentialexposuretosuchvulnerabilities.Givenalloftheuseandabusecases,itisimportanttoanalyzethemtodeterminewhichofthemarethemostcriticalonesandneedtobedocumentedinsecurityrequirements.Theidentificationofthemostcriticalmisuseandabusecasesdrivesthedocumentationofsecurityrequirementsandthenecessarycontrolswheresecurityrisksshouldbemitigated.

Toderivesecurityrequirementsfromuseandmisusecase[20]itisimportanttodefinethefunctionalscenariosandthenegativescenariosandputtheseingraphicalform.Inthecaseofderivationofsecurityrequirementsforauthentication,for

RiskDrivenSecurityRequirements

DerivingSecurityTestRequirementsThroughUseandMisuseCases

OwaspTestingGuidev4

27Introduction

example,thefollowingstep-by-stepmethodologycanbefollowed.

Step1:DescribetheFunctionalScenario:Userauthenticatesbysupplyingausernameandpassword.Theapplicationgrantsaccesstousersbaseduponauthenticationofusercredentialsbytheapplicationandprovidesspecificerrorstotheuserwhenvalidationfails.

Step2:DescribetheNegativeScenario:Attackerbreakstheauthenticationthroughabruteforceordictionaryattackofpasswordsandaccountharvestingvulnerabilitiesintheapplication.Thevalidationerrorsprovidespecificinformationtoanattackertoguesswhichaccountsareactuallyvalidregisteredaccounts(usernames).Thentheattackerwilltrytobruteforcethepasswordforsuchavalidaccount.Abruteforceattacktofourminimumlengthalldigitpasswordscansucceedwithalimitednumberofattempts(i.e.,10^4).

Step3:DescribeFunctionalandNegativeScenariosWithUseandMisuseCase:ThegraphicalexampleinFigurebelowdepictsthederivationofsecurityrequirementsviauseandmisusecases.Thefunctionalscenarioconsistsoftheuseractions(enteringausernameandpassword)andtheapplicationactions(authenticatingtheuserandprovidinganerrormessageifvalidationfails).Themisusecaseconsistsoftheattackeractions,i.e.tryingtobreakauthenticationbybruteforcingthepasswordviaadictionaryattackandbyguessingthevalidusernamesfromerrormessages.Bygraphicallyrepresentingthethreatstotheuseractions(misuses),itispossibletoderivethecountermeasuresastheapplicationactionsthatmitigatesuchthreats.

Step4:ElicitTheSecurityRequirements.Inthiscase,thefollowingsecurityrequirementsforauthenticationarederived:1. Passwordsneedtobealphanumeric,loweranduppercaseandminimumofsevencharacterlength2. Accountsneedtolockoutafterfiveunsuccessfulloginattempt3. Loginerrormessagesneedtobegeneric

Thesesecurityrequirementsneedtobedocumentedandtested.

SecurityTestsIntegratedinDevelopmentandTestingWorkflows

OwaspTestingGuidev4

28Introduction

SecuritytestingduringthedevelopmentphaseoftheSDLCrepresentsthefirstopportunityfordeveloperstoensurethattheindividualsoftwarecomponentstheyhavedevelopedaresecuritytestedbeforetheyareintegratedwithothercomponentsandbuiltintotheapplication.Softwarecomponentsmightconsistofsoftwareartifactssuchasfunctions,methods,andclasses,aswellasapplicationprogramminginterfaces,libraries,andexecutablefiles.Forsecuritytesting,developerscanrelyontheresultsofthesourcecodeanalysistoverifystaticallythatthedevelopedsourcecodedoesnotincludepotentialvulnerabilitiesandiscompliantwiththesecurecodingstandards.Securityunittestscanfurtherverifydynamically(i.e.,atruntime)thatthecomponentsfunctionasexpected.Beforeintegratingbothnewandexistingcodechangesintheapplicationbuild,theresultsofthestaticanddynamicanalysisshouldbereviewedandvalidated.

Thevalidationofsourcecodebeforeintegrationinapplicationbuildsisusuallytheresponsibilityoftheseniordeveloper.Suchseniordevelopersarealsothesubjectmatterexpertsinsoftwaresecurityandtheirroleistoleadthesecurecodereview.Theymustmakedecisionsonwhethertoacceptthecodetobereleasedintheapplicationbuildortorequirefurtherchangesandtesting.Thissecurecodereviewworkflowcanbeenforcedviaformalacceptanceaswellasacheckinaworkflowmanagementtool.Forexample,assumingthetypicaldefectmanagementworkflowusedforfunctionalbugs,securitybugsthathavebeenfixedbyadevelopercanbereportedonadefectorchangemanagementsystem.Thebuildmastercanlookatthetestresultsreportedbythedevelopersinthetoolandgrantapprovalsforcheckinginthecodechangesintotheapplicationbuild.

Aftercomponentsandcodechangesaretestedbydevelopersandcheckedintotheapplicationbuild,themostlikelynextstepinthesoftwaredevelopmentprocessworkflowistoperformtestsontheapplicationasawholeentity.Thisleveloftestingisusuallyreferredtoasintegratedtestandsystemleveltest.Whensecuritytestsarepartofthesetestingactivitiestheycanbeusedtovalidateboththesecurityfunctionalityoftheapplicationasawhole,aswellastheexposuretoapplicationlevelvulnerabilities.Thesesecuritytestsontheapplicationincludebothwhiteboxtesting,suchassourcecodeanalysis,andblackboxtesting,suchaspenetrationtesting.GrayboxtestingissimilartoBlackboxtesting.Inagrayboxtestingitisassumedthatthetesterhassomepartialknowledgeaboutthesessionmanagementoftheapplication,andthatshouldhelpinunderstandingwhetherthelogoutandtimeoutfunctionsareproperlysecured.

Thetargetforthesecuritytestsisthecompletesystemthatwillbepotentiallyattackedandincludesboththewholesourcecodeandtheexecutable.Onepeculiarityofsecuritytestingduringthisphaseisthatitispossibleforsecuritytesterstodeterminewhethervulnerabilitiescanbeexploitedandexposetheapplicationtorealrisks.Theseincludecommonwebapplicationvulnerabilities,aswellassecurityissuesthathavebeenidentifiedearlierintheSDLCwithotheractivitiessuchasthreatmodeling,sourcecodeanalysis,andsecurecodereviews.

Usuallytestingengineers,ratherthensoftwaredevelopers,performsecuritytestswhentheapplicationisinscopeforintegrationsystemtests.Suchtestingengineershavesecurityknowledgeofwebapplicationvulnerabilities,blackboxandwhiteboxsecuritytestingtechniques,andownthevalidationofsecurityrequirementsinthisphase.Inordertoperformsuchsecuritytests,itisaprerequisitethatsecuritytestcasesaredocumentedinthesecuritytestingguidelinesandprocedures.

Atestingengineerwhovalidatesthesecurityoftheapplicationintheintegratedsystemenvironmentmightreleasetheapplicationfortestingintheoperationalenvironment(e.g.,useracceptancetests).AtthisstageoftheSDLC(i.e.,validation),theapplicationfunctionaltestingisusuallyaresponsibilityofQAtesters,whilewhite-hathackersorsecurityconsultantsareusuallyresponsibleforsecuritytesting.Someorganizationsrelyontheirownspecializedethicalhackingteamtoconductsuchtestswhenathirdpartyassessmentisnotrequired(suchasforauditingpurposes).

Sincethesetestsarethelastresortforfixingvulnerabilitiesbeforetheapplicationisreleasedtoproduction,itisimportantthatsuchissuesareaddressedasrecommendedbythetestingteam.Therecommendationscanincludecode,design,orconfigurationchange.Atthislevel,securityauditorsandinformationsecurityofficersdiscussthereportedsecurityissuesandanalyzethepotentialrisksaccordingtoinformationriskmanagementprocedures.Suchproceduresmightrequirethedevelopmentteamtofixallhighriskvulnerabilitiesbeforetheapplicationcanbedeployed,unlesssuchrisksare

SecurityTestingintheDevelopmentWorkflow

SecurityTestingintheTestWorkflow

OwaspTestingGuidev4

29Introduction

acknowledgedandaccepted.

Fromthedeveloper’sperspective,themainobjectiveofsecuritytestsistovalidatethatcodeisbeingdevelopedincompliancewithsecurecodingstandardsrequirements.Developers'owncodingartifacts(suchasfunctions,methods,classes,APIs,andlibraries)needtobefunctionallyvalidatedbeforebeingintegratedintotheapplicationbuild.

Thesecurityrequirementsthatdevelopershavetofollowshouldbedocumentedinsecurecodingstandardsandvalidatedwithstaticanddynamicanalysis.Iftheunittestactivityfollowsasecurecodereview,unittestscanvalidatethatcodechangesrequiredbysecurecodereviewsareproperlyimplemented.Securecodereviewsandsourcecodeanalysisthroughsourcecodeanalysistoolshelpdevelopersinidentifyingsecurityissuesinsourcecodeasitisdeveloped.Byusingunittestsanddynamicanalysis(e.g.,debugging)developerscanvalidatethesecurityfunctionalityofcomponentsaswellasverifythatthecountermeasuresbeingdevelopedmitigateanysecurityriskspreviouslyidentifiedthroughthreatmodelingandsourcecodeanalysis.

Agoodpracticefordevelopersistobuildsecuritytestcasesasagenericsecuritytestsuitethatispartoftheexistingunittestingframework.Agenericsecuritytestsuitecouldbederivedfrompreviouslydefineduseandmisusecasestosecuritytestfunctions,methodsandclasses.Agenericsecuritytestsuitemightincludesecuritytestcasestovalidatebothpositiveandnegativerequirementsforsecuritycontrolssuchas:

Identity,Authentication&AccessControlInputValidation&EncodingEncryptionUserandSessionManagementErrorandExceptionHandlingAuditingandLogging

DevelopersempoweredwithasourcecodeanalysistoolintegratedintotheirIDE,securecodingstandards,andasecurityunittestingframeworkcanassessandverifythesecurityofthesoftwarecomponentsbeingdeveloped.Securitytestcasescanberuntoidentifypotentialsecurityissuesthathaverootcausesinsourcecode:besidesinputandoutputvalidationofparametersenteringandexitingthecomponents,theseissuesincludeauthenticationandauthorizationchecksdonebythecomponent,protectionofthedatawithinthecomponent,secureexceptionanderrorhandling,andsecureauditingandlogging.UnittestframeworkssuchasJunit,Nunit,andCUnitcanbeadaptedtoverifysecuritytestrequirements.Inthecaseofsecurityfunctionaltests,unitleveltestscantestthefunctionalityofsecuritycontrolsatthesoftwarecomponentlevel,suchasfunctions,methods,orclasses.Forexample,atestcasecouldvalidateinputandoutputvalidation(e.g.,variablesanitation)andboundarychecksforvariablesbyassertingtheexpectedfunctionalityofthecomponent.

Thethreatscenariosidentifiedwithuseandmisusecasescanbeusedtodocumenttheproceduresfortestingsoftwarecomponents.Inthecaseofauthenticationcomponents,forexample,securityunittestscanassertthefunctionalityofsettinganaccountlockoutaswellasthefactthatuserinputparameterscannotbeabusedtobypasstheaccountlockout(e.g.,bysettingtheaccountlockoutcountertoanegativenumber).

Atthecomponentlevel,securityunittestscanvalidatepositiveassertionsaswellasnegativeassertions,suchaserrorsandexceptionhandling.Exceptionsshouldbecaughtwithoutleavingthesysteminaninsecurestate,suchaspotentialdenialofservicecausedbyresourcesnotbeingde-allocated(e.g.,connectionhandlesnotclosedwithinafinalstatementblock),aswellaspotentialelevationofprivileges(e.g.,higherprivilegesacquiredbeforetheexceptionisthrownandnotre-settothepreviouslevelbeforeexitingthefunction).Secureerrorhandlingcanvalidatepotentialinformationdisclosureviainformativeerrormessagesandstacktraces.

Unitlevelsecuritytestcasescanbedevelopedbyasecurityengineerwhoisthesubjectmatterexpertinsoftwaresecurityandisalsoresponsibleforvalidatingthatthesecurityissuesinthesourcecodehavebeenfixedandcanbecheckedinto

Developers'SecurityTests

SecurityTestingintheCodingPhase:UnitTests

OwaspTestingGuidev4

30Introduction

theintegratedsystembuild.Typically,themanageroftheapplicationbuildsalsomakessurethatthird-partylibrariesandexecutablefilesaresecurityassessedforpotentialvulnerabilitiesbeforebeingintegratedintheapplicationbuild.

Threatscenariosforcommonvulnerabilitiesthathaverootcausesininsecurecodingcanalsobedocumentedinthedeveloper’ssecuritytestingguide.Whenafixisimplementedforacodingdefectidentifiedwithsourcecodeanalysis,forexample,securitytestcasescanverifythattheimplementationofthecodechangefollowsthesecurecodingrequirementsdocumentedinthesecurecodingstandards.

Sourcecodeanalysisandunittestscanvalidatethatthecodechangemitigatesthevulnerabilityexposedbythepreviouslyidentifiedcodingdefect.Theresultsofautomatedsecurecodeanalysiscanalsobeusedasautomaticcheck-ingatesforversioncontrol,forexamplesoftwareartifactscannotbecheckedintothebuildwithhighormediumseveritycodingissues.

Themainobjectiveofintegratedsystemtestsistovalidatethe“defenseindepth”concept,thatis,thattheimplementationofsecuritycontrolsprovidessecurityatdifferentlayers.Forexample,thelackofinputvalidationwhencallingacomponentintegratedwiththeapplicationisoftenafactorthatcanbetestedwithintegrationtesting.

Theintegrationsystemtestenvironmentisalsothefirstenvironmentwheretesterscansimulaterealattackscenariosascanbepotentiallyexecutedbyamaliciousexternalorinternaluseroftheapplication.Securitytestingatthislevelcanvalidatewhethervulnerabilitiesarerealandcanbeexploitedbyattackers.Forexample,apotentialvulnerabilityfoundinsourcecodecanberatedashighriskbecauseoftheexposuretopotentialmalicioususers,aswellasbecauseofthepotentialimpact(e.g.,accesstoconfidentialinformation).

Realattackscenarioscanbetestedwithbothmanualtestingtechniquesandpenetrationtestingtools.Securitytestsofthistypearealsoreferredtoasethicalhackingtests.Fromthesecuritytestingperspective,theseareriskdriventestsandhavetheobjectiveoftestingtheapplicationintheoperationalenvironment.Thetargetistheapplicationbuildthatisrepresentativeoftheversionoftheapplicationbeingdeployedintoproduction.

Includingsecuritytestingintheintegrationandvalidationphaseiscriticaltoidentifyingvulnerabilitiesduetointegrationofcomponentsaswellasvalidatingtheexposureofsuchvulnerabilities.Applicationsecuritytestingrequiresaspecializedsetofskills,includingbothsoftwareandsecurityknowledge,thatarenottypicalofsecurityengineers.Asaresultorganizationsareoftenrequiredtosecurity-traintheirsoftwaredevelopersonethicalhackingtechniques,securityassessmentproceduresandtools.Arealisticscenarioistodevelopsuchresourcesin-houseanddocumenttheminsecuritytestingguidesandproceduresthattakeintoaccountthedeveloper’ssecuritytestingknowledge.Asocalled“securitytestcasescheatlistorcheck-list”,forexample,canprovidesimpletestcasesandattackvectorsthatcanbeusedbytesterstovalidateexposuretocommonvulnerabilitiessuchasspoofing,informationdisclosures,bufferoverflows,formatstrings,SQLinjectionandXSSinjection,XML,SOAP,canonicalizationissues,denialofserviceandmanagedcodeandActiveXcontrols(e.g.,.NET).Afirstbatteryofthesetestscanbeperformedmanuallywithaverybasicknowledgeofsoftwaresecurity.

Thefirstobjectiveofsecuritytestsmightbethevalidationofasetofminimumsecurityrequirements.Thesesecuritytestcasesmightconsistofmanuallyforcingtheapplicationintoerrorandexceptionalstatesandgatheringknowledgefromtheapplicationbehavior.Forexample,SQLinjectionvulnerabilitiescanbetestedmanuallybyinjectingattackvectorsthroughuserinputandbycheckingifSQLexceptionsarethrownbacktheuser.TheevidenceofaSQLexceptionerrormightbeamanifestationofavulnerabilitythatcanbeexploited.

Amorein-depthsecuritytestmightrequirethetester’sknowledgeofspecializedtestingtechniquesandtools.Besidessourcecodeanalysisandpenetrationtesting,thesetechniquesinclude,forexample,sourcecodeandbinaryfaultinjection,faultpropagationanalysisandcodecoverage,fuzztesting,andreverseengineering.Thesecuritytestingguideshouldprovideproceduresandrecommendtoolsthatcanbeusedbysecuritytesterstoperformsuchin-depthsecurityassessments.

FunctionalTesters'SecurityTests

SecurityTestingDuringtheIntegrationandValidationPhase:IntegratedSystemTestsandOperationTests

OwaspTestingGuidev4

31Introduction

Thenextlevelofsecuritytestingafterintegrationsystemtestsistoperformsecuritytestsintheuseracceptanceenvironment.Thereareuniqueadvantagestoperformingsecuritytestsintheoperationalenvironment.Theuseracceptancetestsenvironment(UAT)istheonethatismostrepresentativeofthereleaseconfiguration,withtheexceptionofthedata(e.g.,testdataisusedinplaceofrealdata).AcharacteristicofsecuritytestinginUATistestingforsecurityconfigurationissues.Insomecasesthesevulnerabilitiesmightrepresenthighrisks.Forexample,theserverthathoststhewebapplicationmightnotbeconfiguredwithminimumprivileges,validSSLcertificateandsecureconfiguration,essentialservicesdisabledandwebrootdirectorynotcleanedfromtestandadministrationwebpages.

Definingthegoalsforthesecuritytestingmetricsandmeasurementsisaprerequisiteforusingsecuritytestingdataforriskanalysisandmanagementprocesses.Forexample,ameasurementsuchasthetotalnumberofvulnerabilitiesfoundwithsecuritytestsmightquantifythesecuritypostureoftheapplication.Thesemeasurementsalsohelptoidentifysecurityobjectivesforsoftwaresecuritytesting.Forexample,reducingthenumberofvulnerabilitiestoanacceptablenumber(minimum)beforetheapplicationisdeployedintoproduction.

Anothermanageablegoalcouldbetocomparetheapplicationsecuritypostureagainstabaselinetoassessimprovementsinapplicationsecurityprocesses.Forexample,thesecuritymetricsbaselinemightconsistofanapplicationthatwastestedonlywithpenetrationtests.Thesecuritydataobtainedfromanapplicationthatwasalsosecuritytestedduringcodingshouldshowanimprovement(e.g.,fewernumberofvulnerabilities)whencomparedwiththebaseline.

Intraditionalsoftwaretesting,thenumberofsoftwaredefects,suchasthebugsfoundinanapplication,couldprovideameasureofsoftwarequality.Similarly,securitytestingcanprovideameasureofsoftwaresecurity.Fromthedefectmanagementandreportingperspective,softwarequalityandsecuritytestingcanusesimilarcategorizationsforrootcausesanddefectremediationefforts.Fromtherootcauseperspective,asecuritydefectcanbeduetoanerrorindesign(e.g.,securityflaws)orduetoanerrorincoding(e.g.,securitybug).Fromtheperspectiveoftheeffortrequiredtofixadefect,bothsecurityandqualitydefectscanbemeasuredintermsofdeveloperhourstoimplementthefix,thetoolsandresourcesrequiredtofix,andthecosttoimplementthefix.

Acharacteristicofsecuritytestdata,comparedtoqualitydata,isthecategorizationintermsofthethreat,theexposureofthevulnerability,andthepotentialimpactposedbythevulnerabilitytodeterminetherisk.Testingapplicationsforsecurityconsistsofmanagingtechnicalriskstomakesurethattheapplicationcountermeasuresmeetacceptablelevels.Forthisreason,securitytestingdataneedstosupportthesecurityriskstrategyatcriticalcheckpointsduringtheSDLC.Forexample,vulnerabilitiesfoundinsourcecodewithsourcecodeanalysisrepresentaninitialmeasureofrisk.Ameasureofrisk(e.g.,high,medium,low)forthevulnerabilitycanbecalculatedbydeterminingtheexposureandlikelihoodfactorsandbyvalidatingthevulnerabilitywithpenetrationtests.Theriskmetricsassociatedtovulnerabilitiesfoundwithsecuritytestsempowerbusinessmanagementtomakeriskmanagementdecisions,suchastodecidewhetherriskscanbeaccepted,mitigated,ortransferredatdifferentlevelswithintheorganization(e.g.,businessaswellastechnicalrisks).

Whenevaluatingthesecuritypostureofanapplicationitisimportanttotakeintoconsiderationcertainfactors,suchasthesizeoftheapplicationbeingdeveloped.Applicationsizehasbeenstatisticallyproventoberelatedtothenumberofissuesfoundintheapplicationduringtesting.Onemeasureofapplicationsizeisthenumberoflinesofcode(LOC)oftheapplication.Typically,softwarequalitydefectsrangefromabout7to10defectsperthousandlinesofnewandchangedcode[21].Sincetestingcanreducetheoverallnumberbyabout25%withonetestalone,itislogicalforlargersizeapplicationstobetestedmoreoftenthansmallersizeapplications.

WhensecuritytestingisdoneinseveralphasesoftheSDLC,thetestdatacanprovethecapabilityofthesecuritytestsindetectingvulnerabilitiesassoonastheyareintroduced.ThetestdatacanalsoprovetheeffectivenessofremovingthevulnerabilitiesbyimplementingcountermeasuresatdifferentcheckpointsoftheSDLC.Ameasurementofthistypeisalsodefinedas“containmentmetrics”andprovidesameasureoftheabilityofasecurityassessmentperformedateachphaseofthedevelopmentprocesstomaintainsecuritywithineachphase.Thesecontainmentmetricsarealsoacriticalfactorinloweringthecostoffixingthevulnerabilities.ItislessexpensivetodealwithvulnerabilitiesinthesamephaseoftheSDLC

SecurityTestDataAnalysisandReporting

GoalsforSecurityTestMetricsandMeasurements

OwaspTestingGuidev4

32Introduction

thattheyarefound,ratherthenfixingthemlaterinanotherphase.

Securitytestmetricscansupportsecurityrisk,cost,anddefectmanagementanalysiswhentheyareassociatedwithtangibleandtimedgoalssuchas:

Reducingtheoverallnumberofvulnerabilitiesby30%Fixingsecurityissuesbyacertaindeadline(e.g.,beforebetarelease)

Securitytestdatacanbeabsolute,suchasthenumberofvulnerabilitiesdetectedduringmanualcodereview,aswellascomparative,suchasthenumberofvulnerabilitiesdetectedincodereviewscomparedtopenetrationtests.Toanswerquestionsaboutthequalityofthesecurityprocess,itisimportanttodetermineabaselineforwhatcouldbeconsideredacceptableandgood.

Securitytestdatacanalsosupportspecificobjectivesofthesecurityanalysis.Theseobjectscouldbecompliancewithsecurityregulationsandinformationsecuritystandards,managementofsecurityprocesses,theidentificationofsecurityrootcausesandprocessimprovements,andsecuritycostbenefitanalysis.

Whensecuritytestdataisreportedithastoprovidemetricstosupporttheanalysis.Thescopeoftheanalysisistheinterpretationoftestdatatofindcluesaboutthesecurityofthesoftwarebeingproducedaswelltheeffectivenessoftheprocess.

Someexamplesofcluessupportedbysecuritytestdatacanbe:

Arevulnerabilitiesreducedtoanacceptablelevelforrelease?Howdoesthesecurityqualityofthisproductcomparewithsimilarsoftwareproducts?Areallsecuritytestrequirementsbeingmet?Whatarethemajorrootcausesofsecurityissues?Hownumerousaresecurityflawscomparedtosecuritybugs?Whichsecurityactivityismosteffectiveinfindingvulnerabilities?Whichteamismoreproductiveinfixingsecuritydefectsandvulnerabilities?Whichpercentageofoverallvulnerabilitiesarehighrisk?Whichtoolsaremosteffectiveindetectingsecurityvulnerabilities?Whichkindofsecuritytestsaremosteffectiveinfindingvulnerabilities(e.g.,whiteboxvs.blackbox)tests?Howmanysecurityissuesarefoundduringsecurecodereviews?Howmanysecurityissuesarefoundduringsecuredesignreviews?

Inordertomakeasoundjudgmentusingthetestingdata,itisimportanttohaveagoodunderstandingofthetestingprocessaswellasthetestingtools.Atooltaxonomyshouldbeadoptedtodecidewhichsecuritytoolstouse.Securitytoolscanbequalifiedasbeinggoodatfindingcommonknownvulnerabilitiestargetingdifferentartifacts.

Theissueisthattheunknownsecurityissuesarenottested.Thefactthatasecuritytestisclearofissuesdoesnotmeanthatthesoftwareorapplicationisgood.Somestudies[22]havedemonstratedthat,atbest,toolscanonlyfind45%ofoverallvulnerabilities.

Eventhemostsophisticatedautomationtoolsarenotamatchforanexperiencedsecuritytester.Justrelyingonsuccessfultestresultsfromautomationtoolswillgivesecuritypractitionersafalsesenseofsecurity.Typically,themoreexperiencedthesecuritytestersarewiththesecuritytestingmethodologyandtestingtools,thebettertheresultsofthesecuritytestandanalysiswillbe.Itisimportantthatmanagersmakinganinvestmentinsecuritytestingtoolsalsoconsideraninvestmentinhiringskilledhumanresourcesaswellassecuritytesttraining.

Thesecuritypostureofanapplicationcanbecharacterizedfromtheperspectiveoftheeffect,suchasnumberofvulnerabilitiesandtheriskratingofthevulnerabilities,aswellasfromtheperspectiveofthecauseororigin,suchascodingerrors,architecturalflaws,andconfigurationissues.

ReportingRequirements

OwaspTestingGuidev4

33Introduction

Vulnerabilitiescanbeclassifiedaccordingtodifferentcriteria.ThemostcommonlyusedvulnerabilityseveritymetricistheForumofIncidentResponseandSecurityTeams(FIRST)CommonVulnerabilityScoringSystem(CVSS),whichiscurrentlyinreleaseversion2withversion3dueforreleaseshortly.

Whenreportingsecuritytestdatathebestpracticeistoincludethefollowinginformation:

ThecategorizationofeachvulnerabilitybytypeThesecuritythreatthattheissueisexposedtoTherootcauseofsecurityissues(e.g.,securitybugs,securityflaw)ThetestingtechniqueusedtofindtheissueTheremediationofthevulnerability(e.g.,thecountermeasure)Theseverityratingofthevulnerability(High,Medium,Lowand/orCVSSscore)

Bydescribingwhatthesecuritythreatis,itwillbepossibletounderstandifandwhy