Thinking Ahead to System Verification and System Validation · verified and validated. At this stage of the system lifecycle, the concepts of system verification and system validation

Copyright©2016byRequirementsExperts. 1

Thinking Ahead to System Verification and System Validation LouisS.WheatcraftRequirementExperts

(281)[email protected]

Note:[Feb2016]Thisisanupdatetothispaperwhichwasoriginallypublishedin2012.Theupdatesarearesultofamaturingoftheauthor’sthoughtsonthesubjectandaclearerunderstandingofwhatismeantwhenusingtheterms“verification”and“validation”.ThemajorchangeistheadditionofamoredetaileddefinitionofthetermsandtheadditionofFigures1&2.Mostoftherestofthechangesweretomaketherestofthetextconsistentwiththeproposeddefinitionstomakeclearwhatthecontextiswhenusingtheseterms.

AbstractSecondonlytore-work,aproject’slargestcostsareinvolvedinsystemverificationandsystemvalidation.Failingtoadequatelyplanforsystemverificationandsystemvalidationfromthebeginningoftheprojectplacestheprojectathighriskforcostoverrunsandscheduleslips.Tomitigatetheserisksprojectteamsmustoutlinetheirconceptsforsystemverificationandsystemvalidationduringthescopedefinitionactivities;addressingthefollowingquestions:

• “Whowillverifythesystemmeetstherequirementsthatdrovethedesign,andwhereandwhenwilltheseactivitiesbedone?”

• “Whowillvalidatethatthedesigned,builtorcoded,systemwillmeetitsintendedpurposeinitsoperationalenvironment,andwhereandwhenwilltheseactiviiesbedone?”

• “Whatfacilities,supportequipment,simulators,emulators,ormodelsarerequiredforsystemverificationandsystemvalidation?”

• “Whataretheinterfacesinvolvedduringsystemverificationandsystemvalidation?”Whenyoustartwritingyourrequirements,addressingsystemverificationastherequirementsarewritteninsuresabettersetofrequirementsandcanreducethecostofsystemverificationandsystemvalidation.Aseachrequirementiswritten,thefollowingquestionsneedtobeaddressed:

• “Isthisrequirementverifiable,.i.e.,istherequirementwrittensuchthatitsintentisclearandasystemverificationactivitycanbeperformedthatwillprovethesystemmeetstherequirement?”

• “Whatwillconstituteproofthatthesystemhasbeenverifiedtomeettherequirement?”

• “Whatprimarymethodwillbeusedtoverifythesystemmeetsthisrequirement?”

Failingtoaddressthesequestionsearlycanresultinunpleasantsurprisesduringyoursystemverificationandsystemvalidationactivities,puttingtheprojectatriskofrework,scheduleslips,andbudgetoverruns.Theemphasisofthispaperistopresentbestpractices,tools,andprovenmethodsprojectteamscanusetoplanforsystemverificationandsystemvalidationfromthebeginningoftheirprojectandthusavoidtherisksofnotdoingso.


Verification and Validation

AquestionIhearoftenis“Whatisdifferencebetweenverificationandvalidation?”

Ingeneral,verificationreferstothebasics(structure)oftheitembeingverified,makingsureitmeetsrequirementsthatdrivethecreationoftheitem,whetheritberulesonwritingwell-formedrequirements,standardsandbestpractices(externalandinternal)onthedesign,orrequirementsonthesystem.Thenvalidationgoesbeyondthebasics(structure)tohowwelltheitem(requirements,design,system)communicatesoraddressesstakeholderneedsandexpectations.

Myuseofthewordsysteminthispaperreferstothesystemofinterest,nomatterwhichlevelthesystemofinterestmayexistwithinthearchitecture(system,subsystem,assembly,component).Requirementsaredevelopedduringtheconceptphaseandrepresentaconceptualviewofthesystemofinterest.Bydesign,Imeanthetransformationoftheconceptualviewofthesystemascommunicatedbythebaselinerequirementsetintoaphysicalrealizationofthesystem.Thisprocessevolvesfromapreliminarydesign,toafinaldesign,andthentodevelopment,whichtransformsthedesignintothephysicalsystem(hardware)orcode(software).

Figure1.Verificationandvalidationaretheprocessesofconfirmingthatartifactsgeneratedduringthetransformationprocessesareacceptable.

Whilethetermsverificationandvalidationarewidelyused,Ifindthatthetruemeaningoftheconceptsthesetermsrepresentareoftenmisunderstoodandthewordsoftenusedinterchangeablywithoutmakingclearthecontextinwhichtheyareusedresultinginambiguity.Withasingleorganization,whenengineersareaskedtodefinethetermsyouwillgetdifferent,oftenconflicting,definitions.Toavoidthisambiguity,eachtermneedstobeprecededbya

Stakeholderneedsandexpectations

TransformationRequirementDevelopment

TransformationDesign Design

OrganizationalRequirement

WritingGuidelines

TransformationBuild,Code,Buy,orReuse

System

VerificationVerificationVerification

Validation

Validation

Validation“Arewebuildingtherightthing?”

“Dowehavetherightdesign?”

OrganizationalDesign

GuidelinesVerification

“Didwebuildtherightthing?”

“Didwebuilditright?”

“Didwedesignitright?”

“Aretherequirementswrittencorrectly?”

“Didwedesignitcorrectly?”

OrganizationalCode,BuildGuidelines“Didwebuildit

correctly?”

Verification

Requirements


modifier(i.e.,thesubject)whichclearlydenotesthepropercontextinwhichthetermisbeingused,specificallyrequirementverificationorrequirementvalidation;designverificationordesignvalidation;systemverificationorsystemvalidationasshowninFigure1.

Arequirementistheresultofaformaltransformationofoneormorestakeholderneedsandexpectations in to the requirement statement. Correspondingly, design is a result of formaltransformation of requirements in to an agree-to design, and a system is a formaltransformationofadesignintothatsystemasshowninFigure1.

AsillustratedinFigure1,theprocessofcreatingarequirementinvolves:• analysingoneormoreselectedneedsandexpectationstoobtainthenecessaryelements

tobeincludedintherequirement;• selectingaformatfortherequirementstatement;• and identifying the characteristics of the desired result against the organizational

guidelinesandrulesbywhichtherequirementstatementistobewritten.

Inthiscontext,RequirementVerificationconfirms,byinspection,thattherequirementcontainsthe necessary elements and possesses the characteristics of awell formed requirement andconforms to the rules set forth in the organization’s requirement development guidelines.RequirementValidation confirms, by inspection and analysis, that the resulting requirementmeets the intent of the stakeholder need from which the requirement is derived. Thus arequirement statement (and sets of requirements) is/are confirmed by both verification andvalidation.

Basedonthisdiscussion,tohelpremovetheambiguityintheuseoftheterms“verification”and“validation”,thefollowingdefinitionsforrequirementverificationandrequirementvalidationareproposedintermsofthecontextofrequirements.

• RequirementVerification:theprocessofensuringtherequirementmeetstherulesandcharacteristicsdefinedforwritingagoodrequirement.Thefocusisonthewordingandstructure of the requirement. “Is the requirement worded or structured correctly inaccordancewiththeorganization’sstandards,processes,andchecklists?”

• RequirementValidation: confirmation the requirement is an agreed-to transformationthat clearly communicates the stakeholder needs and expectations in a languageunderstood by the developers. The focus is on the message the requirement iscommunicating. “Does the requirement clearly and correctly communicate thestakeholder expectations and needs?” “Are we doing the right things?” or “Are webuildingtherightthing?”

RequirementverificationandrequirementvalidationactivitiesshouldbedonecontinuouslyasyoudeveloptherequirementsaswellasdoneaspartofbaselineoftherequirementsetactivitiesperformedduringtheSystemRequirementsReview(SRR)orsimilartypeofgatereview.


Note: Most organizations do not make a distinction between requirement verification vsrequirementvalidation.Rathertheyuseonlythephrase“requirementvalidation”tomeanboth.Using the phrase “requirement verification” often confuses the conversation in that manyinterpret“requirementverification”tohavethesamemeaningas“systemverification”.

Figure1alsocarriestheseconceptsforwardtodesignandrealizationofthesystemunderdevelopment.

Onceyouhaveasetofrequirementsbaselined,therequirementsaretransformedintoadesignofthesystem.Mostorganizationshaveasetofguidelinesor“goldenrules”thatguidethedesignprocess.Theseguidelinesrepresentbestpracticesandlessonslearnedthedesignteamisexpectedtofollow.Aspartofthedesignprocess,thedesignteammaydevelopprototypesorengineeringunits.Theywillusethesetorunteststofinetunetheirdesign.

Afterthedesigniscompleteforthesystem,thereisusuallyagatereviewwherethedesignisbothverifiedandvalidated(e.g.SystemDesignReview(SDR),PreliminaryDesignReview(PDR),CriticalDesignReview(CDR).Inthiscontext,designverificationhastwoaspects:1)Doesthedesignclearlyrepresenttherequirementsthatdrovethedesign?and2)Didthedesignteamfollowtheorganization’sguidelinesfordesign?Alsoaspartofthegatereviewdesignvalidationisaddressedtodeterminewhetherornottheresultingdesignforthesystem,whenimplemented,willresultintheintendedpurposebeingmetintheoperationalenvironmentandthestakeholder’sexpectationsandneedsbeingmet.

Frequently,duringthesegatereviews,thedesignteam“pushesback”onsomeoftherequirementsthatproveddifficulttomeetorweredeemednotfeasible(cost,schedule,technology),andproposeschangestotherequirements.Whenthishappens,notonlydotherequirementsneededtobechanged,butalsothestakeholderexpectationsandneedsfromwhichtherequirementwasderivedmayneedtobeexamined,whichcouldresultinascopechange.

Designverificationanddesignvalidationactivitiesshouldbedoneaspartofacontinuousprocessduringthedesignphaseaswellasduringthethebase-liningofthedesigninthegatereview(s).

Basedonthisdiscussion,tohelpremovetheambiguityintheuseoftheterms“verification”and“validation”,thefollowingdefinitionsfordesignverificationanddesignvalidationareproposed.

• DesignVerification:theprocessofensuringthedesignmeetstherulesandcharacteristicsdefinedfortheorganization’sbestpracticesassociatedwithdesign.Thefocusisonthedesign process. “Did we follow our organizations guidelines for doing the designcorrectly?”Thedesignprocessalso includesensuring thedesignreflects thedesign-torequirements.Thus,designverificationisalsoaconfirmationthedesignisanagreed-totransformationofthedesign-torequirementsintoadesignthatclearlyimplementsthose


requirements correctly. “Does the design clearly and correctly represent therequirements?”“Didwedesignthethingright?”

• DesignValidation:confirmationthedesignwillresultinasystemthatmeetsitsintendedpurposeinitsoperationalenvironment.Willthedesignresultinasystemthatwillmeetthestakeholderexpectations(needs)thatweredefinedduringthescopedefinitionphasethatoccurredatthebeginningoftheproject?Thefocusisonthemessagethedesigniscommunicating.“Howwelldoesthedesignmeettheintentoftherequirements?”“Dowehavetherightdesign?”“Arewedoingtherightthings?”“Willthisdesignresultinthestakeholderexpectationsandneedsbeingmet?”

Oncethedesignisbaselined,thedesignistransformed;viabuild,code,buy,orreuse;intothesystemofinterest.Similartothediscussionforthedesignprocess,mostorganizationshaveasetofguidelinesor“goldenrules”thatguidethebuild(manufactureorcode)process.Theseincludeworkmanshipandqualitycontrolrequirementsontheorganization.

Afterthesystemhasbeenbuiltorcoded,therewillbeagatereviewwherethesystemisbothverifiedandvalidated.Atthisstageofthesystemlifecycle,theconceptsofsystemverificationandsystemvalidationtakeonamoreformalmeaning.Thus,theSystemsEngineering(SE)lifecycleprocessesincludetheprocessesofSystemVerificationandSystemValidation.Eachoftheserepresentasetofactivities(test,demonstration,inspection,analysis)thatcumulatewithoneormoregatereviewsassociatedwiththeacceptanceofthesystembythecustomer.ThusSystemVerificationisaformalSEprocessandhasalegalaspectwherethedeveloperisprovingthebaselinedrequirements,whichareatypeofcontractandarelegallybinding,havebeenmeet.

Inthiscontext,systemverificationhastwoaspects:1)Doesthebuiltorcodedsystemofinterestclearlyrepresenttherequirementsthatdrovethedesign?Didwebuildtherightthing?and2)Didthebuildorcodeteamfollowtheorganizationsguidelinesformanufacturingandcoding?

Followingsystemverification,systemvalidationisperformed.Again,SystemValidationisaformalSEprocessandhasalegalaspectwherethedeveloperisprovingwhetherornotthebuiltorcodedandverifiedsystem,resultsintheintendedpurposebeingmetintheoperationalenvironmentandthestakeholder’sexpectationsandneedsbeingmet.Likethesystemrequirements,thebaselinedstakeholderneedsandrequirementsdefinedduringthescopedefinitionphasecanalsobeconsideredpartofacontractandarelegallybinding.

Basedonthisdiscussion,tohelpremovetheambiguityintheuseoftheterms“verification”and“validation”,thefollowingdefinitionsforsystemverificationandsystemvalidationareproposed.ThesedefinitionsarealsoillustratedinFigure1.

• SystemVerification:SystemVerificationisdoneafterdesignandbuildorcoding,makingsurethedesignedandbuiltorcodedsystemmeetsitsrequirements.Thefocusisonthebuiltorcodedsystemandhowwellitmeetstheagreed-torequirementsetthatdrove


the design and fabrication. Methods used for system verification include: test,demonstration,inspection,oranalysis.“Didwebuildthethingright?”Alsoincludedinsystemverificationisadeterminationthattheteamresponsibleforbuildingorcodingthesystem of interests followed the organization’s rules, guidelines, and best practicesassociatedwithmanufacturingandcoding.Thefocusisonthemanufacturingorcodingprocesses. “Did we follow our organizations guidelines for manufacturing or codingcorrectly?”

• SystemValidation:Systemvalidationoccursaftersystemverificationandmakessurethedesigned, built, and verified system meets its intended purpose in its operationalenvironment.Thefocusisonthecompletedsystemandhowwellitmeetsstakeholderexpectations (needs) thatweredefinedduring the scopedefinitionphase that shouldhaveoccurredatthebeginningoftheproject.“Didwebuildtherightthing?”

Systemverificationandsystemvalidationprocessesandactivitiesaredirectlyrelatedtothecontractualobligationconceptforarequirementstatementandsetofrequirements.Itisthroughtheseactivitiesthatweprovewehavemetboththeagreed-torequirementsandtheagreed-toneedsoftheentitieswhoarethesourceoforownthem.Thisisoftenaccomplishedaspartofcertificationandacceptanceactivities.

VerificationandvalidationandtheSystemsEngineering“V”model

Whiletheprevioussectionpresenteddefinitionsthatareusefulinhelpingaddresstheissuesofambiguityintheuseofthetermsverificationandvalidation,thesystemengineeringprocessismorecomplex.SystemsEngineeringisaniterativeandrecursiveprocess.Requirementsdevelopmentanddesignoccurtop-downasshownontheleftsideoftheSE“V”asshowninFigure2.

Figure2:SystemsEngineer“V”Model

Operation andSupport

Concept Development Production

Development

StakeholderNeeds

Verifies

SystemRequirements)

StakeholderRequirements

SubsystemRequirements

Unit/componentRequirements

SystemIntegrationVerification

SubsystemIntegrationVerification

Unit/componentVerification

Verifies

Verifies

Validates PRODUCT

ProductLifecycle

Start

Operations

Engineering“Vee”Model

SystemsLifeCycleModel

Design

Disposal

VerificationMethods:- Test- Demonstration- Inspection/Observation- Analysis(includesmodels,

simulations, similarity)

Acceptance/OperationalEvaluation andValidation


AsshowninFigure2,Systemsengineeringstartswiththeconceptstagewherestakeholderneeds,expectations,andrequirementsareelicited,documented,andbaselined.Thisispartofdefiningthescopeofthesystemtobedeveloped.Thenthestakeholderneeds,expectationsandrequirementsaretransformedintoasetofsystemrequirementswhicharebaselinedviarequirementsverificationandrequirementsvalidationasdiscussedabove.Oncethesystemrequirementsarebaselined,designresultsinasystemarchitectureinwhichthesubsystemsaredefined.Thisdesignisbaselinedviathedesignverificationanddesignvalidationprocessdiscussedabove.

Foreachsubsystem,theabovecycleofstakeholdersubsystemneeds,expectations,andstakeholderrequirementsaredefinedandtransformedintosubsystemrequirements,whicharebaselinedviarequirementsverificationandrequirementsvalidationasdiscussedabove.Oncethesubsystemrequirementsarebaselined,designresultsinasubsystemarchitectureinwhichtheunits/componentsaredefined.Thisdesignisbaselinedviathedesignverificationanddesignvalidationprocessdiscussedabove.Thisprocesscontinuesuntiltheorganizationmakesabuy,build,code,orreusedecision.

Systemverificationandsystemvalidationoccurbottom-upasshownontherightsideoftheSE“V”asshowninFigure2.

Onceallthecomponentsthatmakeupthesubsystemsaredeveloped,unit/componentverificationandunit/componentvalidationtakeplaceasdescribedabove.Oncetheseactivitiesarecomplete,theunits/componentsareintegratedtogetherandthentheresultingsubsystemsareverifiedandvalidatedasdescribedabove.Oncethesubsystemverificationandsubsystemvalidationactivitiesarecompletethesubsystemsareintegratedtogetherandthensystemverificationactivitiesarecompleted.

Followingsystemverificationactivities,systemvalidationactivitiesareperformed.Thiscouldbedoneintheformofacceptanceand/oroperationevaluationandvalidationactivities.Intheend,proofwillbedocumentedthatcanbeevaluatedbythecustomertodeterminesystemvalidationactivitieshavebeencompletedsuccessfully,stakeholderneedshavebeenmet,andthesystemwilloperateasintendedinitsoperationalenvironment.

Followingthecustomerevaluation,thesystemcanbeacceptedandownershiptransferredtothecustomer.

Thefocusoftherestofthispaperisonsystemverificationandsystemvalidationplanning–addresstherightsideoftheSystemsEngineering“V”.(Formoreinsightintorequirementverificationandrequirementvalidation,seereferences1&2.)

A Winning Product vs. Risk

Attheendofaproject,allprojectteamswanttobeabletosaytheyhavebuilttherightthingandthattheproductmeetsstakeholderexpectationswhileaccomplishingitsintendedpurposeinitsoperationalenvironment.Icallthisdeliveringawinningproduct.Awinningproductis


definedhereinas:“Aproductthatdeliverswhatisneeded,withinbudget,withinschedule,andwiththedesiredquality.”Thegoalofallprojectsshouldbetodeliverawinningproduct.

Asimpleandpracticaldefinitionofriskis:“Anythingthatcanpreventyoufromdeliveringawinningproduct!”Failingtoplanforsystemverificationandsystemvalidationfromthebeginningoftheprojectisamajorrisktotheprojectthatcanresultinmassivecostoverrunsandscheduleslipsaswellasaproductthatdoesnotmeetrequirementsandfailstomeetstakeholderexpectations.Giventheseimpacts,allprojectmanagersneedtomitigatethisriskfromthebeginningoftheirproject.Recognizingthisiscriticaltobeingabletodeliverawinningproduct.

Requirementsbestpracticesincludetheplanningforsystemverificationandsystemvalidationactivitiesfromthebeginningoftheprojectandcontinuingtoaddresssystemverificationandsystemvalidationactivitiesthroughouttheproductlifecycle.Toreducetherisktoyourprojectassociatedwithfailingtoaddressverificationandvalidationactivities,itiscriticalthatyoufollowthesebestpracticesbyincludingsystemverificationandsystemvalidationplanningfromthebeginningofyourprojectstartingwithscopedefinitionandcontinuingthroughouttherequirementdefinition,management,anddesignprocesses.

Thefollowingsectionsintroducethesebestpracticesaswellasdiscusssomeofthetoolsandactivitiesthatwillhelpyouplanforsystemverificationandsystemvalidation.

Impact of requirement defects on system verification and system validation cost and schedule Requirementsarethesingleelementthattiesalltheproductdevelopmentlifecycleprocessestogether.Defectiverequirementshaveadirectandsignificantimpactonyourproject’scostandschedulewhenperformingsystemverificationandsystemvalidationactivities.

AsshowninFigure3[Hooks2001],thecosttofixrequirementdefects(re-work)increasesexponentiallyastheproductlifecycleprogresses.Thischartillustratestheorderofmagnitudecostsoffindingandfixingdefectsasweprogressintheproductlifecycle.Whatthisfigureshowsisthatthecostoffindingandfixingrequirementsdefectsinthecodingphaseis10timesmoreexpensivethanfindingandfixingthemduringtherequirementsphase.FindingdefectsduringDevelopmentTest,15-40timesmore,AcceptanceTest,30-70timesmoreandso-forth.InthecontextofthelifecyclephasesdepictedinFigure3,systemverificationandsystemvalidationactivitiesoccurduringdevelopmentandacceptancetestingaswellasduringinitialoperations.


Figure 3: Cost to fix requirement defects

AsshowninFigure3[Hooks2001],thecosttofixrequirementdefects(re-work)increasesexponentiallyastheproductlifecycleprogresses.Thischartillustratestheorderofmagnitudecostsoffindingandfixingdefectsasweprogressintheproductlifecycle.Whatthisfigureshowsisthatthecostoffindingandfixingrequirementsdefectsinthecodingphaseis10timesmoreexpensivethanfindingandfixingthemduringtherequirementsphase.FindingdefectsduringDevelopmentTest,15-40timesmore,AcceptanceTest,30-70timesmoreandso-forth.InthecontextofthelifecyclephasesdepictedinFigure3,systemverificationandsystemvalidationactivitiesoccurduringdevelopmentandacceptancetestingaswellasduringinitialoperations.

Note:Projectmanagersoftendon’tunderstandtheseconceptsnortheresourcesneededtoaccomplishtheactivitiesassociatedwiththeproductlifecyclephases.Ihaveseencaseswherethecosttodosystemverificationandsystemvalidationactivitiesendedupcostingbetween50%-70%ofthetotaldevelopmentcostforanewsystem.[Hooks2001].Thehigherpercentagesoccurduetohavingapoorsetofrequirementsandthustheresultingrework.Projectmanagerswhoonlybudgetformaybe20%ofthedevelopmentcostsforsystemverificationandsystemvalidationareinforarudeawakingandaresettingtheirprojectupforfailure.

Earlyidentificationofdefectiverequirementspreventsre-workandresultsinsignificantcostsavings.Conversely,allowingthesedefectstogoundetecteduntilsystemverificationandsystemvalidationwillresultinsignificantcostandschedulerisk.

Consideringyoursystemverificationandsystemvalidationstrategyearlyinthedevelopmentlifecycleprovidesafoundationforestimatingyourcostandschedulefortheseactivities.

Resistthetemptationtoreduceyoursystemverificationorsystemvalidationactivitieswhenyouareoverbudgetorhavescheduleproblems.AsshowninFigure3,todosocouldresultin


muchlargercostsandscheduleslipsduetoreworkifrequirementdefectsarenotdiscovereduntilsystemvalidationoroperations.

FailingdorequirementverificationandrequirementvalidationandbaseliningyourrequirementsbeforedesigncanresultinrequirementverificationandrequirementvalidationANDdesignverificationanddesignvalidationactivitiesbeingperformedtogether,duringwhatwassupposedtobejustdesignverificationanddesignvalidationactivities.Thiscouldresultinadesignthatmeetstherequirements(passesdesignverification)butiftheyarethewrongordefectiverequirements,thedesignwillfaildesignvalidation.

Evenworse,failingdorequirementverificationandrequirementvalidationandbaselinebeforedesignandbuildcanresultinrequirementverificationandrequirementvalidationANDdesignverificationanddesignvalidationactivitiesbeingperformedatthesametimeyourfocusshouldbeonsystemverificationandsystemvalidation.Whathappensifyouverifythatyoursystemmeetsitsrequirements,buttherequirementsetisdefectivewithambiguous,missing,orincorrectrequirements?Designverificationanddesignvalidationmaynotexposethesedefects.

Becausethesetofrequirementsisdefective,theydon’tcorrectlyandcompletelycommunicatestakeholderneedsandexpectationsresultinginyoursystemfailingtopasssystemvalidation.Ifyoursystemfailssystemvalidation,itwillnotmeetthestakeholderneedsexpectationsandwillnotaccomplishitsintendedpurposeinitsoperationalenviornment.Youwillnothavedeliveredawinningproduct.

Inthecommercialsector,warrantycostsareamajorconcern.Warrantycostsasafunctionofsalescaneatintoacompany’sprofits.Allowingdefectiverequirementsinyourrequirementsetandfailingtodoadequaterequirementverificationandvalidation,designverificationandvalidation,andsystemverificationandvalidationpriortoproductlaunchincreasestheriskofproblemsnotbeinguncovereduntilaftertheproductisprovidedtocustomersandreleasedintothemarketplace.Ahighdefectrateresultsnotonlyinhighwarrantycostsandreducedprofitability,butalsocanimpactacompany’sreputationresultinginreducedsales.

Note:Theconceptof“systemvalidation”discussedaboveispartoftheformalproductdevelopmentlifecycleaddressedinmosttextbooks.Whatmostpeopledon’trecognizeisthatthereisalsoan“informal”systemvalidationthattakesplaceafteraproductisreleasedforuse.Thisinformalsystemvalidationiscommunicatedtothedevelopingorganizationisvariousformsinadditiontoproductfailuresduringoperationswhichleadstocostlywarrantywork.Theseinformalsystemvalidationactionsinclude,returns,negativecustomercommentsandreviews,recalls,decliningreputation,decreasedsales,salesprojectionsnotbeingmet,etc.Allofthesethingshaveadirectimpactonprofitsandmissionsuccess.Intoday’sworldoflimitedfunding,profitmarginsaregettingtighterandtighter.Companiescannotaffordtofailinformalsystemvalidationanymorethantheycanaffordtofailformalsystemvalidation.

Therisksofcostoverrunsandscheduleslipsassociatedwithfailingtodevelopdefectfreerequirementsandfailingtoplanaheadforsystemverificationandsystemvalidationactivities


aremitigatedbyadoptinggoodrequirementdevelopmentpracticesandplanningaheadforsystemverificationandsystemvalidationactivitiesduringthescopedefinitionandrequirementdevelopmentandmanagementlifecyclephases.

The importance of thinking ahead to system verification and system validation Unfortunately,therearealltoomanyexamplesofaproductbeingdeliveredthatmetthe“customer”requirements,yetdidnotmeetstakeholderexpectationsandwereeithernotusedorrequiredsignificantchangestobeuseful.Whatwentwrong?

Addressing System Verification and System Validation activities during Scope Definition Phase

Oneareatolookatisinthescopedefinitionphaseoftheproject.Howwellhasthescopeofyourprojectbeendefined?Doesitclearlyreflectstakeholderneeds,expectations,andrequirements?Inthefinalanalysis,systemvalidationisbasedonmakingsuretheproductaddressesthestakeholderneeds,expectations,andrequirementsdefinedduringthescopedefinitionphase,agreed-tobytherelevantstakeholders,andbaselinedinthesystemScopeReview(SR)orMissionConceptReview(MCR)beforetherequirementsarewritten.

Thescopeofaprojectconstitutesthevision:the“Need”todeveloporprocureaproductorservice;thegoalsandobjectivesofthestakeholders;informationaboutthestakeholders,especiallycustomersandusersoftheproductorservice;driversandconstraints,andconceptsforhowtheproductwillbedevelopedorpurchased,tested,verified,validated,deployed,used,maintained,anddisposedof.Thescopealsoincludestheidentificationofproductboundaries(interfaces).

Thinkingaheadtosystemverificationandsystemvalidationbeginsduringthescopedefinitionphase.Duringthisphase,inadditiontodefiningstakeholderexpectations,theprojectisdevelopingdraftsoftheirProgram/ProjectManagementPlan(PMP),SystemsEngineeringManagementPlan(SEMP),andtheMasterIntegration,Verification,andValidation(MIVV)Plan.SchedulesandbudgetsarebeingformulatedandtheWorkBreakdownStructure(WBS)isbeingdefined.Becausesystemverificationandsystemvalidationactivitiesrequireasignificantamountoftheproject’sresources,planningfortheseactivitiesmustbeginduringthescopedefinitionphase.

SpecialScopeDefinitionPhaseconsiderations:

• DefineNeed,goals,andobjectives(NGOs)

The“Need”foraprojectdefinesthe“why”–whyarewedoingthis?Whatarewetryingtoaccomplish?The“Need”isbasedonananalysisofaproblemthattheprojectissupposedtosolveforsomestakeholderorgroupofstakeholders.Goalsarethosethingsthatyouplantoaccomplishthatwillresultinmeetingthe“Need”includingMeasuresofEffectiveness(MOEs)-thethingsyouplantousetomeasuresuccess.Objectivesandresultingtechnicalrequirements


areMeasuresofPerformance(MOPs),includingKeyPerformanceParameters(KPPs)thatshowthatyouhavemetthegoals.Objectivesshowthatyouhave“gottenthere”,i.e.,yourproductaccomplisheswhatwasexpectedofitbythestakeholdersandcanfulfillitsintendedpurposeinitsoperationalenviorment.

Requirementvalidationaddresseshowwelltherequirementscommunicatethestakeholderneedsandexpectationsandsystemvalidationactivitiesultimatelyaddresshowwellthedesignedandverifiedproducthasmettheproject’sNGOs.TheprojectteamcannotclaimtheydeliveredawinningproductiftheNeed,goals,andobjectiveswerenotmet.(FormoreinformationondefiningNGOsandotherscopedefinitionbestpractices,seereferences1,2,13,14,and15.)

• Indentifyandinvolverelevantstakeholders

Stakeholdersincludekeyrepresentativesfromvariousorganizationsandgroupsthathavea“stake”or“interest”inyourproject.Fromasystemverificationandsystemvalidationperspective,thestakeholdersthatwillhaveanythingtodowithanyofthevaricationandvalidationactivitiesneedtobeidentifiedandinvolvedfromthebeginningofyourproject.Keystakeholdersthatwillbe“signingoff”andacceptingtheproductmustagreethattheplannedsystemverificationandsystemvalidationactivitieswillresultinsufficient“proof”suchthattheywillapproveandaccepttheproduct.Duringscopedefinition,apreliminarydraftofyourMIVVPlanshouldbedeveloped.

• Identifydriversandconstraints

Driversandconstraintsarethosethingsthatareimposedfromoutsideyourprojectthatyouhavelittlecontrolover,butarerequiredtoshowcompliance.Driversandconstraintsrepresentamajorsourceofrequirements.Systemverificationandsystemvalidationactivitiesmustbecompletedsuccessfullytoshow“proof”thatthesystemiscompliantwiththeidentifieddriversandconstraints.Failingtoshowcompliance,willresultinthesystemfailingcertificationforuse.

Youneedtomakesureyouincludeappropriateindustrystandardsandanyotherstandardsmandatedbygovernmentregulatoryagenciesaspartoftheprojectdriversandconstraints.ForsystemstargetedforuseintheUnitedStates,chemicalcontainmentandemissionsrequirementsfromtheEnvironmentalProtectionAgency(EPA),operatorprotectionrequirementsfromtheOccupationalSafetyandHealthAdministration(OSHA),clinicaltrialsrequirementsfromthefoodandDrugAdministration(FDA),materialsflammabilityrequirementsfromtheFederalAviationAdministration(FAA)orNationalAeronauticsandSpaceAdministration(NASA),oraccessibilityrequirementsmandatedbytheAmericanswithDisabilitiesAct(ADA)areasamplingofthetypeofoutsidedriversthatmustbereflectedintherequirementsetforyoursystem.

Ifyouaredevelopingsystemsforuseinternationally,youneedtofindandincorporatetherelevantstandardsandregulationsinyourrequirements.Yoursystemverificationandsystem


validationplanningmustincludeproducingthedatarequiredbytheregulatoryagenciesto“prove”compliancewiththeirrequirements.

Whatifyouaredevelopingamedicalproduct?Whatsystemverificationandsystemvalidationrequirementswillthegovernmentandmedicalinsurersplaceonyourproduct?Whatextrareviewsbyoutsideexpertsandwhatliabilityinsurancewillberequiredbeforeyoucanconducttestsinvolvinghumanandanimalsubjects?

Otherproductsmayrequireoutsideexpertsforcertification.DoesyourproductrequireanUnderwriters’Laboratories(UL)certificationormaybeyouhaveapressurevesselthatrequiresanAmericanSocietyofMechanicalEngineers(ASME)Class1Certification?

Evenifnotexplicitlystatedbythecustomerorotherstakeholders,beawareofandresearchtherelevantoutsidedriversandconstraints.Complianceisexpectedandfailuretoaddressrelevantdriversandconstraintscanresultinfailureduringsystemverificationandsystemvalidationandproductrejectionbythecustomerevenifthecustomerdidnotexplicitlycommunicatethesedriversandconstraintstoyou.

• Assessthetechnologymaturityofthesystemtobedeveloped

Akeydriverandconstraintistechnology.Tomeetyourobjectives,especiallyhighpriorityMOPsandKPPs,anassessmentofthecurrentmaturitylevelofallthetechnologydevelopmentneedsfortheprojectisnecessary.OutofthisprocessassignaTechnologyReadinessLevel(TRL)foryoursystemanddevelopaTechnologyMaturationPlan.AllrequirementstiedtotheMOPsandKPPsarehighpriority.IftheTRLassociatedwiththerequirementsthatimplementyourMOPsandKPPsislow,theyarealsohigh-riskrequirementsandmustbetracedcloselybyprojectmanagement.Planningforsystemverificationandsystemvalidationofhighpriorityandhigh-riskrequirementsisextremelyimportant.Theassociatedactivitiesandresultingresourceneedscanhavebigimpactsonyourproject’scostandschedule.IncludeinyourplanningaplanformaturationofkeytechnologiesrequirementtomeetyourNGOs.Thisisnecessaryasamajorriskmitigationactivitytoprotectagainstcostandscheduleoverruns.(FormoreinformationonTRLsandthedevelopmentoftechnologydrivenproductsseereference16.)

• Definefeasiblescenariosandconceptstomeetthestakeholderexpectations

Amajoractivityofscopedefinitionisthedevelopmentofasetofscenariosandsystemconceptsthataddresstheneedsandexpectationsofallthestakeholdersandaddresseachoftheproduct’slifecycles--forbothnominalandoff-nominalconditions.ScenariosandsystemconceptsforsystemverificationandsystemvalidationneedtobedefinedanddocumentedsoyouhavetheinformationneededtodevelopyourWBS,budget,schedule,anddraftMIVVplan.Amajorreasonforaproject’scostoverrunsandschedulingdelayscanbecontributedtothefailureoftheprojecttoadequatelyplanaheadforsystemverificationandsystemvalidationactivities.


Whendevelopingscenariosthataddresssystemverificationandsystemvalidationactivities,payparticularattentiontoothersystemsthat“enable”youtoaccomplishyoursystemverificationandsystemvalidationactivities.Youmayneeduniquesupportortestequipment,specialfacilities,ormodelstoperformyoursystemverificationandsystemvalidationactivities.Thesesystemsarereferredtoasenablingsystems.Theseareproductsthatneedtobeidentified,defined,designed,modified,built,procured,verified,andvalidatedintimetomeetyoursystemverificationandsystemvalidationschedule.Willthesesystemsbereadytosupportyoursystemverificationandsystemvalidationactivitieswhenyouneedthem?Earlysystemverificationandsystemvalidationassessmentsduringthescopedefinitionphasewillhelpyouidentifytheseenablingsystemsaswellasadditionalprojectrequirementsneededtosupportsystemverificationandsystemvalidationactivitiesandallowyoutoincludetheminyourprojectbudgetandschedule.

Yoursystemmayneedspecificfeaturesdedicatedtosupportyoursystemverificationandsystemvalidationactivitiesespeciallywhenusingtestordemonstrationasmethodsforsystemverificationandsystemvalidation.Additionalinterfacesmaybeneededtogiveyouaccesstodatathatisneededforsystemverificationandsystemvalidation,butnotnormaloperations.Thiscouldresultintheneedforadditionalconnectors,testpoints,andwiringharnessestoconnecttospecialtestinstrumentationorexternalpower,extradatatodisplayorrecord,oradatabasetogiveyouvisibilityintoaninternalprocess,aninspectionportal,orabrackettoholdapartinatestfixture.[Hooks2001]

Requirementsthatareverifiedbyanalysisusingmodelscanresultinadditionaldevelopmentefforttoproducethosemodels.Modeltypesincludephysical,graphical,mathematical,andstatistical[Larsen2009].Insomecases,themodelsneededforsystemverificationandsystemvalidationcanbeveryexpensivetodevelopandtakeupalargeportionoftheproject’sresources.Inaddition,youwillneedtovalidatethemodel.Todothis,youmayhavetodevelopasimulationofyoursystem.Thissimulationwillbeusedtorunteststocollectdataneededtovalidatethemodelbeforethemodelcanbeusedtoverifyyoursystem’srequirementsorforsystemvalidation.

Whatistheneededfidelityofthesystemyouplantouseforsystemverificationandsystemvalidation?Foryoursystem,youneedtoassesswhichrequirementscanbeverifiedusinganengineeringtestmodel,qualificationunit,delivery-likehardware,oractualdeliveryorflighthardware.Whatfidelityofequipmentisneededduringdevelopment?Qualification?Acceptance?Operations?Eachofthesetypesofequipmentorversionsofyourproductneedstobeincludedinyourbudgetandschedule.

Theseareexamplesoffeaturesoradditionalsystemsthatareneededtomakesystemverificationandsystemvalidationpossibleinsomecasesandreducecostsinothercases.Addressingtheequipmentandfacilitiesneededtoperformsystemverificationandsystemvalidationearlyavoidspossibledelaysinactualproductdelivery.Likemodelsdiscussedabove,youmustverifyandvalidatethisequipmentpriortoyouruseduringyoursystemverificationandsystemvalidationactivities.Planningforsystemverificationandsystemvalidationearly


permitstheexecutionofconcurrentactivitiesthatwillallowyoutocompleteyoursystemverificationandsystemvalidationperyourscheduleandavoiddeliveryscheduledelaysandresultingcostoverruns.

• Defineproductboundariesandexternalinterfaces

Aninterfaceisaboundaryacrosswhichsystemsinteract.Seriousproblemscan,andalltoooftendo,ariseattheinterfaces.Yourprojectisparticularlyvulnerablewheninterfacingwithexternalsystemsoverwhichyouhavelittleornocontrol.Becauseofthis,yourproductisatmostriskattheinterfaces.Identifyinginterfacesfacilitatesdefinitionofyoursystem’sboundariesandclarifiesthedependenciesyoursystemhasonothersystemsanddependenciesothersystemshaveonyoursystem.Indentifyinginterfaceshelpsyouensurecompatibilitybetweenyoursystemandthosewithwhichitinteracts.Identifyinginterfacesalsohelpstoexposepotentialprojectrisks.

Becauseofthis,itisextremelyimportantthatyoudefineyourconceptsforhowyouwillmakesure(verify)yoursystemmeetsallinterfacerequirementsandvalidatethatyoursystemwillworkwithalltheexternalsystemswithwhichitmustinteractintheintendedoperationalenvironmentANDisprotectedfromoutsidethreatsacrossyourinterfaces.

Inyourplanning,besuretoaddressbothinternalandexternalinterfaces.Anemulatororsimulatormaybeneededtocompleteyoursystemverificationandsystemvalidationactivities.Asdiscussedearlier,thesemustbeeitherdevelopedorprocuredandverifiedandvalidatedpriortouseduringyoursystem’sintegration,systemverification,andsystemvalidationactivities.

Addressing System Verification during the Requirement Definition Phase Identificationoftheproposedsystemverificationmethodisanessentialattributeofawell-writtenrequirement.Identificationofthesystemverificationmethodasyoudevelopyourrequirementsimprovesrequirementquality,ensuresyourrequirementsareverifiable,providesthebasisforestimatingsystemverificationcostandschedule,andcanhelpreducethecostoftheseactivities.Carefulassessmentoftheverificationmethodandassociatedimplementationactivitiescanresultinamodificationtotheassociatedsystemverificationactivitiestoreducethesystemverificationcostandtimewhilestillmeetingyoursystem’sNeed,goalsandobjectives.

Duringthescopedefinitionphasediscussedabove,Idiscussedsystemverificationandsystemvalidationfromthe10,000-footlevel.Basicconceptsandscenariosforcompletingyoursystemverificationandsystemvalidationactivitiesweredefined.ThisinformationwasusedtodevelopahighlevelbudgetandschedulefortheseactivitiesinyourPMP,defineyourhighlevelprocessesintheSEMP,andprovidedadetailedprocessdefinitionintheMIVVPlan.

However,thedevilisinthedetails.Asrequirementsaredeveloped,theverificationmethodthatwillbeusedtoshowthesystemmeetstherequirementsmustbeaddressed.Onceyourrequirementshavestabilized,youalsoneedtostartdefiningtheactivitiesneededtoverifyyour


systemmeetsitsrequirements.Let’slookatthecharacteristicsofgoodrequirementsfromasystemverificationperspective.

• Requirementisneeded

Amandatorycharacteristicofarequirementisthatitisneeded.Ifarequirementisnotneeded,whyisitintherequirementset?Byitsveryexistence,arequirementhasadollarvalueassociatedwithit.“Eachrequirementcarriesacost.Itisthereforeessentialthatacompletebutminimumsetofrequirementsbeestablishedfromdefinedstakeholderrequirementsearlyintheprojectlifecycle.Changesinrequirementslaterinthedevelopmentcyclecanhaveasignificantcostimpactontheproject,possiblyresultingincancellation.”[INCOSE2010].

Allrequirementsneedtobemaintained,managed,andverified.Requirementsthatarenotnecessaryresultinincreasedmanagementcostoftherequirementsthat,inturn,increasesoverallprojectcostsandleavesfewerresourcesforneededrequirements.Un-neededrequirementsresultinworkbeingperformedthatisnotnecessary,takingresourcesawayfromtheimplementationofthoserequirementsthatareneeded.Rememberthatintegration,systemverification,andsystemverificationcostscanbe50%ormoreofyourdevelopmentbudget.Youdon’twanttospendtimeandeffortprovingthesystemmeetsrequirementsthatwerenotneeded.Inaddition,implementingrequirementsthatarenotnecessarycanresultindegradedsystemperformanceaswellasintroducingapotentialsourceoffailureandconflict.Werecommendyoutakea“zero-based”approachtodefiningyourrequirementset.Bythiswemeanthatyoudefinetheminimumsetthatisnecessaryandsufficienttomeetthestakeholderneedsandexpectations.Thenifanyonewantstoaddadditionalrequirements,theyhaveto“fight”togettheadditionalrequirement(s)addedtotheset.Insistthateachrequirementthatissubmittedhasrationaledocumentedwiththerequirementthatclearlycommunicatestheneedforthatrequirement.

Totestfortoseeifarequirementisneeded,ask:

1) Whyistherequirementneeded?Whyisitintherequirementset?Ifthereisnorationale,oraweakrationaleforitsexistence,deleteit.

2) Whatwouldhappenifwedeletedthisrequirement–wouldthedeveloperaddressthisconcernanyway?Ifnothingwouldhappenorthedeveloperwouldhavetoaddressthisconcernanywaytomeetotherrequirements,thenwhyisitintheset?Deleteit.

• Requirementisverifiable

Anothermandatorycharacteristicofarequirementisthatitmustbeverifiable.Whyaskforsomethingifyoucan’tproveithasbeenimplementedasintended?Arequirementthatisnotverifiablemayresultintheimplementationofanincorrectsolutionorthesystemmaybeverifiedasdoingsomethingotherthanwasintended.Ifthetrueintentoftherequirementisnotclear,stakeholderexpectationsmaynotbemet.Insistthateachrequirementthatissubmittedhastherecommendedverificationmethoddocumentedwiththerequirement.

Totestwhetherarequirementcanbeverified,ask:


1) Howcanthisrequirementbeverified?Ifnomethodcanbeidentified,canitbere-writtensothatitcanbeverified?

2) Dowewanttoverifythisrequirement?Ifnot,deleteit.3) Isthiswhatwereallywanttoverify?Ifnot,changeittowhatyoudowanttoverifyor

deleteit.

• Requirementisattainable/achievable

Athirdmandatorycharacteristicofarequirementisthatitmustbeattainable/achievablegiventheexistingbudget,schedule,andleveloftechnicalmaturity.Arequirementthatisnotattainablewillfailsystemverification.Ifyouknowyouwillnotbeabletomeettherequirementandthusknowyouwillfailtoverifythatthesystemmeetstherequirement,whystateit?Statingarequirementthatisnotattainablewillresultinwastedeffort,costandscheduleimpacts,aswellasunmetstakeholderexpectations(performance).Iftherequirementisaperformancevalue,youneedtodoanassessmentastothematurityofthetechnologyneededtomeettherequirement.Ifthecurrentmaturityislow,therequirementrepresentsarisktoyourprojectandyouneedaplantomitigatethatriskandmaturethetechnology.Thiscouldrepresentbothacostandscheduleriskaswell.

• Requirementiswrittencorrectly

Ifarequirementispoorlywritten,itwillmostlikelynotbeverifiable.Ifitcontainsambiguousterms,itcanbeunderstoodinmorethanonewayandisthusnotverifiable.Ifitisnotclear,concise,orusespoorgrammarandyouareleavingroomformultipleinterpretations,itisnotverifiable.Ifitisstatednegatively,itmaynotbeverifiable.(Itisdifficulttoprovethatasystemnever,everdoessomething.)Iftherequirementcontainsmorethanonethought,whichthoughtdoyouverify?Thisisespeciallyproblematicifeachthoughtneedsadifferentmethodofverification.

Toguardagainstpoorlywrittenrequirementsmakesureyourteamistrainedtowritedefectfreerequirementsanddefineyourrequirementdevelopmentprocessasonebasedupontheuseofthe“RulesofWritingGoodRequirements.”(Seereference2,aswellasAppendixCoftheNASASystemsEngineeringHandbook,“HowtoWriteGoodRequirements”.[NASA2007].)Inaddition,youcangetacopyofthe“INCOSEGuidetoWritingRequirements,2015”.

• Requirementisdocumentedatthelevelwhereitwillbeverified.

Properflowdown(allocation)ofrequirementstosystemarchitecturallevelsisveryimportanttoeffectivesystemverificationandsystemvalidation.Allrequirementsmustbedocumentedatthelevelwheretheywillbeimplementedandverified.

AsshowninFigure2,theSE“V”model,requirementdefinitionanddesignaretop-downactivitiesduringwhichrequirementsforthesystemasawholearedefinedandthendecomposedintosub-systemsdefinedduringthedesignactivitywhileallocatingrequirementstoeachsubsequentlevelofthearchitecture,descendingdowntheleftsideofthe“V”.


Oncethesystemhasbeendecomposedintothevariousunits/componentsthatmakeupthesystem;integration,systemverificationandsystemvalidationactivitiesbegin,ascendinguptherightsideofthe“V”.Duringintegration,lowerlevelunits/componentsareintegratedtogethertoformthesubsystems.Priortointegrationintosubsystems,thelowerlevelunits/componentsareverifiedandvalidatedandcompatibilityisassessedtoensuretheselowerlevelentitiesmeettherequirementsthatdrovetheirdesignandwillsuccessfullycombineintothesubsystemsatdefinedinterfaces.Oncethesubsystemshavebeenformed,theprocessrepeats.

Priortointegrationintothesystem,thesubsystemsareverifiedandvalidatedandcompatibilityisassessedtoensurethesubsystemsmeettherequirementsthatdrovetheirdesignandwillsuccessfullycombineintothesystematthedefinedinterfaces.

Once,thesubsystemshavebeenintegratedtogethertoformthesystem,systemverificationactivitiesareperformedtoensuretheintegratedsystemsmeetstherequirementsthatdrovethesystemdesign.

Oncetheentiresystemhasbeenverified,thesystemisreadyforsystemvalidationandacceptancebythecustomer.Insomecases,finalsystemvalidationandsystemacceptanceactivitiesarecombined.

Specifyingarequirementatalevelhigherthanwhereitwillbeverified,maymakeitdifficultorimpossibletoverifyatthatlevel.Whenwritingrequirements,alwaysask:“Doesthisrequirementapplytothelevelyouarewritingrequirementsfor?”Therequirementmaybevalid,butnotforthatlevel.Iftherequirementisnotbeingstatedatthecorrectlevel,moveittothecorrectlevel.

• Requirementisallocated(floweddown)tothenextlevelofthesystemarchitecture

Theflowdownofrequirementsfromoneleveltothenextlevelofthearchitectureisanimportantactivitythatneedstobedonecorrectly.Thisflowdownactivityiscalledallocation.Allocationistheprocessofapportioningresourcesorassigningresponsibilityforimplementationofrequirementsfromanupperlevelinthehierarchytopartsatthenextlevelofthesystemarchitecture.Allrequirementsmustbeallocateduntilthefinallevelofthearchitectureisdefined.Onceallocated,thenextlevelwillderivechildrenrequirements,thatwhenimplementedatthatlevel,willcontributetotheparentrequirementbeingmet.

Failingtoallocateyourrequirementscanresultinrequirementsnotbeingimplementedatthenextlevelofthearchitecture.Failingtoallocaterequirementscanalsoresultinmissinglowerlevelrequirements(derivedchildrenrequirementsthatarenecessaryandsufficienttomeettheparentrequirement.)This,inturn,couldresultinthesystemfailingeithersystemverificationorsystemvalidation.

• Requirementaddressesaninternalinterface

Akeypartoftheallocationprocessistheidentificationanddefinitionofinternalinterfaces


amongend-itemswithinyoursystemaswellasexternalinterfacesbetweenyoursystemandtheoutsideworld.Whenfunctionalrequirementsareallocatedtomorethanonepartofthearchitectureatthenextlevel,theremaybeaninterface.Ifso,theinterfacemustbeidentifiedanddefined.Therespectiverequirementsdocumentedforeachpartthenwillincludetheirrespectiveinterfacerequirements.Interfacerequirementsrequirespecialattentiontoensurethattheyarewrittensoastopermitsystemverification.(Theinclusionofareferencetowheretheinterfaceisdefinedisacommonpartofallinterfacerequirements.Thisdefinitionisneededsothattheinterfacerequirementcanbeverified.TheinterfacedefinitionscanbecontainedinanInterfaceAgreementDocument(IAD)orInterfaceControlDocument(ICD).)(Formoreinformationonidentifyinganddefininginterfacesandwritinganddocumentinginterfacerequirements,seereference17.)

• Requirementistraceabletoaparentorsource

Traceabilityistheconceptthatallrequirementsneedtobelinked(traced)totheirparentorsource.Tracingtoaparentorsourcesisalsoamethodtodetermineifarequirementisneeded.Whenthereisnotracetoaparentorsource,itcouldmeanthattherequirementisnotneededandthereisgoldplating.“Goldplating”istheactofaddingfeaturestoasystemthatarenotneeded.Goldplatingisamajorsourceofrequirementscreepandcanimpactthecostandscheduleofyourproject.Thisisespeciallytruewhenitcomestosystemverificationandsystemvalidationactivities.

Oncerequirementshavebeenallocatedtoapartatthenextlevelofthearchitecture,childrequirementsarederivedsuchthatwhenimplementedresultsinimplementationoftheparentorsourcerequirement.Assessmentofwhetherornotthechildrenrequirementslinked(traced)totheparentorsourcerequirementarenecessaryandsufficienttomeettheintentoftheparenttoisneededtodeterminewhetherornotaparentrequirementisproperlyimplemented.Withoutproperallocationandtraceabilitydocumentation,thisassessmentcannotbedone.

Understandingandthencorrectimplementationanddocumentationoftheconceptsofallocationandtraceabilityarethekeystoasuccessfulverificationandvalidationprocess.AsshowninFigure2,SystemsEngineeringVmodel,integrationisabottoms-upprocess.Whenintegratingatonelevel,youneedtomakesuresystemverificationandsystemvalidationarecompleteatthepreviouslevel.Successfulsystemverificationatthelowerlevelisaprerequisitetosuccessfulsystemverificationatthenextlevel.Thisisespeciallytruewhenitcomestointerfaces.

Akeypointisthatbeforeverifyingthesubsystemorsystemmeetsarequirement,ifthatrequirementhaschildrenrequirements,verificationthatthepreviouslevelmeetsthechildrenrequirementsmusthavebeencompletedbeforeverifyingtheparentrequirementhasbeenmet.Proofthatthechildrenrequirementshavebeenmetisneeded(aprerequisite)beforeverifyingthattheparenthasbeenmet.Allocationandtraceabilityallowthistohappen.


Whenplanningsystemverificationatonelevel,alwaysincludeastepinyourprocesstolookatthesystemverificationdocumentationatthepreviousleveltomakesureithasbeensuccessfullycompletedbeforeyoudosystemverificationatyourcurrentlevel.

Planning your system verification and system validation activities Thissectionprovidesmoreofthe“how”associatedwithplanningforsystemverificationandsystemvalidationbyintroducingsomeofthetoolsandactivitiesyoucanusethatwillhelpyouplanforandaccomplishsystemverificationandsystemvalidation.Twoexcellentsourcesthatgointomuchmoredetailcanbefoundinreferences3&4.

Thinking about system verification and system validation while developing your requirements

Asyouwriteyourrequirements,addressthecharacteristicsofgoodrequirementsdescribedintheprevioussection.Ensurethatyourrequirementsare:needed,verifiable,attainable,correctlywritten,atthecorrectlevel,allocated,andtraced.Alsoensurethatallinternalandexternalinterfacesaddressed.

Addressingsystemverificationastherequirementsarewrittenensuresabettersetofrequirements.Thefollowingquestionsneedtobeaddressedaseachrequirementiswritten:

1) “Whatwillbetheprimarymethodusedtoverifythisrequirement?”

TheprimarymethodsofverificationincludeTest,Demonstration,Inspection,andAnalysis.(Note:SomeorganizationsfurtherdefinethevariousmethodsofAnalysistoincludeanalysisbysimilarityandbymodeling.)“Eachrequirementshouldbeverifiablebyasinglemethod.Arequirementrequiringmultiplemethodstoverifyshouldbebrokenintomultiplerequirements.“[INCOSE2010].Somepeoplewillstatemorethanoneverificationmethod,e.g.,TestandAnalysis.ThisisnotneededasitisunderstoodthatverificationbyTestincludessomeanalysisactivitiesandAnalysismayincludesometestingactivities.Thebestpracticeistostateasingleprimarymethodforverification.Note:Makesureyouunderstandthedifferencebetweentest,inspection,andanalysisasactivities,vsTest,Inspection,andAnalysisasmethodsofsystemverification.Theconceptsarecompletelydifferent,yetoftenthisdifferenceisoftennotunderstood.

Riskisamajorconsiderationwhenchoosingthesystemverificationmethod.Thisisaveryimportantissuefromaprojectmanagementstandpoint.SystemverificationbyTestprovidesthemostconfidencethatthesystemmeetsarequirement.WewouldlikeuseTestasourprimarymethodforallrequirements,butwemaynotbeableto.InsomecasesTestisnotappropriate.Inothercases,inmaynotbepossibleuntilthesystemisintegratedandinitsrealoperationalenvironment.Inmanycaseswecan’taffordthecostortimetodoafull-uptestforallourrequirements.Inothercases,aspecialfacilitytosimulateyouroperationalenvironmentorequipmenttotestyourinteractionswithothersubsystemsorsystemsmaynotbeavailablenorcosteffectivetodevelop.


Alloftheseconsiderationsneedtobeaddressedwhendecidingonthesystemverificationmethod.Therefore,youmustdoariskassessmentanddeterminewhichrequirementsrepresentthehighestrisktoyourprojectifnotproperlyimplemented.Oftentherequirementsthatposethehighestrisktoprojectsuccessarerequirementsthattracetothehighpriorityobjectivesdefinedduringscopedefinitionaswellasthesafetyandmissionassurancerequirements.

Whenarequirementposesalowerrisktotheproject,oneoftheotherverificationmethodsmaybeacceptable.Inmyexperienceanalysisisthemostoverusedmethodandpossesthehighestriskwhengoodrationaleforusinganalysisisn’tdefined.Reducingverificationactivitiesduetocostoverrunsorscheduleslipscanaddsignificantrisktoyourproject.

Therationaleforwhyyouareusingaspecificverificationmethodneedstobeincludedinyourplanforeachrequirement.

Becausechoosingthesystemverificationandsystemvalidationmethodisreallyacost,schedule,andriskdecision,theProjectManagerandLeadSystemsEngineerneedtobeinvolvedinthedecisionastowhichsystemverificationmethodwillbeusedandwhichactivitieswillbeincludedaspartofthesystemverificationmethodagreedto.

2) “Whatactivitiesneedtobeperformedaspartofthesystemverificationorsystemvalidation?”

Developascenariooroperationalconcepttodeterminewhichactivities(test,inspection,analysis)needtobeperformedaspartofthechosenverificationorvalidationmethodbeingsuretoaddresswhereandbywhom.Thisprocesswilluncovertheinterface,facility,supportequipment,andinstrumentationissuesassociatedwithyoursystemverificationandsystemvalidationactivities.Wherewillthesystemverificationorsystemvalidationactivitiesbeperformed?WillthesystemunderverificationorvalidationrequiredifferentpowerorcoolingduringaTestorDemonstrationthanneededduringnormaloperations?Willitneedsimulateddatastreams?Howwillyouaddresstheinterfaces–isanemulatororsimulatorneeded?Whatexternalcommandandcontrolcapabilityisneeded?Whatdataisneededtocontroltheactivity?Whatdatamustberecordedandhowfastmustthesystemoutputthisdata?Whowillbeinvolvedinthesystemverificationorsystemvalidationactivity?Whomustobservethetestordemonstrationactivity,signoffthattheactivitywasperformedperprocedure,performtheanalysisofthedataresultingfromthesystemverificationorsystemvalidationactivity,orinspectthesystem?Whattrainingorcertificationsdothepersonnelneedtohavetoparticipateintheirassignedrole?Whatsafetyconsiderationsarethere?Whatresourcesareneededtocompletethisactivity?Howlongwillittake?Howmuchwillitcost?

Thesystemverificationandsystemvalidation(aka“test”)engineerswillaskthesequestions.Projectmanagersneedtoknowtheresourcesneeded,thecost,andscheduleneedstoaccomplishthesystemverificationorsystemvalidationactivitytomakesurethey


havesufficientfundsandtimeaccountedforintheprojectbudgetandschedule.Itisbesttoaddressthesequestionsbothasyouaredefiningyourprojectscopeandasyouaredevelopingyourrequirementsbeforeyouunknowinglyspendprojectresourcesondesigningorbuildingyoursystembasedonanunverifiablerequirementorperformingansystemverificationorsystemvalidationactivitythatcannotbeaccomplishedwithintheproject’sbudgetandschedule.

Thisisaknowledge-basedprocessthatevolveswiththedesignofyoursystem.Someofthesequestionsmaynotbeabletobeansweredbeforesystemdesign.Requirementdevelopment,management,anddesignareanongoingactivities.Evenifyoucannotanswerthesequestionsbeforesomedesigneffort,youcancertainlydosoduringorafterdesignbutbeforemanufacture,code,orbuild.

3) “Whenandatwhatlevelwillyoursystemverificationandsystemvalidationactivitiesbeperformed?”

Aspartofyoursystemverificationandsystemvalidationplanningandconceptdevelopmentefforts,considertiminganddevelopa“when”strategy.Rememberintegration,systemverification,andsystemvalidationarebottoms-upprocessactivities.Whenandwhereinyourproductintegrationlifecyclewillyouverifytherequirementhasbeenmet?

Balancecomponent,subsystem,andsystemlevelverificationactivities.Youmaybetemptedtosaveverificationtimeandmoneybydoingonlysubsystemorsystemlevelverification.Thisstrategyisveryrisky;youmaynotfindcriticalproblemsuntilverylateintheproductintegrationprocessorduringsystemvalidationorcustomeracceptance.Itmaybehard(timeconsuming)totracetheproblemtoitssourceforacomplexsystem.Onceyoufindthesourceoftheproblem,itwillbemoredifficulttoresolve,especiallyifmorethanonecomponentisinvolvedandyouneedtode-integratethesystemaspartoftheanomalyresolution.Asingleproblememergingduringsystem-levelverificationorvalidationmaycancelallsavingsprojectedfromeliminatingcomponentorsubsystemverificationactivities.

Alternatively,budgetandschedulelimitsmaymaketheverificationofeverycomponent-levelrequirementcost-prohibitiveandunnecessary.Again,itisamatterofriskmitigation.Ifthecomponentlevelrequirementtracestohigher-levelrequirementsdealingwithhighpriorityobjectivesormissionorsafety,youwillwanttoverifythosecomponentsmeettheirrequiremens.Alternatively,youmaybeabletoaccepttheriskandwaittodosystemverificationatahigherlevelofthearchitecture-ifpossible.Anotherapproachistousealesscostlyandtimeconsumingmethodofsystemverificationforthelowerriskcomponentrequirements,e.g.,“Analysis”or“Demonstration”ratherthan“Test”.

4) “WhatdoIneedtodotoverifythesysteminthecontextofthenextlevel“super”systeminwhichmysystemisapartofinitsintendedoperationalenvironment?”


Failingtoverifytheintegratedsystemorfailingtoverifythesystemwithitsexternalinterfacesandinitsoperationalenvironmentcanresultinmajorembarrassmentwhenthedeliveredsystemfailstoperformasexpectedanddoesn’taccomplishitsintendedpurposeinitsoperationalenvironment(validation).Eventhougheverypart,component,orsubsystemhasbeenverified,youneedtoensuretheyworktogetherasawhole,theyworkwiththeirexternalinterfaces,andtheyworkintheintendedoperationalenvironmentbeforesystemvalidationactivitiesandbeforeyoudeliverthesystemtoyourcustomer.VerifyinginternalandexternalinterfacesoftenrequiresTestasthepreferredmethodofverification.(UsingAnalysisbysimilarityisveryrisky!)Thecostofthistestingcanbehigh,butfailuretosuccessfullycompletethistestingcanleadtomajor,andevencatastrophic,problems.

5) “Whatconstitutesproofthattherequirementhasbeenverified?”

Proofisderivedfromdocumentationandestablishmentofconfidence.Documenttheexpectationsfortheevidenceneededandlevelofconfidenceinthisevidencebeforeyouandothersarewillingto“signoff”ontheverificationdocumentationfortherequirement.Consideralsowhatevidenceisneededbeforethecustomerwillaccepttheproductfordelivery.

Failingtoaddressthesequestionsearlycanresultinunpleasantsurpriseslateinyoursystemdevelopmentlifecycle,puttingyourprojectatrisk.

Addressing system verification, system validation, certification, and acceptance in Vendor/Supplier Agreements and Statements of Work Ifanend-itemistobeoutsourcedtoavendor/supplier,therequirementsforeachend-itemaretypicallydocumentedinaSystemRequirementDocument(SRD)oraTechnicalDataPackage(TDP).AlongwiththiswillbeaStatementofWork(SOW)thatcontainsrequirementsonthevendor/supplierconcerningdesign,development,systemverification,systemvalidation,andacceptance.TheSOWcontainsprocessandworkmanshiprequirementsleviedonthevendor/supplieronthetaskstheyneedtoperformaspartofthecontracttofabricate,test,transport,andinstallaspecificend-item.TheSRDcontainsrequirementsintendedtocommunicatetothedesignteamthestakeholderneedsandexpectationsinanunambiguousandcleartechnicallanguage,whichcanbethoughofasthe“design-to”requirements.TheTDPincludesequipmentlists,partslists,drawings,andend-itemspecificationrequirementsonthesystemtobesuppliedthatcanbethoughtofasthe"build-to"requirements.Insomecases,theSRDorend-itemrequirementsmaybecontaineddirectlyintheSOW.Themethodofhowthesystemwillbeverifiedthatitmeetsalloftheserequirementsneedstobedefinedaspartofthevendor/supplieragreementprocess.

KeydeliverablesspecifiedintheSOWneedtoincludeanallocationmatrixthatshowshowthevendor/supplierallocatesyourrequirementstothepartsthevendor/supplierisresponsibleforprovidingaswellasatraceabilitymatrixshowinghowthevendor’s/supplier’sderivedrequirementstracetoyourparentrequirements.Inaddition,youneedtoaddresswhatproof


youwillneedfromthevendor/supplierastohowtheyhavemetyourrequirementsandwhatisneededforthecustomertoacceptthesystemfromthevendor/supplier.

Together,theSOWandSRDorTDPresultsinavendor/supplieragreementwhichisacontractbetweenthecustomerandvendor/supplier.Thiscontractneedstoclearlystateyourexpectationsforthesystemverificationandsystemvalidationapproachandscopeoftheeffort.Ifthesystemiscomplex,testingeverypossiblecaseorpathmaybecost-prohibitive.Whataretherisksandwheredoyouwanttospendfundsforsystemverificationandsystemvalidation?Missing,vague,oropen-endedsystemverificationandvalidationplanscanoverrunthesupplier’sbudgetorleaveyou,thecustomer,withoutconfidenceinthesystem.Addressingsystemverificationandsystemvalidationinvendor/supplieragreementsprotectsallpartiesinthecontract.

Thedocumentationthatprovidesproofthattheseactivitieshavebeensuccessfullycompletedneedstobeclearlylistedascontractdeliverables.Thisdocumentationisusedaspartofyouroverallsystemverificationandvalidationactivities.Youcanacceptthesupplier’sdocumentationasproof,orforcriticalitems,youmaywanttoperformyourownsystemverificationandsystemvalidation.Insometexts,acceptingasupplier’sdocumentationasproofthedeliveredcomponentorsystemmeetsitsrequirementsisclassifiedas“verificationbycertification”andistreatedasavalidmethodforsystemverificationandsystemvalidation.

Ifyouareavendor/supplierandyourcustomerswillbeperformingtheirownacceptanceactivities,assessyoursystemverificationandvalidationactivitiesagainsttheiracceptanceplan.Areyoumissingrequirementsthatareobvioustothecustomer?Ifyouarethecustomer,youracceptanceplanmustbecomprehensiveenoughtosatisfyyourorganizationthatthesystemwillperformintheoperationalenvironmentasadvertisedorasdescribedinyoursupplieragreement/contract.Ineithercase,bothorganizationsneedtoensuretheiracceptanceplanreflectstheactualrequirementsanddoesnotintroducepreviouslyunstatedrequirements.[Hooks2001]

Tools to help with your system verification and system validation planning

Thissectionaddressessomeofthekeytoolsthatareusedtohelpwithsystemverificationandsystemvalidationplanning.

• ProgramorProjectManagementPlan(PMP):Definestheoverallprojectactivities,criticalmilestonesandevents,andprocessidentification.OrganizationandWorkBreakdownStructure(WBS)aswellasapreliminarybudgetandschedulearedefined.Thisinformationneedstoincludeyourhigh-levelsystemverificationandsystemvalidationactivities.

• SystemsEngineeringManagementPlan(SEMP):ExpandsonthePMPfromaSystemsEngineeringperspective.TheSEMPestablishestheoverallplanforthetechnicaldevelopmentofyoursystem.TheultimateobjectiveoftheSEMPistoprovideadisciplinedframeworktomeetcost,technicalperformance,quality,andschedule


objectivesfortheprojectorprogram.ItisimportantthattheSEMPestablishtheoverallsystemverificationandsystemvalidationphilosophyfortheprojectorprogram.

• MasterIntegrationVerificationandValidation(MIVV)Plan:ExpandsandimplementsthesystemverificationandsystemvalidationphilosophyoftheprojectorprogramasdefinedinthePMPandSEMP.TheMIVVPlandefinesthedetailedprocessesandapproachthatwillbeusedforsystemverificationandsystemvalidation.TheinformationintheMIVVPlanisusedtomanageandplansystemverificationandsystemvalidationactivities.TheMIVVPlanfurtherdefinestheintegration,systemverification,andsystemvalidationschedulecontainedinthePMPandSEMP.

• SystemRequirementsDocumentorSpecification(SRD/SRS):Containsthesystem’srequirements.Theproposedverificationmethodisakeyattributeofeachrequirement,andneedstobecapturedanddocumentedwiththerequirement.Youcanalsoincludethelevel,lifecyclephase,andsuccesscriteriafortheverificationactivities.ThisinformationwillbeusedtocreatetheRequirementsVerificationMatrix.

• RequirementsVerificationMatrix(RVM):Amatrixthatlistseachrequirement,verificationmethod(Test,Inspection,Demonstration,Analysis),responsibleorganization,level(component,subsystem,system),phase(design,manufacture,integration),locationorfacility,andoftenincludesashortdescriptionofthesuccesscriteriaforsuccessfulverification.TheRVMisoftenincludedaspartoftheSRD/SRS.IftheinformationneededtocreatetheRVMisincludedaspartoftherequirementattributes,thenyourRequirementManagementTool(RMT)willbeabletogeneratetheRVMforyou.

• SystemIntegration,VerificationandValidation(SIVV)Plan:Implementstheproject’sMIVVPlansystemverificationandvalidationactivitiesforagivencomponent,subsystem,orsystemasdocumentedintheSRD/SRSandassociatedRVM.TheSIVVPlancontainsadetailedidentificationofthetaskstobeperformedaspartofverificationandvalidationofthatsystem.Resourcesincludingtestequipment,facilities,andpersonnelareaddressed.AdetailedscheduleconsistentwiththeIntegration,Verification,andValidationScheduledefinedintheMIVVPlanisincluded.

Productstohelpdevelopandimplementthisplanarediscussedbelow.

• VerificationRequirementDefinitionSheet(VRDS):ForeachrequirementintheSRD/SRS/RVM,theVRDStransformstheRVM“what’s”into“hows”–“Howdoyouplanonverifyingtherequirementslistedin,andmeetthesuccesscriteriadefinedin,theRVM?”TheVRDSsdocumentthedetailsofthespecificverificationactivitiesidentifiedintheRVM.AnexampleofandinstructionsforcompletingaVRDSisincludedinAppendixA.

TheVRDScapturesalloftheverificationrequirementsandothersupportinginformationforaspecificrequirementverification.ForeachSystem,thesetofVRDSsdocumenttheverificationrequirementsfromtheperspectiveoftheverificationactivitiesthatare


performedtoensurethesystemmeetsitsrequirements.ThesetofVRDSarecontainedinaVerificationRequirementDocument(VRD).

(Note:SomeprojectsdocumentverificationrequirementsinaseparatesectionoftheSRD/SRS.Idonotadvocatethisapproach.Iadvocateaseparatedocument,VerificationRequirementsDocument(VRD)wheretheverificationrequirementsaredocumentedviatheVRDSsdefinedherein.TheconceptofaseparateVRDcontainingVRDSsforeachrequirementwasusedbyNASA’sConstellationProgramAres1Xproject.Seereference11.)UsingaseparateVRDtodocumentthisinformationallowsyoutobaselineyourrequirementsetandminimizeschangestoyourSRDastheinformationintheVRDmatures.

• TaskDefinitionSheets(TDS):Asanaidtoverificationactivityplanning,eachsystemleaddevelopsalistoftaskstobeperformedtoaccomplishtheneededsystemverificationactivitiesdefinedintheVRDS.Foreachofthosetasks,aTaskDefinitionSheet(TDS)isdeveloped.TheTDSscontaintheinformationneededtodevelopproceduresandanintegratedscheduleforcompletingallthedefinedtasksinvolvedinverification,validation,acceptance,certification,andreadiness.OneTDScanaddressmultiple,relatedVRDSs.WhichVRDSswillbelistedinthetaskobjectivessectionoftheTDS.ThereshouldbeoneTDSforeachverificationorvalidationtaskidentifiedintheSIVVPlan.AnexampleofandinstructionsforcompletingaTDSisincludedinAppendixB.

• TaskPerformanceSheet)(TPS):TheTPSorotherapprovedWorkAuthorizationDocument(WAD)containstheactualprocedureusedtoperformthesystemverificationorvalidationactivitiesdefinedintheTDS.Thisisthedocumentactuallyusedtoaccomplishthesystemverificationorsystemvalidationactivityanddocumenttheresults.TheTPSissignedoffbytheappropriatepersonnelwhoareidentifiedintheapplicableTDS(s).ThecompletedTPScontainsthedatathatrepresentstheproofthatthesystemmeetsarequirementorsetofrequirementsasdefinedintheVRDSsorvalidatedasdefinedintheMIVVPlanandSIVVPlan.

AsystemwillhavemultipleTPSs.TheremaybeoneTPSforverificationbyInspectionwhileanend-itemisatthevendorfacility.CompletionofthatTPSisthenoneoftheitemsrequiredpriortoend-itemacceptanceandshipmenttothecustomer.VerificationbyAnalysismaybeperformedviaasingleTPS–aseparateTPSforeachverificationbyAnalysis.VerificationorvalidationbyTestmaygrouplikerequirementstogetherinthesameTPS,i.e.,leaktestsformultiplejoints/welds.

TPSscalloutchecklistsandproceduresandcontainspecificactionstobeperformed.Fromasystemverificationorsystemvalidationactivitystandpoint,TPSsarepreparedbasedontheinformationcontainedintheSIVVPlan,VRDSs,andTDSthattheTPSisbeingpreparedtosatisfy.AllTPSlineitemsinvolvingasystemverificationorsystemvalidationactivityforaspecificrequirementmustincludeamandatoryinspectionpoint(MIP)thatincludesaQualityAssurance/QualityEngineer(QA/QE)stamp.


Wrap up and Parting Thoughts Systemverificationandsystemvalidationarealargepartofsystemdevelopment.Secondonlytorework,aproject’sbiggestcostsareinvolvedinsystemverificationandsystemvalidation.Failingtoaddresssystemverificationandsystemvalidationearlyintheprojectlifecyclecanresultinunpleasantsurprisestowardstheendoftheproductlifecyclewhensystemverificationandsystemvalidationactivitiesbegin,puttingyourprojectathighriskofcostoverrunsandscheduleslips.

Westartedwithadiscussiononthetermsverificationandvalidationandmadethepointthatthesetermsareambiguousunlessamodifierisused(requirement,design,orsystem)thatclearlycommunicatesthecontextinwhichthetermrefers.

Theemphasisoftherestofthispaperwastopresentbestpracticesandtoolsandprovenmethodsprojectteamscanusetoaddressplanningforsystemverificationandsystemvalidationfromthebeginningofaprojectandavoidtherisksofnotdoingso.

Thecasewasmadeconcerningtheimportanceofplanningforsystemverificationandsystemvalidationstartingatthebeginningofthesystemlifecyclefromariskmitigationstandpointandtheimportanceofprojectmanagementensuringthatrequirementdevelopment,management,anddesignprocessesaddresssystemverificationandsystemvalidationactivityrisks,cost,andschedule.

Inconclusion,avoidtherisksassociatedwithfailingtoadequatelyplanforsystemverificationandsystemvalidationthroughouttheprojectlifecyclebyaddressingthefollowingbestpractices:

• Duringscopedefinition,involvethosepersonnelassociatedwithsystemverificationandsystemvalidationwiththedevelopmentofasetofscenariosoroperationalconceptsusedtosuccessfullycompleteneededsystemverificationandsystemvalidationactivities.ThisknowledgeisdocumentedinthePMP,SEMP,andMIVVPlan.

• Addresssystemverificationandsystemvalidationearlyinthesystemdevelopmentlifecycleasariskmitigationactivityfromaprojectmanagementperspective.

• Ensuringthatyoubeginwithverifiablerequirementsisthekeytoreducingrisklaterinyoursystemdevelopmentlifecycle.Whiledevelopingrequirements,alwaysthinkaheadtosystemverification.Thisimproveseachrequirement’squalityandreducestheriskofunpleasantsurpriseslaterinthesystemdevelopmentlifecycle.

• Usethetoolsdiscussedinthepaper(PIVVplan,RVM,VRDS,TDS,andTPS)tofacilitateplanningforandcompletingsystemverificationandsystemvalidationactivities.

• Ensureprojectmanagementisinvolvedinkeydecisionsassociatedwithbalancingtherisksagainstcostandschedulewhenselectingsystemverificationandsystemvalidationmethodsandthendeterminingatwhatlevel,phase,facilityorlocation,thesystemwillbeverifiedthatitmeetsitsrequirementsitwasdesignedto.


Addressingsystemverificationandsystemvalidationearlyiskeytodeliveringawinningproduct–thefirsttime!!!


References and Further Reading 1. Hooks, I. F. and Farry, K. A., Customer-Centered Products: creating successful products

through smart requirements management; AMACOM Books, NY, NY, 2001. 2. Requirements Experts, Requirements Development and Management, Seminar Workbook.

2012. 3. Larson, Kirkpatrick, Sellers, Thomas, and Verma, Applied Space Systems Engineering,

McGraw-Hill, 2009. 4. Engel, A., Verification, Validation, and Testing of Engineered Systems, Wiley, 2010 5. INCOSE, Systems Engineering Handbook - a guide for system life cycle processes and

activities, Version 3.2, INCOSE-TP-2003-002-03.1, January 2010, ed, Cecilia Haskins. 6. ISO/IEC 15288, System Engineering-System Life Cycle Processes, October 2002. 7. NASA, System Engineering Handbook, SP-2007-6105, Rev. 1, December 2007.

<http://education.ksc.nasa.gov/esmdspacegrant/Documents/NASA%20SP-2007-6105%20Rev%201%20Final%2031Dec2007.pdf>.

8. NASA, Systems Engineering Processes and Requirements, NPR 7123.1A, March 2007 <http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D>.

9. NASA, Space Flight Program and Project Management Requirements, NPR 7120.5D, March 2007. <http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D>.

10. NASA, Space Flight Program and Project Management Handbook, NPR 7120.5, February 2010. < http://www.nasa.gov/pdf/423715main_NPR_7120-5_HB_FINAL-02-25-10.pdf >.

11. NASA, Ares 1-X System Verification Requirements Document (VRD0 for Ares 1-x Flight Test Vehicle (FTV), AI1-SYS-VRD, version 2.02, December 9, 2008

12. Software Engineering Institute, CMMI for Development – Improving processes for better products, Version 1.3, CMMI Product Team, Carnegie Mellon, August 2006.

13. Wheatcraft, L. S., and Hooks, I. F., Scope Magic, 2001. <http://www.complianceautomation.com>.

14. Wheatcraft, L. S., The Importance Of Scope Definition Prior to Developing Space System Requirements. INCOSE INSIGHT, Vol. 4 Issue 4, January 2002. <http://www.complianceautomation.com>.

15. Wheatcraft, L. S., Delivering Quality Products That Meet Customer Expectations. Published in CrossTalk, The Journal of Defense Software Engineering, January 2003, Vol. 16 No. 1. <http://www.complianceautomation.com>.

16. Wheatcraft, L. S., Developing Requirements for Technology-Driven Products. Presented at INCOSE 2005, July 2005. <http://www.complianceautomation.com>.

17. Wheatcraft, L. S., Everything You Wanted to Know About Interfaces – But Were Afraid to Ask, Presented at INCOSE, July 2010.

18. Wheatcraft, L. S., Triple Your Chances of Project Success - Risk and Requirements. Presented at INCOSE 2011, June 2011 and NASA PM Challenge 2012, February 2012.


BIOGRAPHY Lou Wheatcraft has over 40 years experience in the aerospace industry, including 22 years in the United States Air Force. Over the last 15 years, Lou has worked for Compliance Automation (dba Requirements Experts), where he has conducted over 160 seminars on requirement development and management for NASA, Department of Defense (DoD), and industry. Lou has had articles published in the International Council of Systems Engineering (INCOSE) INSIGHT magazine and in DoD’s magazine, CrossTalk.

Lou has made presentations at NASA’s PM Challenge, INCOSE’s International Symposium, and at the local Project Management Institute (PMI) and INCOSE Chapter Meetings. Lou has a BS degree in Electrical Engineering, an MA degree in Computer Information Systems, an MS degree in Environmental Management, and has completed the course work for an MS degree in Studies of the Future.

Lou is a member of INCOSE, co-chair of the INCOSE Requirements Working Group, a member of PMI, the Software Engineering Institute, the World Futures Society, and the National Honor Society of Pi Alpha Alpha. Lou is the recipient of NASA’s Silver Snoopy Award and Public Service Medal and was nominated for the Rotary Stellar Award for his significant contributions to the Nation’s Space Program.

Lou also maintains a blog on requirement development and management topics. You can access his blog articles at http://www.reqexperts.com/blog

If you have further questions, please feel free to contact the author at: [email protected]

Appendix A: Verification Requirement Definition Sheet (VRDS)


[Program or Project Name] [System/Subsystem/Component Name]

Verification Requirement Definition Sheet [Project Logo]

[Requirement Number]

Verification Requirement Number: VR [Rqmt Number] Parent Requirement(s): [Rqmt Number(s)] RequirementText:

System Verification Requirements Verification method (VM): [ ] Test [ ] Demonstration [ ] Inspection [ ] Analysis Descriptionofverificationactivitiestobeperformed:The[VM]shallconsistof…………

SuccessCriterion:Verificationshallbeconsideredsuccessfulwhenthe[VM]shows…...

Rationale:[ReasonfortheVMtobeused]:

System Verification Implementation

Lifecycle Phase: System: [ ] Manufacturing [ ] Build up Integration: [ ] Sys-Sys Interfaces [ ] System Acceptance

Applicable documents: TPS xxxxxxxxx; Drawing: Num: xxxxxxxxxxx Rev: x Date: xx/xx/xxxx Completed children requirement VRDSs: xxxxxxxxxxx

Nonconformance history: N/A

Closure data/documentation required: Signed off TPS and data. Electronic/written confirmation that the requirement verification activity has been successfully completed and the defined success criteria has been achieved. EventsprecedingtheSystemVerificationActivity:xxxxxxxxxEstimatedDurationoftheSystemVerificationActivity:[TBS]

Closure Of the VR [Rqmt Number]

Date Closed: mm/dd/yyyy [ ] Safety Critical (Y/N) xxxx Quality Engineer:

[Name]

xxxx System PM:

[TBS]

xxxx Chief Engineer:

[Name]

Customer Systems Engineer

[Name] Form:VerificationRequirementDefinitionSheet



InstructionsforfillingoutaVRDSThissectionprovidestheinformationneededtofillouttheVRDSforms.Theseformsareusedtospecifythesystemverificationrequirementsastailoredforthesystemverificationactivitiestoprovethesystemmeetstherequirementsthatdrovethesystem’sdesign.• VRDSTitleThetitlecontainsthenameoftheComponent/Subsystem/Systemtherequirementconcerns• RequirementtheSystemVerificationisaddressingThenumberingofeachVRDSiscomprisedoftherequirementnumbercontainedinSystemSpecwithaVR-insertedatthebeginningofthenumber.Includetherequirementtextalongwithanynotesorrationale.• VerificationMethod

• ThemethodisselectedfromDemonstration,Analysis,Inspection,andTestasstatedintheRVMs.

• TheVerificationMethodidentifiesthemethodtoobtainthemeasureofthesuccesscriterion.Supportingactivitiesmaybecalledout(e.g.,“analysisandtest,”incaseswherethesuccesscriterionisaMonteCarlosimulationsupportedbyinputfromtestresults).(Generally,mostifnotallofthefouractivitieswillbeinherentinanyverificationmethod,soitispreferredtofocusonthefinalmethodthatgeneratesthemeasureofthesuccesscriterion,evenwhenmultipleactivitiesareinvolved.)(Rememberthatthemethodsandactivities,maysharethesamenames,buttheconceptsaretotallydifferent.)

• VerificationRequirementsTheverificationrequirementsarerequirementsleviedontheorganizationresponsibleforperformingtheverification.Verificationrequirementsaddresstheactivitiesthatmustbeperformedtoobtaintheinformation(proof)neededtoverifythesystemmeetstherequirementaswellasastatementofthesuccesscriterion.TheserequirementswillbeimplementedviaaTPSorotherformofWorkAuthorizationDocument(WAD).

• Descriptionofverificationactivitiestobeperformed• Oneormore“shall”statements(requirements)statingthemeasure(a

parameter,quantity,orcondition)oftheverificationsuccesscriterionandhowthismeasureisobtained.

• Thispartstateshow(whichactivities)willbeusedtoobtain(generally,notcallingoutaspecifictool)valuesorstatesforthismeasure.

• Thispartstatesrequiredassumptionsandconditionsfortheverification.• Necessaryconditions,assumptions,andotherrequiredstatesorinputsfor

obtainingthemeasureofthesuccesscriterionarestatedclearlyin“shall”statements(requirements).

• Anynecessaryconstraints,certifications,orrequiredtrainingforverificationexecutionpersonnelareidentifiedin“shall”statements(requirements).



• SuccessCriterion• Includeasingle“shall”statement(requirement)statingthesuccesscriterion.This

successcriterionshouldbethesameasstatedintheVRMs.• Thesuccesscriterionincludesthevalueorstateagainstwhichthesuccess

measureobtainedfromtheverificationprocessactivitiesiscomparedtoindicatesuccessfulverification(e.g.;measuredweightshallbelessthan133pounds13ounces;ortelemetryreceivedinthecontrolroom).

• Thesuccesscriterionneedstobemeasurableorobservablebysomeprocess.• Asimple“yes/no”answerisallthatisrequiredtodetermineifthesuccess

criterionhasbeenmet.• Themeasureforthesuccesscriterionisstatedforthe“asbuilt”

implementationofthesystemrequirement.• Nospecialskillsorsubjectmatterexpertisearerequiredtomakethedecision

whetherthesuccesscriterionhasbeensatisfied.• Rationale

Therationaleisan“attribute”oftheverificationrequirements.OnerationaleiswrittenontheVRDSformforeachSystemRequirement,butitappliestoalloftheverificationrequirementsspecifiedonthesheet.Thefocusoftherationalestatementincludesanexplanationandjustificationfor“why”thespecificverificationmethod,successcriterion,measure,constraints,conditions,andassumptionshavebeenselected.Therationaleisespeciallyimportantwhenaspecificverificationmethod“choice”isnottheobviousorexpectedone.Forexample,aspecificsystemrequirementmightbeonethatisuniversallyconsideredtorequirerigorousverificationtesting.However,itmightbeverifiedbyalessstringentmethod(e.g.analysisorinspection)iftherisksofnottestinghavebeenfullyconsideredandacceptedbytheproject.

• VerificationImplementationTheVerificationImplementationportionoftheVRDScontainsthefollowingtypesofinformation:

• LifecyclePhaseTheVerificationPhasestatesinwhichsystemintegrationlifecyclephasethesystemwillbeverifiedtohavemettherequirement.

• ApplicableDocumentsAnydocumentsdealingwiththesystemverificationforthisrequirement.Theseincludethespecificprocedures(TPSorotherWAD)usedtodotheverificationinthecaseofatestordemonstration.Thesealsoincludelowerlevelchildrenrequirementswhoseverificationactivitiesmustbecompletedandsignedoffbeforethesystemverificationactivitiesfortherequirementcanbeperformed.



• NonconformancehistoryNonconformancehistoryincludesthehistoryofpreviousactivitieswhentheabilityoftheSystemtomeetthisrequirementwasnotabletobeproven.

• Closuredata/DocumentationRequiredStatewhatclosuredataand/ordocumentationisrequiredtoprovethesystemhasmeetthisrequirement.

• Event(s)precedingVerificationActivityStateanyeventsthatmustbecompletedandindicatethesystemstateorconfigurationthatmustbeachievedpriortobeingabletoperformsystemverification.Thisincludesverificationthatchildrenrequirementshavebeenmet,installationofthesysteminthefacility,assembly,inspections,checkouts,etc.

• EstimatedDurationoftheVerificationActivityProvideanestimateofthetimeneededtocompletetheactivitiesassociatedwithsystemverificationforthisrequirement.Incaseswhereoneactivitywillinvolvethesystemverificationformultiplerequirements,statethetimetocompletethatactivity.

• ClosureoftheVerificationRequirementThissectioniscompletedoncethesystemverificationactivityhasbeencompletedandtheclosuredataandclosuredocumentationhasbeencompletedanddocumentedinthissectionoftheform.Enterthedateclosed(datetheformissignedbyallfourparties)andindicatewhetherthisverificationincludesasafetycriticalrequirement.EachofthedefinedpartiesneedtosignofftheVRDSindicatingtheiracceptanceoftheverificationactivityresults(proofthesuccesscriteriahasbeenmet)andtheirconfirmationthatsystemmeetsthestatedrequirement.

Appendix B: Task Definition Sheet (TDS)


[Program or Project Name] [System/subsystem/component Name]

Task Definition Sheet [Project Logo]

Task (Inspection/Test/Checkout) Title:

Task Number: Project [System 3 Ltr designation]-XXX TaskDescription:(Purposeofthistask–whyisitneeded?Includeadescriptionoftheequipment/components/assemblies/subsystem/systemsinvolvedinthetask.)

TaskObjectives:(Expectedoutcomesasaresultofdoingthistask.IncludeVRDSsthatareaddressedbythistask.)

Hazardous operation? [ ] Y [ ] N Hazard Analysis Needed? [ ] Y [ ] N

Type of Task: [ ] Mechanical [ ] Integration [ ] Interface [ ] Power [ ] Command/Data [ ] Pressure/Leak [ ] Other: Task Methodology: [ ] Test [ ] Demonstration [ ] Inspection [ ] Analysis LifecyclePhase:System:[]Manufacturing[]BuildupIntegration:[]Sys-SysInterfaces[]SystemAcceptance

Schedule:(Dateswhentaskwillbeperformed.)Start:End:Duration:PredecessorTask(s):(Tasks/installationsthatmustbecompletebeforethistaskcanbeperformed.)

Successor/ParentTask(s):(Tasks/installationsthatrequirethistaskbecompletedbeforetheycanbeperformed.Isthistaskpartofalarger(parent)task?Whichtask?)

Constraints:(Anythingthatlimitstheperformanceofthetask.Alsoincludeimpact/constraintsonotheractivitiesduringtheperformanceofthistask.e.g.,nootherworkinthefacilitywhilethistaskisbeingperformed.)

Codes/Standards: (list any codes or standards this task must be compliant with):

Resources Needed: (utilities (steam, power, LN2, air), other Systems, Portable Equipment, Leak Test Unit, etc. during the performance of this task.)

Documentation: Checklists/Procedures: [ ] Existing [ ] To be Developed [ ] Use Vendor Procedure

Organizations/Personnel: (who will be involved in the performance of this task?) [ ] Project Engineer, [ ] Test Director, [ ] Supplier/Vendor, [ ] Safety, [ ] Quality, Technician, [ ] Other:

Training/Certifications: (State any training/certifications required by personnel involved in the task.)

Form:TaskDefinitionSheet



InstructionsforfillingoutaTDSThissectionprovidestheinformationneededtocompletetheTDSformsusedtospecifythetoplevelinformationneededtodevelopanintegratedverification,acceptance,andreadinessschedule.• Task(Inspection/Test/Checkout)Title:Includetitlethatclearlycommunicatesthenature

ofthetask.• TaskNumber:Includeauniqueidentificationnumberforthetask.Sequentiallynumber

eachTDSforyoursystem.• TaskDescription:Clearlystatethepurpose/reasonforthistask–whyisitneeded?Include

adescriptionoftheequipment/components/assemblies/subsystem/systemsinvolvedinthetask.Whatdoesthetaskconsistof?(startthesentencewithaactiveverb-test,checkout,inspect,etc.

• TaskObjectives:Listtheexpectedoutcomesfromtheperformanceofthistask.Ifverificationisanexpectedoutcome,listtherequirementsthatwillbeverifiedasaresultofdoingthistask.(Note:UpdateanyapplicableVRDSswiththetasknumberofthisTDStoprovidebi-directionaltraceabilitybetweentheverificationrequirementsandthetaskthatwillresultinverification.)

• Hazardousoperation?Checktheappropriateblocktoindicateahazardousoperationandwhetherahazardanalysisisneeded.

• TypeofTask:Checktheappropriateblocktoindicatethetypeoftask(Mechanical,Integration,Interface,Power,Command/Data,Pressure/Leak.Ifnoneoftheseisappropriate,writeinthetypeoftaskinthe"Other"field.

• TaskMethodology:Checktheboxthatbestindicatestheoverallmethodologytobeusedtocompletethistask.(Test,Demonstration,Inspection,Analysis).

• LifecyclePhase:Indicatethelifecyclestagewhereyouplantocompletethistask.• Schedule:Indicatethedateswhenyouplantoperformthetaskandexpectedoverall

duration(hours/days).ThesedatesmustbeconsistentwiththeotheractivitiesincludedintheProject’sMasterSchedule.

• PredecessorTask(s):Indicatewhichprerequisitetasksmustbecompletedbeforethistaskcanbeperformed.

• Successor/ParentTask(s):Indicatewhichtaskscannotbecompletedbeforethistaskisperformed.Indicatewhetherornotthistaskpartofalarger(parent)task?Ifso,indicatewhichtask.

• Constraints:Listanyconstraintsontheperformanceofthistask.Includeanythingthatlimitstheperformanceofthetask.Alsoincludeimpacts/constraintsonotheractivitiesduringtheperformanceofthistask.e.g.,nootherworkintestareawhilethistaskisbeingperformed.

• Codes/Standards:Listanycodesorstandardswithwhichthistaskmustbecompliant.• ResourcesNeeded:Listanyresourcesneededtoperformthistask(utilities:steam,power,

GN2,air)aswellasanyotherSystems,PortableEquipment,LeakTestUnit,etc.thatareneededduringtheperformanceofthistask.



• Documentation:Checklists/Procedures:Indicatewhetherthereisarequirementtodevelopnewchecklistsand/orproceduresbeforedoingthistaskorwhetherexistingchecklistsorprocedurescanbeused.Indicatewhetheryouareplanningtouseanyvendorprocedurestocompletethistask.

• Organizations/Personnel:Indicatewhichkeyorganizationsneedtobeinvolvedintheplanningandperformanceofthistask.(ProjectEngineer,TestDirector,Supplier/Vendor,Safety,Quality,Technician,TestOperations,Others?

• Training/Certifications:Stateanytraining/certificationsrequiredbypersonnelinvolvedinthetask.