Hypothesis Testing - .Non‐Commercial Share Alike License, which permits unrestricted use, distribution,

Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page1

HypothesisTesting

LearningObjectives

Aftercompletingthismodule,thestudentwillbeableto

carryoutastatisticaltestofsignificance calculatetheacceptanceandrejectionregion calculateandinterpretthepvalueofastatistical

test

calculateandinterprettype1andtype2errors calculatethepowerofatest

KnowledgeandSkills

Concepts:nullhypothesis,alternative,teststatistic,rejectionregion,acceptanceregion,pvalue,significancelevel,type1error,type2error,falsepositive,falsenegative,powerofatest

Resamplingmethod Fishersexacttest

Prerequisites

binomialdistribution hypergeometricdistribution Normaldistribution Sampleaverage Samplestandarddeviation macrosinExcel

Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page2

PrologueTheproblemofdecisionmakingisubiquitous.Almostdaily,youcanreadinthenewsaboutstudiesthatleadtorecommendationsbasedonstatisticalevidence.TheU.S.DepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality(http://www.ahrq.gov/)provideshealthcarerecommendations,forinstance,throughitsU.S.PreventiveServicesTaskForce(http://www.ahrq.gov/clinic/uspstfix.htm),anindependentpanelofexpertsinprimarycareandprevention,whichreviewsresearchresultsanddevelopsrecommendations.Theserecommendationsarebasedonanalysesoftensorhundredsofclinicalstudies,andrecommendationsmaychangeasnewevidenceaccumulatesovertime.

Frequently,clinicalstudiesarephrasedintermsofhypothesistesting.Forinstance,ifanewtreatmentforadiseaseisdeveloped,wemightwishtoknowwhetheritperformsbetterthanthecurrenttreatment.Wesetupaclinicaltrialwherepatientsarerandomlyassignedtooneortheothertreatment.Wethencomparethenumberofsuccessfultreatmentsineachgroup.Letsassumethatthetwogroupshavethesamenumberofpatients.Inordertoconcludethatthenewtreatmentisbetterthanthecurrenttreatment,wewouldneedtodemonstratethatthenumberofsuccessfultreatmentsinthenewtreatmentgroupislargerthanthenumberofsuccessfultreatmentsinthecurrenttreatmentgroup.Thequestionishowmuchlargerthenumberofsuccessfultreatmentsinthenewtreatmentgroupwouldneedtobetoconvinceotherinvestigatorsthatthenewtreatmentisindeedbetter.Thesekindsofquestionscanbeansweredwithintheframeworkofhypothesistesting.

InclassActivity1

Assumethecurrenttreatmentforadiseaseissuccessfulin30%ofallcases.Anewtreatmentisbeingdevelopedandapreliminaryclinicaltrialshowedthat5outof10patientsweresuccessfullytreated.Canyouconcludethatthenewtreatmentismoresuccessful?

Ifthenewtreatmentwasnotbetterthanthecurrenttreatment,wewouldhypothesizethatthenewtreatmenthasprobability0.3ofbeingsuccessful.Alternatively,ifthenewtreatmentisbetterthanthecurrenttreatment,wewouldhypothesizethatthenewtreatmenthasprobabilitygreaterthan0.3ofbeingsuccessful.

Ifthenewtreatmenthasthesamelikelihoodofsuccessthanthecurrenttreatment,namelyprobability0.3,thenthenumberofpatientsinthesmallclinicaltrialwhoaretreatedsuccessfullyunderthenewtreatmentisbinomiallydistributedwith10trialsandsuccessprobability0.3.ThefollowingtablewascreatedinEXCELusingtheBINOMDISTfunctionandshowsthisprobabilitydistribution:

Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page3

x P(X=x)

0 0.02821 0.12112 0.23353 0.26684 0.20015 0.10296 0.03687 0.00908 0.00149 0.0001

10 0.0000

Weseethattheprobabilityoffiveormoresuccesseswhenthesuccessprobabilityis0.3is

0.1029 0.0368 0.0090 0.0014 0.0000 0.1502+ + + + =

Thus,itisnotunlikelytosee5(ormore)outof10patientsrecoverwhenthesuccessprobabilityofrecoveryis0.3.Weconcludethatthereisnotenoughevidencetoconcludethatthenewtreatmentisbetter.

Discussinyourgroupthefollowingquestions:

1. Whydidweadduptheprobabilitiesintheaboveexample?2. Wouldyoubeabletoconcludedefinitivelyfromthisstudythatthenewtreatmentisntanybetter?3. Whatwouldbeyournextstepindeterminingwhetherthenewtreatmentisbetter?

Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page4

InclassActivity2

Supposeyouhaveacoininyourpocket.Youwanttodecidewhetherthecoinisfairorbiased.Youhypothesizethatthecoinisfair.Totestthishypothesis,youtossthecoin30times.Thenumberofheadsisbinomiallydistributedwiththenumberoftrialsbeing30andtheprobabilityofheads(success)being0.5.Belowisthehistogramoftheprobabilitydistribution.

Supposetheexperimentresultedin18headsand12tails.Discussthefollowingquestionsinyourgroup:

1. Whatcanyousayaboutthecoin?Isitafaircoinorabiasedcoin?2. Whatwouldyourconclusionbeiftheexperimentresultedin24headsand6tails?3. Whatcriteriadidyouusetomakethedecisionineachofthetwocases?4. Canyoubesurethatyourdecisioniscorrect?

Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page5

SomeTheoryInbothInclassActivities,youhadtomakeadecisionbetweentwoalternatives.Inthefirstcase,youneededtodecidewhetherthenewtreatmentwasbetterthanthecurrenttreatment.Inthesecondcase,youneededtodecidewhetherthecoinwasfairorbiased.Inbothcases,youreliedonaprobabilitymodel,andyoubasedyourdecisiononhowlikelytheoutcomeoftheexperimentwascomparedtotheexpectationofthemodel.Inbothcases,therewasalsothepossibilitythatyouarrivedatthewrongdecision.

Inthefollowing,wewilldiscussthebasicelementsofhypothesistesting.Wewillusetheexampleofthefaircoinversusthebiasedcoinbecauseofitssimplicity.Onehypothesisisthatthecoinisfair,thatis,thattheprobabilityofheadsis0.5.Thealternativehypothesisisthatthecoinisbiased,thatis,theprobabilityofheadsisdifferentfrom0.5.Webaseourdecisionofwhetherornotthecoinisfaironcomparingtheresultofourexperimenttowhatweexpectbasedonaprobabilisticmodel.Namely,ifthefractionofheadsintheexperimentiscloseto1/2,theexperimentprovidesevidenceforthecoinbeingfair;ifthefractionofheadsiseitherloworhigh,theexperimentprovidesevidenceforthecoinbeingbiased.

Thehypothesisthecoinisfairiscalledthenullhypothesisandisdenotedby 0H .Thealternativethe

coinisbiasedisdenotedby 1H .(Wewillsaymoreaboutwhichofthetwohypothesesisthenull

hypothesisandwhichisthealternativelater.)Wesummarizethisas

=

0

1

: 0.5

: 0.5

H p

H p

Wedesignedanexperimentinwhichwetossedthecointhirtytimes.Thedatacollectedintheexperimentprovidedevidencefororagainstthenullhypothesis.Thedatainourexperimentwerethesequenceofheadsandtailsinthethirtytrials.Thedatasuggestthatwecancalculateasinglenumber,namelythenumberofheads,whichwecancompareagainstwhatwewouldexpectunderthenullhypothesis.Thissinglenumberiscalledtheteststatistic.Aprobabilisticmodelfortossingafaircoinallowsustocalculatetheprobabilitydistributionoftheteststatistic.Namely,underthenullhypothesis,thenumberofheadsisbinomiallydistributedwith30trialsandsuccessprobability = 0.5p .Inthe

experiment,weobserved18heads.Howlikelyisitthatweobserve18ormoreheads?IfXdenotesthe

numberofheads,weareaskingfor ( 18)P X ,whichcanbecalculatedbyaddinguptheprobabilitiesof

theevents{ } { } { }= = =18 , 19 ,... 30X X X .Refertothespreadsheet(tabFairCoin)toverifythat

=( 18) 0.1808P X

Citation:Neuhauser,C.HypothesisTesting.Created:November29,2009Revisions:Copyright:2009Neuhauser.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionNonCommercialShareAlikeLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,andallowsotherstotranslate,makeremixes,andproducenewstoriesbasedonthiswork,providedtheoriginalauthorandsourcearecreditedandthenewworkwillcarrythesamelicense.Funding:ThisworkwaspartiallysupportedbyaHHMIProfessorsgrantfromtheHowardHughesMedicalInstitute. Page6

Sincethealternativeistwosided