How study pre-registration can benefit neuroscience

Pia RotshteinSchool of Psychology, University of Birmingham

P.Rotshtein@bham.ac.uk

1

AcknowledgmentCortex RR editorial team

2

ChristopherChambersCardiffUniversity

ZoltanDienesUniversityofSussex

RobMcIntoshUniversityofEdinburgh

Talk Overview

• Problemswitheverydayresearchpracticesinpsychologyandcognitiveneuroscience

• Howstudypre-registrationcanhelp,includingdetailsoftheRegisteredReportsformatandexamples

• Whatcountasanhypothesisinaneuroimgaingstudy?

• ResponsetoFAQsandcriticisms

QUESTIONS

Intheinterestsofdoinggoodscience,whichpartofaresearchstudyshouldbebeyond yourcontrol?

Intheinterestsofadvancingyourcareer,whichpartofaresearchstudyismostimportantforpublishingin‘topjournals’?

Theresults

Theresults

Couldthisbewhyaliensflyrightby?

What’sbestforscience

Transparentandhighqualityresearch,

regardlessofoutcome

What’sbestforscientists

Producingalotof“goodresults”

Sciencehasanincentiveproblem

seeNosek,Spies&Motyl(2012).PerspectivesonPsychologicalScience,7(6):615–631

Whatisa“goodresult”?

“novel”“striking”“clear”“…foundasignificanteffect”“worked”“resultsshowconvincingly…”“majoradvance”“definitive”“beautiful”“breakthrough”“highimpact”

What happens when researchers are pressured to get “good results”?

Publicationbias– suppressionofnegativeorcomplexfindings

©DarioBattisti

Significancechasing– “p-hacking”,selectivereporting

HARKing – hypothesizingafterresultsareknown

Lowstatisticalpower–quantityofpapersoverquality

Lackofreplication – seenasboring,lackinginintellectualprowess

Lackofdatasharing– notime,toohard,noincentive

Generate and specify hypotheses

Design study

Collect dataAnalyse data & test hypotheses

Interpret data

Publish or conduct next experiment

Hypothetico-deductivescientificmethod

Generate and specify hypotheses

Design study

Collect dataAnalyze data & test hypotheses

Interpret data

Publish or conduct next experiment

Hypothetico-deductivescientificmethod

Publication biasLack of data sharing

Low statistical power

Significance chasing

Lack of replication

Inpsychology:1of100papersMakel etal(2012)

~50%chancetodetectmediumeffectsCohen(1962);Sedlmeier andGigerenzer (1989);BezeauandGraves(2001)

~50-100%prevalenceJohnetal(2012)

~50-90%prevalenceJohnetal(2012)Kerr(1998)

~92%positiveFanelli(2010)

~70%failureWicherts etal(2006)

Cyril Pernet

Jeanette Mumford

Whyisthishappening?

Becauseweplacetoomuchimportanceontheresults ofexperimentsandnotenoughontheprocessesthatproducethem

Resultsmakescienceexcitingbutjudgingthequalityofscience(andscientists)accordingtotheresultscondemnsustobeinga“soft”science

Can we fix this? Yes

Philosophy:Whatgiveshypothesis-testingitsscientificvalueis:• theQUESTIONitasks• theQUALITYofthemethodituses• nottheRESULTitproduces

Ifweacceptthisphilosophytheneditorialdecisionsatjournalsshouldbeblind toresults

RegisteredReports

FourcentralaspectsoftheRegisteredReports model:

• Partofthepeerreviewprocesstakesplacebeforeexperimentsareconducted

• Passingthisstageofreviewvirtuallyguaranteespublication

• Originalstudiesandhigh-valuereplicationsarewelcome

• Researchersdecidehypotheses,experimentalprocedures,andmainanalysesbefore datacollection

AuthorssubmitSTAGE1manuscriptwithIntroduction,ProposedMethods&

Analyses,andPilotData(ifapplicable)

Stage1peerreview

Ifreviewsarepositivethenjournaloffers in-principleacceptance(IPA),

regardlessofstudyoutcome(protocolnotpublishedyet)

Howitworks

Arethehypotheseswellfounded?

Arethemethodsandproposedanalysesfeasibleandsufficientlydetailed?

Isthestudywellpowered?(≥90%)

Havetheauthorsincludedsufficientpositivecontrolstoconfirmthatthestudywillprovideafairtest?

Editorialtriagereview

Total submission (44): 70% reject & invite to resubmit

1/18 rejected after stage 1Theoretically not sounds10 awarded IPA

Howitworks

Stage2peerreview Didtheauthorsfollowtheapprovedprotocol?

Didpositivecontrolssucceed?

Aretheconclusionsjustifiedbythedata?Manuscriptpublished!

Authorsdotheresearch

AuthorsresubmitcompletedSTAGE2manuscript:• Introduction andMethods (virtuallyunchanged)• Results(new):Registeredconfirmatoryanalyses

+unregisteredexploratoryanalyses• Discussion(new)• Datadepositedinapublicarchive

Around 8 have been published

Two examples of register reports

16

What we learned?

◈ It was a “high risk & costly” study involving drug intervention and MEG

◈ The theoretical question was straight forward:◈ Are gamma bands oscillation associated with glutamatergic

principal cells

◈ Issues: how much drug to give? What should be the outcome neutral criteria?

◈ Reviewer was not happy with the results (suggesting non registered analyses)

18

19

What we learned?

◈ The theoretical question was complex:◈ Feature vs. category based organization within two regions of

interest anterior temporal, temporal-parietal junction◈ Issues: how to define region of interest◈ How specific the hypothesis need to be◈ What would be a neutral-outcome to assess data quality◈ Not ignoring null results◈ How to report results from whole brain analysis

20

Noneofthesethingsmatter

Whatdoesmatter

THEHYPOTHESESARE CLEAR

THE STUDYIS POWERED

THEQUALITY OFTHE DATA IS

GOOD

THEMETHOD ISSOUND

BUT,whatisanacceptablehypothesisforimagingstudies?

• Myexperimentwillleadtochangeinactivation(/connectivity/amplitudechangeatanytimewindowetc.)somewhereinthebrain

• Myexperimentwillaffectaspecificfunctional/anatomicalstructures

23

FAQ

24

25

1.“IsRegisteredReportssuitableforallsciences?”

• Applicabletoanyareaengagedindeductive,hypothesis-drivenresearchwhereoneormoreofthefollowingproblemsapply:

• Publicationbias• Significancechasing• HARKing – “hypothesizingafterresultsareknown”• Lowstatisticalpower• Lackofdirectreplication• Lackofdatasharing

3.“What’stostopRegisteredReportsfrombecomingadumpinggroundforinconclusivenullresults?”

4.“Won’tthislimitthereportingofserendipitousfindings?”

• aprioripowerrequirements(≥90%)increasereproducibilityofallfindings• Bayesianmethodswelcomed(B<0.33orB>3forsubstantialevidence).A

specialistBayeseditorhasbeenappointedatCortex (ZoltanDienes)

2.“What’stostopresearchersfrom‘pre-registering’astudythattheyhavealreadyconducted?”

• Time-stampedrawdatafilesmustbesubmittedatStage2withbasiclablogandcertificationfromallauthorsthatdatawascollectedafter provisionalacceptance

• SubmittingacompletedstudyatStage1wouldthereforebefraud• Strategywouldbackfireanywaywhenreviewersaskforprotocolamendments• RegisteredReportsaren’tdesignedtopreventfraudbuttoincentivizegoodpractice

• Thearenorestrictionsonthereportingofunregisteredexploratoryanalyses.• Confirmatoryandexploratoryanalyseswillsimplybelabeledcorrectly

5.“Thisistoomuchworkforauthors.Weusuallydon’tdecidehowtoanalyseourdatauntilafterwe’velookedatit”

6.“Thiswillbetoomuchworkforreviewers,whoarealreadyoverstretched”

• It’sasimilaramountofwork,justdoneatadifferenttime– andprovisionalacceptancevirtuallyguaranteesapublication,withoutthepressuretoobtain“goodresults”

• Notnecessary.RRscouldreducesequentialsubmissionofmanuscripts‘downthechain’ofjournals

• RRdoesrequiresindepthreviewofprospectivemethodsandtheoreticalgrounding.

7.“Reviewerscouldstealmyideasatthepre-registrationstageandscoopme”

• OnlyahandfulofpeopleknowabouteachStage1submission• Onceprotocolisaccepted,thejournalcan’trejectyourpaperbecausesomething

similarwaspublished(noveltyisirrelevant)• ManuscriptreceiveddateonpublishedRRwillbethedateofStage1submission• Howdifferentfromgrantapplications,conferencepresentations,seminars?

9.“What’stostopauthorswithprovisionalacceptancepullingtheirmanuscriptaftergettingstunningresultsandsubmittingittoNature/Science/PNAS?”

8.“Doesprovisionalacceptanceabsolutelyguarantee publication?”• No,butitgivespeaceofmindtoauthorsthattheirpaperswon’tberejectedbecause

ofnegativefindings,perceivednovelty/importanceofoutcomes.Nofiledrawer

• Nothing.InthatcasethejournalwillsimplypublishaWithdrawnRegistration,whichwillincludetheabstractandabriefexplanationforthewithdrawalofthepaper

28

10.“Ihavenoideaofwhateffectsizetoexpectinmyexperiment,sohowcanIdoapoweranalysisaspartofStage1?”

• Usuallythereisatleastsomeliterature.Butevenifnot,aminimaleffectsizeoftheoreticalinterestcanbespecified

• Ifminimaleffectsizeisuncertain,optionsareanorthodoxstatisticalapproachwithcorrectedpeeking(Strube etal.,2006,Beh ResMeth,38,24-27)orBayesianmethodstospecifydistributionofpossibleeffectsizes

• PilotresultstoestablisheffectsizearewelcomedinStage1submissions

11.“Sometimesadesignissound,butthedataisgarbagebecauseresearchersruntheexperimentpoorly.Howwillyoudistinguishnegativefindings/weirdresultsduetopoorpracticefromthosethataregenuine?”

• Authorsmustincludeoutcome-neutralconditionsforensuringthattheexperimentsarecapableoftestingthestatedhypotheses(e.g.positivecontrols,manipulationchecks)

12.“Ifpublicationisguaranteedinadvance,whywouldresearchersbotherrunningthestudycarefully?Thisschemewillincentivizefalsenegatives.”

• Runningthestudycarelesslywouldalsoderailtheoutcome-neutralteststhatarenecessaryforfinalacceptance

13.“Whathappensifweneedtochangesomethingaboutourexperimentalproceduresaftertheyareprovisionallyaccepted?”

• Minorchanges(e.g.replacingequipment)canbefootnotedinStage2manuscriptasprotocoldeviations

• Majorchanges(e.g.changingexclusioncriteria)requirewithdrawalandresubmission• Editorialteamdecideswhetherdeviationissufficientlyminortocontinue

15.“IhaveaccesstoanexistingdatasetthatIhaven’tyetanalysed.CanIsubmitthisproposedanalysisasaRegisteredReport?”

• NotatCortex,butotherjournalsareallowingthis…

14.“Someofmyanalyseswilldependontheresults,sohowcanIpre-registereachstepindetail?”(e.g.outlierexclusion,ROIselection)

• Pre-registrationdoesn’trequireeachdecisiontobespecified,onlythedecisiontree• Authorscanpre-registerthecontingencies/rulesforfuturedecisions

16.“Pre-registrationwilldenigrateexploratoryresearch”

• No,whatdenigratesexploratoryresearchisHARKing:presentingexplorationasconfirmation(shoehorningKuhnintoPopper)

• Exploratoryanalysesarewelcometobereported,butunderanexplicitheading.• Exploratoryresearchissimplynotvaluedinitsnativeform,soletsstartdoingso!

17.“HowwillRegisteredReportsincentivizereplicationstudies?”

• Conspiracyofcircumstancestellsusnottobotherdoingdirect(exact)replications

• Methodsectionsareoftentoovaguetoallowprecisereplication• Chroniclackofpowerinnovelresearchmeansthatreplicationsoften

requireverylargesamplessizes• Attemptingtoexactlyrepeatapreviousexperimentcanbeseen(in

psychology)asanactofaggression(cf.physics)• Mostpsych/neurojournalswantnoveltyandseereplicationsas(usually)

unpublishable

• RRs:haveproposedreplicationexperimentreviewedandprovisionallyacceptedbefore youinvestsubstantialresourcesintodoingit;potentiallyinvolveoriginalauthorsinpeerreviewoftheprotocol

RegisteredReportsinformationhubontheOpenScienceFramework

https://osf.io/8mpji/wiki/home/

Google“registeredreports”– tophit

33

Thank you for listening