Opening up the science: Methodological transparency and ...projekt.ht.lu.se/fileadmin/user_upload/humlab/typo3/methodological... · Opening up the science: Methodological transparency

Opening up the science:

Methodological transparency

and reproducibility in second language acquisi7on

research EmmaMarsdenUniversityofYork,UK

Towards Methodological Transparency & Reproducibility in SLA research

1.  Openscience&theIRISdigitalrepositoryofmaterials

2.  RelaConshipbetweentransparencyandquality

3.  ReplicaCon&theimportanceoftransparency

4.  ChallengesandRecommendaCons

Publiclyfundedresearchshouldbemadefreelyavailable

OPENPUBLICATION->citedmore

Wagner,A.B.(2010).

OPENDATA•  CorporaoforalproducCondata,L1andL2:FLLOC,SPLLOC,CHILDES•  Trønso•  CLARIN

->increasedcitaCons Piwowar&Vision(2013)->strongerevidenceandbe]erstaCsCcalreporCng

Wicherts,J.,Bakker,M.,Molenaar,D.(2011).

The s7cks and carrots of open science

Part 1: Open science movement

Part 1: Open science movement

OPENMETHODSMakingmaterialsavailableMaterials=DatacollecContools,instruments,sCmuli,scoringandcodingprocedures,analysisprotocols

Materials design for L2 data collec7on: a lonely, mysterious business

•  Someproblems:

Researcherscreateandkeepowninstruments->re-invenConofwheel,poorsystemaCcityofresearch

Maintenanceandaccesstoinstrumentsisadhoc

PublicaConsjusthavebriefdescripConswithshortsamples

Whatacontrastwithfullinstrument!

Part 1: Open science: METHODS

How much of the methods do we see in journals?

Mai,Z.,&Yuan,B.(2016).Unevenreassemblyoftense,telicityanddiscoursefeaturesinL2acquisiConoftheChineseshì…declefconstrucConbyadultEnglishspeakers.SecondLanguageResearch,32(2),247-276.

Part 1: Open METHODS in SLA

WhatisIRIS?InstrumentsforResearchIntoSecondLanguages q Asustainabledigitalrepositoryq 2200+materialsusedtocollectdata:•  e.g.quesConnaires,grammaCcalityjudgmenttests,observaCon&interviewschedules,wordlists,sound&videofiles,languagetests,pictures,teachingmaterials,sofwarescripts,

• AndANALYSISPROTOCOLSandDATAq Downloadableq Uploadableq Searchable

Part 1: IRIS: Transparency of methods

Marsden,E.,MackeyA.,&Plonsky,L.(2016).TheIRISRepository:AdvancingresearchpracCceandmethodology.InA.Mackey&E.Marsden(Eds.),AdvancingmethodologyandpracFce:TheIRISRepository(pp.1-21).Routledge.‘AcademyResearchProject’AN110002

ScopeofcontentaswideasthefieldofL2research…

L2percepCon

&processing

MoCvaCon,idenCty,&strategies

…indiversecontexts

Languagelearning

PhonologyGrammarVocabularyPragmaCcs

Languageteaching,&

policyBilingualism

…withdiverseresearchaims

…withdiversetypesofdata

Part 1: IRIS: Transparency of METHODS Marsden, Mackey & Plonsky

Materialscanbeadaptedtosuitdifferentcontexts,learners,languages

Promotestransparency

Easiertoevaluatequalityandreliability

Qualityassurance:onlypeer-reviewedpublicaCons

SCmulates&facilitatesreplicaCon

RaKonalebehindIRIS

Part 1: IRIS: Transparency of METHODS Marsden, Mackey & Plonsky

•  2400files->over1100datacollecKoninstruments•  newmaterialscontributedalmostdaily

•  From1300researchers•  Searchableacross330+parameters,

ü researchareaü typeofinstrumentü languagefeatureü L1,L2ü parCcipantcharacterisCcs:age,proficiencyü author

• MaterialsqualifyforIRISifusedfor:peer-reviewedpublicaKonsorPhDs

Part 1: IRIS: Transparency of METHODS Marsden, Plonsky, Mackey

•  SubmissiontoIRISsupportedby37journals•  Inacceptancele]ersfromeditorstoauthors,andinauthorguidelines

• UploadtoIRISinpublicaConguidelinesoftheAmericanAssociaFonofAL•  FormallyendorsedbyBriFshAssociaFonofAL

• OnlyvenueinALthatqualifiesarCclesforCenterforOpenSciencebadges

•  16,500+downloadsofresearchmaterials

• Citedinhandbooks,methodsbooks,posiConpieces&syntheses

• Usedinmanyresearchmethodstrainingcourses

Part 1: IRIS: Transparency of METHODS Marsden, Plonsky & Mackey

Materials used for ‘year abroad’ research?

Part 2: Open methods to improve rigour and replicability…

The“methodologicalturn”inappliedlinguisCcs

“methodologicalissues…demandakindofprofessionalscruKnythatgoesdirectlytothecoreofwhatwedoandwhatweknow…”

Byrnes,2013,p.825“MethodologicalpracCcesandstudyqualityneedtobemeasured,notassumed”

Plonsky,2013

Benefits of transparent instrument design

Part2a:GrammaKcalityjudgmenttests

Indicatehowacceptablethefollowingsentencesarecompletely completelyunacceptable acceptable

1 2 3 41) Idon’tknowwhatisheeaCngfordinnertonight.2) Idon’tknowwhatheiseaCngfordinnertonight.

Part 2: Open methods to improve rigour and replicability

Frequent use of judgment tests because assumed to be…

Easyto…• develop•  administer•  score

CommonconvenConsàgreatercomparabilityacrossstudies

15

Part2a:Transparencyofinstrumentdesign:GrammaCcalityjudgmenttests

JTs are controversial Testsofexplicitorimplicitknowledge?(e.g.,Ellis,2005;GuCerrez,2013;Vafaeeetal.,2016)

TheoreKcalperspecKvesonlinguisKcknowledge->methodologicaldecisions

•  DoinstrucConssay‘howacceptable…’or‘howcorrect…’?•  Scaledvs.dichotomousresponse•  Oralvs.wri]en•  …

16

Part2a:Transparencyofinstrumentdesign:GrammaCcalityjudgmenttests

Methodological synthesis of JTs Howdowedesign,administerandreportonJTs?

K=299studies382JTsTotalsamplesizeinoursynthesis=24,679!

17

Part 2a: Instrument transparency: JTs. Plonsky, Marsden, Gass, Spinner, Crowther (in progress)

%ofJTsavailable

66

22

8

3

1

0 10 20 30 40 50 60 70 80 90 100

No

Yes, in article

Yes, on IRIS

Yes, other

Yes, article + IRIS

Transparency?

18


88%=306instrumentsunavailabletopublicfor

openscruCny.*****

252notavailableforreplicaCon,evenifyouhaveajournalsubscripCon

A‘SpecialCollec:onofJTs’

onIRISisimprovingthis!

•  MedianNwholestudysampleL2learners=47NScontrols=20Overallstudy=60

•  Fieldmedian=60(Plonsky,2013)

•  L1s45%=English15%=Chinese9%=Japanese8%=French

•  TLsEnglish=59%Spanish=17%French=10%

•  77%one-shotdesign->non-developmental,notwithin-subjectoverCme

Consequences of lack of availability Part 2a: Instrument transparency: JTs. Plonsky et al. (in progress)

Imaginethe‘n’ifJTsavailableacross

theglobe?

ThinkoftheL1–L2combinaCons!

ExisCngJTsusedtoinvesCgatechangeover

Cme,e.g.aferteachingor

yearabroad

Transparent design -> clearer opera7onalisa7on of knowledge type

20

41

26

24

20

12

8

7

5

4

1

0 20 40 60 80 100

items randomized

balance of (in)correct

identify errors

time limit

timed

context provided

breaks

rating of confidence

basis of judgment

comprehension Q

Part 2a: Instrument transparency: JTs. Plonsky et al. (in progress)

75%didnotdiscusskindofknowledgeelicited

->thereaderhastoinferwhatkindofknowledgetheresearcherselicited…

withoutfullreporCngofdesignfeatures,withoutseeingtheinstrument!

Consequences of lack of full transparency: Construct validity TimepressuremaKers.

52%wedon’tknowwhetherCmedornotModalitymaKers.

18%wedon’tknowwhetherwri]enorauralBreadthofconstruct-typesandtokensoflinguisFcfeaturetested?

51%wedon’tknowiftherewasonlyone‘version’Ifoneversion,narrowsbreadthofconstructbeingelicited

21

Part 2a: Instrument transparency: JTs. Plonsky et al. (in progress)

• Only15%reportedareliabilitycoefficient(e.g.,Cronbach’salphaforinternalconsistency)

vs45%reporCngreliabilityoverallinSLA(Plonsky,2013)

• Whenreported,meanreliability=0.80(=fieldaverage,Plonsky&Derrick,2016)

• Butreliabilitycalculatedonwholeinstrument• Weneedreliability:

ExcludingdistractorsSeparatelyforgrammaCcalandungrammaCcal

Consequences of lack of transparency: Reliability

22


Reliabilitycoefficientscanbegivenalongside

instrumentsonIRIS.

NewcoefficientspostedeveryCmethatJTused?

Part 2b: Instrument transparency: Self-paced reading

SimilaraimtoJTsEliciCngsensiCvitytogrammaCcalstructureandnormsBut,SPRsidenCfyprecisepointofdifficultyduringparsing,withoutanexplicitjudgment

Part 2b: Instrument transparency: SPRs. Marsden, Thompson, Plonsky (submi?ed)

Self-paced reading: an example

I

SCmulitakenfromMarsden,Cruickshank,Roberts(inprogress)CanwetrainprocessingrouFnesforabstractsyntax?

Part 2b: Instrument transparency: SPRs

Trytounderstandthissentence.

Clickforeachwordtoappear.

YouwillbeaskedaquesKonattheend!


don’t


Part 2b: Instrument transparency: SPRs.


know




what




is



CRITICALREGION


she



CRITICALREGION


having




for




dinner


Part 2b: Instrument transparency: SPRs


tonight



Which picture best matches the sentence?

A B

Theword‘dinner’-aZerthecriCcalregion

Transparency of instrument design: SPR

Methodologicalsynthesis

63studiesreporKngatotalof71SPRtests


Transparency and reproducibility of SPR tests in SLA research

Availability No.ofstudies

FullsKmulionIRIS 3Anotherplace(author’swebsite) 3ExampleinarKcleandonIRIS 4Full-allitems-inarKcle 16JustexamplesinarKcle 28Notavailableatall 16

A‘SpecialCollec:onofSPRs’

onIRISisimprovingthis!


96%offullmaterialsnotyet

availableonpermanent,open

repository

Consequences of poor availability: Agenda limi7ng - L2s?

Language No.studies:L1 No.studies:targetlanguage

EnglishChineseSpanishGreekGermanKoreanDutchJapaneseFrenchRussianArabic

MulKplelanguages

19954333311118

4321116120200


Consequence of poor availability: Agenda limi7ng: Cross-linguis7c influence?

WeknowtheL1ma]ersforonlineprocessing52/71withlearnerswithoneL114comparedacrossdifferentL1s


Consequence of poor availability:

Agenda limi7ng. Who are the par7cipants?

56/71=universitystudents•  Highermeta-linguisCcknowledgecouldaffectreadingCmes

(KeaCng&Jergerski,2015)


Beginner Interm Advanced NearNaCve Bilingual

7 18 50 3 8

• Todate,researchintoprocessingwhencomprehensionishigh•  advancedlearnersorbilinguals• onlyanalyse‘correctlycomprehended’sentences

• Butlearnersprocesslanguageevenwhencomprehensionispoor!

(thoughseeKeaCng,Jegerski&VanPa]en,2016;Xu,2014)

Consequence of poor availability: Agenda limi7ng. Why ‘successful’ comprehension only?


ExisFngSPRscouldbeusedwithotherlearnerpopulaCons

Consequence of poor availability: Agenda limi7ng. Inves7ga7ng learning over 7me?

GrowinginterestintheroleofprocessinginacquisiCon(Altmann&Kamide,Dussias,Gruter,Lew-Williams&Fernald,Kaan,O'Grady,Pickering&Garod,Chang

etal.,2006;Elman,1990;Seidenberg&MacDonald,1999;SpecialissueofLinguisFcApproachestoBilingualism,5,4).

YET,SPRnotyetusedtoinvesCgatedevelopmentofprocessing:76%oneshotstudies24%cross-secConalstudies,withproficiencyasavariableOneawithin-subject,longitudinaldesign

Thoughsee:McManus,K.,&Marsden,E.(2016).L1explicitinstrucConcanimprove

L2onlineandofflineperformance.StudiesinSecondLanguageAcquisiFon,1-34

ExisCngSPRscouldbeusedinlongitudinaldesigns

e.g.before,during,aferyear

abroad


Consequences of lack of transparency: Construct validity - Comprehension of what?

CentraltenantofSPR:processingduringreadingcomprehensionComprehensionquesConusedatendoftrials

QuesConshouldnotfocusa]enConon“criCcalregion”56/71usedCQsBUTwhatdotheseCQsfocusa]enConon?17–noexample34-oneexampleOnly5studiesprovidedmulCpleexamplesofcomprehensionquesCons


OneexampleofaCQdoesnottelluswherea]enConisrepeatedly

focussedduringreading

Consequences of lack of full transparency: Construct validity S7muli design (k items, word frequency, length) Lengthma]ers:affectscomputaConalease(Pienemann&Kessler,2011).Yet…

•  25/71lengthofsentencenotspecified

•  5/71reportednumberofsyllablesperword•  19/29usingsegments:numberofsegmentsnotspecified


AbankofsCmulihelpsinstrument

designandresearch

training

Consequence of lack of transparency:

Data cleaning and analysis protocols Outlierremoval?•  12didnotreportanyoutlierremoval.•  44usedparCcipantRTs•  15usedbothparCcipantRTsanditemRTs.

Outliercutoff?•  14used2.5SD•  5used200ms-2000ms(asrecommendedbyJegerski&KeaCng,butforanadvancedL2reader).•  Widevarietyofranges100ms–25000ms.

Analysealldataorjustcorrectresponses?•  7/56studiesanalysedall•  28/56analysedonlycorrectresponses•  21/56didn’treport


HavinganalysisprotocolsavailablewouldmakeanalysesmoresystemaCcandreproducible

Summary of Part 2: Greater transparency and availability should help:

• Agendase�ngandresearchaims(cross-secConal,L1-L2influence)•  Scopeandpowerofstudies(differentproficiencies,n)• OperaConaliseourconstructs(design:items,Cming,sCmuli)• Comparabilityacrossstudies(datacleaning&analysis)

MoretransparencyandsystemaKcityofmethods->Benefitsforalltypesofvalidity&reliability

‘Be]erreporCng’unlikelytofullyaddressallthese

problems(acrossalljournals&techniques…)

Theactualmaterials

necessary

“ConducCngaresearchstudyagain,inawaythatiseitheridenCcaltotheoriginalprocedureorwithsmallchanges(e.g.,differentparCcipants),totesttheoriginalfindings”(Mackey&Gass,2005:364).

“essenCal…supportfortheory”(Porte,2012)StrongreplicaConmovementinPsychology

•  “ManyLabs”&Reproducibilityprojects•  Pre-registra:onofmaterials&analyses

Part 3: Replica7on research in SLA

Commentaries and calls for replica7on

46publishedcommentaries&callsforreplicaConinL2researchSantos1989…Polio&Gass1997…Porte2012…Vandergirf&Cross2017

23+fromotherdisciplines:Psychology,EducaCon,Sociology,Business,MarkeCng,OrganisaConScience

ReplicaCons…lackingpresCge,originality,orexcitement

(Makeletal2012,ciCngLyndsay&Ehrenberg,1993;Neuliep&Crandall,1993)OthersynthesesofreplicaCons:forPsychology:Makel,Plucker&Hegarty(2012)forEducaConresearch:Makel&Plucker(2014)

01234567

1973

1975

1977

1979

1981

1983

1985

1987

1989

1991

1993

1995

1997

1999

2001

2003

2005

2007

2009

2011

2013

2015

Meanrateof1.5

replicaConsperyear

Norris&Ortega2000?

67replicaConarCcles,in26journals,of70iniCalstudies

Polio&Gass1997?

Howlongdoesittakeforastudytobereplicated?•  mean6.64years(sd6.16)•  range0-37years

Propor7on of replica7on in journals that published most replica7ons

SSLA TheMLJ FLA LL AP

Meanrate Totalrate

TOTALarKcles 562 1009 1528 855 1030

kreplicaKons 13 8 5 5 4

%oftotal 2.31% 0.79% 0.33% 0.58% 0.39% 0.88% 0.70%1973 – 2015, last complete year before synthesis

Comparedtootherdisciplines?1.07%Psychology,butNB:pre-replicaCon-boom&top100journals1%-3%Business,MarkeCng,CommunicaCon0.13%EducaCon(top100,1938-2014)

acrossthe26journalsthathavepublishedreplicaCons=0.26%

What do we replicate? Study design and findings ParKcipants?TheWEIRDestWesternEducatedIndustrializedRichDemocraCc(Mishraetal.2012)…

speaking(1/4)orlearning(1/2)EnglishBut‘ages’and‘proficiency’notreported(67%,37%studies)(Thomas1994)->replicability??

%ofreplicaKonswhoseiniKalfindingswere…

Weneedtoreplicatenullfindingstoowhenpower(n)islow,“nullfindings”≠“noeffect”

Schmidt,F..&Oh,I.-.S.(2016)TheCrisisofConfidenceinResearchFindingsinPsychology:IsLackofReplicaContheRealProblem?OrIsItSomethingElse?ArchivesofScienFficPsychology4,32–37

Null Nullbuttrend Statsigdifferences

other

3 3 83 7

How much do we change when we replicate?

Numberofchanges

changes=reasonsforreplicaKon%studies

0 331 282 213 94 75 1

Meanperstudy(stdev.) 1.34(1.31)

%studieswithchangesbetweenIandR…


Numberofchanges

changes=reasonsforreplicaKon

changesacknowledged,butnotreasonfor

replicaKon%studies

0 33 451 28 312 21 133 9 94 7 15 1 0

Meanperstudy(stdev.)

1.34(1.31) 0.91(1.04)



Numberofchanges

s=reasonsforreplicaKon

sacknowledged,butnotreasonforreplicaKon changesnotacknowledged

byauthors%studies

0 33 45 461 28 31 252 21 13 153 9 9 94 7 1 45 1 0 0Meanperstudy(sd)

1.34(1.31) 0.91(1.04) 1(1.18)


norelaKonshiptowhatthereplicaKoncallsitself(e.g.,‘parKalreplicaKon,’just‘replicaKon’,extension,conceptual)

Do “authorship overlaps” relate to whether findings are suppor7ve of ini7al study?

FindingsinrelaKontoiniKalstudy(%ofreplicaKonstudies)

Authoroverlap?(%totalRstudies)

%notorparKallynotsupporKve

%parKallyorverysupporKve

None(69%) 37 58Some/all(31%) 10 91

Overall 28 68

%“supporCveofIniCal”,asfuncConofauthorshipindependence

How do rela7ons between authorship teams relate to the nature of the conclusions?

FindingsinrelaKontoiniKalstudy(%ofstudies)

Authoroverlap?

(Kstudiestotal)

%notsupporKve

%parKallynot

supporKve%parKallysupporKve

%verysupporKve

%notreported

Nooverlap… (46) 20 17 30 28 4Someoverlap… (21) 5 5 43 48 0Overallk (67) 15 13 34 34 3

%headlinefindingsaccordingtoauthorshipindependencePsychology:92%supporCveifoverlapinauthorship65%supporCveifnooverlapEducaKon:89%supporCvewithoverlap,insamepublicaCon71%withoverlap,innewpublicaCon54%whennoauthoroverlap

vs.SLA:91%58%

QuesConableResearchPracCces(bias,p-hacking)

OrMaterialsavailabilityandfidelitytoiniCalstudy

??

How do studies use the findings and the data of the Ini7al study? Only6%ofReplicaConsprovidedIniCalstudy’seffectsize.

DataSharing!Opendataisassociatedwithstrengthofevidence

andqualityofreporCng

WichertsJM,BakkerM,MolenaarD(2011)WillingnesstoShareResearchDataIsRelatedtotheStrengthoftheEvidenceandtheQualityofReporCngof

StaCsCcalResults.PLoSONE6(11):e26828

Only6%ofReplicaConsusedIniCal’srawdatainanewanalysis

How do Replicators compare their findings to Ini7al Study?

%narraKveComparisons

%menKonedoriginalfindings

%dichotomousdecisionfrom

NullHypothesisSignificance

Test

%descripKvestaKsKcs %unclear %effectsizes

93 90 84 34 6 1

%ofReplicaKonStudiesthatcomparedtheirfindingstoIniKalStudyusing…

How do Replicators compare their findings to Ini7al Study?

LearningfromtheReproducibilityProjectinPsychology:masscoverage:

“inonly36%ofthestudiesweretheoriginalresultsreplicated”[becausep>0.05]

BUT…77%ofreplicaConeffectsizeswerewithina95%predicConintervaloforiginaleffectsize

(PaCletal.2016)

Transparency: How do replicators get hold of materials?

AvailabilityinIniKalStudy

Howpassedontoreplicator?%

%inar:cle

%passedoninprivate

%sharedauthorship %unclear

None(k=12) n/a 25 33 42parCalexamples(k=29) 54 12 31 4onefullinstrument,but

notall(k=26) 77 4 19 0

Only4%ofIniKalstudieshadmaterialsinIRISorsomeotheropenaccess

1) Replica7on: cultural changes in academia

• ReplicaConnotaneasyroute;it’sanessenFalroute

•  FacilitatestudentapprenCceshipmodel

“alotishiddenbehindthefinal[publishedarCcle].”

Roxana,areplicator,SLAgradstudent/traineeteacherfromVasquez&Harvey2010p.436

Part 3: Replica7on

1)only4/67triedtoreplicateastudythathadnullfindings

2)FearthatreplicaConwon’t‘replicatefindingsoforiginal’Howtomake‘nullfindings’morepublishable?Transparentreplica:one.g.viapre-registra:onshouldhelpMethodsfullyreviewedandapproved->InPrincipleAcceptanceIPATHENdatacollectedReviewerscannot‘argueout’onbasisofmethodologicalflawPerspecFvesinPsychologicalScience,Cortex,JournalofChildLanguage

2) Effects of publica7on bias on replica7on effort Part 3: Replica7on

Part 3: Example of a mul7-site replica7on Morgan-Short, Marsden, Heil, et al. (manuscript)

h]ps://osf.io/

Part 3: Replica7on research in SLA

The study: Is comprehension affected by acending to gramma7cal or lexical forms?

Elpuebloazteca,comopuebloprimiCvo,podíaencontrarunasoluciónalosproblemaspresentadosporlasfuerzasdela(La1)naturaleza.Dabamuchaimportanciaasureligión.EnellasuDiosprincipalytodopoderosoeraTonaCuh(elsol)(Sol1).Seloadmirómucho.TonaCuhteníalasbondadesylosdefectosdeloshumanos,peroconungranpodersobrenatural.Segúnla(La2)religiónazteca,elsol(Sol2)TonaCuhnecesitabaqueloalimentaran(-n1)conunasustancia...

Part 3: Example of mul7-site replica7on in SLA. Morgan-Short, Marsden, Heil et al.

THENcomprehensionmeasuredby10mulCplechoicequesCons

ORhearthesameintheoralmodality(andspotforms)

EITHERreadawri]entext(andspotforms):

fromLeowetal.(2008)

AvailableonIRISwww.iris-database.org

Par7cipants: Sites by Modality

704parCcipantsacross7researchsites

•  4sitesranlisteningversion•  UniversityofIllinoisatChicago*•  UniversityofOregon•  SouthamptonUniversity•  KazimierzWielkiUniversity&AdamMickiwiczUniversity

•  3sitesranreadingversion•  UniversityofYork*•  GeorgetownUniversity•  UniversityofSouthCarolina


Demonstra7on of the need for replica7on:

• Clearlyreplicatedfindingsin4sites• Notclearin2sites• Notreplicatedin1site

65


Challenges for mul7-site replica7on

1.  Seekingcollaborators2.  Parityinproficiencyatdifferentsites(Thomas,1994)

3.  CompaCbilityofsofware(EPrime/superlabscripts)4.  ResponsibiliCesfordataentry&analysis

weprovideddetailedprotocols


Benefits of mul7-site replica7on!

1.  N=7042.  Protocolsreadytobeusedagain

FourfurtherreplicaConsongoing:1inChina,1NaCvespeakers,1Heritagelearners,1aferyear

abroad

3.  Havingdifferentsitestemperedourclaims:Oneindividualstudycouldhaveconcludedonbasisofsitewithdifference!Similarfindingsacross6sitessuggestsomethingspecialinone‘odd’site=MorereliablereasonforgeneraCngnewhypothesis

67


CONCLUSIONS: Challenges for open methods

and Recommenda7ons

Some challenges for open materials

Concern1:Openmaterialswillencouragebaduseofmaterials

wesCllneedcriCcalthinkingabout…

• purposeofmaterials

• useoradaptaFonofmaterials

• analysisandinterpretaFon

Osborne,R.(2013).Whyopenaccessmakesnosense.InN.Vincent&C.Wickham(Eds.)DebaCngOpenAccess.(pp96-105).London:TheBriCshAcademy.

Concern 2: Reluctance to share “Others might misuse my materials” •  PutnotesaboutuseonIRIS• Wegatekeeptoreducebadscience:peerreview

“My next study might be ‘scooped’” •  TheexisFngstudyshouldbeopentofullscruCny•  Yourplanwon’tbesameasothers’ “I might be proved wrong” • Good!• but…unlikelytobesoclear-cut•  ->citaCons!

“I don’t have time” •  15minutesofyourCmevs3yearsofPhDstudent’s!•  SharingmagnifiesimpactofCme&publicmoney

“I can’t find my materials or data” •  AllthemorereasonforIRIStoexist!

Concluding remark (1) toresearchers:

•  Ethics:sharingwithall

•  ECc:onestudyindividualismvssyntheFc,acrosscontextsandoverFme(Plonsky2012,Norris&Ortega2005)

•  acollecFvemethodologicalmemory

Ratherthan:I’llsharewithyou(theoreFcalally),

butIwon’tsharewiththem

Concluding remark (2)

toyou,futurereviewersandeditorsofjournalsandbooks:• WithoutseeingFULLmaterials,canreviewersproperlyevaluate?

• Withoutopenaccesstomaterialsanddata,canresearchershaveproperly:

Ø builtonpreviousmethodssystemaFcally?Ø comparedtheirdatatopreviousdata?Ø reducedpotenCalbias?(giventhatindependencehelps)

Ask your journal editors…

toincen:viseandrecogniseopenscience

OpenScienceBadges(fromtheCenterforOpenScience)LanguageLearningadoptedtheschemein2015.StudiesinSecondLanguageAcquisiFon,availablefromnowAppliedLinguisFcs,boardapprovedTheModernLanguageJournal,boardapprovedLinguisFcApproachestoBilingualism,pendingLanguage,InteracFonandAcquisiFon,pending

Ø  Trofimovich&Ellis(2015)EditorialLanguageLearning.[OpenScienceBadges]

Ø  Blohowiak,B.,Cohoon,J.,de-Wit,L..,Farach,F.,Hasselman,F.,&DeHaven,A.(2016).BadgestoacknowledgeopenpracCces.osf.io/tvyxz

Effec7veness of journals recognising open materials with badges

KidwellMC,LazarevićLB,BaranskiE,HardwickeTE,PiechowskiS,FalkenbergL-S,etal.(2016)BadgestoAcknowledgeOpenPracCces:ASimple,Low-Cost,EffecCveMethodforIncreasingTransparency.PLoSBiol14(5):e1002456.doi:10.1371/journal.pbio.1002456

Heh guys, have you seen these on www.iris-database.org ?

Thankyouforlistening

With thanks to: Keycollaborators•  SusanGass,Pa�Spinner,DusCnCrowther(JT)

• AlisonMackey(IRIS)• RosMitchell&FlorenceMyles(Learnerlanguagecorporaprojects)

• KaraMorgan-Short,JeanneHeil,DavidAbugaber(ReplicaCon)

•  LukePlonsky(IRIS,JT,SPR,ReplicaCon)•  SophieThompson(SPR,ReplicaCon)Funders

Phew! I’m glad I went to www.iris-

database.org

Selected references KidwellMC,LazarevićLB,BaranskiE,HardwickeTE,PiechowskiS,FalkenbergL-S,etal.(2016)BadgestoAcknowledgeOpenPracCces:ASimple,Low-Cost,EffecCveMethodforIncreasingTransparency.PLoSBiol14(5):e1002456.doi:10.1371/journal.pbio.1002456

Morgan-Short,K.,Heil,J.,Botero-Moriarty,A.,&Ebert,S.(2012).AllocaConofA]enContoSecondLanguageFormandMeaning.StudiesinSecondLanguageAcquisiFon,34(04),659-685.

OpenScienceCollaboraCon.(2015).EsCmaCngthereproducibilityofpsychologicalscience.Science,349(6251),aac4716.Doi:10.1126/science.aac4716

Plonsky,L.(2012).ReplicaCon,meta-analysis,andgeneralizability.InG.Porte(Ed.),ReplicaFonresearchinappliedlinguisFcs(pp.116-132).NewYork:CambridgeUniversityPress.

Polio,C.,&Gass,S.(1997).ReplicaConandreporCng:Acommentary.StudiesinSecondLanguageAcquisiFon,19(4),499-508.

VanPa]en,B.(1990).A]endingtoformandcontentintheinput:Anexperimentinconsciousness.StudiesinSecondLanguageAcquisiFon,12,287-301.

Wagner,A.B.(2010).OpenaccesscitaConadvantage:Anannotatedbibliography.IssuesinScience,Technology&Librarianship.60:Winter

LouiseCorC-VeerleVandenEynden-LibbyBishop-Ma]hewWoollard-2014ManagingandSharingResearchDataAGuidetoGoodPracCce,Sage

UKDataArchiveMANAGINGANDSHARINGDATAabestpracCceguideforresearchers2ndedh]p://www.admin.ox.ac.uk/media/global/wwwadminoxacuk/localsites/researchdatamanagement/documents/managingsharing.pdf

Klein,R.etal.(2014).InvesCgaCngvariaConinreplicability:A‘‘ManyLabs’’replicaConproject.SocialPsychology45(3),142–152.

Leow,R.P.,Hsieh,H.,&Moreno,N.(2008).A]enContoformandmeaningrevisited.LanguageLearning,58(3),665-695.

Marsden,E.,Mackey,A.,Plonsky,L.(2016)TheIRISrepository:AdvancingresearchpracCceandmethodology.InMackey,A.&MarsdenE.(Eds.)AdvancingMethodologyandPracFce:TheIRISrepositoryofInstrumentsforResearchintoSecondLanguages.Taylor&Francis.

Makel,M.&Plucker,J(2014).Factsaremoreimportantthannovelty:ReplicaConintheEducaConsciences.EducaFonalResearcher,43(6),304–316.

Makel,M.,Plucker,J.&Hegarty,B.(2012).ReplicaConsinpsychologyresearch:Howofendotheyreallyoccur?PerspecFvesinPsychologicalScience,7(6)537–542.

Wicherts,J.,Bakker,M.,Molenaar,D.(2011).WillingnesstoshareresearchdataisrelatedtothestrengthoftheevidenceandthequalityofreporCngofstaCsCcalresults.PLoSOne,6,11,e26828

Documents

Opening up the science: Methodological transparency and ...projekt.ht.lu.se/fileadmin/user_upload/humlab/typo3/methodological... · Opening up the science: Methodological transparency