View
0
Download
0
Category
Preview:
Citation preview
University of ConnecticutOpenCommons@UConn
Doctoral Dissertations University of Connecticut Graduate School
5-10-2013
The Internet and Adolescent Readers: ExploringRelationships Between Online ReadingComprehension, Prior Knowledge, CriticalEvaluation, and Dispositions.J. Gregory McVerry JrSouthern Connecticut State University, jgregmcverry@gmail.com
Follow this and additional works at: https://opencommons.uconn.edu/dissertations
Recommended CitationMcVerry, J. Gregory Jr, "The Internet and Adolescent Readers: Exploring Relationships Between Online Reading Comprehension,Prior Knowledge, Critical Evaluation, and Dispositions." (2013). Doctoral Dissertations. 120.https://opencommons.uconn.edu/dissertations/120
TheInternetandAdolescentReaders:ExploringRelationshipsBetweenOnlineReading
Comprehension,PriorKnowledge,CriticalEvaluation,andDispositions.
J.GregoryMcVerryPh.D
UniversityofConnecticut,2013
Thepurposeofthismixedmethodsstudywastoinvestigatetherolethatbackground
knowledge,criticalevaluationofinformation,andareader’sdispositionsplayinpredictingonline
readingcomprehensionperformance,duringcomprehensiontasksthattakeplaceineitherless
restrictedormorerestrictedinformationspaces.
Sequentialregressionmodelsdemonstratedthat,aftercontrollingforverbalintelligence,
criticalevaluationandpriorknowledgeweresignificantinboththelessrestrictedinformation
spaceandthemorerestrictedinformationspace.Scoresonadispositionmeasurewereonly
significantinthemorerestrictedmodel.
Qualitativeanalysis,usingverbalprotocolmethods,foundthatwerekeyoveralldifferences
inhowskilledonlinereadersnavigateandmonitormeaningduringInternetinquirytasks.Skilled
readersengageinstrategictextassembly.Howeverallparticipantswerenotsuccessfulat
evaluatingorcommunicatingonlineinformation.
Theresultsofthisstudycontributetobothresearchandpractice.Forresearch,theresults
informricherandmorecomplexmodelsofonlinereadingcomprehension.Forpractice,theresults
informteacherschargedwithteachingliteracyinaconstantlyevolvingworld,oneinwhichthe
Internetisincreasinglyimportanttobothreadingandlearning.
ii
TheInternetandAdolescentReaders:ExploringRelationshipsBetweenOnlineReading
Comprehension,PriorKnowledge,CriticalEvaluation,andDispositions.
J.GregoryMcVerry
B.A.,UniversityofHartford,1999
M.Ed.,UniversityofHartford,2001
ADissertation
SubmittedinPartialFulfillmentofthe
RequirementsfortheDegreeof
DoctorofPhilosophy
atthe
UniversityofConnecticut
2013
iv
APPROVALPAGE
DoctorofPhilosophyDissertation
TheInternetandAdolescentReaders:ExploringRelationshipsBetweenOnlineReading
Comprehension,PriorKnowledge,CriticalEvaluation,andDispositions.
Presentedby
JohnGregoryMcVerry
MajorAdvisor_____________________________________________________
DonaldJ.Leu
AssociateAdvisor_____________________________________________________
DouglasK.Hartman
AssociateAdvisor_____________________________________________________
MichaelFagella‐Luby
AssociateAdvisor_____________________________________________________
DouglasKaufman
UniversityofConnecticut
2013
v
ACKNOWLEDGEMENTS
Manyhandshelpedwritethisdissertation.Firsttomylovingwifewhoputupwith
countlesshoursofsleeplessnights,groggymornings,andoccasionaltenseonewaynegotiations
withtechnology.Youhavealwaysbeenmydrivingforce.
Tomychildren,whoprovidedmycenter,wewrotethisforyou.Itisfittingthatontheday
mysecondsonwasbornmyfatherdrovemefromthehospitaltoaschoolfordatacollection,anda
weekbeforeIdefendedwewelcomedourthirdsonintothisworld.Asafamilywecamefullcycle
andallthewhile,John,mynamesake,youwerealwaystheretosneakintomyofficetohelpme
work.
Iameternallygratefultomyfriend,colleague,andadvisorDon.Youtaughtmeinnumerable
lessonsaboutresearch,teaching,andmostimportantlywriting.Ioweyouadebtofgratitudefor
thecountlesshoursofrevisionsandsupport.
AspecialthankstoIanwhostartedandfinishedthisjourneywithme.Wehavespent
countlesshoursdiscussingvitalissuesthatfaceourfield,ourfatherhood,andourfuture.Ithank
youforbeingthereinclassandhelpingmewhentheendgameseemedinsurmountable.
IalsooweaspecialthankstoSue.Youkeptmegroundedinwhatistrulyimportantin
education.Iwanttooffermyheartfeltgratitudeforthecountlesshoursyouspenteditingmywork.
Iwouldalsoliketothankthemembersofmydissertationcommittee.DougKaufman,you
wereaninvaluablesupportwhentimesweretoughanditseemedthedeckswerestackedagainst
me.DougHartmanIhavealwayslookedtoyouasasourceofintellectualguidance.Randyouhave
alwaysguidedmythinkingaboutthinking.Michaelyourinsightintoadolescentliteracywasa
valuedasset.
Finallythankstothemanyfriendsandfamilymemberswhostoodbymeduringthis
process.YouhavehelpedguidemeinmorewaysthanIcanlisthere.Ilookforwardtothankingand
celebratingwitheveryoneofyou.
vi
TableofContents
TableofContents....................................................................................................................................................................vi
ListofFigures...........................................................................................................................................................................ix
ListofTables...............................................................................................................................................................................x
CHAPTERI..................................................................................................................................................................................1OVERVIEWOFTHESTUDY............................................................................................................................................1Introduction..........................................................................................................................................................................1BackgroundoftheStudy.................................................................................................................................................3MoreRestrictedandLessRestrictedInformationSpaces................................................................................4PriorKnowledge.................................................................................................................................................................5CriticalEvaluationofTexts............................................................................................................................................6Dispositions..........................................................................................................................................................................6VerbalIntelligence.............................................................................................................................................................7ResearchQuestions...........................................................................................................................................................8Methods..................................................................................................................................................................................9SettingsandParticipants...........................................................................................................................................9
Materials..............................................................................................................................................................................11Procedures..........................................................................................................................................................................12Analysis................................................................................................................................................................................12SignificanceoftheStudy...............................................................................................................................................13
CHAPTERII..............................................................................................................................................................................14LITERATUREREVIEW....................................................................................................................................................14Introduction........................................................................................................................................................................14TheoreticalPerspectives...............................................................................................................................................14CognitiveFlexibilityTheory...................................................................................................................................15NewLiteracies..............................................................................................................................................................19
PriorResearch...................................................................................................................................................................24ResearchInOnlineReadingComprehension..................................................................................................24Researchonpriorknowledge................................................................................................................................31ResearchonCriticalEvaluation............................................................................................................................35ResearchonDispositions.........................................................................................................................................51ResearchonVerbalIntelligence...........................................................................................................................54
ChapterSummary............................................................................................................................................................56
CHAPTERIII.............................................................................................................................................................................59MethodsandProcedures..............................................................................................................................................59QuantitativeProcedures...............................................................................................................................................60SettingandParticipantSelection.........................................................................................................................60QuantitativeMeasures..............................................................................................................................................61
QuantitativeProcedures...............................................................................................................................................88AdministeringtheIndependentVariableMeasures.....................................................................................88AdministeringtheOnlineReadingComprehensionAssessments.........................................................89
QuantitativeAnalysis......................................................................................................................................................91QualitativeProcedures...................................................................................................................................................93
vii
QualitativeParticipants............................................................................................................................................93QualitativeProcedures.............................................................................................................................................94QualitativeAnalysis....................................................................................................................................................95
ChapterIV..............................................................................................................................................................................102QuantitativeResults.....................................................................................................................................................102DataScreening..........................................................................................................................................................103DescriptiveStatistics..............................................................................................................................................108ResultsForResearchQuestionOne:OnlineReadinginaLessRestrictedInformationSpace113ResultsForResearchQuestionTwo:OnlineReadinginaMoreRestrictedInformationSpace.........................................................................................................................................................................................116
ChapterSummary.........................................................................................................................................................120
ChapterV................................................................................................................................................................................124QualitativeResults........................................................................................................................................................124QualitativeAnalysis:ATwo‐StageApproach....................................................................................................126StageOneThemes.........................................................................................................................................................127Themeone:OverallORCAscoresappearedtoberelatedtostrategictextassembly.................129Themetwo:OverallORCAscoresappearedtoindicatealackofsourceevaluationknowledgeamongparticipants.................................................................................................................................................139Themethree:StudentswhorecalldetailsfrommemoryappeartoperformbetterontheORCAtasks...............................................................................................................................................................................150Themefour:Studentsappearedunpreparedtoengageinthecommunicationstrategiesnecessaryforonlinereadingcomprehensioninacademicsettings...................................................154
StageOneSummary.....................................................................................................................................................162StageTwoThemes........................................................................................................................................................164WhatPatternsofOnlineReadingComprehensionStrategiesAppearAmongHighandLowPerformingOnlineReadersDuringanOnlineReadingComprehensionTaskWithinaLessRestrictedInformationSpace?...........................................................................................................................165WhatPatternsofOnlineReadingComprehensionStrategiesAppearAmongHighandLowPerformingOnlineReadersDuringAnOnlineReadingComprehensionTaskwithinaMoreRestrictedInformationSpace?...........................................................................................................................177
StageTwoSummary....................................................................................................................................................185ChapterSummary.........................................................................................................................................................187
ChapterVI..............................................................................................................................................................................190Introduction.....................................................................................................................................................................190DiscussionoftheQuantitativeResults.................................................................................................................191OnlineReadingComprehensionAssessment:DescriptiveStatistics.................................................191SummaryoftheLessRestrictedModel:ResearchQuestionOne........................................................192SummaryoftheMoreRestrictedModeltoAddressResearchQuestionTwo................................194ExploringTheImplicationofBothModels....................................................................................................195
DiscussionoftheQualitativeResults....................................................................................................................202QualitativeResults:DiscussionofStageOneFindings.............................................................................203QualitativeResults:DiscussionofStageTwoFindings............................................................................209ExploringTheImplicationsOfDifferentPatternsOfProcessingDueToTheInformationSpace.........................................................................................................................................................................................212
Limitations.......................................................................................................................................................................213Instrumentation........................................................................................................................................................214MissingDataandSampleSize.............................................................................................................................216ResearcherBiasinQualitativeAnalysis.........................................................................................................217
viii
AddressingtheGrowingChallengesofOnlineReadingComprehension..............................................217
References:............................................................................................................................................................................221AppendixA:OnlineReadingComprehensionAssessment..........................................................................246AppendixBORCAProtocol.......................................................................................................................................310AppendixC:CriticalOnlineInformationLiteraciesAssessmentItems..................................................313AppendixD:DispositionsofOnlineReadingComprehensionAssessment..........................................340AppendixE:OnlineReadingComprehensionAssessment..........................................................................341
ix
ListofFigures
Figure3.1SampleVerbalComprehensionItems………………………………………...……………………………….62Figure3.2SingleScreenShotExamplefromCOILAssessment……………………………………....…………….64Figure3.3FourScreenShotsExamplefromCOILAssessment…………………………………………………….65Figure3.4FourHyperlinksScreenShotExamplefromCOILAssessment…………………………………….66Figure3.5ScreenShotfromHyperResearch……………………………………………………………………………......99Figure4.1ScatterplotsandHistogramsofResidualPlots…………………………………………………………..106Figure4.2BoxPlotofCook’sDistance………………………………………………………………………………………107Figure5.1ScreenShotofDiscussionBoardDirections........................................................................................156Figure5.2ScreenShotofDiscussionBoardTopics................................................................................................156Figure5.3ScreenShotofWomenintheRevolutionDiscussion......................................................................157
x
ListofTables
Table2.1Descriptive,Validity,andReliabilityofScoresonORCA‐Openassessments…….......................…29Table3.1DemographicDataofSchool……...………………………………………………………………................……..61Table3.2BackgroundknowledgeResponsesTothePrompt:”ListeverythingyouknowabouttheAmericanRevolution”……………………………………......................……….……………………………………………..................…….…63Table3.3Sub‐ConstructsofCredibility………….…………………………………………………................………………66Table3.4Sub‐constructsofRelevancy………..………………………………………………………................……….…….....67Table3.5ResultsoftheContentValidationStudy……….……..………………………………................……………...69Table3.6RevisionsbasedontheContentValidationoftheCOIL……...……….…..……................………...…….72Table3.7ContentValidationResultsforRevisedCOIL.............................................................................................75Table3.8CognitiveLabItemDiscrimination................................................................................................................77Table3.9FinalItemsinValidatedandReliableCOIL................................................................................................79Table3.10Descriptive,Validity,andReliabilityScoresonORCAassessments................................................82Table3.11TaskIntroductionfortheORCAInternetInquiryTasks.....................................................................83Table3.12InternalConsistencyoftheORCA‐LR.........................................................................................................91Table3.13InternalConsistencyoftheORCA‐MR........................................................................................................91Table3.14QualitativeParticipants..................................................................................................................................94Table3.15InitialCodebookforAbductiveAnalysis....................................................................................................96Table4.1SummaryofMissingData...............................................................................................................................104Table4.2Range,Means,andStandardDeviationsforDependentandIndependentVariables.............105Table4.3SkewnessandKurtosisRatiosforDependentandIndependentVariables..................................105Table4.4TransformedBackgroundknowledgeIndependentVariable...........................................................106Table4.5MeanscoresofL,E,S,Cperformance...........................................................................................................108Table4.6BivariateCorrelationsbyItemType..........................................................................................................109Table4.7FrequencyofBackgroundknowledgeScores..........................................................................................111Table4.8DescriptiveStatisticsforCOILItems..........................................................................................................112Table4.9N,MinimumandMaximumScores,Means,StandardDeviations(SD),forDependentandIndependentVariables........................................................................................................................113Table4.10BivariatecorrelationsofDependentandIndependentVariables................................................114Table4.11ResultsofHierarchicalRegressionfortheLessRestrictedORCA.................................................115Table4.12ResultsofHierarchicalRegressionfortheMoreRestrictedORCA...............................................117Table4.13Pearson’srCorrelationsBetweenVerbalIntelligenceandORCAitems...................................118Table4.14Pearson’srcorrelationsbetweenCOILandORCAitems..............................................................119Table4.15Pearson’srcorrelationsbetweenCOILandORCAitems..................................................................119Table4.16Pearson’srcorrelationsbetweenCOILandORCAitems..................................................................120Table5.1QualitativeParticipants..................................................................................................................................126Table5.2FrequencyofNavigationalStrategies........................................................................................................130Table5.3FrequencyofStrategiesUsedtoReadWebsites.....................................................................................133Table5.4NavigationStrategiesUsedWhenTakingaPosition..........................................................................137Table5.5FrequencyofStrategiesUsedtoIdentifyanAuthor.............................................................................141Table5.6FrequencyofStrategiesusedtoJudgeAuthorExpertise....................................................................144Table5.7FrequencyofUsingaStrategyofRecallingDetailsFromMemory................................................151Table5.8DiscussionBoardResponses..........................................................................................................................158Table5.9FrequencyofStrategiesUsedtoDevelopKeywords.........................................................................167Table5.10FrequencyofStrategiestoReadSearchEngineResults..................................................................171Table5.11FrequencyofNavigationStrategiesWhenReadingSearchEnginesDuringtheLessRestrictedTask........................................................................................................................................................................174
xi
Table5.12FrequencyofRelevancyJudgmentsonLessRestrictedTasks........................................................176Table5.13FrequencyofLocatingStrategiesontheMoreRestrictedORCA..................................................179Table5.14FrequencyofStrategyUseonMoreRestrictedLocatingTasks....................................................184
1
CHAPTERI
OVERVIEWOFTHESTUDY
Introduction
TheInternetisquicklybecominganimportantnewcontextforreading.Notoolforliteracy
hasspreadquickerandfasterthantheInternet(Coiro,Knobel,Lankshear,&Leu,2008).Withover
twobillionusersonline(MiniwattsMarketingGroup,2012)andagrowthratethatisexponential,
theInternetisshiftingthesocialpracticesofliteracyandlearning(Lankshear&Knobel,2006)
Thus,understandinghowstudentsreadandcomprehendinformationonlineiscrucialforteaching
andlearning.
Afterall,evidenceisemergingthattheInternetisthetextofchoiceforadolescentreaders.
Forexample,adolescentsnowspendmoretimereadingonlinethanoffline(KaiserFamily
Foundation,2005).Furthermore,ina2001survey90%ofstudentswithInternetaccessreported
usingtheInternetforhomework.70%ofthesestudentsrespondedthattheInternetwastheir
primarysourceofinformation(Lenhart,Horrigan,&Fallows,2004).Thesefundamentalshiftsin
theaccess,use,anddisseminationofinformationhaveledresearcherstocallforrichertheoretical
modelsofreadingcomprehensionthataccountfortheadditionalcomplexityofonline
environments(Alexander,2010).
Thepurposeofthismixedmethodsstudywastoinvestigatetherolethatbackground
knowledge,criticalevaluationofinformation,andareader’sdispositionsplayinpredictingonline
readingcomprehensionperformance,duringcomprehensiontasksthattakeplaceineitherless
restrictedormorerestrictedinformationspaces.Theresultsofthisstudycontributetoboth
researchandpractice.Forresearch,theresultsinformricherandmorecomplexmodelsofonline
readingcomprehension.Forpractice,theresultsinformteacherschargedwithteachingliteracyin
2
aconstantlyevolvingworld,oneinwhichtheInternetisincreasinglyimportanttobothreadingand
learning(Snow,2002).Thismixedmethodstudyexploredthefollowingresearchquestions:
1. When predicting online reading comprehension of a problem solving task within a less
restricted information space, what is the best fit model, after controlling for verbal
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension?
2. When predicting online reading comprehension of a problem solving task within a more
restricted information space, what is the best fit model, after controlling for verbal
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension.
3. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a less
restricted information space?
4. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a more
restricted information space?
Thesequestionswereinvestigatedinthreephases.Inphaseonetheinstrumentsnecessary
forthestudywerecreated.Inphasetwo,regressionanalysiswasusedinanattempttounderstand
factorsthatexplainedvarianceinscoresofanonlinereadingcomprehensionmeasure.Inphase
threeverbalprotocolmethods(Afflerbach,2010)wereusedtoexplorestrategyuseamong
participantswhovariedintheironlinereadingcomprehensionproficiency.
3
BackgroundoftheStudy
Thelargestreviewofcomprehensionresearchhasconcludedthatthedemandsofonline
readingcomprehensionrequirenovelskillsbeyondthosenecessarytounderstandprintedtext
(Snow,2002).Abodyofevidence(Coiro,2011;Coiro&Dobler,2007;Henry,2006;Kuiper,Volman,
&Terwel,2005)isgrowingtosupportthisclaim.However,thesenascentmodelsofonlinereading
comprehensionneedtobeenrichedifwearetobetterunderstandthediversenatureofreading
thattakesplaceonline.
Simultaneously,astheInternethasbecometheinformationaltextofchoiceformany
students,schoolsacrossthenationhavestruggledtoprovideadolescentswithsuccessfulliteracy
classrooms(Snow&Biancarosa,2003).Thus,scholarshavecalledforanagendato“systematically
linkinstructiontothegrowingknowledgebaseonliteracyandinformitwithup‐to‐datedata
relatingtooutcomesandbestpractices.”(CarnegieCouncilonAdvancingAdolescentLiteracy
[CCAAL],2010,p.10).
Yettheserecentcallstoactionpaylittleattentiontothechangingnatureofreading
comprehension.Infact,thereport“ATimetoAct”(CCAAL,2010)notedthatadolescentsmust
understandlongermorecomplicatedtextswithspecializedvocabularyandcomplexgraphical
representations.Howeverthisreviewofresearchonadolescentliteracydidnotdescribehowthese
challengesarefurthercomplicatedwhenreadingonline(Snow,2002).Effortstoimprove
adolescentliteracymustbegintoincludeonlinereadingcomprehensionifwewishtoprepare
studentsforthereadingdemandsofthe21stcentury.
Previousworkinonlinereadingcomprehensionhasoftenassumedalimiteddefinitionof
onlinereading.Littleworkhasevaluatedonlinereadingcomprehensionfromamorecomplex
perspective–onethatincludesvariationsintheextentoftheinformationspace,background
knowledge,thecriticalevaluationofinformation,orareader’sdispositions.Research(Coiro,2011)
4
Kuiper&Volman,2005;Metzger,2007)suggeststhateachoftheseelementsmaybeimportantto
understandifweexpecttobetterunderstandthecomplexnatureofonlinereadingcomprehension.
MoreRestrictedandLessRestrictedInformationSpaces
TheInternetisnotaunidimensionalcontextforreading;itisacomplexmultifaceted
informationalspace.Inordertoenrichourunderstandinganddevelopnewtheoreticalmodelsof
onlinereading,researchersmustinvestigatehowthedemandsofreadingchangeindifferenttypes
ofreadingcontexts(Hartmanetal.,2010).Previousresearchhasnotalwaysevaluatedonline
readingcomprehensionwithinmultiplecontextssuchaswhenthereadingtasksoccurinmore
restrictedorlessrestrictedinformationspaces.
Oneofthegreatestdemandsplacedonthereaderisdealingwithashiftinginformation
space;whenreadingonline,thesizeoftheinformationfieldconstantlyshiftsbasedonthetaskof
thereader(Leu,2000).Morerestrictedinformationspacesuseasmallerfieldofinformation.Less
restrictedinformationspaceshavealargerfieldofinformation.
Forexamplereadinganarticleonanewspaperwebsitetakesplacewithinamorerestricted
informationspacewhencomparedtoanInternetinquiryonthesearchfortheLochnessMonster.
Havingareaderlookforaspecificwebsite,suchastheNewYorkTimes,alsousesamorerestricted
informationsearchtask.Ontheotherhand,havingreaderslocateanyusefulwebsiteonagiven
topicrequiresalessrestrictedinformationspacethanlocatingaspecificwebsite.Studentswho
havetosiftthroughmanymoresearchresultsandkeywordsearchesmaynotautomatically
identifyausefulsource.Clearlytherearefundamentaldifferencesinthereadingdemandsofmore
orlessrestrictedinformationspaces.
Moststudiesofonlinereadingcomprehensionhavefocusedoneitheramorerestricted
informationspace(Coiro&Dobler,2007)oralessrestrictedinformationspace(Deschryver&
Spiro,2010)withoutrecognizingthepossiblealterationsthismightmaketoreadingperformance.
ForexampleCoiro&Dobler(2007)usedamorerestrictedenvironmentbyhavingstudentslocatea
5
specifictigerwebsiteaspartofaverbalprotocolanalysis.OntheotherhandDeschryverandSpiro
(2010)usedalessrestrictedinformationspacebyhavingstudentsconductanopen‐endedsearch
onclimatechange.Todatenostudyhasexaminedhowareader’sperformancechangesbyvarying
therestrictednatureofinformationinonlinespaces.Itisimportantforbothresearchersand
educatorstounderstandhowthereadingdemandsofonlinereadingcomprehensionshiftbasedon
thenatureoftheinformationspace.
PriorKnowledge
Itisalsoimportanttoinvestigatehowpriorknowledgeaffectsstudentperformanceon
tasksthattakeplaceinbothmorerestrictedandlessrestrictedinformationspaces.Prior
knowledgewasoperationalizedasbackgroundknowledgeinthisstudyasonlydomainspecific
knowledgeabouttheAmericanRevolutionwasmeasured.Otherpriorknowledgesuchas
knowledgeoftheInternetwasnotmeasured.
Thestrongeffectthatbackgroundknowledgehasonreadingcomprehensionisoneofthe
moststablefindingsinallofcognitivepsychology(Paris&Stahl,2005).Evenearlyreading
researchersfromGates(1931),Huey(1908),andGray(1939)notedtherelationshipbetween
backgroundknowledgeandreading.Thereforebackgroundknowledgewasincludedinthe
analysisbecauseithashistoricallybeensuchastrongpredictorinmodelsofofflinereading
comprehension(Alexander&Jetton,2004;Pearson,1982).
Whiletheroleofbackgroundknowledgeinthecomprehensionofofflinetextiswell
established,thereadingcommunityhasonlybeguntoinvestigateroleofbackgroundknowledge
duringonlinereadingcomprehension.HillandHannifin(1997)foundthatpriorknowledgeofboth
thetopicandInternetsystemsarerequiredforsuccessfulonlinereading.CoiroandDobler(2007),
fromtheirthinkaloudswithskilled6thgradeonlinereaders,identifiedfourtypesofprior
knowledgeinvolvedduringonlinereadingcomprehension:topic,informationaltextstructures,
websitestructure,andsearchengines.Coiro(2011usedhierarchicallinearregressionandfound
6
backgroundknowledgetopredictasignificantamountofvarianceineachhierarchicalregression
modelofbothofflineandonlinereadingcomprehension.
CriticalEvaluationofTexts
Inaddition,itisimportanttoinvestigatehowthecriticalevaluationofonlineinformation
affectsstudentperformanceontasksthattakeplaceinmorerestrictedandlessrestricted
informationspaces.Oneofthemoreessentialelementstosuccessfulonlinereadingcomprehension
istheabilitytocriticallyevaluateinformation(Kiili,2008).Inanerawhenpublishinghasbecome
theprovinceofaparticipatoryculture,traditionalmarkersofrelevancyandcredibilityarenot
readilyavailabletoreaders(Metzger,Flanagin,&Zwarun,2003).Furthermorestudiesinvestigating
thecriticalevaluationofwebsitessuggestthatmanystudentsstrugglewiththisimportantaspectof
onlinereadingcomprehension.Collegeageparticipants(Tillotson,2002)andmiddleschool
students(Coiro&Dobler,2007)oftenrelyonsuperficialcontenttoreachtheirdecisionwhen
judgingwebsites.ClearlyreadersmustapproachtheInternetwithacriticaleye.Yetwedonotfully
understandtherolethattheevaluationofinformationplaysduringonlinereadingcomprehension.
Whileevidenceisemergingthatcriticalevaluationiscentraltosuccessfulonlinereading
comprehension,researchersdonotknowhowthisroleshiftswithachanginginformation
landscape.Oneofthefundamentalchallengesstudentsfacewhenreadingonlineisthevastamount
ofinformation(Tate&Alexander,1998)availableduringself‐directedtextconstruction(Coiro&
Dobler,2007).Thus,astheinformationspacebecomeslargerandlessrestrictedreadersmayhave
torelymoreheavilyontheirabilitytojudgetherelevancyandcredibilityofwebsites.Forthese
reasonscriticalevaluationwillbeincludedinthisstudybecause,asaskill,itencapsulatesmanyof
thechallengesstudentsfacewhenreadingonline.
Dispositions
Finally,itisimportanttoinvestigatehowdispositionsofonlinereadingcomprehension
affectstudentperformanceontasksthattakeplaceinbothmorerestrictedandlessrestricted
7
informationspaces.Readingisalwaysasituated,affectiveactivity(Brown,Collins,&Duguid,
1989).Thisrequiresanyadequateattemptatmodelingonlinereadingcomprehensiontoinclude
variables,beyondknowledgeandskills(Carr&Claxton,2002),suchasdispositions.Accordingto
Katz(1993)dispositionsarea“tendencytoexhibitfrequently,consciously,andvoluntarily,a
patternofbehaviorthatisdirectedtowardabroadgoal.”
Asonlinereadersengageinself‐directedtextconstruction(Coiro&Dobler,2007)
dispositionsmaybecomeevenmorecriticalforcomprehensionforseveralreasons.Firstthe
Internetisamuchmorecomplexinformationspace(Katz&Rice,2002;Norris,2001)anditis
unlimitedinnature(Alvermann,2004;Gross,2004).Thesefundamentalchangestotexts,andthe
challengestheypresenttoreaders,suggestthatonlinereadingcomprehensionrequiresemerging
habitsofmind(DeSchryver&Spiro,2010).Furthermoresincelearnerswithpositivedispositions
oftenseekoutchallengingtexts(Guthrie,Wigfield,&Percenevich,2004)readerswithpositive
dispositionstowardsonlinereadingcomprehensionmaybeabletocompletetasksthatrequire
increasedtopicalknowledge.Finallystudentswithapositivedispositiontowardsonlinereading
mayexhibitgreateruseofcognitivestrategiesinonlineenvironments(Coiro,2007).Asaresult,a
measureofreaderdispositionsisincludedintheanalysisbecausetraditionalstudiesinreading
haveestablishedtheimportanceofincludingaffectivevariables(Guthrie&Wigfield,1997)in
readingcomprehensionmodels.
Inshort,thisstudyseekstoexaminesomeofthecomplexitiesthatappeartobeapartof
onlinereadingcomprehension:therelativecontributionsofbackgroundknowledge,critical
evaluation,anddispositionsofthereaderduringtasksthattakeplacewithinmorerestrictedand
lessrestrictedinformationspaces.
VerbalIntelligence
Oneofthefundamentaldifferencesofonlinereadingcomprehensionisthatthesetasksare
drivenbytheirproblemsolvingnature(Leu,O’Byrne,Zawilinski,McVerry,Everett‐Cacopardo,
8
2009).Students,engagedinonlinereadingcomprehensiontasksalsohavetoundertakeself‐
directedtextconstruction(Coiro&Dobler,2007)usingwebsiteswithvaryingdegreesof
readability.Thispresentsauniquechallengetostudyingmodelsofonlinereadingcomprehension,
asstudentswillhavevaryingcognitiveabilitiestocreatethesemultiplesourcetexts.
Itiscommoninthefieldofcognitivesciencetocontrolforverbalintelligenceonsearching
andinformationretrievaltaskstocontrolforthesedifferencesincognitiveabilities(Allen,1992).
Verbalintelligencehasbeenshown,usingfactoranalysis,tobesignificantlyrelatedtomeasuresof
conceptattainmentandinformationprocessing(Lemke,Elmer,Klausmeier,&Harris,1967).
Thereforeinordertounderstandthecontributionthatbackgroundknowledge,criticalevaluation
skills,anddispositionsmaketomodelsofonlinereadingcomprehensionverbalintelligencewillbe
controlled.
ResearchQuestions
Thisstudyusesamixedmethoddesign(Creswell,1994)withbothquantitativeand
qualitativeanalyses.Collins,Onwuegbuzie,andSutton(2006)suggestthatbeforemixedmethods
questionscanbedevelopedthegoalsofthestudymustbeidentified.Thegoalofthisstudywas
bothpredictiveanddescriptiveinnature.
OnwuegbuzieandLeech(2006)definepredictionas“usingpre‐existingknowledgeor
theorytoforecastwhatwilloccuratalaterpointintime.”Thereforethisstudywilldrawonboth
thetheoreticalperspectivesinwhichitisframedandpriorresearchtoinvestigatethe
contributionsthatbackgroundknowledge,criticalevaluationofwebsites,anddispositionsmaketo
onlinereadingcomprehensioninbothlessrestrictedandmorerestrictedinformationspaces.
Describingthedifferencesintheprocessesstudentsuseduringonlinereading
comprehensiontasksisalsoagoalofthisstudy.Verbalprotocols(Pressley&Afflerbach,1995),or
think‐alouds,areoftenusedtodescribethecognitiveprocessesusedduringcomprehension
activities(Campbell,2005).Thesedatacanrevealimportantprocessesusedtocompletethe
9
assessment(Ericsson&Simon,1993)andcanbeadaptedtoalsohelpilluminatetheroleof
backgroundknowledge(Palinscar,Magnusson,Pesko,&Hamlin,2005)duringonlinereading
comprehensiontasks.Consequentiallythisstudyutilizedverbalprotocolanalysistoidentify
patternsofcomprehensionprocessesusedduringanonlinereadingcomprehensionassessment.
Fourresearchquestionsguidedthisstudy:
1. When predicting online reading comprehension of a problem solving task within a
less restricted information space, what is the best fit model, after controlling for
verbal intelligence, of the percentage of variance accounted for by each of the
following variables: background knowledge, critical evaluation, and dispositions of
online reading comprehension?
2. When predicting online reading comprehension of a problem solving task within a
more restricted information space, what is the best fit model, after controlling for
verbal intelligence, of the percentage of variance accounted for by each of the
following variables: background knowledge, critical evaluation, and dispositions of
online reading comprehension.
3. What patterns of processing appear among readers with varying levels of online
reading comprehension ability during an online reading comprehension task within a
less restricted information spaces?
4. What patterns of processing appear among readers with varying levels of online
reading comprehension ability during an online reading comprehension task within a
more restricted information space?
Methods
SettingsandParticipants
Thisstudywasconductedwithaconveniencesampleof1317thgradestudents.This
samplewasselectedfromthreeschools.Theschoolswerechosenfromdistrictsthatwere
10
economicallyprivileged,economicallyaverage,andeconomicallychallenged.Oneschoolwas
chosenfromeachSESgroupasdeterminedfromtheConnecticutDistrictReferenceGroups[DRG]
(ConnecticutDepartmentofEducation,2009).Adistrict’sDRGisdeterminedusingavarietyof
economicindicatorsincluding,parentalmedianincome,medianhomevalue,freeorreducedlunch
ratio,parentaleducationlevel,andotherrelatedfactors(ConnecticutDepartmentofEducation,
2009).
Quantitativeparticipants.Theparticipantsinthisstudyinvolved131seventhgraders.
AccordingtoTabachink&Field(2007)asampleof131participantsexceedstheirguidelinesfor
regressionanalysisof100+m,wheremequalsthenumberofpredictors.Thissamplesizeisalso
adequateforaregressionmodelwithonedependentvariableandthreeindependentvariableswith
an=.05,adesiredpowersizeof0.8,andananticipatedeffectsizeof0.15(Sloper,2010).This
anticipatedCohen’sf2isamediumeffect.Theestimateofpowersizewaschosentoensurean
adequateeffect(Sloper,2010).
Qualitativeparticipants.Twelvestudents,fourfromeachparticipatingschool,were
selectedforthequalitativeportionofthestudy.Performanceonthefirstadministrationofthe
onlinereadingcomprehensionassessmentwasusedtodetermineparticipantsinthethinkaloud
activity.Theywereselectedasfollows.
1. First all students in their class were ranked based on scores on the first administration of
the ORCA. Students were divided into two groups: the top 10% of scores and the bottom
10% of scores.
2. These groups were then reviewed by the teacher to allow for the selection of students
who would be comfortable working with an adult on a verbal protocol task.
3. Then four students from each participating school were randomly selected from the list:
two students who scored in the 10% of their class and two students who scored in the
bottom 10% of their class.
11
Then,forthequalitativeanalysis,thetwelveselectedstudentsweredividedintothree
achievement‐levelgroups(highaverage,low)basedonORCAscores,independentofschool.Ten
studentswereincludedinthefinalanalysissincetwostudentshadtoberemovedfromthestudy
duetoerrorsindatacollection.
Materials
Assessmentsusedinthisstudy,exceptverbalintelligence,werecreatedandvalidatedby
theresearcher.Theyincludetwomeasuresofonlinereadingcomprehension:onewithinaless
restrictedcontextandtheotherwithinamorerestrictedcontext.Eachassessmentconsistedoftwo
InternetinquirytasksontheAmericaRevolution.Therewere12scorepointsineachtaskfora
totalof24scorepointsperassessment.
Backgroundknowledgewasestimatedusingathree‐itemtopicalknowledgequestionnaire.
Thequestionnairewasdeliveredusingacomputer‐basedsurvey.Participantswereaskedto
respondtoapromptaskingthemtolistallthefactsordetailstheyknewabouttheAmerican
Revolution.ThesemethodswereadaptedfromCoiro(2011)whoadaptedLeslieandCaldwell’s
(1995)ConceptQuestionTaskandWolfeandGoldman’s(2005)measureoftopic‐specific
backgroundknowledge.
CriticalEvaluationofwebsiteswasmeasuredusingafourteen‐item,multiple‐choicetest,Critical
OnlineInformationLiteracies(COIL).TheCOILfocusedonfourconstructs:authorexpertise,
publisher,evidence,andbias.
TheDispositionsofOnlineReadingComprehensioninstrument(DORC)(O'Byrne&
McVerry,2008)wasusedtomeasurethedispositionsofonlinereadingcomprehensionamong
participants.TheDORCwasacompositescoreoffivescales:reflectivethinking,collaboration,
flexibility,criticalstance,andpersistence.
Verbalintelligencewasmeasuredusingapreviouslyvalidatedassessment,theverbal
comprehensionvocabularytestoftheKitofFactorReferencedCognitiveTest(Ekstrom,French,
12
Harman&Derman,1976).Thistestconsistedofaseriesofvocabularymultiple‐choicequestions.
Themeasurewaschosenbecauseofitsestablishedreliabilityandusebyresearchersstudying
informationalretrievalandlearningfrommultiplesources.Reliabilityestimatesforthetestwhen
workingwith7thgradershaverangedfrom.73‐.86.Theassessmenthasbeenusedbyresearchers
studyinganumberofcontexts:searchpatternsinhypertextwithinCD‐ROMs(Allen,1992);the
learningofsciencecontentwithWebbasedtexts(Wallen,Plass,Brunken,2005);andcomparisons
ofnotetakingstrategieswhilereading(Tuckman,1993).
Procedures
First,participantswereaskedtocompletethemeasureofbackgroundknowledge.Then
studentscompletedthedispositionsofonlinereadingcomprehensionassessmentandtheCOIL
assessment.Next,thestudentscompletedalessrestrictedandamorerestrictedORCAtask.This
wasthefirsthalfofboththelessrestrictedonlinereadingcomprehensionassessmentandthemore
restrictedonlinereadingcomprehensionassessments.ThenthefirsttwoORCAtaskswerescored.
Nextparticipantsforthethink‐aloudswereidentifiedbasedonperformanceonthefirsthalfofthe
ORCA.FinallyallstudentswereadministeredthesecondORCAassessment.
AfterthefirstadministrationoftheORCA,participantsforthethink‐aloudswereidentified.
Studentsthencompletedastructurallypromptedthinkaloudactivity,duringthesecond
administrationofboththelessrestrictedandthemorerestrictedORCA.Studentswerepromptedat
fixedstructurallocationstothinkaloudbytheexperimenters(Afflerbach,2002)whilereading
online.Thestudentswereasked,“Whatwereyouthinking?”atkeydecisionpointssuchasclicking
onalink,leavingawebsite,orenteringkeywords.
Analysis
Toanswerthetwoquantitativequestions,sequential(hierarchical)regressionanalysis
(Tabachink&Field,2007)wasusedtoestimatethebestfitmodelsbetweentheindependent
variableofonlinereadingcomprehensionscoresandthedependentvariablesaftercontrollingfor
13
verbalintelligence.Thiswasdoneforboththelessrestrictedandthemorerestrictedonline
readingtasks.
Toanswermyqualitativequestionsabductive(Onwuegbuzie&Leech,2006)methodsand
constant‐comparative(Bogdan&Biklen,2003;Merriam,1988)methodswere.Thisdualapproach
wasdesignedtoidentifypatternsincognitiveprocessingwhilealsoallowingforthecodebookto
unfoldasthedatawereexamined.
Abductivecodingmethods(Onwuegbuzie&Leech,2006)employbothinductiveand
deductivecodingprocedures.Initialcodingschemeswereinformedbypreviouswork(Leuetal.,
2004;Leuetal.,2009).ThequalitativedatawascodedusingHyperRESEARCH,asoftware‐
packagingtoolthatallowsforthecodingofvideodata.Eachvideocasewasloadedintothe
program.Thevideoswerethenbrokenintoseparateframes.Aunitofanalysisbeganwithstudent
actionortalk.Allrelatedtalkandrepetitiveactionswereincludedinaunit.Theunitendedwhen
talkoractionclearlychanged,suchasclickingon“Go/Search/Enter”orleavingawebsite.
SignificanceoftheStudy
Thisstudycontributestobothliteracyresearchandliteracypractice.Forresearchthis
studyinvestigatedthechangesinthereaderbyfocusingononeofthegreatestchallengesthe
Internetpresentstoreaders:thesizeoftheinformationspace.Byinvestigatingbackground
knowledge,criticalevaluation,anddispositionsofthereaderwhilehavingstudentsreadinboth
morerestrictedandlessrestrictedinformationspacesthisstudyseekstoenrichmodelsofonline
readingcomprehension.
Resultsofthisstudycanalsoofferdirectionstoteachers.Byexaminingthedifferencesin
readingprocessesusedbystudentswhoscorehighonameasureofonlinereadingcomprehension
andthosewhoscorelowthisstudywillidentifysuccessfulstrategiesusedbygoodonlinereaders.
Thisdatacanbeusedbyclassroomteacherswouldwanttomodelandteachtheprocessesusedby
goodreaders.
14
CHAPTERII
LITERATUREREVIEW
Introduction
Thepurposeofthismixedmethodsstudywastoinvestigatetherolethatbackground
knowledge,criticalevaluationofinformation,andareader’sdispositionsplayinpredictingonline
readingcomprehensionperformanceaftercontrollingforverbalintelligence.Thesevariableswere
investigatedintwodifferentmodels.Thecomprehensiontaskstookplaceineitheralessrestricted
oramorerestrictedinformationspace.Thischapterdefinestherelevanttheoreticalperspectives
andexplainshowtheprinciplesofeachtheoreticalperspectivemightshapethepredictionsofthe
variablesineachmodel.FinallyIexploretheliteratureondefiningandmeasuringeachvariable
includedinthestatisticalmodels.
TheoreticalPerspectives
Thisstudyisframedwithinamultiplerealitiesperspective(Labbo&Reinking,1999).This
perspectivesuggeststhatresearchbenefitswhenweframeourworkwithinmultipletheoretical
frameworkssothatwemightcapturemoreofthecomplexityandrichnessthatsurroundsissuesof
literacyandtechnology.Accordingly,thisstudyembracestwodifferenttheoreticalframeworks:
cognitiveflexibilitytheory(Spiro,2004)andnewliteraciesofonlinereadingcomprehension(Leu
etal.,2004).
Bothperspectiveshavehelpedidentifyelementshypothesizedtobeessentialtoreadingin
onlinespaces.Specifically,CognitiveFlexibilityTheoryhassuggestedtheroleofbackground
knowledgeanddispositionsarecentralforstudentstodevelop“advancedwebskillsandopen
mindsets.”(DeSchryverandSpiro,2010).Anewliteraciesofonlinereadingcomprehension
perspectivehassuggestedthatcriticalevaluationofonlineinformationanddispositionswouldalso
becentraltoonlinereadingcomprehension.InthissectionIexploretheliteraturearoundeachof
15
thetheoreticalperspectives.Ithenexplorehowthesetheoriesinfluencethedependentand
independentvariablesselectedforthisstudy.
CognitiveFlexibilityTheory
Cognitiveflexibilitytheory(Spiro,Feltovich,Jacobson,&Coulson,1991)isaconstructivist‐
learningframework,whichbuildsonpreviousschematheories(i.e.Campione,Shapiro,Brown,
1995;Paris,Wasik,Turner,1991).Cognitiveflexibilitytheoryhasshapedtheprediction,inthis
study,ofhowbackgroundknowledgeanddispositionswouldbeexpectedtofunctionwithinmore
andlessrestrictedinformationspaces.
Cognitiveflexibilitytheorysuggeststhatlearningontheweb,especiallywhenmoving
beyondfactfinding,requiresadvancedskillsandopenmindsets(DeSchryver&Spiro,2010).In
otherwords,tasksthattakeplaceinmorerestrictedinformationspacessuchasfindingatrain
schedulewouldplacedifferentcognitivedemandsonthereaderthananonlineinquiryaroundthe
MiddleEastpeaceprocess,whichtakesplaceinalessrestrictedinformationspace.Thereviewof
cognitiveflexibilitytheoryinformedthevariablesofinterestselectedforthisstudy:onlinereading
comprehension,backgroundknowledge,criticalevaluation,anddispositions.
Cognitiveflexibilitytheoryandonlinereadingcomprehension.Ithasbeenarguedthat
Cognitiveflexibilitytheoryisawell‐suitedperspectivefornonlinearlearninginvolvingrandom
accesstechnologysuchastheInternet(Spiro&Jehng,1990).BasedontheprinciplesofCognitive
flexibilitytheory,onlinereadingcomprehensionmaybedefinedastakingplaceinanill‐structured
context.Ill‐structureddomains,suchastheInternet,requireflexiblelearningbecausetheyarenot
guidedbygeneralizablerules(Spiro,Vispoel,Schmitz,Samarapungavan,&Boerger,1987).Inother
wordsthereisnoonewaytonavigateonlinetexts.Insteadoffollowinglinearpagesstudentsbuild
thetextstheyreadbychoosinghyperlinks(Eagleton&Dobler,2006).Thus,onlinetextsrequire
readerstoactivelyconstructmeaningwithnovelskillsandstrategiesthatdonotapplyto
traditionaltext,inmorestructureddomains(Snow,2002).
16
Cognitiveflexibilitytheoristshavesuggestedthesenewskillsarepartofadomainof
“advancedwebexploration”(Deschryver&Spiro,2010)thatneedtomovebeyondsimplysearch
foranswers.Thesewiderangingsearchesunfoldusing“learner‐initiated,complex,reciprocally
adaptive(LICRA)techniques”(p.4).Theseadvancedtechniques,foundinthoseadeptatonline
readingcomprehension,canbeusedfordeeplearninginaspaceofunfetteredinformationand
access(Deschryver&Spiro,2010).
Cognitiveflexibilitytheoryandbackgroundknowledge.CognitiveFlexibilityTheory
explorestheflexibleuseofbackgroundknowledge.Infactcognitiveflexibilitytheoristsdefine
backgroundknowledgeassomethingthatisconstructedbasedonthesituationratherthan
retrieved(Spiro,Feltovich,Jacobson,Coulson,1992).Whenonline,thereaderissituatedinan
environmentwherethosewithadvancedskillscanaccessunlimitedknowledge.
CognitiveFlexibilityTheorywouldsuggestthatinill‐structureddomainssuchasonline
readingcomprehensionactivationofbackgroundknowledgebecomesmoreproblematicastextual
featureschangeonacase‐by‐casenatureasstudentsreadonline(Spiro&DeSchryver,2010).
Readersmaynolongerrelyontheirtemplatesofwhatwebsitesorargumentslooklike.Infactan
overrelianceonbackgroundknowledgemayleadtogreaterdifficulty.
Theaccesstounlimitedamountsofnon‐linearinformation,accordingtotheprinciplesof
cognitiveflexibilitytheory,alsohasimplicationsforbackgroundknowledgeusewhilereading
online.SpiroandDeschryver(2010)arguethatthenooneknowstherolebackgroundknowledge
willplayinaworldwithsomuchexternalmemorystorage.
Thus,studentswhohaveadvancedskillsforreadingonlinemaybeabletousetheInternet
toovercomealackofbackgroundknowledge.Conversely,studentswithhighbackground
knowledge,butlowonlinereadingcomprehensionskillsmaynotbeabletoactivelyconstruct
backgroundknowledgeduringInternetinquirytasks.Finally,studentswhooverlyrelyontheir
17
backgroundknowledgeoftextsinonlineenvironmentsmaymakemistakesasthesetemplatesdo
notalwaystransfertonon‐linearspaces(Spiro&Jehng,1990).
Therefore,inordertoenrichourunderstandingofmodelsofonlinereadingcomprehension
itisimportanttostudytheroleofbackgroundknowledge.Furthermore,cognitiveflexibilitytheory
wouldsuggestthatbackgroundknowledgewillbehavedifferentlyinmodelsofonlinereading
comprehensiondependinguponhowrestrictedtheinformationspaceisforanytask.Intaskswith
lessrestrictedinformationspacesstudentsmayhavetorelymoreontheir“advancedwebskills”
ratherthantopicalbackgroundknowledge.Inmorerestrictedinformationspaces,topical
backgroundknowledgemaybemoreimportantthan“advancedwebskills.”Thusbackground
knowledge,whilesignificantinbothmodels,wouldpredictmoreofthevarianceinthemore
restrictedmodelasstudentswillneedtorelyontopicalknowledgeratherthan“advancedweb
skills.”
Cognitiveflexibilitytheoryandcriticalevaluation.Theill‐structurednatureofonline
textsisamajorsourceofthechallengesinvolvedinthecriticalevaluationofwebsites.Reihand
Belkin(1998)notedthelackofqualitycontrolleadstotheineffectivenessoftraditionalstrategies
toevaluateanonlinetext.Furthermore,FlanniganandMetzger(2000)commentedthateditorsdo
notvetwebsites,leadingtonewchallenges.Finally,Coiro(2003)describedtheinconsistencyof
textfeaturesanddescribedthetextsontheInternetasintertwinedwithhiddeneconomic,social,
personal,andpoliticalagendas.Thesumofthesechallengesindicatesthatmanyofthestrategies
readersaretaughttocomprehendtraditionaltextsmaynotrigidlytransfertotheill‐structured
natureofonlinetexts.
Secondwhenlearnerstrytoapplyasetofrigidstrategiestoanill‐structureddomain,
errorsofoversimplificationoftenoccur(Spiroetal.,1988).Oversimplificationofknowledgemay
leadtoerrorswhenevaluatingwebsites.Forexamplestudentsmayjudgeawebsiteusing
superficialcontent(Coiro,2011).Thismaybebecauserigidstrategyinstructiontaughtforprinted
18
materialsmayleadtoerrorsofoversimplificationwhenreadingonline.Furthermore,cognitive
flexibilitytheorywouldassertthatastabletaxonomyofskillswouldnotguaranteethetransferof
skillsinnewdomainsbecauseoferrorsinoversimplification.Thismayexplainwhyarecentreview
ofcriticalevaluationresearch(Metzger,2007)foundnoevidencethatchecklists,acommon
interventiontoteachcriticalevaluation,improvedjudgments.
Biasinamentalrepresentationcanalsoleadtoalackoftransferofskillsbetweenill‐
structureddomainsontheInternet.Thereforewhatapersonbelievesaboutatopicmayinfluence
howtheyjudgesources.Forexample,DamicoandBaildon(1998),usingthink–aloudprocedures
duringandanInternetInquiryunitofMexican‐Americanmigration,foundthatbeliefsabouta
domainorsubjectinfluencedtheevaluationofclaimsandevidence.
Thus,cognitiveflexibilitytheorywouldsuggestthatcriticalevaluationscoreswillbe
significantpredictorsinboththelessrestrictedandthemorerestrictedmodels.Studentswho
scorewellonbothoftheORCAassessmentsandthemeasureofcriticalevaluationskillsmayhave
“advancedwebskills.”Conversely,studentswhoscorelowonthecriticalevaluationmeasuremay
bemakingerrorsofoversimplificationorhavebiasintheirmentalrepresentation.
Cognitiveflexibilitytheoryanddispositions.AccordingtoCognitiveFlexibilityTheory
dispositionsofthelearnermayalsobeimportantelementstoincludeinamodelofonlinereading
comprehension.Infacttheoristshavesuggestedan“openmindset”isrequiredforonlinereading
comprehension(Spiro,2004).Afterall,accordingtocognitiveflexibilitytheorists,learnerscannot
betaughttosimplyapplyrigidviewsofknowledgetomultiplesituations.Theymustunderstand
thecomplexnatureofknowledgeandbeabletousenovelwaystolearn.Furthermorecognitive
flexibilitytheoristshavenotedbiasinmentalrepresentationcanleadtoalackoftransferofskills
betweenill‐structureddomains(Spiroetal.,1992).Inotherwordsifreadersdonothaveanopen
epistemologicalbeliefaboutknowledgeconstructiontheymaynotsucceedinanill‐structured
19
spacesuchastheInternet.Theseprinciplessuggestdispositionsarecriticaltoonlinereading
comprehension.
Therefore,cognitiveflexibilitytheorywouldsuggestthatotherfactorsaffectperformance
beyondbackgroundknowledgeandcomprehensionskills.Infactcertainlearnersmayhavehabits
ofthoughts(Spiro,2004)ordispositionsthatallowformoreflexiblelearning(Leu,Kinzer,Coiro,&
Cammack,2004).Thisconstructof“openmindsets”isverysimilartothefivescalesmeasuredin
thedispositioninstrument:reflectivethinking,flexibility,persistence,criticalstance,and
collaboration.Thus,studentswithpositivedispositionstowardsreadingonlinemayhavegreater
successinbothlessrestrictedandmorerestrictedinformationspaces.However,accordingto
cognitiveflexibilitytheoryhavingpositivedispositionstowardonlinereadingcomprehension
wouldexplainagreateramountofvarianceinlessrestrictedinformationspacesthanitwouldin
morerestrictedinformationspaces.Thisisbecausealessrestrictedinformationspacewould
requiremoreflexiblelearning,persistence,andagreatercriticalstanceinordertosiftthroughthe
unfetteredinformationandunstructuredsearchesoftheopenInternet.(Schryver&Spiro,2008).
NewLiteracies
Thisstudyisframedwithinabroadperspectiveofanupper‐casetheoryofNewLiteracies
(Coiroetal.,2008)aswellasamorespecific,lower‐casetheoryofthenewliteraciesofonline
readingcomprehension.Anupper‐caseNewLiteraciestheoryisusedtocapturecommonalities
amongdiverseareasofinquiryinthisareaincludingworkinsocialpractices(Street,2003),
Discourses(Gee&Green,2007),comprehension(Castek,2008),andotherareas.Fourcommon
principlescurrentlyappeartodefineanupper‐casetheoryofNewLiteracies(Coiroetal,2008):
ICTsrequireustobringnewpotentialstotheireffectiveuse.
Newliteraciesarecentraltofullcivic,economic,andpersonalparticipationina
globalizedcommunity.
Newliteraciesaredeicticandchangeregularly.
20
Newliteraciesaremultiple,multimodal,andmultifaceted.
WithinthisbroadlyconceivednotionofNewLiteracies,manyareactivelypursuingmore
specificareasofresearch,informedbyseparatelower‐casetheoriesofnewliteracies(Leu,Kinzer,
Coiro,Castek,&Henry,inpress).
Newliteraciesofonlinereadingcomprehension.Thenewliteraciesofonlinereading
comprehension(Leuetal.,2009),areonesuchlower‐casetheory(Coiroetal.,2008).This
perspectivedefinesonlinereadingcomprehensionasaproblem‐basedlearningprocess,which
includestheskills,strategies,anddispositionsrequiredtolocate,evaluate,synthesize,and
communicateonlineinformation.Thisperspectivesuggeststhattheshiftingnatureofhowtextsare
constructed,evaluated,andaccessedrequiresustoaltertraditionalmodelsofreading
comprehension(Leuetal.,2004).Thetheoryofnewliteraciesofonlinereadingcomprehension
informedtheinclusionoftheindependentvariablesofbackgroundknowledge,criticalevaluation,
anddispositions.
Newliteraciesofonlinereadingcomprehensionandbackgroundknowledge.Thereis
noconsistentbodyofevidenceexplainingtheroleofbackgroundknowledgeduringonlinereading
comprehension.Whileempiricalstudiesofonlinereadingcomprehensionhaveincluded
backgroundknowledgeasavariableinmodelsofonlinereadingcomprehensionresearchershave
justbeguntoexplorethetheoreticalimplications.Overallsomearguethatbackgroundknowledge
willplayadiminishedrole(Coiro,2011).Others(Hartman,Morsnik,&Zheng,2010)arguethatnew
formsofbackgroundknowledgewillbereprioritized.
Coiro(2011)suggestedthatresearchersmustrethinktherolethatbackgroundknowledge
playsduringonlinereadingcomprehension.Shefoundthat“whiletopic‐specificbackground
knowledgeusuallyplaysasignificantroleinmostofflinereadingtasks,itappearedtoplaya
relativelyminorroleinaseriesofthreeonlinereadingtasks,”includedinherstudy(Coiro,2007p.
262).Coirosuggestedtheimportanceofbackgroundknowledgemaychangebasedonthe
21
informationspace.Specifically,Coiroarguedthatthepossibilityexiststhat“backgroundknowledge
isindeedafunctionofthespecificityofonlinetasksorthenon‐linearnatureofonlinetask”(Coiro,
2007p.246).Theseresultssuggestthatinlessrestrictedtasksstudentsmayfinditmoredifficultto
relyontopicalbackgroundknowledgeandbecomemoredependenton“advancedwebskills.”
Hartmanetal.(2010)suggestnewformsofknowledgearerequiredforcognitivemodelsof
onlinereadingcomprehension.Thisnewknowledgeaddsnewcomplexitiesratherthansupplants
traditionalviewsofknowledgethatincluded:declarative,proceduralandconditionalknowledge
(Paris,Wasik,andTurner,1990).
Hartmanetal.(2010)arguethatonlinereadingcomprehensionrequiresgoalknowledge,
locationknowledge,andidentityknowledge.Goalknowledge,orknowingwhy,providesstudents
withacontinuedsenseofpurposeduringonlinereadingcomprehensiontasks.Location
knowledge,orknowingwhere,referstoknowingthelocationofsearchenginefeaturesandthe
basicsofInternetsearching.Identityknowledge,orknowingwho,isknowledgeofthebasicsofhow
authorsconstructandrepresentonlineidentities.
Thereforeatheoryofnewliteraciesofonlinereadingcomprehensionwouldsuggestthat
priortopicalknowledgemaynotcontributethesameamountofvariancetomodelsofonline
readingcomprehensionastheyhaveinthepast.Instead,asstatedbyCoiro(2011)priortopical
knowledge,mayhaveareducedroleor,asstatedbyHartmanetal.(2010)newformsof
backgroundknowledgewillbecomemoreimportant.Basedonthesetheoriespriortopical
knowledgewillplayastrongerroleinmorerestrictedenvironmentsasstudentswillneed
increasedbackgroundknowledgetomakemorefinitejudgments.
Newliteraciesofonlinereadingcomprehensionandcriticalevaluation.Whileanew
literaciesofonlinereadingcomprehensionperspectivesuggeststhatquestioning,locating,
evaluating,synthesizing,andcommunicationareallcentralconstructsforinvestigation,only
criticalevaluationofonlineinformationwaschosenforthisstudy.Therewereseveralreasonsfor
22
thisdecision.First,duetotheself‐directedtextconstruction(Coiro&Dobler,2007)thejudgingof
therelevanceofsources(Braaschetal.,2009)isfundamentalwhenreadingonline.Asreaders
selecttextstoreadtheymustconstantlyjudgetherelevancyandcredibilityofsources.Thus,
criticalevaluationbecomesespeciallyimportantduringonlinereading.Inaddition,whilesome
workisbeginninginthisarea(Damico&Baildon,2007;Kiilietal.,2008;Quintana,Zhang,&
Krajcik,2005)weknowlittleabouttheroleofcriticalevaluationduringonlinereading
comprehension.
Theprinciplesofnewliteraciesofonlinereadingcomprehensionwouldsuggestthata
measureofcriticalevaluationskillswouldexplainasignificantamountofthevarianceinboth
restrictedandlessrestrictedinformationspaces.However,lessrestrictedinformationspaces
requiregreaterself‐directedtextconstructionwhichmakesthecriticalevaluationoftextsmore
important.Inalessrestrictedinformationspacetheparticipantshavetobuildthetextwithno
guidanceandhavetoconsidertherelevancyandreliabilityofthesources.Inthemorerestricted
tasktheparticipantsknowthesourcetheyaretryingtolocate.Thereforereaderswithgreater
criticalevaluationskillsmayhavehighersuccessinlessrestrictedspacesandreaderswithless
proficientskillswillperformworse.Thus,eventhoughatheoryofonlinereadingcomprehension
predictsthatcriticalevaluationskillswillbeasignificantpredictorinbothmodels,thescoreson
thecriticalevaluationmeasuremayhavemorepredictivevaluewithinlessrestrictedinformation
spaces.
Newliteraciesofonlinereadingcomprehensionanddispositions.Anewliteraciesof
onlinereadingcomprehensionperspectivewouldalsosuggestthatdispositionsmaybecentralto
readinginbothlessrestrictedandmorerestrictedonlineinformationspaces.O’ByrneandMcVerry
(2008)identifiedfivelearningdispositionsthatarecentraltoonlinereadingcomprehension:
reflectivethinking,takingacriticalstance,persistence,flexibility,andcollaboration.Whilethese
23
dispositionsmaybeimportantforalllearningtaskstheill‐definednatureoftheInternetmaymake
thesedispositions,orhabitsofmind(Spiro,2004),morecentraltoonlinereadingcomprehension.
Accordingtotheprinciplesofonlinereadingcomprehensionreaderdispositionswould
predictsignificantvarianceinbothlessrestrictedandmorerestrictedinformationspaces.Research
isemergingthatdemonstratesaffectivevariablesarepositivelyrelatedtostrategyusewithin
onlinereadingtasks(Dwyer,2010;Tsai,2004;Tsai&Lin,2004;Hofman,Wu,Krajcik,andSoloway,
2003).InfactLeuetal.(2004)suggestthatnewdispositionsarecentraltomeaningmakingin
onlineenvironments.Thereforescoresonaselfreportmeasureofdispositionsofonlinereading
comprehensionareexpectedtomakeasignificantpredictioninmodelsusingbothlessrestricted
andmorerestrictedinformationspaces.
Thispredictiondiffersfromthatofcognitiveflexibilitytheory.Basedonthetheoriesof
cognitiveflexibilitytheorydispositionswouldbeasignificantpredictorinbothmodels,butabetter
predictorinthelessrestrictedmodel.Basedonthenewliteraciesofonlinereadingcomprehension
dispositionswouldbeasignificantandstrongpredictorinbothmodels.Thisdifferenceis
attributedtothegreateremphasisnewliteraciesofonlinereadingcomprehensionplaceson
dispositionsinalltheoreticaldefinitions.
Summaryoftheoreticalperspectives.Twotheoreticalperspectiveswereusedtoguide
thisstudy:cognitiveflexibilitytheoryandnewliteraciesofonlinereadingcomprehension.These
perspectiveshelpedtoguidetheselectionofvariablesofinterest.Furthermoretheywereusedto
makepredictionsabouthowthevariablesofinterestwillbehaveinmodelsofonlinereading
comprehensionthatusebothalessrestrictedinformationspaceandamorerestrictedinformation
space.
Inthisstudypredictionsfromthetwotheoreticalperspectivesarecloselyaligned.Thetwo
theoreticalperspectivesCognitiveFlexibilityTheoryandnewliteraciesofonlinereading
comprehensiontheoryhavebeenusedtoexplorerecentshiftsinourliteracypractices.Themost
24
importantalignments,intermsofthisstudy,areasignificantbutreducedroleforbackground
knowledgeandtheincreasedimportanceofcriticalevaluation.Theoneareaofdisagreementwas
intheareaofdispositions.Basedontheprinciplesofcognitiveflexibilitytheory,dispositionswould
notplayassignificantaroleinthemorerestrictedinformationspaceasthestructurednatureof
thetaskwouldlimitthepotentialfordeeplearningontheWeb.Conversely,atheoryofnew
literaciesofonlinereadingcomprehension,wouldsuggestthatdispositionsarejustascriticalinthe
morerestrictedspacesastheyareinthelessrestrictedspaces
Cognitiveflexibilitytheoryledtoanumberofpredictions:(a)backgroundknowledge,while
significantinbothmodels,wouldnotbethestrongestpredictorineithermodel;(b)scoresona
measureofcriticalevaluationwouldbesignificantinbothmodels,butpredictmoreofthevariance
inthelessrestrictedmodel;and(c)scoresonaself‐reportofdispositionsmeasurewouldexplaina
significantamountofvarianceinbothmodels,butwouldbeabetterpredictorinthelessrestricted
model.
Inthisstudy,anewliteraciesofonlinereadingcomprehensiontheoryledtoanumberof
predictions:(a)backgroundknowledgewillbesignificant,butnotbeasastrongofapredictorasin
offlinecomprehensionresearch;(b)criticalevaluationskillswillbesignificantinbothmodels,but
explaingreatervarianceinthelessrestrictedmodel;and(c)scoresonaself‐reportofdispositions
measurewouldexplainasignificantamountofvarianceinbothmodels.
PriorResearch
Inadditiontothesetheoreticalperspectives,severalareasofresearchalsoinformedthe
designofthisstudy:(a)onlinereadingcomprehension,(b)backgroundknowledge,(c)critical
evaluation,(d)dispositions,and(e)verbalintelligence.Eachwillbeexploredbelow.
ResearchInOnlineReadingComprehension
Researchsuggeststhatonlinereadingcomprehensionisdifferent,andmaybemore
complex,thantraditionalreadingcomprehension(Coiro&Dobler,2007;Hartmanetal.,2010;Leu
25
etal.,2009).Thesestudieshavefoundseveralcommonresults:scoresonmeasuresofonline
readingcomprehensionarenotisomorphicwithofflinereadingcomprehensionmeasures(Coiro,
2011,Leuetal.,2005);newstrategiesarerequiredinonlinespaces(Chen,2011;Schmar‐Dobler,
2003);newknowledgeandattitudesarerequiredforonlinereadingcomprehension(Bilal,2001;
2002;Deschryver&Spiro,2010);thenatureoftheinformationspacemayaffectthenatureof
onlinereadingcomprehension;andassessmentsofonlinereadingcomprehensionhaveahistoryof
beingvalidandreliable.
Scoresonmeasuresofonlinereadingcomprehensionarenotisomorphicwithoffline
readingcomprehensionmeasures.Leuetal.(2005)suggestthatreadingcomprehensionand
onlinereadingcomprehensionarenotisomorphic.Theyfoundnosignificantstatisticalcorrelation
amongtheDegreesofReadingPower(TouchstoneAppliedScienceAssociates,2004)testanda
validatedmeasureofonlinereadingcomprehensionadministeredto89seventhgradestudents.
Theonlinereadingcomprehensionmeasure(ORCA‐Blog)requiredstudentstolocate,evaluate,
synthesize,andcommunicateinformation.TheDegreesofReadingPower(DRP)test,isacriterion‐
referencedmeasureusedwidelyasameasureofreadingcomprehension.
Coiro(2011)foundthatknowingastudent’sonlinereadingcomprehensionability
predictedasignificantamountofvarianceoverandaboveofflinereadingcomprehensionand
backgroundknowledge,butanadditional16%ofindependentvariancewascontributedby
knowingstudents’onlinereadingcomprehensionability.Participantsincluded118seventh‐grade
studentsfromConnecticut.StudentswereadministeredtwodifferentversionsoftheORCA,one
usedasadependentvariableandtheotherasanindependentvariable.Participantsalsocompleted
abackgroundknowledgemeasure.ScoresontheConnecticutMasteryTest(StateofConnecticut,
2010)wereusedasameasureofreadingcomprehension.Thedatasuggestthatadditionalskillsare
requiredforonlinereadingcomprehension,beyondthoserequiredforofflinereading
26
comprehension.Thesestudiessuggestthattheremaynotbeahighcorrelationbetweenofflineand
onlinereadingcomprehension.
Newstrategiesarerequiredinonlinespaces.Researchhasalsofoundthatonline
readingcomprehensionmayrequirenewskillsandstrategies.Schmar‐Dobler(2003)investigated
strategyuseamongfifthgradersastheysearchedforbothexplicitandimplicitinformationonthe
Internet.Datasourcesincludedobservations,think‐aloudprotocols,andpost‐readinginterviews.
Schmar‐Doblerconcludedstudentsusedmanyofthesamestrategiesusedduringtraditionalonline
readingactivities.However,shenotedthatnewnavigationalstrategieswererequiredtoreadin
onlinespaces.Thisstudysuggeststhatonlinereadingcomprehensionrequiresnewstrategies.
Anotherstudy,usingqualitativethinkaloudmethodswith12proficientsixthgrade
students,concludedthatonlinereadingcomprehensionandofflinereadingcomprehensionare
similar,butonlinereadingcomprehensionwasalsomorecomplex(Coiro&Dobler,2007).Inthis
studystudentscompletedtwoseparateonlinereadingcomprehensiontasks.Thefirsttaskhad
studentsreadwithinawebsite.Thesecondtaskhadstudentsanswerveryspecificquestionsusing
asearchengine.Datasourcesincludedthink‐aloudprotocols,fieldobservations,andsemi‐
structuredinterviews.Theauthorsconcludedthatonlinereadingcomprehensionrequireda
processofself‐directedtextconstructionthataddsadditionalcomplexitiestotraditional
comprehension.
Deschryver(2010)conductedathinkaloudprocedurewithadvancedweblearners,skilled
collegeagestudentsandcomparedtheseresultswithstudiesinvestigatingschoolagechildren.
DeschryverconcludedthatdifferencesexistamongexpertusersoftheInternet.Specifically,
advancedlearnerscansynthesizenewlearningthatmovesbeyondwhatisalreadyknownabouta
topic.
Chen(2010)usingqualitativemethodscomparedtheonlinereadingcomprehension
strategiesofupperelementarystudentswithandwithoutlearningdisabilities.Datasources
27
included:surveys,structuredmetacognitiveinterviews,observations,readingcomprehension
activities,andonlinesearchtasksthatwereadministeredto119studentsinthefifthandsixth
grades.Chenconcludedthatthelackofnavigationalstrategiestodealwithnon‐linearreading
impededcomprehension.
Newknowledgeandattitudesarerequiredforonlinereadingcomprehension.
Researchhasalsodemonstratedthatnewknowledgeandattitudesmayberequiredforonline
readingcomprehension.Bilal(2000;2001)workedwithapproximately25seventhgradersto
explorethecognitive,affective,andpsychomotordomainsoflearningasparticipantssearchedthe
InternetusingYahooligans.BilalcreatedaWebTraversalMeasuretoquantifysearchbehaviors.
Otherdatasourcesincluded:screenshotrecordings,teacherassessmentsofstudentattributesand
studentexitinterviews.Bilalconcludedthatabilitytorecoverfrom‘‘breakdowns,’’navigational
style,andone’sfocusontaskwerekeytosuccessfulonlinereading.Thisindicatesthatonline
readingperformancemaygobeyondskillsandrequirelearnerdispositions.
Insummarythesestudiespresentagrowingcorpusofworkthatreadingcomprehensionis
fundamentallydifferentandmorecomplexinonlinespaces.Specificallynewskills,knowledge,and
dispositionsareneeded.Acrossallofthestudiestherewasanincreasednoteoftherolenavigation
playsduringreadingcomprehension.Manyofthestudiesnotedtheimportanceofgoalknowledge
(Bilal,2000;2001;Schmar‐Dobler,2003)throughselfdirectedtextconstruction(Coiro&Dobler,
2003).
Thenatureoftheinformationspacemayaffectthenatureofonlinereading
comprehension.Severalstudieshaveusedvariouslevelsofarestrictedinformationspaceduring
onlinereadingwithoutcontrollingforthepotentialconsequences(Chen,2010;Coiro&Dobler,
2007;Deschryver,2010;Leuetal.,2005;Schmar‐Dobler;2003).Todatenostudyhasexaminedthe
differencesinperformancebasedontherestrictednatureoftheinformationspaces.
28
Themajorityofstudiesusedamorerestrictedinformationspacetomeasureonlinereading
comprehension.Schmar‐Dobler(2003)usedamorerestrictedinformationspacebylimitingthe
tasktoquestionsfromclass.CoiroandDobler(2007)usedarestrictedspaceofjustasingle
websiteanddirectedquestions.Leuetal.(2005)usedamorerestrictedspacebyhavingstudents
lookforspecificanswerstofocusedInternetqueries.Theassessmentsinthesestudieshad
studentsanswerspecificquestionsandevaluatespecificpagesratherthanconductsearchersfor
topicalinformation.
Fewstudiesutilizedlessrestrictedinformationspaces.Deschryver(2010)usedaless
restrictedinformationspacebyhavingstudentsconductanopen‐endedsearch.Chen(2010)useda
morerestrictedspaceforthinkaloudactivitiesandalessrestrictedspaceforhisquantitative
measures.
Whilepreviousstudieshaveusedavarietyoftypesofinformationspaceintheirresearch,
therearenostudiesontherolethatmoreorlessrestrictedinformationspaceplaysduringonline
readingcomprehensiontasks.Thisstudyseekstoexplorestudentperformanceinbothless
restrictedandmorerestrictedspaces.TherestrictednatureoftheInternetinquirytaskmay
influencetheskills,strategies,anddispositionsofthereader.
Assessmentsofonlinereadingcomprehensionhaveahistoryofbeingvalidand
reliable.Validandreliableassessmentsofonlinereadingcomprehensionhavebeenusedin
previousworkthatattemptedtocapturethenatureofreadingonlinethroughperformancebased
assessmenttasks.ThepurposeofeachORCAmeasurethathasbeenpreviouslyusedwastocapture
“real‐time”onlinereadingproductsandprocessesduringreadingontheopenInternet,adynamic
andunboundeddigitalinformationenvironment.TheseassessmentsincludedORCA‐Instant
Message(ORCA‐IM),ORCA‐Blog,ORCA‐ScenariosIandII,ORCA‐Iditarod,andORCA‐IditarodRevised.
EachoftheseassessmentshasbeenshowntobevalidandreliableasshowninTable2.1.
29
Table2.1Descriptive,Validity,andReliabilityofScoresonORCA‐OpenassessmentsInstrument/#Items
ScoreRanges
N Validity*
Reliability**
ORCA‐IM(12) 3‐36of38 12 66.5% .85‐.95ORCA‐Blog(10) 0‐30of32 89 59.2% .84ORCAScenarioI(20) 0‐56of60 120 51.7% .92ORCAScenarioII(20) 0‐56of60 120 44.1% .91ORCAIditarod(17) 0‐33of42 220 53.1% .88ORCAIditarodRevised(13) 0‐20of30 373 41.6% .73*%ofvarianceexplainedbyacompositescore,ameasureofconstructvalidity.**Cronbach’salphareliabilitycoefficient,ameasureofreliability.
ThefirstattempttocreateavalidandreliablemeasurewastheORCA‐IM(Leuetal,2005).
TocompletetheORCA‐IMstudentsworkedindividuallyonfourInternetreadingtasks.Anonline
researcherintroducedtheactivityoverinstantmessage.Asecondresearchersatwiththestudent
andcollectedfieldnotes.WhileORCA‐IM,resultedinasuccessfulprotocolfordatacollectionthe
amountofresourcesitrequired,tworesearchersperadministration,madetheORCA‐IMverylabor
intensive.
ThenextassessmentdevelopedwastheORCA‐Blog(Leuetal,2005).ThegoalofORCA‐Blog
wastorepeatthedesignofORCA‐IMbutwithproceduresthatallowedscalingupthrough
simultaneousadministration.StudentswhocompletedtheORCA‐Bloghadtoanswerthree
informationrequestspostedonablogsitebyfictitiousteachersrequestingonlineresourcesfor
humanbodysystems.TheORCA‐Blog,capturedarangeofachievementlevelsinonlinereading
comprehension.Howeveritdidnotaccountforthemultipledimensionsofcriticalevaluationorthe
synthesisofmultiplesources.TheORCA‐Bloghadstudentsevaluateasinglewebsiteonoverall
reliability.Furthermorestudentswerenotrequiredtointegratemultiplesourcesintheirsynthesis
tasks.Thesearekeyskillsthatneedtobemeasuredduringonlinereadingcomprehension
assessments.
30
Thenextiterationofassessments,ORCA‐ScenarioI&II,wasdevelopedbyCoiro(2007;
2011).Theinstrumentinvolvedtwoparallelmeasuresofonlinereadingcomprehension.Each
instrumentincluded20open‐endeditemsconstructedtomeasurethestudents’readingability
duringthelocation,evaluation,synthesis,andcommunicationofonlineinformation.The
assessmentwasdeliveredoveraprivatequizinterfacethatrequiredindividualusersignon.The
ORCA‐ScenarioI&IIwerebothvalidandreliablebututilizedaveryrestrictedinformationspacein
ordertoensureaparallelitemformat.Thetesttookplaceinaquizinterfaceandstudentswere
givenverytargetedsearchestocompleteorwereprovidedwebsitestoevaluate.
TwoORCA‐IditarodmeasuresweredevelopedaspartofanIESsupportedresearchgrant
project(TeachingInternetComprehensiontoAdolescents[TICAProject],Leu&Reinking,2005).
Theinstrumentsrequiredstudentstouseanonlineassessmenttool,SurveyMonkey.Thetasks
askedstudentstolocate,evaluate,synthesize,andcommunicateinformation.Theinformation
requestedwasveryrestrictedinnature.Forexamplestudentswereaskedtofindsinglefactssuch
as,“WhoholdstheIditarodrecord?”Alsothecriticalevaluationtaskwasrestrictedtoasingle
website.
Summaryofresearchinonlinereadingcomprehension.Todate,researchinonline
readingcomprehensionisadvancing.Yetuniquechallengesremain.Firstwedonotknowhowthe
restrictednatureoftheinformationspacemayaffectonlinereadingcomprehensionscores.
Furthermoreeffortstocapturethecollaborativenatureofcommunicationduringonlinereading
comprehension,suchasonORCA‐Blogcreatedifficultiesforassessmentincludingecological
validityandcumbersomescoring.Effortsweretriedtoutilizeinstantmessagingandblogging.Yetit
wasimpossibletorecreatethecollaborativenatureofthesecommunicationtools.Thescoringfor
onlinereadingcomprehensionassessments,involvedthecodingofvideoscreencasts.Thismeant
eachassessmentwouldtakefiftyminutestoscore.Finallytheshiftingnatureofonlinewebsites
31
threatensthestabilityoftheassessments.Inmanyoftheseassessments,secondaryreplication
studiescannotbecompletedaswebsitesthatwereusedintheassessmentsarenolongeravailable.
Theassessmentsdesignedforthepresentstudytriedtoaccountfromthelessonslearnedin
pastdevelopment.Firstandforemosttwodifferentmodelswillbetestedtoaccountforshifting
informationspaces.Onemodelwillusealessrestrictedspace.Thesecondmodelwilluseamore
restrictedspace.
Anotherimprovementistorequirethereadingofmultiplesources.Inmanyprevious
versionsoftheORCAstudentsdidnothavetoutilizemultiplesources.Thisdoesnotreflecttrue
onlinereadingcomprehension.Inthisstudyboththelessrestrictedandmorerestrictedtasks
requiredthereadingofmultipleonlinesources.
Inaddition,multipleelementsofsourcingskillswereincludedinthecriticalevaluation
items.Inpreviousstudies,criticalevaluationskillswerelimitedtoexaminingjusttheauthor.This
studyalsoincludestheevaluationoftheevidenceusedbyanauthor.
Finallyeffortsweremadetodrawonthecollaborativenatureofonlinecommunication.In
previousORCAversionscommunicationtaskswerelimitedtoblogsorinstantmessaging.Both
assessmentversionsinthepresentstudyrequiredstudentstoreplytopreviouspostsona
discussionboard.Specificallystudentshavetousetheinformationtheylearntorespondtoanother
post.
Researchonpriorknowledge
Therolethatpriorknowledgeplaysintraditional,offlinereadingcomprehensioniswell
known,stable,andsignificant(Chiesi,Spilich,&Voss,1979;Spilich,Vesonder,Chiesi,&Voss,1979).
Howevertheresultsfromrecentstudiesinvestigatingbackgroundknowledgeandonlinereading
comprehensionhaveshownasomewhatmixedpatternofresults.ThisincludesstudiesofInternet
inquiry(Bilal2000;2001;Hill&Hannafin,1997),non‐linearhypertextreading(Tabati&Shore,
32
2005),andonlinereadingcomprehension.(Coiro,2011;).Thesestudiesdonotprovideconclusive
evidencethatbackgroundknowledgeplaysastrongroleinonlinereadingcomprehension.
HillandHannafin(1997)usedacasestudymethodwithfourcollegestudentstoinvestigate
strategyuse.Theyconcludedthatstudents’backgroundknowledgeaffectedmetacognitivestrategy
use.Thisbackgroundknowledgeincludedbothtopicandsystemsknowledge.HillandHannafin
(1997)askedparticipantstolocatematerialsonasubjectoftheirchoosing.Theyreportedthat
previousexperiencewiththeInternetpredictedperformanceonthetask.Inthestudypriortool
expertise,asmeasuredusingaself‐reportfrequencyofusemeasure,wasmorepredictivethan
domainknowledge.Inonlinereadingcomprehensionassessment,familiaritywithavailableWeb‐
basedtoolsmaybetterpredictsuccessthanpriordomainknowledgeandexperience.
TheresultsofbothofBilal’sstudies(2000;2001)foundthatbackgroundknowledgedidnot
influencesuccess.Resultssuggestedneitherdomainknowledgenortopicknowledgeinfluencedthe
success(Bilal2000;2001)ofstudentsatInternetinquiry.Inthe2001study,fourstudentswitha
highermeanscoreofmeasuresoftopicknowledgewereunsuccessfulintheirsearcheswhilenine
studentswhoscoredloweronameasureoftopicknowledgeweresuccessful.Thisindicatesthat
backgroundknowledgemaynotbeascriticalduringonlinereadingcomprehension.
Otherstudies,however,havefoundthatbackgroundknowledgeplayedasignificantrole.
TabatiandShore(2005)conductedastudycomparingthesearchingbehaviorsofexpertsand
novices.Participantsincluded10novices,9intermediates,and10experts.Usingverbalprotocol
andsurvivalanalysis,amethodforanalyzingdatabasedontheoccurrenceofanevent,(Allison,
2010),theyconcludedthatmostsignificantdifferencesinpatternsofsearchbetweennovicesand
expertswerefoundinthecognitive,metacognitive,andbackgroundknowledgestrategies.The
researchersfoundasignificantcorrelationbetweenmetacognitivestrategiesandbackground
knowledge(r=0.54,p=0.003).Specificallytheyfoundsignificantcorrelationsbetweenreflection
33
anddomainknowledge(r=0.45,p=0.01),reflectionandsystemknowledge(r=0.41,p=0.03),
andmonitoringanddomainknowledge(r=0.45,p=0.01).
Finallysomestudieshavefoundthattherolebackgroundknowledgeplaysinonline
readingcomprehensionmaybedeterminedbyastudent’sonlinereadingcomprehensionability.
Coiro(2011)foundthatbackgroundknowledgeexplainedasignificantamountofvariancewhen
enteredintoaregressionmodelpredictingonlinereadingcomprehensionassessment.However
Coiroalsofoundaninteractioneffectbetweenbackgroundknowledgeandonlinereading
comprehensionability.Coiroconcludedthattheinteractionmaysuggestthattopic‐specific
backgroundknowledgeaffectsstudentswithhighandaverageonlinereadingabilitylessthanthose
withlowonlinereadingability.
Thefivestudiesofonlinereadingcomprehensionthatinvestigatedbackgroundknowledge,
eitherwithqualitativeanalysis(Bilal2000;2001;Hill&Hannifin;1997)orstatisticaltesting(Coiro,
2011;Tabati&Shore,2005)havefoundsomewhatcompetingresultsastotheroleofbackground
knowledge.Thisisinsuchstarkcontrasttostudiesoftraditionalreadingcomprehension.Some
researchfoundastrongroleforbackgroundknowledge(Hill&Hannafin,1997;Tabati&Shore
2005)whileotherworkfoundadiminishedroleforbackgroundknowledge(Bilal,2000;2001;
Coiro,2011).Thishasledsomeresearcherstospeculatethattheexpertiseoftheuser,including
theirabilitytousetheInternet(Coiro,2011;Hill&Hannifin,1997),mayinfluencetheroleof
backgroundknowledge.
Measuresofpriorknowledgeinonlinereadingcomprehensionstudies.Therehave
beentwomajormethodsformeasuringpriorknowledgeinstudiesthatexaminedpriorknowledge
andonlinereadingcomprehension:self‐reportsurveysandpriorknowledgequestionnaires.The
typesofpriorknowledgemeasuredincludeddomain,ortopicknowledge,andsystemknowledge,
inotherwordstaskknowledge.
34
HallandHannifin(1997)usedaquestionnairetojudgebothtopicknowledgeandsystem
knowledge.Fourquestionswererelatedtoparticipants'priorsubjectknowledge.Participantswere
askedtoexplaintheirsearchtaskandaskedtoratethemselvesonascaleofnovicetoexpert.
Systemknowledgewasratedusingaself‐reportfrequencymeasureoncommonsearchtools.The
scalewasalongacontinuumofneverusedtodailyuse.
Bilal(2000;2001)usedateachersurveyofstudents’backgroundknowledge.Thesurvey
consistedofquestionsabouttopicknowledge(alligators).Thesurveyalsoaskedaboutdomain
knowledge.Finallyteachersalsoratedstudents’readingability.
TabatiandShore(2005)alsousedself–reportsofself‐efficacytoserveasascorefor
systemknowledge.Participantswereaskedtoratethemselvesinthreeareasofcomputerliteracy:
information‐seekingknowledge,computerknowledge,andknowledgeoftheWeb.Afivepoint
Likertscaleonadimensionofpoortoexcellentwasused.Notopicbackgroundknowledgemeasure
wasused.
Coiro(2011)usedapriorknowledgemeasurethatmovedbeyondself‐reportscores.Asix‐
itemquestionnairewasused.Fourofthequestionsaskedstudentsabouttopic‐specificknowledge.
Twoofthequestionsaskedabouttaskknowledge,specificallyanimationandwebsitereliability.
ItemscoringprocedureswereinformedbyLeslieandCaldwell’s(1995)ConceptQuestionTaskand
WolfeandGoldman’s(2005)measureoftopic‐specificpriorknowledge.Thequestionnairewas
administeredorallyonaone‐to‐onebasis.
Summaryofresearchonpriorknowledgeinbothofflineandonlinereading
comprehension.Priorknowledgeplaysanimportantroleinofflinereadingcomprehension.Its
roleinonlinereadingcomprehensionislessclear.Thediversityintheresultsexploringtheroleof
priorknowledgeduringonlinereadingmayhaveasmuchtodowiththetypesofmeasuresusedto
measurepriorknowledgeastheydowithonlinereadingcomprehension.Thusitwasdecidedto
35
operationalizepriorknowledgeasbackgroundknowledgeforthisstudy.Thisreferstoonlydomain
specificknowledgeabouttheAmericanRevolution.
Acrossthestudiesthemajoritymeasuredpriorknowledgeusingself‐reportmeasures.This
maybeproblematicastheseitemsareopentobias.Studiesthatcomefromlibraryandinformation
sciencefavorsystemknowledge.Studiesfromareadingperspectivefavortopicknowledge.Given
thefocusofthisstudy,themethodsemployedbyCoiro(2011)seemmostpromising.Howeverthe
one‐on‐oneadministrationformatwasnotpractical.Thereforethebackgroundknowledgemeasure
inthisstudydrewonthemethodsusedbyCoirobututilizedanonlineformforsystematicdata
collection.
ResearchonCriticalEvaluation
FoundationalOverview.Avarietyoffieldshaveexploredthecriticalevaluationof
information.Beforebeginningasystematicreviewofresearchofthecriticalevaluationofwebsites,
itisusefultohaveabroadsenseoftheseveralfieldsthathaveprovidedthefoundationforthis
research.Theseinclude:informationprocessing,informationretrieval,designtheory,critical
thinking,anddocumentrepresentation.
Informationprocessing.Theemergenceofthecomputerinthe1950’sledcognitive
sciencetometaphorsofthinkingthatcomparedthemindandthecomputer(Wilson&Myers,
2000).Thuscomputerorientedinformationprocessingmodelsprovidednewwaystothinkabout
thinking(Johnson‐Laird,1988).Informationprocessingtheoriesbelieveinformationisreceived
fromexternalstimuli(websites),itreceivesattention,theinformationisstoredinshorttermor
longtermmemorywhereitiscombinedwithpreviouslystoredinformation,andfinallyaresponse
(judgment)isgenerated.Muchoftheearlyresearchonthecriticalevaluationorcredibilityof
websitesdrawsoninformationprocessingmodels(Tate&Alexander,1996).Specifically,
frameworksofinformationretrievalanddesigntheoryweresituatedininformationprocessing
perspectives.
36
Informationretrieval.Informationretrieval(IR)frameworksencompasscognitive
processesinvolvedinthesearching,organizing,andaccessingofinformation.Muchoftheliterature
fromlibraryscienceadoptsanIRframeworkandexplorescriticalevaluationasacredibility
judgmentthatinvolvesinformationqualityandcognitiveauthority.
TheIRliteratureontheevaluationofwebsitesdrawsonfiveprinciplesofinformation
qualityidentifiedfromliteratureontheevaluationofprinttexts:accuracy,authority,objectivity,
currency,andcoverage(Tate&Alexander,1996).Thesefiveelementsofcredibilityhavebeenquite
stableoverthelastdecade.Forexample,inarecentreviewoftheliteratureMetzger(2007)found
themostconsistentelementsofinformationqualitywerefoundtobe:believability,accuracy,
trustworthiness,bias,andcompletenessofinformation.Thesefiveprinciples,similartothe
characteristicsoutlinedbyTateandAlexander,weredrawnfromTaylor’s(1986)valueadded
modelofjudginginformationquality.
Taylordefinedqualityas“ausercriterion,whichhastodowithexcellenceorinsomecases
truthfulnessinlabeling”(p.62).Taylorpositedthatinformationsystemsandinformationhad
specificvaluessomeweretangibleandcouldbeseen,whileotherssuchasreliabilitywere
intangible.Therewerespecificvaluesthatcouldbebuiltupovertime.Tayloridentifiedfiveofthese
valuesthathaveconsistentlybeenfoundintheliteraturethattakesanIRperspective:accuracy,or
thedegreetowhichtheinformationistrue;comprehensiveness,orthecompletenessofcoverage;
currency,orhowrecenttheinformationisreliability,orthetrustsomeoneputsintheinformation;
andvalidity,orhowusefultheinformationistothetask.
AnotherelementcommontoresearchinlibrarysciencethattakesanIRframeworkis
cognitiveauthority.CognitiveauthorityasdefinedbyWilson(1983)influencesmanyofthe
theoreticalarticles(Tate&Alexander,1996)orstudies(Fritch&Cromwell,2001,Reih,2002;Reih
&Belkin,1998)investigatingthecriticalevaluationofwebsitesthroughrecognitionthat
informationfromdifferentsourceshasvaryingreliability.ReihandBelkin(1998)summarized
37
Wilson’sexternaltestsforcognitiveauthorityoftextsasinvolving:personalauthorityin
recognitionoftheauthor;institutionalauthorityinrecognitionofthepublisher;textualtype
authorityinplacingvalueinthetypeoftext;andintrinsicplausibilityauthorityinplacingauthority
inthecontent.
Designtheory.Someoftheearliestempiricalworkonthecriticalevaluationofwebsites
developedfromresearchersinvestigatingwhatmakespeoplebelievesomewebsitesoverothers
(Foggetal.,2001;Foggetal.,2003;Fox,2006;Fox&Raihne,2002).Designtheoryalignsclosely
withthetheoreticalframeworkfoundinIRsuchasWilson’s(1983)cognitiveauthorityandthe
valuesofinformationqualityidentifiedbyTaylor(1986).Themajortheoreticaldifferenceisinthe
audience.IRresearchfocusedonimprovementsinsystemsandusers.Designtheorylooksto
increasetheperceivedcredibilityoftextsinordertoincreasemarketvalue.
Criticalthinking.Researchinsciencehasoftenusedacriticalthinkingframeworkto
investigatehowreadersevaluatescienceargumentsonline(Brem,Russell,Weems,2001;Graesser
etal.,2007;Sanchez,Wiley,&Goldman,2007).Bremetal.defineargumentationastheemployment
ofcriticalthinkingskillsintheevaluationofspecificclaimsbutalsoframedtheirresearchin
situatedcognition.AccordingtoGraesseretal.criticalthinking,“requireslearnerstoevaluatethe
truthandrelevanceofinformation,tothinkaboutthequalityofinformationsources,totracethelikely
implicationsofevidenceandclaims,andtoaskhowtheinformationislinkedtothelearner’sgoalsand
largerconceptualframeworks”(p.3).Theythensuggestthatcriticalthinkingrequiresacriticalstance
thatrequiresreaderstobesuspectofallinformationtheyencounter.
Documentrepresentation.Theoriesofdocumentrepresentationbuildoffofearlierworkin
thereader’sconstructionofdocumentmodelswithsingletexts(Kintsch,1998;Kintsch,&VanDijk,
1978)buttrytoaccountforthemultiplesourcesreadwhenconductingInternetinquiries.Theories
ofdocumentrepresentationsuggestthatreadersconstructadocumentmodel.Itdefersfromearlier
documentmodelsthatsuggestedatwo‐phaseconstruction–integration(Kintsch,1998)that
38
involvesthetextbase,orinternalmeaning,andthesituationalmodel,whichcombinesthe
informationinthetextwithbackgroundknowledge,byaddingonanadditionallayertheintertext
model(Britt,Perfetti,Sandak,&Rouet,2007).Theintertextmodelincludesinformationabouthow
thetextsarerelatedandinformationaboutcharacteristicsofthesources(Braten,Strømsø,&Britt,
2010.Itistheintertextmodel,andmorespecificallytheskillofsourcing,thatalignswithother
operationalizeddefinitionsofcriticalevaluation.Sourcing“istheskillofgatheringinformation
aboutasourceandusingittoformconclusionsaboutadocument,especiallyconclusionsregarding
credibility”(Britt&Gabrys,2002p.171).
Definingthecriticalevaluationofwebsites.Therearemultipledefinitionsofcritical
evaluation(Coiro,2007)thatdrawonmanydifferenttheoreticalperspectives.Constructsusedto
measuretheseprocessesandjudgmentsvaryacrossdifferentresearchfieldsandinclude:
evaluation,judgment,andcriticality.Iwillusethetermcriticalevaluationtoexploresimilarities
anddifferencesinhowvariousconstructswereoperationalized.Overallstudiesandtheoretical
articlesagreethattheconstructofevaluationdrawsheavilyoncognitiveworkinvestigatingthe
evaluationprocessandcredibilityjudgments.
Evaluation.IndefiningevaluationFitzgerald(1999)drawsonBloom’staxonomy(Bloom,
Engelhart,Furst,Hill,Krathwohl,1956),whichdefinedevaluationasajudgmentinvolvedinthe
evaluationofcriteria,values,andstandards.Krathwohl(2002)expandedontheseideascreateda
two‐dimensiontaxonomythatseparatesknowledge(factual,procedural,conditional,and
metacognitive)andcognitiveprocesses(understand,apply,analyze,evaluate,andcreate).The
revisedtaxonomyalsodefinesthecognitiveprocessofevaluationasthemakingofjudgments
basedonstandardsandcriteriathatinvolvethecognitiveprocessesofcheckingandcritiquing.
Mostresearchhasdefinedtheprocessofevaluationasinvolvingjudgments(Flanagin,
Metzger,&Miriam,2000;Foggetal.,2001;Reih,2002;Reih&Belkin,1998;Zhang&Duke,2007)
orasadecisioninvolvingtheprocessesofevaluation(Flanigan,1999).Eventhoseresearcherswho
39
tookanewliteraciesperspective,whichadaptedamoresociologicalperspective,stillaccountedfor
ajudgmentofclaimsandevidence(Damico&Baildon,1998;Damico&Baildon,2006)ordeveloped
taxonomiesgroundedinthecredibilityjudgmentliterature(Coiro,2007;Coiro&Dobler,2006,
Zawilinskietal.,2007).
Judgment.Theworkdoneinthefieldofjudgment,decision‐making,andchoice(Hogarth,
1987;Goldstein&Hogarth,1997,Rachlin,1989)influencesthedefinitionsofjudgmentusedby
researchersinvestigatingthecriticalevaluationofwebsites.AccordingtoRachlin(1989),“…a
judgmentisalwaysaguideformakingadecision,whichleadstoachoice,whichthenproducesan
outcome”(p.43).Hogarth(1987)identifiedtwotypesofjudgment:predictivejudgmentsand
evaluativejudgments.Researchershaveoftenadaptedthesetwotypesofjudgmentsintheir
explanationsofcriticalevaluation(Reih,2002).
Thepredictivejudgment,orpredictiveinference(Coiro,2007;Coiro&Dobler,2007)
involvesthejudgingoftherelevanceofalink.Thereaderisdecidingifthatlinkwillbeuseful,
makesapredictivejudgmentandfollowswithachoicebyclickingonalink.Atthatpointthereader
wouldthenmakeanevaluativejudgment.
Asstatedearlier,researcherswhoadoptanIRand/oralibraryscienceperspectivelookto
thisevaluativejudgmentasusingbothinformationqualityandcognitiveauthority(Fritch&
Cromwell,2001,Reih,2002;Reih&Belkin,1998)ascriteria.Thecriteriausedbyresearchersto
operationalizeinformationqualityandcognitiveauthoritysharemanycommonelements.
Criticality.Critical,intermsofcriticalevaluationcanmeanmanythingstomanydifferent
people.Intermsofthedefinitionscollectedforthisstudy,criticalreferstothreeseparateentitiesin
termsofeducationalresearch:criticalthinking,criticalreading,andcriticalliteracy.
AsnotedbyFitzgerald(1999),manywritersequatecriticalthinkingwithevaluationwhile
themajorityofresearchersviewevaluationasoneofasubsetofhigherorderskillsinvolvedin
criticalthinking.Intermsofjudgingtext,criticalthinkingis“analyticalthinkingfortheprocessof
40
evaluatingsources”(Hickey,1990p.175).Itinvolvesanalyzingfacts,generatingandorganizingof
ideasdefendingopinions,comparisons,drawinginferences,evaluatingfact/opinion,problem
solving,setofdispositionstodrawonthoseskills(Bremetal.,2001;Coiro,2008;Fitzgerald,1999).
Coiro(2008)usesacriticalthinkinglenstodrawonherdefinitionofcriticalevaluationas“readers
applyingtheircriticalthinkingabilitiesto:(a)question,analyze,andcomparetheresourcesthey
located;(b)judgethequalityofinformationonvariouscharacteristics;and(c)defendtheir
opinionswithevidencefrommultiplesourcesandtheirbackgroundknowledge”(p.47).
Inhiswork,Spache(1964)wrotethatcriticalreadingisasetofskillsthatextendsbeyond
bothfunctionalliteracyskillsandhighercomprehension.Theseskillsincludeinvestigatingsources,
recognizinganauthor’spurpose,distinguishingfactfromopinion,drawinginferences,judgments,
anddetectingpropaganda.Therearemanyparallelsevidentbetweendefinitionsofcriticalthinking
andcriticalreadingfoundinmuchoftheliterature(c.f.Coiro2003,2008;Robinson,1964,Russell,
1963).
Infactmanyliteracyresearchersinthefieldofcriticalreadinghavelongheldthatcritical
readingcannotbeseparatedfromcriticalthinking(Ennis,1962;Wolf,King&Huck,1968).Thisis
evidentinresearchoncriticalevaluationrootedinlibraryinformationsciences,information
retrieval,andmoresocio‐cognitiveviewsofnewliteracies.Manyofthecriticalreadingskillsare
evidentinthecheckliststhatarecommonlyusedtoteachcriticalevaluationofwebsites.Cervetti,
Pardales,andDamico(2001)arguethatthisconnectionisrootedinliberal‐humanisttraditionsthat
areatoddswiththetraditionsofcriticalliteracy.
Unlikecriticalreading,criticalliteracyisrootedmoreinsocio‐culturalviewsofreadingthat
viewresponsetothetextaslessapersonalextractionsofauthor’sintentandmorerootedinsocial,
historical,andculturalpractices(Freebody&Luke,1990;Lankshear&Knobel,1998;Luke,2000;
Mellor&Patterson,2004).Influencesoncriticalliteracyemergedfromavarietyoftraditions.
CriticalsocialtheoriesofNewCriticismschoolsofthought,focusedonusinglanguageresourcesto
41
createamorejustsociety(Cevettietal.,2001).PostcolonialandMarxistviewsrestonthe
assumptionthatweliveinaworldofunequalpowerandtextsareusedtoeitherreinforceor
challengethesepowerstructures(Friere,1970).
Recentlyviewsofcriticalliteracyhavedrawnofideasofpost‐structuralismthatexamine
therelationshipbetweenpower,discourses,andculture(Mellor&Patterson,2004).Howmeaning
isconstructed,isthenconnectedtopowerrelationshipswithinspecificcommunitiesofpractice
(Cevettietal.,2001).LankshearandKnobel(1998)arguethatdefinitionsofliteracymustthen
considerthreeelements:theoperational,thecritical,andthecultural.
Whilecriticalliteracydrawsonavarietyofhistoricalschoolsofthoughtitisunitedinthe
ideathatliteracyisasocialpracticeandnotasetofneutralpsychologicalskills.TothisendLuke
andFreebody(1990)createdthefour‐resourcesmodeltocreatecriticalliteracypedagogy.They
suggestthattherearefournecessary,butinsufficient,rolesreadersmusttakeinapost‐modern
world:code‐breaker,meaningmaker,textuser,andtextcritic.Coiro(2008),whilerootingherwork
incriticalreading,givesapassingnodtocriticalliteracybysuggestingitwillbenecessaryfor
studentstocomprehendtheincreasinglyimage‐drivenwebsites.MurrayandMcPherson(2006)
suggestthattheunderstandingofonlinetextsandwebsiteswillrequiregreatercriticalliteracy
skills.Stone(2007)suggeststhatwemustthinkofcriticalreadingofwebsitesasinvolvingmore
thanevaluationoftruthbutalsolooktoseehowstudentsusepopularwebsitesintheireveryday
lives.
Summary:Drawingonmultipleperspectivesanddefinitionstodefinethecritical
evaluationofwebsites.Itisclearthatthecriticalevaluationoftexts,whetherthedefinitionis
rootedincriticalthinking,criticalreading,orcriticalliteracyinvolvesajudgment.However,
assumingthatauthorityandcredibilityareinherentfeaturesoftextsignorescertaincontextual
elementstoreading.Thepurpose,culture,andpracticesofreaderswillinfluencewhatjudgments
anddecisionstheymake.
42
Idefinecriticalevaluationofwebsitesasacontextualprocessofexamining,adopting,and
changingperspectivesinordertojudgetherelevancyandreliabilityofawebsite.First,critical
evaluationiscontextualbecauseitmayrequirespecificcontentknowledgeormaybeinfluencedby
students’epistemicbelief(Damico&Baildon,2007;Bremetal.,2001).
Second,criticalevaluationisarecursiveprocess.Thecriteriaandjudgmentsinvolvedin
criticalevaluationhavetobecontinuousthroughoutInternetinquiry.Itisnotaspecificstageor
stepinasimpletaxonomy.Criticalevaluationtakesrepeatedquestioning,goalsetting,andavariety
ofmetacognitiveskills(Graesseretal.,2008;Zhang&Duke,2007).
Third,criticalevaluationinvolvestheexamining,adopting,andchangingofperspectives
becausealltextsaresocially,historically,andculturallysituated(Lankshear&Knobel,1998).
Focusingcriticalevaluationassimplyidentifyinganauthor’smessage,intentandbias“privileges
thefactualandobjective”(Fabos,2008p.843).Encouragingstudentstounderstandthedifferent
perspectivesthatshape“truth”willallowthemtojudgerelevancyandreliabilityandmayavoid
errorsofoversimplificationsuchasrejectingasourcebecauseitisbiased.
Finally,thecriticalevaluationofwebsitesinvolvesthejudgmentofrelevancyandreliability
overthejudgmentofcredibilityandauthoritysimplybecausetextsarenotneutralentities.The
reliabilityandrelevancyofasourcemaychangebasedoncontent,contextandpurpose.
ThisdefinitionwasappliedtothedesignoftheCriticalOnlineInformationLiteracies
measureofwebsiteevaluationusedinthisstudy.Theoriginalassessmentincludedscalesto
measurebothrelevancyandcredibilityjudgments.Furthermoreitincludeditemsthatlookedat
theauthor,publisher,credibilityofevidence,andbias.
Methodsforidentifyingcriticalevaluationskillsandstrategies.Themethodologies
usedtostudythecriticalevaluationofwebsitescontinuouslyevolve.Themajorfocusofstudieshas
beentoidentifytheskillsandstrategiesusedtoevaluatewebsites.Threebasicmethodshavebeen
used:
43
1)self‐reportsurveysandquestionnairestounderstandthefrequencyandtypeofskills
studentsuseincriticalevaluation;
2)verbalprotocolanalysis[VPA]andinterviewstoidentifytheskillsandstrategiesusedby
students;and
3)casestudiestoexploresocio‐culturalfactorsofcriticalevaluation.
Self‐reportsurveysandquestionnaires.Theearlieststudiesinvestigatingthecritical
evaluationofwebsitesinvolvedself‐reportsurveysandfrequencyquestionnaires(Flanagin&
Metzger,2000;Foggetal,2001;Fox&Raihne,n.d.Fox&Raihne,2002,Metzger,Flanagin,Zwarum,
n.d.;PrincetonSurveyAssociates,2005).Thegoalsofthesemeasuresweretounderstand:(a)What
makesawebsitecredible?(b)Whatweretheperceivedlevelsoftrustindifferenttypesofmedia?
and(c)Whatwasthefrequencyofskillsused?Themajorityofskillssurveyedinthesestudies
sharedmanysimilarities,fromlistsofskillsusedforprintbasedmedia(i.e.Tate&Alexander,
1996)whichincludedaccuracy,authority,objectivity,currency,andcoverage.
Acrosstheself‐reportstudiesavarietyofmethodswereused.Surveyswerecreatedand
validated(Foggetal.,2001)usingexploratoryfactoranalysis.Othersurveysreliedondescriptive
statisticsfromlargesamplestodrawconclusions(Fox,2006;Fox&Raihne,2002;Princeton
ReviewBoard,2002;2005)whileothersuseddescriptivestatisticsasdependentvariablesfor
furtherstudy(Flannigan&Metzger,2000).SincetheInternetisaratheryoungphenomenonitis
understandablethatsomeoftheearlieststudieswouldrelyonself‐reportsurveystoidentify
trendsandfactorsthatinfluencethecriticalevaluationofwebsites.
Earlyresearchintowebsitecredibility,usingsurveys,hadthegoalofmakingcommercial
websitesmorecredible.Forexample,Foggetal.(2001)createdandadministeredasurveyto1,441
participantsranking51elementsofcredibilityonaLikertscale.Thecreationofthe51itemsthat
mayinfluencecredibilitywentthroughfourrigorousstagesofcontentvalidation.Thesurveywas
44
thenadministeredtoparticipantswhorankedtheitemsonaseven‐pointscalefromlessbelievable
tomorebelievable.Theresearchers,usingvarimaxrotation,andEigenvaluesgreaterthan1.73
foundthatfivefactorsincreasedwebsitecredibility:real‐worldfeel,easeofuse,expertise,
trustworthiness,andtailoring.Incontrasttwofactorshurtperceivedcredibility:commercialism
andamateurism.Fromthesefindingstheresearchersconcludedwithasetofdesignprinciplesthat
couldbeusedtoincreasetheperceivedcredibilityofwebsites.Itmustbenoted,however,that
researchersonlyexplained15.1%ofthevarianceintheirmodel,whichcouldindicatethatthe
predictedfactorloadingsdonotadequatelyexplainthevarianceinthepopulation.
Anotherpurposeofsurveyswastocomparewebsitecredibilitywithothermediatypes
(PrincetonSurveyResearchAssociates,2002;2005;Flanigan&Metzger,2000).ThePrinceton
SurveyAssociates(2002)conductedasurveyof1,051adultsinordertoidentifyareasthatcould
improvewebsitecredibility.Theresultsshowedthatonly29%ofrespondentstrustedwebsites
thatsoldproducts.Inthestudy,33%ofrespondentstrustedwebsitesthatreviewedproductsand
services.Thesescoredweredrasticallylowerthanothermediatypesandprivateentities.
FlaniganandMetzger(2000)alsoconductedasurveytocompareperceivedcredibilityof
websiteswithothermediaandalsotounderstandtheverificationstrategiesusedbypeoplebased
ondemographics,typeofmedia,andInternetexperience.UsingarepeatedmeasureANOVAthe
resultsshowednewspaperswereperceivedtohaveasignificanthigherlevelofcredibilitythan
othermedia(F‐114.12,df‐4,2428,p<.001,~=.1).
InthesurveyconductedbyPrincetonSurveyResearchAssociates(2002)80%of
respondentsstatedthattrustwasveryimportant,80%statedthateaseofnavigationwas
important,and80%statedthatbeingabletoidentifythesourceoftheinformationwasvery
important.Only32%ofrespondentssaiditwasimportanttoknowtheauthorofawebsite.Finally,
65%respondedthatknowingthatthewebsitewasupdatedfrequentlyisveryimportant.Inthe
followupstudy(PrincetonSurveyResearchAssociates,2005)thecredibilityofwebsitesseemedto
45
declinewithonly21%ofrespondentstrustingwebsitesthatsoldproducts.Therewasalsoan
increaseintheelementsofwebsitesimportanttotrust.Therewasastatisticallysignificantincrease
inthenumberofrespondentssayingitwasimportanttoidentifythesourceofinformation,the
frequencyofupdates,andknowingwhoownsawebsite.
Studiesinvestigatinghowpeopletrusthealthinformationonlinehavealsobeenconducted.
In2002,500“healthseekers”wereaskedabouttheirsearchhabitsinphoneinterviews(Fox,
2006).Over50%ofrespondentsreportedcheckingcheckthesourceordateoftheinformation
theyreadonlinewebsitesformedicaladvice“onlysometimes,”“hardlyever,”or“never.”One
quarterseemedtojudgethecredibility“mostofthetime”andanother25%“allofthetime.”49%of
respondentswholookforinformationonmultiplesitesrespondedthatverifyinginformationgave
themalotmoreconfidence,and38%saidthatitgavethemalittlemoreconfidence.Inthe2006
surveyseventy‐fivepercentofrespondentsreportedevaluatinginformationsometimes,hardly,
andnever(Fox,2006).Thisrepresentsa25%increaseoverthe50%ofadultswhoreportednot
beingvigilantorconcernedfromthe2002survey.Theauthorsspeculatethatthisfallintheself‐
reportsofevaluationcouldbecausedbytheincreaseineighteenandplususers(Fox,2006)who
grewupwiththeInternetandwhoplacemoretrustinwebsitecredibility.
Studieswerealsodone,usingself‐reportquestionnaires,tounderstandthecriteriastudents
usetoevaluatewebsitesinacademicsettings.Tillotson(2002)collected499questionnairesfrom
collegeundergraduatestudentsinordertoinvestigateiftheyrecognizedaneedtoevaluate
websites,theextenttowhichtheyevaluatedwebsites,andthetypeofcriteriausedtoevaluate
websites.Theresultsshowedthatstudentshaveanascentapproachtoevaluatingwebsites.He
foundthat38%ofstudentsrespondedtheyhaveneverfoundmisleadinginformationonline;the
averagestudentreportedusingfewerthantwocriteriatoevaluateawebsite.Theresultsshowthat
themajorityofstudentswhoreportedusingcriteriausedsourcereliability.Onlytwenty‐fiveofthe
46
studentsreportedusingtheURLorwebaddresstomakeajudgment.Furthermore,overhalfthe
studentswhousedcontenttoevaluateawebsiteuseditastheonlycriteria.
Thestudiesusingself‐reportmethodologiesareingeneralagreementofthetypesofskills
neededorusedbystudentstoevaluateonlineinformationandthelackoftheseskillsinInternet
usersregardlessofage.Theseskillsdonotdiffergreatlyfromevaluationskillsidentifiedusing
printmedia.Skillscommonacrossalltheself‐reportstudiesinclude:(a)identifyingtheauthoror
sponsor;(b)examiningtheURL;(c)usingformatorappearanceofthewebsite;(d)checkingthe
currencyofinformation;(e)checkingtheaccuracyusingasecondarysource;(f)examiningbias;and
(g)usingcontenttojudgeawebsite.Itmustbenoted,however,thatmanyoftheseskills,because
thesewereself‐reportmeasureswerenotdirectlyobserved.Thestudiesthereforearesubjectto
errorsinbothoverandunder‐reporting.Furthermore,acrossthestudiesparticipantsrarelyused
morethanoneskilltojudgeawebsite.Finally,theskillsreportedwiththegreatestfrequencies
includedidentifyingtheauthorandusingsuperficialelementssuchastheappearanceofawebsite
orthecontent.
Theresultsofresearchstudiesusingself‐reportmethodsalsoprovideinsightintohow
readerscriticallyevaluatewebsites.Thesestudiesareingeneralagreementthatreadersdonot
evaluatewebsiteswithgreatfrequency.Acrossthestudiesthatinvestigatedthecriteriatojudge
websites,evaluatingthesourcewasconsistentlymentionedasthemostfrequentskillusedby
participants.Howeveracrossallthesurveysveryfewreadersusedmultiplecriteria.Forexample,
FlaniganandMetzger(2000)usedmeanverificationscorestoanalyzecredibilityverification
strategies.Acrossallthestrategiesrespondentsreportedusingskillsbetween“never”to
“sometimes.”withmostscoresfallingbetween“never”to“rarely.”Thisindicatesthatparticipants
didnotevaluatethesourcestheyread.
Theself‐reportstudiesalsoshowdifferencesintheresponsesbasedonage.Younger
participantsoftenreportedgreaterfrequencyofInternetuse,butalsoplacedgreatertrustin
47
Internetsources.Thiscouldindicateaneasingoffearswithincreaseduse;however,thisfinding
mayalsoindicatethatwhilestudentsareso‐called“digitalnatives”theyarenotinformationsavvy
(Bennet,Maton,&Kervin,2008).InotherwordsstudentsmaybeabletodownloadMP3’sorcreate
amash‐upforYouTube,butthatdoesnotmeantheyareskilledinusingtheInternetduring
problembasedinquirytasks.
Also,theresearchshowsdifferentresultsintrendsovertime.Thestudiesconductedbythe
PrincetonSurveyResearchAssociates(2002,2005)showanincreaseinthedistrustofwebsites.On
theotherhand,thePewInternetandAmericanLifeSurveys(Fox&Rainie,2002;Fox,2006)show
anuptickinthenumberofrespondentswho“never”or“rarely”evaluatewebsites.Thismaybe
explainedbydifferencesinthereadingtask.ThePrincetonSurveyfocusedprimarilyonwebsites
thatprovidedproductsorservicesforpurchase.ThePewstudieswereconcernedwithhealth.This
couldindicatethatthecriticalevaluationskillsusedbystudentsaretaskanddomainspecific.Yet
thestudiesindicatethatastheInternetmaturesthecriticalevaluationskillsexhibitedbepeople
arealsoevolving.
Finally,themajorityofstudiesusingself‐reportdatafocusedonadultpopulations.The
studiesmaynotprovideinsightintothecognitiveprocessesofyoungerreaders.Compoundingthis
issueisthelackofresearchinacademicsettings.Onlyafewstudiescollectedforhisreviewdealt
withstudentpopulations,andofthosestudiesallusedundergraduatestudents.Agreatereffort
needstobemadetounderstandthereadinghabitsofyoungerlearners.
Interviews,casestudies,verbalprotocolanalysis.Avarietyofqualitativemethodshave
beenusedtoidentifytheskillsandstrategiesreadersusetocriticallyevaluatewebsites.These
methodsvarybasedonthepositioningandepistemologicalstancesoftheresearcher,butoverall
seektounderstandtheprocessofjudgingawebsite,identifyingthecognitiveskillsusedinwebsite
evaluation,andidentifyingexternalsocialfactorsthataffecthowpeoplejudgewebsites.
48
Acrossthestudiescollectedforthisreviewthreequalitativemethodswereused:interviews
(Reih,2002;Reih&Belkin,1998),casestudies(Damico&Baildon,2006),andverbalprotocol
analysis(Coiro,2007;Damico&Baildon,2007;Reih&Belkin,1998,Zawilinskietal.,2007).
Researchusingsemi‐structuredinterviewdataaskedparticipantstoindicatewhichskillstheyuse
whilereadingonline(Merriam,1998).CaseStudies(Yin,2003)usedexamplesfromlargersamples
toexploretherelationshipbetweenindividualdispositionsandcriticalevaluationskills.Finally
verbalprotocolanalysis(Afflerbach&Pressley,1995)hadparticipantsthinkaloudandexplain
theirdecisions,astheyreadonline(Zawilinskietal.,2007).
Twomajorprocedureswerecommonacrossthequalitativestudies:websiterankingsor
givingstudentsataskandhavingthemcompleteanInternetinquiry.Inwebsiterankingtasks
(Brem,Russell,&Weems,2003;Foggetal.,2003)studentsweregivenalistofwebsitestorankon
ascale.Studentstooknotes,whichwerelateranalyzed.Participantsdiscussedtheserankingsin
interviewsorduringverbalprotocolanalysis,ortheywroteexplanationsoftheirrankings.
AssigningInternetinquirytaskstoparticipantswasalsoaprocedureusedinqualitative
studies.Thesetaskswerethenrecorded,oftenwithscreencapturesoftware,andusedinanalysis
withverbalprotocolanalysis(Reih,2002;Zawilinskietal.,2007),semi‐structuredinterviews(Reih
&Belkin,1998),orcasestudies(Damico&Baildon,1998,2007).Thetasksvariedontherestricted
natureoftheInternet.Someresearchershadstudentsevaluateasinglewebsite(Zawilinskietal.,
2007);othersusedalistedifpre‐selectedwebsites(Agosto,2002;Sanchezetal.,2006)whilesome
studiesgavestudentsopenaccesstothefullweb(Coiro&Dobler,2007;Damico&Baildon,1998,
2007;Reih2002,Reih&Belkin,1998,Zawilinskietal.,2007).TheInternetisanunboundedspace,
byrestrictingstudentstoasinglewebsitetheresultsofthesestudiesmaybelimited.
Theproceduresandmethodsusedtoinvestigatecriticalevaluationvariedbutacrossthe
individualprojectssomeimportantcommonalitiesemerge.Firstthereisacommonagreementthat
newskills,strategiesanddispositionsareneededabovethoserequiredforlinearprintreading.
49
Secondthecriticalevaluationofwebsitesmayrequireamoreflexibleworldview.Thirdsocial
factorsmayplayamoredominantrolethancognitivefactorsintheevaluationofwebsites.Finally,
newscreencapturesoftwarewillserveasanimportantmethodologicaltoolinthestudieson
criticalevaluationTheresultsofthesestudieshelptoinformwhatcriticalevaluationskillsshould
beinitiallymeasured.Basedontheliteraturereviewitwasdecidedtoexaminetwoscalesof
evaluation:credibilityandrelevancy.
Assessmentofcriticalevaluationskillsandstrategyuse.Measuringcriticalevaluation
isachallengeforresearchers.Onlyafewstudiesattemptedtomeasurestudents’judgmentsof
onlinetexts.Fourformatswerecommonlyusedtoassesscriticalevaluationskills:websiteranking,
writtenstatements,andonlinereadingcomprehensionassessments[ORCA].
Themostpopularassessmenttoolwastherankingofwebsites(Graesseretal.,2006;
Sanchezetal.,2006;Zhang&Duke,2007).Basicallystudentsaregivenalistofwebsitesandthen
havetorankthemfromleastreliabletomostreliable.Thesearethenscoredagainstapre‐
determinedlistofrankings.Graesseretal.(2007)andSanchezetal.(2006)createdamockGoogle
pagewithsevenwebsites:threereliablesites,threeunreliablesites,andoneambiguoussite.The
goalwastocreateanaturalisticenvironmentbutlimitthenumberofWebpagesparticipantscould
read.Afterafifty‐minuteinquirystudentswereaskedtorankthesites1‐7andratetheinformation
onthewebsitesonasix‐pointscale.Zhang&Duke(2007)gavestudentsalistoffourwebsitesand
hadthemrankwebsitesfromonetofour.Theyalsohadstudentsdecideiftheinformationona
singlewebsitewastrustworthy.
Writtenstatementswerealsoacommonmeasurementtool.Theywereoftenusedin
conjunctionwithwebsiterankings(Zhang&Duke,2007)orasmeasurementsoflearninggains
(Graesseretal.,2007&Sanchezetal.,2006).ZhangandDukehadparticipantswriteaparagraph
explainingtheirhighestrankingandanadditionalparagraphexplainingtheirlowestranking.
50
Sanchezetal.hadstudentsalsojustifytheirrankings.Furthermoretherewasnodifference
inthejudgmentscoresofstudentsonthesinglewebsite,butthereasoningscoreontheSanchezet
al.,(nd)foundthatstudentstrainedintheSEEKmethodweremorelikelytojustifytheirrankings
usingevidenceinthewebsiteandinformationaboutthesourcewhereasparticipantsinthecontrol
simplyusedcontent.
Graesseretal.,(nd),testingawebtutorbasedontheSEEKmethod,hadstudentswritean
essayonthetopicofinquiry.Theseessayswereanalyzedtoevaluatestudents’useofacritical
stance.Therewasasignificanteffectontreatmenteventhoughtherewerenosignificant
differencesintherankingofwebsitesbetweentreatmentandcontrol.Inotherwordsstudents
couldnotcorrectlyrankwebsitesbuttherewassignificantimprovementintheirjustificationof
theirrankings.Thiswouldsuggeststudentsweremorecognizantoftheskillstheyshouldapplybut
appliedtheskillsincorrectly.
Researchershavealsodevelopedonlinereadingcomprehensionassessmentsthatembed
criticalevaluationscalesintofullInternetinquirytasks.Leuetal.,(2005)embeddedcritical
evaluationitemsintoalargerassessmentofonlinereadingcomprehension.Thisassessmenthad
students:locatetwowebsiteswhentheyweregivenapartialdescriptionofthewebsiteorURL,
evaluatethetwositesandchoosethebestsource,andexplaintheirreasoning.Thesecondtask
askedstudentstolocateawebsitewithagraphicoftherespiratorysystemthatmetspecific
criteria,andcommunicatehowsomeoneshouldcheckforaccuracy.
Coiro(2011)alsoincludedcriticalevaluationinherORCAassessments.Factoranalysisfor
bothORCASrevealedthataone‐factorsolutionwasmostappropriatewhichmightsuggestthatthe
processesofonlinereadingcomprehensionmaynotbeindependentofeachother
TheORCA‐Iditarod(Coiro,Castek,Henry,&Malloy,20007)hadstudentsreadaboutand
thentakeastandontheIditarod.Tomeasurecriticalevaluationskillsweregivenonesourceand
51
hadtoansweraseriesofquestion.UnlikeCoiro(2011)thescoresonthisORCAdidnotloadon
one‐scaleusingprincipalcomponentanalysis.(McVerry,O’Byrne,&Roberts,2009).
Asummaryofcriticalevaluationassessments.Initialworkintheassessmentofcritical
evaluationskillsneedstocontinue.Websiterankinghasbeenthemostpopularmethodtoassess
criticalevaluation.Howevernopsychometricpropertiesoftheseassessmentswerereportedinthe
artifactscollectedforthisreview.Thereforeitwasdecidednottousewebsiterankingasamethod
formeasuringcriticalevaluationskills.
Thefactthatresearchersfindstatisticaldifferences,afterinterventionsinwritten
statementsbutnotinconcurrentwebsiterankingisalsointeresting(Zhang&Duke,2007).This
couldindicatestudentsaremakingerrorsofoversimplification.Theyknowwhatresponsesabout
strategyusetogiveandparrottheseresponsesbackwithoutactuallyapplyingthestrategiesto
websites.Thereforeitwasdecidednottousewrittenjustificationofwebsiteevaluationinthe
measureofcriticalevaluationskills.
Furthermoreindicationsthatonlinereadingcomprehensionskillsmaynotbeindependent
ofeachotherwillmakemeasuringcriticalevaluationskillschallenging.Independenceofitemsisan
assumptiononalmostallreadingcomprehensiontests.Ofnotewerethefactorloadingsonthe
ORCA‐Iditarod.Relevancyjudgmentsloadedwithlocatingitemsandcredibilityjudgmentsloaded
withevaluationitems.Ifreadingonlinedoesinvolveasetofmultidimensionalskillsthen
developingassessmentswillbeauniquechallenge.
ResearchonDispositions
Currentmodelsofreadingcomprehension(Alexander&Jetton,2002;Snow,2002)have
notedtheimportanceofbothaffectiveandcognitivevariables(Baker&Wigfield,1999;Guthrie&
Wigfield,1997).Thesedispositionsandopenmindsets(Deschreyver&Spiro,2010)arecentralto
onlinereadingcomprehension(Leuetal.,2004).
52
AccordingtoClaxton(1999),theprocessoflearningrequirescapabilitiesbutthese
capabilitiescannotaccountforallthelearningthatmusttakeplace.Learningmustalsoinvolve
specificdispositions,oraffectivevariables,whicharea“domainofhumanattributesnotattributed
toknowledge,skill,orbehavior”(Katz,1988,p.30).Carr&Claxton(2002)definedispositionsasa
“tendencytoedit,select,adapt,andrespondtotheenvironmentinarecurrent,characteristickind
ofway.”Learningdispositionsare“patternsofbehaviors,situatedinthecontextofthe
environment,thatwhenrecognizedanddevelopedbythosewhocanmanipulatetheenvironment
mayleadtogainsintheacquisitionofknowledge,skillsandunderstandings”(O’Byrne,&McVerry,
2009).
Duetothenatureofonlineinformation(Alvermann,2004;Gross,2004)dispositionsmight
beevenmoresignificantasindividualsreadonlineinformation(Coiro,2011).Thisisduetothe
increasedneedtofocusonthegoalofthetask,evaluatingthesourcesbeingread,andhavingthe
persistenceduringInternetsearches.Inthisstudylearninghasbeenviewedasaninteractionof
students’capabilitiesanddispositions(Carr&Claxton,2002)astheyreadinanonlinespace.
Recentstudieshaveinvestigatedstudents’onlinereadingcomprehensionability(Coiro,
2007;Henry,2007;Castek,2008).Yetwedonotknowhowdispositionsaffectonlinereading
comprehensionbasedontherestrictednatureoftheinformationspace.Inotherwordsare
dispositionsmoreimportantinlessrestrictedspacesormorerestrictedspaces?Thisstudywill
investigatedispositionsinbothlessrestrictedandmorerestrictedinformationspaces.
Measuringdispositionsofonlinereadingcomprehension.McVerryandO’Bryne(2009)
usingprotocolandfieldnotesfromtheTICAproject(Leuetal.2007‐2011)identifiedfive
dispositionsthatarecentraltoonlinereadingcomprehension:(a)reflectivethinking,(b)critical
stance,(c)collaboration,(d)flexibility,and(e)persistence.O’ByrneandMcVerrythencreatedand
validatedaself‐reportinstrumenttomeasurethesedispositions.Usingexploratoryfactoranalysis
53
thesefivescaleswereidentified.Theinstrumentasawholewasshowntohaveadequate
reliability.Howevertwoscales,flexibilityandcriticalstancedidnothaveadequatereliability.
Researchershavecritiquedassessmentmethodsthatusedself‐reportsurveysand
interviewsformeasuringdispositions(Claxton&Carr,2004).Theyclaimthatdispositionsareso
situated(Sadler,2002)theycannotbemeasuredwhenthecontextisstrippedawaybecausethey
donotprovideanadequatelyrichcontextfortheirmeasurement.Theseresearchers(Carr&
Claxton,2004;Sadler,2002)suggestthatobservationsoverextendedperiodsoftimearetheonly
waytotrulyassesslearningdispositions.Whilethesetoolsmaybettercapturelearning
dispositionstheyalsocarryreliabilityandpracticalityissues.
Anotherrecentadvancementinthemeasurementofdispositionsisfacialrecognition
software(D’Mello&Graesser,2010).Thesetoolstrackstudents’affectivestates(boredom,
flow/engagement,confusion,frustration,andneutral)inreal‐timebymonitoringconversational
cues,grossbodylanguage,facialfeatures,andthelanguageoftheirresponsesduringinteractions
withanintelligenttutoringsystem.Trackingemotionalstatesmaybethefutureofmeasuring
dispositions.Howeverthecostandtrainingofthesoftwaremakestheinstrumentimpracticalat
thistime.Thereforethisstudywillrelyonaself‐reportmeasureasitisthemostcosteffective,
practical,andvalidtoolavailable.
Dispositionssummary.Newdispositionsarerequiredforlearnerstoreadinonline
environments.These“openingmindsets”willbecriticalasinquiriesmovebeyondsimple“findthe
answertasks”(Spiro&Deschryver,2010)andintolessrestrictedinformationspaces.Dispositions
willbecentraltobuildingknowledgeinthemomentthroughtheactof“readingwithmousein
hand.”(McWilliams&Clinton,2012).Inotherwordsstudentswillneedtobeflexiblein
constructingknowledgeonthefly,usereflectivethinkingtoremembertheirgoal,bepersistentin
searchingforthegoal,andhaveacriticalstancetoquestiontheunlimitedamountofinformation
online.
54
ThisstudyusedtherevisedinstrumentdevelopedbyO’Byrne&McVerry(2009).The
instrumentincludesadditionalitemsfortheflexibilityandcriticalstancescales.Whilethereare
concernsaboutself‐reportmeasures,otherdispositionassessmentswereimpractical.Classroom
observationswouldrequiretoomuchtimeandhavenotbeenshowntobereliableandfacial
recognitionprogramsarenotcurrentlyreadilyavailable.
Thisstudywilltesttherelativecontributionofdispositionsinamodelthatincludesaless
restrictedinformationspaceandamodelthatincludesamorerestrictedinformationspace.Itis
hypothesized,fromboththeoreticalperspectives,thatdispositionswillbeasignificantpredictorin
boththelessrestrictedmodelandthemorerestrictedmodel.Howeverbasedontheprinciplesof
cognitiveflexibilitytheoryitishypothesizedthatdispositionswillbeastrongerpredictorinthe
lessrestrictedinformationspace.
ResearchonVerbalIntelligence
Thereisalongtraditionofresearchindicatingthatverbalintelligencehasaconnection
bothtooverallintelligenceandreadingcomprehension(Thorndike,1974).Sincethegoalofthis
studywastoexaminetherolebackgroundknowledge,criticalevaluation,anddispositionsplayin
predictingscoresonameasureofonlinereadingcomprehensionitwasdecidedtopartitionoutthe
variancecausedbyverbalintelligence.UsinghierarchicalregressionmethodssimilartoAnderson
etal.(1988),itwasdecidedtocontrolforverbalintelligencegivenitshighcorrelationswith
readingcomprehension.
Definingverbalintelligence.Hunt(1978)definedverbalintelligenceasinvolving
“processesbasedonknowledge.”WhichHuntnoted,“Theabilitytodealwithwordsandthe
conceptstheyrepresentimpliestheacquisitionofinformation”(p.109).Usingschematheory,from
aninformationprocessingperspective,Huntclaimedthatverbalintelligenceindicateda“deep
structurerepresentationofalinguisticstatementofthethoughtsinvolved”(p.110).Hunt,
55
Lunneborg,andLewis(1975)alsofoundthatknowingaperson’sverbalintelligencecanpredict
theirabilitytomanipulatestimulirapidly.
Previousstudiesthatexaminesearchingforinformationormultimedialearningcontroluse
verbalintelligenceasavariableofinterest.Allen(1992)controlledforverbalintelligence,usingthe
verbalcomprehensionmeasureofTheKitofFactorReferencedTesttoexaminehowuserssearch
CD‐ROMS.Allenstated,“Theabilitytoselectappropriatesearchvocabulary,toexplorealternative
expressionsofideas,andtounderstandthecontentofretrievedmaterials,iscentraltosuch
searching.”ThesameconclusioncouldbedrawnforsearchingtheInternetforinformation.
Ithasbeenarguedthatsynonymvocabularytestsarebestformeasuringverbal
intelligence.Carrol(1974)arguedthatverbalintelligenceneedstomeasurelexiosemantic
informationstoredinlong‐termmemory.Hesuggestedthatanyotherformofassessment,besides
asynonymtest,wouldconflateverbalintelligencemeasureswithothervariables.Furthermore
vocabularytests,suchasverbalcomprehensionmeasureofTheKitofFactorReferencedTest
correlatehighlywiththecapacityofverbalworkingmemory(Avons,Wragg,Cupples,&Lovegrove,
1998;Gathercole&Baddeley,1993;Gathercole,Service,Hitch,Adams,&Martin,1999;Masoura&
Gathercole,1999).Infactrecentreviewsofcomprehensionresearchfound“Vocabularyknowledge
andsyntacticcompetence,accountformoreofthevarianceinreadingcomprehensionthando
individualdifferences”(Snow,2002,p.84).
Summaryofverbalintelligence.Giventhatthegoalofthisstudyistoexaminethe
contributionsthatbackgroundknowledge,criticalevaluationskills,anddispositionsmaketo
modelsofonlinereadingcomprehensionitwasdecidedtocontrolforverbalintelligence.Thiswill
allowthemodelstoexaminevariancebeyondthatcausedbydifferencesinlong‐term,
lexiosemanticmemory.Verbalintelligencewasmeasuredusingapreviouslyvalidatedassessment,
theverbalcomprehensionvocabularytestoftheKitofFactorReferencedCognitiveTest(Ekstrom,
56
French,Harman&Derman,1976).Thisisapreviouslyvalidatedmeasuretestedwith7thgraders.
Theinternalconsistencyofthemeasurehasrangedfrom.68‐.88.
ChapterSummary
Insummarythischapterreviewedtheliteraturebaseofthedependentandindependent
variablesincludedinthisstudy.Theliteraturereviewforthisstudyhelpedtoguidethedesignand
otherdecisionsusedinthisstudy.Empirically,researchhasdemonstratedthattherearenewskills,
strategiesanddispositionsrequiredforonlinereadingcomprehension.Unfortunatelyfewstudies
haveexaminedhowtherestrictednatureoftheinformationspaceshapestheroleofcognitiveand
affectivevariables.
Findingsfromtheinvestigationintobackgroundknowledge,criticalevaluationskills,and
dispositionsallindicatethatmoreworkisneededifwearetoenrichourtheoreticalmodelsof
readingcomprehension.Firsttheresultssurroundingthecontributionsofbackgroundknowledge
haveoftenbeencontradictory.Thisstudywilladdtoourunderstandingofbackgroundknowledge
andonlinereadingcomprehension.Byfocusingoncontentknowledge,andnotsystemknowledge,
theresultsmaydemonstrateifbackgroundknowledgeisastrongpredictorofcomprehensionin
onlinespaces.
Intermsofcriticalevaluationskillsthereisverylittleresearchexploringtheroleitplaysin
termsofoverallonlinereadingcomprehension.Thisstudypredicts,aftercontrollingforverbal
intelligence,thatitwillbethesinglegreatestfactorinpredictingperformanceinamodelthatalso
includesbackgroundknowledgeanddispositions.Thereviewofresearchalsoindicatedthatthere
wasnoexistingmeasureofcriticalevaluationskills.Basedonareviewoftheevidenceitwas
decidedtocreateanewmeasurethathadstudentsevaluatemultiplesourcesinaforcedresponse
assessment.
57
Theworkondispositionsofonlinereadingcomprehensionisinitsinfantstages.Thisstudy
hopestocontinuethework.ItwasdecidedthatarevisedDORC(O’Byrne&McVerry,2009)would
beusedinthisstudy.
Thiscurrentstudyseekstobuildonpreviousworkbyansweringfourresearchquestions:
1. When predicting online reading comprehension of a problem solving task within a less
restricted information space, what is the best fit model, after controlling for verbal
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension?
2. When predicting online reading comprehension of a problem solving task within a more
restricted information space, what is the best fit model, after controlling for verbal
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension.
3. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a less
restricted information spaces?
4. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a more
restricted information space?
Thepresentstudywillcontributeinsightstotheexistingliteratureinvestigatingonline
readingcomprehension.Mostspecificallythisstudywilltaketherestrictednatureofthe
informationspaceintoaccountasitinvestigatesthecontributionsthatreadercharacteristicsof
verbalintelligence,backgroundknowledge,criticalevaluationskills,anddispositions,maketo
58
modelsofonlinereadingcomprehension.Bytestingmultiplemodelsofonlinereading
comprehensionassessmentthisstudywilladdtotheresearchbaseoftheliteracyfield.
Thisstudyalsowillhelpadvanceeffortstomeasureonlinereadingcomprehensionskills.As
partofthestudytwomeasuresofonlinereadingcomprehensionandameasureofcritical
evaluationskillswillbecreatedandvalidated.Furthermoreapreviousinstrumentmeasuring
dispositionsofonlinereadingcomprehensionwillberevisedandtested.
Thisstudywillalsoofferimportantinsightsintothestrategyuseofstudentswithvarying
degreesofonlinereadingcomprehensionability.Thequantitativeresultswillbeusedtoidentify
participantsinthethinkaloudportionofthisstudy.Thisdatawillbecrucialinexploringthe
findingsfoundinthequantitativemodels.Thedatawillalsoofferimportantinsightsintothetypes
ofstrategyinstructionthatstudentwillneedinordertomakemeaninginanetworkedsociety.
Thisstudywillcontributetoexistingliterature;advanceeffortstomeasureonlinereading
comprehension,andofferinsightsintotheskillsstrategiesusedbystudentsengagedinonline
inquirytasks.Theseconclusionswillhelptheresearchcommunitydevelopmorerobustmodelsof
readingcomprehensionwhileofferingimportantinstructionalstrategiesforeducators.
59
CHAPTERIII
MethodsandProcedures
Thisstudysoughttoexaminesomeofthecomplexitiesthatappeartobeapartofonline
readingcomprehension.Specificallythisstudyexaminedtherelativecontributionsmadebya
reader’sbackgroundknowledge,criticalevaluationskills,anddispositionsintwomodelstopredict
scoresonanonlinereadingcomprehensionmeasure.Thefirstmodelusedalessrestricted
informationspacewherestudentscouldlocateanysourceonthetopic.Thesecondmodeluseda
morerestrictedinformationspacethatrequiredstudentstolocatespecificsources.Thestudyused
amixed‐methoddesign(Tashakkori&Teddlie,2003)thatcombinedqualitativeandquantitative
researchapproaches.
Thepresentstudywasconductedinaseriesofthreephases.Inphaseone,multiple
instrumentsweredevelopedtoserveasindependentanddependentvariables.Inphasetwo,
regressionanalysiswasusedinanattempttounderstandfactorsthatexplainedvarianceestimates
inscoresofanonlinereadingcomprehensionmeasure.Inphasethree,verbalprotocolmethods
(Afflerbach,2002)wereusedtoexplorestrategyuseamongparticipantswhovariedintheironline
readingcomprehensionproficiency.Thesetwoapproaches:aregressionanalysisandverbal
protocolanalysisledtoaninterpretationofthedatathroughaconvergenceofbothquantitative
andqualitativedata.Thismixedmethodstudyexploredthefollowingresearchquestions:
1. When predicting online reading comprehension of a problem solving task within a less
restricted information space, what is the best fit model, after controlling for verbal
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension?
2. When predicting online reading comprehension of a problem solving task within a more
restricted information space, what is the best fit model, after controlling for verbal
60
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension.
3. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a less
restricted information space?
4. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a more
restricted information space?
QuantitativeProcedures
SettingandParticipantSelection
Therewere131seventhgradestudentsinitiallyincludedinthequantitativeproceduresof
thisstudy.Seventhgradewaschosenbecauseitrepresentsapointsomewherenearthemiddleof
grades1‐12andthusresultsmaybesomewhatsimilartostudentsfromawiderrangeofadjacent
gradelevels.Furthermore,theuseoftheInternetasatoolforreadinginformationaltextismore
commonatthisgradelevel(Fox&Rainie,2002).Sevenschooldistricts,fromaconveniencesample,
wererecruitedthroughpersonalcontacts.Fromthisinitialsamplethreeschoolswereselectedto
ensurerepresentationfromhigh,medium,andlowsocio‐economicdistrictsasmeasuredbythe
DistrictReferenceGroup[DRG](StateofConnecticut,2010).
DRGsrepresentastatisticalcategorycreatedbythestateofConnecticutforstatistical
reporting.DRGsarecalculatedthroughmedianfamilyincome,educationandoccupationlevelof
parents,familystructure,homelanguage,andoverallenrollment(ConnecticutStateDepartmentof
Education,2010)andrangefromlevelsA‐I,withAbeingthehighestSESschooldistrictandIbeing
thelowest.Toensureareasonablyrepresentativesample,acrossDRGgroups,thethreeschools
selectedforthisstudyincludedaDRGBschool,aDRGEschool,andaDRGIschooldistrict.The
61
threeschooldistrictsnotonlyvariedontheirDRGreferencegroupbutalsoontheirdemographic
make‐upandlevelofspecialservicesoffered.ThisisexplainedinTable3.1.
Table 3.1 Demographic Data of Schools School Reduced
LunchPriority District
Non-English Speaking Home
ESL Services Provided
Special Education
Non-Asian Minority
1 DRG I 78% Yes 30% 21% 17.3% 64%2 DRG E 35.2% No 7.6% 1.3% 12.7% 21%3 DRG B 5.3% No 1.8% 0% 9.2% 3.7%
School African-American
Caucasian Hispanic Asian
1 DRG I 11.1% 16.7% 69.4% 2.8%2 DRG E 11.1% 64% 20% 8%3 DRG B 2.3% 92.9% 0% 2.8%
QuantitativeMeasures
Verbalintelligence.Previousresearchhasshownthatverbalintelligencecorrelates
stronglywithreadingcomprehension(Curtis,1987).Studiesthatexaminesearchingfor
informationormultimedialearningcontrolhaveusedverbalintelligenceasacontrollingvariable.
Allen(1992),forexample,controlledforverbalintelligence,usingtheverbalcomprehension
measureofTheKitofFactorReferencedTesttoexaminehowuserssearchCD‐ROMS.Verbal
intelligenceinthepresentstudywasalsomeasuredusingtheverbalcomprehensionvocabulary
testoftheKitofFactorReferencedCognitiveTest(Ekstrom,French,Harman&Derman,1976).The
testisdesignedforstudentsfrom7thto12thgrade.Reliabilityinpreviousadministrationshas
variedfrom.76‐.89.
Theverbalcomprehensiontestconsistsof36forcedresponseitemswiththreedistractors.
Thetestisorganizedintwo,18‐questionparts.Studentshavefourminutestocompleteeachpart.
Eachquestioncontainsfourmultiple‐choicesynonymsforatargetword.SeeFigure3.1foran
exampleitem.Studentsreceivedonepointforeachcorrectanswer.
62
Figure3.1SampleVerbalComprehensionItems
Backgroundknowledge.Backgroundknowledgewasestimatedusingathree‐itemtopical
knowledgequestionnaire.Thequestionnairewasdeliveredusingacomputer‐basedsurvey.
Participantswereaskedtorespondtoapromptaskingthemtolistallthefactsordetailstheyknew
abouttheAmericanRevolution.Respondentshadtoclickabutton,“Submit,”tomoveon.Asecond
screenthenasked,“TrytothinkofoneortwomoredetailsabouttheAmericanRevolution.Donot
worryifyoucannotrememberanymore.Justtype,"MoveOn."Oncetheyhit“Submit”students
werebroughttoathirdscreenandasked,“Thinkrealhard.Isthereanythingelseyoucan
rememberabouttheAmericanRevolution?Itisokayifyoucannot,justtype,Moveon."
Backgroundknowledgescoresweredeterminedbasedonthetotalnumberofideaunits
(Leslie&Caldwell,1995)studentsprovided.Anideaunitwasdefinedasaproposition.Scoringof
ideaunitswasinformedbyCoiro’s(2012)adaptationofWolfeandGoldman’s(2005)measureof
topic‐specificbackgroundknowledge.Onepointwasgivenforanaccurateandrelevantdetail;a
halfpointwasawardedforanyideaunitthatsomewhataccurateorrelevant;azerowasgivenfor
anyideaunitthatwasinaccurateorirrelevant.Table3.2givesexamplesofeachtypeofresponse.
63
Table3.2BackgroundknowledgeResponsesTothePrompt:”ListeverythingyouknowabouttheAmericanRevolution”Inaccurateorirrelevantstatement(0points)
Generallyaccuratestatement(.5points)
Specificallyaccuratestatement(1point)
Peoplelivedinfear? ItwasawarinAmerica
TheAmericanrevolutioninvolvedGeorgeWashington
Therewerethirtycoloniesduringthewar.
usaarmywasfightingforamerican
becauseofthefamoussaying"notaxationwithoutrepresentation."
Tofreetheslavesinthesouth
Itwasawarinthe1700
Theamericanrevolutionwaswhenourcountryfoughtagainstthebritishrule
Tocalculateinter‐raterreliabilityasecondraterscored20%oftheresponsesrandomly
selected.Firsttheresearcherandtheraterscoredananchorsetoffiveexamples.Thentheyeach
scoredthe20%samplealone.Inter‐raterreliabilitycoefficientsfortotalscoresonthebackground
knowledgemeasurewasr=.82.Theyrangedacrossthethreepromptsfrom.79‐.87.
Criticalevaluation.TheCriticalOnlineInformationLiteracies(COIL)instrumentwas
basedonmeasuresdevelopedbyKiili,Laurinen&Marttunen,(2008),Brem,Russell,&Weems
(2001),andLeuetal.(2010).ThefinalinstrumentwasdeliveredusingSurveyMonkey,anonline
surveytool.Theitemsmeasuredeachofthefollowingconstructs:author,bias,publisher,and
source.
Thereweretwotypesofitems.Thefirstconsistedofscreenshotsofwebsites.Thesewere
eithersinglewebsitesorfourscreenshots.Thesecondtypeofitemrequiredstudentstoopenlinks
tofourdifferentwebsites.Thesewebsitesincludedsecondarylinkstoauthorandpublisher
information.Allofthewebsitesinvolved,eitherstaticordynamic,wererecreationsofactual
websitesthatwerehostedonauniversityownedserver.Anexampleofeachitemtypeisavailable
64
inFigures3.2,3.3,and3.4.Thetaskspresentedineachitemweresituatedinactivitiesthat
adolescentswouldbeengagedinastheysearchedforonlineinformation
Figure3.2.SingleScreenShotExamplefromCOILAssessment
65
Figure3.3FourScreenShotExamplefromCOILAssessment.Note:Thefourscreenshotswere
presentedvertically.
66
Figure3.4FourHyperlinkScreenShotExamplefromCOILAssessment.
Theinstrumentunderwentthreephasesofdevelopment.PhaseIinvolveddefining
constructsandcontentvalidation.PhaseIIinvolvedapilottestforinstrumentvalidation.PhaseIII
usedtheresultsofthevalidationstudytoinformfinalitemdevelopmentbeforetheinstrumentwas
used.Thisthree‐phaseprocessledtothedevelopmentofthefinal14‐itemassessment.
PhaseI.TheconstructionoftheCOILbeganwithaliteraturereviewtodeterminefactorsthat
influencethecredibilityandrelevancyjudgmentofsources(Judd,Farrow&Tims,2006;Kiili,
Laurinen&Marttunen,2008).Previousworkbyotherresearcherswasusedtoidentifysub
constructsthatinfluencedcredibilityandrelevancyjudgments.(SeeTables3.3and3.4).These
subconstructswereusedtodevelopmultiple‐choiceitemsforcredibilityandrelevancy.
67
Table3.3Sub‐ConstructsofCredibility
Sub‐Construct Definition
Evaluateauthor Judgingthecredibilityofawebsitebasedondetailsabouttheauthor(Harris,1997)
Evaluatesourceofclaim
Judgingthecredibilityofatextbasedonthesourceofinformationthatisincluded(Rieh&Belkin,1998;Strømsø&Bråten,2010)
Evaluatebias Judgingthecredibilityofawebsitebasedonaninclinationtowardholdingaparticularperspective(Coiro,2003;Fabos,2008)
Evaluatecontent Judgingthecredibilityofwebsitebasedoncompletenessofinformation(Harris,1997;Kiilietal.,2008)
Evaluateargument
Identifyingandcomparingperspectives(Kiili etal.,2008)
Evaluateaccuracy Comparingclaimswithasecondarytext(Meola,2004)
Table3.4Sub‐constructsofrelevancy
Sub‐Construct Definition
Evaluaterelevanceoftopic Identifyingwebsitesorsearchresultsthatwillhelpansweraquestion(Kiili,etal.,2008)
Evaluaterelevanceofawebsite
Identifyinghyperlinksorheadingsthatwillansweraquestion(Kiili,etal.,2008)
Evaluatepurpose Identifyingthepurposeorintendedaudienceofatext(Harris,1997)
Evaluatecurrency Judgingawebsitebasedonageofpublication(Meola,2004;Kiilietal.,2008)
EvaluateUsability Judgingawebsitebasedoneaseofuse andreadability(Meola,2004;Kiilietal.,2008)
68
Contentvalidationtechniques.Inordertoestablishitemvalidity,theinstrumentunderwent
acontentvalidationphasewithexpertsfamiliarwithcriticalevaluationresearchtodevelop
definitionsfortheconstructs(McKenzie,Wood,Kotecki,Clark,&Brey,1999).Thesixexperts
includedprofessorsandgraduatestudentsfamiliarwithresearchinthecriticalevaluationofonline
information.Theexpertsratedthedimensionalityofeachofthetwentymultiple‐choiceitemsby
indicatingwhichoftheconstructandsubconstructstheitemmeasured.Itemsidentifiedby90%of
participantsasmeasuringthehypothesizedconstructwerekeptforfurtheranalysis(Gable&
Wolfe,1993,McKenzieetal.,1999).
AContentValidityIndex(CVI)(Rubio,Berg‐Weger,Tebb,Lee,&Rauch,2003)wascreated
foreachitemusingthefeedbackprovidedbytheexpertstotestformultidimensionalityofitems.A
CVIiscalculatedbyhavingeachraterrankanitemfromone(irrelevant)tofour(extremely
relevant).TheCVIistheproportionofitemsthatreceivedatleastathreeorfourbytheraters.For
inclusioninthefirstiterationoftheinstrument,theCVIforeachitemneededtoexceedathreshold
of0.70(Rubioetal.,2003).Finally,theexpertswereencouragedtoleavewrittenfeedbackthatwas
usedtoensuretheadequacyandaccuracyofdefinitionsofconstructsanditemsconstructed
(McKenzieetal.,1999).Table3.5liststheresultsfromthecontentvalidationstudy.Itemsarelisted
intheordertheyappearedontheassessment.
69
Table3.5ResultsoftheContentValidationStudy
Item %Correct
%ofExpertsWhoCorrectlyIdentifiedTheConstruct
CVI Comment
1 85 87.5author 2.875 Someconfusionoverauthorandsource.Collapsedsourceintocontentforsecondround.
2 54 90usability 2.33 Expertsfeltreadabilityistoosubjectiveandusabilitytoosituated.Thisitemandsubconstructweredeleted.
3 62.5 12.5argument NA‐ Somereviewerslikedargumentasasubconstructbutitcausedconfusion.Theitemswerecollapsedundercontent.
4 88 100currency 2.67 Somearguedcopyrightofwebsiteisnotclearindicatorofdateofinfo.MovedanswerchoiceawayfromAvalanche.
5 100 62purpose 1.33 Peopleliketheitembutfeelpurposeisreallyarelevancyjudgment.CVIwouldbehigherifpurposewasconsideredarelevancyjudgment.
6 65 64relevancy 2.2 Thesearchresultsneedtobelessrelevantifthisisarelevancyjudgment.Betterdistracterswerepicked.
7 90 34relevancy 1.8 Mostreviewersfeltthatknowingwebsitegenreswasnotatimportanttomeasuringcriticalevaluation.Theitemwasdeleted.
8 87.5 66currency 2.8 Itemkept.Examinedwhypeopledidnotpickcurrency.
9 87.5 12.5argument NA EditeditemsoitiswhichwebsiteusesthebestdetailstosupporttheclaimPlutoisnotaplanet.Collapsedargumentundercontent.
10 100% 83source 2.75 Sourcewascollapsedundercontent
11 85 85.7author 2.75 Nochanges
12 28% 20usability NA Expertswereunsureofthegradelevel,againsomecommentedthatreadinglevelissubjective.Deletedallusabilityitems.
13 83.3 50argument 2.33 Peopleliketheitem.Thiswascollapsedunder
70
content.
14 75 85.7relevanceoftopic
2.8 Nochanges
15 100 57.1purpose 2.25 Againexpertsfeltthatevaluatingpurposewasarelevancyjudgment.
16 50 .25usability NA Deleteditemormakeitarelevancyjudgment
17 85.7 42.8argument 2.33 Argumentwascollapsedintocontent.
18 85.7 75relevanceofhyperlinkonawebsite
2.67 Renamedsubconstruct
19 100 75relevanceofinfo 2.67 Changedthenameofothersubconstruct.
20 50 50source 2.33 Sourcewascollapsedundercontent.
ResultsofphaseI.Asaresultofthecontentvalidation,revisionsweremadetoitemsto
matchsubconstructshypothesizedbythepanelofexperts.ThesechangesarelistedinTable3.6.
Theexpertsfeltthatpurposewasmoreofarelevancyjudgmentratherthanacredibilityjudgment.
Furthermorethepanelcouldnotagreeonwhichitemsmeasuresthesubconstructof
evaluatecontent.Thecontentvalidityindexfortheseitemswastoolow.Giventhedisagreement
overthesubconstructofpurposeandthelowCVIscoresforevaluatingadecisionwasmadeto
revisetheconstructofrelevancy.Thereforethesubconstructofpurposewasmovedfromthe
constructofcredibilitytotherelevanceconstruct,andthesubconstructofevaluatecontentwas
deleted.Thenewlydefinedevaluaterelevanceconstructwasnowdefinedbytwonew
subconstructs.Thisincludedevaluatesearchresultsandevaluateinformationonawebsite.This
ledtotherevisionofitems:6,14,18,and19.
Additionallytheexpertsfeltthatthesubconstructofusabilitywastoosubjective.Theitems
askedstudentstoevaluatethereadinglevelsofwebsites.Thepanelfeltthissubconstructwastoo
dependentonindividualdifferences.Thusthesubconstructofusabilitywasdropped.Theitems
71
werereplacedwithitemstorepresentthesubconstructofbias.Thisledtotherevisionofitemstwo
and12.
Item20wasalsorevisedduetothelownumberofexpertsscoringtheitemcorrectly.Inthe
originalitemtheparticipantswerepresentedwithfourscreenshotsofwebsites.Theywereasked
whichwebsiteusedinformationfromthemostreliablesource.Inthereviseditemspecificclaims
andtheirsourceswerehighlighted.Thesubconstructwasrenamed“evaluatesources”toevaluate
sourcesofinformation.
72
Table3.6RevisionsbasedontheContentValidationoftheCOIL
DraftAssessmentIteminContentValidationStudy
AssessmentItemsIncludedInPilotInstrument
1.Evaluateauthor 1.Evaluateauthor
2.Evaluateusability 2.Evaluatebias
3.Evaluateargument 3.Evaluateargument
4.Evaluatecurrency 4.Evaluatecurrency
5.Evaluatepurpose 5.Evaluatepurpose
6.Evaluaterelevanceoftopic 6.Evaluaterelevancysearchresults
7.Evaluatecontent 7.Evaluateauthorexpertise
8.Evaluatecurrency 8.Evaluatecurrency
9.Evaluateaccuracyofclaims 9.Evaluateaccuracyofclaims
10.Evaluatesourceorclaim 10.Evaluatesourceofclaim
11.Evaluateauthor 11.Evaluateauthor
12.Evaluateusability 12.Evaluatebias
13.Evaluateargument 13.Evaluateargument
14.Evaluaterelevanceoftopic 14.Evaluaterelevanceofsearchresults
15.Evaluatepurpose 15.Evaluatepurpose
16.Evaluatecontent 16.Evaluaterelevanceofsearchresults
17.Evaluateaccuracyofclaims 17.Evaluateaccuracyofclaims
18.Evaluaterelevanceoftopic 18.Evaluaterelevanceofinformationonwebsite
19.Evaluaterelevanceofwebsite 19.Evaluaterelevanceofinformationonwebsite
20.Evaluatesources 20.Evaluatesourceofinformation
73
PhaseII..Totestthereliability,orinternalconsistencyofthescales,acoefficientalphawas
calculated(Pettetal.,2003)usingresultsfromanotherstudy(O’Byrne,2011).Apaperandpencil
versionofthetwentyitemswasadministeredto197seventhgraders.Theachievedcoefficient
alphaforthetwohypothesizedscaleswas.358forrelevancyand.312forcredibility.Thecombined
instrumenthadanoverallcoefficientalphaof.339.Investigationoftheinter‐itemcorrelations
revealedthatthecorrelationsbetweentheitemsinthescalewerelowornegativelycorrelated.
Becausethisassessmentdidnotmeetthethresholdof0.70foracoefficientalphathemeasurewas
showntobenotreliable.
PhaseIII.Giventheinadequatereliabilityofthelastiterationadecisionwasmadetorevise
theinstrument.Itwasdecidedtomakethreechanges:(a)reducethenumberofscalesand
subscales;(b)makedistractorseasiertorecognize;and(c)simplifythetestingformat.
Adecisionwasmadetocreateitemshypothesizedonlytomeasurecredibility.Thisdecision
wasboththeoreticalandpractical.Inboththeoreticalpieces(Coiro,2003;Hartmanetal.,2010)
andinpastinstrumentvalidationstudies(McVerry,O’Byrne,andRobbins,2009)researchersinthe
fieldofnewliteraciesofonlinereadingcomprehensionhavesuggestedthatwhilearelevance
judgmentisanevaluationofwebsites,thecognitiveprocessesinvolveddiffersinimportantways
fromjudgingcredibility.InfactMcVerryetal.(2009)foundthatscoresonrelevancyitems
hypothesizedtoloadwithothercriticalevaluationscoresactuallyloadedonscalescontainingthe
locatingscores.
Reducingthenumberofconstructsalsohasadvantagesforimprovingthepracticalityofthe
instrument.Thetotalnumberofitemsisreducedwhichmakesiteasiertoadminister.Whilefewer
itemsreducetheoverallcoefficientalpha(Cronbach,1972)aone‐factormodelmayhavegreater
internalconsistencythanatwo‐factormodel.Thusitwasdecidedtoonlymeasuretheevaluation
subconstructsofauthorexpertise,publishercredibility,sourcecredibility,andauthorbias.
74
PhaseIII:Contentvalidation.Thisphaseoftheinstrumentwasvalidatedinthreesteps.The
firststepinvolvedanadditionalroundofacontentvalidationsurveybyanexpertpanel.The
secondstepusedstructurallypromptedthinkalouds(Afflerbach,2002)duringcognitivelabs
(Ericsson&Simon,1999)toelicitfeedbackfrom7thgradestudentsonitemformatanddifficulty.
Finallytheinstrumentwasadministeredto1207thgradersfromhigh,low,andmediumSESschool
districtstotestforreliability.
Thefirststepinrevisingtheinstrumentinvolvedanotherrigorouscontentvalidation
process(McKenzieetal.,,1999).First12itemsweredeveloped,threeitemsforeachofthenew
constructs.Then18expertswereidentified.Theexpertsratedtheconstructvalidityofeachofthe
twentymultiple‐choiceitemsbychoosingwhichoftheconstructtheitemmeasured.Allitems
identifiedby90%ofparticipantsasmeasuringthehypothesizedconstructwerekeptforfurther
analysis(Gable&Wolfe,1993,McKenzieetal.,1999).NextaContentValidityIndex(CVI)(Rubioet
al.,2003)wascalculatedforeachitem.TheCVIwascalculatedasaproportionofexpertswho
indicatedifanitemwasextremelyrelevant(4)orveryrelevant(3)onafourpointLikertscale.
FollowingthecalculationofaCVI,aContentValidityRatiowasalsocalculatedtoevaluate
theextenttowhichameasurerepresentsagivenconstruct(McKenzieetal.,1999).Tocalculatea
CVR,youfirstdeterminethedifferencebetweenthenumberofexpertswhomarkedanitemas
essentialandhalfthetotalnumberofexperts.ThentheCVRiscalculatedbydividingthisnumber
byhalfthetotalnumberofexperts.ForinclusioninthefinalversionoftheinstrumenttheCVRfor
eachitemwasrequiredtoexceed0.70(McKenzieetal.,1999).Thus,tobeincludeditemswere
requiredtohaveaCVIof2.67andaCVRofatleast0.70.Table3.7showstheinitialresultsofthis
process.Items2,5,6,8,and11failedtomeeteithertheCVIand/ortheCVRcriteria.Theseitems
wererevised.
75
Table3.7
ContentValidationResultsforRevisedCOIL
Item %OfParticipantCorrectlyRespondingtoItem
ConstructMeasured
%WhoIdentifiedtheConstruct
CVI CVR Comment
1 100 Author 94.1% 3.5 1.0
2 88 Author 40.0% NA NA Confusionbetweenauthor,source,andpublisher.Itemreworded.
3 66.6% Bias 93.3% 3.15 .866
4 93.3% Publisher 100% 3.4 1.0
5 100% Author 0.0% NA NA Hypothesizedpointofviewasundertheconstructofauthorbutallparticipantsrankeditasbias.Theconstructwasrevised.
6 93.3% Source 93.3% 3.08 .858 Neededmoreplausibledistractors
7 85.7% Publisher 92.2% 3.30 .858
8 80.0% Author 80.0% 3.33 .867 Itemwasrevisedtoaddeasierdistractorsandfocusrespondentstothepublisherandnottheauthor.
9 84.6% Source 92.9% 3.23 .857
10 50% Publisher 83.3% 3.16 1.0 Neededmorediscriminantdistractors
11 85.7% Bias 76.7% Neededtochoosewebsiteswithaclearauthorandpublisher.Confusionbetweenthetwoconstructs.
12 72.7 Source 92.9% 3.23 .857
76
PhaseIII:Revisionsbasedoncontentvalidation.Basedonthecontentvalidationresultsthere
appearedtobesomeconfusionoverthedefinitionsforseveralsubconstructs.Specificallytherewas
confusionoverauthor,source,andpublisher.Thiswasevidentinitem2,asonly40%ofexperts
correctlyidentifiedtheconstruct.Torectifythissituationseveralrevisionsweremade.First,
insteadofprovidingstudentswithawebsiteandaskingthemwhichsiteuseddetailsfrommore
reliablesources,aspecificdetailwaschosenfromeachsite.Items6,9,12wererevisedsothata
specificdetailineachwebsitewashighlighted.Eachincludedthesourceoftheinformation.
Second,onquestionsaskingaboutthecredibilityofpublishersthecorrectanswersanddistractors
wererevisedtoincludeonlytheheaderortheaboutuspageaboutapublisher.
PhaseIII:Cognitivelabs.Nextaseriesofcognitivelabs(Ericsson&Simon,1999)were
conductedwitheightstudentsfromanaverageSESschooldistrict.Structurallypromptedthink‐
alouds(Afflerbach,2002)wereused.Thisallowedtheresearchertotestitemformat,wording,and
difficultywiththetargetpopulationforalloftheitems.
Usingthinksaloudprocedures,studentswerefirstaskedtoindicatewhichanswerchoice
theythoughtwascorrectforeachitem.Theythenwereaskedwhytheythoughteachoftheother
answerchoiceswereincorrect.Nextstudentswereaskedtoofferfeedbackoneachitemformat.
PhaseIII:Cognitivelabsresults.Basedonitemdescriptivesandthecognitivelabresultsthe
instrumentneededfurtherrefinement.SeeTable3.8forstudentperformanceonthetwelve‐item
assessment.Aoneindicatesacorrectanswer.Azeroindicatesanincorrectanswer.Themeanscore
isanestimateofitemdiscriminationasthepercentageofstudentswhoscoredcorrectlyonthe
item.Theseresultsindicatedthat3,8,and11hadtoberevised.
77
Table3.8CognitiveLabItemDiscrimination
Item 1 2 3 4 5 6 7 8 9 10 11 12 Total
1 0 0 0 1 1 0 0 1 0 0 1 5
1 1 0 0 1 0 0 0 1 0 0 1 5
1 1 0 1 1 1 0 0 1 1 1 0 8
1 1 0 1 1 0 1 0 0 1 0 1 7
1 0 0 1 1 1 1 0 0 1 0 0 6
1 1 0 1 1 1 1 0 0 0 1 1 8
1 0 1 1 1 1 1 0 0 0 0 1 7
Mean 1 .63 .13 .63 .88 .75 .57 0 .48 .43 .25 .71 6.30
Thethinkalouddataalsosuggesteditemrefinement.Thecognitivelabquestionsthat
examinedbiasby“askingwhichwebsitesincludestrongwordsandimagestoinfluencethe
audience”neededmorerecognizabledistractors.Studentshaddifficultyunderstandinghow
informationwasshapedbyanauthororpublisher.Questionsthree,five,andelevenwererevisedto
makedistractorsmorerecognizable.
Studentresponsesalsoindicatedthatsomeindividualitemsneededtoberevised.No
studentsscoredcorrectlyonquestioneightcorrectly.Thisquestionhadstudentsevaluatean
author’sexpertisegivenherbiographypage.Thedistractorswererevisedtomakethequestion
easier.
Thestudentsfromthethink‐aloudreportedoneaseofusewiththeSurveyMonkey
interface.Theycouldclickonembeddedlinks,andcoulddelineatebetweentheitemstemand
forcedchoiceresponses.Theywerealsoasked,andgiventheoption,tohavetheirwebsitesopenin
newwindowsortabs.Allofthestudentsreportedthattheypreferredwebsitestoopeninnewtabs
ratherthanwindows.Studentsalsosuggestedthewordingonsomeitemsneededrevisions.
78
PhaseIII:Finalinstrumentationandadministration.Therevised12itemswerethen
administeredtothe131studentsinthestudy.Studentstooktheassessmentinoneclassperiod,in
theirclassroomusinglaptops.
Areliabilityanalysiswasrunfollowingtheadministrationofthe12‐itemassessment.A
coefficientalpha(Cronbach,1972)wascalculatedat.43.Thisassessmentdidnotmeetthe
thresholdof0.70foracoefficientalphathemeasurewasshowntobenotreliable.Thelow
reliabilitymayhavebeenduetotheshortassessmentlength.
UsingtheSpearman‐Brownprophecythetestwouldhavetoincreasebyafactorof3.16.
Giventhecomplexityoftheassessmentandthelimitationsofclassroomschedulesa40to50‐item
assessmentwouldnotbepossible.Thusthedecisionwasmadetomorethandoublethe
assessmentlengthbycreating16newitems,foureachfor,publisher,author,bias,andsource.The
itemswerecreatedbymirroringitemsthatwereacceptablefromthecontentvalidationsurvey.
Theseitemswerethenadministeredtothestudentsinthestudybeforetheycompletedanyother
additionalassessmentsandafteraweekoftheinitialbatteryoforiginalitems.
PhaseIII:instrumentationresults.Areliabilityanalysiswasthenconducted.Oneofthe
additionalnewitemsmeasuringauthorexpertisewasremovedfromtheassessmentaprioridueto
anerrorinadministration.Twenty‐sevenitemswereincludedinthefinalanalysiswithanN=110.
Thecoefficientalphawithall27itemswas0.560.Thiswasnotacceptableforresearchgivenitdid
notpassourthresholdforaninternalconsistencyvalueof0.700(Peterson,1994).
Nextanexaminationofthedescriptivestatisticsoftheitemsrevealedthatmanyitemshad
meanscoreoflessthan0.20.Thiswouldmeanlessthan20%ofparticipantsscoredtheitem
correctly.Anyitemwithameanscoreoflessthan0.20wasremovedfromtheanalysis;thisleadto
theremovaloffiveitems.SeeTable3.10foradescriptionofthefinalitemsincludedinthe
assessment.Thecoefficientalphawasthenrecalculatedwithoutthefiveremoveditems.Internal
consistencyfortheremainingitemswascalculatedat.618.
79
Thentheinter‐itemcorrelationmatrixwasreviewed.Anyitemthathadamajorityof
negativeinter‐itemcorrelationswasdeleted;thisleadtotheremovalofsevenadditionalitems.A
reliabilityanalysiswasconductedandthecoefficientalphawasnow0.713.Afterexaminingthe
“scalesifitemdeleted”table,ameasurecalculatedbySPSSsoftware,thedecisionwasmadeto
removethreeadditionalitems.Thecoefficientwasrecalculatedfortheremaining14itemsofthe
assessmentat0.722.Thiscoefficientalphaisacceptable.ThefinalCOILassessmentconsistedofthe
14itemslistedinTable3.9.SeeAppendixCforthecompleteinstrument.Therewerefouritems
measuringauthorexpertise,fouritemsmeasuringpublishercredibility,fouritemsmeasuringbias,
andtwoitemsmeasuringsourcecredibility.Sixoftheitemshadonlyonescreenshot,sixofthe
itemshadfourscreenshots,andtwooftheitemsrequiredparticipantstoclickonfourhyperlinks.
Table 3.9 Final Items in Validated and Reliable COILItems In Final Version Items Deleted1. Evaluate author: Where would you click to learn more about an author?*
3. Evaluate bias: Which website uses strong words or images to persuade the audience?**
2. Evaluate author: Which websites about Chihuahuas and asthma was created by the most knowledgeable author?***
6. Evaluate Source: Which website uses details from the least reliable source?**
4. Evaluate publisher: Where do you click to learn more about a publisher?*
7. Evaluate publisher: Which website was created by a more reliable publisher?***
5. Evaluate bias: Think about the author’s point of view. What may influence the way he thinks about energy drinks?*
8. Evaluate Author: Given the author's profile page what is her expertise?**
10. Evaluate publisher: Which publisher creates a website with the most credible medical information?**
9. Evaluate Source: Which websites uses details that are from the most reliable source about healthy snacks?***
14. Evaluate bias: Think about the authors' point of view. How does the authors' point of view influence the words and images used on the website?***
11. Evaluate Bias: Which website uses strong words, phrases, or images to persuade readers?***
15. Evaluate author: Look at this website. What is the author's expertise?*
12. Evaluate Sources: Which website uses information from the most reliable source?**
80
16. Evaluate bias: Which website uses strong words or images to persuade the audience?**
13. Evaluate publisher: Who is the publisher of this website?#
18: Evaluate bias: Which website uses strong words or images to persuade the audience?**19: Evaluate author: Which author is the best expert on volcanoes?**
17. Evaluate author. Look at these websites. Which website was created by the author with the most expertise on mosquito ringtones?***20: Evaluate author: Where would you click to learn more about an author?*
21: Evaluate bias: Think about the author's point of view. How does her point of view shape the words and images on this website?#
23: Evaluate Source: Where would you click to learn more about the sources an author used?*
22: Evaluate publisher: Which website was created by a more reliable publisher?***
25: Evaluate Publisher: Which website about smoking hazards was created by the most reliable publisher?**26: Evaluate Source: Which source used in the websites is the most reliable source to answer the question, "What killed the dinosaurs?"**
24: Evaluate Source: Which discussion board post uses details from the most reliable source?*
27: Evaluate Publisher: Who is the publisher of this website?*Notes*‐onescreenshot,#‐clickononelink,**‐fourscreenshots,***‐clickonfourlinks
Onlinereadingcomprehensionassessments.Twomeasuresofonlinereading
comprehensionwerecreatedforthisstudy:ORCAlessrestricted(ORCA‐LR)andORCAmore
restricted(ORCA‐MR).Theassessmentsweredraftedandunderwentnumerousvalidityand
reliabilitytests.Intheinitialiteration,eachassessmentconsistedoftwoseparateInternetinquiry
activities(Leuetal,2007).Eachactivityrequiredstudentstolocate,evaluate,synthesize,and
communicateinformation.Onthelessrestrictedtaskstudentscoulduseanywebsiteonagiven
topic.Onthemorerestrictedtasksstudentswereaskedtolocatespecificwebsitesonagiventopic.
EachparticipantthushadascoreforORCA‐LRandORCA‐MR
81
Thedomainfortheonlinereadingcomprehensiontasksusedinthisstudywassocial
studies.Thisdomainwaschosenbecausepreviouswork(Perfetti,Britt,Georgi,&Mason,1994;Van
Sledright,2002)hashighlightedtheimportanceofreadingandevaluatinginformationacross
multiplesources.(Weinberg,1991).Thus,thedisciplinarydemandsofsocialstudiesfitnicelywith
theskillsrequiredtoreadinonlineenvironmentswherereadersoftenhavetolocate,evaluate,and
synthesizeacrossseveralsources.
TopicsfortheInternetinquiryactivitiesrevolvedaroundtheAmericanRevolution.This
topicwaschosenbecauseitiscoveredinfifthgradeinthestateofConnecticut(StateofConnecticut
DepartmentofEducation,2009).Thus,studentsinthestudyarelikelytohavehadexposuretothe
topicandcontent.Alsostudents,bytheseventhgrade,areexpectedtohavecompetencein
accessinginformationfrommultiplesources,evaluatingarguments,andcommunicatinganswers
usingdigitaltexts(StateofConnecticutDepartmentofEducation,2009).Theoriginaltopicsforthe
activitieswere:(a)thetechnologiesoftheAmericanrevolution‐lessrestricted,(b)theaccuracyof
Leutze’spaintingofGeorgeWashingtonCrossingtheDelaware‐lessrestricted,(c)theturningpoint
oftheAmericanrevolution‐morerestricted,(d)thecausesoftheAmericanrevolution‐more
restricted.
Onthelessrestrictedtasksstudentscouldfindanysourcerelevanttotheirtopic.So,for
example,studentswereaskedtolocateanywebsitewithinformationabouttheaccuracyofthe
Leutze’spaintingofGeorgeWashingtonCrossingtheDelaware.Onthemorerestrictedtasks
studentswereaskedtofindaspecificarticleorwebpagelocatedataspecificwebsite.Forexample
studentswereaskedtofindthewebpage“BattleofSaratoga”onthewebsite
AmericanRevolution.orgforthemorerestrictedtask,theturningpointoftheAmericanRevolution.
TheassessmentscreatedforthisstudywerebasedonORCAassessmentsdevelopedfor
previousstudies(Coiro,2012;Leuetal,2012;2008).Thisformathasstudentsfirstlocate
information,synthesizethisinformation,evaluatethesourcesoftheinformation,takeaposition,
82
andthemcommunicatewhatwaslearned.Table3.10showsthatpreviousadministrationsofonline
readingcomprehensionassessmentsusingthisformathavedemonstratedgoodestimatesofboth
validityandreliability
Table3.10.Descriptive,Validity,andReliabilityScoresonORCAassessments.Instrument/# Items Range N * Validity ** ReliabilityORCA-IM 3-36 of 38 12 66.5% .85-.95
ORCA-Blog 0-30 of 32 89 59.2% .84ORCA-I 0-56 of 60 120 51.7% .92ORCA-II 0-56 of 60 120 44.1% .91ORCA-Iditarod 0-33 of 42 220 53.1% .88ORCA-Iditarod, revised
0-20 of 30 373 41.6% .73
*‐Asmeasuredusingproportionoftotalvarianceexplained**‐AsmeasuredusingCoefficientAlpha(Chronbach,1972)
Eachactivityoriginallyconsistedoffourlocatingscorepoints,fourevaluationscorepoints,
foursynthesisscorepoints,andfourcommunicationscorepoints.Theactivitiesbeganwithatask
introduction,eachofwhichcontainedaproblemandadescription.Thetaskintroductionswere
designedtobeasparallelaspossibleacrossthetasks,regardlessoftherestrictednatureofthe
locatingtasks.ThesearepresentedinTable3.11.
ThelocatingitemsdifferedbetweentheORCA‐LRandtheORCA‐MR.Onthelessrestricted
tasksstudentscouldlocateanyfourrelevantsources.Onthemorerestrictedtasksstudentshadto
locatefourspecificsources.
83
Table3.11TaskIntroductionfortheORCAInternetInquiryTasksTask Less Restricted Task Introductions More Restricted Task IntroductionTask A Introduction
The problem: Is the painting of George Washington crossing the Delaware River historically accurate?Mr. Barnes's history class is debating the accuracy of the painting "George Washington Crossing the Delaware" by Emanuel Gottlieb Leutze.
They are debating if the painting is accurate, or truthful, on their class discussion board.
You will do research online and decide if the painting is accurate.
The problem: What was the turning point of the American Revolution?
Mr. Barnes's history class is debating the turning point of the American revolution on the class discussion board.
You will do research to help them answer the question: What was the turning point of the American Revolution?
Task B Introduction
The problem: What role did some women play in the American Revolution?In Mr. Barnes's history class they are discussing women and the American Revolution. His class is posting messages on the class discussion board.You will do research to answer the question: What role did women play during the American Revolution?
The problem: What were the causes of the American Revolution?Mr. Barnes's history class is talking about the causes of the American Revolution on their discussion board.You will do research online to answer the question: What were the main causes of the American Revolution?
84
Foreachinquiryactivity,studentshadtolocatetwowebsitesonthetopic.Ontheless
restrictedtaskthiswastolocateanytwo,relevantsources.Onthemorerestrictedtaskstudents
wereaskedtofindtwo,specificallydefinedsources.Nexttheyhadtocombinethemainideasofthe
twowebsitesintoasynthesisstatement.Thentheylocatedanadditionalsetoftwowebsites,
followingthesamerestrictedparameters.Nexttheywroteanothersynthesisstatementcombining
whattheyreadonthetwowebsites.
Alloftheremainingitems,regardlessoftherestrictednatureoftheinformationspace,were
parallelfromthispointforward.Thestudentswererequiredtolocateanauthorofoneoftheir
websites,evaluatetheauthor’sexpertise,evaluatetheevidenceusedbytheauthor,andevaluate
theauthor’spointofview.Nextstudentstookapositionandbackeduptheirclaimwithevidence
fromwhattheyread.Finallystudentswererequiredtologontoaclassdiscussionboard,makean
originalpost,explainingwhattheylearned,andrespondtooneotherpersononthediscussion
board.Theitemswerescoredwitha1forcorrectanda0forincorrect.Thedichotomousrubric
usedtoscoreeachitemisavailableinAppendixB.
ORCAcontentvalidation.EachORCAassessmentunderwentathree‐stepvalidation
process.Firstapanelofsocialstudiesexpertsevaluatedthemeasuresduringafocusgroup.Next
themeasureswereevaluatedbyafocusgroupofonlinereadingcomprehensionexperts.Finallya
seriesofcognitivelabs(Ericsson&Simon,1999)wereconducted.
Socialstudiesexpertpanelvalidation.Aftertheinitialinquiryactivitiesweredeveloped,
theassessmentsweregiventoafocusgroupthathadexpertiseinthefieldofsocialstudiesand
technologies.ExpertswererecruitedusingTwitter,asocialmediatool.Arecruitmentcallwentout
usingseveralhashtags,amethodforcategorizingmessages:#sschat‐socialstudieschat,#edtech‐
educationaltechnology,and#edchat‐generaleducationdiscussion.Sevenpeoplerespondedtothe
85
call.Thefinalpanelconsistedoffourexperts.Twoweredoctoralstudents,andtwowereclassroom
teachers.
Thepanelmembersthenreviewedthematerials.Theywereaskedtocompleteeachofthe
fourversionsoftheORCA.Theresearcherthenmetwiththepanelmembersinavideo–conference.
Thepanelmemberswereaskedtoassessthevalidityoftheinstrument.Interviewquestions
focusedonthetasksbeingspecifictothedomainofsocialstudies.Theywerealsoaskedifthe
onlinereadingcomprehensionskillsassessedinthetasksmirroredtheskillsusedbyexpertsinthe
field.Finallytheexpertpanelwasaskedifthetopicsincludedinthetasksweresuitedforthetarget
populationofseventhgrade.
Overall,thecontentexpertsfelttheassessmentsadequatelyrepresentedinquirytasks
requiredinthedomainofsocialstudies.Theyremarkedthatevaluatingtheauthorandsourcesof
evidencewereimportantskillsatthislevel.Onthecommunicationtasktheexpertswanted
studentstoalsonotewheretheyfoundtheirinformation.
Theexpertsdidrecommendthatthetopicoftechnologyandtherevolutionarywarwas
unsuitableforthetask.Theyfeltthatwebsitesaboutthistopicwerescarce;andthosethat
studentswouldfindwereabovethenormalreadinglevelofaseventhgrader.Theexperts
recommendedchoosinganewtopic.TechnologyandtheAmericanRevolutionwasreplacedwith
theRoleofWomenDuringtheAmericanRevolution.
Onlinereadingcomprehensionexpertpanel.Thenextstepincontentvalidationinvolved
expertsinthefieldofonlinereadingcomprehension.TwodoctoralstudentsfromtheNew
LiteraciesResearchLabattheUniversityofConnecticutwererecruitedtovalidatetheORCA
activities.Theymetwiththeresearcherandwentthrougheachtask.Thereviewersfeltthatthe
fouractivitiesadequatelycapturedtheelementsoflocating,evaluating,synthesizing,and
communicating.
86
Cognitivelabs.Thefinalstepincontentvalidationwasaseriesofcognitivelabs(Ericsson&
Simon,1999).EightstudentsfromanaverageSESschooldistrictwereselectedtocomplete
structurallypromptedthink‐alouds(Afflerbach,2010).Astructurallypromptedthinkaloud
interruptsstudentsatkeydecisionpoints.Theirscreenactionsandvoiceswerecapturedusing
IshowU.Thisisascreenrecordingsoftwarethatrecordsallactiononthescreenalongwiththe
student’sandresearcher’svoices.Studentsfirstcompletedatrainingtasktofamiliarizethemselves
withcompletingathink‐aloud.Thestudentsthencompletedonemorerestrictedtaskandoneless
restrictedtaskwithastructurallypromptedthinkaloud.Afterthestudentscompletedallofthe
think‐aloudstheywereasked:(a)howtheassessmentscomparedtohowtheyusuallyreadand
writeinsocialstudies;(b)howtheyopenmultiplewindows;(c)howtheynormallytakenotes
whenconductingonlineresearch.
ThethinkalouddatarevealednumerousissueswiththeformatoftheORCAassessment.
Firstonlocatingtasksthestudentsstruggledwithlocatingtherestrictedwebsitesgivenonlythe
titleofthepagewithouttheURLextensions(.com,.org).Forexample,onewebpagetheywere
askedtofindwaslocatedonawebsiteAmericanRevolution.Thereweremultiplewebsiteswiththe
titleAmericanRevolution.Thusitwasdecidedtoaddtheextensiontoallwebsitetitlesonrestricted
tasks.Studentswerenowaskedto,“Findthewebpage‘TheBattleofSaratoga’onthewebsite
AmericanRevolution.org,”insteadof“Findthewebpage‘TheBattleofSaratoga’onthewebsite
AmericanRevolution.”
Theparticipantsalsofeltthatfindingfourwebsiteswasveryredundant.Furthermoremany
studentscouldnotcompletethetaskinfortyminutes.Itwasdecidedtoreducethenumberof
searchtasksfromfindingfourwebsitestofindingthreewebsites.Thisleadtotherevisionofall
fourversionsoftheinstrumentfroma16‐itemassessment(fourlocate,fourevaluate,four
synthesize,andfourcommunicatescorepoints)toa12‐itemassessment(threelocate,three
evaluate,threesynthesize,andthreecommunicatescorepoints).
87
Thefinalrevisionaroundthelocatingtasksinvolvedaddingahelpfeature.Whenstudents
couldnotlocateawebsitetheywereunabletoproceedtoadditionalitems.Inotherwordsthey
couldnotsummarizeawebsitetheycouldnotfind.Thiscouldleadtodependencyissuesasalow
scoreonasynthesispointmightreflectaninabilitytolocateapage.Thus,ahelpfeaturewasadded
toeachlocatingtask.Students,aftersearchingforfiveminutes,couldrespondthattheywere
unabletolocateawebsite.TheycouldsubmitIDK(Idon’tknow)asananswer.Thentheywere
asked,“DidyoutypeIDK:YesorNo?”Ifstudentsindicated,“Yes”theywouldthenbegiventhelink
toanappropriatewebsite.
Thestudentsallfeltthatthesynthesistaskswerealsoredundant.Studentswereaskedto
combineinformationfromthefirsttwowebsitestheyfound,thesecondtwowebsitestheyfound,
andthentakeaposition.Thestudentsfeltthatthefinalsynthesisstatementtheywrotemirrored
thecommunicationtaskthataskedthemtoexplainwhattheylearned.Inordertoreducethe
feelingofredundancyboththesynthesistasksandthecommunicationtaskswererevised.Onthe
synthesistask,studentsnowhadtosummarizethekeyideasonthefirstwebsitetheyfound.Onthe
secondsynthesistaskstudentshadtocombinethemainideasfromthesecondtwowebsitesthey
found.Onthethirdsynthesistaskstudentshadtotakeapositionandprovideevidenceusing
informationthattheyread.
Thecommunicationtaskswerealsorevised.Thestudentsfeltbeingaskedtotakeaposition
onthecommunicationtaskwastoosimilartotakingapositiononthesynthesistask.Thereforethe
assessmentwasrevisedsostudentsnolongerhadtodoaninitialpostonthediscussionboard,
statingtheiropinionandthenreplyingtoanotherpostonthediscussionboard.Insteadthe
studentsnowhadtoagreeordisagreetooneofthetwostudentresponses,usingevidencefromthe
websitestheyread,explainingwheretheyfoundtheinformation.
ThefinalversionsoftheassessmentsareavailableinAppendixA.TheORCAlessrestricted
andtheORCAmorerestrictedeachconsistedoftwotasks.Therewere12possiblescorepointsin
88
eachtask.Thustherewere24possiblescorepointsfortheORCAlessrestricted.Therewerealso24
possiblescorepointsfortheORCAmorerestricted.
Thedispositionsofonlinereadingcomprehensionmeasure.TheDispositionsofOnline
ReadingComprehensioninstrument(DORC)(O'Byrne&McVerry,2008)wasusedtoestimatethe
dispositionsofonlinereadingcomprehensionamongparticipants.Inthisinstrument,dispositions
ofonlinereadingcomprehensionweremeasuredusingaseriesoffive‐pointLikertscalesinaself‐
reportsurvey.Studentsansweredquestionsthatfellonfivescales:reflectivethinking,critical
stance,flexibility,collaboration,andpersistence.Thesefivescaleswerecombinedtoforma
compositescoreofastudent’sdispositiontowardsonlinereadingcomprehension.Previouswork
establishedareliabilityestimateof.72andcontentvaliditywasensuredthroughthreeroundsof
review,followedbyrevisions,usingexpertpanels(O'Byrne&McVerry,2008).Thefullinstrument
isavailableinAppendixD.
QuantitativeProcedures
AdministeringtheIndependentVariableMeasures
Participantsfirstcompletedthebackgroundknowledgemeasure,thedispositionsofonline
readingcomprehensionmeasure,andtheCOIL,whichmeasuredcriticalevaluationskills.Students
weregivenaccesstoalaptopcomputerduringtheirnormallyscheduledclassperiod.Students
wereassignedacomputerandtheirnumberwasrecorded.Screenrecordingsoftware,IShowU,also
capturedallstudentactivitytoprotectagainstdataloss.
Duringthefirstsessionstudentscompletedalloftheindependentmeasuresbeforethe
onlinereadingcomprehensionassessments.Iwouldvisiteachclassandcompletethebackground
knowledgemeasure,thedispositionsofonlinereadingcomprehension,andtheCOIL.The
backgroundknowledgemeasurewasadministeredfirsttoensurethatnolearninggainsfromthe
inquiryactivitiesledtohigherbackgroundknowledgescores.Studentsthencompletedthe
89
dispositionsofonlinereadingcomprehension.Finallystudentsfinishedthefirsttwelveitemsofthe
COIL.
Followingthisfirstsession,theCOILwasanalyzedtocheckforinternalconsistencyasa
measureofadequatereliability.Asoutlinedabove,theinitial,12‐itemCOILdidnothaveadequate
reliability.OnceitwasevidentthatmoreitemswouldbenecessaryIreturnedtotheschoolto
administeranadditional15items.
Theteachersunderwenttrainingsothattheymightadministerthemeasureofverbal
comprehension.Theycompletedtheassessmentwiththeresearcherandwentovertheprocedures.
Theteachersthengavetheassessmenttotheentireclassina10‐minutesession.Ineachclassroom
thecompletionoftheverbalabilitymeasurewasdoneafterallotherassessmentswere
administered.
AdministeringtheOnlineReadingComprehensionAssessments
AllfourORCAtaskswereadministeredtotheparticipants(n=131)followingaprotocol(see
AppendixC).Theorderofthefourtaskswasrandomlyassignedtoparticipantstoprotectagainsta
learningeffect.Thetaskswereassignedtoensurestudentscompletedamorerestrictedorless
restrictedtaskfollowedbyamorerestrictedtaskorlessrestrictedtask,respectively.Therewere
12differentcombinations.Thecombinationswererandomlyassigned.
ForexampleastudentmightbeassignedtheDelawaretask(lessrestricted)followedbythe
Causestask(morerestricted).TheywouldthencompletetheWomentask(lessrestricted)and
finishwiththeTurningPointtask(morerestricted).TheirscoreontheoverallORCAlessrestricted
wouldbeasumofthescorepointsonboththeDelawareandWomentask.TheirscoreontheORCA
morerestrictedwouldbethesumofthescorepointsonboththeTurningPointandCausestask.
Studentscompletedtheassessmentinfour40‐minuteclassperiods.Theycompletedone
taskperclass.Ateachschoolsitestudentscompletedtwoofthetaskswithinthesamefive‐day
academicweek.Thentoreducethepossibilityofalearningeffect,thestudentsdidnotcompletethe
90
remainingtwotasksforthreeweeks.Theresearcherreturnedtotheschoolsandadministeredthe
finaltwotaskswithinone,five‐daycalendarweek.
Theresearcherbrought25laptopstotheclassrooms.Thecomputerswerepre‐distributed
onstudentdesks.Theresearcherpreparedanentranceslipforeachstudent.Ontheslipwasthe
URLfortheirassessment.Theentranceslipalsohadausernameandpassword.Eachstudentwas
assignedauniqueusernameandpasswordforuseonthediscussionboard.Astheinstructorand
researcherpassedouttheactivitycardstheywouldhitrecordonIShowU,ascreenrecording
software.
Onceeverystudentwasgiventheirmachineandactivitycardtheywereaskedtoenter
theirURLandnavigatetotheirassignedtask.Theresearcherthenreadthroughthetwodirection
screenswiththestudents.(SeeAppendixB).Theresearcherthenwalkedaroundtheroomwhile
studentsbegan.
Theprotocolprovidedstudentswithfiveminutesforeachsearchtask.Theresearcher
wouldremindstudentsatthefiveminutemarktotype“IDK,”ifneeded,andthenclickonthelinkto
gotothecorrectwebsite.Whenstudentsfinishedatasktheywouldraisetheirhandandthe
researcherwouldstopthescreen‐recordingsoftware.
ORCAReliabilityAnalysis.AfterthecompletionofthefirsttwoORCAinquirytasks,one
lessrestrictedandonemorerestricted,theresearcherscoredthetasksusingarubric.(See
AppendixB).Toensurereliability,20%(n=40)ofthetaskswererandomlychosenandgivento
anotherdoctoralstudentstudyingonlinereadingcomprehension.Afterscoringfivetogetherthe
ratersseparatedandscoredtheremaining35tasksalone.Agreementwascalculatedusingasimple
percentageanditexceeded93%.Differenceswereresolvedthroughdiscussion.
TheresearcherthenscoredtheORCAtasks.Testsforinternalconsistencywereconducted
usingacoefficientalpha(Cronbach,1972).Reliabilitytestingwasdoneattwolevels..Asstated,the
ORCAlessrestricted(ORCA‐LR)consistedoftheDelawareandWomentasks.TheORCAmore
91
restricted(ORCA‐MR)consistedoftheTurningPointandCausestasks.Firsteachindividualtask
consistingoftwelvescorepointswasevaluated.ThentheORCAlessrestricted,whichconsistedof
theDelawareandWomentasksforacombined24scorepoints,wasevaluated.ThentheORCAmore
restricted,whichconsistedofTurningPointandCausestasksforacombined24scorepointswas
evaluated.Table3.12presentsthereliabilityestimatesfortheORCA‐LR.Table3.13presentsthe
reliabilityestimatesfortheORCA‐MR.
Table3.12InternalConsistencyoftheORCA‐LRORCAInquiryTask Internal
ConsistencyDelaware(lessrestrictedtaskA) .615Causes(lessrestrictedtaskB) .713ORCA Less Restricted Total .722
Table3.13InternalConsistencyoftheORCA‐LRandORCA‐MRORCA Inquiry Task Internal
ConsistencyTurning Point (more restricted task A) .763Causes (more restricted task B) .769ORCA More Restricted Total .804
QuantitativeAnalysis
Thegoalofthequantitativeportionofthisstudywastoexaminetheuniquecontributions
thatbackgroundknowledge,criticalevaluationskills,anddispositionsmadetomodelsofonline
readingcomprehensionoverandaboveverbalability.Thisstudyexaminedtwomodels.Thefirst
involvedalessrestrictedtask(ORCA‐LR).Thesecondinvolvedamorerestrictedtask(ORCA‐MR).
Thequantitativeportionofthestudyexploredtworesearchquestions:
1. When predicting online reading comprehension of a problem solving task within a less
restricted information space, what is the best fit model, after controlling for verbal
92
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension?
2. When predicting online reading comprehension of a problem solving task within a more
restricted information space, what is the best fit model, after controlling for verbal
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension.
Toanswerthetwoquantitativequestions,sequential(hierarchical)regressionanalysis
(Tabachink&Field,2001)wasusedtoestimatethebestfitmodelsbetweentheindependent
variableofonlinereadingcomprehensionscoresandthedependentvariables,aftercontrolling
forverbalintelligence.Hierarchicalregressionanalysisallowstheenteringofvariablesinone
model,andthenadditionalvariablesareaddedinthesecondmodel.Giventhatthegoalofthe
quantitativeportionofthisstudywastoexaminemodelfit,thevariableswereaddedtothe
modelusingthe“enter”method.The“enter”methodentersallofthevariablesthatwereadded
tothemodelatthesametime.Thiswaschosenoverastepwiseregressioninordertoensure
alloftheindependentvariableswereincludedinthefinalmodel.Inthestepwisemethod,
variablesthatdonotmakeasignificantcontributiontothemodel,afteradditionalvariablesare
added,arethenremoved.
Beforetheregressionmodelswerecalculatedallofthedependentandindependent
variableswereexaminedtoensuretheymettheassumptionsnecessaryforaregressionanalysis.
Firsttheskewnessandkurtosisofthevariableswereexaminedtoensuretheassumptionofnormal
distributionwasmet.Thenthedatawasexaminedforoutliers.Testswerealsoruntoensurethe
assumptionsofhomoscedasticityandmulticollinearityweremet.Thesetestsarereportedin
chapterfour.
93
QualitativeProcedures
QualitativeParticipants
Thepurposeofthequalitativephaseofthismixedmethodsstudieswastoexplorethe
strategyusebystudentswhovaryintheironlinereadingcomprehensionability.Twelvestudents,
fourfromeachparticipatingschool,wereselectedforthequalitativeportionofthestudy.
Performanceonthefirstadministrationoftheonlinereadingcomprehensionassessment,including
oneORCA‐LRandoneORCA‐MRtask,wasusedtodetermineparticipantsinthethinkaloud
activity.Theywereselectedasfollows.
1. First all students in their class were ranked based on scores on the first administration of
the ORCA. Students were divided into two groups: the top 10% of scores and the bottom
10% of scores.
2. These groups were then reviewed by the teacher to allow for the selection of students
who would be comfortable working with an adult on a verbal protocol task.
3. Then four students from each participating school were randomly selected from the list:
two students who scored in the 10% of their class and two students who scored in the
bottom 10% of their class.
Then,forthequalitativeanalysis,thetwelveselectedstudentsweredividedintothree
achievement‐levelgroups(high,average,low)basedonORCAscores,independentofschool.They
wererankedusingacombinedscoreofonelessrestrictedtaskandonemorerestrictedtask.The
studentswererankedfrom1(lowest)to12(highest)andbrokenintothreegroupsoffour.Thislist
resultedinamedianscoreof10.5,outof24.Themiddlegroupincludedfourstudentswithintwo
scoresofthemedian.Thelowgroupincludedscoresmorethantwoscoresbelowthemedian.The
highgroupincludedfourstudentswithscoresmorethantwoscoresawayfromthemedian.
Tenstudentswereincludedinthefinalanalysissincetwostudentshadtoberemovedfrom
thestudyduetoerrorsindatacollection.Thisincludedonestudentinthehighgroupandone
94
studentinthemiddlegroup.Table3.14presentsthelistofstudents(usingpseudonyms)andtheir
performancelevelsonthevariablesofinterestbygroup.
Table3.14QualitativeParticipantsPseudonym Verbal PK ORCA1 ORCA2 Total SchoolHigh Isabella 13 1.5 8 9 17 3Olivia 17 2 9 10 19 3Ava 23 2 9 9 18 3 Medium Alexander 8 2 7 8 15 3Sophia 5 0.5 4 7 13 1Jacob 8 2 7 8 15 2Ethan 7 1.5 2 7 9 2 Low Emma 2 0 1 1 2 2 Michael 3 5 5 3 8 1Jaydan 7 .5 4 1 5 2*=ORCAScoreconsistedofonelessrestrictedandonemorerestrictedtask.QualitativeProcedures
Afterparticipantsforthequalitativethinkaloudwereselectedfromeachresearchsite,the
researchermetwithstudentsone‐on‐oneandtheycompletedasingleonlineinquirytask,using
structuredthinkaloudprocedures,overtwoconsecutivedays.First,thestudentsunderwenta
think‐aloudtrainingsession.Theresearchermodeledtoshowstudentshowtothinkaloudwhile
findingthecapitalofFrance.ThenthestudentswereaskedtopracticebyfindingthecapitalofNew
YorkState.
Studentsthencompletedastructurallypromptedthinkaloudactivitywithinanonline
inquirytask,beingpromptedatfixedstructurallocationstothinkaloudbytheresearcher
(Afflerbach,2002)whilereadingonline.Thestudentswereasked,“Whatwereyouthinking?”atkey
decisionpointssuchasclickingonalink,leavingawebsite,orenteringkeywords.Theiractions
wererecordedusingIshowU.
95
QualitativeAnalysis
Thereweretwogoalsforthequalitativeportionofthisstudy.Thefirstgoalwastocompare
strategydifferencesbetweenthelessrestrictedandmorerestrictedconditions.Thesecondgoal
wastoexaminestrategydifferencesbetweengoodandpooronlinereadersastheycompleted
onlinereadingcomprehensiontasks.Thus,atwo‐stageapproach,utilizingthemethods
recommendedbyAfflerbach(2002)andLeuetal.(2009)wereusedtoanswerthethirdandfourth
researchquestions:
3. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a less
restricted information space?
4. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a more
restricted information space?
Thefirststageofanalysisexamineddifferencesinstrategyuseamongparticipantsbasedon
therestrictednatureoftheinformationspaceandtheironlinereadingcomprehensionability.The
secondstageofanalysisinvolvedamorein‐depthanalysisofpatternsfoundinstageone.Thegoals
ofthissecondstageofanalysisweretwofold:1)toprovideillustrativeexamplesthatmirroredthe
quantitativefindings;2)todeterminetheunderlyingpatternsamongstrategyuseofparticipants
withvaryingdegreesofonlinereadingcomprehensionability.
Stageonedatawereanalyzedfirstthroughthelensofthecomponentsofonlinereading
comprehension:locating,evaluating,synthesizing,andcommunicating.Thisstagebeganby
examiningfrequencytablesofrawcodesforpatterns.Thensuccessivepassesweremadethrough
thedatatoallowfordatareductionandthemestoemerge.Thepurposeofthisstagewastoidentify
patternsinstrategyusethatcouldbeexploredduringthemorein‐depthstage‐twoanalysis.
96
Thesecondstageofanalysisbuiltupontheresultsofthefirststage.Thegoalofthemore
finiteanalysiswastoidentifythemesthatcutacrossthecomponentsofonlinereading
comprehension.Thisstageinvolvedmakingadditionalpassesthroughthedatatocompare
instancesofcodingacrossparticipants.Thepatternsandthemesfromstageonewerethenrefined
andreorganizedataintonewpatterns.Patternidentificationinvolvedaniterativeprocessof
reorganizingthedatafromstageoneandreworkinggroupingssothatthecategorystructures
definedthemesinthedata.
Theanalysisofthethinkalouddatainbothstageswascompletedfollowingbothabductive
(Onwuegbuzie&Leech,2006)methodsandconstant‐comparative(Bogdan&Biklen,2003;
Merriam,1988)methods.Thisdualapproachwasdesignedtoidentifypatternsincognitive
processingwhilealsoallowingforthecodebooktounfoldasthedatawereexamined.
Abductivecoding.Abductivecodingmethods(Onwuegbuzie&Leech,2006)employboth
inductiveanddeductivecodingprocedures.Initialcodingschemeswereinformedbyprevious
work(Leuetal.,2004;Leuetal.,2009).Usingthiswork,alistofexpectedstrategieswascreatedfor
eachofthe12scorepointsineachtask.Theinitialcodebookofexpectedstrategiesappearsin
Table3.15.
ThequalitativedatawascodedusingHyperRESEARCH,asoftwaretoolthatallowsforthe
codingofvideodata.Eachvideocasewasloadedintotheprogram.Thevideoswerethenbroken
intoseparateframes.AseriesofframeswascodedusingthecodebookaspresentedinFigure3.5.
Innumerousinstances,researcherannotationswerealsoaddedtoeachinstanceofthecode.
Table3.15
Initialcodebookforabductiveanalysis
Category CodeLocate K-Locate-Keyword Entry K1-Copy and paste exact words from task
K2-Type exact words from taskK3-Use keywords from taskK4-Revise keywords
97
SR-Locate Read Search Results SR1-Cursor movement indicates reading of results.SR2-Clicks on first linkSR3-Clicks on a link on first pageSR4-Moves to second pageSR5-Returns to search results
LW-Locate Website LW1-Locates website and judges it relevantLW2-Locates website and judges it irrelevantLW3-Skims website
Evaluate CEE-Evaluate Author Expertise CEE1- Finds the author in the text
CEE2- Finds the author on an about us/bio pageCEE3- Infers the author from the text.CEE4- Does a secondary search for the author.CEE5-Does not locate the authorCEE6-Uses an authoritative title to judge the authorCEE7- Uses supporting details from content of the website.CEE8- Uses institutional information to judge author expertiseCEE9-Uses background knowledge to judge author expertise
CES-Evaluate Author1s use of Evidence CES1- Uses the source of claims/evidence to judge use of evidenceCES2-Uses a bibliography or reference to judge use of evidenceCES3-Uses a secondary source to verify information.CES4-Uses overall quantity of content to judge use of evidence
CEP-Evaluate Author Expertise
CEP1- Uses authors perceived level of expertise to describe point of view.CEP2- Uses authors prior experience to describe point of view.CEP3- Uses authors institutional affiliation to describe point of view.CEP4- Uses content of website to describe point of view.
98
SynthesizeSEM-Locate two important details SEM1-Copy and paste important details
SEM2-Copy and paste entire source.SEM3- Copy and paste irrelevant detailsSEM3-Paraphrase important details.SEM4-Paraphrase irrelevant details.
SEC-Combine information from two sources
SEC1- Navigate between multiple tabs/windows.SEC2-Cursor movement provides evidence of reading two sources.SEC3-Copy and paste details from two sourcesSEC4-Copy and paste details from one sourceSEC5-Paraphrase details from two sourcesSEC6-Paraphrase details from one source
SEP-Take a position with evidence SEP1-Student makes a specific claim related to task.SEP2-Student copies and past details in reference to claim.SEP3-Student paraphrases details in reference to claim.SEP4-Student copies and pastes information without making claim.
CommunicateCDB-Correctly use discussion board CDB1-Student can log in to discussion board
CDB2-Student navigates to correct discussion.CDB3- Student responds to another post.CDB4-Student replies to discussion.
CDB-Engage in dialogue CED-Student agreed or disagreed with an initial post.CED-Student did not disagree.
CDE-Provide source of evidence CDE1-Student links to a sourceCDE2-Student refers to a specific sourceCDE3-Student refers to source in general (i.e. websites I read)
99
Figure3.5ScreenShotfromHyperRESEARCH
Followingabductivemethods,additionalcodeswereaddedtothecodebookduring
analysis.Thesecodesdevelopedinductivelyasnewpatternsemergedthatwereeithernot
representedinacodeorcouldnotberepresentedexclusivelybyoneexistingcode.Forexample
manystudentscompletedasearchbyusingtheauto‐fillfeatureofGoogle,whichpredictsasearch
string.Thiscodewasaddedtothecodebook.
Constant‐ComparativeAnalysis.Analyticinductivemethods(Bogdan&Biklen,2003;
Robinson,1951)wereusedrecursivelyacrossthevideodatasources.Afterallofthecaseswere
coded,reportswererunasapreliminaryexaminationofpatternsthatdevelopedusingconstant
comparativeanalyticmethods.First,anoverallreportwasrunonallofthesubareas:locate,
evaluate,synthesize,andcommunicate.Thefrequenciesofthecodeswerematchedwithaspecific
timestampinthevideo.Thisdatawasusedforstageoneanalysis.Ithenwentbacktowatchthe
codedvideosegmentsofeachcodeinstance.Finally,Icomparedtheannotationsofeachcode
100
instance.Aspatternsacrossthefrequencies,videosegments,andannotationsemergedtheywere
noted.
Thenextstepinthestageoneanalysiswastocrosscheckindividualcasesbasedonthe
restrictednatureoftheinformationspaceandthesuccessofstudentsontheORCAinstruments.
Forexample,ifahighperformingstudentexhibitedatendencyto“navigatemultiplewindows”I
examinedifthiswassimilaracrossallparticipantsorjusthighperformingparticipants.Ifthecode
exhibitedstabilityateitherlevelitremainedinthecodebook.
ThetenparticipantsweresplitintothirdsbasedonORCAscores.Theyweredefinedas
highperformingonlinereaders,averageperformingonlinereaders,andlowperformingonline
readers.Areportwasrunforeachsubscale:locate,evaluate,synthesize,andcommunicateforeach
groupofparticipants.Initially,thefrequencyofcodesforeachgroupwascomparedtoother
groups.Then,usingtheinitial,overallpatternsfoundinthefirstpassofthedata,Itriedto
determineifthesamepatternsheldtruebasedonsuccessonthelessrestrictedandmore
restrictedtasks.
Thenextstepinthestageoneanalysiswastoidentifypatternsandthemesfromtheraw
codesandfrequencymatrices.Thefirstlevelofpatternidentificationinvolvedanexaminationof
thefrequencymatrices.Ifdifferencesemergedinthefrequencyofcodesbetweenhigher,average,
andlowerperformingstudentsIreturnedtothevideoforacloserexamination.Eachinstancesof
thecodewasreexaminedandadditionalannotationsmadefocusedonstudentstrategyuse.The
goalofstageoneanalysiswastoidentifypatternsinstrategyusethatledtooverallperformanceon
theORCAassessmentsregardlessoftherestrictednatureoftheinformationspace.Thethemesand
patternsfoundinthestageoneanalysiswerethenusedtoinformthestagetwoanalysis.
Thegoalofthestagetwoanalysiswastodeterminepatternsofstrategyuseamong
participantsbasedontheirabilityandtherestrictednatureoftheinformationspace..Inorderto
condensethepatternsandthemesfromstageoneintomoreexclusivethemesrecursive,analytic
101
inductivemethodswereused(Bogdan&Biklen,2003).Usingthetimestampsfromtheinitialcode
Ireturnedtothevideoandsystematicallyandconsistentlyexaminedthedatatoconfirm,
disconfirm,andgeneratenewpatternsandthemes.
102
ChapterIV
QuantitativeResults
Thisstudyexploredtherelationshipsofbackgroundknowledge,criticalevaluationskills,
anddispositionsofonlinereadingcomprehensioninpredictingonlinereadingcomprehension
whenverbalintelligencewascontrolled.Theserelationshipsweretestedintwodifferentreading
“spaces:”alessrestrictedenvironment,i.e.,wherereaderscouldchoosetheirsources;andamore
restrictedenvironment,i.e.,wherereadershadtolocatespecificwebsitesthatwereprovided.
Thequantitativeportionofthisstudysoughttoanswertwoquestions:
1. When predicting online reading comprehension of a problem solving task within a less
restricted information space, what is the best fit model, after controlling for verbal
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension?
2. When predicting online reading comprehension of a problem solving task within a more
restricted information space, what is the best fit model, after controlling for verbal
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension.
Sequential(hierarchical)regression(Tabachink&Fidell,2001)wasusedtoallowforthe
researchertoenterverbalintelligencetothemodelapriori.Ameasureofverbalintelligencewas
usedasitispresumedtobecausallyrelatedtoreadingcomprehension.Artley(1944)notedthat
verbalintelligenceisakeycomponenttocomprehension.Cain,Oakhill,andBryant(2004)found
thatmeasuresofverbalabilityexplainedthegreatestamountofvarianceinmeasuresofreading
comprehension.Finally,Baddeley,Logie,andNimo‐Smith(1985)alsofoundverbalabilitytobea
strongpredictorofreadingcomprehension.
103
Sincethisstudywantedtoexaminehowbackgroundknowledge,criticalevaluationskills,
anddispositionscontributedtomodelsofonlinereadingcomprehensionitwasdecidedtocontrol
forverbalability.Theremainingindependentvariables:backgroundknowledge,criticalevaluation
skills,anddispositionswereenteredinthesecondsteptotestthebestfitoverandaboveverbal
ability.
Thischapterisorganizedinfoursections.Thefirstsectionreviewsthedatascreening
proceduresforallofthevariables.Thesecondsectionprovidesdescriptivestatisticsfor
independentanddependentvariablesafterthedatascreening.Thethirdsectiondescribesthe
resultsoftheregressionmodelforthelessrestrictedinformationspace.Thefinalsectionprovides
theresultsoftheregressionmodelforthemorerestrictedinformationspace.
DataScreening
Scoresforthepredictorvariableswerecollectedfor131students.However,dueto
absencesandlossofdata33studentswereremovedwithlist‐wisedeletion.Thisleftasampleof98
participants.Thisexceedstheminimumsamplesizeof77forahierarchicalregressionwithan
effectsizeof0.15,adesiredpowerof.80,andoneindependentvariableinthefirststepandthree
independentvariablesinthesecondstep(Sloper,2011).ThisanticipatedCohen’sf2of0.15isa
mediumeffect.Theestimateofpowersizeof.80waschosentoensureanadequateeffect(Sloper,
2010).
Theestimatedeffectsizewaschosenbasedonthereviewoftheresearch.Iexamined
previousstudiesthatusedsimilarregressiontechniques.Iftheresearcherdidnotpresentaneffect
sizeIcalculatedtheeffectsizefromthegivendata.Icalculatedaneffectsize(Cohen’sf2)of.0740
fromthedatapresentedbyBråten,Strømsø,&Britt(2009).Thisstudyusedhierarchicalregression
toexploresourceevaluation.Theyusedintertextual(multipletext)comprehensionasdependent
variableandbackgroundknowledgeasanindependentvariable.Inthesamestudy,Bråten,
104
Strømsø,&Britt(2009)addedtrustindocumenttypetotheirintertextualcomprehensionmodel.
Thismodelhadaneffectsize(Cohen’sf2)of.025316.
IalsousedthedatapresentedbyCoiro(2011).Thisstudyexaminedtheunique
contributionanonlinereadingcomprehensionmeasurehadinpredictingperformanceonan
additionalmeasureofonlinecomprehensionaftercontrollingforbackgroundknowledgeand
offlinereadingability.Whenofflinereadingabilitywasaddedtotheregressionmodelwith
backgroundknowledgeandthescoresonthefirstmeasureofonlinereadingcomprehensionthere
wasaneffectsize(Cohen’sf2)of0.1286whenofflinereadingabilitywasaddedtoaregression
modelwithonlinereadingabilityandbackgroundknowledge.Whenanadditionalmeasureof
onlinereadingcomprehensionwasaddedtothemodeltherewasaneffectsizeof.366.Therefore
myestimatedeffectsizeofCohen’sF2=0.15isaconservativeestimate.
Missingdata.Eachcasewasexaminedforcompletenessofdata.Tobeincludedinthefinal
samplestudentsneededtohavecompletedeightdifferentmeasuresonsevendifferentschooldays.
Themajorityofdatalossinthesamplewasduetostudentabsencesononeofthesevendifferent
schooldays.(SeeTable4.1forasummaryofmissingdata.)Asecondarysourceoflostdatawas
softwaremalfunctionorusererrorthatdidnotresultinscoresbeingrecorded.
Table4.1SummaryofMissingData Measure N Missing School 1 School 2 School 3ORCA 19 9 5 5Background knowledge
9 2 3 4
COIL 13 3 6 8Dispositions 10 3 4 3Total N missing 33 14 8 11
Descriptivestatistics.Means,rangesandstandarddeviationsforthemeasurespriorto
datascreeningarereportedinTable4.2.Afterexaminingthedescriptivestatistics,testswererun
toensurethattheassumptionsforamultipleregressionanalysisweremet.Examinationof
105
skewnessandkurtosisscoresandoutlierswereusedtoassesstheassumptionofnormality,
linearity,andhomoscedascity.Durbin‐Watsonscoreswereusedtotestindependenceofvariables.
Finally,collinearitydiagnosticswereusedtoassessmulticollinearityissues.
Table4.2Range,Means,andStandardDeviationsforDependentandIndependentVariables N Minimum Maximum Mean SDVerbal Intelligence 98 0 24 10.906 5.197Background knowledge 98 0 11 1.67 2.13COIL 98 1.0 11 5.187 2.094Dispositions 98 1.95 5.00 3.419 .4889Less Restricted 98 0 18 8.394 4.117More Restricted 98 0 20 7.15 4.48 Normality.Toassessthenormalityoftheunivariatedistributions,skewnessandkurtosis
statisticswerecalculatedforeachofthepredictoranddependentvariables.Theseresultsare
presentedinTable4.3.Therewasleptokurticdistributionindicatingahighprobabilityforextreme
valuesinthebackgroundknowledgemeasure(3.569).Thelargenumberofstudents(n=58)who
scoredlessthanoneontheirrecalloffactsfromtheAmericanRevolutionmayexplaintheextreme
violationofnormality.
Table4.3SkewnessandKurtosisRatiosforDependentandIndependentVariablesVariable Skewness Ratio Kurtosis RatioVerbal Intelligence 1.37 -.467Background knowledge 1.74 3.59COIL .138 -.559Dispositions .50 1.794Less Restricted ORCA -.578 .136More Restricted ORCA -.438 .576
Inordertomeettheassumptionofnormalityofunivariatedistributionsthebackground
knowledgemeasurewastransformedusinglogarithmictransformation(Osborne,2010).This
resultedinanormaldistributionasindicatedinTable4.4.This,however,didreducemuchofthe
varianceinscores
106
Table4.4TransformedBackgroundknowledgeIndependentVariable Minimum Maximum Mean SD Skewness KurtosisBackground knowledge
1.00 1.32 1.063 .0725 1.269 .425
Linearity,homoscedascity,andindependence.Todetermineiftheassumptionsof
linearityandhomoscedascityweremettheresidualplotswereexamined(seeFigure4.1).The
Durbin‐Watsonstatisticwasusedtotesttheassumptionofindependenceofvariables.TheDurbin‐
Watsonstatisticforboththemorerestricted(1.689)andthelessrestricted(1.706)satisfiedthe
assumptionofindependenceofvariables.
Figure4.1.Scatterplotsandhistogramsofresidualplots.
Outliers.Outliers,whicharepoorfitsfortheregressionmodel,canlowermultiple
correlations.Totestforoutliers,Cook’sdistancemeasurewasused(Tabachink&Fidell,2001).
107
FirstCook’sdistancewascalculatedforthelessrestrictedregressionmodel.Thesescoreswere
thenanalyzedinaboxplotgraphFigure4.2presentstheboxplotofCook’sdistance.
Figure4.2.Boxplotofcook’sdistance.
Anexaminationoftheboxplotrevealedsevenoutliers.Theblacklinerepresentsthe
mediandistance.Theboxrepresentsthefirstandthirdquartilesdistances.Tobeconsideredan
outlierthescoreshadtobeatadistancegreaterfromthemedianscoremorethan1.5timesfrom
theinterquartilerange.Thesesevenscoresareplottedindividuallyascircles.Bothregression
models,lessrestrictedandmorerestrictedwerethentestedwithandwithoutthesevenoutliers.
Theremovaloftheoutliersaffectedthevariablesofinterest.
Nexttwodifferentmodelswererun.Thefirstmodelremovedallsevenoutliers.Thesecond
modelremovedthethreemostextreme.Themodelswiththethreeextremeoutliersremovedwere
thesamewithallsevenoutliersremoved.Thereforeitwasdecidedtorunthemodelswithoutthe
threemostextremeoutliers.Thisleftafinalsampleof95participants.
Multicollinearity.Collinearitydiagnosticsofthebivariatecorrelationsandrelated
statisticsindicatedsomemulticollinearityconcerns.Notolerancevaluewaslessthan0.2andno
varianceinflationfactorsweregreaterthan4.However,inthemorerestrictedmodel,the
108
eigenvalueforthefifthfactor(.002)andtheconditioningindicesexceeded30(47.34).Intheless
restrictedmodeltheeigenvaluealsoapproachedzero(.002)andtheconditioningindicesexceeded
30(47.957).Thisincreaseinmulticollinearitycouldleadtoinflatedstandarderrorsforthe
coefficients.Totesttheeffectofcollinearitythemodelswererunwithandwithoutthedispositions
variables.WithoutdispositionstheR2=.447,andwithdispositionsincludedinthemodeltheR2=
.431.Sincethisdifferencewasnegligibleitwasdecidedtokeepallindependentvariablesinthe
model.
DescriptiveStatistics
Onlinereadingcomprehensionassessment.BasedonthemeansoftheORCA
assessments,themorerestrictedenvironmentwasamoredifficultinformationspaceforstudents
touse.Anexaminationofdescriptivestatisticsrevealedadifferenceinoverallperformancebased
onSESoftheschool.Oneachitemtype(Locate,Evaluate,Synthesize,andCommunicate)students
couldearnamaximumofsixpoints.Anexaminationofmeanscoresfortheitems(seeTable4.5)
indicatedthatinbothtypesofinformationspacescriticalevaluationofwebsiteshadthelowest
meanscore.
Table4.5MeanscoresofL,E,S,Cperformance n Less Restricted More Restricted M SD M SDLocate 95 3.509 1.483 2.775 1.508Evaluate 95 .9554 1.101 .8725 .9917Synthesize 95 2.732 1.483 2.683 1.622Communicate 95 1.554 1.328 1.206 1.269Total 95 8.394 4.117 7.15 4.48
Apairedsamplet‐testwasconductedtocomparethemeanscoresonoftheoftheORCA‐LR
andtheORCA‐MR.Thetestwassignificant(t=4.088,p=.000)withameandifference1.406,SD.33).
ThiswouldindicatethattherewasasignificantdifferenceinstudentscoresontheORCA‐LRand
109
theORCA‐MR.Anexaminationofthehistogramsindicatesthatstudentsdidnotscoreashighonthe
ORCA‐MR.SeeFigure4.1..
Aonewayrepeatedmeasuresanalysisofvariancewasconductedtocomparestudents’
scoresineachofthefourskillareasontheORCA‐LR.Themultivariatestatisticsrevealeda
significanteffectforLESC(Locate,Evaluate,Synthesize,Evaluate)skillareas,Wilks’Lambda=.133,
F(1,93)=408.72p<.005.,multivariatepartialetasquared=.818.Ananalysisofpairwise
comparisonsusingaBonferronicorrectiontocontrolforTypeIerrorfoundasignificantdifference
betweeneachofthefourskillareasandeachoftheothersskills(p<.05forallpairwise
comparisons).Itemsrequiringstudentstolocateinformationweretheeasiest(M=3.509SD=
1.403),followedbysynthesisitems(M=2.732,SD=1.483),communicateitems(M=1.554,SD
=1.328),andfinallyevaluateitems(MD=.9544,SD=1.101),whichwerethehardest.
Aonewayrepeatedmeasuresanalysisofvariancewasconductedtocomparestudents’
scoresineachofthefourskillareasontheORCA‐MR.Themultivariatestatisticsrevealeda
significanteffectforLESCskillareas,Wilks’Lambda=.214,F(1,93)=818.68p<.005.,multivariate
partialetasquared=.214.AnanalysisofpairwisecomparisonsusingaBonferronicorrectionto
controlforTypeIerrorfoundasignificantdifferencebetweeneachofthefourskillsareasandeach
oftheothersskills(p<.05forallpairwisecomparisons).Itemsrequiringstudentstolocate
informationweretheeasiest(M=2.755SD=1.508),followedbysynthesisitems(M=2.683,
SD=1.633),followedbycommunicateitems(M=1.206,SD=1.269),andfinallyevaluateitems
(MD=.8725,SD=.9917),whichwerethehardest.
Thebivariatecorrelationforthetwotestformatswas.658,whichwassignificant,p=.00.
ThebivariatecorrelationspresentedinTable4.6demonstratedthatthemajorityoftheitemson
thelessrestrictedwererelated.Thecommunicationitems,however,didnotsignificantlycorrelate
withtheotheritems.Theitemsonmorerestrictedwerealsorelated.Onceagainthe
communicationitemsdidnotcorrelatewiththeotheritemsexceptfortheevaluationitems.Across
110
theformats(ORCA‐MRandORCA‐LR)alloftheitemscorrelatedsignificantlywiththeir
counterparts.
Table4.6BivariateCorrelationsbyItemType LR-
EvaluateLR-Synthe-size
LR-Comm-unicate
MR-Locate
MR-Evaluate
MR-Synthesize
MR Comm-unicate
LR-Locate .295* .552** -.107 .356** .301** .370** .180LR-Evaluate .556** -.135 .346** .620** .442** .282**LR-Synthesize .064 .441** .565** .607** .262*LR-Communicate .042 .003 -.018 .214*MR-Locate .463** .618** .110MR-Evaluate .618** .246*MR-Synthesize .198
VerbalIntelligence.TheverbalcomprehensionvocabularytestoftheKitofFactor
ReferencedCognitiveTest(Ekstrom,French,Harman&Derman,1976)wasusedasameasureof
verbalintelligence.Themeanforthetestwas10.726withastandarddeviationof5.374.
Backgroundknowledge.Scoresonthebackgroundknowledgemeasureaboutthe
AmericanRevolutionwerelow.Themeanscorewas1.758withastandarddeviationof2.20.Forty‐
threeofthe95finalparticipantsincludedinthesurveycouldnotrecallatleast1relevantideaunit
abouttheAmericanRevolution.Thefrequencyofbackgroundknowledgescoresarepresentedin
Table4.7.
111
Table4.7FrequencyofBackgroundknowledgeScoresScore Frequency0 340.5 81 101.5 52 122.5 13 63.5 44+ 15
CriticalEvaluation.Afourteenitemmultiple‐choiceassessmentwasusedtomeasure
students’abilitytoevaluatewebsites.Themeanscorewas5.1053withastandarddeviationof
2.013.PerformanceontheCOILvariedbyconstructanditemtype.Themeanscoresofeachitemon
theCOILarepresentedinTable4.8.
112
Table4.8DescriptiveStatisticsforCOILItemsItems In Final Version Mean SD1. Evaluate author: Where would you click to learn more about an author?*
.84 .37
2. Evaluate author: Which websites about Chihuahuas and asthma was created by the most knowledgeable author?***
.38 .49
4. Evaluate publisher: Where do you click to learn more about a publisher?*
.62 .49
5. Evaluate bias: Think about the author’s point of view. What may influence the way he thinks about energy drinks?*
.60 .49
10. Evaluate publisher: Which publisher creates a website with the most credible medical information?**
.32 .47
14. Evaluate bias: Think about the authors' point of view. How does the authors' point of view influence the words and images used on the website?***
.48 .50
15. Evaluate author: Look at this website. What is the author's expertise?*
.71 .45
16. Evaluate bias: Which website uses strong words or images to persuade the audience?**
.52 .50
18: Evaluate bias: Which website uses strong words or images to persuade the audience?**
.28 .45
19: Evaluate author: Which author is the best expert on volcanoes?** .44 .5023: Evaluate Source: Where would you click to learn more about the sources an author used?*
.60 .49
25: Evaluate Publisher: Which website about smoking hazards was created by the most reliable publisher?**
.30 .46
26: Evaluate Source: Which source used in the websites is the most reliable source to answer the question, "What killed the dinosaurs?"**
.33 .47
27: Evaluate Publisher: Who is the publisher of this website?* .47 .50Notes*‐onescreenshot,#‐clickononelink,**‐fourscreenshots,***‐clickonfourlinks
DispositionsofOnlineReadingComprehension.Dispositionsofonlinereading
comprehensionweremeasuredusingafive‐pointLikertscaleselfreportsurvey.Students
answeredquestionsthatfellonfivescales:reflectivethinking,criticalstance,flexibility,
collaboration,andpersistence.Thesefivescaleswerecombinedtoformacompositescoreofa
student’sdispositiontowardsonlinereadingcomprehension.MeanscoresontheDORCwere3.382
withastandarddeviationof.471.
113
ResultsForResearchQuestionOne:OnlineReadinginaLessRestrictedInformationSpace
Thefirstresearchquestionexploredthebest‐fitmodelforonlinereadingcomprehensionina
lessrestrictedinformationspace.Itasked:
Whenpredictingonlinereadingcomprehensionofaproblemsolvingtaskwithina less
restricted information space, what is the best fitmodel, after controlling for verbal
intelligence, of the percentage of variance accounted for by each of the following
variables:backgroundknowledge,criticalevaluation,anddispositionsofonlinereading
comprehension?
Scoresforthefinalanalysisincluded95studentsafterthreeoutliersweredeleted.Means,
rangesandstandarddeviationsforthemeasuresarereportedinTable4.9.Bivariatecorrelations
forthedependentandindependentvariablesarelistedinTable4.10.InordertoanswerResearch
Question1ahierarchicalregressionwasused.TotalscoresonthelessrestrictedORCA(ORCA‐LR)
wereenteredasthedependentvariable.Verbalabilitywasenteredinthefirststep.Thenthe
independentvariables:backgroundknowledgescores,criticalevaluationscores,andthenewly
computeddispositionscoreswereaddedinasecondstep.
Table4.9N,MinimumandMaximumScores,Means,StandardDeviations(SD),forDependentandIndependentVariables N Minimum Maximum Mean SDVerbal Intelligence 95 0 24 10.726 5.374Background knowledge 95 0 11 1.758 2.20COIL 95 1.0 9 5.1053 2.013Dispositions 95 1.88 4.56 3.382 .47114ORCA-LR 95 0 20 6.8421 4.16ORCA-MR 95 0 18 8.3053 3.98
114
Table4.10BivariatecorrelationsofDependentandIndependentVariables ORCA-
MRVerbal Intelligence Background
knowledgeCOIL Dispositions
ORCA-LR .658** .422** 379** .462** .195ORCA-MR .414** .436** .574** .264*Verbal .329** .263** -.088Background knowledge
.314** .128
Critical Evaluation
.122
Note:**=Correlationsignificantatthe.01level,*=significantat.05level
Resultsofthelinearregression,aspresentedinTable4.11,indicated,firstthatverbal
intelligence,asmeasuredbyavocabularytest,explained16.9%ofthevariance,whichwas
significant,F(1,93)=20.530,p<.001.Afterverbalabilitywasaccountedfor,thethreepredictor
variablesofbackgroundknowledge,evaluationskills,andscoresonaself‐reportmeasureof
dispositionsofonlinereadingcomprehensionexplainedanadditional22.4%ofthevarianceonthe
scoresofthelessrestrictedORCA,whichwassignificantF(1,92)=15.415p<.001.Inthefullmodel,
scoresonacriticalevaluationmeasureexplainedasignificantamountofuniquevariance(β=.335
p<.000).Backgroundknowledgealsocontributedsignificantlytothepredictionofscoresonthe
lessrestrictedORCA(β=.211p=.021).Dispositionsofonlinereadingcomprehensiondidnot
contributesignificantlytopredictionsinthemodel(β=.117p=.162).Theadditionofthevariables
inthesecondstepofthelessrestrictedmodelhadalargeeffect,Cohen’sF=.358.
115
Table4.11ResultsofHierarchicalRegressionfortheLessRestrictedORCAModel β Sig ΔR2 ΔFModel 1 .205 21.030*Verbal Comprehension .429 .00 Model 2 .224 11.454*Verbal Comprehension .270 .003Critical Evaluation .335 .000Background knowledge .241 .007Disposition .124 .137Note:*=p<.01 Interactioneffectsrefertothecombinedeffectsofvariablesonthedependentvariableand
theyshouldbeconsideredinmultipleregressions(Pedhazur&Schmelkin,1991).Howeverwhen
testinginteractionsitisimportanttocentertheinteractionstoreducethechanceof
multicollinearity(Tabachink&Fidell,2001).Firstthefourindependentvariableswerecentered
(Aiken&West,1991;Judd&McClelland,1989)bysubtractingthemeanfromeachscore.This
resultedineachvariablehavingameanofzero(Aiken&West,1991).Thenaninteractiontermwas
computedforeachvariable.Finallyahierarchicalregressionwasrunwiththeeachcentered
variableinthefirstmodelandthecenteredvariableandinteractionterminthenextmodel.This
testediftheinteractionvariableshouldbeintroducedtothefullmodel(Aiken&West,1991).
Resultsoftheseanalysesfoundnosignificantinteractionbetweenverbalabilityandcritical
evaluationscorest(93)=1.298,p=.198;nosignificantinteractionbetweenverbalabilityand
backgroundknowledget(93)=‐.368,p=.713;nosignificantinteractionverbalabilityand
dispositionst(93)=‐.408,p=.684;nosignificantinteractionbetweencriticalevaluationscoresand
backgroundknowledget(93)=1.340,p=.184;nosignificantinteractionbetweendispositionsand
backgroundknowledget(93)=1.024,p=.308.
Summaryofresultstoresearchquestionone.Theresultsfromresearchquestionone
indicatethataftercontrollingforverbalintelligence,scoresonameasureofstudents’abilityto
criticallyevaluatewebsitesisthebestpredictorofperformanceonanonlinereading
116
comprehensionassessmentinalessrestrictedspace(β=.334,p=.000).Backgroundknowledge
wasalsoasignificantpredictorofperformanceonanonlinereadingcomprehensionassessmentin
alessrestrictedspace(β=.241,p=.007).Dispositionsdidnotmakeasignificantpredictionin
performanceonanonlinereadingcomprehensionassessmentinalessrestrictedspace(β=.124,
p=.137).
ResultsForResearchQuestionTwo:OnlineReadinginaMoreRestrictedInformationSpace
Thesecondresearchquestionexploredthebestfitmodelforonlinereadingcomprehensionin
amorerestrictedinformationspace.Itasked:
Whenpredictingonlinereadingcomprehensionofaproblemsolvingtaskwithinamore
restrictedinformationspace,whatisthebestfitmodel,aftercontrollingforverbal
intelligence,ofthepercentageofvarianceaccountedforbyeachofthefollowing
variables:backgroundknowledge,criticalevaluation,anddispositionsofonlinereading
comprehension.
ToanswerResearchQuestiontwo,anotherhierarchicalregressionwascalculated.
Verbalabilitywasaddedinthefirststep;followedbytheotherindependentvariablesinthe
secondstep.TotalscoresonthemorerestrictedORCA(ORCA‐MR)wereenteredasthe
dependentvariable.Verbalabilitywasenteredinthefirststep.Thentheindependent
variables:backgroundknowledgescores,criticalevaluationscores,andthenewlycomputed
dispositionscoreswereaddedinasecondstep.
Inthefirststep,verbalabilityexplained15.2%ofthevariance,whichwassignificant,F(1,94)
=17.024,p<.001.Afterverbalabilitywasaccountedfor,thethreepredictorvariablesof
backgroundknowledge,evaluationskills,andscoresonaself‐reportmeasureofdispositionsof
onlinereadingcomprehensionexplainedanadditional38.1%ofthevarianceonthescoresofthe
morerestrictedORCA,whichwassignificantF(1,92)=26.328p<.000.
117
Inthefullmodelscores,aspresentedinTable4.12,acriticalevaluationmeasureexplained
asignificantamountofuniquevariance(β=.499p<.000).Backgroundknowledgealsocontributed
significantlytothepredictionofscoresonthemorerestrictedORCA(β=.189p=.017).Dispositions
alsomadeasignificantpredictiontothemodel(β=.291p=.006).Thisexplanationhadalarge
effect,Cohen’sF=.631.
Table4.12ResultsofHierarchicalRegressionfortheMoreRestrictedORCAModel β Sig ΔR2 ΔFModel 1 .152 17.024*Verbal Intelligence .414 .00 Model 2 .381 25.008*Verbal Intelligence .219 .006Critical Evaluation .499 .000Background knowledge .189 .017Disposition .206 .006Note:*=p<.01 Interactioneffectswereonceagaintestedtoseeiftheyshouldbeaddedtothemodel.
Resultsoftheseanalysesfoundnosignificantinteractionbetweenverbalabilityandcritical
evaluationscorest(96)=.104,p=.917;nosignificantinteractionbetweenverbalabilityand
backgroundknowledget(104)=‐.132,p=.895;nosignificantinteractionverbalabilityand
dispositionst(96)=‐.869,p=.387;nosignificantinteractionbetweencriticalevaluationscoresand
backgroundknowledget(97)=.281,p=.780;nosignificantinteractionbetweendispositionsand
backgroundknowledget(93)=.936,p=.351.
Summaryofresultstoresearchquestiontwo.Onceagain,aftercontrollingforverbal
intelligencescores,ameasureofcriticalevaluationofwebsitesmadethelargestpredictionin
performanceonameasureofonlinereadingcomprehensioninamorerestrictedenvironment(β=
.499p<.000).Backgroundknowledgealsocontributedsignificantlytothepredictionofscoreson
themorerestrictedORCA(β=.189p=.017).Adifferencebetweenthelessrestrictedandthemore
118
restrictedenvironmentwerescoresontheself‐reportmeasureofdispositions.Dispositionsof
onlinereadingcomprehensionmadeasignificantpredictiontothemorerestrictedmodel(β=.291
p=.006).
Exploringbi‐variatecorrelations.Additionalanalyseswereconductedtofurtherexplorethe
relationshipsbetweenthedependentandindependentvariables.Backgroundknowledgewas
significantlycorrelatedwithbothversionsoftheORCA.ThePearsoncorrelationbetween
backgroundknowledgeandtheORCA‐LRwas.370p=.000.Thecorrelationbetweenbackground
knowledgeandtheORCA‐MRwas.433p=.000.Table4.13liststhePearsoncorrelationsforitem
typesoneachformat.Thebackgroundknowledgemeasurewassignificantlycorrelatedwithevery
itemtypeexceptforthecommunicationitems.
Table4.13Pearson’srcorrelationsbetweenVerbalIntelligenceandORCAitems Locate Evaluate Synthesize CommunicateORCA-LR .239* .372** .375** .028ORCA-MR .250* .431** .420** .157Note*=significantat.05level;**=significantat.01level
ScoresonthecriticalevaluationmeasureweresignificantlycorrelatedwiththeORCA.The
PearsoncorrelationfortheORCA‐LRwas.462,p=.000.ThecorrelationfortheORCA‐MRwas.574
p=.000.Table4.14liststhePearsoncorrelationsforitemtypesoneachformat.Thecritical
evaluationmeasurewassignificantlycorrelatedwitheveryitemtypeexceptforthecommunication
items.
119
Table4.14Pearson’srcorrelationsbetweenCOILandORCAitems Locate Evaluate Synthesize CommunicateORCA-LR .235* .414** .397** .065ORCA-MR .432* .534** .575** .109Note*=significantat.05level;**=significantat.01level
Self‐reportscoresontheDORCdidnothaveasstrongacorrelationwiththeORCAasthe
otherindependentvariables.TheDORCandtheORCA‐LRhadaweakandnon‐significant
correlationof.122p=.248.TheDORCandtheORCA‐MRhadaweakbutsignificantcorrelationof
.212p=.042.TheDORCdidnotsignificantlycorrelatewithanyitemtypeontheORCA‐LRbutdid
haveweak,butsignificantcorrelationswiththesynthesisandevaluationitemsontheORCA‐MR.
ThesecorrelationsaredisplayedinTable4.15.
Table4.15Pearson’srcorrelationsbetweenCOILandORCAitems Locate Evaluate Synthesize CommunicateORCA-LR .169 .090 .093 -.014ORCA-MR .213* .225* .131 .065Note*=significantat.05level;
ExaminingDispositionScores.GiventhelowcorrelationbetweentheDORCscoresand
theORCAscoresadecisionwasmadetoexaminethecorrelationsofthesubscalesoftheDORC:
reflectivethinking,criticalstance,collaboration,flexibility,andperformanceontheORCAThedata,
asdisplayedinTable4.16,demonstratedthatscoresonthecollaborationsubscalehadanegative
correlationwithscoresonboththeORCAlessrestrictedbutnottheORCAmorerestricted.
120
Table4.16BivariateCorrelationsofDORCSubscalesandORCA ORCA-LR ORCA-MRReflective Thinking .195 .269**Critical Stance .164 .231*Collaboration -.115 .009Flexibility .035 .122Persistence .213* .236*Note*=significantat.05level;**=significantat.01level
ChapterSummary
Thischapterpresentedtheresultsofastudyseekingtofurtherenrichourunderstandings
aboutmodelsofonlinereadingcomprehension.Tworesearchquestionswereaddressedinthe
quantitativeportionofthisstudy:
1. Whenpredictingonlinereadingcomprehensionofaproblemsolvingtaskwithina
lessrestrictedinformationspace,whatisthebestfitmodel,aftercontrollingfor
verbalintelligence,ofthepercentageofvarianceaccountedforbyeachofthe
followingvariables:backgroundknowledge,criticalevaluation,anddispositionsof
onlinereadingcomprehension?
2. When predicting online reading comprehension of a problem solving task within a more
restricted information space, what is the best fit model, after controlling for verbal
intelligence, of the percentage of variance accounted for by each of the following
variables: background knowledge, critical evaluation, and dispositions of online reading
comprehension?
Specificallythisstudyexaminedtherelativecontributionsofbackgroundknowledge,
criticalevaluationskills,anddispositionsofonlinereadingcomprehensionaftercontrollingfor
verbalintelligence.Thesevariablesweretestedintwodifferentinformationspaces.Thefirst
context,alessrestrictedinformationspace,allowedstudentstosearchforanyrelevantwebsite.
121
Thesecondinformationcontext,amorerestrictedinformationspace,requiredparticipantstofind
specificsources.
Sequential(hierarchical)regression(Tabachink&Fidell,2001)wasusedtoallowthe
researchertoenterverbalintelligencetothemodelapriori.Avocabularytestwasusedtomeasure
verbalintelligence.Ineachmodelthescoresontheverbalintelligencemeasureexplaineda
significantamountofthevariance,andwereenteredinthefirststep.Inthelessrestrictedmodel
verbalintelligenceexplained16.9%ofthevariance,whichwassignificant,F(1,93)=20.530,p<.001.
Inthemorerestrictedmodelverbalintelligenceexplained15.2%ofthevariance,whichwas
significant,F(1,94)=17.024,p<.001.Theadditionofcriticalevaluationscores,background
knowledgescores,anddispositionscoresinthenextstepexplainedanadditional22.4%variancein
thelessrestrictedmodelandanadditional38.9%ofthevarianceinthemorerestrictedmodel.
Inbothmodels,lessrestrictedandmorerestricted,criticalevaluationmadethelargest
contributionafteraccountingforverbalintelligence.Inthelessrestrictedmodelscoresona
measureofcriticalevaluationwereasignificantpredictor(β=.334,p=.000)ofonlinereading
comprehension.Inthemorerestrictedmodelcriticalevaluationwasalsoasignificantpredictorof
performanceonameasureofonlinereadingcomprehension(β=.499p<.000).
Thisresultwasconsistentwithcognitiveflexibilitytheory.Itwaspredictedthatstudents
whoscoredwellonameasureofcriticalevaluationskillsmayhave“advancedwebskills”andthus
wouldscorehigherontheORCAassessments.Furthermorestudentswhoscoredlowonthe
measuremayhavemadeerrorsofoversimplificationorhavehadbiasintheirmental
representationofthesource.Thiswouldleadtoscoresonthecriticalevaluationmeasurebeinga
strongpredictorontheORCAassessment.
Thisresultwasalsoconsistentwithatheoryofnewliteraciesofonlinereading
comprehension.Criticalevaluationofwebsitesisakeycomponentofonlinereading
122
comprehension.Inbothmodelsscoresonthecriticalevaluationinstrumenthadthelargest
predictiveweight.
Inbothmodels,lessrestrictedandmorerestricted,backgroundknowledgescoresalso
madeasignificantprediction.Backgroundknowledgewasasignificantpredictorofperformanceon
anonlinereadingcomprehensionassessmentinalessrestrictedspace(β=.241,p=.007).).
Backgroundknowledgealsocontributedsignificantlytothepredictionofscoresonthemore
restrictedORCA(β=.189p=.017).
Itwaspredicted,basedontheprinciplesofcognitiveflexibilitytheory,thatbackground
knowledgewouldbeasignificantpredictorinbothmodels.Furthermorethepredictionwasthat
backgroundknowledgewouldnotbethestrongestpredictorineitherthelessrestrictedorthe
morerestrictedmodelasstudentswouldhavetorelymoreon“advancedwebskills”ratherthan
topicalknowledge.
Itwaspredicted,basedontheprinciplesofatheoryofnewliteraciesofonlinereading
comprehensionthatbackgroundknowledgewouldhavearoleinpredictingperformanceona
measureofonlinereadingcomprehensionbutitwouldnotbethestrongestpredictorof
performance.Resultsofthisstudyconfirmedthisprediction.Scoresonanassessmentofcritical
evaluationskillswerethestrongestpredictors.
Scoresonaself‐reportmeasureofdispositionsofonlinereadingcomprehensionwereonly
significantpredictorsinthemorerestrictedmodel.Dispositionsdidnotmakeasignificant
predictioninperformanceonanonlinereadingcomprehensionassessmentinalessrestricted
space(β=.124,p=.137).Dispositionsofonlinereadingcomprehensionmadeasignificant
predictiontothemorerestrictedmodel(β=.291p=.006).
Theseresultsdonotalignwiththepredictionsmadebasedoncognitiveflexibilitytheory.It
waspredictedthatdispositionswouldbeasignificantpredictorinthelessrestrictedORCA.This
predictionwasbasedontheideathatstudentswho“haveopenmindsets”woulddobetterinthe
123
unfetteredenvironmentofthelessrestrictedinformationandthatstructuredsearchesofthemore
restrictedORCA,whilebenefiting,”wouldnotallowforserendipitouslearning.Howevertheinverse
wasfoundintheresultsofthisstudy.Scoresonthedispositionsofonlinereadingcomprehension
weresignificantintheORCA‐MRmodelbutnottheORCA‐LRmodel.
Theresultsalsodidnotalignwiththepredictionsmadebasedonatheoryofnewliteracies
ofonlinereadingcomprehension.Basedonthistheorythepredictionwasmadethatdispositions
wouldbeasignificantpredictorinbothmodels.Howeverdispositionswereonlyasignificant
predictorinthemorerestrictedmodel.
124
ChapterV
QualitativeResults
Inadditiontothetwoquantitativeresearchquestions(numbers1and2)thatwere
previouslyexploredinChapterIV,thisstudyalsoexploredtwoqualitativeresearchquestions
(numbers3and4)usingthinkalouddata:
3. What patterns of online reading comprehension strategies appear among high
and low performing online readers during an online reading comprehension task
within a less restricted information space?
4. What patterns of online reading comprehension strategies appear among high
and low performing online readers during an online reading comprehension task
within a more restricted information space?
Iutilizedatwo‐stageanalysis.Firstthedatawereexaminedforthemesandpatterns
betweenhighandlowperformersacrosstheoverallORCAassessments.Thentoanswerquestions
threeandfourthedatawereexaminedforthemesandpatternsbetweenhighandlowperformers:
first,withinalessrestrictedinformationspaceandthenwithinamorerestrictedinformation
space.Thus,thestageoneanalysisconsistedoflookingforpatternsofstrategyusethatledto
betteroverallperformancewhencompletingonlinereadingcomprehensionassessments.Thenin
stagetwothesepatternswereexaminedmorecloselytoseeifthereweredifferencesbasedonthe
natureoftheinformationspace.
Twelvestudents,fourfromeachparticipatingschool,wereselectedforthequalitative
portionofthestudy.Performanceonthefirstadministrationoftheonlinereadingcomprehension
assessmentwasusedtodetermineparticipantsinthethinkaloudactivity.Theywereselectedas
follows.
125
1. First all students in their class were ranked based on scores on the first administration
of the ORCA. Students were divided into two groups: top 10% of scores and the
bottom 10%
2. These groups were then reviewed by the teacher to allow for the selection of students
who would be comfortable working with an adult on a verbal protocol task.
3. Then four students from each participating school were randomly selected from the
list: two students who scored in the 10% of their class and two students who scored
in the bottom 10% of their class.
Then,forthequalitativeanalysis,thetwelveselectedstudentsweredividedintothree
achievement‐levelgroups(highaverage,low)basedonORCAscores,independentofschool.They
wererankedusingacombinedscoreofonelessrestrictedtaskandonemorerestrictedtask.The
studentswererankedfrom1(lowest)to12(highest)andbrokenintothreegroupsoffour.Thislist
resultedinamedianscoreof10.5,outof24.Themiddlegroupincludedfourstudentswithintwo
scoresofthemedian.Thelowgroupincludedscoresmorethantwoscoresbelowthemedian.The
highgroupincludedfourstudentswithscoresmorethantwoscoresawayfromthemedian.
Tenstudentswereincludedinthefinalanalysissincetwostudentshadtoberemovedfrom
thestudyduetoerrorsindatacollection.Thisincludedonestudentinthehighgroupandone
studentinthemiddlegroup.Table5.1liststhescoresofeachparticipant.
Thestudentsthencompletedtwoadditionalthink‐aloudsessionswiththeresearcher.One
taskinvolvedamorerestrictedinformationspace,wherestudentshadtofindspecificsources,and
theothertaskinvolvedalessrestrictedinformationspace,wherestudentscouldselectanysource.
Studentswerescoredusingadichotomousrubric.ThisrubricappearsinAppendixA.
126
Table5.1QualitativeParticipantsPseudonym Verbal PK ORCA1 ORCA2 Total SchoolHigh Isabella 13 1.5 8 9 17 3Olivia 17 2 9 10 19 3Ava 23 2 9 9 18 3 Medium Alexander 8 2 7 8 15 3Sophia 5 0.5 4 7 13 1Jacob 8 2 7 8 15 2Ethan 7 1.5 2 7 9 2 Low Emma 2 0 1 1 2 2 Michael 3 5 5 3 8 1Jaydan 7 .5 4 1 5 2Note:ORCA1andORCA2representsthetwoassessmentsadministeredtoselectstudentsforthe
thinkaloudtask.Itincludedbothamorerestrictedandalessrestrictedtask.
QualitativeAnalysis:ATwo‐StageApproach
Thischapterpresentstheresultsofthequalitativeanalysisorganizedintwoseparate
stages.Stageoneinvolvedlookingatthedataglobally.Thegoalofstageonewastoidentify
patternsofstrategyusethatleadtosuccessontheORCAregardlessofthenatureoftheinformation
space.
Firstfrequencytablesofrawcodeswereexaminedforpatterns.Thecodingscheme
consistedofobservablereadingstrategies.Thensuccessivepassesthroughthedatawere
conductedbyreexaminingthetimestampofcodinginstances.Iwouldreturntothevideoat
specifictimestampsandcomparecodinginstancesacrosscases.Thisallowedfordatareduction
andpatternstoemerge.Thedatawerethenreexaminedinordertogrouppatternsuntilthemes
emerged.Thisapproachleadtothemes,whichconsistedofpatterns,andeachpattern,consistedof
observedbehaviorsfromfrequenciesofcodes.
127
Thesecondstageofanalysisbuiltupontheresultsofthefirststage.Thegoalofthesecond
stagewastoexaminethethemesfromstageonetoanswerresearchquestionsthreeandfour.This
stageinvolvedmakingadditionalpassesthroughthedatatoseeifpatternsemergedthatwere
differentbasedonthenatureoftheinformationspace:morerestrictedorlessrestricted.Pattern
identificationinvolvedaniterativeprocessofreorganizingthedatafromstageoneandreworking
groupingssothatthepatternstructuresdefinedthemesinthedata.
Examiningpatternsinthesecondstageofdataanalysisrevealedadifferenceamong
strategyusebyhighandlowperformers.Itwasevident,byexaminingthedatathattherestricted
natureoftheinformationspaceonlyaffectedpatternsofstrategyuseonthelocatingtasks.
Thereforethesepatternswerethengroupedintothemessurroundingspecificprocessesinvolved
inlocatinginformationduringtheORCAtasks.
StageOneThemes
Thegoalofstageonewastoidentifypatternsofprocessesthatleadtooverallincreased
performanceontheORCAassessment.Recursive,analyticinductivemethods(Angrosino,&Mays
dePerez,2000;Bogdan&Biklen,2003)wereusedtomakeinitialpassesthroughcodedvideosof
studentthinkaloudstoidentifypatterns.Thepatternsthatemergedwereevidentacrossall
elementsofonlinereadingcomprehension:locating,evaluating,synthesizing,andcommunicating.
Thepatternsthatemergedwerealsoevidentinboththelessrestrictedandthemorerestricted
tasks.
Thesepatternswerefurtherdistilledassuccessivepassesweremadetoorganizethe
patternsintothemes.Intheend,fourglobalthemesemergedthatrepresentedcentraltrendsinthe
data.Thesethemesandtheirassociatedpatternsconsistedofthefollowing:
128
Themeone:OverallORCAscoresappearedtoberelatedtostrategictextassembly.
1. Studentswithgreaternavigationalskillswerebetteratreadingmultiple
sources.
2. Studentswhousedcomprehensionmonitoringstrategiesweremore
successfulontheORCAtasks.
3. Studentsmoresuccessfulatsynthesizingonlineinformationreturnedto
theirsourceswithgreaterfrequency.
Themetwo:OverallORCAscoresappearedtoindicatealackofsourceevaluationknowledgeamongparticipants.
1. Fewstudentsweresuccessfulatevaluatingauthorexpertise,evidence
usedbyanauthor,andauthor’spointsofview.
2. Studentsmadeerrorsofoversimplificationduringonlinereading
comprehensiontasks.
Themethree:StudentswhorecalldetailsfrommemoryappeartoperformbetterontheORCAtasks.
1. Students who recalled details from memory when combing multiple sources
may be better at synthesis tasks.
2. Students who recalled details from memory may be better at taking a
position.
Themefour:Studentsappearedunpreparedtoengageinthecommunicationstrategiesnecessaryforonlinereadingcomprehensioninacademicsettings.
1. Studentresponsesfailedtoprovideadequateinformation,especiallyevidence
fromwhattheyhaveread.
2. Studentsdidnotusetheaffordancesofonlinecommunicationspaces.
129
Themeone:OverallORCAscoresappearedtoberelatedtostrategictextassembly.
Acleardifferenceemergedbetweenstudentswithhighlevelsofonlinereading
comprehensionandtheirlessskilledpeersaroundanabilitytostrategicallyassembletextswhile
engaged.inInternetinquiry.Afteralltheconstructionoftextsinsituisanoveldifferenceofonline
readingcomprehension(Leuetal,2004).Thusthisthemewasdefinedastheabilitytoreadfor
meaningwhileflexiblyapplyingbothnavigationstrategiesandcomprehensionmonitoring
strategies.
Threepatternsinformedthistheme:(a)Studentswithgreaternavigationalskillswere
betteratreadingmultiplesources;(b)Studentswhousedcomprehensionmonitoringstrategies
weremoresuccessfulontheORCAtasks;(c)studentsmoresuccessfulattakingapositionwhile
synthesizingonlineinformationusedmorenavigationstrategies.
Patternone:Studentswithgreaternavigationalskillswerebetteratreadingmultiple
sources.Oneofthefirstelementsrequiredforstrategictextassemblyistheabilitytomanipulate
multiplewindowsandtabs.Participantswhoscoredinthehighestperforminggroupontheonline
readingcomprehensionassessmentweremorelikelytoutilizemultipletabsasatooltomove
betweenthetaskandtheirsources.Thisallowedthemtocreateusefulmultiplesourcetexts
throughsefl‐directedtextconstruction(Coiro&Dobler,2008).Inaddition,theyweremorelikely
toclickonadditionallinksattheinitialwebpagetheyfoundbasedonsearchresults.Thisallowed
participantstohaveawiderselectionofsourcestoselectduringtextassemblage.Thesepatterns
aredisplayedinTable5.2.Thispatternincludedtwostrategies:(a)utilizingtabstonavigate
betweentaskandsourceand(b)usingtabstonavigatebetweenmultiplesources.
130
Table 5.2 Frequency of Navigational Strategies ParticipantsgroupedbyORCAscores
Movesbetweentaskandsource
Navigatebetweenmultipletabsorwindows.
High Isabella 20 4Olivia 29 11Ava 14 6TOTAL 64 21 Medium Sophia 16 2Alexander 13 6Jacob 11 3Ethan 11 3TOTAL 51 14 Low Emma 21 2Jaydan 13 3Michael 15 1TOTAL 49 6 Utilizingtabstonavigatebetweentaskandsource.Akeyoveralldifferencebetweenhigh
andlowperformerswasthefrequencyinwhichstudentsmovedbetweenthetaskandthesource
orbetweenthetaskandmultiplesourcesbyusingtabs.Thestudentswhosuccessfullyusedtab
browsingseemedtorelyonthetooltousenavigationalstrategies.Thisstrategysupportedthe
strategictextassemblyofskilledonlinereaders.Successfulstudentsusedwindowsortabbrowsing
tonavigatebetweensourcesandtasksandbetweenmultipleopensources.Thesestrategies
supportedtheirnavigationasstudentsusedthetabstomovebacktothetasktoremembertheir
purposeinreadingasource.
131
Theadventoftabbedbrowsinghasalteredhowmanyreadonline.Theseparticipantswere
infantswhenMozillaintroducedtabbrowsingtoFirefoxin2000.Thusitwasnotasurprisetoseea
relianceontabsacrossallthreegroups.Stillitisclearthatstudentswhoutilizenavigational
strategiessuchastheeffectiveuseoftabbedbrowsingperformedbetter.Forexamplestudentsin
boththehighandmiddlegroupswhoweremoresuccessfulmovedbetweenthetaskandtheir
sourceswithgreaterfrequencyandease.
AvausedtabssuccessfullywhenshehadtoparaphrasetwosourcesontheCausestask.Ava
beganthesecondsynthesisitemontheCausestaskbyreadingtheitem.Shehadfourtabsopen:the
taskandherthreesources.AfterreadingtheitemAvathenclickedonthetabforthesecondsource
thatshefound.Avareadthewebsitefor12seconds,andthenclickedbackonthetabtothetask.
Shethenreadthetaskaskinghertocombinetheinformationsheread.Avanextclickedonthe
secondsourceshefound.Shescrolleddownandreadthewebsiteforafewseconds.Avathen
typed:
Therewasn’tjustoneeventthatcausedtheAmericanrevolution,thereweremany
eventssuchastheFrenchandindianwar,andwhenthebritishstartedtocontrol
howamericansboughtteaand…
IthenaskedherwhereshegotherinformationfromandAvaclickedonherthirdsourceand“said
thisone,andthisone”assheclickedonhersecondsource.Avathenmousedoverlinkstovarious
causesonhersecondsourceandthenclickedbacktoherthirdwebsiteshefound.Shereadthe
pageandthenclickedbackonhertaskandaddedthewords,“stampact.”Byclickingontheactivity
tabtorereadthetaskAvawasengagingincursorcontrol.Thishelpedherincompletingthe
synthesistasks.
Oliviawasaskillednavigatoroftabbedbrowsing.Shewouldconstantlyreferbacktothe
taskinordertoremindherselfofthegoal.Afterre‐readingthetaskOliviawouldmovebacktoher
sourceanddecideifitwasrelevant.
132
Michael,inthelowgroup,ontheotherhand,didnothaveastrongunderstandingoftab
browsing.Heoftenwouldaccidentlycloseouttabshewasusing.Twiceheclosedoutofthetasktab
andthetaskhadtoberestarted.IntheseinstancesIforwardedhimtohislastquestion.Michael
actuallyopenedanadditionalwindowtoconductasearch.Hethenaskedmeifhecould,“Exitoutof
thesetwo,”whilemousingoverthe“Closex”intheupperlefthandcorner.Iexplained,“Youmight
needthoselater,”andMichaelminimizedthewindowwiththetask.Laterheforgotheminimized
thetaskandhadtoberemindedbyme.
Emma,inthelowgroup,seemstohaveusedtabsquiteoften.Howeveranexaminationof
thevideodatarevealedthatEmmawasnotefficientinheruseoftabstoreadmultiplesources.She
wouldoftenclickthroughthetabs,losingherplace,forgettingwhatsourceshewastryingtoread.
Usingtabstonavigatebetweenmultiplesources.Anotherkeydifferencethatledto
higheroverallperformancewasnavigatingbetweensources.ForexampleIsabella,inthehigh
group,usedthetaskinterfaceasanote‐takingtoolassheread.Isabellawouldseamlesslymove
betweenuptofiveopentabsinherbrowser.Ontheturningpointtask,forexample,shehadfive
tabsopen.Shewouldmovethroughthesetabsandfinddetailsoneachpageduringthesynthesis
tasks.Whenaskedaboutmovingbetweentabs,Isabellasaid,“Itiseasiertofindinformationthis
way.”
Alexander,inthemiddlegroup,alsousedtabstoparaphrasetworelevantdetailsfromeach
sourcehefound.Whileworkingonthecausestask,Alexandersaid,“ItsaystotakenotessoIam
goingtolookatallthreeandtakenotesfromeachofthem.”Alexanderthenclickedonthetabfor
thefirstsourcehefound.Heclickedbackontothetaskandtypedintwodetails.Hethenclickedon
atabtothethirdsourcehefound.Alexspentafewsecondsreadingthesource.Hethentabbedback
tothetaskandaddedathirddetail.
Michael,ontheTurningPointtask,wasnotassuccessfulatsynthesizinginformationfrom
twosources.InfactMichaelneverclickedonthetabstomovebetweensources.Onthesecond
133
synthesisitemMichaelquicklyreadthetaskandclickedonthetabforthewebpageaboutthe
TurningPointoftheRevolutiononsocialstudeisforkids.com.Hethenbegantoparaphrasethe
detailsfromthesource.Besidesafewphrases,Michael’sresponseisalmostverbatimfromthe
website.Inordertogetthewording,names,anddatescorrectMichaeltoggledbackandforth
betweenhissourceandtaskatotalofsixtimes.Michaelneverclickedonthesecondsource.
Patterntwo:Studentswhousedcomprehensionmonitoringstrategiesweremore
successfulontheORCAtasks.AnotherkeydifferenceinoverallsuccessontheORCAwastheuse
ofcomprehensionmonitoringstrategiesbyskimmingwebsitesorreadingforsustainedperiods.
Comprehensionmonitoringwascentraltostrategictextassembly.Thispatternisdisplayedin
Table5.3.Twostrategieswereevidentinthispattern:(a)usingastrategyofskimmingwebsitesto
identifykeydetails;(b)andengagedreadingwithinawebsite.
Table5.3FrequencyofComprehensionMonitoringStrategiesUsedtoReadWebsitesParticipantsgroupedbyORCAscores
Skimswebsite
Engagedreadingofrelevantwebsite
Summarizesawebsite
Usesinternalsearchengine
High Isabella 9 1 0 0Olivia 15 4 1 0Ava 10 3 0 0TOTAL 34 9 1 0 Medium Sophia 13 5 0 0Alexander 15 0 1 0Jacob 6 5 0 1Ethan 10 0 0 1TOTAL 44 10 1 2 Low Emma 2 2 0 0Jaydan 8 3 0 0Michael 4 1 0 0TOTAL 14 6 0 0
134
Usingastrategyofskimmingwebsitestoidentifykeydetails.Skimmingandscanning
wasdefinedasquicklyscrollingupanddownawebsite.Overallstudentswhoweremoresuccessful
ontasksskimmedmoresources,lookingforrelevantdetailstointegrateintheirresponses.This
wasevidentinboththehighandmiddlegroups.Studentsinthelowestperforminggroupdidnot
skimwithgreatfrequency.
Participantswhoperformedwellskimmedandscannedinformationmorewithinawebsite.
Inessenceeffectiveonlinereadersquicklyskimmedwebsites,checkedthesourceagainsttheir
purposeandmovedonfromirrelevantsources..Olivia,forexample,wasveryadeptatquickly
readingawebsiteanddisregardingirrelevantinformation.Shewouldoftenskimandscana
websitebeforeshereadthesource.Whenaskedshesaid,“Iliketoquicklylooktoseeifithasany
linksorinformationIcanuse.”
Studentsinthemiddlegroupalsousedastrategyofskimmingwebsites.Themean
frequencyforthemiddlegroup(10.4),however,wasnotquiteashighasthatforthetoptiergroup
(11.3).WhenSophiawasworkingontheTurningPointtasksheclickedonthesite
SocialStudiesforkids.com.Hercursorthenscrolledovertheheadingsandthefirstfewsentencesof
thearticle.Whenaskedwhatshewasdoing,Sophiasaid,“Ithadinformationaboutthe
revolutionarywar.Iamgoingtoreadit.”
Studentswhowerelesssuccessfulalsoskimmedandscannedsources.However,theyused
thestrategywithlessfrequencythantheirmoreaccomplishedpeers.Ethanforexamplelandedon
aWikipediaarticleduringtheTurningPointtask.Hequicklyscrolleddowntothebottomandback
up.Hethenrecordedhisanswerascorrectwithoutactuallyreadingthewebsite.
Jaydandidnotalwaysscanhiswebsitesforrelevantinformation.DuringtheDelawaretask
JaydanlandedontheSocialStudiesForKidswebsite.Hescrolledupanddownandsaid,“Ithasalot
moreinformationaboutit.Soitmighthavemorefacts.”AfterskimmingthepageJaydanwentonto
135
readthesite.YetonthesametaskhelandedonaWikipediaarticle,selectedtheentiretextsand
copiedthatashisresponsetoasynthesisprompt.
Engagedreadingofwebsites.Engagedreadingofsourceswasdefinedasaperiodof
readingbeyondtensecondsthatwasaccompaniedwitheithercursormovementorreadingaloud.
Basicallystudentswhoreliedonstrategictextassemblyhadtheconditionalknowledgeofwhento
slowdownaswellasspeedup.Inadditiontoskimmingandscanning,studentswhoweremore
successfulontheORCAtaskspentmoretimereadingatselectedlocations.
Olivia,amemberofthehighestscoringgroup,heavilyinvestedtimeinthereadingof
sources.OntheWomentaskshelandedonawebpage,AmericanAthenasonthewebsite
AmericanRevolution.org.Oliviaspentoverfiveminutesreadingthesource.Shescrolledthroughthe
sourcemultipletimes.Olivia’sapproachtotheCausestaskwasverysimilar.Sheoftenspent
minutesonasource,wherehercounterpartswouldoftenreadasourceforlessthanaminute.Once
Oliviafinishedreadingasourceshewouldprovideaverbalsummary.
Studentsinthemiddlegroupalsousedastrategyofengagedreading.Theywerealsomore
likelytoreadasourceoutloudwhencomparedtothehighgroup.Thisstrategywasmoreprevalent
withSophiawhospentmoretimeonsourcesthananystudentbesidesOlivia.WhenSophialanded
onthewebpagetitled,“What’sWrongwiththisPainting”duringtheDelawaretasksheexplained
thatshewaslookingforinformationabouttheaccuracyofthepainting.Sophiathenproceededto
readthewebsiteoutloud.Sophiachosetoreadallofherwebsitesverbally.
Alexander,alsoamemberofthemiddlegroup,spentsubstantialtimereadingasource.
WhenAlexanderlandedonasourceandchoseitasrelevanthewouldreadthesourcetohimself.
Alexander,likeOlivia,wouldthenprovideaverbalsummary.Forexample,ontheDelawaretaskhe
wasreadingasite.OncehefinishedAlexanderstoppedandsaid,“Ireadalittlebitofthisandit
says,‘itcontainsanoftendiscussedhistoricalinaccuracy.”
136
Studentsinthelowestperforminggroupalsousedastrategyofengagedreadingon
websites.Howeverthisstrategywasemployedwithlessfrequencythantheirmoresuccessful
peers.Onceagain,likestudentsinthemiddlegroup,thisstrategywasnotedmostoftenwhen
studentswouldreadasourceoutloud.Alsothetimespentonsourcesbythesestudentswasmuch
lessthanthetimespentbystudentsinhigherachievementgroupssuchasOliviaandSophia.
Jaydan,forexample,ontheDelawaretask,foundhimselfonapageonthe
socialstudiesforkids.comwebsite.Hestartedbyreadingthefirstparagraphoutloud.Jaydan,
however,didnotreadanydetailsthatweresignificanttothetask.
Emmadisplayedarudimentaryuseofaengagedreadingonawebsitetoidentifykeyideas.
Shewouldstoptoreadasource,butEmmaoftenmovedquicklyfromthesourcebacktothetask
withoutreadingtheentirepieceandEmmamissedkeydetailsthatwouldanswerthequestion
posedinthetask.ForexampleontheWomenintherevolutiontaskEmmaspentonlyafewsecond
witheachsource.
Patternthree:Studentsmoresuccessfulatsynthesizingonlineinformationreturned
totheirsourceswithgreaterfrequency.Returningtosourceswasanotherkeycomponentof
strategictextassembly.Thisrequiredbothnavigationalskillsandcomprehensionmonitoring.
Thispatterninvolvedtwostrategies:navigatingtoasourcetoreadwhiletakingapositionand
navigatingtothesourcetocopyandpastedetails.Thefrequenciesofthesestrategiesaredisplayed
inTable5.4.
137
Table5.4NavigationStrategiesUsedWhenTakingaPositionParticipants grouped by ORCA scores
Student returns to the source
Student copies and past details in reference to claim.
High
Isabella 1 2Olivia 19 0Ava 1 0TOTAL 21 2
Medium
Sophia 3 1Alexander 4 0Jacob 0 0Ethan 0 0TOTAL 7 1
Low
Emma 1 1Jaydan 0 0Michael 0 0TOTAL 1 1
Navigatingtoasourcetoreadwhiletakingaposition.Participantswhoreferredbackto
thesourcestheyfoundduringthefinalsynthesisitemenhancedtheirperformanceontheORCA
tasks.Thisstrategywasdefinedasclickingonasourceforanygiventimeduringthetakeaposition
items.Itshouldbenotedthateverystudentinthehighestperforminggroupreturnedtothesource
aminimumofonetime.Twooutofthreestudentsinthemiddlegroupusedthisstrategy,andonly
onestudentinthelowestgroupusedthisstrategy.
Studentsinthehighestperforminggroupusedastrategyofreturningtothesource.Olivia
continuedherpatternoflongsustainedreadingonthefinalsynthesistask.Shetookatotaloffour
138
minutestocompletetheitem,thelongestofanyoftheparticipants.Oliviabeganbyreadingthe
item,andtyping,“Thewomen.”IaskedhertoexplainherthinkingandOliviathenclickedontothe
lastsourceshefoundandsaid,“therolethattheyplayed,umtheyfoughtfortheircountrytoo,but
bydoingotherthingslikestickingupfortheirhomes.”Oliviathenclickedbackonthetaskand
typed:
Thewomenfoughtadifferentpartofthewarathome,theyprotectedtheirhome
frominvaders,theyevendecidedtostopdoingsomethingsinordertosavetheir
countrythatwasatbattle.
Oliviathenmovedontothequestionaskinghertosupportherposition.Shereadsilently,ona
singlesource,forapproximatelyoneandahalfminutes.Shethenclickedbackonthetask,and
typed,“Theyendedupsigningadeclarationtostopusingteatohelpthewar,alsotostopthe
Englishimportsthatwerecoming.”
Onlytwostudentsinthemiddlegroupsreturnedtothesource.Alexander,forexample
navigatedtohissourceduringtheCausestask.Hebeganthetakeapositionitembywriting,“Ithink
thattheywerethatAmericanpeoplewereveryindependent...”Alexanderthenclickedonthe
AmericanPersuaderswebsiteandscrolledthroughthepage.Heclickedbackontheaskandadded,
“andtheBostonTeaparty,BostonMassacre,andtheStampAct.Alexanderthenmovedontothe
secondquestion.Hetyped,“Ifoundmyfirstexampleinwordsfrom.”Alexanderthenclickedonthe
tabtohissecondsource,socialstudiesforkids.com.HecopiedandpastedtheURL.Hethenrepeated
thisprocessforthethirdwebsitehefound.
Studentsinthelowestgroupdidnotreturntothesourcewhiletakingaposition.Only
Emmadidonce.JaydanorMichaeldidnotusethisstrategy.
Navigatingtothesourcetocopyandpastedetails.Studentsalsoreliedoncopyingand
pastinginthefinalsynthesistaskasastrategy.Thisstrategywasdefinedascopyingandpastinga
139
detail,andnottheURL,fromthewebsite.Theuseofthisstrategywasnotakeyindicatorofoverall
ORCAperformance.Onestudent,ineachofthegroups,usedthisstrategy.
IsabellacopiedandpasteddetailsfromasourceontheTurningPointtask.Shebeganthe
taskbyansweringthefirstquestionwith,“WhenthebattleswouldhappentheAmericanswere
takingoverandwinningmorethantheBrisith”.Shethenmovedonthesecondquestioninthetask.
Sheclickedononeofhersourcesandcopiedandpasted,“AmericanVictoryatSaratoga.”Isabella
thenadded“…becausethatisoneoftheBattlestheywon.”Shethenclickedonanothersourceand
copiedandpasted“TheBattleofBennington”Isabellathenfinishedthesentencewith“…another
battletheywon.”
Sophiaalsoreliedonacopyandpastestrategy.Whenshestartedthethirdsynthesisitem
ontheTurningPointtaskSophiaimmediatelytyped,“WhentheAmericansandFrenchhad
surroundedtheBritish.”Shethenquicklyclickedononeofhersourcesandcopyandpastedforthe
secondhalfoftheitem.
Emmawastheonlystudentinthelowestperforminggrouptouseastrategyofcopyand
paste.Shedidnotemploythestrategywell.Forexample,onthecausestaskshebeganbyreading
thetask.Emmathenclickedonthetabtooneofhersources.Emmathenproceededtoreadfortwo
minutes.Shethencopiedandpastedalinefromthetaskforherposition.Shethenclickedbackto
hersource,andimmediatelymovedbacktothetaskandtyped,“IDK,”afterthesentenceshecopied.
Themetwo:OverallORCAscoresappearedtoindicatealackofsourceevaluationknowledge
amongparticipants.
Theresultsfromthequalitativeportionofthisstudysupporttheconclusionthatsource
evaluationmaybethemostcriticalcomponenttoonlinereadingcomprehension.Sourcing
strategiesfellalongthetwocontinuumsofrelevancyjudgmentsandcredibilityjudgments.Stage
oneanalysisrevealedthatstudentswhousedmorestrategiestomakecredibilityjudgmentsonthe
140
ORCAtasksoutperformedtheirpeersregardlessofthenatureofthetask.Strategiesforrelevancy
judgmentswereaffectedbythetaskandwereanalyzedinstagetwo.
Twopatternsinformedthistheme:(a)fewstudentsweresuccessfulatevaluatingauthor
expertise,evidenceusedbyanauthor,orauthor’spointsofview;(b)studentsmadeerrorsof
oversimplificationwhenevaluatingonlinesources.
Patternone:Fewstudentsweresuccessfulatevaluatingauthorexpertise,evidence
usedbyanauthor,orauthor’spointsofview.Thejudgingofwebsitecredibilityinvolved
complexsourcingstrategies.Thepatternthatemergedamongtheparticipantswasagenerallackof
effectivestrategiestoevaluateauthorexpertise,evidenceusedbyanauthor,orauthor’spointof
view.
Strategiesusedtoevaluateauthorexpertise.Evaluatingauthorexpertisewasatwo‐step
process.Thefirststepwasidentifyinganauthor.Whilereadingbooksstudentsknowtolookonthe
frontcover,butincomplexonlineenvironmentsthestudentsstrugglewithcorrectlyidentifyingthe
author.ThisisevidentinTable5.5.Inthisstudystudentsusedastrategyofidentifyingtheauthor
inthetext,usedastrategytoidentifytheauthoronanaboutuspage,orhadaninabilitytofindthe
author.
141
Table5.5FrequencyofStrategiesUsedtoIdentifyanAuthorParticipants grouped by ORCA scores
Finds the author in the text
Finds the author on an about us/bio page
Infers the author from the text.
Does a secondary search for the author.
Does not locate the author
High Isabella 1 0 0 0 1Olivia 2 0 0 0 0Ava 2 3 0 0 0TOTAL 6 3 0 0 0 Medium Sophia 2 1 0 0 0Alexander 1 0 0 0 0Jacob 1 1 0 0 0Ethan 1 0 0 0 1TOTAL 7 2 0 0 1 Low Emma 2 0 0 0 2Jaydan 4 1 0 0 0Michael 1 0 0 0 1TOTAL 5 1 0 0 5
Usingastrategyofidentifyingtheauthorinthetext.Themostfrequentstrategydisplayedby
thestudentswastoidentifytheauthorinatext.Thisstrategyinvolvedlookingforinformation
underanarticletitle,theheaderorthefooterofthepage.Allofthestudentsinthehighperforming
groupwereabletoidentifytheauthoronatleastoneoftheirtasks.Therewasalsonodifference
betweenthehigh,medium,andlowgroupsintheuseofastrategytoidentifytheauthorinthetext.
Theuseofthisstrategywasusuallypredicatedmorebythesourcethanbytheabilityofthe
student.Inotherwordsiftheauthorwasreadilyavailableonthepagethestudentswouldnotapply
morecomplexstrategiessuchaslookingforanaboutuspage.Forexamplemanyofthestudents
chosetoevaluatethewebsiteAmericanRevolutionPersuaders.Thewebsiteclearlyliststheauthor
asJeremyJonesinthetitle.Yetifyougotothewebsite’shomepagebyremovingthefileextension
142
intheURLyouarebroughttoabusiness.TheAmericanRevolutionPersuaderswasachild’sessay
hostedonaparent’swebsite.
Usingastrategyofidentifyingtheauthoronanaboutuspage.Acrossthetasksonlyfive
studentsclickedonlinksthatwouldprovideadditionalinformationabouttheauthor.Thisstrategy,
likemostofthecriticalevaluationstrategieswasnotparticulartoanyspecificgroupof
participants.
Avatriedtoutilizethestrategyonbothofhertasks.Shewasnot,however,always
successful.ForexampleontheDelawaretaskAvachosetoreviewthearticle“What’swrongwith
thispainting?”hostedontheDelawareCrossingParkwebsite.WhenlookingfortheauthorAvafirst
clickedonthehomelink.Shethenclickedonalinktositeusage,whichhaddetailsaboutrenting
parkspace.Avathenclickedonthecontactuslinkfollowedbythe“aboutus”link.Once,onthe
“aboutus”pageAvascrolledtothebottomandcopiedinformationaboutthePennsylvania
Governor.Sheenteredthisinformationastheauthor.
SophiaalsotriedtoidentifytheauthorontheDelawareCrossingParkwebsite.Sophia
scrolledupanddownthewebpage,“What’swrongwiththispainting.”Sheexplainedthatshewas
tryingtofindtheauthor.NextSophiaclickedonthe“aboutus”link.Whenshecouldnotfindthe
authorSophialeftthepageandchoseasourcethatlistedtheauthorunderthetitle.
Jacobwasabletousethestrategyoffindinginformationonasecondarylink.Duringthe
CausestaskJacobwasonthewebsiteamericanhistorycentral.com.Whilehelookedfortheauthor
hescrolledtothebottomofthepage,andreadthefooteraloud,“Multieductator,Inc.”“Ohwait,”he
addedasheclickedona“contactus”link.Thisopenedupanemailprogram.Jacobclosedthe
programandclickedonthe“aboutus”link.Hereadthepageoutloudandsaid,“Thereitis.”Jacob
thenentered“multieducatorinc”ashisauthor.
Inabilitytoapplyastrategytofindanauthor.Sixoftheparticipantswereunabletoidentify
theauthorinatleastoneoftheirtasks.Ethancouldnotlocatetheauthoronhistaskaboutthe
143
historicalaccuraciesofGottlieb’spainting.EthanchosetoidentifytheauthorofaWikipediaarticle
hefound.Ethanbeganbyclickingbetweentwowebsites:theWikipediaarticleandanothersource.
Ashewasmovingthroughthepages,scrollingupanddown,theresearcherasked,“Whatareyou
thinking?”Ethan,replies,“Icannotfindtheauthor,theauthortooneofthewebsites.”Hethen
typed“IDK”intheboxfortheauthor.Itmustbenotedhedidnotclickanylinksoneithersite.
Ethanjustscrolledupanddownthepages.
Michaelwhenlookingforanauthorforthe“womenintherevolution”taskquicklytabbed
throughallofhissources.HesettledonawebsitefortheStonyFieldBattlefieldStateHistoricPark.
Thesiteclearlylistedtheauthoratthetopofthewebpageas,GillianCourtney,ParkRanger.
MichaelincorrectlylistedtheauthorsasGillianCourtneyandParkRanger.
Highperformingstudentsdidnotalwaysidentifytheauthor.Oliviaalsostruggledononeof
hertasks.Shereadthetaskandthenclickedonthetabtooneofhersources,historycentral.com.
Shethenscrolledupanddownthepage.TheresearcheraskedOlivia,“Whatareyoulookingfor?”
Sheresponded,“Informationabouttheauthor.”Oliviathenpausedonthecitationlistedatthe
bottomofthepage.Shethenclickedbackovertothetaskandtypedinthename,“Pheobe
Hanafore.”Thiswasnottheauthorofthewebsitebutwasasourcecitedinthewebsite.
IsabellaalsocouldnotidentifytheauthorontheTurningPointtask.Isabellawaslookingfor
theauthorofthewebsiteAmericanRevolution.org.Shespentoneminuteandvisitedthreeseparate
pagesonthewebsitelookingfortheauthor.Isabellaclickedonheroriginalsourceaboutabattlein
therevolution.Shethenclickedonmoreinformationaboutthebattleandfinallythewebsite’s
homepage.Aftershecouldnotlocatetheauthororpublisherofthewebsitesherecordedthe
answer,“websitemakers”astheauthor.
Thesecondstepinevaluatingauthorexpertiserequiredreaderstojudgetheauthor
expertiseusingcommonmarkersofauthoritysuchasoccupation,institutionalaffiliation,or
education.Overalltheparticipantsstruggledtoevaluatetheexpertiseoftheauthor.Thepatternis
144
displayedinTable5.6.Onlythreestudentscorrectlyrespondedtoatleastonepromptabout
expertise.Onlyonestudent,Ava,earnedbothscorepointsforevaluatingauthorexpertise.Thetwo
mostdominantstrategiesusedwerejudgingexpertisebyusingthecontentofthewebsiteorby
focusingonspecificdetails.Studentswhosuccessfullyjudgedauthorexpertisereliedoneffective
markersofexpertise.
Table5.6FrequencyofStrategiesUsedtoJudgeAuthorExpertiseParticipants grouped by ORCA scores
Uses an authoritative title to judge the author
Uses supporting details from content of the website.
Uses institutional information to judge author expertise
Uses background knowledge to judge author expertise
Uses completeness of information to judge author expertise
High Isabella 0 0 2 1 1Olivia 0 1 0 0 3Ava 1 0 2 0 0TOTAL 1 1 2 1 4 Medium Sophia 0 0 0 0 1Alexander 0 1 0 0 2Jacob 0 1 0 0 0Ethan 0 1 0 0 0TOTAL 0 3 0 0 3 Low Emma 0 2 0 0 0Jaydan 2 0 1 0 2Michael 0 1 0 0 1TOTAL 0 2 0 0 3
Usingeffectivemarkersofexpertise.Studentswhocorrectlyevaluatedauthorexpertiserelied
ontwomajorstrategies:institutionalaffiliationandauthor’soccupation.Ava,whocouldnot
identifytheauthorsofawebsitepublishedbytheStateofPennsylvania(shelistedthegovernoras
author)notedthattheauthorswereownersofahistoricalparksotheymustbeexperts.Onher
nexttask,theTurningPointoftheRevolution,Avafocusedontheauthor’soccupation.Shenoted
145
thatDanWhite,creatorofsocialstudiesforkids.comisan“educationalprofessionalwithapassion
forsocialstudies.”JaydanalsonotedthatasasocialstudiesteacherDanWhiteisanexpert.
Sophiausedthepublisheraffiliation,backgroundknowledge,andcontentofsecondary
sourcestoevaluateauthorexpertise.Shewrote:
Ithinkheisbecauseitisa.organdthoseareprettyhelpful.AlsoofwhatIknow
thisallsoundsright.Anotherreasonisthattheyhavealotofinformationthatthe
otherwebsitesdidn't.
SophiafirstfocusedinontheURLextension.Whilenotaguaranteeofexpertiseitdiddemonstrate
ahigherlevelofsourceknowledge.Shethencheckedthedetailsusedbytheauthor.Finallyshe
mentionedthatthewebsitehadinformationfromotherwebsites.Itmustbenotedthatatnotime
didSophiaactuallyevaluatetheauthor.She,likeherpeers,putthemostvalueinthecontentof
websites.
Strategiesusedwhenevaluatingevidenceusedbytheauthor.Anotherimportant
strategy,whenreadingmultiplesourcesonline,isevaluatingthecredibilityofasourcebyjudging
theevidencecitedbytheauthor(Goldman,2010).Thisisalsoanimportantstrategyforthe
disciplineofsocialstudiesasstudentsreadonlinetexts(Manderino,2011).Thisstrategywas
definedasjudgingthecredibilityofthesourcesofclaimsorevidenceusedbyanauthorofa
website.
Overalltherewerenomajordifferencebetweenstudentswhoweremoresuccessfulonthe
ORCAtasksandstudentswhowerelesssuccessfulontheORCAtask.Onlyonestudentsexhibiteda
strategythatevaluatedtheevidenceinawebsite.Exceptforthisstudent,alloftheparticipantsdid
notexhibitstrategiestoevaluatethesourceofevidencewithinawebsite.Thereforetheonly
patternevidentinthedatawasatotallackofstrategyusetojudgetheevidenceusedbyanauthor.
Examiningtheonecaseofsuccessfulstrategyusetojudgeevidenceusedbyanauthor.Sophia
wassuccessfulinevaluatingevidenceusedbytheauthor.Sophia’sstrategywastocheckthe
146
evidenceagainstasecondarysource.Sheresponded,“Becauses|hegivefacts.Andtheother
websitessaythesamething.soHemusthavehisFactsCorrect.”
Strategiesusedwhenevaluatingauthor’spointofview.Participantsinthestudy
displayedthefeweststrategieswhentheywereaskedtoevaluatehowanauthor’spointofview
mayhaveshapedthewordsorimagestheauthorusedonawebsite.Infact,sevenofthestudents
answeredthequestionwithavariationof“Idonotknow.”Furthermoreonlytwosuccessfully
judgedtheauthor’spointofviewforeitherthelessrestrictedorthemorerestrictedtask.Therefore
itisimpossibletodrawadistinctiononstrategyuse.Onceagaintheonlypatternthatemergedin
thedatawasstudents’inabilitytojudgeanauthor’spointofview.
Themostcommonphenomenonnotedwasalackofunderstandingoraninabilitytoanswer
thequestion.Emma,Ava,andMichaelallput“IDK”or“Idonotknow”astheiranswer.Emmabegan
byreadingthequestion.Shethentabbedbackovertohersource,andspentafewsecondsonthe
page.Emmathenclickedonthetabforthetaskandtyped,“IDK.”Emmaalsoput“IDKforher
response.Shesimplyreadthetaskandneverreferredbacktohersourcesformorethanafew
seconds.Infact,onherothertaskEmmacompletedallofthecriticalevaluationquestionswithout
evertabbingbacktohersources.Michaelfinishedthefirsttwoquestionsandpaused.The
researcherasked,“Doyouknowwhatthisquestionisasking?”Michaelresponds,“Ihavenoidea.”
Hethenproceededtotype“IDK.”
Usingauthor’spurposetojudgeauthor’spointofview.Thenextgroupofstudentsreliedon
theauthor’spurposetoinferhisorherpointofview.Whiletheydidnotspecificallyinferhowan
author’pointofviewaffectsherversionofthetruththestudentshadarudimentaryunderstanding
thateverytexthasagoal.Ethan,ontheDelawaretask,forexample,wrote,“toinformreadersthat
GeorgeWashingtondidreallycrosstheDelaware.”ForEthanpointofviewwasconfirmedwitha
genrebasedideaofpurpose.Hereadaninformationalwebsitethereforetheauthor’spointofview
wastoinform.
147
Isabellaalsousedauthor’spurposetodefinetheauthor’spointofview.OntheDelaware
tasksheexclaimedthat,“Ittellsyouthedetailsofthepainting.”OntheTurningPointtaskIsabella
wasevaluatingthewebsiteAmericanRevolution.organdwrote,“Yes,heshowsmanypicturesand
heisnotdefendingonesideheistellingitlikeitis.”IneachcaseIsabellafocusedonauthor’s
purpose.
Examiningthetwocasesofsuccessfulstrategyusetoevaluatingpointofview.OnlyOliviaand
Sophiacorrectlyinferredtheinfluenceofanauthor’spointofview.Sophiacorrectlyinferredthe
author’spointofviewononetask,andOliviaonbothofherassignedtasks.Sophiawasevaluating
aclassroomwebsiteSedivy.tripod.comfortheTurningPointtask.Shewrote,“Theauthorithinkis
tryingtogiveusinformationonthetopic,andshowinguswhathisclasscandoandLearn.”Sophia
focusedontheteacher’sdesiretohighlightwhathisclasswasdoingandhowthisinfluencedthe
designofthewebsite.
Oliviaalsocorrectlyinferredtheauthor’spointofview.Olivia,whenevaluatingawebsite
fortheWomentaskwrote:
Phebetakesthepointofviewshetookbecauseshe’sstickingupforthewomento
showtheycandowaymorethanpeoplethinktheycando.
Oliviaidentifiedtheauthorunderlyingpointofviewinadvancingtherolewomenhaveplayedin
history.Onhernexttask,Oliviadiscussedtheauthor'spointofviewbyidentifyingrevulsionfor
war:
Theauthor'spointofviewdoesinfluencethewordsandimagesusedonthewebsite
becausehispointofviewseemstogoalongthesamepathashiswordsandimages
becauseyoucantellhe'sagainstwhatthe3majoreventsdidtotheUnitedStatesof
Americaandothercountriesinvolvedinthewarbyhiswordchoiceandbythelast
sentenceortwowhenhesaysthathehopesthattheworldwillneverseeanother
AmericanRevolutioneveragain.
148
Patterntwo:Studentsmadeerrorsofoversimplificationwhenevaluatingonline
sources.Inbothlessrestrictedandmorerestrictedtasksandregardlessofabilitylevel,students
madeerrorsofoversimplification(Spiro,2004).Anoversimplificationisamisconceptionlearners
havebasedonpreviousintroductorylearning.(Spiro&Feltovitch,1996).Intermsofonlinereading
comprehension,errorsofoversimplificationoccurwhenusersviewwebsitesasregular,well‐
definedstructures.Instancesofoversimplificationoccurredwhen:(a)studentsoverlyreliedonthe
amountofcontentatwebsites,and(b)studentsoverlyreliedonwebsitefeatures.
Studentsmadeerrorsofoversimplificationbyoverlyrelyingontheamountofcontent
ofwebsites.Thishappenedwhenstudentsusedtheamountinformationatawebsitetoevaluate
authorexpertiseandwhentheyusedtheamountofinformationatawebsitetoevaluatethe
evidenceusedbyanauthor.Themostcommon,yetunsuccessful,strategyforevaluatingexpertise
wasusingthecompletenessofinformationtojudgeanauthor.Thisstrategywasnoted11times
acrossthestudy.Studentsinallgroupsreliedonthisineffectivestrategy(seeTable5.8).
Alexanderreliedontheamountofcontentofthewebsitetojudgeauthorexpertise.Onthe
CausestaskhewasreviewingthewebsiteAmericanRevolutionPersuadersandresponded,“Idon
thinktheauthorisanexpertbutI’msureheknowalotaboutitifheknewthreecausesandcould
supportthem.”
Sophiaalsousedastrategyofrelyingontheamountofcontenttojudgeauthorexpertise.
OntheTurningPointtask,forexample,sheresponded:
No,becausehedidhavesomefactsbutnotallnortheMainfactsontheAmerican
Revolution.Itwasverylittlewritinganddidn'tsayanythingabouttheTurning
Points.
Sophiawasevaluatingtheauthorexpertiseusingtheamountofinformationandnotcommon
markerssuchaseducation,institutionalaffiliation,orexperience.
149
Michaelalsoreliedonthecompletenessofthecontenttojudgeauthorexpertise.Hewas
evaluatingawebsitebyaparkrangerforthewomenoftherevolutiontask.Michaelwroteinhis
response,“Theauthorisbecausehewrotealotanditseamshedidalotofresearch.”Michael’s
responsealsoreflectedhisactions,ashewouldoftenchoosethewebsiteswiththegreatestamount
ofinformation.
Studentsalsomadeerrorsofoversimplificationbyoverlyrelyingontheamountofcontent
atwebsiteswhenjudgingevidenceusedbyanauthor.Onceagainstudentsreliedmoreonthe
quantityandnotthequalityofinformationinthesources.Isabella,forexample,ontheTurning
Pointtask,whenevaluatingawebpageontheAmericanrevolution.orgsitecommentedthatthe
evidenceusedinthewebsitewascrediblebecause,“Hetalksabouteverypartofthebattle.For
exampleduringitbeforeitandafterit.”
Michaelwasalsoillustrativeofstudentswhorelyonthequantityofinformation.Michael
responded:
Yesandno,theauthorusesconvincingevidencebecauseitlookslikeheknowswhat
heistalkingaboutandhedidnotwritealothecouldhavewroteabouttheteaparty
ortheBostonmassacre.
Hewasnotevaluatingtheevidenceintermsofitssource,butinsteadintermsofitcompleteness.
Studentsmadeerrorsofoversimplificationbyrelyingonwebsitefeatures.Thesecond
mostcommonstrategyused,alsounsuccessfully,bytheparticipantstojudgeauthorexpertise
focusedoneitherspecificdetailsortheauthor’sorganizationofthewebsite.Ethan,ontheTurning
Pointtaskforexample,saidtheauthorDavidWhitewasanexpertbecause,“Heusedspecificdetails
andgreatpunctuation.”
Emmanoted,onthecausestask,thattheauthorJeremyJoneswasnotanexpertbecause,
“thereisnopersuasivelanguageused.”Oliviafocusedontheamountoftheinformationandalso
specificdetails:
150
Yes,becausePhebeputalotofinformationintheparagraphsabouteachwoman
whoservedintheAmericanRevolutionalsosheusesspecificdatesofwhen
somethingmajorhappened.
Jaydanalsousedtheinformationtojudgethecredibilityofsources.Hedidnothowever
focusontheoverallquantitybutlikemanyofhispeers,onspecificdetailsinthewebsite.When
evaluatingtheevidenceusedonthewebsiteSocialstudiesforkidsJaydanreplied,“Yesheuses
convincingevidence.HestatedthatthereisnotjustoneTurningPointthattherewasmany.”Then
whenevaluatingasiteforthecrossingtheDelawaretaskJaydanwrote,“Hesaysthatitwasacold
night.Alsohowthecrossingwasasneakattack.”Ineachcaseheneverinvestigatedthesourceof
theevidence.
Otherstudentsunsuccessfullyusedawebsite’sorganizationorfeaturestojudgethe
evidenceusedbyanauthor.EthancommentedthattheauthorofSocialstudiesforkidsusedcredible
evidencebecause,“heusedgreatfactsandeasytofollowwordsandsentences.”Alexanderalso
focusedontheformatofthewebsitebywriting,“Ithinkhedoesuseconvincingevidencebecause
hewritesafullparagraphbackingupwhathethinksthecausesare.”
Themethree:Studentswhorecalldetailsfrommemoryappeartoperformbetteronthe
ORCAtasks.
Thedatafromthisthinkaloudstudyindicatedthatacleardifferencebetweenstudents
whoperformwellandstudentswhodidnotperformwellwastheabilitytorecallinformation
theyhadjustread.Twopatternsinformedthistheme:(a)studentswhorecalleddetailsfrom
memorywhencombingmultiplesourcesmaybebetteratsynthesistasks;and(b)studentswho
recalleddetailsfrommemorymaybebetterattakingaposition.Thesepatternsareexploredin
Table5.7.Whilethesestrategieswereusedbyeverygroupofparticipantsstudentsinthehigher
groupsrecalledmoredetailswithgreateraccuracythantheirpeers.
151
Table 5.7 Frequency of Using a Strategy of Recalling Details From Memory Participants grouped by ORCA scores
Recalls details from memory when combing multiple sources
Recalls details from memory when taking a position
High Isabella 1 2 Olivia 1 2 Ava 2 14 TOTAL 4 18 Medium Sophia 0 2 Alexander 0 3 Jacob 1 4 Ethan 0 3 TOTAL 1 12 Low Emma 0 2 Jaydan 0 3 Michael 0 2 TOTAL 0 7
Patternone:Studentswhorecalleddetailsfrommemorywhencombingmultiple
sourcesmaybebetteratsynthesistasks.Allofthestudentsinthehighgrouprecalledatleast
onedetailfrommemorywhencombiningmultiplesources.However,onlyonestudentinthe
middleandlowgroupsrecalledadetailfrommemorywhencombininginformationfrommultiple
sources.
Ava,inthehighgroup,recalleddetailsfrommemoryonthecrossingtheDelawaretask.Ava
clickedonthetaskandthenonthelastsourcesheread.Shereadquickly;scrolledtothebottomof
thepage,andclickedbackonthetask.Iaskedher,“Whatareyouthinking?”Avaresponds,“Like
thepaintingshowsitwasdaytimelikewithlight,butitwasactuallydoneinthesnowstormat
night.Herresponsecombinedinformationfrombothofthesourcesshereadbutsheonlyreviewed
onesourceduringthesecondsynthesistask.
152
Jacob,inthemiddlegroup,whenhewasonthesecondsynthesisforthecausestaskwas
asked,“Whatareyouthinking?”Jacobrespondedbysaying,“Ihavetocombineallthis,”ashe
tabbedthroughallofhisopenedsources.Jacobrecalledthefirstdetailfrommemorybyalmost
typingasentencefromverbatimthat,“Therewasnosingularcauseofthewar.”
Patterntwo:studentswhorecalleddetailsfrommemorymaybebetterattakinga
position.Therewerenotgreatdifferencesintheuseofthisstrategyacrossallofthe
participants.Therewere,however,differencesinthenumberofdetailsrecalledbysome
students.Ava,forexample,wasabletorecall14separateideasacrosshertwotasks.Sheonly
referredbacktooneofhersourceswhentakingapositionforabrieffewseconds.Duringthe
DelawaretaskAvawrote:
1.no,therearemanythinswrongwiththepainting2.thepaintinghasadaytime
scene,butitwasactuallydoneatnightinasnowstorm.Alsotheboatssizeand
shapeisn’thistoricallycorrect,andifgeorgeactuallystoodupIit,hewouldhave
drowned.
Onhernexttask,Causes,Avarespondedwithoutreturningtohersources:
ithinkthebritishtryingtocontroleverythinganditmadetheamericansfrustrated
becausetheycametoamericatogetfreedom,nottobebossedaround.2.thebritish
increasedtaxesonthethingsthattheamericansneededorboughtalot,likestamps
andtea
IneachcaseAvarecalledallofthesedetailsfrommemory.Avaneverclickedbackontothesources
shefound.Inherresponsesshedirectlyaddressedthequestionposedinthetask.
Studentsinthemiddle‐performinggroupalsoreliedonmemorytorecalldetails.However,
onlySophiaandAlexanderwereaselaborateasstudentsinthehigherperforminggroup.Sophia
beginsthetaskbyaskingme,“Inthisquestiontheyareaskingifitisright?”Ireplied,“Youare
takingaposition.Tellmewhatyoubelieve.”
153
Sophiathenstartedtypingherresponsebyaddingtheword“No.”AssheistypingIasked
heraboutheranswerandSophiarepliedbysaying,“Nobecausetheylikegivegoodpoints.Asin
likeitswinter,soitsnotgoingtobelikewiththesunout.”Sophiathenwrote:
1.No,becauseitswinterandthesunwouldn'tbeout.Thewaterwouldn'tbeas
mellowasitshows.Andtheygivefactthatitdoesn'teventalkplaceintheDelaware
River.2.IfoundthatthesettingthepositionthatGeorgeWashingtonisStandingis
wrong,andnotaccurate.Thethingthepeoplearedoingandevenhowmanypeople
weretherewherewrongto,soIthink.
ItisclearthatSophiakeptthegoalinmindasshetookaposition.Shedirectlyansweredthe
inquiryquestionposedandincludedtwodetailsasevidence.
Alexanderalsorecalleddetailsfrommemory.OntheDelawaretaskherepliedtothe
questionpromptwith:
1.Idon’tthinkit’saccuratebecausethepointtheyraisedaboutitbeingunaccurate
wasgood.2.Ifoundmywebsitesayingtheydidn’tthinktheboatwouldbeabletotay
afloatwithsomnypassengersaboardthesmallboat.
InthisresponseAlexanderisclearlyawareofhisgoal.Hespecificallyaddressesthetask.
Ontheotherhand,somestudentsinthemiddlegroupsuchasJacobonlyimplicitly
addressedthetask.Jacob,forexample,onthecrossingtheDelawaretask,spokeofthelackof
camerastocapturethehistoricalmoment.Jacobwrote:
1.NoIdonotelieveitisbecausehewasn’ttheretoseeithappenohehadtoguess.
2.thattherewasnoonewithcamerastokeeptheimagecorrect
ThefirstsourcethatJacobfoundforthetaskmentionedacamera.Specificallythesite,
AmericanHistory.orgopenedwiththeline:
Wouldn'tithavebeengreatifagroupofnewsreporterswithhightechcamerasand
soundequipmentlinedtheshoresoftheDelawareRiver…
154
Jacobusedthisdetailthroughoutthetaskashismainevidencetosupporttheclaimthatthe
paintingwasnothistoricallyaccurate.
Studentsinthelowestperforminggroupalsousedastrategyofrecallingdetailsfrom
memory.Thesestudentshoweverwerenotassuccessfulatrecallinganumberofrelevantdetails.
Jaydan,forexample,reliedonmemorytodevelopaclaimandprovideevidence.Onbothofhis
ORCAtasksJaydanneverleftthescreenofthetaskthataskedhimtotakeapositiononwhathe
read.Hequicklytypedaresponseandmovedon.OntheTurningPointtaskhewrote:
1.TheTurningPointwasatsaratoga.SomeothersweretheBattleofBennington.2.
itsaysthatthebattleofbenningtonwasgoingtobeasneakattackbutitdidntwork.
OnthecrossingtheDelawaretaskJaydanwrote,“Yesithinkso.‐‐Theysaiditwasarealhappening
andtheysaidwhenitwaspainted.”
Michaelstruggledtousedetailsfrommemoryinhisposts.OntheWomenintheRevolution
taskMichaelwrote,“1.Thaycleanedcookedandweremedic’s.2.IDK”Hedidnottrytouseanother
strategysuchasreturningtohisoriginalsource.
Emmaalsorecalledsparsedetailsfrommemoryinhertakeapositiontask.Onthesame
WomenintheRevolutiontaskEmmawrote,“1.MostBattledwiththemen.2.InonewebsiteI
searchedittoldaboutwomenstayinghomebutthousandsbattled
Themefour:Studentsappearedunpreparedtoengageinthecommunicationstrategies
necessaryforonlinereadingcomprehensioninacademicsettings.
Thedatafromthisthinkaloudportionofthestudyindicatedthatstudentswerenotfully
preparedtoengageinthecommunicationstrategiesnecessaryforonlinereadingcomprehension.
Thisthemewasevidentinstudentresponses:(a)thatfailedtoprovideadequateinformation,
especiallyfromtheevidencethatwasread;and(b)whenstudentsdidnotusetheaffordancesof
onlinecommunicationspaces.
155
Studentresponsesfailedtoprovideadequateinformation,especiallyfromthe
evidencethatwasread.Thestudentresponsestothecommunicationtaskswereshort,
unfocused,andoftendidnotincludeevidencefromwhattheyread.Exceptfortwostudentsno
participantprovidedinformationabouttheirsources.
Whenstudentsfinishedthefinalsynthesistasktheywouldclick“Next”andthenwere
broughttoadirectionscreen(seeFigure5.1).Onthepagewerescreenshotswithdirectionsto
logintothediscussionboard.Atthebottomofthepagewasalinktothediscussionboard.Once
theyloggedonstudentscouldchoosefromthefourtopics(seeFigure5.2).Thenwhentheyclicked
onatopictheysawascreenwithanoriginalpostfromafictitiousteacherandthentworeplies
fromstudents(seeFigure5.3).
Icreatedtworesponsesbyfictionalstudentsundereachtopic.Thesewerebriefstatements
thatcontainedlittleornoinformationthatcouldbeusedbyparticipantsinformulatingtheir
answers.Itisunknown,however,ifthebrevityanddiscourseintheinitialpostsinfluencedthe
lengthandcontentofthepostsbypeers.
157
Figure5.3.ScreenShotofWomenintheRevolutionDiscussion
Thetaskaskedstudentstoagreeordisagreewithamessagepostedbyanotherstudent.The
studentresponsesaredisplayedinTable5.8.Itisclearthatstudentresponses,evenwhentheytake
apositionareshortanddonotuseevidence.Table5.8alsoshowsthatstudentsinthehighest
performinggroupsdidnotengageinanycommunicationstrategiesthatwouldhaveincreasedtheir
performanceontheORCAtask.
158
Table5.8DiscussionBoardResponsesStudent MoreRestricted Lessrestricted Isabella TurningPoint‐Agree Delaware‐disagreeithappenedin
thedeadofnight,andtheystartedinlateafternoon.
Olivia Causes‐TheBostonTeaPartyandFrenchandIndianWarcausedtheAmericanrevolutionbecausetheBritishmadetheAmericanspaytheirshareofthewardebtoftheFrenchandIndianWar.AlsotheBritishputtaxesonteawhentheyknewpeopledidn’thavealotofmoneytospareonuselesstaxeslikentea.
Women‐ActuallythewomendidplayabigpartoftheAmericanRevolution.Theycamtogethertofightathometosavehurtsoilders,keephomessafeabdtostopimportsofgoodsfromGreatBritian
Ava Alsothestampactandsugaractbecauseitincreasedpricesofthingsamericansusedeveryday.
Delaware‐Ithinkthepaintingisn’taccurate,becauseoftheboats,thefactthatGeorgeisstandingup,ifhedidthathewouldhavedrowned.
Sophia TurningPoint‐TheBattleofSaratogawastheTurningPointifthisdidd’thappentherewouldhavebeenachancetheAmericanslost.
Delaware‐Thistomeiswrongbecausethepaintinggivesthewrongfacts.Theplaceandwhathe’sdoingisallwrong.Andthepaintingsendsoutthewrongfacts.Sothisdoesn’tshowwhatitreallylookedlikedwhathappened.Butitdoesshowhimleadinghismen!
Alexander IagreebecauseIfoundtheBritishmadeAmericanspayabigshareofthewardebtfromtheFrenchIndianWarandIagreewithyouransweroftheBostonTeaPartybecausethatwaswenAmericansprotestedtheBritishsothatwasabigcause.
Delaware‐ Ithinkthewarwasreal,butIdonotthinkthepaintingisaccurate.IthoughtthepaintingwasunaccruatebecauseofmanyreasonsIfoundonmywebsitestosupportmythoughts
Jacob Idisagreebecausetherearemanymorecausesthantheses.
Idisagreebecausebeleivthebattleactuallyhappenedbutthepaintingisfakebecausehewastatthebattlepaintingsoitisallcorrect.
Ethan TurningPoint‐youareextremly Delaware‐Thepaintingisrealandif
159
rightaboutthatquotebecauseifyougoonwikipedia.organdtypeintheTurningPointoftherevolutionitwillsaythebatteofsaratogainthatsentence
yougotometmuseum.comandtypeinGeorgeWashingtoncrossestheDelawareriveritwilltellyouthetruefactsaboutthishistoricalevent.
Emma N/A Womendidhavearoleanditwas
major.Somestayedhomeandwerehousewivesotherswenttobattle.
Jaydan TheBattleofBenningtonwasanotherTurningPoint.IttookplaceinNewyork.IthappenedAugust161777
Delaware‐Thepictureshowsthattherewasiceanditwasacoldnight.Sohismenweretiredandcoldsoitwashardtofight.
Michael ThebattleofSaratogawasthe
TurningPointoftheAmericanRevolutionnotthebattleofBennington
Thaywereusedasmedics’theycookedandcleaned
Thetaskaskedstudentstoagreeordisagreewithamessagepostedbyanotherstudentand
thenincludeevidencefromwhattheyread.Studentswerealsoaskedwheretheyfoundtheir
information.
InthetopperforminggroupIsabellatookapositionononetaskbutincludednoevidence.
InhernextresponseIsabellatookapositionandincludedevidence.Olivia’sandAva’sresponses,
whilebrief,tookapositionandprovidedevidence.Nooneinthetopgroupincludedanydetails
aboutthesourceoftheirinformation.
Sophia,inthemiddlegroup,forexample,ontheDelawaretask,respondedtoapostbya
fictitiousstudentBrianBwhowrote,“Thepaintingisfake.Thebattlenevertookplace.”Sophia
repliedback:
Thistomeiswrongbecausethepaintinggivesthewrongfacts.Theplaceandwhat
he’sdoingisallwrong.Andthepaintingsendsoutthewrongfacts.Sothisdoesn’t
160
showwhatitreallylookedlikedwhathappened.Butitdoesshowhimleadinghis
men!
BasedonSophia’sresponseitishardtoinferifshewasstatingthatBrianB’spositionwaswrongor
ifthepaintingwaswrong.
EthantookapositionontheTurningPointtask.Inhisprompthestatedthattheauthorwas
“extremlycorrect.”EthanthenwentontoexplainhowtheBattleofSaratogawastheTurningPoint.
However,thepostthatEthanwasrespondingtotookthepositionthattheBattleofBenningtonwas
theTurningPoint.Therefore,eventhoughEthantookapositionhewasrespondingtotheincorrect
prompt.InotherwordsEthanwasarguingthatthefictitiousstudentwhostartedthethreadwas
correctbutthennegatedthispositionwithhisevidence.
Otherstudentshadamoreimplicitposition.Theymadeaclaim,buttheparticipantsdidnot
explicitlyagreeordisagreewithapreviouspost.Emmaforexample,ontheWomentask,
respondedtoJuliowhooriginallyposted,“WomenplayedamajorroledintheAmerican
Revolution.Theydidlotsofstuff.Emmarespondedwith,“Womendidhavearoleanditwasmajor.”
Michaeltookasimilarapproachandposted,“Thayweremedics’theycookedandcleaned.
Intermsofexplainingwherestudentsfoundtheirinformationtheresponseswereonceagain
inadequate.Onlyonestudent,Ethan,actuallyincludedanyinformationabouthissource.One
student,Alexander,impliedtheuseofasource.Themajorityofthestudents,however,
communicatedevidencefromthesourcestheyreadbutdidnotprovideanydetailson“whereyou
gotyourinformationfrom,”aswasaskedinthetaskinstructions.
OnlyEthanusedastrategyofembeddingacitedasourceinhisdiscussionboardpost.In
eachtaskhementionedawebsite,butdidnotprovideanaddresstoaspecificwebpage.Onthe
crossingtheDelawaretaskEthanfirstloggedin.Hemousedoverallfourtopics.Iaskedhim,“What
doyouneedtodo?”Ethanresponded,“Youneedtoclickonwhichonetheyweretalkingabout.”He
161
thenclickedonthecorrectdiscussion,scrolleddownandreadthetask.Heclickedthequotebutton
andtyped,“Thepaintingisrealandifyougotometmuseum.com….”
Alexanderusedastrategyofimplyingasource.Hemadeavaguereferencetothewebsites
hereadinbothofhisdiscussionboardposts.OnthecausestaskAlexanderreferredtothesources
helocated.Hewrote,“IagreebecauseIfoundtheBritishmadeAmericanspayabigshareofthe
wardebt….”OntheDelawaretaskAlexanderwrote,“Ithoughtthepaintingwasunaccruatebecause
ofmanyreasonsIfoundonmywebsitestosupportmythoughts.”Ineachoftheseexamples
Alexanderdemonstratedsomeknowledgeofhowtousesources.
Studentsdidnotusetheaffordancesofonlinecommunicationspaces.Theparticipants
didnotusethehypertextfeaturesthatwereavailabletothem.Thisisakeystrategytoimproving
communicationinonlinespaces.(Burnett&Meyers,2006).Thediscussionboardtoolincludeda
commontexteditor,whichallowedstudentstobold,underline,anduselists.Itranscribedand
examinedstudentresponses(SeeTable5.8)andnooneusedtheeditortoincludehyperlinksto
sources.OnlyEthanincludedatypedURLwhenhewrote,“Thepaintingisrealandifyougoto
metmuseum.com.”Thislackofusingonlinewritingtoolsindicatesthatstudentusedsurfaceonly
textualfeaturestocommunicate(Burnett&Meyers,2006).
Evidencefromstudentverbalfeedbackalsosupportsthepatternthatstudentsdonotuse
textualfeaturestocommunicateideasinonlinespaces.Nostudentsvoicedanyattemptofusing
hypertextwhenpromptedbytheresearcher.Asstudentsweretypingtheirresponses,Iasked,
“Whydidyouwriteitthisway”or“Whydidyouformatyourresponsethisway.”Eachansweronly
referredtothecontentandnotthedesignoftheresponse.Notonerespondentmentionedtheuse
oftextualandmultimodalelementstoimprovetheirabilitytocommunicateinadigitalage
(Merchant,2007).
162
StageOneSummary
Stageoneanalysishighlightedgeneralthemesthataffectedoverallperformanceonthe
ORCAtaskregardlessoftherestrictednatureoftheinformationspaceandmayhelptoexplain
someofthequantitativefindings.ThefirstthemesuggestedthatoverallORCAscoreswererelated
toanabilitytostrategicallyassembletextsduringonlinereadingcomprehensiontasks..Thishelps
toilluminatethefindingsfromthequantitativeportionofthestudy.Firstmeanscoresonthe
ORCA‐MRwerelowerthanmeanscoresontheORCA‐LR.Thiscouldindicatethatnavigation
requiredwithinawebsitemadetasksmoredifficultforstudentswhocouldnotsuccessfullyuse
textassemblystrategies..Theresultsofthisqualitativestudymayindicatethatparticipantswith
greaternavigationalstrategiesmayhavescoredhigherinamorerestrictedinformationspace.The
observationsofthestudentswhoscoredthehighestontheORCAindicatethattheymademany
morenavigationalmovesthanthestudentswhoscoredinthelowestgroup.Thefirstthememay
alsoillustratewhycriticalevaluationscoreswerethestrongestpredictorsinbothmodels.The
criticalevaluationinstrumentrequiredstudentstomakenavigationalchoices.Thereforestudents
whoreadwithcursorcontrolmayhavedonewellonbothassessments.
Thesecondthemeappearedtoindicateanoveralllackofsourceevaluationknowledge.This
themeindicatedthatthemajorityofstudentswerenotpreparedtoevaluateonlineinformation.
Theresultmayexplainthelowmeanscoreof5.10onthe12‐itemassessmentwith9beingthe
highestscoreontheCriticalOnlineInformationLiteraciesassessment.Thepatternsofstrategyuse
aroundtheevaluationofonlinesourcesmayalsoexplainwhyscoresonthecriticalevaluation
measurewerethestrongestpredictorofonlinereadingcomprehensionscores.Itisclearbasedon
thequalitativeobservationsthatstudentsingenerallackedsourcingskills.Itisalsoevidentfrom
thequalitativedatathatcriticalevaluationiscentraltoonlinereadingcomprehension.Therefore
scoresofthestudentswhodidwellontheCOILmayhavebeencorrelatedwithahighscoreonthe
ORCA.ConverselylowCOILscoresmayhavebeencorrelatedwithalowscoreontheORCA.Finally
163
theillustrativeexamplesincludedinthequalitativeportionofthisstudydemonstratewhythe
evaluateitemswerethemostdifficultforstudentsinboththeORCA‐LRandtheORCA‐MR.Infactin
examiningthetenstudentsincludedinthestudytherewereonlytwoexamplesofstudentswho
successfullyjudgedauthorexpertiseandauthorpointofview.Ifthispatternheldtrueacrossallof
thequantitativeparticipants,theevaluateitemsoftheORCA‐LRandtheORCA‐MRwouldbethe
mostdifficult.
Thethirdthemeindicatedthattheabilitytorecalldetailswhenreadingmultiplesourcesor
takingapositionwasakeyindicatoronsuccessontheORCAtask.Studentsinthehighestgroup
reliedonastrategyofrecallingdetailsmorethanstudentsinthelowestgroup.Thereforethis
studymayprovideevidencethatmoresuccessfulonlinereadersengagein“flexibleassemblage”
(Deschryver&Spiro,2008p.15)ofknowledgebycreating“schemaatthemoment.”Thiswould
indicatethattheroleofbackgroundknowledgemaynotbeasimportantaswetransitionaway
fromaprintbasedworld.ThismaysimplybeduetotheInternetbeingtheworld’slargestexternal
storageofhumanknowledge.Inotherwordswhyremembersomethingifyoucan“GoogleIt”.
Thefourththemeindicatedthatstudentswereunpreparedtoengageinthecommunication
strategiesnecessaryforonlinereadingcomprehensioninacademicsettings.Thisresultisreflected
inthequantitativefindingsofthestudyinthatmeanscoresonthecommunicationitemswerethe
secondlowestonboththeORCA‐LR(1.55)andtheORCA‐MR(1.206).Itwasevidentfromthe
qualitativedatathatstudentsoftenfailedtoprovideevidencefromwhattheyhadread.Ifthis
patternheldtrueacrossallthequantitativeparticipantsitwouldhelpexplainthelow
communicationscores.Similarlyinthisstudystudentsdidnotfullyusetheaffordancesofonline
communicationspaces.Ifthispatternheldtrueinthequantitativedataitmighthelpexplainwhy
manystudentsdidnotreceivepointsforexplainingwheretheyfoundtheirinformation.
Thepurposeofstageoneanalysiswastoidentifydifferencesinstrategyuseacrossthe
ORCAtasksregardlessoftherestrictednatureoftheinformationspace.Thisanalysisexamined
164
differencesinstrategyusebasedonperformanceontheORCAThefollowingstrategieswere
utilizedbythehigherperformingstudents:
utilizing tabs to navigate between task and source;
using tabs to navigate between multiple sources;
using a strategy of skimming websites to identify key details;
engaged reading within a website;
navigating to a source to read while taking a position;
navigating to the source to copy and paste details;
identifying the author on an about us page;
evaluating author using effective markers of expertise;
checking evidence against a secondary source;
inferring an author’s point of view;
recalling details from memory when combining sources;
recalling details from memory when taking a position.
StageTwoThemes
Themesthatemergedfromstageone,highlighteddifferencesinstrategyusethataffected
overallORCAperformance.Thegoalofthestagetwoanalysiswastoexaminethedatainorderto
answerresearchquestionsthreeandfourtoseeifhighandlowperformingstudentsutilize
differentstrategiesinlessrestrictedinformationspacesandmorerestrictedinformationspaces.
Onceagainrecursive,analyticinductivemethods(Angrosino,&MaysdePerez,2000;Bogdan&
Bilken,2003)wereusedtomakeadditionalpassesthroughcodedvideosofstudentthinkaloudsto
identifypatterns.
Itwasevidentfromthisinitialanalysisthatonlyasingledifferenceappearedbetweenhigh
andlowperformingstudents,inrelationtothenatureoftheinformationspace,andthisappeared
onthelocatingitems.Thus,inthestagetwoanalysis,Ireturnedtothedatatolookfordifferent
165
patternsofstrategyuseonthelocatingtasks.Thisanalysis,usingrecursive,analyticinductive
methods(Angrosino,&MaysdePerez,2000;Bogdan&Biklen,2003)wasconductedinrelationto
researchquestionsthreeandfour:
ResearchQuestionThree:Whatpatternsofonlinereadingcomprehensionstrategies
appearamonghighandlowperformingonlinereadersduringanonlinereading
comprehensiontaskwithinalessrestrictedinformationspace?
ResearchQuestionFour:Whatpatternsofonlinereadingcomprehensionstrategiesappear
amonghighandlowperformingonlinereadersduringanonlinereadingcomprehension
taskwithinamorerestrictedinformationspace?
WhatPatternsofOnlineReadingComprehensionStrategiesAppearAmongHighandLow
PerformingOnlineReadersDuringanOnlineReadingComprehensionTaskWithinaLess
RestrictedInformationSpace?
Inthestagetwoanalysisitwasclearthattherestrictednatureoftheinformationspaceonly
affectedstrategyuseonthelocatingtasks.Thereweretwolessrestrictedtasksincludedinthe
ORCAassessment.Studentshadtocompleteoneofthetwoforthequalitativepartofthisstudy.
ThefirsttopicwastheroleofWomenintheAmericanRevolution(Women).Thesecondtopicwas
thehistoricalaccuracyofEmmanuelLeutze’spaintingofGeorgeWashingtoncrossingtheDelaware
(Delaware).Studentswereasked,forexampleto,“Findonewebsiteaboutthehistoricalaccuracy
ofEmmanuelLeutze’spaintingofGeorgeWashingtoncrossingtheDelaware.”Threepatterns
emergedwhenexaminingthedifferencesofstrategyuseamonghighandlowperformersonthe
lessrestrictedORCAtask:
1. StudentswhoperformedbetteronthelessrestrictedORCAusedmoreeffectivestrategies
forenteringkeywords.
2. StudentswhoperformedbetteronthelessrestrictedORCAusedmoreeffectivestrategies
forreadingsearchresults.
166
3. StudentswhoperformedbetteronthelessrestrictedORCAmademoreaccuraterelevancy
judgments.
Patternone:StudentswhoperformedbetteronthelessrestrictedORCAusedmore
effectivestrategiesforenteringkeywords.Acleardifferenceofstrategyuseamongstudents
whoweremoresuccessfulandlesssuccessfulonthelessrestrictedtaskappearedinthestrategies
usedtoenterkeywords.Threestrategieswereevidentinthispattern:(a)moresuccessfulstudents
includedboththetopicandfocusofthetaskwhentheyenteredkeywords;and(b)moresuccessful
studentsincludedboththetopicandfocuswhentheycopiedandpastedkeywords;(c)andsearch
enginefeaturesoftenimpededeffectivesearches.
Moresuccessfulstudentsincludedboththetopicandfocusofthetaskwhenthey
enteredkeywords.ThemajorityofstudentsontheORCAlessrestrictedtaskreliedonastrategyof
usingkeywordsfromthetask.ThepatternisdisplayedinTable5.9.Yetthestudentswhowere
successfulatkeywordentryincludedbothtopic(e.g.paintingofGeorgeWashingtonCrossingthe
Delaware)andfocus(historicallyaccurate)(Eagleton&Guinee,2002).Theinclusionofbothtopic
andfocusposedinthetaskwasamoreeffectivestrategyforlocatingwebsitesonthelessrestricted
task.
167
Table5.9FrequencyofStrategiesUsedtoDevelopKeywordsParticipantsgroupedbyORCAscores
Usekeywordsfromtask
Copyandpastewordsfromtask
UsesSearchEngineTools
Revisekeywords
IncludedTopicandFocus
High Isabella 1 0 2 0 0Olivia 3 0 0 0 3Ava 3 0 1 0 3Total 7 0 3 0 6 Medium Sophia 2 2 1 4 1Alexander 4 0 2 2 1Jacob 0 3 0 2 0Ethan 1 1 1 1 1Total 6 5 4 9 3 Low Emma 3 0 2 2 0Jaydan 3 0 0 1 1Michael 2 0 1 1 1Total 7 1 3 4 1
Successfulkeywordentryvariedsomewhatforparticipantsinthehighestgroupofscores.
Studentswhorememberedboththetopicandthefocus(Eagleton&Guinee,2002)demonstrated
moresuccessfulkeywordentry.Ava,ahighperformingstudent,forexample,begantheDelaware
taskbygoingtoGoogle.Shethentypedinthekeywords,“GeorgeWashingtoncrossingtheDelaware
accuracy.”Avathensaid,“Iaddedaccuracybecauseitmeanstruthful.”
Oliviawasalsosuccessfulonthekeywordtask.ItmustbenotedthatontheWomeninthe
AmericanRevolution,lessrestrictedtask,thereweremoresearchstringsthatwouldreturn
relevantsites.OnherfirstkeywordentryOliviaused,“WomenintheAmericanRevolution.”Onher
nexttwoqueriesOliviausedthephrase,“WomenandtheAmericanRevolutionWar.”
168
IsabelladidnotbeginhersearchstringswitheffectivekeywordsonthePaintingtask.On
herfirstsearchIsabellaused“EmanualeGottliebLeutzepictureonGeorgeWashington.”Then
Isabellaused,EmanuelGottliebLeutzePaintingofGeorgeWashington.Finally,whenlookingforthe
thirdwebsite,Isabella,usedthesearchterm,“PaintingofGeorgeWashingtonbyEmmanuel.”I
askedherwhatshewasthinkingandIsabellasaid,“Iamlookingforinfoonthepainting.”Whileshe
mentionedthetopicIsabelladidnotmakereferencetothefocusofthetask.
Nostudentsinthemiddlegroupbegantheirinitialsearchstringwithboththetopicandthe
focus.AlexandersearchedjustforthetopicontheDelawaretask.Forexample,Alexanderbeganhis
firstsearchlookingfor“GeorgeWashingtonCrossingtheDelaware.”Thisloadedwebsiteswithonly
historicalinformation.Alexanderwasunabletofindrelevantwebsitesusinghiskeywords.
StudentsinthelowestgroupalsostruggledwithkeywordentryontheDelawaretaskbut
wereabletoenterkeywordsontheWomenintheRevolutionTask.Ethanbeganbytyping“George
WashingtonCrossingtheDelaware.”Theauto‐fillsuggestionsgavehimresultsfor“George
WashingtonCrossingDelawareRiverhistoricallyaccurate.”Ethanthendeletedthewords:
historicallyaccurate.Hisnexttwosearcheswereavariationoftheartistsnameandthepainting.
Jaydan,lookedforinformationusingthesearchterms“thepaintingofGeorgeWashington
crossingtheDelaware.”Thisreturnedsearchresultsabouttheactualpainting.Websitesthat
discussedthehistoricalaccuracyofthepaintingwerenotinthetopsearchresults.
OntheWomentaskthelowperformingstudentsfairedbetter.Emmabeganwiththekey
words“therolewomenplayedintheAmericanRevolution.”Sheincludedthetopic“womeninthe
AmericanRevolution”andthefocus“role.”Onhernexttwosearchessheused,“womenandthe
AmericanRevolution.”Michaelalsobeganwith“womenoftheAmericanrevolution”andfollowed
thisupwith“womenintheAmericanRevolution.”
Moresuccessfulstudentsincludedboththetopicandfocuswhentheycopiedand
pastedkeywords.Onlythreestudentsusedastrategyofcopyandpastesoitishardtodraw
169
patternsaboutthisstrategyuse.Noneofthehighestperformingstudentsusedthiskeyword
strategy.However,Sophia,amid‐levelstudent,includedthetopicandfocusinthetask.Sophiaused
acopypastestrategytoentertheauthor’snamebutthentypedtherestofherkeywords.Sherelied
oncopyandpastethroughoutthetasksforhardspellingssuchasnamesandcities.Sheincluded
boththetopicandfocusinherfinaltaskandsucceededonfindingawebsiteforthesecondsearch
item.
Studentswhocopiedandpastedkeywordswithoutthetopicandfocusdidnotsucceed.
Jacobforexample,ontheDelawaretaskonlysearchedfor“EmanuelGottliebLeutze’sGeorge
WashingtonCrossingtheDelaware.”Relevantsearchresultswerefurtherdownthelist.Ethanalso
wasunsuccessfulatusingacopyandpastestrategy.Hecopiedandpasted,EmanuelGottlieb
Leutze’spaintingofGeorgeWashington.
Searchenginefeaturesoftenimpededeffectivesearches.Therewasgreateruseofsearch
toolsbuiltintothesearchengineonthelessrestrictedtask.Yettherelianceonthesearchengine
toolsactuallyhinderedstudentssearchingabilitiesratherthanhelptoscaffoldtheirInternet
inquiries.
Isabellausedtheauto‐fillfeature.Auto‐filltriestopredictsearchterms.Itgivesyouadrop
downlistofoptionsandfillsinthesearchbarwiththe“bestprediction.”Shetyped“George
Washingtoncrossing…”andthenchose“GeorgeWashingtoncrossingdelawareriver”fromthe
suggestedsearchers.Therewerebetterkeywordssuggestedbyauto‐fill.Thesesearchterms
includedboththetopicandthefocus.Isabelladidnotusethese.Shepickedthefirstauto‐fill
suggestion.OnthenextsearchIsabellatyped,“GeorgeWashington…”andselected“George
Washingtonpainting”Shethentyped“de…”andselected“GeorgeWashingtonpaintingdelaware
river.”
Alexanderusedtheauto‐filltofinishhiskeywords.Thesekeywords,however,includedonly
thetopicandnotthefocus.Alexanderbeganbytyping,“GeorgeWashingtoncr…”hethenletthe
170
auto‐fillfinishtherest.Alexandersearchedfor“GeorgeWashingtonCrossingtheDelawareRiver.”
OnhisthirdsearchtaskAlexanderagainreliedontheauto‐fill.Hetyped“GeorgeWashington
crossingDelawareriver.”thenselected“GeorgeWashingtoncrossingtheDelawareriverfacts”from
theauto‐fillsuggestedsearches.
Sophiaalsousedtheautofill.Shetyped,“GeorgeWashingtoncrossing.”Shethenselected
“GeorgeWashingtoncrossingthedelawarepainting.”Thesuggestedsearchtermsdidnotincluded
anyinformationabouttheaccuracyofthepainting.
Michaelalsousedtheauto‐fillfeature.Hebeganbytyping“womenoftheam..”Hethen
selectedthesuggestedsearchtermof,“womenoftheAmericanrevolution.”Onceheclickedonthe
searchtermhewasbroughttotheresultpage.
Asecondsearchenginefeaturethatstudentsreliedonwasthesuggestedspelling.There
werenotenoughinstancestodrawclearpatterns,butonceagainthesuggestedspellingoften
servedasahindranceratherthanasupportivescaffoldsearchingtheInternet.
Emma,asecondlanguagestudent,usedthesuggestedspellingfeature.Sheoriginally
searchedfor,“therolewomenplayedintheAmericanrevolution.Googledisplayedresultsforthe
correctspellingbutgavelinkstosearchesusingtheincorrectandthecorrectspelling.Emmachose
theincorrectspellinglink.Thepagethatloadeddisplayedalinktothesearchresultswiththe
correctspelling.Emmaeventuallypickedthecorrectlink.
Ava,ahighperformingstudent,wastheonlystudenttousethesearchenginefeatures
correctly.AvausedthecorrectedspellingtoolsbuiltintotheGooglesearchengine.Sheforgota
spacebetweenDelawareandpainting.Googledisplayedtheresultsforthecorrectspellingbut
providedalinktothesearchresultsforboththecorrectandincorrectspelling.Avaclickedonthe
linkforthecorrectspelling.
PatternTwo:StudentswhoperformedbetteronthelessrestrictedORCAusedmore
effectivestrategiesforreadingsearchresults.Thesecondpatternofstrategyuseamong
171
studentsmoresuccessfulonthelessrestrictedORCAwastheuseofmoreeffectivestrategiesfor
readingsearchresults.Twopatternsofstrategyusewerenoted:(a)activelyreadingsearchresults;
and(b)navigatingbacktosearchresultsonthelessrestrictedtasks.
Activelyreadingsearchresults.Thequalitativeresultssuggestthathowstudentsread
searchresultsmaybeoneofthemoredefiningindicatorsoftheirperformanceontheonline
readingcomprehensionassessmentsinlessrestrictedinformationspaces.Thesepatternsare
displayedintable5.10,whichshowsthefrequencyofsearchenginereadingstrategiesnoted.
Overallstudentsinthehighestandmiddleperforminggroupsactivelyreadsearchresults,returned
tothesearchresults,andclickedonmorelinks.
Table5.10FrequencyofStrategiestoReadSearchEngineResults
ParticipantsgroupedbyORCAscores
Cursormovementorverbalizationindicatesreadingofresults.
Clicksonfirstlinkwithoutskimming
Movestosecondpage
Clicksonasearchresult
HighIsabella 2 0 0 3Olivia 6 0 0 8Ava 2 0 0 2TOTAL 10 0 0 13
MiddleSophia 5 0 0 7Alexander 5 0 0 5Jacob 1 1 0 7Ethan 3 0 0 0TOTAL 14 1 0 19 LowEmma 3 0 0 3Jaydan 1 1 0 3Michael 2 0 0 3TOTAL 6 1 0 9
172
Theactivereadingofsearchresultswasdefinedasreadingmorethanthefirstoronelink.It
requiredeitheraverbalizationorcursormovement.Thisstrategywasmoreprevalentamongthe
highestperformingparticipants.Olivia,forexample,ontheWomentaskbeganbyenteringinthe
searchterms,“womenandtheAmericanrevolution.”Shethenspent39secondsreadingthesearch
results.Iasked,“Whatareyouthinking?”Oliviahoveredoverasearchresultfor“AmericanAthena”
andsaid,“Ithinkthisisitbecauseitsaysamericanrevolution.org.
OnthenextsearchtaskOliviausedthesamekeywords.IagainaskedOliviawhatshewas
thinking.Oliviareplied,referringtothesearchresults,“Iamjustgoingthrough,like,andreading
thelittledescription,kinda,underneath.”After30secondssheclickedonasearchresult.
Avaalsospentaconsiderableamountoftimethinkingaboutthesearchresultsshewould
clickon.OntheDelawaretaskshebeganbyenteringthekeywords,“georgewashingtonpainting
crossingthedelawareaccuracy.”Hercursorhoveredoverthefirstthreesearchresultsmovingleft
torightassheread.Avathenclickedonasearchresulttoawebpagetitled,“DidGeorge
WashingtonreallystandupinhisboatwhencrossingtheDelaware”onthewebsite“ushistory.org.”
Iasked,“Whydidyouclickonthatone?”Avaexplains,“umm.Itlookedlikethemostrelevantto
whatItypedin.”
Studentsinthelowestperforminggroupsdiddisplaysomerudimentaryreadingofsearch
results,butdidnotthinkaboutthelinkstheyclickedon.Emmaforexampleclickedonasuggested
spellinglinkandexpectedittotakehertoawebsite.Whenaskedaboutwhysheclickedonthelink
Emmasaid,“becauseithadlikeeverythingItyped.”ThenEmmamoveddownthepagewithher
cursor.ShefocusedonanotherlinktoanirrelevantwebsitebecausetheURLendedin.org.Emma
didnotclickthelink.Shethenclickedonthecorrectspellingandsaid,“IamnotsureifIshould
clickonthisbecauseitisWikipedia...wellsometimespeopleputtheirownopinions.Iheardthatthe
173
guywhocreatedWikipediawentthroughandchangedeverything.”Emmathenclickedon
Wikipediaanduseditashersource.
Jacobmovedquicklythroughsearchresultswithoutactivelyreadingthem.Hebeganby
copyingandpasting,“EmanuelGottliebLetutze’spaintingofGeorgeWashingtoncrossingthe
Delaware”fromthetask.Hethenclickedsearchandimmediatelyclickedonthefirstlinkinthe
results,readingonlythewebsitetitleoutloud.Jacobdidrealizethattheresultwasirrelevantand
didnotchooseabettersite.
OnthenextsearchtaskJacobusedthesamekeywords.Onceonthesearchresultspagehe
quicklysaid,“Thisseemslikeagoodone,”andclickedonthefourthsearchresult.Iasked,“Which
didyoupick?”andJacobreplied,“Globalwholesaleart.”Hedidnotmaketheconnectionfromthe
searchresultsthatthiswasacommercialwebsite.Onhisnextclickhealsomadethesamemistake
andjustsaid,“Iwilltrythisone,”andJacobclickedon“allposters.com.”Itisapparentthathewas
onaclickandhuntmission.
Michaelreliedonlyonthetitleofthelinkstochoosethesearchresultstoclickon.Hedid
notreadthedescriptionunderthelinkortheURL.OntheWomentaskMichaelclickedthesecond
linkonthesearchresultspage.Hesaidhechoseit,“becauseitsaidwomenintheAmerican
Revolution.”Heleftthepageandthenextlinkhepickedwasalsosolelybasedonthetitle.
Emmaalsodidnotspendtoomuchtimejudgingtherelevancyofhersources.Onthe
WomentasksheclickedonasearchresultfortheWikipediaarticleonthetopic.Sheskimmedthe
articleanddecided,“Ithasinformationonthetopic.”Emmarepeatedthepatternwhensearching
forhernexttwosources.Sheautomaticallywentwiththefirstsourcesheclickedon.Itmustbe
notedthattheseresultswererelevanttothetask.Emma,however,unlikehermoresuccessful
peers,didnotcomparesourcesbeforechoosingoneshebelievedtoberelevant.
Navigatingbacktosearchresultsonthelessrestrictedtasks.Therewasadifferencein
thenumberoftimesstudentsreturnedtosearchresultsontheORCAlessrestrictedtaskswith
174
cursorcontrol.ThefrequencyofthesenavigationstrategiesaredisplayedinTable5.11.Studentsin
thetwohighestperforminggroupsfrequentlyreturnedtotheirsearchresultstofindabetter
source,usingthecursor.Onlyonestudentinthelowgroupreturnedtosearchresults.
Table5.11FrequencyofNavigationstrategieswhenreadingsearchenginesduringthelessrestrictedtaskParticipantsgroupedbyORCAscores
Cursormovementverbalizationindicatesreadingofresults.
Returnstosearchresults
Isabella 2 0Olivia 6 4Ava 2 0TOTAL 10 4
Sophia 5 4Alexander 5 3Jacob 1 4Ethan 3 0TOTAL 14 11 Emma 3 0Jaydan 1 0Michael 2 1TOTAL 6 1
AvaandIsabella,bothhighperformers,didnotreturntotheirsearchresultsontheless
restrictedtask.This,however,wasaresultoftheirgoodkeywordentryandtheircarefulreadingof
searchresults.Ava,forexample,alwaysincludedtheword“accuracy”inherkeywordsonthe
Delawaretask.Shealsocarefullyreadsearchresultsbeforeselectingalinktoclickon.
Oliviawasverymethodologicalinreturningtosearchresults.Aftershereadawebsiteshe
wouldgobacktothesearchresultstomakesuretherewasnotabetteroption.Forexampleduring
theWomentask,afterspendingafewminutesonasiteOlivialeftbecauseit,“Itdidn’thavethat
175
muchinformation.Othersiteshadlikepicturesandstuff.”OnhernextsiteOlivialeftbecausethe
website,“Didn’treallytalkaboutwomenANDtherevolution.”
Oneofthecleardifferencesbetweenstudentsinthemiddlegroupandstudentsinthe
lowestgroupwasnavigatingbacktosearchresults.Alexanderreturnedtohissearchresultsonhis
thirdsearchtask.Thefirstwebsitehevisitedwasaprintcompany.AsheleftAlexandersaid,“This
oneismostlylikeselling.Sobackoutofthat”Hereturnedtothesearchresultsandsaid,“Clickon
theoneaboveitbecauseitsaysWashingtoncrossingtheDelawarebythatname”(referringtothe
artist).
Sophiaalsoreturnedtohersearchresults.AfterenteringherkeywordsontheDelaware
task(missinganyclaimsaboutaccuracy)Sophiafirstclickedonalinktoamuseumsite.Asshehit
thebackbuttonSophiasaid,“Itjusttalksabouthowprettythepictureis.”
Jacobreturnedtohissearchresultsmorethanonce.AsJacobprogressedhissearchterms
improved.BythethirdsearchtaskJacobwasincludingthewordaccuracyinhiskeywords.After
readingamuseumwebsiteJacobleftaftersaying,“Itdoesn’thavemuchonaccuracy.”
InthelowestgrouponlyMichaelreturnedtohissearchresultsonthelessrestrictedtask.
Hedidnotverbalizehisdecision.Hejustwentbacktothesearchresultsaftervisitingarelevant
pagethatwouldhaveworkedforthetask.
Patternthree:StudentswhoperformedbetterontheORCAlessrestrictedtaskmore
frequentlywereabletodeterminethatwebsiteswereirrelevant.Successfulstudentswere
abletodeterminethatwebsiteswereirrelevant.ThispatternisdetailedinTable5.12.Twomajor
strategieswerenotedinthelocatingofrelevantwebsites:(a)Studentswhosuccessfullymade
relevancyjudgmentsduringthelessrestrictedORCAskimmedwebsites;and(b)Studentswho
choseirrelevantwebsitesreliedonastrategyofchoosingthefirstlink.
176
Table5.12FrequencyofRelevancyJudgmentsonlessrestrictedtasksParticipantsgroupedbyORCAscores
Locateswebsiteandjudgesitrelevant
Locateswebsiteandjudgesitirrelevant
Isabella 1 0Ava 3 0Olivia 4 4TOTAL 8 4 Sophia 2 4Alexander 1 5Jacob 3 5Ethan 2 1TOTAL 8 15 Emma 3 0Jaydan 3 0Michael 2 2TOTAL 8 2
StudentswhosuccessfullymaderelevancyjudgmentsduringthelessrestrictedORCA
skimmedwebsites.Oliviaisagreatexampleofastudentwhospenttimeconsideringtherelevancy
ofeachwebsitesheread.ShebegantheWomentaskbyfirstclickingonawebsiteentitled
“AmericanRevolution.”WhensheleftthesiteIaskedher,“Whatareyouthinking?”Sherepliedthat
thesite“…didn’thavealotofinformation,likeotherwebsites.Thathad,like,picturesandfacts.”
Oliviathenclickedonanotherwebsitethatlistedlinkstospecificbiographicalinformationabout
famouswomenoftherevolution.SheleftthewebsiteandIasked,“Whydidn’tyoupickthat
website?”Oliviarespondedthatitdidn’treallytalkabout,“…womenANDtheAmericanrevolution.
Itjusttalksaboutwomenwhofoughtinit.”Oliviathenwentbackandrereadthetask.Shethen
lookedattwomoresources,anddecidedafterallthatthewebsiteshefirstclickedonwasthemost
relevant.
177
Sophiaalsospentconsiderabletimeconsideringtherelevancyofhersourcesonthe
Delawaretask.ShebeganbyfirstclickingontheWikipediaarticleaboutthebattle.Sophiaquickly
returnedtohersearchresultsandchangedthekeywords.IaskedSophiawhysheleftthepageand
shestated,“Idon’tthinkithadtherightinformation.”Sophiathenskimmedoverthesearchresult
andclickedonthefirstlinkbecauseshesaid,“itcamefromamuseum.”Sophiathensaid,“Itjust
talksabouthowbigthepictureis,”andshereturnedbacktothesearchresults.Sophiarejectedthe
museumwebpageasirrelevantonceshereadit.Sophiathenclickedonawebsitetitled,“Whatis
wrongwiththispainting?”Sophiapickedthissourceasbeingrelevanttothetopic.Byexamining
threedifferentsourcesSophiamaderelevancyjudgmentsuntilshelocatedausefulsource.
Studentswhochoseirrelevantwebsitesreliedonastrategyofchoosingthefirstlink.On
theotherhandJaydanrepresentsthetypeofreaderwhodidnotjudgetherelevancyofsourcesand
quicklychoseawebsitefromtheirsearchresults.Thispatternwascommoninthelowestgroupof
performers.Jaydanatfirstdidnotlikethesearchresultshegotfor“GeorgeWashingtonCrossing
theDelaware”andaddedtothekeywords“wasitaccurate.”Jaydanthenclickedonafirstlinktoa
Wikipediaarticleaboutthecrossing.BeforeJaydanevenskimmedthearticlehecopiedandpasted
thelinksintothetask.Jaydanrepeatedthispatternonthenexttwosearchtasks.Healways
acceptedthefirstwebsiteheclickedonasrelevant.
Emma,alsointhelowestgroup,neverjudgedanywebsiteasirrelevantontheless
restrictedtask.OntheWomentaskEmmaalwayswentwithherfirstclick.Sheneverreturnedto
thesearchresultsafterreadingawebsite.
WhatPatternsofOnlineReadingComprehensionStrategiesAppearAmongHighandLow
PerformingOnlineReadersDuringAnOnlineReadingComprehensionTaskwithinaMore
RestrictedInformationSpace?
Stagetwoanalysisrevealedthattheonlydifferencebetweenstrategypatternsappearedin
thelocatingtasks.Thereweretwo,morerestrictedtasksincludedintheORCA.Studentsinthe
178
qualitativeportioncompletedoneofthetwo.Thetwotopicswere:a)thecausesoftheAmerican
Revolution(Causes)andb)theTurningPointoftheAmericanRevolution(TurningPoint).
OntheCausestaskstudentshadtofindthreedifferentwebsites.Thefirsttaskasked
studentsto,“FindthewebpageCausesofRevolutionaryWaronthewebsite
SocialStudiesforKids.com.”Thesecondtaskasked,“FindthewebpageListofCausesoftheWaronthe
websitehistorycentral.com.”Thefinallocatingtaskaskedstudentsto,“Findthewebsitetitled
AmericanRevolutionPersuaders.”
StudentsalsohadtofindthreewebsitesfortheTurningPointrestrictedtask.Thefirsttask
said,“CandacepostedthequestiontothewebsiteYahooAnswers,‘WhatistheTurningPointofthe
AmericanRevolution?’Locatethisdiscussion.”Thedirectionsforfindingthesecondwebsiteasked,
“FindthearticleabouttheBattleofBenningtononthewebsiteTheAmericanRevolution.org.”Onthe
finalsearchtaskstudentswereaskedtofindthearticlesabouttheBattleofSaratogaonthewebsite
SocialStudiesForKids.com.”
Stagetwoanalysisfoundthatthepatternofstrategyusetolocatewebsitesdifferedonthe
morerestrictedtasks.Therestrictednatureofthetask,thatishavingstudentslookforaspecific
source,reducedthenumberofinstancesofkeyworduse.Insteadmanyofthestudentswent
directlytoawebsiteandsearchedwithinthesitefortherequiredinformation.Threepatterns
emergedintheanalysisofthemorerestrictedtask:
1. StudentswhoperformedbetteronthemorerestrictedORCAusedmoreeffectivestrategies
tosearchwithinawebsite.
2. StudentswhoperformedbetteronthemorerestrictedORCAusedkeywordsincludingboth
thetopicandthesource.
3. StudentswhoperformedbetteronthemorerestrictedORCAmademoreaccuraterelevancy
judgments.
179
Patternone:StudentswhoperformedbetteronthemorerestrictedORCAusedmore
effectivestrategiestosearchwithinawebsite.Studentswhoweresuccessfulonlocating
informationduringthemorerestrictedtaskdemonstratedtheabilitytosearchwithinawebpage.
Oftenthisrequiredbetterforwardinferencing(Coiro&Dobler,2007),orpredictingwherelinks
willtakeyou,whenstudentssearchedwithinawebsite.Thestudentsinthehighestperforming
groupsweremoreadeptatlocatingapagewithinawebsite.Studentsinthelowestgroupoften
struggledtofindawebpagewithinasite.ThesepatternsareillustratedinTable5.13.
Table5.13FrequencyofLocatingStrategiesontheMoreRestrictedORCA
Navigatesdirectlytothesource
Searchessuccessfullywithinawebsiteforasource
Usesbothtopicandsourceinkeywords
Copiesandpastesexactwordsfromtask
UsesSearchEngineTools
Reviseskeywords
High Olivia 2 3 4 0 1 0Ava 2 2 1 0 0 0Isabella 2 1 1 4 1 2TOTAL 6 6 6 4 2 2 Medium Sophia 2 2 1 0 1 0Alexander 1 1 0 1 0 0Jacob 0 2 1 0 1 0Ethan 0 1 0 0 1 0Total 3 6 2 1 3 0 Low Emma 0 0 0 0 0 2Jaydan 0 0 0 0 1 0Michael 0 0 1 0 2 0TOTAL 0 0 1 0 3 2
180
Studentsinthehighestperforminggrouptypicallywentdirectlytothewebsitebytyping
theURLintothenavigationbarofthebrowser.Theythensearchedwithinthewebsite.Oliviabegan
bytypingtheURLtosocialstudiesforkids.cominthenavigationbar.Shethenclickedbackonthe
task.Oliviaclickedbackonhersourceandscrolledupanddownthepageandsaid,“Iamtryingto
thinkofwhereitwouldbe.”Sheclickedon“USatWar”intheleftmenubar.NextOliviaclickedon
“RevolutionaryWar.”Shescrolledupanddownthepage,andclickedonthe“2”atthebottomofthe
pagetogotothenextpage.NextOliviaclickedbackandsaid,“Ireadallthedescriptionsandthisis
best.”SheclickedonalinkabouttheAmericanRevolutionandfoundthelinktothecorrectpage
thatexplainedthecausesoftheRevolutionaryWar.
Avaalsosearchedwithinapagebymakingforwardinferencesaboutlinks.Shestartedby
firstsearchingjustforthesourceusingGoogle.OnceshefoundthesourceAvasearchedwithinthe
source.Avasearchedforthespecifiedwebsite,SocialStudiesForKids,inGoogleratherthanthe
webpagethatlistedcausesoftheAmericanRevolution.OnceonthewebsiteAvascrolleddownthe
homepageandclickedonalinkintheleftnavigationbartitled“U.S.atWar.”Shethenscrolleddown
andclickedonthe“AmericanRevolution.”Shethenreadthepageanddidnotseealinkto“causes
ofthewar”inthelefttoolbar.Shedidnotfindtheexactlinkbutwasabletoinferthecausesofthe
waronthewebsite.
OnhernextsearchtaskAvaenteredintheURLaddressdirectlyintothenavigationbar.She
waslookingforthewebpageonthewebsiteAmericanRevolution.org.OnceshewasonthepageAva
movedhercursorthroughthesidenavigationbar.Shefoundthecorrectlinkandsaid,“Thisisit.”
Studentsinthemiddlegroupweresomewhatsuccessfulatsearchingwithinawebsite.They
performedinasimilarfashiontotheirpeerswhohadhigheronlinereadingcomprehensionability.
AlexanderforexamplemadeatotalofnineclicksbeforehelocatedthecorrectpageontheCauses
task.HeenteredtheURLtosocialstudiesforkids.cominthenavigationbar.Alexanderthensaid,“I
amgoingtolookaroundonthistab.ItsayslinksandstuffsoIamgoingtogotohistory.”Hethen
181
scrolledupanddownandsaid,“Iamgoingtolookforcausesoftherevolutionorcausesofthe
AmericanRevolution.”Alexanderthenclickedon,“USGovernment”andscrolledupanddownthe
page.Hethenclickedbackonthetaskandsaid,“Iamlookingforcausesoftherevolutionarywar.
Hethenclickedon“warsaroundtheworld.”Alexanderscrolledupanddown;heclickedonalink
forasecondpageofresults.ThelinktotheAmericanRevolutionwasatthetopofthepage.
Alexanderclickedonit.Hescrolledupanddownthepageandsaid,“MaybeifIclickontimeline.”
Heclickedonthelink,scrolledupanddownthepage,anddidnotfindthewebpage.Hethensaid,
“I’llgoback…,”andheclickedthebackarrow.AshehoveredoveralinkAlexanderadded,“…and
maybethisandfindinformationinit.”HeclickedonalinktotheAmericanRevolution.Thisbrought
Alexandertoapageoffofthewebsite.Hesaid,“Notit,”andclickedthebackarrow.Hemousedover
anadditionallinkandsaid,“Thatonelookslikeaspecificperson.SoIamgoingtodothatone.”He
clickedonalinkabouttheRevolutionandthenclickedonthelinktotherequiredwebpagethatwas
listedontherightofthepageinamenuframe.
Sophiawasabletolocatethespecifiedwebsite.Shewaslookingforthewebpage“Causesof
War”onthewebsitehistorycentral.com.Shecopiedandpastedhistorycentral.comintothe
navigationbar.Shesaid,“OnthewebsiteIgotIamjustgoingtolookforListofCauses.”Sophia
scrolledupanddownthepageandthenfoundthelinktoAmericanRevolutioninthelefttoolbar.A
popupwindowofferingafreeiPadopenedup.Sophiawasabletocloseitwithoutclickingonanyof
thehiddenlinks.SheskimmedthepageandclickedonthelinktoCausesoftheWar.
Jacobfoundtwoofhisthreesourcesbysearchingwithinapage.OntheCausestaskhe
foundthewebpageonsocialstudiesforkids.combyusingtheinternalsearchengine.Hewasthe
onlystudentinthestudytoutilizethisstrategy.Ashewasskimmingthepagehesaid,“Iamgoing
totrysearchinginthisthing.”Hethenmovedhiscursortothesearchbar.
Thestudentsinthelowestgroupdidnotperforminasimilarfashionasstudentsinthe
middleandthehighgroup.Theirsearchstrategiesweremorelimitedandlesssuccessful.No
182
studentfoundthepageswithinawebsite.Twostudentsacceptedthehomepageofawebsiteasthe
correctpagetheywerelookingfor.Onestudenthadtobedirectedtothesourcesbyme.
Jaydanwasunabletofindthecorrectwebsites.Hesubmittedthehomepagesasthecorrect
sites.Whenhegottothesynthesissitehesaid,“Ican’tfindit.Maybehistory.”AfterafewminutesI
foundbothwebsitesforhimusingtheinternalsearchengine.
Emmawasunabletofindanyofthetargetedwebsites.Shesearchedfortwowebpages
withinawebsite.OntheCausestasksheenteredtheURLforsocialstudiesforkids.comandthe
submittedthehomepageURLasananswer.ThenshemovedontotheAmericanRevolution.orgtask
andonceagainsubmittedthehomepageasheranswertothelocatetask.WhenEmmamovedonto
thesynthesistasksheclickedonsocialstudiesforkids.com.Shethenclickedonalinkto“How
Presidentsgetelected”andsaid,“UmmIdon’tknowwhattodo.”Ifoundthetworequiredpagesfor
her.
Patterntwo:StudentswhoperformedbetteronthemorerestrictedORCAused
keywordsincludingboththetopicandthesource.Higherperformingandmiddlelevelstudents
typicallyusedtheURLtolocateinformationonthemorerestrictedtask.Studentsinthelowest
performinggroupsmoreoftenusedasearchengine.Whentheydid,thosewhosearchedforboth
thetitleofthewebpageandthesourceweresuccessfulatthetask.Isabella,forexample,onthe
TurningPointtask,putthekeywords“AmericanRevolution.orgbattleofbennington”inGoogle.She
wasabletofindtherequiredsource.
Middleandlowperformingstudentswhosearchedforthetitleofwebsiteswithoutthe
sourcenamewerelesssuccessfulonthetask.Sophia,forexample,usedthewords,“TurningPoint
oftheAmericanRevolution.”However,shedidnotmentionthespecificsource,YahooAnswersthat
studentsweretaskedtofind.Whilethecorrectwebsiteappearedonthefirstpageoftheresultsit
wasfurtherdowninthesearchresults.
183
Emmaalsoneverincludedanyinformationaboutthesourceinheranswer.Sheenteredin
thekeywords“listofcauses”whenlookingforawebsitefortheCausestask.Shethenspenttime
readingtheresultssaying,“Iamjusttryingtofigureoutwhichitwouldbe.Emmaneverincluded
anyinformationaboutthesource,andneverfoundthecorrectwebsite.
Patternthree:StudentswhoperformedbetteronthemorerestrictedORCAmade
moreaccuraterelevancyjudgments.Studentswhoweremoresuccessfulontherestrictedtask
weremoreabletojudgewebsitestheyvisitedasirrelevant.Thisstrategywasdefinedasnavigating
toawebsiteorwebpageandthenleavingafterdeterminingitwasnotuseful.Studentsinboththe
highandmediumgroupsutilizedthisstrategy.Nostudentsinthelowestthirdjudgedawebsite
theyvisitedasirrelevant.ThepatternisexploredinTable5.14.
184
Table5.14FrequencyofStrategyUseonMoreRestrictedLocatingTasksParticipantsgroupedbyORCAscores
Locateswebsiteandjudgesitirrelevant
LocateswebsitebytypingURLinNavigationbar
Locatesspecifiedwebpagewithinawebsite
Isabella 2 0 1Olivia 0 2 2Ava 1 0 1TOTAL 3 2 4 Sophia 0 0 2Alexander 1 4 2Jacob 0 4 2Ethan 0 0 0TOTAL 1 8 6 Emma 0 2 1Jaydan 0 3 0Michael 0 1 1TOTAL 0 6 2
IsabellaontheTurningPointtaskcouldjudgeawebsiteirrelevant.Whileshewas
searchingfortheYahooanswersdiscussionsheclickedthrougheachwebsiteandjudgedthemto
beirrelevant.Atonepointtheresearchstoppedandaskedher,“Whydidyoupickthatone”when
sheclickedonanirrelevantwebsite,“Itlookedokay,”sheresponded,referringtothesearchresults,
“butitwasashop.”ThewebsitesoldhistoryDVDs.EventhoughIsabelladidnotfindthespecified
websiteshedidexhibitbetterstrategicreadingintermsofjudgingwebsiterelevancy.
OntheotherhandstudentswhowerenotassuccessfulontheORCAtaskshadatendencyto
acceptthefirstlinktheychosefromthesearchresultsortocopyandpasteaURLintothe
navigationbarasthelegitimatesourcethetaskaskthemtofind.ForexampleEthan,whenlooking
fortheYahoodiscussionboard,firstclickedonanarticleonWikipediaabouttheAmerican
185
Revolution.Hesays,“ThishassomestuffIamlookingfor,”andhecopiedthelinktothe
SurveyMonkeytask.LaterinthetaskwhenEthanwaslookingfortheAmericanPersuaderswebsite
heclickedonawebsitesellingstudentessays.Seeingthetitleofthearticlehecopiedandpasted
andsubmittedtheURLascorrect.
StageTwoSummary
Thepurposeofstagetwoanalysiswastoexaminethedatatoseeifthenatureofthe
informationspaceaffectedpatternsofstrategyuseamonghighandlowperformersontheORCA
assessment.Initialanalysisdeterminedthatthenatureoftheinformationspaceonlyaffected
strategyuseonthelocatingtasks.Thereforestagetwoanalysisexaminedifferencesinstrategyuse
onthelocatingtasksinthelessrestrictedspace.Thenstagetwoanalysisexamineddifferencesin
locatingstrategyuseamorerestrictedinformationspace.Thesedifferenceswereanalyzedby
comparingstudentsontheirperformancelevel.
Onthelessrestrictedtasksstudentswhoperformedbetterthantheirpeersusedspecific
strategies.Firsttheyweremoreaptatdevelopingkeywords.Thesekeywordsincludedboththe
topicandthefocus.Theyalsoreadsearchresults.Finallymoreproficientstudentsmademore
accuraterelevancyjudgmentsinthelessrestrictedenvironment.Thefollowingstrategieswere
notedamongbetterperformingstudentsinthelessrestrictedlocatingtasks:
using keyword that included both the topic and focus;
copying and pasting keywords that include both the topic and focus;
actively reading search results;
navigating back to search results;
skimming websites to make accurate relevancy judgment.
OnthemorerestrictedORCAtasksstudentswhoperformedbetterthantheirpeersused
specificstrategies.Moresuccessfulstudentsgenerallysearchedwithinawebsiteratherthanuseda
searchengine.Whenstudentsdiduseasearchengine,thosewhoincludedatopicandsource
186
performedwell.Similartothelessrestrictedtask,studentswhomademorerelevancyjudgments
onthemorerestrictedtaskoutscoredtheirpeersontheORCAassessments.Thefollowing
strategieswerenotedamongbetterperformingstudentsinthemorerestrictedlocatingtasks:
navigating directly to a source;
searching for a specific source;
searching within a website;
using keywords with topic and source;
making accurate relevancy judgments.
Thestagetwofindingsalsoilluminatedthequantitativefindingsofthisstudy.Specifically
thestagetwothemesmayexplaindifferencesinscoresontheontheORCA‐LR(8.394)andthe
ORCA‐MR(7.15).Stagetwoanalysisrevealedthatmoreproficientusersutilizedverydifferent
strategiestosearchwithinasourceonthemorerestrictedtask.Onthelessrestrictedtaskboth
studentsinthehighandmiddlegroupsearchedforwebsitesusingeffectivekeywords.Thefactthat
thelocatingtasksonthemorerestrictedORCArequiredspecificskillsonlyobservedinthemost
proficientreadercouldhavecontributedtodifferencesinthemeanscoresforthesixlocating
scorepointsontheORCA‐LRwas3.599.Themeanscoreforthesixlocatingscorepointsonthe
ORCA‐MRwas2.755.
Furthermorethestagetworesultsmayexplainwhyscoresontheself‐reportdispositions
measureweresignificantintheORCA‐MRmodelbutnotintheORCA‐LRmodel.Basedonthethink
alouddatasearchingwithinasourcewasastrategyprevalentamongproficientusersduringORCA‐
MR.Theinstancesofthisstrategyuseindicatethattheabilitytosearchwithinawebsiterequired
morereflectivethinking,persistenceandflexibility.Thesearethreesubscalesofthedispositionsof
onlinereadingcomprehensioninstrument.ThestudentswhoweremoresuccessfulontheORCA‐
MRtaskdemonstratedanabilitybeyondthatofcognitiveskills.
187
ChapterSummary
Thegoalofthequalitativeportionofthisstudywastoexaminedifferencesofonlinereading
comprehensionstrategyuseamonghighandlowperformersonanassessmentofonlinereading
comprehensioninalessrestrictedinformationandalsoinamorerestrictedinformationspace.
ThisanalysiswasdesignedtoexplainthequantitativedifferencesfoundinChapterFour.Atwo‐
stagequalitativeanalysiswasutilizedtoexaminethink‐alouddata.
Analysisinstageoneidentifiedactionsthataffectedperformanceregardlessofthe
informationspace.Theseincludedgreatersourceknowledgeandcursorcontrol.Furthermorestage
oneanalysishowalackofcriticalevaluationandcommunicationskillslimitedstudent
performance.
Thestageoneresultshelpedtoillustratesomeofthequantitativefindingsofthisstudy.
Moststrikinglythethinkalouddataexploredthedifficultystudentshadwithboththeevaluation
andcommunicationitems.Statisticalevidencefoundthesetobethehardestitemclustersonboth
theORCA‐LRandtheORCA‐MR.Stageoneresultsrevealedthatstudentsseldomhadthestrategies
toevaluateonlineinformationandwereunpreparedtocommunicateinonlinespaces.
Thestageoneresultsmayalsohelptoexplainwhybackgroundknowledge,while
significant,wasnotthebestpredictorofperformanceineithertheORCA‐LRortheORCA‐MR.
Studentswhodidwellinbothformatscouldquickly“reassemble”informationtheyreadwhile
synthesizingsources.Thisabilityrequiredgreatercursorcontrolratherthanbackground
knowledge.
Instagetwothequalitativeanalysisidentifieddifferentpatternsofstrategyusebasedon
therestrictedinformationspace.Theonlydifferenceinstrategyuseamonghighperforming
studentsandlowperformingstudentswasontheitemsmeasuringlocatingskills.Ontheless
restrictedtasksstudentswhosearchedforatopicandfocus,readsearchresults,andmade
relevancyjudgmentsoutperformedtheirpeers.Onthemorerestrictedtasksstudentswho
188
successfullysearchedwithinwebsitesoutperformedtheirpeersintherestrictedinformationspace.
InfactthemostproficientusersnavigateddirectlytothesourcebyenteringtheURLintothe
navigationbar.Whenstudentsdidusekeywordsthosestudentswhoincludedthetopicandsource
outperformedtheirpeers.Finallystudentswhomademorerelevancyjudgmentsdidwellonthe
ORCAmorerestrictedlocatingitems.
Thestagetworesultshelptohighlightmanyofthequantitativefindingsofthisstudy.Most
importantlytheymayshedlightonthedifferencesintheoverallmeanscoresoftheORCA‐LR
(M=8.394SD=4.117)andscoresoftheORCA‐MR(M=7.15SD=4.48).Theresultsofthethinkaloud
studyindicatethatitwasmoredifficultforstudentstosearchforaspecificsourceratherthan
informationonagiventopic.
Thisstudyalsohelpedtoidentifystrategiesthatareusedbymoreproficientstudentsas
theyusetheInternettolearn.Thisstudytookplaceinthedisciplineofsocialstudies.Thestrategies
thatwereobservedinthemoreproficientstudentsinclude:
Locating strategies on a less restricted task
o using keyword that included both the topic and focus;
o copying and pasting keywords that include both the topic and focus;
o actively reading search results;
o navigating back to search results;
o skimming websites to make accurate relevancy judgment.
Locating strategies on a more restricted task
navigating directly to a source;
searching for a specific source;
searching within a website;
using keywords with topic and source;
making accurate relevancy judgments. utilizing tabs to navigate between task and source;
189
Strategies regardless of the restricted nature of the information space
o using tabs to navigate between multiple sources;
o using a strategy of skimming websites to identify key details;
o engaged reading within a website;
o navigating to a source to read while taking a position;
o navigating to the source to copy and paste details;
o identifying the author on an about us page;
o evaluating author using effective markers of expertise;
o checking evidence against a secondary source;
o inferring an author’s point of view;
o recalling details from memory when combining sources;
o recalling details from memory when taking a position.
190
ChapterVI
DISCUSSIONOFTHEFINDINGS
Introduction
Thepurposeofthismixedmethodsstudywastoinvestigatetherolethatbackground
knowledge,criticalevaluationofinformation,andareader’sdispositionsplayinpredictingonline
readingcomprehensionperformance,duringcomprehensiontasksthattakeplaceineitherless
restrictedormorerestrictedinformationspaces.Specificallythisstudysoughttoanswerfour
questions:
1. When predicting online reading comprehension during a problem solving task within
a less restricted information space, what is the best fit model, after controlling for
verbal intelligence, of the percentage of variance accounted for by each of the
following variables: background knowledge, critical evaluation, and dispositions of
online reading comprehension?
2. When predicting online reading comprehension of a problem solving task within a
more restricted information space, what is the best fit model, after controlling for
verbal intelligence, of the percentage of variance accounted for by each of the
following variables: background knowledge, critical evaluation, and dispositions of
online reading comprehension.
3. What patterns of processing appear among readers with varying levels of online
reading comprehension ability during an online reading comprehension task within a
less restricted information space?
4. What patterns of processing appear among readers with varying levels of online
reading comprehension ability during an online reading comprehension task within a
more restricted information space?
191
Quantitativemethodswereusedtoaddressthefirsttworesearchquestions.Hierarchical
regression,controllingforverbalintelligence,wasusedtotesttwodifferentmodels.Thefirstmodel
includedalessrestrictedinformationspace.Thesecondmodelinvolvedamorerestrictedspace.In
bothcasesscoresonabackgroundknowledgemeasure,acriticalevaluationmeasure,anda
dispositionmeasurewereusedtopredictscoresoneitheranonlinereadingcomprehension
assessmentinalessrestrictedinformationspaceandanonlinereadingcomprehensionassessment
inamorerestrictedinformationspace.
Qualitativemethodswereusedtoanswerthesecondtworesearchquestions.Verbal
protocolanalysisusingabductive(Onwuegbuzie&Leech,2006)codingmethodsfollowedbya
constant‐comparative(Bogdan&Biklen,2003;Merriam,1988)analysiswereused.Thegoalofthe
qualitativecomponentwastoanalyzepatternsofprocessingamongstudentswithvaryingdegrees
ofonlinereadingcomprehensionability.Tenparticipantseachcompletedtwoonlinereading
comprehensionassessmenttasks:alessrestrictedtaskandamorerestrictedtask.Screencasts
weremadeoftheactivitiesandpatternsofstrategyusewereidentifiedusingconstantcomparative
methods.
Theuseofthismixedmethodapproachallowedmetounderstandnotonlytheunique
contributionsthevariablesofinterestmadetothemodelbutalsohowthesevariablesmightlookin
classroomcontexts.Byexaminingboththequantitativeandqualitativefindingstheimplications
forresearchandclassroompracticeemerge.
DiscussionoftheQuantitativeResults
OnlineReadingComprehensionAssessment:DescriptiveStatistics
Thedescriptivestatisticsrevealimportantinsightsintotheresultsofthisstudy.Asnoted
statisticaltestingfoundsignificantdifferencesbetweenthemeanscoresofthetwoORCAformats
(t=4.088,p=.000).Thuswouldindicatethattherewasasignificantdifferenceinstudentscoreson
192
theORCA‐LRandtheORCA‐MRwithscoresonORCA‐MRbeingsignificantlylower.Aoneway
repeatedmeasuresanalysisofvariancewasconductedtocomparestudents’scoresineachofthe
fourskillareasontheORCA‐LRandtheORCA‐MRrespectively.Inbothmodels,themeanscores
weresignificantlydifferent.Followuppost‐hoctestsfound,forboththeORCA‐LRandtheORCA‐
MR,thatlocationitemsweretheeasiestitems,followedbysynthesisitems,thencommunicate
items,andfinallyevaluateitems.
Theseresultssomewhatreplicaterecentstudiesthathaveevaluatedpatternsinthescores
ofLocate,Evaluate,Synthesize,andCommunicateitems.ForzaniandBurlingame(2012),for
example,whenanalyzingarelatedORCAassessmentfoundthatsynthesiswastheeasiestskillarea,
followedbylocate,thencommunicateitemsand,finally,byevaluateitems.Bothofthesestudies
foundthatcommunicateitemsandevaluateitemswerethehardestandhadtheloweststudent
scores.
Thesefindingshaveimportantimplicationsforinstructionandresearchinonlinereading
comprehension.Intermsofclassroompracticeitappearsthatstudentsstruggletocommunicate
whattheylearnandtoevaluatethesourcestheyreadonline.Thisisacriticalissueforschoolsas
thesearebothareasemphasizedintheCommonCoreStateStandards(CCSS,2010).Futurestudies
shouldbeconductedtoseeifsimilarpatternsofstudentscoresarereplicated.Iftheyare,
instructionalstudiesshouldbeconductedtoevaluatehowbesttoteachtheseareaswherestudents
performlowest.
SummaryoftheLessRestrictedModel:ResearchQuestionOne
Researchquestiononeasked:
Whenpredictingonlinereadingcomprehensionduringaproblemsolving
taskwithinalessrestrictedinformationspace,whatisthebestfitmodel,
aftercontrollingforverbalintelligence,ofthepercentageofvariance
193
accountedforbyeachofthefollowingvariables:backgroundknowledge,
criticalevaluation,anddispositionsofonlinereadingcomprehension?
Resultsofthelinearregressionindicatedfirst,thatverbalability,asmeasuredbya
vocabularytest,explained16.9%ofthevariance,whichwassignificant,F(1,93)=20.530,p<.001.In
thefullmodel,scoresonacriticalevaluationmeasureexplainedasignificantamountofunique
variance(β=.335p<.000).Backgroundknowledgealsocontributedsignificantlytotheprediction
ofscoresontheunrestrictedORCA(β=.211p=.021).Dispositionsofonlinereading
comprehensiondidnotcontributesignificantlytopredictionsinthemodel(β=.117p=.162).The
totalexplanationhadalargeeffect,Cohen’sF=.358.
Inthelessrestrictedmodelscoresonthecriticalevaluationmeasurehadthelargest
predictiveweight.ThisfindingisconsistentwithbothCognitiveFlexibilityTheory(CFT)thenew
literaciesofonlinereadingcomprehension.ItwaspredictedbasedonCFTthatscoresonthe
criticalevaluationassessmentwouldbeasignificantpredictorinthelessrestrictedmodel.Thisis
duetostudentswith“advancedwebskills”performingbetterinanopenInternetspace.
ThepredictivepowerofCOILscoresalsoalignedwithatheoryofnewliteraciesofonline
readingcomprehension.TheprinciplesofthebroadercapitalletterNewLiteracieshasrecognized
thecentralrolethatcriticalliteraciesplayindigitalenvironments(Leu,Kinzer,Coiro,Castek,&
Henry,2013).Morespecificallythenewliteraciesofonlinereadingcomprehensiontheoristshave
alsolongarguedthatthecriticalevaluationofonlinesourcesisakeydifference(Coiro,2003;Coiro
etal,2008;Leuetal.,2004).
Inthelessrestrictedmodelbackgroundknowledgewasasignificant,butnotthelargest,
predictor.ThisresultisalsocongruentwiththepredictionsmadebasedonbothCognitive
FlexibilityTheoryandatheoryofnewliteraciesofonlinereadingcomprehension.Thesetheories
havesuggestedthattheroleofbackgroundknowledgeinpredictingreadingcomprehension
changesinonlineinformationcontexts.Whilebackgroundknowledgehasbeenoneofthemost
194
stablefindingsofcomprehensionresearchthisstudymayaddcredencetotheclaimthatwemust
rethinktheroleofbackgroundknowledgewhendevelopingtheoreticalmodelsofonlinereading
comprehension.
Inthelessrestrictedmodelscoresonaself‐reportmeasureofdispositionsofonlinereading
comprehensionwerenotasignificantpredictor.Thisresultdidnotalignwiththepredictionsmade
fromeithertheoreticalperspective.CognitiveFlexibilityTheoryaguesthatstudentswith“open
mindsets”arebetterpreparedtoreadintheill‐structuredspaceoftheInternet.Newliteraciesof
onlinereadingcomprehensionarguesthatnewdispositionsarecentraltomakingmeaningonline.
Yetinthelessrestrictedmodelascoreonadispositionsmeasurewasnotasignificantpredictor.
SummaryoftheMoreRestrictedModeltoAddressResearchQuestionTwo
Researchquestiontwoasked:
Whenpredictingonlinereadingcomprehensionofaproblemsolvingtask
withinamorerestrictedinformationspace,whatisthebestfitmodel,after
controllingforverbalintelligence,ofthepercentageofvarianceaccounted
forbyeachofthefollowingvariables:backgroundknowledge,critical
evaluation,anddispositionsofonlinereadingcomprehension.
Resultsforthelinearregressionindicatethataftercontrollingforverbalintelligence,the
threepredictorvariablesofbackgroundknowledge,evaluationskills,andscoresonaself‐report
measureofdispositionsofonlinereadingcomprehensionexplainedanadditional38.1%ofthe
varianceonthescoresofthemorerestrictedORCA,whichwassignificantF(1,92)=26.328p<.000.
Inthefullmodelscoresonacriticalevaluationmeasureexplainedasignificantamountofunique
variance(β=.499p<.000).Backgroundknowledgealsocontributedsignificantlytotheprediction
ofscoresontheunrestrictedORCA(β=.189p=.017).Dispositionsalsomadeasignificant
predictiontothemodel(β=.291p=.006).Thistotalexplanationhadalargeeffect,Cohen’sF=.631.
195
Inthemorerestrictedmodel,similartothelessrestrictedmodel,scoresontheCOIL,a
measureofcriticalevaluationskills,wasasignificantpredictorwiththelargestbetaweightinthe
model.ThisfindingmirroredthepredictionsthatweremadebasedonbothCFTandatheoryof
newliteraciesofonlinereadingcomprehension.Thiswouldonceagainsuggesttheabilityto
flexiblyapplynewstrategiesinavarietyofsituationsisacentralrequirementforsuccesswhen
makingmeaninginonlinespaces.
Inthemorerestrictedmodelbackgroundknowledgewasasignificant,butnotthelargest,
predictorofperformanceontheORCA‐MR.Thisresultdidnotalignwiththepredictionmadebased
ontheprinciplesofCognitiveFlexibilityTheory.Itwaspredictedthatbackgroundknowledge
wouldbesignificantinbothmodels.Howeveritwasbelievedthatbackgroundknowledgewouldbe
thestrongestpredictorinthemorerestrictedspace,aslearnerswouldnotbenefitfrom“flexible
schemaassembly”and“serendipitous”learningwhilelookingforspecificsources.Inotherwords
becausestudentscouldnotbuilduptheirknowledgewhileconductingopenInternetsearches
thosewithhigherbackgroundknowledgewouldhaveagreaterchanceofsuccesslookingfor
specificsources.Yetinthemorerestrictedmodelitwascriticalevaluationandnotbackground
knowledgethatwasthestrongestpredictor.
Inthemorerestrictedmodel,scoresonaself‐reportmeasureofdispositionsofonline
readingcomprehensionwereasignificantpredictorofperformanceontheORCA‐MR.Thisdiffered
fromtheresultsoftheORCA‐LRmodel.Thisfindingalignedwithpredictionsmadebasedon
CognitiveFlexibilityTheoryandatheoryofnewliteraciesofonlinereadingcomprehension.InCFT
itisarguedthatreaderswith“openmindsets”willfairbetterinonlinespaces(Spiro&Deschryver,
2010).Newliteraciesofonlinereadingcomprehensiontheoristsalsoarguethatnewdispositions
arerequiredtomakemeaningwhenreadingonline(Leuetal.,2004).
ExploringTheImplicationofBothModels
196
Theoreticalmodelsofreadingcomprehension.Theresultsofthesetwomodelsindicate
theneedtoexpandourmodelsofreadingcomprehensiontoaccountfortheriseofonlinetexts.
Mosttheoreticalmodelsofreadingaredrawnonprinciplesofstaticandconsistentconstructs.For
exampletheheuristicofreader,text,andactivity(Snow,2002)reliesonwell‐foundedresults
establishedthroughdecadesofcomprehensionresearch.Yetthisstudy,andsimilarwork,callsinto
questionstaticmodelsofreadingcomprehension.
Overthepastthreedecadesresearchershaveidentifiedstablefactorstoincludeinmodels
ofreadingcomprehension(Snow,2002).Yetinadigitalagewhennewtextsandtoolsforliteracy
emergeeveryday(Coiroetal.,2008)researcherscannolonersolelyrelyonstablefindings.Asnew
textandliteracypracticesemergewiththeshiftfrompagetopixel(Hartmanetal.,2010)new
variablesofinterestwillemergeandconfoundourpreviousmodels.Thesefactors,forexample,
couldincludethecentralityofevaluationandnavigationasnotedinthisstudy.Theywillalso
includevariablesyettobeidentifiedastheInternetcontinuestoevolve.Wecannolongerrelyon
staticmodelsofreadingcomprehension.
Insteadweneedtocontinuetodevelopdynamicmodelsofreadingcomprehension
(Hartmanetal.,2010)thatcanaccountforconstantchange(Leu,2000).Thisstudyhelpsto
illuminatesomepossibleelementsofdynamicmodelsofreadingcomprehension.Firstinthese
modelstheevaluationofmultiplesourcesasafluidtextmustbeatthecenterofreading.Alsoin
thesenewtheoreticalmodelsthereadermayrelyonactivelyconstructingknowledgeandtextsin
themoment(DeschryverandSpiro)ratherthansolelyrelyingonbackgroundknowledge.Finally
thesemodelsmayhavetoaccountforepistemologicalprocessesandhabitsofmindinorderto
accountfornewdispositions.
Criticalevaluation.Themostcompellingfindingfromthequantitativefindingsofthis
studyisthecentralroleofcriticalevaluation.Aftercontrollingforverbalcomprehension,scoreson
theCOILassessmentwerethebestpredictors(β=.429)ofscoresonthelessrestrictedinformation
197
spaceandalsoonthemorerestrictedmodel(β=.499).Thefactthatcriticalevaluation,not
backgroundknowledge,wasthebestpredictorofonlinereadingcomprehensionmaylend
additionalsupporttostudiessuggestingthatonlinereadingcomprehensionisdifferentfromoffline
readingcomprehension.Thisfindingaddsweighttorecentresearch(Braten,Stromso,&
Samuelstuen,200;Coiro,2011;Goldman,Braasch,Wiley,Graesser,&Brodowinska,
2012).Goldmanetal.,(2012)usingathinkaloudmethodamongadultsfoundthatexpertreaders
usedmoreevaluativestrategieswhencomparedtonovices.Coiro(2011)foundthatafter
controllingforbackgroundknowledgeandofflinereadingabilitythatknowingastudent’sonline
readingcomprhensionskillpredictedadditionalvarianceinscoresonanonlinereading
comprehensionassessement.Herresultsindicatethatnewknowledge,skills,anddispositionsare
involvedinonlinereadingcomprehensionassessment.Thisstudysuggeststhat,inaddition,the
relativecontributionofcriticalevaluationskillsmaybegreaterinonlinereadingcomprehension,
comparedtoofflinereadingcomprehension.
Braten,Stromso,andSamuelstuen(2008)foundthatknowingastudents’sourcingskills
bestpredictedtheirabilitywithmultiplesourcecomprehensioninofflinereadingcontexts.
Similarlythisstudyfoundthatknowingstudents’criticalevaluationskillsbestpredictstheironline
readingcomprehensionability.Sinceonlinereadingcomprehensiontypicallyinvolvesmultiple
sourcereading,thismayindicatethatmultiplesourcereadingisanimportantdifferencebetween
offlineandonlinereadingcomprehension.Furthermoreitmayindicatethatevaluatingsourcesis
centraltobothmultiplesourcereadingofflineandonlinereadingcomprehension.However,given
thatmultiplesourcereadingofflineusuallyinvolvespre‐vettedsourcessuchasnewspapersand
journals,additionalevaluationskillsmayberequiredinonlineenvironments.
AssessingCriticalEvaluation.Perhapsoneofthegreatestchallengesthatliteracy
researchersfaceisthefurtherrefinementanddevelopmentofmeasurestoevaluatecritical
evaluationskills.Thefinal12itemCriticalOnlineInformationLiteraciesassessment(COIL)hada
198
coefficientalphaof.71.Whilethislevelofareliabilityestimateisacceptableonemighthavegreater
confidenceinthemeasurehaditbeenhigher.
Researchmustalsobeconductedtounderstandhowitemformateffectsitemdifficulty.The
COILcontainedforcedresponseanswersthatincludedentirescreenshotsofwebsites.Thisviolates
basicprinciplesofcomprehensionassessmentthatcallforshortdistractors(Fuchs,Fuchs,&
Maxwell,1988;Keenan&Betjemann,2008).ItemsintheCOILwerealsopresentedindifferent
formats.Someitemscontainedsinglescreenshots,otheritemscontainedfourscreenshots,and
finallyitemscouldhavehadfourhyperlinkstorecreationsofactualwebsites.Thisitemformatmay
haveinfluenceditemloadings.Futurestudiesshouldbeconductedusingitemresponsetheory
(Baker,2001)totestifdistractorlengthanditemformatinfluencesitemdifficulty.
Backgroundknowledge.Basedonthelinearregressionmodelsinthisstudy,background
knowledgewasasignificantpredictorofscoresonthelessrestrictedORCA(β=.241).Itwasalsoa
significantpredictorofscoresonthemorerestrictedORCA(β=.189).Backgroundknowledgeisone
ofthemostwidelyacceptedpredictorsofreadingcomprehension.Ithasbeendemonstratedthat
backgroundknowledgeaffectscomprehension(Anderson&Pearson,1984;Kintsch,Patel,&
Ericson,1999;Voss,Fincher,Keifer,Green,Post,1985).Theoverarchingtheoryisthatgoodreaders
relatetheirbackgroundknowledgetothetextandthetask.
Thisstudymaycallintoquestionthelong‐standingdominanceofbackgroundknowledgein
comprehension.Ithasbeenargued,whenexaminingprintbasedcomprehensionthatknowledgeis
comprehension(Willingham,2010).Yetwhenexaminingtheresultsofthisstudyitisclearthat
newknowledge,skills,andstrategiesmaybejustas,orevenmoreimportant,thanbackground
knowledge.Thisfindingcallsintoquestionmuchofourlong‐standingbeliefsabouttheroleor
backgroundknowledge.
Infactthereisongoingdebateastotherolebackgroundknowledgewillplayinonline
readingcomprehension.TheresultsofthisstudyparallelthosesuchasHillandHannifin(1997)
199
andCoiro(2012)whothatfoundbackgroundknowledgepredictedperformanceinonlinereading
comprehensiontasks.However,Bilal(2000;2001)foundthatbackgroundknowledgedidnotplaya
significantroleindeterminingsuccesswhenconductingInternetinquires.Thisstudyadds
evidencetothedebate.
Mostoftheresearchontheroleofbackgroundknowledgeassumesacognitivestructure
basedontheprinciplesofinformationprocessingtheories(Kirschner,Sweller,&Clark,2006).This
includesacognitivearchitecturewithelementsoflong‐termmemory,short‐termmemory,and
workingmemory.Basedonthistheorybackgroundknowledgeisconsideredtobeanelementof
long‐termmemory.YettheresultsofthisstudyandrecentfindingsbyBilal(2000;2001)andCoiro
(2012)maycallintoquestionthesetheoreticalmodels.Insteadofrecallingdetailsfromlong‐term
memorystorageonlinereadersmaybecreatingwhatCFTtheoristshavelabeled“schemaatthe
moment”whichiscreatedthrough“flexibleassemblage”ofinformationencounteredonlineinreal‐
time.
Theseobservationsmayalignbetterwithsituatedcognitionmodels,whichsuggestthat
knowledgeexists“insitu,inseparablefromcontext,activity,people,culture,andlanguage.”
(McVerry,2010,para.1)Theliteracycommunitymaybenefitfromalineofresearchintotheroleof
backgroundknowledgewithresearchdesignsthatdrawonprinciplesofsituatedcognition
(Greeno,1989)aswellastraditionalinformationprocessingtheories(Kirschner,Sweller,&Clark,
2006;LaBerge,&Samuels,1974).
Futurestudiesshouldalsoincludemeasuresofsystemsknowledgeandnotsimplycontent
knowledge.Oneofthekeydifferencesbetweenstudiesthatfoundaroleforbackgroundknowledge
(Coiro,2012;Hill&Hannafin,1997)andthosethatdidnot(Bilal,2000;2001)wastheinclusionof
itemstomeasuresystemsknowledgeabouttheInternet.Thisstudyonlyincludedameasureof
topicalknowledgeontheAmericanRevolution.Followupstudiesmightincludebackground
knowledgescoresfromabasicnavigationtest.
200
Dispositionsofonlinereadingcomprehension.Thefactthatscoresontheself‐report
measureofdispositionsofonlinereadingcomprehensionwerenotasignificantpredictorinthe
ORCAlessrestrictedmodel(β=.124)butwereasignificantpredictorinthemorerestrictedmodel
(β=.206)isaveryinterestingoutcome.Thesefindingsmayindicatethattheinformationspacehas
aninfluenceduringonlinereading.Furthermore,theresultsmaysuggestthatmoretargetedand
specializedsearches,whichutilizemorerestrictedspaces,requirereaderswhocanadapttonew
situations.
ThebivariatecorrelationsoftheDORCwereveryinformativeinexploringthisfinding.Of
mostinterestwasthenegativecorrelationoftheitemsonthecollaborationsubscale.Collaboration
hadnocorrelationwiththelessrestrictedORCAandanalmostnon‐existentandinsignificant
correlationwiththemorerestrictedORCA.
Yetatthesametimerecentworkhashighlightedhowimportantcollaborationisduringin
onlinereadingcomprehension(Coiro,Castek,andGuzniczak,2011;Kiili,Laurinen,Marttunen,&
Leu,2011;O’Byrne,2011;Zawilinski2011).Ineachofthesestudies,collaborationimproved
studentperformancewhenconductingInternetinquiryactivities.Kiilietal.(2011)foundthat
collaborativegroupsreadingonlinecomparedmultipleperspectivesandsoughtdeepermeaning
thanindividualsreadingonline.O’Byrnefoundthatstudentsworkingingroupstocreatespoof
websitesimprovedtheirevaluationskills.FinallyZawilinskifoundthatfirstgradersandfifth
gradersbloggingtogetherimprovedcommunicationskills.Theseresultshaveleadresearchers
(Coiro,Castek,&Guzniczak,2011;Killietal.,2011)toarguethatwemustviewonlinereading
comprehensionasacollaborativesocialpractice.
Itisclearbasedonthesestudiesthatopennesstocollaborationisacentraldispositionto
onlinereadingcomprehension.YetinthisstudythecollaborationelementsoftheDORCwerenot
significantlycorrelatedwithperformanceontheORCA.Thismaybeanartifactofthesolitary
201
natureofthetestingenvironment.Inotherwordstheassessmentdesigndidnotallowstudentsto
takeadvantageoftheaffordancesthatawillingnesstocollaborateprovides.
Futureworkshouldalsobeconductedtofurtherrefinemeasuresofdispositionsforonline
readingcomprehension.Thisstudyusedapreviouslyvalidatedinstrumentthatincludedfive
subscales:reflectivethinking,criticalstance,persistence,flexibilityandcollaboration.Otherwork
beingconducted(Putman,inpress)hasexaminedmotivation,self‐efficacy,valueandanxiety.
Furthermoresomeresearchershavecritiquedself‐reportmeasuresofdispositions(Carr&
Claxton,2002).Themostpromisingdirectionforthefieldmaybenotininstrumentationbutin
facialrecognitionsoftware(D’Mello&Graesser,2010).Thisresearchcancapturerealtime
indicatorsofdispositionsinplaybycapturingsubtledifferenceinfacialexpression,whichcan
recognizepersistence,frustration,andsuccess.
ResearchshouldalsobeconductedthatmeasuresthepredictivevalueoftheDORCwhen
studentsarecompletingonlinereadingcomprehensiontasksinpairsoringroups.Giventhat
recentworkhasfoundsuchastrongroleforcollaborationisassistingonlinereading
comprehensionitwouldbebeneficialtothereadingresearchcommunitytounderstandhow
collaborationchangescomprehension.Weneedtoidentifytheaffectivevariablesthatpredict
successincollaborativeenvironments.
Summaryofthequantitativeimplications.Thisstudyexaminedmodelsofonline
readingcomprehension.Specificallyitexaminedtherelativecontributionsthatcriticalevaluation,
backgroundknowledge,anddispositionsmaketoonlinereadingcomprehensionaftercontrolling
forverbalintelligence.Thismodelwastestedinbothalessrestrictedinformationspace,where
studentscouldsearchforanywebsite,andamorerestrictedinformation,wherestudentshadto
findspecificwebsites.Inbothmodelscriticalevaluationscoreswerethelargestpredictorsof
performance.Backgroundknowledgewasalsoasignificantpredictorinbothmodels.Thescoreon
thedispositionmeasurewasonlyasignificantpredictorinthemorerestrictedmodel.
202
Theseresultsindicatethatresearchersandteachersmightmorecarefullyconsiderthe
differentspacesinwhichonlinereadingcomprehensiontakesplace.Wewillneedanincreased
effortinunderstandinghowtheinformationfieldinfluencestheperformanceofstudentsduring
onlinereadingcomprehension.Wealsoneedtodevelopmodelsofinstructionthatprepare
studentsforthe“openmindsets”and“advancedwebskills”necessaryforonlinereading
comprehension(Deschryver&Spiro,2010,p.4).
Afterall,theCommonCoreStateStandardscallforonlineresearchandmediaskillstobe
embeddedinallcontentareasandacrossallclassrooms.Theresultsofthisstudyindicatethata
large‐scaleeffortmustbetakentopreparestudentsforonlinereadingcomprehension.Basedon
themeanORCAscoresandtheCOILscoresitisclearthatstudentsareillpreparedtomake
meaninginonlinespaces.Theresearchcommunityandeducatorsalikemustworktogetherto
developnewinstructionalroutinestoaddressthischallenge.
DiscussionoftheQualitativeResults
Thepurposeofthequalitativestudywastoinvestigatedifferencesofstrategyuseamong
studentswithvaryingdegreesofonlinereadingcomprehensionproficiency.Tenstudentswere
includedinthefinalanalysis.EachparticipantcompletedalessrestrictedORCAandamore
restrictedORCAtask.Usingverbalprotocolanalysis(Afflerbach,2002)andabductivecoding
methodsfollowedbyconstantcomparativemethods(Bogdan&Biklen,2003;Merriam,1988),this
studysoughttoanswerthethirdandfourthresearchquestions:
3. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a less
restricted information space?
4. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a more
restricted information space?
203
Toanswerthesequestionsatwo‐stageanalysiswasused.Inthefirststage,patternsof
processingwereidentifiedthatimprovedoverallperformanceontheORCAassessmentsregardless
ofthenatureoftheinformationspace.Inthesecondstageofanalysis,patternsofprocessingwere
identifiedthatchangedbasedonthenatureoftheinformationspace.Bothstagesofanalysishelped
toilluminatethequantitativefindings.
QualitativeResults:DiscussionofStageOneFindings
Stageoneanalysisidentifiedprocessingpatternsthatledtoincreasesinoverall
performanceregardlessoftheinformationspace.Fourthemeswereidentifiedinthisstageof
analysis:(a)overallORCAscoresappearedtoberelatedtostrategictextassembly;(b)overall
ORCAscoresindicatedalackofsourceevaluationknowledgeamongparticipants;(c)studentswho
recalleddetailsfrommemoryappearedtoperformbetterontheORCAtasks;and(d)students
appearedunpreparedtoengageinthecommunicationstrategiesnecessaryforonlinereading
comprehensioninacademicsettings.
OverallORCAscoresappearedtoberelatedtostrategictextassembly.Thefirsttheme
instageoneanalysishighlightedhowimportantnavigationalandcomprehensionmonitoringskills
wereindeterminingsuccessatonlinereadingcomprehension.Students,whonavigatedbetween
sources,returnedtothetaskandalteredtheircursorspeedwhilereadingoutperformedtheirpeers
whodotheselessfrequently.
Thisfindingisconsistentwithpreviousresearchthathasfoundreadingonlinerequires
similarbutmorecomplexreadingskills(Coiro&Dobler,2007;Goldmanetal.,2012).Oneofthese
newcomplexitiesmaybeintheabilitytonavigateinformationalspaces.InfactGoldmanetal.,
usingthinkaloudprocedurestocomparecollegeageexpertandnovicereadersfoundnavigationto
beacrucialdifference.Similartothisstudy,Goldmanetal.foundthatexpertonlinereadersmade
significantlymorenavigationalchoicesbacktoreliablesourcesandawayfromunreliablesources.
McWilliamsandClinton(2010)arguedthat“readingwithmouseinhand”isafundamental
204
differenceinonlinespaces.O’Hanlon(2002)arguesthatstudentsentercollegewithoutbasic
navigationalabilitiestoconductonlineresearch.Ifstudentsaretograduatehighschoolwiththe
onlineresearchandmediaskillstobecollegeandcareerready(CCSS,2010)theneffortsmustbe
undertakentoincreasethefullrangeofnavigationalskills.
Comprehensionmonitoringwasdefinedinthisstudyasalsoknowingwhentoskimand
scansourcesandwhentoslowdownforengagedreading.Inthisstudytheexpertreadersused
strategiestoskimsourcesandusedstrategiesofengagedreadingwithagreaterfrequencythan
theirpeers.ThisissimilartotheresultsfoundGoldmanetal.(2012)whereexpertundergraduate
readersspentconsiderablemoretimereadingreliablesourcesandlesstimereadingunreliable
sourcesthantheirpeers.
Thisfirsttheme,thatperformanceontheORCArequiresstrategictextassembly,mayalso
informthequantitativefindingsofthisstudy.Inboththelessrestrictedandthemorerestricted
modelstudents’scoresonthecriticalevaluationmeasurewerethestrongestpredictorof
performanceontheORCA‐LRandORCA‐MR.Thiscouldbeduetothenumberofnavigational
choicesrequiredintheCOILinstrument.TheCOILconsistedoffourdifferentitemtypes:astatic
screenshotofasinglewebpage,astaticimageoffourscreenshots,awebpagewithhyperlinks,and
listoffourhyperlinks.Thusstudentswhohaveahigherabilitytoreadwithcursorcontrolmay
havesucceededontheCOILaswellastheORCA.
Animportantcontributionthatemergedfromthemeoneishighlightingtheimportanceof
navigationalskillsacrossallelementsofonlinereadingcomprehension.Thisstudyfoundthat
studentswhomovebetweenthetaskandthesource,especiallytheparticipantswhousedtabbed
browsing,outperformedtheirpeersontheORCAtasks.Thisstudyalsofoundthatexpertstudents
movedbetweensourceswithagreaterfrequencythantheirnovicepeers.Thesestudentsmadeuse
ofnavigationalstrategiesthroughouttheORCAassessmentbutpatternsespeciallyemergedduring
thesynthesisandcommunicationtasks.
205
Manypreviousstudieshaveoperationalizednavigationalstrategiesaspartofsearchengine
use(Bilal,2000;2001;HillandHannifin,1997).Yetthisstudy,inalignmentwiththeworkby
Goldmanandherpeers,foundthatnavigationalstrategiesarejustasimportantacrossallelements
ofonlinereadingcomprehension.
Implications.Theroleofnavigationalabilityfoundinthisstudymayhaveimportant
consequencesforclassroompractice.Firstofallthenextgenerationofhighstakestestingwillbe
deliveredelectronically.Thetestsaretobecompatiblewithdesktop,laptop,andtabletcomputing
(SBAC,2012;PARCC,2012).Giventhatthisstudyfoundthattheabilitytonavigateaninformation
spaceiscriticaltoperformanceteacherswillneedtoensurethatstudentsarepreparedforthis
typeoftestingenvironment.Itisalsoclearthateducatorsneedtoincreasetheirinstructioninthe
useofmultipleonlinesourcesinclassroom.Basedonthefindingsofthisthememuchofthis
instructionshouldfocusongoaldirectednavigationalstrategies.
OverallORCAscoresappearedtoindicatealackofsourceevaluationknowledge
amongparticipants.Thesecondthemeindicatedthatthemajorityofstudentssimplydonot
evaluatesourcestheyencounteronlineandiftheydo,theyrelyonverysuperficialmarkersof
relevancyandreliability.Thefactthatveryfewdifferencesinpatternsofprocessingwereidentified
amongstudentsbasedontheirproficiencylevelisstriking.Inrealityfewstudentshadtheabilityto
evaluateauthor’sexpertise,evidenceusedbytheauthor,andauthor’spointofview.Whenstudents
didtrytodothese,theyoftenmadeerrorsofoversimplification.
Thisthemereflectsmanyoftherecentfindingsofresearchinonlinereading
comprehension.Braaschetal,(2009)foundthatmiddleschoolstudentsrarelyevaluatedthe
usefulnessofsources.Leuetal.(2007)foundstudentshadaninabilitytoidentifyspoofsites.
Goldmanetal.(2012)alsofoundnosignificantdifferenceinthewebsiterankingsofreliability
amongexpertandnoviceundergraduatereaders.Coiro(2012)usingacasestudyanalysisofthree
206
students,foundthatthestudentseitherlackedcriticalevaluationskillsorreliedonsurfacelevel
informationtojudgesources.
Thisthemealsohelpedtoinformthequantitativefindingsofthisstudy.Itisconsistentwith
outcomesonboththeCOILassessmentandtheORCAassessments.IntermsoftheCOILassessment
thethinkalouddataillustrateswhymeanscoreswerelow.Itisclearfromthesetenparticipants
thatsofewcoulduseprocessesforsourceevaluationcorrectly.Infactonlytwostudentswas
successfullyabletojudgeanauthor’spointofview.This,addsadditionalevidenceforthevalidity
ofthismeasure.Thequalitativestudyalsohelpstoexplainwhytheevaluateitemswerethemost
difficultforstudentsinboththeORCA‐LRandtheORCA‐MR.
Implications.Giventhatstudentsrarelyutilizedcriticalevaluationstrategies,andwhen
studentsdidusethesestrategiestheyoftenmadeerrorsofoversimplification,itisclearthat
currentclassroompracticestoteachsourceevaluationmaybeinsufficient.Thequalitativeportion
ofthisstudyindicatedthattheseerrorsofoversimplificationmayhelptoexplaincommonresults
instudiesthatusecriticalevaluationmeasures.Inthisstudyandinsimilarresearchinvestigating
criticalevaluation,studentsoftenreliedonsurfacelevelfeaturestojudgesources.
Studies(Goldmanetal.,2012;Sanchezetal.,2006;Zhang&Duke,2007),forexample,that
usewebsiterankingstocomparenoviceandexpertreadersfindnosignificantdifferenceintheir
evaluationability(Goldmanetal.,2012).FurthermoreZhang&Duke(2007)andSanchezetal.
(2006)foundthatstudents’performanceatrankingwebsitesdidnotsignificantlyimprove,
followinganintervention,butstudentwrittenjustificationsdidsignificantlyimprove.Theseresults
mirrorthelackofevaluationskillsnotedinthisstudy.
Acrossallofthestudiesthateitherusemeasuresofcriticalevaluation(Goldmanetal,
2012;Sanchezetal.,2006;Wiley,2009;Zhang&Duke,2007;)orverbalprotocolstudies(Coiro,
2011;Coiro&Dobler,2007)studentsmadeerrorsofoversimplificationbyrelyingonsurfacelevel
207
textfeatures.Thepatternsinthisthemesupportthisconclusion.Inthisstudystudentsalsooverly
reliedonwebsitecontentandwebsitedesigntojudgeonlinesources.
Giventherelativestabilityofstudentsoverlyrelyingontextualfeaturestojudgewebsites
therehavebeencallstomoveawayfromatop‐downtaxonomyapproachofteachingcritical
evaluationskills(Goldmanetal.,2012).Insteadresearcherscallfortheoreticalandinstructional
modelsthatviewsourceevaluationasmoreintegraltothemeaningmakingprocess(Goldmanet
al.,2012).Onealternativetotopdowntaxonomieswouldbetodrawontheprinciplesof
instructionoutlinedinCognitiveFlexibilityTheory(Spiro,2004).Thiscallsformultiple
representationsofmaterialindiversecases.Suchanapproachwouldalignwellwiththetheoretical
modeloutlinedbyGoldmanetal.(2012).
StudentswhorecalldetailsfrommemoryappeartoperformbetterontheORCA
tasks.Thethirdthemeinstageoneanalysisindicatedthatstudentswhorecalleddetailsofwhat
theyreadduringtheORCAtasksoutperformedtheirpeers.Thesepatternswereespeciallyevident
whenstudentscombinedinformationfromwhattheyread.Thisthemewasalsoobservedwhen
studentstookapositiononthefinalinquirytask.
Thisresultreflectsrecentworkexaminingtheroleofshorttermandworkingmemory
duringInternetinquiry.LabergeandScafalia(2013)foundworkingmemorycapacitytobeastrong
predictorofperformanceinadultsengagedinreadingasinglewebsite.Herder&Juvina,(2004)
foundthatweaknavigationalchoiceswereassociatedwithlowerscoresonmeasuresofworking
memory.Withsimilarfindingsthisstudyfoundthatstudentswhoareabletorecallmoredetails
aboutthesourcestheyreadmayperformbetteratonlinereadingcomprehensiontasks.
Implications.Thisfindingsupportstheideathatresearchshouldcontinuetoinvestigate
therolememoryplaysinonlinereadingcomprehension.Thisstudyexaminedtherolebackground
knowledge,asoperationalizedascontentknowledge,playedinpredictingonlinereading
comprehensionscores.Howevergiventhatthethinkalouddatafoundstudentswhorecalled
208
detailsfrommemoryoutperformedstudentswhodidnot,futureresearchisneededintotheroleof
memory.Specifically,studiesshouldbecreatedthatexaminethecombinedpredictivepowerof
multiplesourcesofbackgroundknowledgesuchascontentknowledgeandsystemsknowledgeof
searchengines.Studiescouldalsoexaminethepredictivecapacityofshort‐termmemorycapacity,
andworkingmemorycapacity.Thiswouldprovideaclearerpictureoftheroleofvariouselements
thatmightcontributetobackgroundknowledgeuseduringonlinereadingcomprehension.
Studentsappearedunpreparedtoengageinthecommunicationstrategiesnecessary
foronlinereadingcomprehensioninacademicsettings.Thefourththemefromthestageone
analysissuggeststhatstudentsareillpreparedtoconveymeaninginformalonlineacademic
settings.Thiswasespeciallytruewhenstudentsdidnotincludetheinformationtheyreadintheir
finalpostsonasimulateddiscussionboard.
Theparticipantsinthisstudyrarelyusedevidencetobackupclaimsandonlyonestudent
includedanyinformationidentifyingasource.Theparticipantsinthisstudylackedthebasic
discoursemovesandargumentative(Tillman,1956)structuresnecessaryforacademicwriting.
Thiswouldindicatethatstudentsneedadditionalinstructionintheuseofargumentativetext
structureinonlinespaces.
Studentsalsodidnotfullyusetheaffordancesavailableinanonlinecommunicationtoolto
improvetheirwriting.Nostudentembeddedhyperlinkstosourcesorimages.Infacttheonly
multimodalfeatureusedwereemoticons,orsmallsmileyfaces.Thisstudycouldindicatethat
students,withoutformalinstruction,maybemorebiasedtocasualwritinginonlinespaces.
Theresultsofthisstudyreflectsimilarresearchintoacademicwritinginonlinespaces.
Whilethereissignificantevidenceofhighqualitycommunicationstrategiesininformalspaces
(Black,2007;Stone,2007)thereisagrowingbodyofevidencetosupportthatstudentsarenot
preparedforacademicdiscourseinonlinecontexts.ForzaniandBurlingame(2012),inevaluating
itemdifficultyonanORCAassessment,foundthatscoresoncommunicationitemswerethesecond
209
lowest,behindevaluationscores.Theirenvironmenttestedbothwikiandemailcommunication
tools.Thisstudyreachedthesameconclusionusingadiscussionboardformat.
Thistheme,alackofcommunicationstrategies,alsoinformsthequantitativefindingsofthis
study.MeanscoresonboththeORCA‐LRandtheORCA‐MRwerelowforthecommunicationitems.
Onlyevaluationscoreswerelower.Thequalitativeinstancesandanexaminationofthestudent
discussionboardpostsputthesescoresintofocus.Studentsaresimplyillpreparedforacademic
writinginonlinespaces.
Implications.Weneedadditionalefforttoincreasetheamountofopportunitiesstudents
havetoutilizeonlinecommunicationtoolsintheirclassrooms.Everystudentshouldhaveaccessto
schoolassigned,child‐safe,email.Everyteacherevaluationplanshouldmeasureateacher’sability
toutilizehybrid‐teachingenvironments.Giventhat70%ofcurrentcollegestudentswilltakean
onlineclassduringtheirsecondaryeducationalcareer(Sloan,2010)itishardtoarguethathigh
schoolscangraduatestudentswhoarecollegeandcareerreadyiftheycannotuseonlinespacesfor
academic,argumentativewriting.
QualitativeResults:DiscussionofStageTwoFindings
Thepurposeofthestagetwoanalysiswastoexamineifthereweredifferencesinpatterns
ofprocessingthatwereaffectedbytheinformationspace.Whereasstageoneanalysisidentified
patternsofprocessingthataffectedoverallperformance,stagetwoanalysisfocusedonresearch
questionsthreeandfour:
3. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a less
restricted information space?
4. What patterns of processing appear among readers with varying levels of online reading
comprehension ability during an online reading comprehension task within a more
restricted information space?
210
Ananalysisofthedataindicatedthattheonlydifferencesinthepatternsofprocessing,basedon
theinformationspace,appearedonthelocatingitems.
Whatpatternsofprocessingappearamongreaderswithvaryinglevelsofonline
readingcomprehensionabilityduringanonlinereadingcomprehensiontaskwithinaless
restrictedinformationspace?Threepatternswereidentifiedinthelessrestrictedinformation
space.FirststudentswhoperformedbetterontheORCA‐LRusedmoreeffectivestrategiesfor
enteringkeywords.AlsostudentswhoperformedbetterontheORCA‐LRusedmoreeffective
strategiesforreadingsearchresults.FinallystudentswhoperformedbetterontheORCA‐LRmade
morerelevancyjudgments.
Thesethreepatternsreflectmuchofthecurrentworkinvestigatingstudentsearchhabits
(Coiro&Dobler2007;Reih,2004;Tabatabai&Shore,2005).Whilekeywordselectionoftenbegins
withgeneralterms,researchhasfoundmoreskillfulreadersrevisekeywordduringthesearch
process(Rieh,2004).FurthermoreTabatabai&Shore(2005)foundthatexpertsusedmore
keywordsthannovices.Coiro&Dobler(2007)foundthatlocatingskillsmightaddtothe
complexityofonlinereading.Thisstudybuildsonthesefindingsbysuggestingskilledusersmay
adapttheirsearchingbehaviortotheinformationspace.
Theexaminationofprocessingpatternsonthelessrestrictedlocatingtasksmayalso
illuminatethequantitativefindingsofthisstudy.Specificallytheresultsofthethinkaloudindicated
thatstudentswhomademorerelevancyjudgmentsoutperformedtheirpeers.Giventhatthere
weremoreacceptableanswersontheORCA‐LRtaskwhencomparedtotheORCA‐MRthismay
haveleadtothedifferencesinmeanscoresonthelocatingitemsontheORCA‐LR(3.599)andthe
ORCA‐MR(2.755).Itmayhavecontributedtotheoverall,andsignificant,differencesinscoreson
theORCA‐LRandtheORCA‐MR.OntheORCA‐LRitemsstudentsoftenchosetoevaluatewebsites
wheretheauthorwaseasilyidentifiable.ThiswasnotanoptionontheORCA‐MRitems.Thismay
haveloweredevaluationitems.OntheORCA‐MRsynthesisitemstheinabilitytofindtherequired
211
sourcemayhaveimpactedthewebsitesthatwereusedtocombinedetails.Thesefactorsmayhave
leadtosignificantlylowerscoresontheORCA‐MR.
Whatpatternsofprocessingappearamongreaderswithvaryinglevelsofonline
readingcomprehensionabilityduringanonlinereadingcomprehensiontaskwithinamore
restrictedinformationspace?Thereweremajordifferencesinthepatternsofprocessingonthe
ORCA‐MRwhencomparedtotheORCA‐LR.OnthemorerestrictedORCAstudentswhoused
strategiestosearchwithinawebpageoutperformedtheirpeerswhodidnot.Whenstudentsdid
usesearchenginesthosewhoincludedboththetopicandthesourcescoredbetterthantheirpeers.
Finally,andcomparabletotheORCA‐LR,studentswhomademorerelevancyjudgments
outperformedtheirpeers.
Theseresultsalsoreflectfindingsinpreviousresearch.Successfulreaders infer which link
may be most useful on a webpage (Coiro & Dobler, 2008; Henry, 2006) during online reading
comprehension. Furthermore researchers (McDonald & Stevenson, 1996; Rouet, 2006) have found that
more skilled online readers efficiently scan for relevant information within websites. Goldman et al,
(2012) found that expert readers could navigate within a website when compared to their novice peers.
Similarly, in this study there was a clear difference between the proficient online readers and their peers.
More successful online readers could navigate directly to a website and search within that site. Less
proficient online readers often searched for a website using a search engine and accepted their first click
on the homepage as a relevant source.
The patterns of processing on the locating tasks of the ORCA-MR may also help to explain the
quantitative findings of this study. Specifically the ability of proficient readers to keep digging within a
source until they found the specified website may explain why dispositions scores were significant in the
ORCA-MR model but not the ORCA-LR model. Based on the think aloud data searching within a source
required more reflective thinking, persistence, and flexibility. These variables were measured by the
dispositions of online reading comprehension assessment.
212
ExploringTheImplicationsOfDifferentPatternsOfProcessingDueToTheInformation
Space
Implicationsforresearch.Inthisstudyonlytheinformationspace,specificallythe
locatingtasks,wererestricted.Thisinfluencedhowthevariablesofinterestbehavedinthemodels.
Ifotherelementsoftheinformationspace,beyondthelocatingtasks,wererestrictedthemodels
mayhavebeendifferent.Therearetwoadditionalmethodsforrestrictingtheinformationspace
thatmightbeattemptedinfuturework:restrictingtheonlinereadingcomprehensionassessment
itemsandrestrictingtheoverallinquirytask.
Futureworkmustcontinuetoseewhattherelativecontributionsofbackground
knowledge,criticalevaluationskills,anddispositionscoresplayinpredictingscoresononline
readingcomprehensionassessmentwhentherestrictednatureofevaluation,synthesisand
communicationitemsismanipulated.Forexamplestudentscouldbegiventhesourcetoevaluate
ratherthanchoosingasourcetoevaluate.Givingstudentsanotebooktoolwithscaffoldedprompts
couldrestrictthesynthesisitems.Finallyproviding“sentencestarters”couldrestrict
communicationitems.Eachoftheseelementswouldfurtherrestricttheinformationspacesbeing
usedbystudentsandcouldinfluencethecontributionsthatbackgroundknowledge,critical
evaluationskills,anddispositionscoresplayinpredictingscoresduringonlinereading
comprehension.
Futureworkalsoneedstocontinuetoexplorehowtaskdifferencesaffectonlinereading
comprehension.Itiswellknownthatthenatureofthetaskinfluencescomprehension(Snow,
2002).Thereforestudiescouldbeconductedthatrestricttheinquirytaskratherthantheelements
ofonlinereadingcomprehension.ForexampleThisstudywouldbeinformedbyfutureworkthat
thatteststhesameinquirytask,suchastheCrossingtheDelawaretask,inbothalessrestrictedand
morerestrictedinformationspace.Thiswouldhelpusdrawadistinctionbetweentaskdifferences
andtheinformationspace.
213
Infutureworktheinquirypromptitselfshouldbeevaluatedbasedonhowitaffectsthe
contributionthatbackgroundknowledge,criticalevaluationskills,anddispositionscoresplayin
predictingscoresononlinereadingcomprehensionassessment.Inthesestudiestheinquiry
promptandtaskcouldbethevariedasafactorindeterminingthenatureoftheinformationspace.
Forexampleonlinereadingcomprehensionassessmentscouldbebuiltthataskstudentstofinda
specificanswertoamorerestrictedquestionsuchas,“Whatisthedistancetothemoon?”Other
assessmentscouldasklessrestrictedquestionssuchas,“Whatisthebestwaytofightglobal
hunger?”Researchmayfindthatrestrictingtheinformationspacebyalteringtheinquiryquestion
mayaffectthecontributionthatbackgroundknowledge,criticalevaluationskills,anddisposition
scoresplayinpredictingonlinereadingcomprehensionperformance
Implicationsforpractice.Thestagetwothemesalsoaddtothegrowingevidencethat
teachingstudentstolocatesourcesontheInternetshouldbecomeacentralpracticeintoday’s
classroom.Thepatternsfoundinthisstudyaddtoresearchwhichconcludedthatgoodonline
readersgenerateeffectivekeywordsearchstrategies(Bilal,2000;Kuiper&Volman,2008),make
forwardinferences(Coiro,2011)inordertojudgetherelevancyoflinks(Henry,2006),and
effectivelyskimandscansourcestofinddetailswithinasource(Rouet,2006;Rouetetal.,2011).
Thefindingsfromthisstudysuggestthatweneedanincreasedinstructionalfocusonteaching
theseskillswithindifferenttypesofinformationspaces.Thepatternsofprocessingonthelocating
taskswerequitestarkbetweenthemoreproficientonlinereadersandtheirpeers.
Limitations
Thisstudyofseventhgradestudentsinthreediverseschooldistrictswasdesignedtotest
thecontributionsthatbackgroundknowledge,criticalevaluationskills,anddispositionsofonline
readingcomprehensionmadeonscoresofameasureofonlineaftercontrollingforverbal
intelligence.Twodifferentmodelsweretested,alessrestrictedinformationspaceandamore
restrictedinformationspace.Furthermorequalitativeanalyseswereconductedtoexaminethe
214
differencesinstrategyuseamongstudentswhovariedontheironlinereadingcomprehension
ability.Potentiallimitationstothisstudyneedtoberecognizedinanyinterpretation.
Instrumentation
Thetheoreticalmodelsinthisstudyareasonlygoodastheinstrumentsusedtocalculate
studentscores.Thisstudyrequiredthecreationandvalidationoffourseparateinstruments.
Thereforeresultsfromthisstudyshouldnotbegeneralizeduntiltheinstrumentscanundergo
furthertestingandthemodelsverifiedwithconfirmatorystudies.Therearelimitationstothe
ORCAassessments,thebackgroundknowledgemeasure,andthedispositionsofonlinereading
comprehensionmeasurethatmaylimittheinterpretivepowerofthisstudy.
Onlinereadingcomprehensionassessments.ThereliabilitiesfortheORCA‐LRand
ORCIA‐MRwereacceptablebutlow.ReliabilityforthelessrestrictedORCAwasestimatedbya
coefficientalphaof.72(Cronbach,192).ThecoefficientalphaforthemorerestrictedORCAwas
.804.Theseareacceptableforresearch(Nunnally,1978).Howeverreliabilityestimatesinthis
rangecansometimesleadtotheoverestimationofeffectsizesinmultipleregressionanalyses
(Osborne,2002).Infactinmultipleregressionmodels,“Withtheadditionofoneindependent
variablewithlessthanperfectreliability,eachsucceedingvariableenteredhastheopportunityto
claimpartoftheerrorvarianceleftoverbytheunreliablevariable”(Osborne&Waters,2002,para.
12).Giventheexploratorynatureofthisstudy,andthefactthatreliabilityestimatesexceedthose
requiredforresearch(Nunnally,1978)theinternalconsistencyofthequantitativemeasureswas
deemedadequate.
Backgroundknowledgemeasure.Thebackgroundknowledgemeasureusedinthisstudy
haslimitationsassociatedwiththevariable’suseintheregressionmodel.Thebiggestthreattothe
studywastheoverallnumberofstudentswhoreceivedazeroonthemeasureindicatingthatthey
couldnotrecallanyinformationabouttheAmericanRevolution.Thispresentsanumberofissues.
215
Firstitishighlyunlikelythatthestudentsdidnothaveanybackgroundknowledgeofthe
AmericanRevolution.Thisstudy,byusingthebackgroundknowledgemeasure,mayhave
underestimatedparticipants’backgroundknowledge.Thebackgroundknowledgemeasureusedan
onlineform,inagroupsetting,toaskstudentstorecallasmanydetailsaspossibleaboutthe
AmericanRevolution.Itmaybethatanothermethodsuchasaconstructedresponsetestor
administeringthesameinstrumentorallyandoneononewouldhavebeenabettermethodto
estimatebackgroundknowledge.
Thefrequencyoflowscoresontheassessmentalsoviolatedtheassumptionofnormality
requiredforregressionmodels.Thescoreshadtobetransformedduetoexcessivelyhighkurtosis.
Theuseofthislogarithmictransformationalsolimitstheinterpretationsofthisstudy.Thevariable
usedinthemodelwasnottheobservedbutrathertheresultofatransformationandmaycontain
newbiases(Beauchamp&Olson,1973).Thusanyinterpretationofthemodel,especiallyofthe
backgroundknowledgevariableshouldbeundertakenwithcaution.
Thelackofvariabilityofthetransformedbackgroundknowledgevariablealsopresentsa
limitationtointerpretation.Afterthetransformationthestandarddeviationforthebackground
knowledgescorewas.0725.Thislackofvariancethreatenstheoverallregressionmodels.Lower
variancecanleadtolargerstandarderrors(Osborne&Waters,2002).Inflatedstandarderrorscan
leadtoincreasedTypeIerrorrate.
Theformatofthebackgroundknowledgeassessmentmayhaveledtothelowmeanscore
andoveralllowvarianceinthescores.Thetaskhadstudentsansweraprompttolistallthatthey
knewabouttheAmericanRevolution.Theywerethengiventwomorechancestoanswerthesame
prompt.Thisformatmaynothaveelicitedallofthebackgroundknowledgestudentshadaboutthe
AmericanRevolution.Futureworkshouldtrytoelicitadditionalbackgroundknowledgeconnected
tothedisciplinaryliteraciesofsocialstudies(Shanahan&Shanahan,2008).Onepossiblemethod
wouldbetoprovidestudentswithblankconceptmapswithconceptsassociatedwiththeAmerican
216
Revolution.TherecouldbedifferentareasforleadersoftheRevolution,battles,causes,dates,etc.
Thismethodofassessmentmaybebetteralignedwiththetypesofknowledgeneededinthe
discipline.
DispositionsofOnlineReadingComprehensionMeasure.Thebiggestlimitationofthe
dispositionsofonlinereadingcomprehensionmeasure(DORC)wasthetreatmentofordinaldataas
intervaldataintheregressionmodels.TheDORCusedafivepointLikertscale.Labowitz(1967;
1970)arguesthatthevalueoftreatingordinaldata,suchastheDORCscores,asintervaldatain
multivariateanalysisoutweighstheriskofincludingordinaldatainregressionmodels.
Theoperationaldefinitionsoftheconstructsmeasuredbythedispositionsinstrumentalso
presentalimitationtothestudy.ItmaybethattheDORCfailedtocapturetheepistemologicaland
ontologicalstancesof“openmindsets”(Deschryver&Spiro,2010)necessaryforonlinereading
comprehension.Intermsoftherestrictednatureoftheinformationspace,forexample,itmaybe
thatdifferentmindsetsareneededtosucceedinarestrictedspacewhencomparedtoaless
restrictedspace.Thereforeareaderwhohastheabilitytoadoptmultipleperspectivesmaysucceed
atonlinereadingcomprehensiontasks.TheitemsincludedintheDORCdonotattempttomeasure
thesetypesofepistemologicalshiftsinthereader.
Futuremeasuresofdispositionsshouldtrytocaptureboththe“openmindsets”andthe
abilityofreaderstoshiftworldviewsandmindsetsbasedontheinformationspace.Thiswould
requirethecreationofatwo‐scaleinstrumentwithitemsindicatinganopenmindsetoraclosed
mindset.Thisinstrumentcouldthenbeappliedtomultiplemodels.
MissingDataandSampleSize
Anotherlimitationtothisstudyisthehighrateofmissingdata.Thesamplestartedwith
131students.Afterthreeoutlierswereremovedthefinalnequaled95students.Thishighrateof
attritioncouldleadtoimportantstudentsbeingleftoutofthemodel.Theattritionratewassimilar
acrossthethreeresearchsites.Thelargenumberofmissingdatawasduemainlytooneresearcher
217
beingresponsiblefortheadministrationofsomanyassessments.Itwasimpossibletoensureall
thestudentscompletedeachtaskandallabsenteeswereaccountedforinamanageabletimeframe.
Asmallernumberofstudentswerelosttocorrupteddata.
Themissingdataledtoasmallersamplesize.Inordertoensuretherewasenoughpowerin
themodelthetenstudentswhoconductedtheverbalprotocolportionofthestudywereincludedin
thefinalsample.Themeanscores,ontheORCA‐LRandtheORCA‐MRwerehigherforthe
qualitativeparticipantsthantheoverallmeanscoresontheORCA‐LRandtheORCA‐MRofstudents
whodidnotparticipateintheverbalprotocols.Itcouldbethatelicitingstudentstothinkaloud
aboutthestrategiestheyusedresultedinraisingtheirperformance(Gerjets,Kammerer,&Werner,
2010).
ResearcherBiasinQualitativeAnalysis
Astheonlypersoncodingandanalyzingdatathequalitativeportionofthisstudycould
havebeenthreatenedbymyinherentepistemologicalandontologicalviews.Mydesiretofind
differencesamongstudentstrategyusecouldhaveinfluencedmyresults.Inordertocontrolfor
thisacodebookwasdevelopedandanotherresearcherutilizedtoensureinter‐raterreliability.I
wasalsoabletoprovideself‐checksthroughoutthestudytoensurepatternsIfoundwere
consistentwiththedata.DuringtheanalysisprocessIwasabletorepeatedlyreturntothevideo
dataforself‐checks.Iwouldensurethatideaunitsonlyfitaspecificcode.Oncethecodeswere
collapsedintopatternsIreturnedtothevideodatatoensurethatpatternswereuniqueanddid
notoverlap.OncethepatternswerecollapsedintothemesIreturnedtothevideodatatoensure
thethemesencompassedallofthepatternstheycaptured.Theuseofaninter‐raterreliabilitycheck
andtherecursiveselfcheckshelpedtominimizethethreatofresearcherbias.
AddressingtheGrowingChallengesofOnlineReadingComprehension
Theresultsofthisstudycontributetobothresearchandpractice.Forresearch,theresults
informricherandmorecomplexmodelsofonlinereadingcomprehension.Thiswasoneofthefew
218
studiestoconsiderhowtherestrictednatureofonlinereadingcomprehensiontasksinfluences
studentperformance.Itsuggeststhatthenatureoftheinformationspaceforonlinereading
comprehensiondoes,indeed,makeadifference.
Recentcallstoaddressadolescentliteracyhaveemergedacrossthenation(CCAAL,2010;
CouncilofStateGovernments,2010;Snow&Biancarossa,2003).Infactthesecallshaveculminated
intotheCommonCoreStateStandardsplacingaheavyemphasisonpreparingstudentswhoare
“collegeandcareerready”(CCSS,2010,p.5).Whilethisisanoblecause,researchershavenoted
thatnewstatestandardsplacetooheavyanemphasisonofflinedefinitionsoftext(Draper,2012;
Leuetal.,2011).Infactthisstudyhelpstodemonstratethatanycalltoactiontoimprove
adolescentliteracymustplaceastrongemphasisononlinereadingcomprehensionandthenature
oftheinformationspace.
AsthesecallsforimprovedadolescentliteracyhaveemergedtheInternethasgrowninits
dominanceasthetextofchoiceforourstudents.Adolescentsnowspendmoretimereadingonline
thanoffline(KaiserFamilyFoundation,2005).FurthermoretheInternetistheprimarysource
whendoinghomeworktheInternetforhomework.Thereforeanyeffortstoimproveadolescent
literacymustalsoincludeeffortstoimprovemeaningmakinginonlinespaces,andtheliteracy
researchcommunityneedsmorerobustmodelsofcomprehensioninonlinecontexts.
Theseonlinespacesareverydiverse.Infactoneofthegreatestdemandsplacedonthe
readerwhenreadingonlineistheshiftinginformationspace(Leu,2002)Afterallwhenreading
onlinethesizeoftheinformationfieldconstantlyshiftsbasedonthetaskofthereader.Thisstudy
soughttoenrichthetheoreticalmodelsofonlinereadingcomprehensionbyexaminingshifting
informationspaces.
Inthisstudyitwasevidentthattherestrictednatureoftheinformationspaceaffectsonline
readingcomprehension.Usingbothquantitativeandqualitativemethodsthisstudydemonstrated
thatreadercharacteristicsbehavedifferentlywhentheinformationspaceisalteredduringonline
219
readingcomprehensiontasks.Innaturalsettingstheinformationspaceusedinonlinereading
comprehensiontaskswillalwayschange.Thusresearchersneedtodeveloptheoreticalmodelsof
readingthataccountforthisconstantflux(Hartmanetal.,2010;Leuetal.,2004).
Itisalsoclearfromthisstudythatlongheldassumptionsaboutreadingcomprehension
mayhavetobereconsidered.Specificallytherolebackgroundknowledgeplaysindetermining
comprehensionmaybelessenedinonlinespaces.Inthisstudyscoresonacriticalevaluation
measure,notbackgroundknowledge,werethestrongestpredictorofperformanceonboththeless
restrictedmodelofonlinereadingcomprehensionandthemorerestrictedmodelofonlinereading
comprehension.Thusitisclearthatcriticalevaluationskillsarecentraltosuccessfulonlinereading
comprehension(Coiro,2003).
Unfortunatelystudieshaveconsistentlyshownthatstudentsareillequippedtoevaluate
onlineinformation(Bennett,Maton,&Kervin,2008;Goldmanetal.,2012;Jewitt,2008;Johnson&
Kaye,1998;Livingstone,2004;Leuetal.,2007bRieh&Belkin,1998;).Boththequantitativeand
qualitativedataofthisdutysupportthisconclusion.Ifwearetoaddressadolescentliteracyitis
clearthataconcertedeffortisneededtoteachstudentscriticalevaluationskills.Thisissuewillbe
paramountforbotheducatorsandresearchers.
Forpractice,theresultsinformteacherschargedwithteachingliteracyinaconstantly
evolvingworld,oneinwhichtheInternetisincreasinglyimportanttobothreadingandlearning
(Snow,2002).Thisstudyhighlightsthatteachersneedtoconsiderfourcriticalareasforonline
readingcomprehension:searchstrategies,navigationalstrategies,evaluationstrategiesand
communicationstrategies.Whilebuildinginstructionalroutinesthatfocusonthesestrategiesthis
studyalsosuggeststhatteachersbuildinopportunitiestomeasureanddevelopdispositions
centraltoonlinereadingcomprehension.
Ifwearetomeetthecalltoimproveadolescentliteracyonlinereadingcomprehension
cannotbeignore.Itisacriticalissuefacingliteracyresearchersandeducators.AstheInternet’s
221
References
Afflerbach, P. P. (1990). The influence of prior knowledge on expert readers’ main idea
construction strategies. Reading Research Quarterly, 25, 31-46.
Afflerbach, P. (2002). The use of think-aloud protocols and verbal reports as research
methodology. In M. Kamil, P. Mosenthal, P.D. Reason, & R. Barr, (Eds.), Methods of
literacy research (pp. 87-103). Mahwah, NJ: Erlbaum.
Agosto, D. (2002). Bounded rationality and satisficing in young people’s web-based decision
making. Journal of the American Society for Information Science and Technology. 53,
16-27.
Alexander, P. A., & Fox, E. (2004). A historical perspective on reading research and practice. In
R. B. Ruddell, N. J. Unrau (Eds.), Theoretical models and processes in reading (5th ed.,
pp. 33-68). Newark, NJ: International Reading Association.
Alexander, P. A., & Fox, E. (2011). Adolescents as readers. In M. Kamil, P. D. Pearson, E.B.
Moje, & P. Afflerbach (Eds.), Handbook of reading research (Vol. 4, pp. 157-176).
New York, NY: Routlage.
Allen, B. (1992). Cognitive differences in end user searching of a CD-ROM index. In N. J.
Belkin, P. Ingwersen, & A. M. Pejtersen (Eds.), Proceedings of the 15th Annual
International ACM SIGIR Conference on Research and Development in Information
Retrieval (pp. 298-309). ACM, New York, NY, USA, DOI=10.1145/133160.133212
http://doi.acm.org/10.1145/133160.133212
Allen, I.E., & Seaman, J. (2010). Class differences: Online education in the United States, 2010.
Retrieved from The Sloan Consortium website:
http://sloanconsortium.org/sites/default/files/class_differences.pdf
222
Alvermann, D. E. (2004). Media, information communication technologies, and youth literacies:
A cultural studies perspective. American Behavioral Scientist, 48, 78-83. doi:
10.1177/0002764204267271.
Anderson, R. C., & Pearson, P. D. (1984). A schema-theoretic view of basic processes in reading
comprehension. In P. D. Pearson (Ed.), Handbook of Reading Research (pp. 255-291).
New York: Longman.
Anderson, R. C., Reynolds, R. E., Schallert, D. L., & Goetz, E. T. (1977). Frameworks for
comprehending discourse. American Educational Research Journal, 14, 367-381.
Avons, S. E., Wragg, C. A., Cupples, L., & Lovegrove, W. J. (1998). Measures of phonological
short-term memory and their relationship to vocabulary development. Applied
Psycholinguistics, 19, 583-602.
Baker, F. B. (1985). The basics of item response theory. Portsmouth, NH: Heinemann.
Baker, L., & Wigfield, A. (1999). Dimensions of children’s motivation for reading and their
relations to reading activity and reading achievement. Reading Research Quarterly, 34,
452-477. doi: 10.1598/RRQ.34.4.4
Beauchamp, J. J., & Olson, J. S. (1973). Corrections for bias in regression estimates after
logarithmic transformation. Ecology, 54, 1403-1407.
Berners-Lee, T. (1989). Information Management: A Proposal, CERN, March 1989. Retrieved
from http://www.w3.org/History/1989/proposal.html.
Biancarosa, C., & Snow, C. E. (2006). Reading next—A vision for action and research in middle
and high school literacy: A report to Carnegie Corporation of New York (2nd ed.).
Washington, DC: Alliance for Excellent Education.. Retrieved from
http://www.all4ed.org/files/ReadingNext.pdf
223
Bilal, D. (2000). Children's use of the Yahooligans! Web search engine: I. Cognitive, physical,
and affective behaviors on fact-based search tasks. Journal of the American Society for
Information Science, 51, 646-665.
Bilal, D. (2001). Children’s use of the Yahooligans! Web search engine: II. Cognitive and
physical behaviors on research tasks. Journal of the American Society for Information
Science, 52, 118-136.
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy
of educational objectives: Handbook I: Cognitive domain. New York, NY: Longman.
Bogdan, R. C., & Biklen, S. K. (2006). Qualitative research for education: An introduction to
theories and methods (5th ed.). Boston, MA: Allyn & Bacon.
Braasch, J. L., Lawless, K. A., Goldman, S. R., Manning, F. H., Gomez, K. W., & MacLeod, S.
M. (2009). Evaluating search results: An empirical analysis of middle school students'
use of source attributes to select useful sources. Journal of Educational Computing
Research, 41, 63-82.
Bråten, I., Strømsø, H.I., & Britt, M.A. (2009). Trust matters: Examining the role of
source evaluation in students' construction of meaning within and across multiple
texts. Reading Research Quarterly, 44, 6-28. doi: 10.1598/RRQ.44.1.1
Bråten, I., Strømsø, H. I., & Samuelstuen, M. (2008). Are sophisticated students always better?
The role of topic-specific personal epistemology in the understanding of multiple
expository texts. Contemporary Educational Psychology, 33, 814-840. doi:
10.1016/j.cedpsych.2008.02.001.
Brem, S., Russell, J., & Weems, L. (2001). Science on the Web: Student evaluations of scientific
arguments. Discourse Processes, 32, 191-213. doi: 10.1207/S15326950DP3202&3_06.
224
Britt, M. A., & Gabrys G. (2002) Implications of document-level literacy skills for Web site
design. Behavior Research Methods, Instruments, & Computers, 34, 170-176.
Britt, M. A., Perfetti, C. A., Sandak, R., & Rouet, J.-F. (1999). Content integration and source
separation in learning from multiple texts. In S. R. Goldman, A. C. Graesser, & P. v. d.
Broek (Eds.), Narrative comprehension, causality, and coherence: Essays in honor of
Tom Trabasso. Mahwah, NJ: Erlbaum.
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning.
Educational Researcher, 18(1), 32-42. doi: 10.3102/0013189X018001032
Campbell, J. R. (2005). Single instrument, multiple measures: Considering the use of multiple
item formats to assess. In S. G. Paris & S. S. Stahl (Eds.), Children's reading
comprehension and assessment (pp. 347-368). Mahwah, NJ: Erlbaum.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for
research. Belmont, CA: Wadsworth.
Campione, J. C., Shapiro, A. M., & Brown, A. L. (1995). Forms of transfer in a community of
learners: Flexible learning and understanding. In A. McKeough, J. Lupart, & A. Marini
(Eds.), Teaching for transfer: Fostering generalization in learning (pp. 35-68). Mahwah,
NJ: Erlbaum.
Carnegie Council on Advancing Adolescent Literacy. (2010). Time to act: An agenda for
advancing adolescent literacy for college and career success. New York: Carnegie
Corporation of New York.
Carr, M. (1999). Being a learner: Five learning dispositions for early childhood. Early Childhood
Practice, 1, (1). 81-99.
225
Carr, M., & Claxton, G. (2002). Tracking the development of learning dispositions. Assessment
in Education: Principles, Policy & Practice, 9, 9-37. doi: 10.1080/09695940220119148
Carroll, J. B. Psychometric tests as cognitive tasks: A new "structure of intellect." In L. B.
Resnick (Ed.), The nature of intelligence. Hillsdale, NJ : Erlbaum, 1976.
Cervetti, G., Pardales, M. J., & Damico, J. S. (2001, April). A tale of differences: Comparing the
traditions, perspectives, and educational goals of critical reading and critical literacy.
Reading Online, 4(9). Retrieved from
http://www.readingonline.org/articles/art_index.asp?HREF=/articles/cervetti/index.html
Chiesi, H.L., Spilich, G.J., & Voss, J.F. (1979). Acquisition of domain-related knowledge in
relation to high and low domain knowledge. Journal of Verbal Learning and Verbal
Behavior, 18, 257-273.
Coiro, J. (2003). Rethinking comprehension strategies to better prepare students for critically
evaluating content on the Internet. [Electronic Version]. New England Reading
Association Journal. Retrieved from http://ctell1.uconn.edu/coiro/nera2003.pdf
Coiro, J. (2011). Predicting reading comprehension on the Internet: Contributions of offline
reading skills, online reading skills, and prior knowledge. Journal of Literacy Research,
43, 352-392. doi: 10.1177/1086296X11421979
Coiro, J., Castek, J., & Guzniczak, L. (2011). Uncovering online reading comprehension
processes: Two adolescents reading independently and collaboratively on the Internet. In
P.J. Dunston, L.B. Gambrell, K. Headley, S.K. Fullerton, & P.M. Stecker (Eds.), Sixtieth
yearbook of the Literacy Research Association (pp. 354–369).
226
Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by
sixth-grade skilled readers to search for and locate information on the Internet.. Reading
Research Quarterly, 42, 214–257.
Coiro, J., Knobel, M., Lankshear, C., & Leu, D. J. (2008). Central issues in new literacies and
new literacies research. Lawrence Erlbaum Associates, Taylor & Francis Group.
Retrieved from http://eprints.jcu.edu.au/7369/
Coiro, J., Knobel, M., Lankshear, C. & Leu, D. (2008). Central issues in new literacies and new
literacies research. In J. Coiro, M. Knobel, C. Lankshear, & D. Leu (Eds.), Handbook of
research on new literacies (pp. 25-32). Mahwah, NJ: Erlbaum.
Collins, K. M. T., Onwuegbuzie, A. J., & Sutton, I. L. (2006). A model incorporating the
rationale and purpose for conducting mixed methods research in special education and
beyond. Learning Disabilities: A Contemporary Journal, 4(1), 67-100.
Conley, M.W. (2008). Cognitive strategy instruction for adolescents: What we know about the
promise, what we don't know about the potential. Harvard Educational Review, 78, 84-
106.
Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for
field settings. Chicago, IL: Rand McNally.
Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed
methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed
methods in social and behavioral research (pp. 209-240). Thousand Oaks, CA: Sage.
Cronbach, L. J. , Gleser, C.G., Nanda, H., & Rajaratnam, N. (1972). The dependability of
behavioral measurements: Theory of generalizability for scores and profiles. New York,
NY: John Wiley.
227
Damico, J., & Baildon, M. (2007). Examining ways readers engage with websites during think-
aloud sessions. Journal of Adolescent & Adult Literacy. 51, 254-263. doi:
10.1598/JAAL.51.3.5
Davis, E. A. (2000). Scaffolding students' knowledge integration: Prompts for reflection in KIE.
International Journal of Science Education, 22, 819-837.
DeSchryver, M. & Spiro, R. (2008). New forms of deep learning on the Web: Meeting the
challenge of cognitive load in conditions of unfettered exploration. In R. Zheng (Ed.),
Cognitive effects of multimedia learning (pp. 134-152). Hershey, PA: IGI Global.
DeSchryver, M. & Spiro, R. (in preparation). CFT on the Web: Illustrated theoretical cognitive
and metacognitive processes for advanced nonlinear knowledge acquisition of ill-
structured topics.
D'Mello, S., & Graesser, A. (2009). Automatic detection of learner's affect from gross body
language. Applied Artificial Intelligence, 23, 123-150.
Dwyer, B. (2010). Scaffolding Internet reading: A study of a disadvantaged school community in
Ireland (Doctoral dissertation, University of Nottingham, Nottingham, UK). Retrieved
from http://etheses.nottingham.ac.uk/2426/1/523046.pdf.
Eagleton, M. B. & Dobler, E. (2007). Reading the Web: Strategies for Internet inquiry. New
York, NY: Guilford.
Ekstrom, R. B., French, J. W., & Harman, H. H. (1979). Cognitive factors: Their identification
and replication. Multivariate Behavioral Research Monographs, 79(2), 3-84.
Ennis, RH. (1962). A concept of critical thinking. Harvard Educational Review, 32, 81-111.
228
Ennis, R.H. (1987). A taxonomy of critical thinking dispositions and abilities. In J. B. Baron &
R.J. Sternberg (Eds.), Teaching thinking skills: Theory and practice (pp. 9–26), New
York, NY: W.H. Freeman.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.)
Cambridge, MA: MIT Press.
Fincher-Kiefer, R., Post, T. A., Greene, T.R., & Voss, J.E. (1988). On the role prior knowledge
and task demands in the processing of text. Journal of Memory and Language, 27, 416-
428.
Fitzgerald, M. A. (1999) Evaluating information: An information literacy challenge. School
Library Media Research (Vol. 2). Retrieved from
http://www.ala.org/ala/mgrps/divs/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume2
1999/vol2fitzgerald.cfm
Flanagin, A. J., & Metzger, M. J. (2000). Perceptions of Internet information credibility.
Journalism and Mass Communication Quarterly, 77, 515–540.
doi: 10.1177/107769900007700304
Fogg, B.J., Marshall, J., Laraki, O., Osipovich, A., Varma, .Fang, N. , . . . Treinen, M. (2001).
What makes web sites credible? A report on a large quantitative study. Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems, Seattle, WA. 61-68,
doi: 10.1145/365024.365037
Fogg, B. J., Soohoo, C., Danielson, D. R., Marable, L., Stanford, J., & Tauber, E. R. (2003).
How do users evaluate the credibility of Web sites? A study with over 2,500 participants.
Proceedings of the Conference on Designing for User Experiences. Retrieved from
http://dl.acm.org/citation.cfm?id=997097
229
Fox, S., & Rainie, L. (2002). Vital decisions: How Internet users decide what information to trust
when they or their loved ones are sick. Washington, DC: Pew Internet & American Life
Project. Retrieved from
http://www.pewinternet.org/pdfs/PIP_Vital_decisions_May2002.pdf
Freebody, P., & Luke, A. (1990). Literacies programs: Debates and demands in cultural context.
Prospect: Australian Journal of TESOL, 5(7), 7-16.
Freire, P. (1970). The adult literacy process as cultural action for freedom. Harvard Educational
Review, 40, 205-225.
Fritch, J. W., & Cromwell, R. L. (2001). Evaluating Internet resources: Identity, affiliation, and
cognitive authority in a networked world. Journal of the American Society for
Information Science and Technology, 52,, 499–507.
Fritch, J. W., & Cromwell, R.L. (2002). Delving deeper into evaluation: Exploring cognitive
authority on the Internet. Reference Services Review, 30, 242–254.
Gable, R. K., & Wolf, M. B. (1993). Instrument development in the affective domain: Measuring
attitudes and values in corporate and school settings. (2nd ed.). Norwell, MA: Kluwer
Academic Publishers.
Gates, A. I. (1931). What do we know about optimum lengths of lines in reading?. The Journal
of Educational Research, 23, 1-7.
Gathercole, S. E., & Baddeley, A. D. (1993). Working memory and language. East Sussex, UK:
Psychology Press.
Gathercole, S. E., Service, E., Hitch, G. J., Adams, A -M., & Martin, A. J. (1999). Phonological
short‐term memory and vocabulary development: Further evidence on the nature of the
relationship. Applied Cognitive Psychology, 13, 65-77.
230
Gee, J. P. (2004). Situated Language and Learning: A Critique of Traditional Schooling.
London: Routledge.
Gee, J. P., & Green, J. L. (1998). Discourse analysis, learning, and social practice: A
methodological study. Review of Research in Education, 23(1), 119-169.
Goldman, S. R., Braasch, J. L., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012).
Comprehending and learning from Internet sources: Processing patterns of better and
poorer learners. Reading Research Quarterly, 47, 356-381. doi: 10.1002/RRQ.027
Goldstein, W. M., & Hogarth, R. M. (Eds.). (1997). Research on judgment and decision making:
Currents, connections, and controversies. New York: Cambridge University Press.
Graesser, A. C., Wiley, J., Goldman, S. R., O’Reilly, T., Jeon, M., & McDaniel, B. (2007).
SEEK Web tutor: Fostering a critical stance while exploring the causes of volcanic
eruption. Metacognition and Learning, 2, 89-105.
Gross, E. (2004). Adolescent Internet use: What we expect, what teens report. Journal of Applied
Developmental Psychology, 25, 633-649. doi: 10.1016/j.appdev.2004.09.005
Guthrie, J. T., & Wigfield, A. (Eds.). (1997). Reading engagement: Motivating readers through
integrated instruction. Newark, DE: International Reading Association.
Guthrie, J. T., Wigfield, A., & Perencevich, K. C. (Eds.). (2004). Motivating reading
comprehension: Concept-Oriented reading instruction. Mahwah, NJ: Erlbaum.
Harris, F. J. (2008). Challenges to teaching credibility assessment in contemporary schooling.. In
M. J. Metzger & A. J. Flanigan (Eds.), Digital Media, Youth, and Credibility, (pp. 155-
180). Cambridge, MA: MIT Press.. doi: 10.1162/dmal.9780262562324.155
Hartman, D.K., Morsink, P.M., & Zheng, J. (2010). From print to pixels: The evolution of
cognitive conceptions of reading comprehension. In E.A. Baker (Ed.), Multiple
231
Perspectives on New Literacies Research and Instruction (pp. 131-164). New York:
Guilford.
Herder, E., & Juvina, I. (2004). Discovery of individual user navigation styles. Proceedings of
the AH 2004 Workshop Eindhoven, The Netherlands. pp. 40-49. Retrieved from
http://www.dcs.bbk.ac.uk/%7Egmagoulas/AH2004_Workshop/Proceedings.htm
Henry, L. A. (2006). SEARCHing for an Answer: The critical role of new literacies while
reading on the Internet. The Reading Teacher, 59, 614-627. doi: 10.1598/RT.59.7.1
Hill, J.R., & Hannafin, M. J. (1997). Cognitive strategies and learning from the World Wide
Web. Educational Technology Research & Development, 45(4), 37-64.
Hirsch, E. D. (2003). Reading comprehension requires knowledge – of words and the world:
Scientific insights into the fourth-grade slump and stagnant reading comprehension.
American Educator, 27(1), 10-29.
Hoffman, J. L., Wu, H-K., Krajcik, J. S., & Soloway, E. (2003). The nature of middle school
learners' science content understandings with the use of on‐line resources. Journal of
Research in Science Teaching, 40, 323-346.
Hogarth, R. M. (1987). Judgment and choice: The psychology of decision. New York, NY: John
Wiley.
Johnston, P.H. (1984). Prior knowledge and reading comprehension test bias. Reading Research
Quarterly, 19, 219-239.
Judd, V. C., Farrow, L. I., & Tims, B. J. (2006). Evaluating public web site information: A
process and an instrument. Reference Services Review, 34, 12-32. doi:
10.1108/00907320510631571
232
Katz, L.G. (1993). Dispositions: Definitions and implications for early childhood practices.
Perspectives from ERIC/EECE: A monograph series, no. 4. Urbana, IL: ERIC
Clearinghouse on ECCE. Retrieved from http://www.eric.ed.gov/PDFS/ED360104.pdf
Katz, J., & Rice, R. (2002). Social consequences of Internet use: Access, involvement, and
interaction. Cambridge MA: MIT Retrieved from
http://www.citeulike.org/user/maxsenges/article/228698
Kay, K. (2009) Middle schools preparing young people for 21st century life and work. Middle
School Journal, 40(5), 41-54. Retrieved from
http://cesa7ita2009.pbworks.com/f/middle%20schools%20preparing%20young%20peopl
e%20for%20work%20in%20the%2021st%20century.pdf
Kiili, C., Laurinen, L., & Marttunen, M. (2008). Students evaluating Internet sources: From
versatile evaluators to uncritical readers. Journal of Educational Computing Research,
39, 75-95. doi: 10.2190/EC.39.1.e
Kiili, C., Laurinen, L., Marttunen, M., & Leu, D. (2011). Working on understanding:
collaborative reading on the Web. Manuscript submitted for publication.
Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction-
integration model. Psychological Review, 95, 163-182.
Kintsch, W. (1998) Comprehension: A paradigm for cognition. New York: Cambridge
University Press.
Kintsch, W., & van Dijk, T. A. (1978). Toward a model of text comprehension and production.
Psychological Review, 85, 363-394.
Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into
practice, 41, 212-218.
233
Kress, G. (2000). Multimodality. In B. Cope & M. Kalantzis (Eds.). Multiliteracies: Literacy
learning and the design of social futures (pp. 179-200). New York, NY: Routlage.
Kuiper, E. & Volman, M. (2008). The Web as a source of information for students in K-12
education. In J. Coiro, M. Knobel, D. Leu, & C. Lankshear (Eds.). Handbook of Research
on New Literacies (pp. 241-266). Mahwah, NJ: Erlbaum.
Kuiper, E., Volman, M., & Terwel, J. (2005). The Web as an information resource in K-12
education: Strategies for supporting students in searching and processing information.
Review of Educational Research, 75, 285-328.
Labbo, L. D., & Reinking, D. (1999). Negotiating the multiple realities of technology in literacy
research and instruction. Reading Research Quarterly, 34, 478-492. doi:
10.1598/RRQ.34.4.5.
Lankshear, C. & Knobel, M. (1998, April). Critical literacies and new technologies. Paper
presented at the Annual Meeting of the American Educational Research Association, San
Diego, CA.
Lankshear, C. & Knobel, M. (2006). New literacies: Everyday practices and classroom learning
(2nd ed.). Maidenhead, UK: Open University Press.
Lankshear, C., & Knobel, M. (2007a). Researching New Literacies : Web 2 .0 Practices and
Insider Perspectives, 1-31.
Lankshear, C., & Knobel, M. (2007b) Researching new literacies: Web 2.0 practices and insider
perspectives. E-Learning and Digital Media, 4, 224-240. Retrieved from
http://dx.doi.org/10.2304/elea.2007.4.3.224
Lawless, K. A., & Schrader, P. G. (2008). Where do we go now? Understanding research on
navigation in complex digital environments. In J. Coiro, M. Knobel, D. Leu, & C.
234
Lankshear (Eds.). Handbook of Research on New Literacies (pp. 267-296). Mahwah, NJ:
Erlbaum.
Lee, C. D., Spratley, A. (2010). Reading in the disciplines: The challenges of adolescent literacy.
New York, NY: Carnegie Corporation of New York.
Lemke, E. A., Klausmeier, H. J., & Harris, C. W. (1967). Relationship of selected cognitive
abilities to concept attainment and information processing. Journal of Educational
Psychology, 58, 27-35.
Lenhart, A., Horrigan, J., & Fallows, D. (2004). Content Creation Online: 44% of US Internet
users have contributed their thoughts and their files to the online world. Pew Internet &
American Life Project. Washington, DC.
Leslie, L. & Caldwell, J. (1995). Qualitative Reading Inventory – II. Glenview, IL: Harper
Collins.
Leu, D. J. (2000). Literacy and technology: Deictic consequences for literacy education in an
information age. In M.L. Kamil, P. Mosenthal, P.D. Pearson, & R. Barr (Eds.),
Handbook of reading research: Volume III (pp. 743-770). Mahwah, NJ:
Erlbaum.
Leu, D. J., Castek, J., Hartman, D., Coiro, J., Henry, L., Kulikowich, J., & Lyver, S. (2005).
Evaluating the development of scientific knowledge and new forms of reading
comprehension during online learning. Final report presented to the North Central
Regional Educational Laboratory/Learning Point Associates. Retrieved from
http://www.newliteracies.uconn.edu/ncrel_files/FinalNCRELReport.pdf
Leu, D. J., Kinzer, C. K., Coiro, J., & Cammack, D. (2004). Towards a theory of new literacies
emerging from the Internet and other ICT. In R. B. Ruddell & N. J. Unrau (Eds.),
235
Theoretical models and processes of reading, (5th ed., pp. 1570-1613). Newark, NJ:
International Reading Association.
Leu, D. J., Reinking, D., Hutchinson, A., McVerry, J. G., Robbins, K., Rogers, A., Malloy, J.,
O’Byrne, W. I., & Zawilinski, L. (2008, December). The TICA project: Teaching the
new literacies of online reading comprehension to adolescents: An alternative symposium
conducted at the National Reading Conference, Orlando, FL.
Leu, D. J., O’Byrne, W. I., Zawilinski, L., McVerry, J. G., & Everett-Cacopardo, H. (2009).
Expanding the new literacies conversation. Educational Researcher, 38, 264-269. doi:
10.3102/0013189X09336676
Leu, D. J., Zawilinski, L., Castek, J., Banerjee, M., Housand, B., Liu, Y., et al. (2007). What is
new about the new literacies of online reading comprehension? In L. Rush, J. Eakle, & A.
Berger, (Eds.). Secondary school literacy: What research reveals for
classroom practices. (37-68). Urbana, IL: National Council of Teachers of English.
Library of Congress. (2008). About the library. Retrieved from http://www.loc.gov/about/
Luke, A. (2000). Critical literacy in Australia: A matter of context and standpoint. Journal of
Adolescent & Adult Literacy, 43, 448-461.
Manderino, M. L. (2011). Reading across multiple multimodal sources in historical
inquiry (Doctoral dissertation, University of Michigan). Retrieved from
http://indigo.uic.edu/bitstream/handle/10027/9605/Manderino_Michael.pdf?sequence=1
Masoura, E. V., & Gathercole, S. E. (1999). Phonological short-term memory and foreign
language learning. International Journal of Psychology, 34, 383-388.
doi:10.1080/002075999399738
236
McKenzie, J. F., Wood, M. L., Kotecki, J. E., Clark, J. K., & Brey, R. A. (1999). Establishing
content validity: Using qualitative and quantitative steps. American Journal of Health
Behavior, 23, 311-318.
McWilliams, J., & Clinton, C. (2013). Reimagining and Reinventing the English classroom in
the digital age. In H. Jenkins & W. Kelley, with K. Clinton, J. McWilliams, R. Pitts-
Wiley, & E. B. Reilly (Eds). Reading in a participatory culture: Remixing Moby Dick in
the English classroom (pp. 185-196). New York, NY: Teachers College Press.
McVerry, J. G., O’Byrne, W. I., & Robbins, K. (2009, April). Validating instruments used in the
TICA Project. Presented at the Annual Meeting of the American Educational Research
Association Annual Meeting, San Diego, CA.
Mellor, B., & Patterson, A. (2004). Poststructuralism in English classrooms: Critical literacy and
after. International Journal of Qualitative Studies in Education, 17, 85–102.
Merriam, S.B. (1988). Case study research in education: A qualitative approach. San Francisco,
CA: Jossey-Bass.
Metzger, M. (2007). Making sense of credibility on the Web: Models for evaluating online
information and recommendations for future research. Journal of the American Society for
Information Science and Technology, 58, 2078-2091.
Metzger, J. M., Flanagin, J. A., & Zwarun, L. (2003). College student Web use, perceptions of
information credibility, and verification behavior. Computers & Education, 41, 271-290.
doi: 10.1016/S0360-1315(03)00049-6
Mills, E. (2005, October 8). Google ETA? 300 years to index the world's info. CNET. Retrieved
from http://news.cnet.com/Google-ETA-300-years-to-index-the-worlds-info/2100-
1024_3-5891779.html
237
Miniwatts Marketing Group. (2012). Internet World Stats. Retrieved from
http://www.internetworldstats.com/
Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A
framework for integrating technology in teacher knowledge. Teachers College Record,
108, 1017-1054.
Moje, E. B. (2009). A call for new research on new and multi-literacies. Research in the
Teaching of English, 43, 348-362.
Murray, D. E., & McPherson, P. (2006). Scaffolding instruction for reading the Web. Language
Teaching Research, 10, 131-156.
New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard
Educational Review, 66, 60-92.
Norris, P. (2001). Digital divide?: Civic engagement, information poverty, and the Internet
worldwide. Cambridge University Press. Retrieved from
http://portal.acm.org/citation.cfm?id=51619.
O’Byrne, W.I. (2011). Facilitating critical evaluation skills through content creation:
Empowering adolescents as readers and writers of online information. Unpublished
doctoral dissertation, University of Connecticut, Storrs.
O'Byrne, W. I., & McVerry, J. G. (2009). Measuring the dispositions of online reading
comprehension: A preliminary validation study. In K. Leander, D. Rowe, D. Dickson, M.
Hundley, R. Jiménez, & V. Risko (Eds.), The 57th National Reading Conference
Yearbook (pp.58-84). Oak Creek, WI: National Reading Conference.
O'Hanlon, N. (2002). Net knowledge: Performance of new college students on an Internet skills
proficiency test. The Internet and Higher Education, 5, 55-66.
238
Onwuegbuzie, A. J., & Leech, N. L. (2006). Linking Research Questions to Mixed Methods Data
Analysis procedures, The Qualitative Report, 11, 474-498.
Osborne, J. W., Christensen, W. R., & Gunter, J. (April, 2001). Educational psychology from a
statistician’s perspective: A review of the power and goodness of educational
psychology research. Paper presented at the national meeting of the American
Education Research Association, Seattle, WA.
Osborne, J., & Waters, J. (2002). Four assumptions of multiple regression that researchers
should always test. Practical Assessment, Research & Evaluation, 8(2). Retrieved
from http://PAREonline.net/getvn.asp?v=8&n=2
Palincsar, A. S., Magnusson, S. J., Pesko, E., & Hamlin, M. (2005). Attending to the nature of
subject matter in text comprehension Children's Reading Comprehension and
Assessment, 257. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension
and assessment (pp. 257–278). Mahwah, NJ: Erlbaum.
Paris, S. G., & Stahl, S. A. (2005). Children's reading comprehension and assessment..
Routledge. Mahwah, NJ: Erlbaum.
http://books.google.com/books?hl=en&lr=&id=hhLS9ax8IocC&pgis=1
Paris, S. G., Wasik, B. A., & Turner, J. C. (1991). The development of strategic readers. In R.
Barr, M. L. Kamil, P. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading
research:Volume II (pp. 609-640). White Plains, NY: Longman.
Partnership for 21st Century Skills (2008). 21st century skills, education and competitiveness: A
resource and policy guide. Tuczon, AZ: Author. Retrieved from
http://www.p21.org/storage/documents/21st_century_skills_education_and_competitiven
ess_guide.pdf
239
Pearson, P. D. (1982). A context for instructional research on reading comprehension (Technical
Report No. 230) Cambridge, MA: Bolt, Beranek and Newman.
Pearson, P. D., Hansen, J., & Gordon, C. J. (1979). The effect of prior knowledge on young
children's comprehension of explicit and implicit information. Journal of Reading
Behavior, 11, 201-209.
Perfetti, C. A., Britt, M. A., Rouet, J.-F., Georgi, M. C., & Mason, R. A. (1994). How students
use texts to learn and reason about historical Uncertainty. In J. F. Voss & M. Carretero
(Eds.), Cognitive and instructionalpProcesses in history and social sciences (pp. 257-
284). Mahwah, N.J.: Lawrence Erlbaum Associates.
Peterson, R. A. (1994). A meta-analysis of Cronbach's coefficient alpha. Journal of Consumer
Research, 381-391.
Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis. The use of
factor analysis for instrument development in health care research. Thousand Oaks, CA:
Sage.
Pressley, M., & Afflerbach, P. (1995). Verbal protocols for reading: The nature of
constructively responsive reading. Hillsdale, NJ: Erlbaum.
Princeton Survey Research Associates International. (2005). Leap of faith: Using the Internet
despite the dangers. Princeton, NJ: Author. Retrieved from
http://www.consumerwebwatch.org/pdfs/princeton.pdf
Quintana, C., Zhang, M., & Krajcik, J. (2005). A framework for supporting metacognitive
aspects of online inquiry through software-based scaffolding. Educational Psychologist,
40, 235–244.
240
Rachlin, H. (1989). Judgment, decision, and choice: A cognitive/behavioral synthesis. New
York, NY: W. J. Freeman.
Rideout, V. J., Foehr, U. G., & Roberts, D. F. (2010). Generation M2: Media in the lives of 8- to
18-year-olds: A Kaiser family foundation study. Retrieved from the Henry J. Kaiser
Family Foundation website: http://www.kff.org/entmedia/upload/8010.pdf
Rieh,S.Y. (2002). Judgment of information quality and cognitive authority in the Web. Journal
of the American Society for Information Science and Technology, 53, 145–161.
Rieh, S. Y., & Belkin, N. J. (1998). Understanding judgment of information quality and
cognitive authority in the WWW. In C. M. Preston (Ed.), Proceedings of the 61st Annual
Meeting of the ASIS (pp. 279-289). Silver Spring, MD: American Society for Information
Science.
Robinson, H. M. (1964). Developing critical readers. In R. G. Stauffer (Ed.), Dimensions of
critical reading: Proceedings of the Annual Education and Reading Conferences, (pp. 1-
12). Newark, DE: University of Delaware.
Rouet, J., Britt, M. A., Mason, R. A., & Perfetti, C. A. (1996). Using Multiple Sources of
Evidence to Reason About History. Journal of Educational Psychology, 88, 478-493.
Rouet, J. F., Ros, C., Goumi, A., Macedo-Rouet, M., & Dinet, J. (2011). The influence of surface
and deep cues on primary and secondary school students' assessment of relevance in Web
menus. Learning and Instruction, 21, 205-219.
Rubio, D. M., Berg-Weger, M., Tebb, S. S., Lee, E. S., & Rauch, S. (2003). Objectifying content
validity: Conducting a content validity study in social work research. Social Work
Research, 27, 94-104.
Russell, D. (1961). Children learn to read (2nd ed.). Boston, MA: Ginn.
241
Sadler, D. R. (2002). Learning dispositions: Can we really assess them? Assessment in
Education: Principles, Policy & Practice, 9, 45-51.
Sanchez, C.A., Wiley, J., & Goldman, S. R. (2006). Teaching students to evaluate source
reliability during Internet research tasks. In S. A. Barab, K.E. Hay, & D.T. Hickey (Eds.).
Proceedings of the seventh international conference on the learning sciences (pp. 662-
666). Mahwah, NJ: Erlbaum.
Schmar-Dobler, E. (2003). Reading on the Internet: The link between literacy and technology.
Journal of Adolescent & Adult Literacy. [Online Serial]. Retrieved from
http://www.readingonline.org/newliteracies/jaal/9-03_column
Shanahan, T., & Shanahan, C. (2008). Teaching disciplinary literacy to adolescents: Rethinking
content-area literacy. Harvard Education Review. 78, 40-61.
Shirky, C. (1995). Voices from the Net. Emeryville, CA: Ziff-Davis Press.
Snow, C. (2002). Reading for understanding: Toward an R&D program in reading
comprehension. New York: Rand Corporation.
Snow, C. E., & Biancarosa, G. (2003). Adolescent literacy and the achievement gap: What do we
know and where do we go from here?. New York, NY: Carnegie Corporation.
Soper, D. S. (2010). The Free Statistics Calculators Website. Retrieved from
http://www.danielsoper.com/statcalc/
Spache, G. D. (1964). Reading in the elementary school. Boston, MA: Allyn & Bacon.
Spilich, G. J., Vesonder, G. T., Chiesi, H. L., & Voss, J. F. (1979). Text processing of domain-
related information for individuals with high and low domain knowledge. Journal of
Verbal Learning and Verbal Behavior 18, 275-290.
242
Spiro, R. (2004). Principled pluralism for adaptive flexibility in teaching and learning to read. In
R. B. Ruddell & N. Unrau (Eds.), Theoretical Models and Processes of Reading (pp. 654-
659). Newark, DE: International Reading Association.
Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson, D. K. (1988). Cognitive flexibility
theory: Advanced knowledge acquisition in ill-structured domains (Tech. Rep. No. 441).
Urbana-Champaign, IL: University of Illinois, Center for the Study of Reading. Retrieved
from http://eric.ed.gov
Spiro, R. J., Feltovich, P. J., Jacobson, M. I., & Coulson, R. L. (1991). Cognitive flexibility,
constructivism, and hypertext: Random access instruction for advanced knowledge
acquisition in ill-structured domains. Educational Technology, 35, 24-33.
Spiro, R. J., Feltovich, P. L., Jacobson, M. J., & Coulson, R. L. (1992). Cognitive flexibility,
constructivism, and hypertext: Random access instruction for advanced knowledge
acquisition in ill-structured domains. In T. Duffy & D. Jonassesn (Eds.), Constructivism
and the technology of instruction (pp. 57-76). Hillsdale, NJ: Erlbaum.
Spiro, R. J. & Jehng, J-C. (1990). Cognitive flexibility and hypertext: Theory and technology for
the non-linear and multidimensional traversal of complex subject matter. In D. Nix & R.
Spiro (Eds.), Cognition, education, and multimedia: Exploring ideas in high technology
(pp. 163-206). Hillsdale, NJ: Erlbaum.
Spiro, R., Vispoel, W., Schmitz, J., Samarapungavan, A., & Boerger, A. (1987). Knowledge
acquisition for application: Cognitive flexibility and transfer in complex content domains.
[Electronic Version] In B. Britton & S. Glynn (Eds.), Executive control processes in
reading (pp. 177–199). Hillsdale, NJ: Erlbaum.
243
Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the WWW: The role of
metacognition in the formation of documents models. International Journal of Computer-
Supported Collaborative Learning, 2, 191-210.
Stadtler, M., & Bromme, R. (2008). Effects of the metacognitive computer-tool met.a.ware on
the web search of laypersons. Computers in Human Behavior, 24, 716-737.
Stahl, S. A., Hynd, C. R., Britton, B.K., McNish, M. M., & Bosquet, D. (1996). What happens
when students read multiple source documents in history?. Reading Research Quarterly,
31, 430–456.
State of Connecticut Department of Education. (2009). Connecticut's Social Studies
Frameworks. Hartford, CT: Author.
Street, B. (2003). What's 'new' in New Literacy studies? Critical approaches to literacy in theory
and practice. Current Issues in Comparative Education. 5(2), 77- 91.
Strømsø, H. I., & Bråten, I. (2009). Beliefs about knowledge and knowing and multiple‐text
comprehension among upper secondary students. Educational Psychology, 29, 425-445.
Stone, J. C. (2007). Popular websites in adolescents‘ out-of-school lives: Critical lessons on
literacy. In M. Knobel & C. Lankshear (Eds.), A new literacies sampler (pp. 49-65). New
York, NY: Peter Lang.
Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston, MA:
Allyn & Bacon.
Tashakkori, A., & Teddlie, C. (2008). Foundations of mixed methods research: Integrating
quantitative and qualitative approaches in the social and behavioral sciences. Thousand
Oaks, CA: Sage.
244
Tate, M., & Alexander, J. (1996). Teaching critical evaluation skills for World Wide Web
resources. Computers in Libraries, 16(10), 49-52.
Taylor, R. S. (1986). Value-added processes in information systems. Norwood, NJ: Ablex.
Tillman, H. (2003). Evaluating quality on the net. The selected works of Hope Tillman.
(Unpublished manuscript) Retrieved from http://www.hopetillman.com/findqual.html
Tillotson, J. (2002). Web site evaluation: A survey of undergraduates. Online Information
Review, 26, 392-403.
Tsai, M. J., & Tsai, C. C. (2010). Junior high school students’ Internet usage and self-efficacy: A
re-examination of the gender gap. Computers & Education, 54, 1182-1192.
Tuckman, B. W. (1993). The coded elaborative outline as a strategy to help students learn from
text. The Journal of Experimental Education, 62, 5-13.
VanSledright, B. (2002). Fifth graders investigating history in the classroom: Results from a
researcher-practitioner design experiment. The Elementary School Journal, 103, 131-160.
Wallen, E., Plass, J. L., & Brünken, R. (2005). The function of annotations in the comprehension
of scientific texts: Cognitive load effects and the impact of verbal ability. Educational
Technology Research and Development, 53(3), 59-71.
Wilson, B. G., & Myers, K. M. (2000). Situated cognition in theoretical and practical context. In
D. H. Jonassen & S. M. Lund (Eds.), Theoretical foundations of learning environments
(pp. 57-88). Mahwah, NJ: Erlbaum.
Wilson, P. (1983). Second-hand knowledge: An inquiry into cognitive authority. Westport, CT:
Greenwood Press.
245
Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in
the evaluation of documentary and pictorial evidence. Journal of Educational
Psychology, 83, 73-87.
Wolfe, M. B. & Goldman, S. R. (2005). Relationships between adolescents’ text processing and
reasoning. Cognition & Instruction, 23, 467-502.
Wolf, W., King, M. L., Huck, C. S. (1968). Teaching critical reading to elementary school
children Reading Research Quarterly, 3, 435-498.
Zawilinski, L. (2011). An exploration of a collaborative blogging approach to literacy and
learning: A mixed method study. (Unpublished doctoral dissertation). University of
Connecticut, Storrs.
Zawilinski. L., Carter, A., O’Byrne, I. W., McVerry, G., Nierlich, T., & Leu, D. (2007,
December). Toward a taxonomy of online reading comprehension strategies. Paper
presented at the 57th Annual National Reading Conference. Austin, TX.
Recommended