The Internet and Adolescent Readers: Exploring ... - CORE

University of ConnecticutOpenCommons@UConn

Doctoral Dissertations University of Connecticut Graduate School

5-10-2013

The Internet and Adolescent Readers: ExploringRelationships Between Online ReadingComprehension, Prior Knowledge, CriticalEvaluation, and Dispositions.J. Gregory McVerry JrSouthern Connecticut State University, [email protected]

Follow this and additional works at: https://opencommons.uconn.edu/dissertations

Recommended CitationMcVerry, J. Gregory Jr, "The Internet and Adolescent Readers: Exploring Relationships Between Online Reading Comprehension,Prior Knowledge, Critical Evaluation, and Dispositions." (2013). Doctoral Dissertations. 120.https://opencommons.uconn.edu/dissertations/120

http://lib.uconn.edu/?utm_source=opencommons.uconn.edu%2Fdissertations%2F120&utm_medium=PDF&utm_campaign=PDFCoverPages

http://lib.uconn.edu/?utm_source=opencommons.uconn.edu%2Fdissertations%2F120&utm_medium=PDF&utm_campaign=PDFCoverPages

https://opencommons.uconn.edu?utm_source=opencommons.uconn.edu%2Fdissertations%2F120&utm_medium=PDF&utm_campaign=PDFCoverPages

https://opencommons.uconn.edu/dissertations?utm_source=opencommons.uconn.edu%2Fdissertations%2F120&utm_medium=PDF&utm_campaign=PDFCoverPages

https://opencommons.uconn.edu/gs?utm_source=opencommons.uconn.edu%2Fdissertations%2F120&utm_medium=PDF&utm_campaign=PDFCoverPages

https://opencommons.uconn.edu/dissertations?utm_source=opencommons.uconn.edu%2Fdissertations%2F120&utm_medium=PDF&utm_campaign=PDFCoverPages

https://opencommons.uconn.edu/dissertations/120?utm_source=opencommons.uconn.edu%2Fdissertations%2F120&utm_medium=PDF&utm_campaign=PDFCoverPages

TheInternetandAdolescentReaders:ExploringRelationshipsBetweenOnlineReading

Comprehension,PriorKnowledge,CriticalEvaluation,andDispositions.

J.GregoryMcVerryPh.D

UniversityofConnecticut,2013

Thepurposeofthismixedmethodsstudywastoinvestigatetherolethatbackground

knowledge,criticalevaluationofinformation,andareader’sdispositionsplayinpredictingonline

readingcomprehensionperformance,duringcomprehensiontasksthattakeplaceineitherless

restrictedormorerestrictedinformationspaces.

Sequentialregressionmodelsdemonstratedthat,aftercontrollingforverbalintelligence,

criticalevaluationandpriorknowledgeweresignificantinboththelessrestrictedinformation

spaceandthemorerestrictedinformationspace.Scoresonadispositionmeasurewereonly

significantinthemorerestrictedmodel.

Qualitativeanalysis,usingverbalprotocolmethods,foundthatwerekeyoveralldifferences

inhowskilledonlinereadersnavigateandmonitormeaningduringInternetinquirytasks.Skilled

readersengageinstrategictextassembly.Howeverallparticipantswerenotsuccessfulat

evaluatingorcommunicatingonlineinformation.

Theresultsofthisstudycontributetobothresearchandpractice.Forresearch,theresults

informricherandmorecomplexmodelsofonlinereadingcomprehension.Forpractice,theresults

informteacherschargedwithteachingliteracyinaconstantlyevolvingworld,oneinwhichthe

Internetisincreasinglyimportanttobothreadingandlearning.

ii



J.GregoryMcVerry

B.A.,UniversityofHartford,1999

M.Ed.,UniversityofHartford,2001

ADissertation

SubmittedinPartialFulfillmentofthe

RequirementsfortheDegreeof

DoctorofPhilosophy

atthe

UniversityofConnecticut

2013

iii

CopyrightbyJohnGregoryMcVerry

2013

iv

APPROVALPAGE

DoctorofPhilosophyDissertation



Presentedby

JohnGregoryMcVerry

MajorAdvisor_____________________________________________________

DonaldJ.Leu

AssociateAdvisor_____________________________________________________

DouglasK.Hartman


MichaelFagella‐Luby


DouglasKaufman

UniversityofConnecticut

2013

v

ACKNOWLEDGEMENTS

Manyhandshelpedwritethisdissertation.Firsttomylovingwifewhoputupwith

countlesshoursofsleeplessnights,groggymornings,andoccasionaltenseonewaynegotiations

withtechnology.Youhavealwaysbeenmydrivingforce.

Tomychildren,whoprovidedmycenter,wewrotethisforyou.Itisfittingthatontheday

mysecondsonwasbornmyfatherdrovemefromthehospitaltoaschoolfordatacollection,anda

weekbeforeIdefendedwewelcomedourthirdsonintothisworld.Asafamilywecamefullcycle

andallthewhile,John,mynamesake,youwerealwaystheretosneakintomyofficetohelpme

work.

Iameternallygratefultomyfriend,colleague,andadvisorDon.Youtaughtmeinnumerable

lessonsaboutresearch,teaching,andmostimportantlywriting.Ioweyouadebtofgratitudefor

thecountlesshoursofrevisionsandsupport.

AspecialthankstoIanwhostartedandfinishedthisjourneywithme.Wehavespent

countlesshoursdiscussingvitalissuesthatfaceourfield,ourfatherhood,andourfuture.Ithank

youforbeingthereinclassandhelpingmewhentheendgameseemedinsurmountable.

IalsooweaspecialthankstoSue.Youkeptmegroundedinwhatistrulyimportantin

education.Iwanttooffermyheartfeltgratitudeforthecountlesshoursyouspenteditingmywork.

Iwouldalsoliketothankthemembersofmydissertationcommittee.DougKaufman,you

wereaninvaluablesupportwhentimesweretoughanditseemedthedeckswerestackedagainst

me.DougHartmanIhavealwayslookedtoyouasasourceofintellectualguidance.Randyouhave

alwaysguidedmythinkingaboutthinking.Michaelyourinsightintoadolescentliteracywasa

valuedasset.

Finallythankstothemanyfriendsandfamilymemberswhostoodbymeduringthis

process.YouhavehelpedguidemeinmorewaysthanIcanlisthere.Ilookforwardtothankingand

celebratingwitheveryoneofyou.

vi

TableofContents

TableofContents....................................................................................................................................................................vi

ListofFigures...........................................................................................................................................................................ix

ListofTables...............................................................................................................................................................................x

CHAPTERI..................................................................................................................................................................................1OVERVIEWOFTHESTUDY............................................................................................................................................1Introduction..........................................................................................................................................................................1BackgroundoftheStudy.................................................................................................................................................3MoreRestrictedandLessRestrictedInformationSpaces................................................................................4PriorKnowledge.................................................................................................................................................................5CriticalEvaluationofTexts............................................................................................................................................6Dispositions..........................................................................................................................................................................6VerbalIntelligence.............................................................................................................................................................7ResearchQuestions...........................................................................................................................................................8Methods..................................................................................................................................................................................9SettingsandParticipants...........................................................................................................................................9

Materials..............................................................................................................................................................................11Procedures..........................................................................................................................................................................12Analysis................................................................................................................................................................................12SignificanceoftheStudy...............................................................................................................................................13

CHAPTERII..............................................................................................................................................................................14LITERATUREREVIEW....................................................................................................................................................14Introduction........................................................................................................................................................................14TheoreticalPerspectives...............................................................................................................................................14CognitiveFlexibilityTheory...................................................................................................................................15NewLiteracies..............................................................................................................................................................19

PriorResearch...................................................................................................................................................................24ResearchInOnlineReadingComprehension..................................................................................................24Researchonpriorknowledge................................................................................................................................31ResearchonCriticalEvaluation............................................................................................................................35ResearchonDispositions.........................................................................................................................................51ResearchonVerbalIntelligence...........................................................................................................................54

ChapterSummary............................................................................................................................................................56

CHAPTERIII.............................................................................................................................................................................59MethodsandProcedures..............................................................................................................................................59QuantitativeProcedures...............................................................................................................................................60SettingandParticipantSelection.........................................................................................................................60QuantitativeMeasures..............................................................................................................................................61

QuantitativeProcedures...............................................................................................................................................88AdministeringtheIndependentVariableMeasures.....................................................................................88AdministeringtheOnlineReadingComprehensionAssessments.........................................................89

QuantitativeAnalysis......................................................................................................................................................91QualitativeProcedures...................................................................................................................................................93

vii

QualitativeParticipants............................................................................................................................................93QualitativeProcedures.............................................................................................................................................94QualitativeAnalysis....................................................................................................................................................95

ChapterIV..............................................................................................................................................................................102QuantitativeResults.....................................................................................................................................................102DataScreening..........................................................................................................................................................103DescriptiveStatistics..............................................................................................................................................108ResultsForResearchQuestionOne:OnlineReadinginaLessRestrictedInformationSpace113ResultsForResearchQuestionTwo:OnlineReadinginaMoreRestrictedInformationSpace.........................................................................................................................................................................................116

ChapterSummary.........................................................................................................................................................120

ChapterV................................................................................................................................................................................124QualitativeResults........................................................................................................................................................124QualitativeAnalysis:ATwo‐StageApproach....................................................................................................126StageOneThemes.........................................................................................................................................................127Themeone:OverallORCAscoresappearedtoberelatedtostrategictextassembly.................129Themetwo:OverallORCAscoresappearedtoindicatealackofsourceevaluationknowledgeamongparticipants.................................................................................................................................................139Themethree:StudentswhorecalldetailsfrommemoryappeartoperformbetterontheORCAtasks...............................................................................................................................................................................150Themefour:Studentsappearedunpreparedtoengageinthecommunicationstrategiesnecessaryforonlinereadingcomprehensioninacademicsettings...................................................154

StageOneSummary.....................................................................................................................................................162StageTwoThemes........................................................................................................................................................164WhatPatternsofOnlineReadingComprehensionStrategiesAppearAmongHighandLowPerformingOnlineReadersDuringanOnlineReadingComprehensionTaskWithinaLessRestrictedInformationSpace?...........................................................................................................................165WhatPatternsofOnlineReadingComprehensionStrategiesAppearAmongHighandLowPerformingOnlineReadersDuringAnOnlineReadingComprehensionTaskwithinaMoreRestrictedInformationSpace?...........................................................................................................................177

StageTwoSummary....................................................................................................................................................185ChapterSummary.........................................................................................................................................................187

ChapterVI..............................................................................................................................................................................190Introduction.....................................................................................................................................................................190DiscussionoftheQuantitativeResults.................................................................................................................191OnlineReadingComprehensionAssessment:DescriptiveStatistics.................................................191SummaryoftheLessRestrictedModel:ResearchQuestionOne........................................................192SummaryoftheMoreRestrictedModeltoAddressResearchQuestionTwo................................194ExploringTheImplicationofBothModels....................................................................................................195

DiscussionoftheQualitativeResults....................................................................................................................202QualitativeResults:DiscussionofStageOneFindings.............................................................................203QualitativeResults:DiscussionofStageTwoFindings............................................................................209ExploringTheImplicationsOfDifferentPatternsOfProcessingDueToTheInformationSpace.........................................................................................................................................................................................212

Limitations.......................................................................................................................................................................213Instrumentation........................................................................................................................................................214MissingDataandSampleSize.............................................................................................................................216ResearcherBiasinQualitativeAnalysis.........................................................................................................217

viii

AddressingtheGrowingChallengesofOnlineReadingComprehension..............................................217

References:............................................................................................................................................................................221AppendixA:OnlineReadingComprehensionAssessment..........................................................................246AppendixBORCAProtocol.......................................................................................................................................310AppendixC:CriticalOnlineInformationLiteraciesAssessmentItems..................................................313AppendixD:DispositionsofOnlineReadingComprehensionAssessment..........................................340AppendixE:OnlineReadingComprehensionAssessment..........................................................................341

ix

ListofFigures

Figure3.1SampleVerbalComprehensionItems………………………………………...……………………………….62Figure3.2SingleScreenShotExamplefromCOILAssessment……………………………………....…………….64Figure3.3FourScreenShotsExamplefromCOILAssessment…………………………………………………….65Figure3.4FourHyperlinksScreenShotExamplefromCOILAssessment…………………………………….66Figure3.5ScreenShotfromHyperResearch……………………………………………………………………………......99Figure4.1ScatterplotsandHistogramsofResidualPlots…………………………………………………………..106Figure4.2BoxPlotofCook’sDistance………………………………………………………………………………………107Figure5.1ScreenShotofDiscussionBoardDirections........................................................................................156Figure5.2ScreenShotofDiscussionBoardTopics................................................................................................156Figure5.3ScreenShotofWomenintheRevolutionDiscussion......................................................................157

x

ListofTables

Table2.1Descriptive,Validity,andReliabilityofScoresonORCA‐Openassessments…….......................…29Table3.1DemographicDataofSchool……...………………………………………………………………................……..61Table3.2BackgroundknowledgeResponsesTothePrompt:”ListeverythingyouknowabouttheAmericanRevolution”……………………………………......................……….……………………………………………..................…….…63Table3.3Sub‐ConstructsofCredibility………….…………………………………………………................………………66Table3.4Sub‐constructsofRelevancy………..………………………………………………………................……….…….....67Table3.5ResultsoftheContentValidationStudy……….……..………………………………................……………...69Table3.6RevisionsbasedontheContentValidationoftheCOIL……...……….…..……................………...…….72Table3.7ContentValidationResultsforRevisedCOIL.............................................................................................75Table3.8CognitiveLabItemDiscrimination................................................................................................................77Table3.9FinalItemsinValidatedandReliableCOIL................................................................................................79Table3.10Descriptive,Validity,andReliabilityScoresonORCAassessments................................................82Table3.11TaskIntroductionfortheORCAInternetInquiryTasks.....................................................................83Table3.12InternalConsistencyoftheORCA‐LR.........................................................................................................91Table3.13InternalConsistencyoftheORCA‐MR........................................................................................................91Table3.14QualitativeParticipants..................................................................................................................................94Table3.15InitialCodebookforAbductiveAnalysis....................................................................................................96Table4.1SummaryofMissingData...............................................................................................................................104Table4.2Range,Means,andStandardDeviationsforDependentandIndependentVariables.............105Table4.3SkewnessandKurtosisRatiosforDependentandIndependentVariables..................................105Table4.4TransformedBackgroundknowledgeIndependentVariable...........................................................106Table4.5MeanscoresofL,E,S,Cperformance...........................................................................................................108Table4.6BivariateCorrelationsbyItemType..........................................................................................................109Table4.7FrequencyofBackgroundknowledgeScores..........................................................................................111Table4.8DescriptiveStatisticsforCOILItems..........................................................................................................112Table4.9N,MinimumandMaximumScores,Means,StandardDeviations(SD),forDependentandIndependentVariables........................................................................................................................113Table4.10BivariatecorrelationsofDependentandIndependentVariables................................................114Table4.11ResultsofHierarchicalRegressionfortheLessRestrictedORCA.................................................115Table4.12ResultsofHierarchicalRegressionfortheMoreRestrictedORCA...............................................117Table4.13Pearson’srCorrelationsBetweenVerbalIntelligenceandORCAitems...................................118Table4.14Pearson’srcorrelationsbetweenCOILandORCAitems..............................................................119Table4.15Pearson’srcorrelationsbetweenCOILandORCAitems..................................................................119Table4.16Pearson’srcorrelationsbetweenCOILandORCAitems..................................................................120Table5.1QualitativeParticipants..................................................................................................................................126Table5.2FrequencyofNavigationalStrategies........................................................................................................130Table5.3FrequencyofStrategiesUsedtoReadWebsites.....................................................................................133Table5.4NavigationStrategiesUsedWhenTakingaPosition..........................................................................137Table5.5FrequencyofStrategiesUsedtoIdentifyanAuthor.............................................................................141Table5.6FrequencyofStrategiesusedtoJudgeAuthorExpertise....................................................................144Table5.7FrequencyofUsingaStrategyofRecallingDetailsFromMemory................................................151Table5.8DiscussionBoardResponses..........................................................................................................................158Table5.9FrequencyofStrategiesUsedtoDevelopKeywords.........................................................................167Table5.10FrequencyofStrategiestoReadSearchEngineResults..................................................................171Table5.11FrequencyofNavigationStrategiesWhenReadingSearchEnginesDuringtheLessRestrictedTask........................................................................................................................................................................174

xi

Table5.12FrequencyofRelevancyJudgmentsonLessRestrictedTasks........................................................176Table5.13FrequencyofLocatingStrategiesontheMoreRestrictedORCA..................................................179Table5.14FrequencyofStrategyUseonMoreRestrictedLocatingTasks....................................................184

1

CHAPTERI

OVERVIEWOFTHESTUDY

Introduction

TheInternetisquicklybecominganimportantnewcontextforreading.Notoolforliteracy

hasspreadquickerandfasterthantheInternet(Coiro,Knobel,Lankshear,&Leu,2008).Withover

twobillionusersonline(MiniwattsMarketingGroup,2012)andagrowthratethatisexponential,

theInternetisshiftingthesocialpracticesofliteracyandlearning(Lankshear&Knobel,2006)

Thus,understandinghowstudentsreadandcomprehendinformationonlineiscrucialforteaching

andlearning.

Afterall,evidenceisemergingthattheInternetisthetextofchoiceforadolescentreaders.

Forexample,adolescentsnowspendmoretimereadingonlinethanoffline(KaiserFamily

Foundation,2005).Furthermore,ina2001survey90%ofstudentswithInternetaccessreported

usingtheInternetforhomework.70%ofthesestudentsrespondedthattheInternetwastheir

primarysourceofinformation(Lenhart,Horrigan,&Fallows,2004).Thesefundamentalshiftsin

theaccess,use,anddisseminationofinformationhaveledresearcherstocallforrichertheoretical

modelsofreadingcomprehensionthataccountfortheadditionalcomplexityofonline

environments(Alexander,2010).




restrictedormorerestrictedinformationspaces.Theresultsofthisstudycontributetoboth

researchandpractice.Forresearch,theresultsinformricherandmorecomplexmodelsofonline

readingcomprehension.Forpractice,theresultsinformteacherschargedwithteachingliteracyin

2

aconstantlyevolvingworld,oneinwhichtheInternetisincreasinglyimportanttobothreadingand

learning(Snow,2002).Thismixedmethodstudyexploredthefollowingresearchquestions:

1. When predicting online reading comprehension of a problem solving task within a less

restricted information space, what is the best fit model, after controlling for verbal

intelligence, of the percentage of variance accounted for by each of the following

variables: background knowledge, critical evaluation, and dispositions of online reading

comprehension?

2. When predicting online reading comprehension of a problem solving task within a more




comprehension.

3. What patterns of processing appear among readers with varying levels of online reading

comprehension ability during an online reading comprehension task within a less

restricted information space?


comprehension ability during an online reading comprehension task within a more


Thesequestionswereinvestigatedinthreephases.Inphaseonetheinstrumentsnecessary

forthestudywerecreated.Inphasetwo,regressionanalysiswasusedinanattempttounderstand

factorsthatexplainedvarianceinscoresofanonlinereadingcomprehensionmeasure.Inphase

threeverbalprotocolmethods(Afflerbach,2010)wereusedtoexplorestrategyuseamong

participantswhovariedintheironlinereadingcomprehensionproficiency.

3

BackgroundoftheStudy

Thelargestreviewofcomprehensionresearchhasconcludedthatthedemandsofonline

readingcomprehensionrequirenovelskillsbeyondthosenecessarytounderstandprintedtext

(Snow,2002).Abodyofevidence(Coiro,2011;Coiro&Dobler,2007;Henry,2006;Kuiper,Volman,

&Terwel,2005)isgrowingtosupportthisclaim.However,thesenascentmodelsofonlinereading

comprehensionneedtobeenrichedifwearetobetterunderstandthediversenatureofreading

thattakesplaceonline.

Simultaneously,astheInternethasbecometheinformationaltextofchoiceformany

students,schoolsacrossthenationhavestruggledtoprovideadolescentswithsuccessfulliteracy

classrooms(Snow&Biancarosa,2003).Thus,scholarshavecalledforanagendato“systematically

linkinstructiontothegrowingknowledgebaseonliteracyandinformitwithup‐to‐datedata

relatingtooutcomesandbestpractices.”(CarnegieCouncilonAdvancingAdolescentLiteracy

[CCAAL],2010,p.10).

Yettheserecentcallstoactionpaylittleattentiontothechangingnatureofreading

comprehension.Infact,thereport“ATimetoAct”(CCAAL,2010)notedthatadolescentsmust

understandlongermorecomplicatedtextswithspecializedvocabularyandcomplexgraphical

representations.Howeverthisreviewofresearchonadolescentliteracydidnotdescribehowthese

challengesarefurthercomplicatedwhenreadingonline(Snow,2002).Effortstoimprove

adolescentliteracymustbegintoincludeonlinereadingcomprehensionifwewishtoprepare

studentsforthereadingdemandsofthe21stcentury.

Previousworkinonlinereadingcomprehensionhasoftenassumedalimiteddefinitionof

onlinereading.Littleworkhasevaluatedonlinereadingcomprehensionfromamorecomplex

perspective–onethatincludesvariationsintheextentoftheinformationspace,background

knowledge,thecriticalevaluationofinformation,orareader’sdispositions.Research(Coiro,2011)

4

Kuiper&Volman,2005;Metzger,2007)suggeststhateachoftheseelementsmaybeimportantto

understandifweexpecttobetterunderstandthecomplexnatureofonlinereadingcomprehension.

MoreRestrictedandLessRestrictedInformationSpaces

TheInternetisnotaunidimensionalcontextforreading;itisacomplexmultifaceted

informationalspace.Inordertoenrichourunderstandinganddevelopnewtheoreticalmodelsof

onlinereading,researchersmustinvestigatehowthedemandsofreadingchangeindifferenttypes

ofreadingcontexts(Hartmanetal.,2010).Previousresearchhasnotalwaysevaluatedonline

readingcomprehensionwithinmultiplecontextssuchaswhenthereadingtasksoccurinmore

restrictedorlessrestrictedinformationspaces.

Oneofthegreatestdemandsplacedonthereaderisdealingwithashiftinginformation

space;whenreadingonline,thesizeoftheinformationfieldconstantlyshiftsbasedonthetaskof

thereader(Leu,2000).Morerestrictedinformationspacesuseasmallerfieldofinformation.Less

restrictedinformationspaceshavealargerfieldofinformation.

Forexamplereadinganarticleonanewspaperwebsitetakesplacewithinamorerestricted

informationspacewhencomparedtoanInternetinquiryonthesearchfortheLochnessMonster.

Havingareaderlookforaspecificwebsite,suchastheNewYorkTimes,alsousesamorerestricted

informationsearchtask.Ontheotherhand,havingreaderslocateanyusefulwebsiteonagiven

topicrequiresalessrestrictedinformationspacethanlocatingaspecificwebsite.Studentswho

havetosiftthroughmanymoresearchresultsandkeywordsearchesmaynotautomatically

identifyausefulsource.Clearlytherearefundamentaldifferencesinthereadingdemandsofmore

orlessrestrictedinformationspaces.

Moststudiesofonlinereadingcomprehensionhavefocusedoneitheramorerestricted

informationspace(Coiro&Dobler,2007)oralessrestrictedinformationspace(Deschryver&

Spiro,2010)withoutrecognizingthepossiblealterationsthismightmaketoreadingperformance.

ForexampleCoiro&Dobler(2007)usedamorerestrictedenvironmentbyhavingstudentslocatea

5

specifictigerwebsiteaspartofaverbalprotocolanalysis.OntheotherhandDeschryverandSpiro

(2010)usedalessrestrictedinformationspacebyhavingstudentsconductanopen‐endedsearch

onclimatechange.Todatenostudyhasexaminedhowareader’sperformancechangesbyvarying

therestrictednatureofinformationinonlinespaces.Itisimportantforbothresearchersand

educatorstounderstandhowthereadingdemandsofonlinereadingcomprehensionshiftbasedon

thenatureoftheinformationspace.

PriorKnowledge

Itisalsoimportanttoinvestigatehowpriorknowledgeaffectsstudentperformanceon

tasksthattakeplaceinbothmorerestrictedandlessrestrictedinformationspaces.Prior

knowledgewasoperationalizedasbackgroundknowledgeinthisstudyasonlydomainspecific

knowledgeabouttheAmericanRevolutionwasmeasured.Otherpriorknowledgesuchas

knowledgeoftheInternetwasnotmeasured.

Thestrongeffectthatbackgroundknowledgehasonreadingcomprehensionisoneofthe

moststablefindingsinallofcognitivepsychology(Paris&Stahl,2005).Evenearlyreading

researchersfromGates(1931),Huey(1908),andGray(1939)notedtherelationshipbetween

backgroundknowledgeandreading.Thereforebackgroundknowledgewasincludedinthe

analysisbecauseithashistoricallybeensuchastrongpredictorinmodelsofofflinereading

comprehension(Alexander&Jetton,2004;Pearson,1982).

Whiletheroleofbackgroundknowledgeinthecomprehensionofofflinetextiswell

established,thereadingcommunityhasonlybeguntoinvestigateroleofbackgroundknowledge

duringonlinereadingcomprehension.HillandHannifin(1997)foundthatpriorknowledgeofboth

thetopicandInternetsystemsarerequiredforsuccessfulonlinereading.CoiroandDobler(2007),

fromtheirthinkaloudswithskilled6thgradeonlinereaders,identifiedfourtypesofprior

knowledgeinvolvedduringonlinereadingcomprehension:topic,informationaltextstructures,

websitestructure,andsearchengines.Coiro(2011usedhierarchicallinearregressionandfound

6

backgroundknowledgetopredictasignificantamountofvarianceineachhierarchicalregression

modelofbothofflineandonlinereadingcomprehension.

CriticalEvaluationofTexts

Inaddition,itisimportanttoinvestigatehowthecriticalevaluationofonlineinformation

affectsstudentperformanceontasksthattakeplaceinmorerestrictedandlessrestricted

informationspaces.Oneofthemoreessentialelementstosuccessfulonlinereadingcomprehension

istheabilitytocriticallyevaluateinformation(Kiili,2008).Inanerawhenpublishinghasbecome

theprovinceofaparticipatoryculture,traditionalmarkersofrelevancyandcredibilityarenot

readilyavailabletoreaders(Metzger,Flanagin,&Zwarun,2003).Furthermorestudiesinvestigating

thecriticalevaluationofwebsitessuggestthatmanystudentsstrugglewiththisimportantaspectof

onlinereadingcomprehension.Collegeageparticipants(Tillotson,2002)andmiddleschool

students(Coiro&Dobler,2007)oftenrelyonsuperficialcontenttoreachtheirdecisionwhen

judgingwebsites.ClearlyreadersmustapproachtheInternetwithacriticaleye.Yetwedonotfully

understandtherolethattheevaluationofinformationplaysduringonlinereadingcomprehension.

Whileevidenceisemergingthatcriticalevaluationiscentraltosuccessfulonlinereading

comprehension,researchersdonotknowhowthisroleshiftswithachanginginformation

landscape.Oneofthefundamentalchallengesstudentsfacewhenreadingonlineisthevastamount

ofinformation(Tate&Alexander,1998)availableduringself‐directedtextconstruction(Coiro&

Dobler,2007).Thus,astheinformationspacebecomeslargerandlessrestrictedreadersmayhave

torelymoreheavilyontheirabilitytojudgetherelevancyandcredibilityofwebsites.Forthese

reasonscriticalevaluationwillbeincludedinthisstudybecause,asaskill,itencapsulatesmanyof

thechallengesstudentsfacewhenreadingonline.

Dispositions

Finally,itisimportanttoinvestigatehowdispositionsofonlinereadingcomprehension

affectstudentperformanceontasksthattakeplaceinbothmorerestrictedandlessrestricted

7

informationspaces.Readingisalwaysasituated,affectiveactivity(Brown,Collins,&Duguid,

1989).Thisrequiresanyadequateattemptatmodelingonlinereadingcomprehensiontoinclude

variables,beyondknowledgeandskills(Carr&Claxton,2002),suchasdispositions.Accordingto

Katz(1993)dispositionsarea“tendencytoexhibitfrequently,consciously,andvoluntarily,a

patternofbehaviorthatisdirectedtowardabroadgoal.”

Asonlinereadersengageinself‐directedtextconstruction(Coiro&Dobler,2007)

dispositionsmaybecomeevenmorecriticalforcomprehensionforseveralreasons.Firstthe

Internetisamuchmorecomplexinformationspace(Katz&Rice,2002;Norris,2001)anditis

unlimitedinnature(Alvermann,2004;Gross,2004).Thesefundamentalchangestotexts,andthe

challengestheypresenttoreaders,suggestthatonlinereadingcomprehensionrequiresemerging

habitsofmind(DeSchryver&Spiro,2010).Furthermoresincelearnerswithpositivedispositions

oftenseekoutchallengingtexts(Guthrie,Wigfield,&Percenevich,2004)readerswithpositive

dispositionstowardsonlinereadingcomprehensionmaybeabletocompletetasksthatrequire

increasedtopicalknowledge.Finallystudentswithapositivedispositiontowardsonlinereading

mayexhibitgreateruseofcognitivestrategiesinonlineenvironments(Coiro,2007).Asaresult,a

measureofreaderdispositionsisincludedintheanalysisbecausetraditionalstudiesinreading

haveestablishedtheimportanceofincludingaffectivevariables(Guthrie&Wigfield,1997)in

readingcomprehensionmodels.

Inshort,thisstudyseekstoexaminesomeofthecomplexitiesthatappeartobeapartof

onlinereadingcomprehension:therelativecontributionsofbackgroundknowledge,critical

evaluation,anddispositionsofthereaderduringtasksthattakeplacewithinmorerestrictedand

lessrestrictedinformationspaces.

VerbalIntelligence

Oneofthefundamentaldifferencesofonlinereadingcomprehensionisthatthesetasksare

drivenbytheirproblemsolvingnature(Leu,O’Byrne,Zawilinski,McVerry,Everett‐Cacopardo,

8

2009).Students,engagedinonlinereadingcomprehensiontasksalsohavetoundertakeself‐

directedtextconstruction(Coiro&Dobler,2007)usingwebsiteswithvaryingdegreesof

readability.Thispresentsauniquechallengetostudyingmodelsofonlinereadingcomprehension,

asstudentswillhavevaryingcognitiveabilitiestocreatethesemultiplesourcetexts.

Itiscommoninthefieldofcognitivesciencetocontrolforverbalintelligenceonsearching

andinformationretrievaltaskstocontrolforthesedifferencesincognitiveabilities(Allen,1992).

Verbalintelligencehasbeenshown,usingfactoranalysis,tobesignificantlyrelatedtomeasuresof

conceptattainmentandinformationprocessing(Lemke,Elmer,Klausmeier,&Harris,1967).

Thereforeinordertounderstandthecontributionthatbackgroundknowledge,criticalevaluation

skills,anddispositionsmaketomodelsofonlinereadingcomprehensionverbalintelligencewillbe

controlled.

ResearchQuestions

Thisstudyusesamixedmethoddesign(Creswell,1994)withbothquantitativeand

qualitativeanalyses.Collins,Onwuegbuzie,andSutton(2006)suggestthatbeforemixedmethods

questionscanbedevelopedthegoalsofthestudymustbeidentified.Thegoalofthisstudywas

bothpredictiveanddescriptiveinnature.

OnwuegbuzieandLeech(2006)definepredictionas“usingpre‐existingknowledgeor

theorytoforecastwhatwilloccuratalaterpointintime.”Thereforethisstudywilldrawonboth

thetheoreticalperspectivesinwhichitisframedandpriorresearchtoinvestigatethe

contributionsthatbackgroundknowledge,criticalevaluationofwebsites,anddispositionsmaketo

onlinereadingcomprehensioninbothlessrestrictedandmorerestrictedinformationspaces.

Describingthedifferencesintheprocessesstudentsuseduringonlinereading

comprehensiontasksisalsoagoalofthisstudy.Verbalprotocols(Pressley&Afflerbach,1995),or

think‐alouds,areoftenusedtodescribethecognitiveprocessesusedduringcomprehension

activities(Campbell,2005).Thesedatacanrevealimportantprocessesusedtocompletethe

9

assessment(Ericsson&Simon,1993)andcanbeadaptedtoalsohelpilluminatetheroleof

backgroundknowledge(Palinscar,Magnusson,Pesko,&Hamlin,2005)duringonlinereading

comprehensiontasks.Consequentiallythisstudyutilizedverbalprotocolanalysistoidentify

patternsofcomprehensionprocessesusedduringanonlinereadingcomprehensionassessment.

Fourresearchquestionsguidedthisstudy:

1. When predicting online reading comprehension of a problem solving task within a

less restricted information space, what is the best fit model, after controlling for

verbal intelligence, of the percentage of variance accounted for by each of the

following variables: background knowledge, critical evaluation, and dispositions of

online reading comprehension?


more restricted information space, what is the best fit model, after controlling for



online reading comprehension.

3. What patterns of processing appear among readers with varying levels of online

reading comprehension ability during an online reading comprehension task within a

less restricted information spaces?



more restricted information space?

Methods

SettingsandParticipants

Thisstudywasconductedwithaconveniencesampleof1317thgradestudents.This

samplewasselectedfromthreeschools.Theschoolswerechosenfromdistrictsthatwere

10

economicallyprivileged,economicallyaverage,andeconomicallychallenged.Oneschoolwas

chosenfromeachSESgroupasdeterminedfromtheConnecticutDistrictReferenceGroups[DRG]

(ConnecticutDepartmentofEducation,2009).Adistrict’sDRGisdeterminedusingavarietyof

economicindicatorsincluding,parentalmedianincome,medianhomevalue,freeorreducedlunch

ratio,parentaleducationlevel,andotherrelatedfactors(ConnecticutDepartmentofEducation,

2009).

Quantitativeparticipants.Theparticipantsinthisstudyinvolved131seventhgraders.

AccordingtoTabachink&Field(2007)asampleof131participantsexceedstheirguidelinesfor

regressionanalysisof100+m,wheremequalsthenumberofpredictors.Thissamplesizeisalso

adequateforaregressionmodelwithonedependentvariableandthreeindependentvariableswith

an=.05,adesiredpowersizeof0.8,andananticipatedeffectsizeof0.15(Sloper,2010).This

anticipatedCohen’sf2isamediumeffect.Theestimateofpowersizewaschosentoensurean

adequateeffect(Sloper,2010).

Qualitativeparticipants.Twelvestudents,fourfromeachparticipatingschool,were

selectedforthequalitativeportionofthestudy.Performanceonthefirstadministrationofthe

onlinereadingcomprehensionassessmentwasusedtodetermineparticipantsinthethinkaloud

activity.Theywereselectedasfollows.

1. First all students in their class were ranked based on scores on the first administration of

the ORCA. Students were divided into two groups: the top 10% of scores and the bottom

10% of scores.

2. These groups were then reviewed by the teacher to allow for the selection of students

who would be comfortable working with an adult on a verbal protocol task.

3. Then four students from each participating school were randomly selected from the list:

two students who scored in the 10% of their class and two students who scored in the

bottom 10% of their class.

11

Then,forthequalitativeanalysis,thetwelveselectedstudentsweredividedintothree

achievement‐levelgroups(highaverage,low)basedonORCAscores,independentofschool.Ten

studentswereincludedinthefinalanalysissincetwostudentshadtoberemovedfromthestudy

duetoerrorsindatacollection.

Materials

Assessmentsusedinthisstudy,exceptverbalintelligence,werecreatedandvalidatedby

theresearcher.Theyincludetwomeasuresofonlinereadingcomprehension:onewithinaless

restrictedcontextandtheotherwithinamorerestrictedcontext.Eachassessmentconsistedoftwo

InternetinquirytasksontheAmericaRevolution.Therewere12scorepointsineachtaskfora

totalof24scorepointsperassessment.

Backgroundknowledgewasestimatedusingathree‐itemtopicalknowledgequestionnaire.

Thequestionnairewasdeliveredusingacomputer‐basedsurvey.Participantswereaskedto

respondtoapromptaskingthemtolistallthefactsordetailstheyknewabouttheAmerican

Revolution.ThesemethodswereadaptedfromCoiro(2011)whoadaptedLeslieandCaldwell’s

(1995)ConceptQuestionTaskandWolfeandGoldman’s(2005)measureoftopic‐specific

backgroundknowledge.

CriticalEvaluationofwebsiteswasmeasuredusingafourteen‐item,multiple‐choicetest,Critical

OnlineInformationLiteracies(COIL).TheCOILfocusedonfourconstructs:authorexpertise,

publisher,evidence,andbias.

TheDispositionsofOnlineReadingComprehensioninstrument(DORC)(O'Byrne&

McVerry,2008)wasusedtomeasurethedispositionsofonlinereadingcomprehensionamong

participants.TheDORCwasacompositescoreoffivescales:reflectivethinking,collaboration,

flexibility,criticalstance,andpersistence.

Verbalintelligencewasmeasuredusingapreviouslyvalidatedassessment,theverbal

comprehensionvocabularytestoftheKitofFactorReferencedCognitiveTest(Ekstrom,French,

12

Harman&Derman,1976).Thistestconsistedofaseriesofvocabularymultiple‐choicequestions.

Themeasurewaschosenbecauseofitsestablishedreliabilityandusebyresearchersstudying

informationalretrievalandlearningfrommultiplesources.Reliabilityestimatesforthetestwhen

workingwith7thgradershaverangedfrom.73‐.86.Theassessmenthasbeenusedbyresearchers

studyinganumberofcontexts:searchpatternsinhypertextwithinCD‐ROMs(Allen,1992);the

learningofsciencecontentwithWebbasedtexts(Wallen,Plass,Brunken,2005);andcomparisons

ofnotetakingstrategieswhilereading(Tuckman,1993).

Procedures

First,participantswereaskedtocompletethemeasureofbackgroundknowledge.Then

studentscompletedthedispositionsofonlinereadingcomprehensionassessmentandtheCOIL

assessment.Next,thestudentscompletedalessrestrictedandamorerestrictedORCAtask.This

wasthefirsthalfofboththelessrestrictedonlinereadingcomprehensionassessmentandthemore

restrictedonlinereadingcomprehensionassessments.ThenthefirsttwoORCAtaskswerescored.

Nextparticipantsforthethink‐aloudswereidentifiedbasedonperformanceonthefirsthalfofthe

ORCA.FinallyallstudentswereadministeredthesecondORCAassessment.

AfterthefirstadministrationoftheORCA,participantsforthethink‐aloudswereidentified.

Studentsthencompletedastructurallypromptedthinkaloudactivity,duringthesecond

administrationofboththelessrestrictedandthemorerestrictedORCA.Studentswerepromptedat

fixedstructurallocationstothinkaloudbytheexperimenters(Afflerbach,2002)whilereading

online.Thestudentswereasked,“Whatwereyouthinking?”atkeydecisionpointssuchasclicking

onalink,leavingawebsite,orenteringkeywords.

Analysis

Toanswerthetwoquantitativequestions,sequential(hierarchical)regressionanalysis

(Tabachink&Field,2007)wasusedtoestimatethebestfitmodelsbetweentheindependent

variableofonlinereadingcomprehensionscoresandthedependentvariablesaftercontrollingfor

13

verbalintelligence.Thiswasdoneforboththelessrestrictedandthemorerestrictedonline

readingtasks.

Toanswermyqualitativequestionsabductive(Onwuegbuzie&Leech,2006)methodsand

constant‐comparative(Bogdan&Biklen,2003;Merriam,1988)methodswere.Thisdualapproach

wasdesignedtoidentifypatternsincognitiveprocessingwhilealsoallowingforthecodebookto

unfoldasthedatawereexamined.

Abductivecodingmethods(Onwuegbuzie&Leech,2006)employbothinductiveand

deductivecodingprocedures.Initialcodingschemeswereinformedbypreviouswork(Leuetal.,

2004;Leuetal.,2009).ThequalitativedatawascodedusingHyperRESEARCH,asoftware‐

packagingtoolthatallowsforthecodingofvideodata.Eachvideocasewasloadedintothe

program.Thevideoswerethenbrokenintoseparateframes.Aunitofanalysisbeganwithstudent

actionortalk.Allrelatedtalkandrepetitiveactionswereincludedinaunit.Theunitendedwhen

talkoractionclearlychanged,suchasclickingon“Go/Search/Enter”orleavingawebsite.

SignificanceoftheStudy

Thisstudycontributestobothliteracyresearchandliteracypractice.Forresearchthis

studyinvestigatedthechangesinthereaderbyfocusingononeofthegreatestchallengesthe

Internetpresentstoreaders:thesizeoftheinformationspace.Byinvestigatingbackground

knowledge,criticalevaluation,anddispositionsofthereaderwhilehavingstudentsreadinboth

morerestrictedandlessrestrictedinformationspacesthisstudyseekstoenrichmodelsofonline

readingcomprehension.

Resultsofthisstudycanalsoofferdirectionstoteachers.Byexaminingthedifferencesin

readingprocessesusedbystudentswhoscorehighonameasureofonlinereadingcomprehension

andthosewhoscorelowthisstudywillidentifysuccessfulstrategiesusedbygoodonlinereaders.

Thisdatacanbeusedbyclassroomteacherswouldwanttomodelandteachtheprocessesusedby

goodreaders.

14

CHAPTERII

LITERATUREREVIEW

Introduction



readingcomprehensionperformanceaftercontrollingforverbalintelligence.Thesevariableswere

investigatedintwodifferentmodels.Thecomprehensiontaskstookplaceineitheralessrestricted

oramorerestrictedinformationspace.Thischapterdefinestherelevanttheoreticalperspectives

andexplainshowtheprinciplesofeachtheoreticalperspectivemightshapethepredictionsofthe

variablesineachmodel.FinallyIexploretheliteratureondefiningandmeasuringeachvariable

includedinthestatisticalmodels.

TheoreticalPerspectives

Thisstudyisframedwithinamultiplerealitiesperspective(Labbo&Reinking,1999).This

perspectivesuggeststhatresearchbenefitswhenweframeourworkwithinmultipletheoretical

frameworkssothatwemightcapturemoreofthecomplexityandrichnessthatsurroundsissuesof

literacyandtechnology.Accordingly,thisstudyembracestwodifferenttheoreticalframeworks:

cognitiveflexibilitytheory(Spiro,2004)andnewliteraciesofonlinereadingcomprehension(Leu

etal.,2004).

Bothperspectiveshavehelpedidentifyelementshypothesizedtobeessentialtoreadingin

onlinespaces.Specifically,CognitiveFlexibilityTheoryhassuggestedtheroleofbackground

knowledgeanddispositionsarecentralforstudentstodevelop“advancedwebskillsandopen

mindsets.”(DeSchryverandSpiro,2010).Anewliteraciesofonlinereadingcomprehension

perspectivehassuggestedthatcriticalevaluationofonlineinformationanddispositionswouldalso

becentraltoonlinereadingcomprehension.InthissectionIexploretheliteraturearoundeachof

15

thetheoreticalperspectives.Ithenexplorehowthesetheoriesinfluencethedependentand

independentvariablesselectedforthisstudy.

CognitiveFlexibilityTheory

Cognitiveflexibilitytheory(Spiro,Feltovich,Jacobson,&Coulson,1991)isaconstructivist‐

learningframework,whichbuildsonpreviousschematheories(i.e.Campione,Shapiro,Brown,

1995;Paris,Wasik,Turner,1991).Cognitiveflexibilitytheoryhasshapedtheprediction,inthis

study,ofhowbackgroundknowledgeanddispositionswouldbeexpectedtofunctionwithinmore

andlessrestrictedinformationspaces.

Cognitiveflexibilitytheorysuggeststhatlearningontheweb,especiallywhenmoving

beyondfactfinding,requiresadvancedskillsandopenmindsets(DeSchryver&Spiro,2010).In

otherwords,tasksthattakeplaceinmorerestrictedinformationspacessuchasfindingatrain

schedulewouldplacedifferentcognitivedemandsonthereaderthananonlineinquiryaroundthe

MiddleEastpeaceprocess,whichtakesplaceinalessrestrictedinformationspace.Thereviewof

cognitiveflexibilitytheoryinformedthevariablesofinterestselectedforthisstudy:onlinereading

comprehension,backgroundknowledge,criticalevaluation,anddispositions.

Cognitiveflexibilitytheoryandonlinereadingcomprehension.Ithasbeenarguedthat

Cognitiveflexibilitytheoryisawell‐suitedperspectivefornonlinearlearninginvolvingrandom

accesstechnologysuchastheInternet(Spiro&Jehng,1990).BasedontheprinciplesofCognitive

flexibilitytheory,onlinereadingcomprehensionmaybedefinedastakingplaceinanill‐structured

context.Ill‐structureddomains,suchastheInternet,requireflexiblelearningbecausetheyarenot

guidedbygeneralizablerules(Spiro,Vispoel,Schmitz,Samarapungavan,&Boerger,1987).Inother

wordsthereisnoonewaytonavigateonlinetexts.Insteadoffollowinglinearpagesstudentsbuild

thetextstheyreadbychoosinghyperlinks(Eagleton&Dobler,2006).Thus,onlinetextsrequire

readerstoactivelyconstructmeaningwithnovelskillsandstrategiesthatdonotapplyto

traditionaltext,inmorestructureddomains(Snow,2002).

16

Cognitiveflexibilitytheoristshavesuggestedthesenewskillsarepartofadomainof

“advancedwebexploration”(Deschryver&Spiro,2010)thatneedtomovebeyondsimplysearch

foranswers.Thesewiderangingsearchesunfoldusing“learner‐initiated,complex,reciprocally

adaptive(LICRA)techniques”(p.4).Theseadvancedtechniques,foundinthoseadeptatonline

readingcomprehension,canbeusedfordeeplearninginaspaceofunfetteredinformationand

access(Deschryver&Spiro,2010).

Cognitiveflexibilitytheoryandbackgroundknowledge.CognitiveFlexibilityTheory

explorestheflexibleuseofbackgroundknowledge.Infactcognitiveflexibilitytheoristsdefine

backgroundknowledgeassomethingthatisconstructedbasedonthesituationratherthan

retrieved(Spiro,Feltovich,Jacobson,Coulson,1992).Whenonline,thereaderissituatedinan

environmentwherethosewithadvancedskillscanaccessunlimitedknowledge.

CognitiveFlexibilityTheorywouldsuggestthatinill‐structureddomainssuchasonline

readingcomprehensionactivationofbackgroundknowledgebecomesmoreproblematicastextual

featureschangeonacase‐by‐casenatureasstudentsreadonline(Spiro&DeSchryver,2010).

Readersmaynolongerrelyontheirtemplatesofwhatwebsitesorargumentslooklike.Infactan

overrelianceonbackgroundknowledgemayleadtogreaterdifficulty.

Theaccesstounlimitedamountsofnon‐linearinformation,accordingtotheprinciplesof

cognitiveflexibilitytheory,alsohasimplicationsforbackgroundknowledgeusewhilereading

online.SpiroandDeschryver(2010)arguethatthenooneknowstherolebackgroundknowledge

willplayinaworldwithsomuchexternalmemorystorage.

Thus,studentswhohaveadvancedskillsforreadingonlinemaybeabletousetheInternet

toovercomealackofbackgroundknowledge.Conversely,studentswithhighbackground

knowledge,butlowonlinereadingcomprehensionskillsmaynotbeabletoactivelyconstruct

backgroundknowledgeduringInternetinquirytasks.Finally,studentswhooverlyrelyontheir

17

backgroundknowledgeoftextsinonlineenvironmentsmaymakemistakesasthesetemplatesdo

notalwaystransfertonon‐linearspaces(Spiro&Jehng,1990).

Therefore,inordertoenrichourunderstandingofmodelsofonlinereadingcomprehension

itisimportanttostudytheroleofbackgroundknowledge.Furthermore,cognitiveflexibilitytheory

wouldsuggestthatbackgroundknowledgewillbehavedifferentlyinmodelsofonlinereading

comprehensiondependinguponhowrestrictedtheinformationspaceisforanytask.Intaskswith

lessrestrictedinformationspacesstudentsmayhavetorelymoreontheir“advancedwebskills”

ratherthantopicalbackgroundknowledge.Inmorerestrictedinformationspaces,topical

backgroundknowledgemaybemoreimportantthan“advancedwebskills.”Thusbackground

knowledge,whilesignificantinbothmodels,wouldpredictmoreofthevarianceinthemore

restrictedmodelasstudentswillneedtorelyontopicalknowledgeratherthan“advancedweb

skills.”

Cognitiveflexibilitytheoryandcriticalevaluation.Theill‐structurednatureofonline

textsisamajorsourceofthechallengesinvolvedinthecriticalevaluationofwebsites.Reihand

Belkin(1998)notedthelackofqualitycontrolleadstotheineffectivenessoftraditionalstrategies

toevaluateanonlinetext.Furthermore,FlanniganandMetzger(2000)commentedthateditorsdo

notvetwebsites,leadingtonewchallenges.Finally,Coiro(2003)describedtheinconsistencyof

textfeaturesanddescribedthetextsontheInternetasintertwinedwithhiddeneconomic,social,

personal,andpoliticalagendas.Thesumofthesechallengesindicatesthatmanyofthestrategies

readersaretaughttocomprehendtraditionaltextsmaynotrigidlytransfertotheill‐structured

natureofonlinetexts.

Secondwhenlearnerstrytoapplyasetofrigidstrategiestoanill‐structureddomain,

errorsofoversimplificationoftenoccur(Spiroetal.,1988).Oversimplificationofknowledgemay

leadtoerrorswhenevaluatingwebsites.Forexamplestudentsmayjudgeawebsiteusing

superficialcontent(Coiro,2011).Thismaybebecauserigidstrategyinstructiontaughtforprinted

18

materialsmayleadtoerrorsofoversimplificationwhenreadingonline.Furthermore,cognitive

flexibilitytheorywouldassertthatastabletaxonomyofskillswouldnotguaranteethetransferof

skillsinnewdomainsbecauseoferrorsinoversimplification.Thismayexplainwhyarecentreview

ofcriticalevaluationresearch(Metzger,2007)foundnoevidencethatchecklists,acommon

interventiontoteachcriticalevaluation,improvedjudgments.

Biasinamentalrepresentationcanalsoleadtoalackoftransferofskillsbetweenill‐

structureddomainsontheInternet.Thereforewhatapersonbelievesaboutatopicmayinfluence

howtheyjudgesources.Forexample,DamicoandBaildon(1998),usingthink–aloudprocedures

duringandanInternetInquiryunitofMexican‐Americanmigration,foundthatbeliefsabouta

domainorsubjectinfluencedtheevaluationofclaimsandevidence.

Thus,cognitiveflexibilitytheorywouldsuggestthatcriticalevaluationscoreswillbe

significantpredictorsinboththelessrestrictedandthemorerestrictedmodels.Studentswho

scorewellonbothoftheORCAassessmentsandthemeasureofcriticalevaluationskillsmayhave

“advancedwebskills.”Conversely,studentswhoscorelowonthecriticalevaluationmeasuremay

bemakingerrorsofoversimplificationorhavebiasintheirmentalrepresentation.

Cognitiveflexibilitytheoryanddispositions.AccordingtoCognitiveFlexibilityTheory

dispositionsofthelearnermayalsobeimportantelementstoincludeinamodelofonlinereading

comprehension.Infacttheoristshavesuggestedan“openmindset”isrequiredforonlinereading

comprehension(Spiro,2004).Afterall,accordingtocognitiveflexibilitytheorists,learnerscannot

betaughttosimplyapplyrigidviewsofknowledgetomultiplesituations.Theymustunderstand

thecomplexnatureofknowledgeandbeabletousenovelwaystolearn.Furthermorecognitive

flexibilitytheoristshavenotedbiasinmentalrepresentationcanleadtoalackoftransferofskills

betweenill‐structureddomains(Spiroetal.,1992).Inotherwordsifreadersdonothaveanopen

epistemologicalbeliefaboutknowledgeconstructiontheymaynotsucceedinanill‐structured

19

spacesuchastheInternet.Theseprinciplessuggestdispositionsarecriticaltoonlinereading

comprehension.

Therefore,cognitiveflexibilitytheorywouldsuggestthatotherfactorsaffectperformance

beyondbackgroundknowledgeandcomprehensionskills.Infactcertainlearnersmayhavehabits

ofthoughts(Spiro,2004)ordispositionsthatallowformoreflexiblelearning(Leu,Kinzer,Coiro,&

Cammack,2004).Thisconstructof“openmindsets”isverysimilartothefivescalesmeasuredin

thedispositioninstrument:reflectivethinking,flexibility,persistence,criticalstance,and

collaboration.Thus,studentswithpositivedispositionstowardsreadingonlinemayhavegreater

successinbothlessrestrictedandmorerestrictedinformationspaces.However,accordingto

cognitiveflexibilitytheoryhavingpositivedispositionstowardonlinereadingcomprehension

wouldexplainagreateramountofvarianceinlessrestrictedinformationspacesthanitwouldin

morerestrictedinformationspaces.Thisisbecausealessrestrictedinformationspacewould

requiremoreflexiblelearning,persistence,andagreatercriticalstanceinordertosiftthroughthe

unfetteredinformationandunstructuredsearchesoftheopenInternet.(Schryver&Spiro,2008).

NewLiteracies

Thisstudyisframedwithinabroadperspectiveofanupper‐casetheoryofNewLiteracies

(Coiroetal.,2008)aswellasamorespecific,lower‐casetheoryofthenewliteraciesofonline

readingcomprehension.Anupper‐caseNewLiteraciestheoryisusedtocapturecommonalities

amongdiverseareasofinquiryinthisareaincludingworkinsocialpractices(Street,2003),

Discourses(Gee&Green,2007),comprehension(Castek,2008),andotherareas.Fourcommon

principlescurrentlyappeartodefineanupper‐casetheoryofNewLiteracies(Coiroetal,2008):

ICTsrequireustobringnewpotentialstotheireffectiveuse.

Newliteraciesarecentraltofullcivic,economic,andpersonalparticipationina

globalizedcommunity.

Newliteraciesaredeicticandchangeregularly.

20

Newliteraciesaremultiple,multimodal,andmultifaceted.

WithinthisbroadlyconceivednotionofNewLiteracies,manyareactivelypursuingmore

specificareasofresearch,informedbyseparatelower‐casetheoriesofnewliteracies(Leu,Kinzer,

Coiro,Castek,&Henry,inpress).

Newliteraciesofonlinereadingcomprehension.Thenewliteraciesofonlinereading

comprehension(Leuetal.,2009),areonesuchlower‐casetheory(Coiroetal.,2008).This

perspectivedefinesonlinereadingcomprehensionasaproblem‐basedlearningprocess,which

includestheskills,strategies,anddispositionsrequiredtolocate,evaluate,synthesize,and

communicateonlineinformation.Thisperspectivesuggeststhattheshiftingnatureofhowtextsare

constructed,evaluated,andaccessedrequiresustoaltertraditionalmodelsofreading

comprehension(Leuetal.,2004).Thetheoryofnewliteraciesofonlinereadingcomprehension

informedtheinclusionoftheindependentvariablesofbackgroundknowledge,criticalevaluation,

anddispositions.

Newliteraciesofonlinereadingcomprehensionandbackgroundknowledge.Thereis

noconsistentbodyofevidenceexplainingtheroleofbackgroundknowledgeduringonlinereading

comprehension.Whileempiricalstudiesofonlinereadingcomprehensionhaveincluded

backgroundknowledgeasavariableinmodelsofonlinereadingcomprehensionresearchershave

justbeguntoexplorethetheoreticalimplications.Overallsomearguethatbackgroundknowledge

willplayadiminishedrole(Coiro,2011).Others(Hartman,Morsnik,&Zheng,2010)arguethatnew

formsofbackgroundknowledgewillbereprioritized.

Coiro(2011)suggestedthatresearchersmustrethinktherolethatbackgroundknowledge

playsduringonlinereadingcomprehension.Shefoundthat“whiletopic‐specificbackground

knowledgeusuallyplaysasignificantroleinmostofflinereadingtasks,itappearedtoplaya

relativelyminorroleinaseriesofthreeonlinereadingtasks,”includedinherstudy(Coiro,2007p.

262).Coirosuggestedtheimportanceofbackgroundknowledgemaychangebasedonthe

21

informationspace.Specifically,Coiroarguedthatthepossibilityexiststhat“backgroundknowledge

isindeedafunctionofthespecificityofonlinetasksorthenon‐linearnatureofonlinetask”(Coiro,

2007p.246).Theseresultssuggestthatinlessrestrictedtasksstudentsmayfinditmoredifficultto

relyontopicalbackgroundknowledgeandbecomemoredependenton“advancedwebskills.”

Hartmanetal.(2010)suggestnewformsofknowledgearerequiredforcognitivemodelsof

onlinereadingcomprehension.Thisnewknowledgeaddsnewcomplexitiesratherthansupplants

traditionalviewsofknowledgethatincluded:declarative,proceduralandconditionalknowledge

(Paris,Wasik,andTurner,1990).

Hartmanetal.(2010)arguethatonlinereadingcomprehensionrequiresgoalknowledge,

locationknowledge,andidentityknowledge.Goalknowledge,orknowingwhy,providesstudents

withacontinuedsenseofpurposeduringonlinereadingcomprehensiontasks.Location

knowledge,orknowingwhere,referstoknowingthelocationofsearchenginefeaturesandthe

basicsofInternetsearching.Identityknowledge,orknowingwho,isknowledgeofthebasicsofhow

authorsconstructandrepresentonlineidentities.

Thereforeatheoryofnewliteraciesofonlinereadingcomprehensionwouldsuggestthat

priortopicalknowledgemaynotcontributethesameamountofvariancetomodelsofonline

readingcomprehensionastheyhaveinthepast.Instead,asstatedbyCoiro(2011)priortopical

knowledge,mayhaveareducedroleor,asstatedbyHartmanetal.(2010)newformsof

backgroundknowledgewillbecomemoreimportant.Basedonthesetheoriespriortopical

knowledgewillplayastrongerroleinmorerestrictedenvironmentsasstudentswillneed

increasedbackgroundknowledgetomakemorefinitejudgments.

Newliteraciesofonlinereadingcomprehensionandcriticalevaluation.Whileanew

literaciesofonlinereadingcomprehensionperspectivesuggeststhatquestioning,locating,

evaluating,synthesizing,andcommunicationareallcentralconstructsforinvestigation,only

criticalevaluationofonlineinformationwaschosenforthisstudy.Therewereseveralreasonsfor

22

thisdecision.First,duetotheself‐directedtextconstruction(Coiro&Dobler,2007)thejudgingof

therelevanceofsources(Braaschetal.,2009)isfundamentalwhenreadingonline.Asreaders

selecttextstoreadtheymustconstantlyjudgetherelevancyandcredibilityofsources.Thus,

criticalevaluationbecomesespeciallyimportantduringonlinereading.Inaddition,whilesome

workisbeginninginthisarea(Damico&Baildon,2007;Kiilietal.,2008;Quintana,Zhang,&

Krajcik,2005)weknowlittleabouttheroleofcriticalevaluationduringonlinereading

comprehension.

Theprinciplesofnewliteraciesofonlinereadingcomprehensionwouldsuggestthata

measureofcriticalevaluationskillswouldexplainasignificantamountofthevarianceinboth

restrictedandlessrestrictedinformationspaces.However,lessrestrictedinformationspaces

requiregreaterself‐directedtextconstructionwhichmakesthecriticalevaluationoftextsmore

important.Inalessrestrictedinformationspacetheparticipantshavetobuildthetextwithno

guidanceandhavetoconsidertherelevancyandreliabilityofthesources.Inthemorerestricted

tasktheparticipantsknowthesourcetheyaretryingtolocate.Thereforereaderswithgreater

criticalevaluationskillsmayhavehighersuccessinlessrestrictedspacesandreaderswithless

proficientskillswillperformworse.Thus,eventhoughatheoryofonlinereadingcomprehension

predictsthatcriticalevaluationskillswillbeasignificantpredictorinbothmodels,thescoreson

thecriticalevaluationmeasuremayhavemorepredictivevaluewithinlessrestrictedinformation

spaces.

Newliteraciesofonlinereadingcomprehensionanddispositions.Anewliteraciesof

onlinereadingcomprehensionperspectivewouldalsosuggestthatdispositionsmaybecentralto

readinginbothlessrestrictedandmorerestrictedonlineinformationspaces.O’ByrneandMcVerry

(2008)identifiedfivelearningdispositionsthatarecentraltoonlinereadingcomprehension:

reflectivethinking,takingacriticalstance,persistence,flexibility,andcollaboration.Whilethese

23

dispositionsmaybeimportantforalllearningtaskstheill‐definednatureoftheInternetmaymake

thesedispositions,orhabitsofmind(Spiro,2004),morecentraltoonlinereadingcomprehension.

Accordingtotheprinciplesofonlinereadingcomprehensionreaderdispositionswould

predictsignificantvarianceinbothlessrestrictedandmorerestrictedinformationspaces.Research

isemergingthatdemonstratesaffectivevariablesarepositivelyrelatedtostrategyusewithin

onlinereadingtasks(Dwyer,2010;Tsai,2004;Tsai&Lin,2004;Hofman,Wu,Krajcik,andSoloway,

2003).InfactLeuetal.(2004)suggestthatnewdispositionsarecentraltomeaningmakingin

onlineenvironments.Thereforescoresonaselfreportmeasureofdispositionsofonlinereading

comprehensionareexpectedtomakeasignificantpredictioninmodelsusingbothlessrestricted

andmorerestrictedinformationspaces.

Thispredictiondiffersfromthatofcognitiveflexibilitytheory.Basedonthetheoriesof

cognitiveflexibilitytheorydispositionswouldbeasignificantpredictorinbothmodels,butabetter

predictorinthelessrestrictedmodel.Basedonthenewliteraciesofonlinereadingcomprehension

dispositionswouldbeasignificantandstrongpredictorinbothmodels.Thisdifferenceis

attributedtothegreateremphasisnewliteraciesofonlinereadingcomprehensionplaceson

dispositionsinalltheoreticaldefinitions.

Summaryoftheoreticalperspectives.Twotheoreticalperspectiveswereusedtoguide

thisstudy:cognitiveflexibilitytheoryandnewliteraciesofonlinereadingcomprehension.These

perspectiveshelpedtoguidetheselectionofvariablesofinterest.Furthermoretheywereusedto

makepredictionsabouthowthevariablesofinterestwillbehaveinmodelsofonlinereading

comprehensionthatusebothalessrestrictedinformationspaceandamorerestrictedinformation

space.

Inthisstudypredictionsfromthetwotheoreticalperspectivesarecloselyaligned.Thetwo

theoreticalperspectivesCognitiveFlexibilityTheoryandnewliteraciesofonlinereading

comprehensiontheoryhavebeenusedtoexplorerecentshiftsinourliteracypractices.Themost

24

importantalignments,intermsofthisstudy,areasignificantbutreducedroleforbackground

knowledgeandtheincreasedimportanceofcriticalevaluation.Theoneareaofdisagreementwas

intheareaofdispositions.Basedontheprinciplesofcognitiveflexibilitytheory,dispositionswould

notplayassignificantaroleinthemorerestrictedinformationspaceasthestructurednatureof

thetaskwouldlimitthepotentialfordeeplearningontheWeb.Conversely,atheoryofnew

literaciesofonlinereadingcomprehension,wouldsuggestthatdispositionsarejustascriticalinthe

morerestrictedspacesastheyareinthelessrestrictedspaces

Cognitiveflexibilitytheoryledtoanumberofpredictions:(a)backgroundknowledge,while

significantinbothmodels,wouldnotbethestrongestpredictorineithermodel;(b)scoresona

measureofcriticalevaluationwouldbesignificantinbothmodels,butpredictmoreofthevariance

inthelessrestrictedmodel;and(c)scoresonaself‐reportofdispositionsmeasurewouldexplaina

significantamountofvarianceinbothmodels,butwouldbeabetterpredictorinthelessrestricted

model.

Inthisstudy,anewliteraciesofonlinereadingcomprehensiontheoryledtoanumberof

predictions:(a)backgroundknowledgewillbesignificant,butnotbeasastrongofapredictorasin

offlinecomprehensionresearch;(b)criticalevaluationskillswillbesignificantinbothmodels,but

explaingreatervarianceinthelessrestrictedmodel;and(c)scoresonaself‐reportofdispositions

measurewouldexplainasignificantamountofvarianceinbothmodels.

PriorResearch

Inadditiontothesetheoreticalperspectives,severalareasofresearchalsoinformedthe

designofthisstudy:(a)onlinereadingcomprehension,(b)backgroundknowledge,(c)critical

evaluation,(d)dispositions,and(e)verbalintelligence.Eachwillbeexploredbelow.

ResearchInOnlineReadingComprehension

Researchsuggeststhatonlinereadingcomprehensionisdifferent,andmaybemore

complex,thantraditionalreadingcomprehension(Coiro&Dobler,2007;Hartmanetal.,2010;Leu

25

etal.,2009).Thesestudieshavefoundseveralcommonresults:scoresonmeasuresofonline

readingcomprehensionarenotisomorphicwithofflinereadingcomprehensionmeasures(Coiro,

2011,Leuetal.,2005);newstrategiesarerequiredinonlinespaces(Chen,2011;Schmar‐Dobler,

2003);newknowledgeandattitudesarerequiredforonlinereadingcomprehension(Bilal,2001;

2002;Deschryver&Spiro,2010);thenatureoftheinformationspacemayaffectthenatureof

onlinereadingcomprehension;andassessmentsofonlinereadingcomprehensionhaveahistoryof

beingvalidandreliable.

Scoresonmeasuresofonlinereadingcomprehensionarenotisomorphicwithoffline

readingcomprehensionmeasures.Leuetal.(2005)suggestthatreadingcomprehensionand

onlinereadingcomprehensionarenotisomorphic.Theyfoundnosignificantstatisticalcorrelation

amongtheDegreesofReadingPower(TouchstoneAppliedScienceAssociates,2004)testanda

validatedmeasureofonlinereadingcomprehensionadministeredto89seventhgradestudents.

Theonlinereadingcomprehensionmeasure(ORCA‐Blog)requiredstudentstolocate,evaluate,

synthesize,andcommunicateinformation.TheDegreesofReadingPower(DRP)test,isacriterion‐

referencedmeasureusedwidelyasameasureofreadingcomprehension.

Coiro(2011)foundthatknowingastudent’sonlinereadingcomprehensionability

predictedasignificantamountofvarianceoverandaboveofflinereadingcomprehensionand

backgroundknowledge,butanadditional16%ofindependentvariancewascontributedby

knowingstudents’onlinereadingcomprehensionability.Participantsincluded118seventh‐grade

studentsfromConnecticut.StudentswereadministeredtwodifferentversionsoftheORCA,one

usedasadependentvariableandtheotherasanindependentvariable.Participantsalsocompleted

abackgroundknowledgemeasure.ScoresontheConnecticutMasteryTest(StateofConnecticut,

2010)wereusedasameasureofreadingcomprehension.Thedatasuggestthatadditionalskillsare

requiredforonlinereadingcomprehension,beyondthoserequiredforofflinereading

26

comprehension.Thesestudiessuggestthattheremaynotbeahighcorrelationbetweenofflineand

onlinereadingcomprehension.

Newstrategiesarerequiredinonlinespaces.Researchhasalsofoundthatonline

readingcomprehensionmayrequirenewskillsandstrategies.Schmar‐Dobler(2003)investigated

strategyuseamongfifthgradersastheysearchedforbothexplicitandimplicitinformationonthe

Internet.Datasourcesincludedobservations,think‐aloudprotocols,andpost‐readinginterviews.

Schmar‐Doblerconcludedstudentsusedmanyofthesamestrategiesusedduringtraditionalonline

readingactivities.However,shenotedthatnewnavigationalstrategieswererequiredtoreadin

onlinespaces.Thisstudysuggeststhatonlinereadingcomprehensionrequiresnewstrategies.

Anotherstudy,usingqualitativethinkaloudmethodswith12proficientsixthgrade

students,concludedthatonlinereadingcomprehensionandofflinereadingcomprehensionare

similar,butonlinereadingcomprehensionwasalsomorecomplex(Coiro&Dobler,2007).Inthis

studystudentscompletedtwoseparateonlinereadingcomprehensiontasks.Thefirsttaskhad

studentsreadwithinawebsite.Thesecondtaskhadstudentsanswerveryspecificquestionsusing

asearchengine.Datasourcesincludedthink‐aloudprotocols,fieldobservations,andsemi‐

structuredinterviews.Theauthorsconcludedthatonlinereadingcomprehensionrequireda

processofself‐directedtextconstructionthataddsadditionalcomplexitiestotraditional

comprehension.

Deschryver(2010)conductedathinkaloudprocedurewithadvancedweblearners,skilled

collegeagestudentsandcomparedtheseresultswithstudiesinvestigatingschoolagechildren.

DeschryverconcludedthatdifferencesexistamongexpertusersoftheInternet.Specifically,

advancedlearnerscansynthesizenewlearningthatmovesbeyondwhatisalreadyknownabouta

topic.

Chen(2010)usingqualitativemethodscomparedtheonlinereadingcomprehension

strategiesofupperelementarystudentswithandwithoutlearningdisabilities.Datasources

27

included:surveys,structuredmetacognitiveinterviews,observations,readingcomprehension

activities,andonlinesearchtasksthatwereadministeredto119studentsinthefifthandsixth

grades.Chenconcludedthatthelackofnavigationalstrategiestodealwithnon‐linearreading

impededcomprehension.

Newknowledgeandattitudesarerequiredforonlinereadingcomprehension.

Researchhasalsodemonstratedthatnewknowledgeandattitudesmayberequiredforonline

readingcomprehension.Bilal(2000;2001)workedwithapproximately25seventhgradersto

explorethecognitive,affective,andpsychomotordomainsoflearningasparticipantssearchedthe

InternetusingYahooligans.BilalcreatedaWebTraversalMeasuretoquantifysearchbehaviors.

Otherdatasourcesincluded:screenshotrecordings,teacherassessmentsofstudentattributesand

studentexitinterviews.Bilalconcludedthatabilitytorecoverfrom‘‘breakdowns,’’navigational

style,andone’sfocusontaskwerekeytosuccessfulonlinereading.Thisindicatesthatonline

readingperformancemaygobeyondskillsandrequirelearnerdispositions.

Insummarythesestudiespresentagrowingcorpusofworkthatreadingcomprehensionis

fundamentallydifferentandmorecomplexinonlinespaces.Specificallynewskills,knowledge,and

dispositionsareneeded.Acrossallofthestudiestherewasanincreasednoteoftherolenavigation

playsduringreadingcomprehension.Manyofthestudiesnotedtheimportanceofgoalknowledge

(Bilal,2000;2001;Schmar‐Dobler,2003)throughselfdirectedtextconstruction(Coiro&Dobler,

2003).

Thenatureoftheinformationspacemayaffectthenatureofonlinereading

comprehension.Severalstudieshaveusedvariouslevelsofarestrictedinformationspaceduring

onlinereadingwithoutcontrollingforthepotentialconsequences(Chen,2010;Coiro&Dobler,

2007;Deschryver,2010;Leuetal.,2005;Schmar‐Dobler;2003).Todatenostudyhasexaminedthe

differencesinperformancebasedontherestrictednatureoftheinformationspaces.

28

Themajorityofstudiesusedamorerestrictedinformationspacetomeasureonlinereading

comprehension.Schmar‐Dobler(2003)usedamorerestrictedinformationspacebylimitingthe

tasktoquestionsfromclass.CoiroandDobler(2007)usedarestrictedspaceofjustasingle

websiteanddirectedquestions.Leuetal.(2005)usedamorerestrictedspacebyhavingstudents

lookforspecificanswerstofocusedInternetqueries.Theassessmentsinthesestudieshad

studentsanswerspecificquestionsandevaluatespecificpagesratherthanconductsearchersfor

topicalinformation.

Fewstudiesutilizedlessrestrictedinformationspaces.Deschryver(2010)usedaless

restrictedinformationspacebyhavingstudentsconductanopen‐endedsearch.Chen(2010)useda

morerestrictedspaceforthinkaloudactivitiesandalessrestrictedspaceforhisquantitative

measures.

Whilepreviousstudieshaveusedavarietyoftypesofinformationspaceintheirresearch,

therearenostudiesontherolethatmoreorlessrestrictedinformationspaceplaysduringonline

readingcomprehensiontasks.Thisstudyseekstoexplorestudentperformanceinbothless

restrictedandmorerestrictedspaces.TherestrictednatureoftheInternetinquirytaskmay

influencetheskills,strategies,anddispositionsofthereader.

Assessmentsofonlinereadingcomprehensionhaveahistoryofbeingvalidand

reliable.Validandreliableassessmentsofonlinereadingcomprehensionhavebeenusedin

previousworkthatattemptedtocapturethenatureofreadingonlinethroughperformancebased

assessmenttasks.ThepurposeofeachORCAmeasurethathasbeenpreviouslyusedwastocapture

“real‐time”onlinereadingproductsandprocessesduringreadingontheopenInternet,adynamic

andunboundeddigitalinformationenvironment.TheseassessmentsincludedORCA‐Instant

Message(ORCA‐IM),ORCA‐Blog,ORCA‐ScenariosIandII,ORCA‐Iditarod,andORCA‐IditarodRevised.

EachoftheseassessmentshasbeenshowntobevalidandreliableasshowninTable2.1.

29

Table2.1Descriptive,Validity,andReliabilityofScoresonORCA‐OpenassessmentsInstrument/#Items

ScoreRanges

N Validity*

Reliability**

ORCA‐IM(12) 3‐36of38 12 66.5% .85‐.95ORCA‐Blog(10) 0‐30of32 89 59.2% .84ORCAScenarioI(20) 0‐56of60 120 51.7% .92ORCAScenarioII(20) 0‐56of60 120 44.1% .91ORCAIditarod(17) 0‐33of42 220 53.1% .88ORCAIditarodRevised(13) 0‐20of30 373 41.6% .73*%ofvarianceexplainedbyacompositescore,ameasureofconstructvalidity.**Cronbach’salphareliabilitycoefficient,ameasureofreliability.

ThefirstattempttocreateavalidandreliablemeasurewastheORCA‐IM(Leuetal,2005).

TocompletetheORCA‐IMstudentsworkedindividuallyonfourInternetreadingtasks.Anonline

researcherintroducedtheactivityoverinstantmessage.Asecondresearchersatwiththestudent

andcollectedfieldnotes.WhileORCA‐IM,resultedinasuccessfulprotocolfordatacollectionthe

amountofresourcesitrequired,tworesearchersperadministration,madetheORCA‐IMverylabor

intensive.

ThenextassessmentdevelopedwastheORCA‐Blog(Leuetal,2005).ThegoalofORCA‐Blog

wastorepeatthedesignofORCA‐IMbutwithproceduresthatallowedscalingupthrough

simultaneousadministration.StudentswhocompletedtheORCA‐Bloghadtoanswerthree

informationrequestspostedonablogsitebyfictitiousteachersrequestingonlineresourcesfor

humanbodysystems.TheORCA‐Blog,capturedarangeofachievementlevelsinonlinereading

comprehension.Howeveritdidnotaccountforthemultipledimensionsofcriticalevaluationorthe

synthesisofmultiplesources.TheORCA‐Bloghadstudentsevaluateasinglewebsiteonoverall

reliability.Furthermorestudentswerenotrequiredtointegratemultiplesourcesintheirsynthesis

tasks.Thesearekeyskillsthatneedtobemeasuredduringonlinereadingcomprehension

assessments.

30

Thenextiterationofassessments,ORCA‐ScenarioI&II,wasdevelopedbyCoiro(2007;

2011).Theinstrumentinvolvedtwoparallelmeasuresofonlinereadingcomprehension.Each

instrumentincluded20open‐endeditemsconstructedtomeasurethestudents’readingability

duringthelocation,evaluation,synthesis,andcommunicationofonlineinformation.The

assessmentwasdeliveredoveraprivatequizinterfacethatrequiredindividualusersignon.The

ORCA‐ScenarioI&IIwerebothvalidandreliablebututilizedaveryrestrictedinformationspacein

ordertoensureaparallelitemformat.Thetesttookplaceinaquizinterfaceandstudentswere

givenverytargetedsearchestocompleteorwereprovidedwebsitestoevaluate.

TwoORCA‐IditarodmeasuresweredevelopedaspartofanIESsupportedresearchgrant

project(TeachingInternetComprehensiontoAdolescents[TICAProject],Leu&Reinking,2005).

Theinstrumentsrequiredstudentstouseanonlineassessmenttool,SurveyMonkey.Thetasks

askedstudentstolocate,evaluate,synthesize,andcommunicateinformation.Theinformation

requestedwasveryrestrictedinnature.Forexamplestudentswereaskedtofindsinglefactssuch

as,“WhoholdstheIditarodrecord?”Alsothecriticalevaluationtaskwasrestrictedtoasingle

website.

Summaryofresearchinonlinereadingcomprehension.Todate,researchinonline

readingcomprehensionisadvancing.Yetuniquechallengesremain.Firstwedonotknowhowthe

restrictednatureoftheinformationspacemayaffectonlinereadingcomprehensionscores.

Furthermoreeffortstocapturethecollaborativenatureofcommunicationduringonlinereading

comprehension,suchasonORCA‐Blogcreatedifficultiesforassessmentincludingecological

validityandcumbersomescoring.Effortsweretriedtoutilizeinstantmessagingandblogging.Yetit

wasimpossibletorecreatethecollaborativenatureofthesecommunicationtools.Thescoringfor

onlinereadingcomprehensionassessments,involvedthecodingofvideoscreencasts.Thismeant

eachassessmentwouldtakefiftyminutestoscore.Finallytheshiftingnatureofonlinewebsites

31

threatensthestabilityoftheassessments.Inmanyoftheseassessments,secondaryreplication

studiescannotbecompletedaswebsitesthatwereusedintheassessmentsarenolongeravailable.

Theassessmentsdesignedforthepresentstudytriedtoaccountfromthelessonslearnedin

pastdevelopment.Firstandforemosttwodifferentmodelswillbetestedtoaccountforshifting

informationspaces.Onemodelwillusealessrestrictedspace.Thesecondmodelwilluseamore

restrictedspace.

Anotherimprovementistorequirethereadingofmultiplesources.Inmanyprevious

versionsoftheORCAstudentsdidnothavetoutilizemultiplesources.Thisdoesnotreflecttrue

onlinereadingcomprehension.Inthisstudyboththelessrestrictedandmorerestrictedtasks

requiredthereadingofmultipleonlinesources.

Inaddition,multipleelementsofsourcingskillswereincludedinthecriticalevaluation

items.Inpreviousstudies,criticalevaluationskillswerelimitedtoexaminingjusttheauthor.This

studyalsoincludestheevaluationoftheevidenceusedbyanauthor.

Finallyeffortsweremadetodrawonthecollaborativenatureofonlinecommunication.In

previousORCAversionscommunicationtaskswerelimitedtoblogsorinstantmessaging.Both

assessmentversionsinthepresentstudyrequiredstudentstoreplytopreviouspostsona

discussionboard.Specificallystudentshavetousetheinformationtheylearntorespondtoanother

post.

Researchonpriorknowledge

Therolethatpriorknowledgeplaysintraditional,offlinereadingcomprehensioniswell

known,stable,andsignificant(Chiesi,Spilich,&Voss,1979;Spilich,Vesonder,Chiesi,&Voss,1979).

Howevertheresultsfromrecentstudiesinvestigatingbackgroundknowledgeandonlinereading

comprehensionhaveshownasomewhatmixedpatternofresults.ThisincludesstudiesofInternet

inquiry(Bilal2000;2001;Hill&Hannafin,1997),non‐linearhypertextreading(Tabati&Shore,

32

2005),andonlinereadingcomprehension.(Coiro,2011;).Thesestudiesdonotprovideconclusive

evidencethatbackgroundknowledgeplaysastrongroleinonlinereadingcomprehension.

HillandHannafin(1997)usedacasestudymethodwithfourcollegestudentstoinvestigate

strategyuse.Theyconcludedthatstudents’backgroundknowledgeaffectedmetacognitivestrategy

use.Thisbackgroundknowledgeincludedbothtopicandsystemsknowledge.HillandHannafin

(1997)askedparticipantstolocatematerialsonasubjectoftheirchoosing.Theyreportedthat

previousexperiencewiththeInternetpredictedperformanceonthetask.Inthestudypriortool

expertise,asmeasuredusingaself‐reportfrequencyofusemeasure,wasmorepredictivethan

domainknowledge.Inonlinereadingcomprehensionassessment,familiaritywithavailableWeb‐

basedtoolsmaybetterpredictsuccessthanpriordomainknowledgeandexperience.

TheresultsofbothofBilal’sstudies(2000;2001)foundthatbackgroundknowledgedidnot

influencesuccess.Resultssuggestedneitherdomainknowledgenortopicknowledgeinfluencedthe

success(Bilal2000;2001)ofstudentsatInternetinquiry.Inthe2001study,fourstudentswitha

highermeanscoreofmeasuresoftopicknowledgewereunsuccessfulintheirsearcheswhilenine

studentswhoscoredloweronameasureoftopicknowledgeweresuccessful.Thisindicatesthat

backgroundknowledgemaynotbeascriticalduringonlinereadingcomprehension.

Otherstudies,however,havefoundthatbackgroundknowledgeplayedasignificantrole.

TabatiandShore(2005)conductedastudycomparingthesearchingbehaviorsofexpertsand

novices.Participantsincluded10novices,9intermediates,and10experts.Usingverbalprotocol

andsurvivalanalysis,amethodforanalyzingdatabasedontheoccurrenceofanevent,(Allison,

2010),theyconcludedthatmostsignificantdifferencesinpatternsofsearchbetweennovicesand

expertswerefoundinthecognitive,metacognitive,andbackgroundknowledgestrategies.The

researchersfoundasignificantcorrelationbetweenmetacognitivestrategiesandbackground

knowledge(r=0.54,p=0.003).Specificallytheyfoundsignificantcorrelationsbetweenreflection

33

anddomainknowledge(r=0.45,p=0.01),reflectionandsystemknowledge(r=0.41,p=0.03),

andmonitoringanddomainknowledge(r=0.45,p=0.01).

Finallysomestudieshavefoundthattherolebackgroundknowledgeplaysinonline

readingcomprehensionmaybedeterminedbyastudent’sonlinereadingcomprehensionability.

Coiro(2011)foundthatbackgroundknowledgeexplainedasignificantamountofvariancewhen

enteredintoaregressionmodelpredictingonlinereadingcomprehensionassessment.However

Coiroalsofoundaninteractioneffectbetweenbackgroundknowledgeandonlinereading

comprehensionability.Coiroconcludedthattheinteractionmaysuggestthattopic‐specific

backgroundknowledgeaffectsstudentswithhighandaverageonlinereadingabilitylessthanthose

withlowonlinereadingability.

Thefivestudiesofonlinereadingcomprehensionthatinvestigatedbackgroundknowledge,

eitherwithqualitativeanalysis(Bilal2000;2001;Hill&Hannifin;1997)orstatisticaltesting(Coiro,

2011;Tabati&Shore,2005)havefoundsomewhatcompetingresultsastotheroleofbackground

knowledge.Thisisinsuchstarkcontrasttostudiesoftraditionalreadingcomprehension.Some

researchfoundastrongroleforbackgroundknowledge(Hill&Hannafin,1997;Tabati&Shore

2005)whileotherworkfoundadiminishedroleforbackgroundknowledge(Bilal,2000;2001;

Coiro,2011).Thishasledsomeresearcherstospeculatethattheexpertiseoftheuser,including

theirabilitytousetheInternet(Coiro,2011;Hill&Hannifin,1997),mayinfluencetheroleof


Measuresofpriorknowledgeinonlinereadingcomprehensionstudies.Therehave

beentwomajormethodsformeasuringpriorknowledgeinstudiesthatexaminedpriorknowledge

andonlinereadingcomprehension:self‐reportsurveysandpriorknowledgequestionnaires.The

typesofpriorknowledgemeasuredincludeddomain,ortopicknowledge,andsystemknowledge,

inotherwordstaskknowledge.

34

HallandHannifin(1997)usedaquestionnairetojudgebothtopicknowledgeandsystem

knowledge.Fourquestionswererelatedtoparticipants'priorsubjectknowledge.Participantswere

askedtoexplaintheirsearchtaskandaskedtoratethemselvesonascaleofnovicetoexpert.

Systemknowledgewasratedusingaself‐reportfrequencymeasureoncommonsearchtools.The

scalewasalongacontinuumofneverusedtodailyuse.

Bilal(2000;2001)usedateachersurveyofstudents’backgroundknowledge.Thesurvey

consistedofquestionsabouttopicknowledge(alligators).Thesurveyalsoaskedaboutdomain

knowledge.Finallyteachersalsoratedstudents’readingability.

TabatiandShore(2005)alsousedself–reportsofself‐efficacytoserveasascorefor

systemknowledge.Participantswereaskedtoratethemselvesinthreeareasofcomputerliteracy:

information‐seekingknowledge,computerknowledge,andknowledgeoftheWeb.Afivepoint

Likertscaleonadimensionofpoortoexcellentwasused.Notopicbackgroundknowledgemeasure

wasused.

Coiro(2011)usedapriorknowledgemeasurethatmovedbeyondself‐reportscores.Asix‐

itemquestionnairewasused.Fourofthequestionsaskedstudentsabouttopic‐specificknowledge.

Twoofthequestionsaskedabouttaskknowledge,specificallyanimationandwebsitereliability.

ItemscoringprocedureswereinformedbyLeslieandCaldwell’s(1995)ConceptQuestionTaskand

WolfeandGoldman’s(2005)measureoftopic‐specificpriorknowledge.Thequestionnairewas

administeredorallyonaone‐to‐onebasis.

Summaryofresearchonpriorknowledgeinbothofflineandonlinereading

comprehension.Priorknowledgeplaysanimportantroleinofflinereadingcomprehension.Its

roleinonlinereadingcomprehensionislessclear.Thediversityintheresultsexploringtheroleof

priorknowledgeduringonlinereadingmayhaveasmuchtodowiththetypesofmeasuresusedto

measurepriorknowledgeastheydowithonlinereadingcomprehension.Thusitwasdecidedto

35

operationalizepriorknowledgeasbackgroundknowledgeforthisstudy.Thisreferstoonlydomain

specificknowledgeabouttheAmericanRevolution.

Acrossthestudiesthemajoritymeasuredpriorknowledgeusingself‐reportmeasures.This

maybeproblematicastheseitemsareopentobias.Studiesthatcomefromlibraryandinformation

sciencefavorsystemknowledge.Studiesfromareadingperspectivefavortopicknowledge.Given

thefocusofthisstudy,themethodsemployedbyCoiro(2011)seemmostpromising.Howeverthe

one‐on‐oneadministrationformatwasnotpractical.Thereforethebackgroundknowledgemeasure

inthisstudydrewonthemethodsusedbyCoirobututilizedanonlineformforsystematicdata

collection.

ResearchonCriticalEvaluation

FoundationalOverview.Avarietyoffieldshaveexploredthecriticalevaluationof

information.Beforebeginningasystematicreviewofresearchofthecriticalevaluationofwebsites,

itisusefultohaveabroadsenseoftheseveralfieldsthathaveprovidedthefoundationforthis

research.Theseinclude:informationprocessing,informationretrieval,designtheory,critical

thinking,anddocumentrepresentation.

Informationprocessing.Theemergenceofthecomputerinthe1950’sledcognitive

sciencetometaphorsofthinkingthatcomparedthemindandthecomputer(Wilson&Myers,

2000).Thuscomputerorientedinformationprocessingmodelsprovidednewwaystothinkabout

thinking(Johnson‐Laird,1988).Informationprocessingtheoriesbelieveinformationisreceived

fromexternalstimuli(websites),itreceivesattention,theinformationisstoredinshorttermor

longtermmemorywhereitiscombinedwithpreviouslystoredinformation,andfinallyaresponse

(judgment)isgenerated.Muchoftheearlyresearchonthecriticalevaluationorcredibilityof

websitesdrawsoninformationprocessingmodels(Tate&Alexander,1996).Specifically,

frameworksofinformationretrievalanddesigntheoryweresituatedininformationprocessing

perspectives.

36

Informationretrieval.Informationretrieval(IR)frameworksencompasscognitive

processesinvolvedinthesearching,organizing,andaccessingofinformation.Muchoftheliterature

fromlibraryscienceadoptsanIRframeworkandexplorescriticalevaluationasacredibility

judgmentthatinvolvesinformationqualityandcognitiveauthority.

TheIRliteratureontheevaluationofwebsitesdrawsonfiveprinciplesofinformation

qualityidentifiedfromliteratureontheevaluationofprinttexts:accuracy,authority,objectivity,

currency,andcoverage(Tate&Alexander,1996).Thesefiveelementsofcredibilityhavebeenquite

stableoverthelastdecade.Forexample,inarecentreviewoftheliteratureMetzger(2007)found

themostconsistentelementsofinformationqualitywerefoundtobe:believability,accuracy,

trustworthiness,bias,andcompletenessofinformation.Thesefiveprinciples,similartothe

characteristicsoutlinedbyTateandAlexander,weredrawnfromTaylor’s(1986)valueadded

modelofjudginginformationquality.

Taylordefinedqualityas“ausercriterion,whichhastodowithexcellenceorinsomecases

truthfulnessinlabeling”(p.62).Taylorpositedthatinformationsystemsandinformationhad

specificvaluessomeweretangibleandcouldbeseen,whileotherssuchasreliabilitywere

intangible.Therewerespecificvaluesthatcouldbebuiltupovertime.Tayloridentifiedfiveofthese

valuesthathaveconsistentlybeenfoundintheliteraturethattakesanIRperspective:accuracy,or

thedegreetowhichtheinformationistrue;comprehensiveness,orthecompletenessofcoverage;

currency,orhowrecenttheinformationisreliability,orthetrustsomeoneputsintheinformation;

andvalidity,orhowusefultheinformationistothetask.

AnotherelementcommontoresearchinlibrarysciencethattakesanIRframeworkis

cognitiveauthority.CognitiveauthorityasdefinedbyWilson(1983)influencesmanyofthe

theoreticalarticles(Tate&Alexander,1996)orstudies(Fritch&Cromwell,2001,Reih,2002;Reih

&Belkin,1998)investigatingthecriticalevaluationofwebsitesthroughrecognitionthat

informationfromdifferentsourceshasvaryingreliability.ReihandBelkin(1998)summarized

37

Wilson’sexternaltestsforcognitiveauthorityoftextsasinvolving:personalauthorityin

recognitionoftheauthor;institutionalauthorityinrecognitionofthepublisher;textualtype

authorityinplacingvalueinthetypeoftext;andintrinsicplausibilityauthorityinplacingauthority

inthecontent.

Designtheory.Someoftheearliestempiricalworkonthecriticalevaluationofwebsites

developedfromresearchersinvestigatingwhatmakespeoplebelievesomewebsitesoverothers

(Foggetal.,2001;Foggetal.,2003;Fox,2006;Fox&Raihne,2002).Designtheoryalignsclosely

withthetheoreticalframeworkfoundinIRsuchasWilson’s(1983)cognitiveauthorityandthe

valuesofinformationqualityidentifiedbyTaylor(1986).Themajortheoreticaldifferenceisinthe

audience.IRresearchfocusedonimprovementsinsystemsandusers.Designtheorylooksto

increasetheperceivedcredibilityoftextsinordertoincreasemarketvalue.

Criticalthinking.Researchinsciencehasoftenusedacriticalthinkingframeworkto

investigatehowreadersevaluatescienceargumentsonline(Brem,Russell,Weems,2001;Graesser

etal.,2007;Sanchez,Wiley,&Goldman,2007).Bremetal.defineargumentationastheemployment

ofcriticalthinkingskillsintheevaluationofspecificclaimsbutalsoframedtheirresearchin

situatedcognition.AccordingtoGraesseretal.criticalthinking,“requireslearnerstoevaluatethe

truthandrelevanceofinformation,tothinkaboutthequalityofinformationsources,totracethelikely

implicationsofevidenceandclaims,andtoaskhowtheinformationislinkedtothelearner’sgoalsand

largerconceptualframeworks”(p.3).Theythensuggestthatcriticalthinkingrequiresacriticalstance

thatrequiresreaderstobesuspectofallinformationtheyencounter.

Documentrepresentation.Theoriesofdocumentrepresentationbuildoffofearlierworkin

thereader’sconstructionofdocumentmodelswithsingletexts(Kintsch,1998;Kintsch,&VanDijk,

1978)buttrytoaccountforthemultiplesourcesreadwhenconductingInternetinquiries.Theories

ofdocumentrepresentationsuggestthatreadersconstructadocumentmodel.Itdefersfromearlier

documentmodelsthatsuggestedatwo‐phaseconstruction–integration(Kintsch,1998)that

38

involvesthetextbase,orinternalmeaning,andthesituationalmodel,whichcombinesthe

informationinthetextwithbackgroundknowledge,byaddingonanadditionallayertheintertext

model(Britt,Perfetti,Sandak,&Rouet,2007).Theintertextmodelincludesinformationabouthow

thetextsarerelatedandinformationaboutcharacteristicsofthesources(Braten,Strømsø,&Britt,

2010.Itistheintertextmodel,andmorespecificallytheskillofsourcing,thatalignswithother

operationalizeddefinitionsofcriticalevaluation.Sourcing“istheskillofgatheringinformation

aboutasourceandusingittoformconclusionsaboutadocument,especiallyconclusionsregarding

credibility”(Britt&Gabrys,2002p.171).

Definingthecriticalevaluationofwebsites.Therearemultipledefinitionsofcritical

evaluation(Coiro,2007)thatdrawonmanydifferenttheoreticalperspectives.Constructsusedto

measuretheseprocessesandjudgmentsvaryacrossdifferentresearchfieldsandinclude:

evaluation,judgment,andcriticality.Iwillusethetermcriticalevaluationtoexploresimilarities

anddifferencesinhowvariousconstructswereoperationalized.Overallstudiesandtheoretical

articlesagreethattheconstructofevaluationdrawsheavilyoncognitiveworkinvestigatingthe

evaluationprocessandcredibilityjudgments.

Evaluation.IndefiningevaluationFitzgerald(1999)drawsonBloom’staxonomy(Bloom,

Engelhart,Furst,Hill,Krathwohl,1956),whichdefinedevaluationasajudgmentinvolvedinthe

evaluationofcriteria,values,andstandards.Krathwohl(2002)expandedontheseideascreateda

two‐dimensiontaxonomythatseparatesknowledge(factual,procedural,conditional,and

metacognitive)andcognitiveprocesses(understand,apply,analyze,evaluate,andcreate).The

revisedtaxonomyalsodefinesthecognitiveprocessofevaluationasthemakingofjudgments

basedonstandardsandcriteriathatinvolvethecognitiveprocessesofcheckingandcritiquing.

Mostresearchhasdefinedtheprocessofevaluationasinvolvingjudgments(Flanagin,

Metzger,&Miriam,2000;Foggetal.,2001;Reih,2002;Reih&Belkin,1998;Zhang&Duke,2007)

orasadecisioninvolvingtheprocessesofevaluation(Flanigan,1999).Eventhoseresearcherswho

39

tookanewliteraciesperspective,whichadaptedamoresociologicalperspective,stillaccountedfor

ajudgmentofclaimsandevidence(Damico&Baildon,1998;Damico&Baildon,2006)ordeveloped

taxonomiesgroundedinthecredibilityjudgmentliterature(Coiro,2007;Coiro&Dobler,2006,

Zawilinskietal.,2007).

Judgment.Theworkdoneinthefieldofjudgment,decision‐making,andchoice(Hogarth,

1987;Goldstein&Hogarth,1997,Rachlin,1989)influencesthedefinitionsofjudgmentusedby

researchersinvestigatingthecriticalevaluationofwebsites.AccordingtoRachlin(1989),“…a

judgmentisalwaysaguideformakingadecision,whichleadstoachoice,whichthenproducesan

outcome”(p.43).Hogarth(1987)identifiedtwotypesofjudgment:predictivejudgmentsand

evaluativejudgments.Researchershaveoftenadaptedthesetwotypesofjudgmentsintheir

explanationsofcriticalevaluation(Reih,2002).

Thepredictivejudgment,orpredictiveinference(Coiro,2007;Coiro&Dobler,2007)

involvesthejudgingoftherelevanceofalink.Thereaderisdecidingifthatlinkwillbeuseful,

makesapredictivejudgmentandfollowswithachoicebyclickingonalink.Atthatpointthereader

wouldthenmakeanevaluativejudgment.

Asstatedearlier,researcherswhoadoptanIRand/oralibraryscienceperspectivelookto

thisevaluativejudgmentasusingbothinformationqualityandcognitiveauthority(Fritch&

Cromwell,2001,Reih,2002;Reih&Belkin,1998)ascriteria.Thecriteriausedbyresearchersto

operationalizeinformationqualityandcognitiveauthoritysharemanycommonelements.

Criticality.Critical,intermsofcriticalevaluationcanmeanmanythingstomanydifferent

people.Intermsofthedefinitionscollectedforthisstudy,criticalreferstothreeseparateentitiesin

termsofeducationalresearch:criticalthinking,criticalreading,andcriticalliteracy.

AsnotedbyFitzgerald(1999),manywritersequatecriticalthinkingwithevaluationwhile

themajorityofresearchersviewevaluationasoneofasubsetofhigherorderskillsinvolvedin

criticalthinking.Intermsofjudgingtext,criticalthinkingis“analyticalthinkingfortheprocessof

40

evaluatingsources”(Hickey,1990p.175).Itinvolvesanalyzingfacts,generatingandorganizingof

ideasdefendingopinions,comparisons,drawinginferences,evaluatingfact/opinion,problem

solving,setofdispositionstodrawonthoseskills(Bremetal.,2001;Coiro,2008;Fitzgerald,1999).

Coiro(2008)usesacriticalthinkinglenstodrawonherdefinitionofcriticalevaluationas“readers

applyingtheircriticalthinkingabilitiesto:(a)question,analyze,andcomparetheresourcesthey

located;(b)judgethequalityofinformationonvariouscharacteristics;and(c)defendtheir

opinionswithevidencefrommultiplesourcesandtheirbackgroundknowledge”(p.47).

Inhiswork,Spache(1964)wrotethatcriticalreadingisasetofskillsthatextendsbeyond

bothfunctionalliteracyskillsandhighercomprehension.Theseskillsincludeinvestigatingsources,

recognizinganauthor’spurpose,distinguishingfactfromopinion,drawinginferences,judgments,

anddetectingpropaganda.Therearemanyparallelsevidentbetweendefinitionsofcriticalthinking

andcriticalreadingfoundinmuchoftheliterature(c.f.Coiro2003,2008;Robinson,1964,Russell,

1963).

Infactmanyliteracyresearchersinthefieldofcriticalreadinghavelongheldthatcritical

readingcannotbeseparatedfromcriticalthinking(Ennis,1962;Wolf,King&Huck,1968).Thisis

evidentinresearchoncriticalevaluationrootedinlibraryinformationsciences,information

retrieval,andmoresocio‐cognitiveviewsofnewliteracies.Manyofthecriticalreadingskillsare

evidentinthecheckliststhatarecommonlyusedtoteachcriticalevaluationofwebsites.Cervetti,

Pardales,andDamico(2001)arguethatthisconnectionisrootedinliberal‐humanisttraditionsthat

areatoddswiththetraditionsofcriticalliteracy.

Unlikecriticalreading,criticalliteracyisrootedmoreinsocio‐culturalviewsofreadingthat

viewresponsetothetextaslessapersonalextractionsofauthor’sintentandmorerootedinsocial,

historical,andculturalpractices(Freebody&Luke,1990;Lankshear&Knobel,1998;Luke,2000;

Mellor&Patterson,2004).Influencesoncriticalliteracyemergedfromavarietyoftraditions.

CriticalsocialtheoriesofNewCriticismschoolsofthought,focusedonusinglanguageresourcesto

41

createamorejustsociety(Cevettietal.,2001).PostcolonialandMarxistviewsrestonthe

assumptionthatweliveinaworldofunequalpowerandtextsareusedtoeitherreinforceor

challengethesepowerstructures(Friere,1970).

Recentlyviewsofcriticalliteracyhavedrawnofideasofpost‐structuralismthatexamine

therelationshipbetweenpower,discourses,andculture(Mellor&Patterson,2004).Howmeaning

isconstructed,isthenconnectedtopowerrelationshipswithinspecificcommunitiesofpractice

(Cevettietal.,2001).LankshearandKnobel(1998)arguethatdefinitionsofliteracymustthen

considerthreeelements:theoperational,thecritical,andthecultural.

Whilecriticalliteracydrawsonavarietyofhistoricalschoolsofthoughtitisunitedinthe

ideathatliteracyisasocialpracticeandnotasetofneutralpsychologicalskills.TothisendLuke

andFreebody(1990)createdthefour‐resourcesmodeltocreatecriticalliteracypedagogy.They

suggestthattherearefournecessary,butinsufficient,rolesreadersmusttakeinapost‐modern

world:code‐breaker,meaningmaker,textuser,andtextcritic.Coiro(2008),whilerootingherwork

incriticalreading,givesapassingnodtocriticalliteracybysuggestingitwillbenecessaryfor

studentstocomprehendtheincreasinglyimage‐drivenwebsites.MurrayandMcPherson(2006)

suggestthattheunderstandingofonlinetextsandwebsiteswillrequiregreatercriticalliteracy

skills.Stone(2007)suggeststhatwemustthinkofcriticalreadingofwebsitesasinvolvingmore

thanevaluationoftruthbutalsolooktoseehowstudentsusepopularwebsitesintheireveryday

lives.

Summary:Drawingonmultipleperspectivesanddefinitionstodefinethecritical

evaluationofwebsites.Itisclearthatthecriticalevaluationoftexts,whetherthedefinitionis

rootedincriticalthinking,criticalreading,orcriticalliteracyinvolvesajudgment.However,

assumingthatauthorityandcredibilityareinherentfeaturesoftextsignorescertaincontextual

elementstoreading.Thepurpose,culture,andpracticesofreaderswillinfluencewhatjudgments

anddecisionstheymake.

42

Idefinecriticalevaluationofwebsitesasacontextualprocessofexamining,adopting,and

changingperspectivesinordertojudgetherelevancyandreliabilityofawebsite.First,critical

evaluationiscontextualbecauseitmayrequirespecificcontentknowledgeormaybeinfluencedby

students’epistemicbelief(Damico&Baildon,2007;Bremetal.,2001).

Second,criticalevaluationisarecursiveprocess.Thecriteriaandjudgmentsinvolvedin

criticalevaluationhavetobecontinuousthroughoutInternetinquiry.Itisnotaspecificstageor

stepinasimpletaxonomy.Criticalevaluationtakesrepeatedquestioning,goalsetting,andavariety

ofmetacognitiveskills(Graesseretal.,2008;Zhang&Duke,2007).

Third,criticalevaluationinvolvestheexamining,adopting,andchangingofperspectives

becausealltextsaresocially,historically,andculturallysituated(Lankshear&Knobel,1998).

Focusingcriticalevaluationassimplyidentifyinganauthor’smessage,intentandbias“privileges

thefactualandobjective”(Fabos,2008p.843).Encouragingstudentstounderstandthedifferent

perspectivesthatshape“truth”willallowthemtojudgerelevancyandreliabilityandmayavoid

errorsofoversimplificationsuchasrejectingasourcebecauseitisbiased.

Finally,thecriticalevaluationofwebsitesinvolvesthejudgmentofrelevancyandreliability

overthejudgmentofcredibilityandauthoritysimplybecausetextsarenotneutralentities.The

reliabilityandrelevancyofasourcemaychangebasedoncontent,contextandpurpose.

ThisdefinitionwasappliedtothedesignoftheCriticalOnlineInformationLiteracies

measureofwebsiteevaluationusedinthisstudy.Theoriginalassessmentincludedscalesto

measurebothrelevancyandcredibilityjudgments.Furthermoreitincludeditemsthatlookedat

theauthor,publisher,credibilityofevidence,andbias.

Methodsforidentifyingcriticalevaluationskillsandstrategies.Themethodologies

usedtostudythecriticalevaluationofwebsitescontinuouslyevolve.Themajorfocusofstudieshas

beentoidentifytheskillsandstrategiesusedtoevaluatewebsites.Threebasicmethodshavebeen

used:

43

1)self‐reportsurveysandquestionnairestounderstandthefrequencyandtypeofskills

studentsuseincriticalevaluation;

2)verbalprotocolanalysis[VPA]andinterviewstoidentifytheskillsandstrategiesusedby

students;and

3)casestudiestoexploresocio‐culturalfactorsofcriticalevaluation.

Self‐reportsurveysandquestionnaires.Theearlieststudiesinvestigatingthecritical

evaluationofwebsitesinvolvedself‐reportsurveysandfrequencyquestionnaires(Flanagin&

Metzger,2000;Foggetal,2001;Fox&Raihne,n.d.Fox&Raihne,2002,Metzger,Flanagin,Zwarum,

n.d.;PrincetonSurveyAssociates,2005).Thegoalsofthesemeasuresweretounderstand:(a)What

makesawebsitecredible?(b)Whatweretheperceivedlevelsoftrustindifferenttypesofmedia?

and(c)Whatwasthefrequencyofskillsused?Themajorityofskillssurveyedinthesestudies

sharedmanysimilarities,fromlistsofskillsusedforprintbasedmedia(i.e.Tate&Alexander,

1996)whichincludedaccuracy,authority,objectivity,currency,andcoverage.

Acrosstheself‐reportstudiesavarietyofmethodswereused.Surveyswerecreatedand

validated(Foggetal.,2001)usingexploratoryfactoranalysis.Othersurveysreliedondescriptive

statisticsfromlargesamplestodrawconclusions(Fox,2006;Fox&Raihne,2002;Princeton

ReviewBoard,2002;2005)whileothersuseddescriptivestatisticsasdependentvariablesfor

furtherstudy(Flannigan&Metzger,2000).SincetheInternetisaratheryoungphenomenonitis

understandablethatsomeoftheearlieststudieswouldrelyonself‐reportsurveystoidentify

trendsandfactorsthatinfluencethecriticalevaluationofwebsites.

Earlyresearchintowebsitecredibility,usingsurveys,hadthegoalofmakingcommercial

websitesmorecredible.Forexample,Foggetal.(2001)createdandadministeredasurveyto1,441

participantsranking51elementsofcredibilityonaLikertscale.Thecreationofthe51itemsthat

mayinfluencecredibilitywentthroughfourrigorousstagesofcontentvalidation.Thesurveywas

44

thenadministeredtoparticipantswhorankedtheitemsonaseven‐pointscalefromlessbelievable

tomorebelievable.Theresearchers,usingvarimaxrotation,andEigenvaluesgreaterthan1.73

foundthatfivefactorsincreasedwebsitecredibility:real‐worldfeel,easeofuse,expertise,

trustworthiness,andtailoring.Incontrasttwofactorshurtperceivedcredibility:commercialism

andamateurism.Fromthesefindingstheresearchersconcludedwithasetofdesignprinciplesthat

couldbeusedtoincreasetheperceivedcredibilityofwebsites.Itmustbenoted,however,that

researchersonlyexplained15.1%ofthevarianceintheirmodel,whichcouldindicatethatthe

predictedfactorloadingsdonotadequatelyexplainthevarianceinthepopulation.

Anotherpurposeofsurveyswastocomparewebsitecredibilitywithothermediatypes

(PrincetonSurveyResearchAssociates,2002;2005;Flanigan&Metzger,2000).ThePrinceton

SurveyAssociates(2002)conductedasurveyof1,051adultsinordertoidentifyareasthatcould

improvewebsitecredibility.Theresultsshowedthatonly29%ofrespondentstrustedwebsites

thatsoldproducts.Inthestudy,33%ofrespondentstrustedwebsitesthatreviewedproductsand

services.Thesescoredweredrasticallylowerthanothermediatypesandprivateentities.

FlaniganandMetzger(2000)alsoconductedasurveytocompareperceivedcredibilityof

websiteswithothermediaandalsotounderstandtheverificationstrategiesusedbypeoplebased

ondemographics,typeofmedia,andInternetexperience.UsingarepeatedmeasureANOVAthe

resultsshowednewspaperswereperceivedtohaveasignificanthigherlevelofcredibilitythan

othermedia(F‐114.12,df‐4,2428,p<.001,~=.1).

InthesurveyconductedbyPrincetonSurveyResearchAssociates(2002)80%of

respondentsstatedthattrustwasveryimportant,80%statedthateaseofnavigationwas

important,and80%statedthatbeingabletoidentifythesourceoftheinformationwasvery

important.Only32%ofrespondentssaiditwasimportanttoknowtheauthorofawebsite.Finally,

65%respondedthatknowingthatthewebsitewasupdatedfrequentlyisveryimportant.Inthe

followupstudy(PrincetonSurveyResearchAssociates,2005)thecredibilityofwebsitesseemedto

45

declinewithonly21%ofrespondentstrustingwebsitesthatsoldproducts.Therewasalsoan

increaseintheelementsofwebsitesimportanttotrust.Therewasastatisticallysignificantincrease

inthenumberofrespondentssayingitwasimportanttoidentifythesourceofinformation,the

frequencyofupdates,andknowingwhoownsawebsite.

Studiesinvestigatinghowpeopletrusthealthinformationonlinehavealsobeenconducted.

In2002,500“healthseekers”wereaskedabouttheirsearchhabitsinphoneinterviews(Fox,

2006).Over50%ofrespondentsreportedcheckingcheckthesourceordateoftheinformation

theyreadonlinewebsitesformedicaladvice“onlysometimes,”“hardlyever,”or“never.”One

quarterseemedtojudgethecredibility“mostofthetime”andanother25%“allofthetime.”49%of

respondentswholookforinformationonmultiplesitesrespondedthatverifyinginformationgave

themalotmoreconfidence,and38%saidthatitgavethemalittlemoreconfidence.Inthe2006

surveyseventy‐fivepercentofrespondentsreportedevaluatinginformationsometimes,hardly,

andnever(Fox,2006).Thisrepresentsa25%increaseoverthe50%ofadultswhoreportednot

beingvigilantorconcernedfromthe2002survey.Theauthorsspeculatethatthisfallintheself‐

reportsofevaluationcouldbecausedbytheincreaseineighteenandplususers(Fox,2006)who

grewupwiththeInternetandwhoplacemoretrustinwebsitecredibility.

Studieswerealsodone,usingself‐reportquestionnaires,tounderstandthecriteriastudents

usetoevaluatewebsitesinacademicsettings.Tillotson(2002)collected499questionnairesfrom

collegeundergraduatestudentsinordertoinvestigateiftheyrecognizedaneedtoevaluate

websites,theextenttowhichtheyevaluatedwebsites,andthetypeofcriteriausedtoevaluate

websites.Theresultsshowedthatstudentshaveanascentapproachtoevaluatingwebsites.He

foundthat38%ofstudentsrespondedtheyhaveneverfoundmisleadinginformationonline;the

averagestudentreportedusingfewerthantwocriteriatoevaluateawebsite.Theresultsshowthat

themajorityofstudentswhoreportedusingcriteriausedsourcereliability.Onlytwenty‐fiveofthe

46

studentsreportedusingtheURLorwebaddresstomakeajudgment.Furthermore,overhalfthe

studentswhousedcontenttoevaluateawebsiteuseditastheonlycriteria.

Thestudiesusingself‐reportmethodologiesareingeneralagreementofthetypesofskills

neededorusedbystudentstoevaluateonlineinformationandthelackoftheseskillsinInternet

usersregardlessofage.Theseskillsdonotdiffergreatlyfromevaluationskillsidentifiedusing

printmedia.Skillscommonacrossalltheself‐reportstudiesinclude:(a)identifyingtheauthoror

sponsor;(b)examiningtheURL;(c)usingformatorappearanceofthewebsite;(d)checkingthe

currencyofinformation;(e)checkingtheaccuracyusingasecondarysource;(f)examiningbias;and

(g)usingcontenttojudgeawebsite.Itmustbenoted,however,thatmanyoftheseskills,because

thesewereself‐reportmeasureswerenotdirectlyobserved.Thestudiesthereforearesubjectto

errorsinbothoverandunder‐reporting.Furthermore,acrossthestudiesparticipantsrarelyused

morethanoneskilltojudgeawebsite.Finally,theskillsreportedwiththegreatestfrequencies

includedidentifyingtheauthorandusingsuperficialelementssuchastheappearanceofawebsite

orthecontent.

Theresultsofresearchstudiesusingself‐reportmethodsalsoprovideinsightintohow

readerscriticallyevaluatewebsites.Thesestudiesareingeneralagreementthatreadersdonot

evaluatewebsiteswithgreatfrequency.Acrossthestudiesthatinvestigatedthecriteriatojudge

websites,evaluatingthesourcewasconsistentlymentionedasthemostfrequentskillusedby

participants.Howeveracrossallthesurveysveryfewreadersusedmultiplecriteria.Forexample,

FlaniganandMetzger(2000)usedmeanverificationscorestoanalyzecredibilityverification

strategies.Acrossallthestrategiesrespondentsreportedusingskillsbetween“never”to

“sometimes.”withmostscoresfallingbetween“never”to“rarely.”Thisindicatesthatparticipants

didnotevaluatethesourcestheyread.

Theself‐reportstudiesalsoshowdifferencesintheresponsesbasedonage.Younger

participantsoftenreportedgreaterfrequencyofInternetuse,butalsoplacedgreatertrustin

47

Internetsources.Thiscouldindicateaneasingoffearswithincreaseduse;however,thisfinding

mayalsoindicatethatwhilestudentsareso‐called“digitalnatives”theyarenotinformationsavvy

(Bennet,Maton,&Kervin,2008).InotherwordsstudentsmaybeabletodownloadMP3’sorcreate

amash‐upforYouTube,butthatdoesnotmeantheyareskilledinusingtheInternetduring

problembasedinquirytasks.

Also,theresearchshowsdifferentresultsintrendsovertime.Thestudiesconductedbythe

PrincetonSurveyResearchAssociates(2002,2005)showanincreaseinthedistrustofwebsites.On

theotherhand,thePewInternetandAmericanLifeSurveys(Fox&Rainie,2002;Fox,2006)show

anuptickinthenumberofrespondentswho“never”or“rarely”evaluatewebsites.Thismaybe

explainedbydifferencesinthereadingtask.ThePrincetonSurveyfocusedprimarilyonwebsites

thatprovidedproductsorservicesforpurchase.ThePewstudieswereconcernedwithhealth.This

couldindicatethatthecriticalevaluationskillsusedbystudentsaretaskanddomainspecific.Yet

thestudiesindicatethatastheInternetmaturesthecriticalevaluationskillsexhibitedbepeople

arealsoevolving.

Finally,themajorityofstudiesusingself‐reportdatafocusedonadultpopulations.The

studiesmaynotprovideinsightintothecognitiveprocessesofyoungerreaders.Compoundingthis

issueisthelackofresearchinacademicsettings.Onlyafewstudiescollectedforhisreviewdealt

withstudentpopulations,andofthosestudiesallusedundergraduatestudents.Agreatereffort

needstobemadetounderstandthereadinghabitsofyoungerlearners.

Interviews,casestudies,verbalprotocolanalysis.Avarietyofqualitativemethodshave

beenusedtoidentifytheskillsandstrategiesreadersusetocriticallyevaluatewebsites.These

methodsvarybasedonthepositioningandepistemologicalstancesoftheresearcher,butoverall

seektounderstandtheprocessofjudgingawebsite,identifyingthecognitiveskillsusedinwebsite

evaluation,andidentifyingexternalsocialfactorsthataffecthowpeoplejudgewebsites.

48

Acrossthestudiescollectedforthisreviewthreequalitativemethodswereused:interviews

(Reih,2002;Reih&Belkin,1998),casestudies(Damico&Baildon,2006),andverbalprotocol

analysis(Coiro,2007;Damico&Baildon,2007;Reih&Belkin,1998,Zawilinskietal.,2007).

Researchusingsemi‐structuredinterviewdataaskedparticipantstoindicatewhichskillstheyuse

whilereadingonline(Merriam,1998).CaseStudies(Yin,2003)usedexamplesfromlargersamples

toexploretherelationshipbetweenindividualdispositionsandcriticalevaluationskills.Finally

verbalprotocolanalysis(Afflerbach&Pressley,1995)hadparticipantsthinkaloudandexplain

theirdecisions,astheyreadonline(Zawilinskietal.,2007).

Twomajorprocedureswerecommonacrossthequalitativestudies:websiterankingsor

givingstudentsataskandhavingthemcompleteanInternetinquiry.Inwebsiterankingtasks

(Brem,Russell,&Weems,2003;Foggetal.,2003)studentsweregivenalistofwebsitestorankon

ascale.Studentstooknotes,whichwerelateranalyzed.Participantsdiscussedtheserankingsin

interviewsorduringverbalprotocolanalysis,ortheywroteexplanationsoftheirrankings.

AssigningInternetinquirytaskstoparticipantswasalsoaprocedureusedinqualitative

studies.Thesetaskswerethenrecorded,oftenwithscreencapturesoftware,andusedinanalysis

withverbalprotocolanalysis(Reih,2002;Zawilinskietal.,2007),semi‐structuredinterviews(Reih

&Belkin,1998),orcasestudies(Damico&Baildon,1998,2007).Thetasksvariedontherestricted

natureoftheInternet.Someresearchershadstudentsevaluateasinglewebsite(Zawilinskietal.,

2007);othersusedalistedifpre‐selectedwebsites(Agosto,2002;Sanchezetal.,2006)whilesome

studiesgavestudentsopenaccesstothefullweb(Coiro&Dobler,2007;Damico&Baildon,1998,

2007;Reih2002,Reih&Belkin,1998,Zawilinskietal.,2007).TheInternetisanunboundedspace,

byrestrictingstudentstoasinglewebsitetheresultsofthesestudiesmaybelimited.

Theproceduresandmethodsusedtoinvestigatecriticalevaluationvariedbutacrossthe

individualprojectssomeimportantcommonalitiesemerge.Firstthereisacommonagreementthat

newskills,strategiesanddispositionsareneededabovethoserequiredforlinearprintreading.

49

Secondthecriticalevaluationofwebsitesmayrequireamoreflexibleworldview.Thirdsocial

factorsmayplayamoredominantrolethancognitivefactorsintheevaluationofwebsites.Finally,

newscreencapturesoftwarewillserveasanimportantmethodologicaltoolinthestudieson

criticalevaluationTheresultsofthesestudieshelptoinformwhatcriticalevaluationskillsshould

beinitiallymeasured.Basedontheliteraturereviewitwasdecidedtoexaminetwoscalesof

evaluation:credibilityandrelevancy.

Assessmentofcriticalevaluationskillsandstrategyuse.Measuringcriticalevaluation

isachallengeforresearchers.Onlyafewstudiesattemptedtomeasurestudents’judgmentsof

onlinetexts.Fourformatswerecommonlyusedtoassesscriticalevaluationskills:websiteranking,

writtenstatements,andonlinereadingcomprehensionassessments[ORCA].

Themostpopularassessmenttoolwastherankingofwebsites(Graesseretal.,2006;

Sanchezetal.,2006;Zhang&Duke,2007).Basicallystudentsaregivenalistofwebsitesandthen

havetorankthemfromleastreliabletomostreliable.Thesearethenscoredagainstapre‐

determinedlistofrankings.Graesseretal.(2007)andSanchezetal.(2006)createdamockGoogle

pagewithsevenwebsites:threereliablesites,threeunreliablesites,andoneambiguoussite.The

goalwastocreateanaturalisticenvironmentbutlimitthenumberofWebpagesparticipantscould

read.Afterafifty‐minuteinquirystudentswereaskedtorankthesites1‐7andratetheinformation

onthewebsitesonasix‐pointscale.Zhang&Duke(2007)gavestudentsalistoffourwebsitesand

hadthemrankwebsitesfromonetofour.Theyalsohadstudentsdecideiftheinformationona

singlewebsitewastrustworthy.

Writtenstatementswerealsoacommonmeasurementtool.Theywereoftenusedin

conjunctionwithwebsiterankings(Zhang&Duke,2007)orasmeasurementsoflearninggains

(Graesseretal.,2007&Sanchezetal.,2006).ZhangandDukehadparticipantswriteaparagraph

explainingtheirhighestrankingandanadditionalparagraphexplainingtheirlowestranking.

50

Sanchezetal.hadstudentsalsojustifytheirrankings.Furthermoretherewasnodifference

inthejudgmentscoresofstudentsonthesinglewebsite,butthereasoningscoreontheSanchezet

al.,(nd)foundthatstudentstrainedintheSEEKmethodweremorelikelytojustifytheirrankings

usingevidenceinthewebsiteandinformationaboutthesourcewhereasparticipantsinthecontrol

simplyusedcontent.

Graesseretal.,(nd),testingawebtutorbasedontheSEEKmethod,hadstudentswritean

essayonthetopicofinquiry.Theseessayswereanalyzedtoevaluatestudents’useofacritical

stance.Therewasasignificanteffectontreatmenteventhoughtherewerenosignificant

differencesintherankingofwebsitesbetweentreatmentandcontrol.Inotherwordsstudents

couldnotcorrectlyrankwebsitesbuttherewassignificantimprovementintheirjustificationof

theirrankings.Thiswouldsuggeststudentsweremorecognizantoftheskillstheyshouldapplybut

appliedtheskillsincorrectly.

Researchershavealsodevelopedonlinereadingcomprehensionassessmentsthatembed

criticalevaluationscalesintofullInternetinquirytasks.Leuetal.,(2005)embeddedcritical

evaluationitemsintoalargerassessmentofonlinereadingcomprehension.Thisassessmenthad

students:locatetwowebsiteswhentheyweregivenapartialdescriptionofthewebsiteorURL,

evaluatethetwositesandchoosethebestsource,andexplaintheirreasoning.Thesecondtask

askedstudentstolocateawebsitewithagraphicoftherespiratorysystemthatmetspecific

criteria,andcommunicatehowsomeoneshouldcheckforaccuracy.

Coiro(2011)alsoincludedcriticalevaluationinherORCAassessments.Factoranalysisfor

bothORCASrevealedthataone‐factorsolutionwasmostappropriatewhichmightsuggestthatthe

processesofonlinereadingcomprehensionmaynotbeindependentofeachother

TheORCA‐Iditarod(Coiro,Castek,Henry,&Malloy,20007)hadstudentsreadaboutand

thentakeastandontheIditarod.Tomeasurecriticalevaluationskillsweregivenonesourceand

51

hadtoansweraseriesofquestion.UnlikeCoiro(2011)thescoresonthisORCAdidnotloadon

one‐scaleusingprincipalcomponentanalysis.(McVerry,O’Byrne,&Roberts,2009).

Asummaryofcriticalevaluationassessments.Initialworkintheassessmentofcritical

evaluationskillsneedstocontinue.Websiterankinghasbeenthemostpopularmethodtoassess

criticalevaluation.Howevernopsychometricpropertiesoftheseassessmentswerereportedinthe

artifactscollectedforthisreview.Thereforeitwasdecidednottousewebsiterankingasamethod

formeasuringcriticalevaluationskills.

Thefactthatresearchersfindstatisticaldifferences,afterinterventionsinwritten

statementsbutnotinconcurrentwebsiterankingisalsointeresting(Zhang&Duke,2007).This

couldindicatestudentsaremakingerrorsofoversimplification.Theyknowwhatresponsesabout

strategyusetogiveandparrottheseresponsesbackwithoutactuallyapplyingthestrategiesto

websites.Thereforeitwasdecidednottousewrittenjustificationofwebsiteevaluationinthe

measureofcriticalevaluationskills.

Furthermoreindicationsthatonlinereadingcomprehensionskillsmaynotbeindependent

ofeachotherwillmakemeasuringcriticalevaluationskillschallenging.Independenceofitemsisan

assumptiononalmostallreadingcomprehensiontests.Ofnotewerethefactorloadingsonthe

ORCA‐Iditarod.Relevancyjudgmentsloadedwithlocatingitemsandcredibilityjudgmentsloaded

withevaluationitems.Ifreadingonlinedoesinvolveasetofmultidimensionalskillsthen

developingassessmentswillbeauniquechallenge.

ResearchonDispositions

Currentmodelsofreadingcomprehension(Alexander&Jetton,2002;Snow,2002)have

notedtheimportanceofbothaffectiveandcognitivevariables(Baker&Wigfield,1999;Guthrie&

Wigfield,1997).Thesedispositionsandopenmindsets(Deschreyver&Spiro,2010)arecentralto

onlinereadingcomprehension(Leuetal.,2004).

52

AccordingtoClaxton(1999),theprocessoflearningrequirescapabilitiesbutthese

capabilitiescannotaccountforallthelearningthatmusttakeplace.Learningmustalsoinvolve

specificdispositions,oraffectivevariables,whicharea“domainofhumanattributesnotattributed

toknowledge,skill,orbehavior”(Katz,1988,p.30).Carr&Claxton(2002)definedispositionsasa

“tendencytoedit,select,adapt,andrespondtotheenvironmentinarecurrent,characteristickind

ofway.”Learningdispositionsare“patternsofbehaviors,situatedinthecontextofthe

environment,thatwhenrecognizedanddevelopedbythosewhocanmanipulatetheenvironment

mayleadtogainsintheacquisitionofknowledge,skillsandunderstandings”(O’Byrne,&McVerry,

2009).

Duetothenatureofonlineinformation(Alvermann,2004;Gross,2004)dispositionsmight

beevenmoresignificantasindividualsreadonlineinformation(Coiro,2011).Thisisduetothe

increasedneedtofocusonthegoalofthetask,evaluatingthesourcesbeingread,andhavingthe

persistenceduringInternetsearches.Inthisstudylearninghasbeenviewedasaninteractionof

students’capabilitiesanddispositions(Carr&Claxton,2002)astheyreadinanonlinespace.

Recentstudieshaveinvestigatedstudents’onlinereadingcomprehensionability(Coiro,

2007;Henry,2007;Castek,2008).Yetwedonotknowhowdispositionsaffectonlinereading

comprehensionbasedontherestrictednatureoftheinformationspace.Inotherwordsare

dispositionsmoreimportantinlessrestrictedspacesormorerestrictedspaces?Thisstudywill

investigatedispositionsinbothlessrestrictedandmorerestrictedinformationspaces.

Measuringdispositionsofonlinereadingcomprehension.McVerryandO’Bryne(2009)

usingprotocolandfieldnotesfromtheTICAproject(Leuetal.2007‐2011)identifiedfive

dispositionsthatarecentraltoonlinereadingcomprehension:(a)reflectivethinking,(b)critical

stance,(c)collaboration,(d)flexibility,and(e)persistence.O’ByrneandMcVerrythencreatedand

validatedaself‐reportinstrumenttomeasurethesedispositions.Usingexploratoryfactoranalysis

53

thesefivescaleswereidentified.Theinstrumentasawholewasshowntohaveadequate

reliability.Howevertwoscales,flexibilityandcriticalstancedidnothaveadequatereliability.

Researchershavecritiquedassessmentmethodsthatusedself‐reportsurveysand

interviewsformeasuringdispositions(Claxton&Carr,2004).Theyclaimthatdispositionsareso

situated(Sadler,2002)theycannotbemeasuredwhenthecontextisstrippedawaybecausethey

donotprovideanadequatelyrichcontextfortheirmeasurement.Theseresearchers(Carr&

Claxton,2004;Sadler,2002)suggestthatobservationsoverextendedperiodsoftimearetheonly

waytotrulyassesslearningdispositions.Whilethesetoolsmaybettercapturelearning

dispositionstheyalsocarryreliabilityandpracticalityissues.

Anotherrecentadvancementinthemeasurementofdispositionsisfacialrecognition

software(D’Mello&Graesser,2010).Thesetoolstrackstudents’affectivestates(boredom,

flow/engagement,confusion,frustration,andneutral)inreal‐timebymonitoringconversational

cues,grossbodylanguage,facialfeatures,andthelanguageoftheirresponsesduringinteractions

withanintelligenttutoringsystem.Trackingemotionalstatesmaybethefutureofmeasuring

dispositions.Howeverthecostandtrainingofthesoftwaremakestheinstrumentimpracticalat

thistime.Thereforethisstudywillrelyonaself‐reportmeasureasitisthemostcosteffective,

practical,andvalidtoolavailable.

Dispositionssummary.Newdispositionsarerequiredforlearnerstoreadinonline

environments.These“openingmindsets”willbecriticalasinquiriesmovebeyondsimple“findthe

answertasks”(Spiro&Deschryver,2010)andintolessrestrictedinformationspaces.Dispositions

willbecentraltobuildingknowledgeinthemomentthroughtheactof“readingwithmousein

hand.”(McWilliams&Clinton,2012).Inotherwordsstudentswillneedtobeflexiblein

constructingknowledgeonthefly,usereflectivethinkingtoremembertheirgoal,bepersistentin

searchingforthegoal,andhaveacriticalstancetoquestiontheunlimitedamountofinformation

online.

54

ThisstudyusedtherevisedinstrumentdevelopedbyO’Byrne&McVerry(2009).The

instrumentincludesadditionalitemsfortheflexibilityandcriticalstancescales.Whilethereare

concernsaboutself‐reportmeasures,otherdispositionassessmentswereimpractical.Classroom

observationswouldrequiretoomuchtimeandhavenotbeenshowntobereliableandfacial

recognitionprogramsarenotcurrentlyreadilyavailable.

Thisstudywilltesttherelativecontributionofdispositionsinamodelthatincludesaless

restrictedinformationspaceandamodelthatincludesamorerestrictedinformationspace.Itis

hypothesized,fromboththeoreticalperspectives,thatdispositionswillbeasignificantpredictorin

boththelessrestrictedmodelandthemorerestrictedmodel.Howeverbasedontheprinciplesof

cognitiveflexibilitytheoryitishypothesizedthatdispositionswillbeastrongerpredictorinthe

lessrestrictedinformationspace.

ResearchonVerbalIntelligence

Thereisalongtraditionofresearchindicatingthatverbalintelligencehasaconnection

bothtooverallintelligenceandreadingcomprehension(Thorndike,1974).Sincethegoalofthis

studywastoexaminetherolebackgroundknowledge,criticalevaluation,anddispositionsplayin

predictingscoresonameasureofonlinereadingcomprehensionitwasdecidedtopartitionoutthe

variancecausedbyverbalintelligence.UsinghierarchicalregressionmethodssimilartoAnderson

etal.(1988),itwasdecidedtocontrolforverbalintelligencegivenitshighcorrelationswith

readingcomprehension.

Definingverbalintelligence.Hunt(1978)definedverbalintelligenceasinvolving

“processesbasedonknowledge.”WhichHuntnoted,“Theabilitytodealwithwordsandthe

conceptstheyrepresentimpliestheacquisitionofinformation”(p.109).Usingschematheory,from

aninformationprocessingperspective,Huntclaimedthatverbalintelligenceindicateda“deep

structurerepresentationofalinguisticstatementofthethoughtsinvolved”(p.110).Hunt,

55

Lunneborg,andLewis(1975)alsofoundthatknowingaperson’sverbalintelligencecanpredict

theirabilitytomanipulatestimulirapidly.

Previousstudiesthatexaminesearchingforinformationormultimedialearningcontroluse

verbalintelligenceasavariableofinterest.Allen(1992)controlledforverbalintelligence,usingthe

verbalcomprehensionmeasureofTheKitofFactorReferencedTesttoexaminehowuserssearch

CD‐ROMS.Allenstated,“Theabilitytoselectappropriatesearchvocabulary,toexplorealternative

expressionsofideas,andtounderstandthecontentofretrievedmaterials,iscentraltosuch

searching.”ThesameconclusioncouldbedrawnforsearchingtheInternetforinformation.

Ithasbeenarguedthatsynonymvocabularytestsarebestformeasuringverbal

intelligence.Carrol(1974)arguedthatverbalintelligenceneedstomeasurelexiosemantic

informationstoredinlong‐termmemory.Hesuggestedthatanyotherformofassessment,besides

asynonymtest,wouldconflateverbalintelligencemeasureswithothervariables.Furthermore

vocabularytests,suchasverbalcomprehensionmeasureofTheKitofFactorReferencedTest

correlatehighlywiththecapacityofverbalworkingmemory(Avons,Wragg,Cupples,&Lovegrove,

1998;Gathercole&Baddeley,1993;Gathercole,Service,Hitch,Adams,&Martin,1999;Masoura&

Gathercole,1999).Infactrecentreviewsofcomprehensionresearchfound“Vocabularyknowledge

andsyntacticcompetence,accountformoreofthevarianceinreadingcomprehensionthando

individualdifferences”(Snow,2002,p.84).

Summaryofverbalintelligence.Giventhatthegoalofthisstudyistoexaminethe

contributionsthatbackgroundknowledge,criticalevaluationskills,anddispositionsmaketo

modelsofonlinereadingcomprehensionitwasdecidedtocontrolforverbalintelligence.Thiswill

allowthemodelstoexaminevariancebeyondthatcausedbydifferencesinlong‐term,

lexiosemanticmemory.Verbalintelligencewasmeasuredusingapreviouslyvalidatedassessment,

theverbalcomprehensionvocabularytestoftheKitofFactorReferencedCognitiveTest(Ekstrom,

56

French,Harman&Derman,1976).Thisisapreviouslyvalidatedmeasuretestedwith7thgraders.

Theinternalconsistencyofthemeasurehasrangedfrom.68‐.88.

ChapterSummary

Insummarythischapterreviewedtheliteraturebaseofthedependentandindependent

variablesincludedinthisstudy.Theliteraturereviewforthisstudyhelpedtoguidethedesignand

otherdecisionsusedinthisstudy.Empirically,researchhasdemonstratedthattherearenewskills,

strategiesanddispositionsrequiredforonlinereadingcomprehension.Unfortunatelyfewstudies

haveexaminedhowtherestrictednatureoftheinformationspaceshapestheroleofcognitiveand

affectivevariables.

Findingsfromtheinvestigationintobackgroundknowledge,criticalevaluationskills,and

dispositionsallindicatethatmoreworkisneededifwearetoenrichourtheoreticalmodelsof

readingcomprehension.Firsttheresultssurroundingthecontributionsofbackgroundknowledge

haveoftenbeencontradictory.Thisstudywilladdtoourunderstandingofbackgroundknowledge

andonlinereadingcomprehension.Byfocusingoncontentknowledge,andnotsystemknowledge,

theresultsmaydemonstrateifbackgroundknowledgeisastrongpredictorofcomprehensionin

onlinespaces.

Intermsofcriticalevaluationskillsthereisverylittleresearchexploringtheroleitplaysin

termsofoverallonlinereadingcomprehension.Thisstudypredicts,aftercontrollingforverbal

intelligence,thatitwillbethesinglegreatestfactorinpredictingperformanceinamodelthatalso

includesbackgroundknowledgeanddispositions.Thereviewofresearchalsoindicatedthatthere

wasnoexistingmeasureofcriticalevaluationskills.Basedonareviewoftheevidenceitwas

decidedtocreateanewmeasurethathadstudentsevaluatemultiplesourcesinaforcedresponse

assessment.

57

Theworkondispositionsofonlinereadingcomprehensionisinitsinfantstages.Thisstudy

hopestocontinuethework.ItwasdecidedthatarevisedDORC(O’Byrne&McVerry,2009)would

beusedinthisstudy.

Thiscurrentstudyseekstobuildonpreviousworkbyansweringfourresearchquestions:





comprehension?





comprehension.



restricted information spaces?




Thepresentstudywillcontributeinsightstotheexistingliteratureinvestigatingonline

readingcomprehension.Mostspecificallythisstudywilltaketherestrictednatureofthe

informationspaceintoaccountasitinvestigatesthecontributionsthatreadercharacteristicsof

verbalintelligence,backgroundknowledge,criticalevaluationskills,anddispositions,maketo

58

modelsofonlinereadingcomprehension.Bytestingmultiplemodelsofonlinereading

comprehensionassessmentthisstudywilladdtotheresearchbaseoftheliteracyfield.

Thisstudyalsowillhelpadvanceeffortstomeasureonlinereadingcomprehensionskills.As

partofthestudytwomeasuresofonlinereadingcomprehensionandameasureofcritical

evaluationskillswillbecreatedandvalidated.Furthermoreapreviousinstrumentmeasuring

dispositionsofonlinereadingcomprehensionwillberevisedandtested.

Thisstudywillalsoofferimportantinsightsintothestrategyuseofstudentswithvarying

degreesofonlinereadingcomprehensionability.Thequantitativeresultswillbeusedtoidentify

participantsinthethinkaloudportionofthisstudy.Thisdatawillbecrucialinexploringthe

findingsfoundinthequantitativemodels.Thedatawillalsoofferimportantinsightsintothetypes

ofstrategyinstructionthatstudentwillneedinordertomakemeaninginanetworkedsociety.

Thisstudywillcontributetoexistingliterature;advanceeffortstomeasureonlinereading

comprehension,andofferinsightsintotheskillsstrategiesusedbystudentsengagedinonline

inquirytasks.Theseconclusionswillhelptheresearchcommunitydevelopmorerobustmodelsof

readingcomprehensionwhileofferingimportantinstructionalstrategiesforeducators.

59

CHAPTERIII

MethodsandProcedures

Thisstudysoughttoexaminesomeofthecomplexitiesthatappeartobeapartofonline

readingcomprehension.Specificallythisstudyexaminedtherelativecontributionsmadebya

reader’sbackgroundknowledge,criticalevaluationskills,anddispositionsintwomodelstopredict

scoresonanonlinereadingcomprehensionmeasure.Thefirstmodelusedalessrestricted

informationspacewherestudentscouldlocateanysourceonthetopic.Thesecondmodeluseda

morerestrictedinformationspacethatrequiredstudentstolocatespecificsources.Thestudyused

amixed‐methoddesign(Tashakkori&Teddlie,2003)thatcombinedqualitativeandquantitative

researchapproaches.

Thepresentstudywasconductedinaseriesofthreephases.Inphaseone,multiple

instrumentsweredevelopedtoserveasindependentanddependentvariables.Inphasetwo,

regressionanalysiswasusedinanattempttounderstandfactorsthatexplainedvarianceestimates

inscoresofanonlinereadingcomprehensionmeasure.Inphasethree,verbalprotocolmethods

(Afflerbach,2002)wereusedtoexplorestrategyuseamongparticipantswhovariedintheironline

readingcomprehensionproficiency.Thesetwoapproaches:aregressionanalysisandverbal

protocolanalysisledtoaninterpretationofthedatathroughaconvergenceofbothquantitative

andqualitativedata.Thismixedmethodstudyexploredthefollowingresearchquestions:





comprehension?



60



comprehension.







QuantitativeProcedures

SettingandParticipantSelection

Therewere131seventhgradestudentsinitiallyincludedinthequantitativeproceduresof

thisstudy.Seventhgradewaschosenbecauseitrepresentsapointsomewherenearthemiddleof

grades1‐12andthusresultsmaybesomewhatsimilartostudentsfromawiderrangeofadjacent

gradelevels.Furthermore,theuseoftheInternetasatoolforreadinginformationaltextismore

commonatthisgradelevel(Fox&Rainie,2002).Sevenschooldistricts,fromaconveniencesample,

wererecruitedthroughpersonalcontacts.Fromthisinitialsamplethreeschoolswereselectedto

ensurerepresentationfromhigh,medium,andlowsocio‐economicdistrictsasmeasuredbythe

DistrictReferenceGroup[DRG](StateofConnecticut,2010).

DRGsrepresentastatisticalcategorycreatedbythestateofConnecticutforstatistical

reporting.DRGsarecalculatedthroughmedianfamilyincome,educationandoccupationlevelof

parents,familystructure,homelanguage,andoverallenrollment(ConnecticutStateDepartmentof

Education,2010)andrangefromlevelsA‐I,withAbeingthehighestSESschooldistrictandIbeing

thelowest.Toensureareasonablyrepresentativesample,acrossDRGgroups,thethreeschools

selectedforthisstudyincludedaDRGBschool,aDRGEschool,andaDRGIschooldistrict.The

61

threeschooldistrictsnotonlyvariedontheirDRGreferencegroupbutalsoontheirdemographic

make‐upandlevelofspecialservicesoffered.ThisisexplainedinTable3.1.

Table 3.1 Demographic Data of Schools School Reduced

LunchPriority District

Non-English Speaking Home

ESL Services Provided

Special Education

Non-Asian Minority

1 DRG I 78% Yes 30% 21% 17.3% 64%2 DRG E 35.2% No 7.6% 1.3% 12.7% 21%3 DRG B 5.3% No 1.8% 0% 9.2% 3.7%

School African-American

Caucasian Hispanic Asian

1 DRG I 11.1% 16.7% 69.4% 2.8%2 DRG E 11.1% 64% 20% 8%3 DRG B 2.3% 92.9% 0% 2.8%

QuantitativeMeasures

Verbalintelligence.Previousresearchhasshownthatverbalintelligencecorrelates

stronglywithreadingcomprehension(Curtis,1987).Studiesthatexaminesearchingfor

informationormultimedialearningcontrolhaveusedverbalintelligenceasacontrollingvariable.

Allen(1992),forexample,controlledforverbalintelligence,usingtheverbalcomprehension

measureofTheKitofFactorReferencedTesttoexaminehowuserssearchCD‐ROMS.Verbal

intelligenceinthepresentstudywasalsomeasuredusingtheverbalcomprehensionvocabulary

testoftheKitofFactorReferencedCognitiveTest(Ekstrom,French,Harman&Derman,1976).The

testisdesignedforstudentsfrom7thto12thgrade.Reliabilityinpreviousadministrationshas

variedfrom.76‐.89.

Theverbalcomprehensiontestconsistsof36forcedresponseitemswiththreedistractors.

Thetestisorganizedintwo,18‐questionparts.Studentshavefourminutestocompleteeachpart.

Eachquestioncontainsfourmultiple‐choicesynonymsforatargetword.SeeFigure3.1foran

exampleitem.Studentsreceivedonepointforeachcorrectanswer.

62

Figure3.1SampleVerbalComprehensionItems

Backgroundknowledge.Backgroundknowledgewasestimatedusingathree‐itemtopical

knowledgequestionnaire.Thequestionnairewasdeliveredusingacomputer‐basedsurvey.

Participantswereaskedtorespondtoapromptaskingthemtolistallthefactsordetailstheyknew

abouttheAmericanRevolution.Respondentshadtoclickabutton,“Submit,”tomoveon.Asecond

screenthenasked,“TrytothinkofoneortwomoredetailsabouttheAmericanRevolution.Donot

worryifyoucannotrememberanymore.Justtype,"MoveOn."Oncetheyhit“Submit”students

werebroughttoathirdscreenandasked,“Thinkrealhard.Isthereanythingelseyoucan

rememberabouttheAmericanRevolution?Itisokayifyoucannot,justtype,Moveon."

Backgroundknowledgescoresweredeterminedbasedonthetotalnumberofideaunits

(Leslie&Caldwell,1995)studentsprovided.Anideaunitwasdefinedasaproposition.Scoringof

ideaunitswasinformedbyCoiro’s(2012)adaptationofWolfeandGoldman’s(2005)measureof

topic‐specificbackgroundknowledge.Onepointwasgivenforanaccurateandrelevantdetail;a

halfpointwasawardedforanyideaunitthatsomewhataccurateorrelevant;azerowasgivenfor

anyideaunitthatwasinaccurateorirrelevant.Table3.2givesexamplesofeachtypeofresponse.

63

Table3.2BackgroundknowledgeResponsesTothePrompt:”ListeverythingyouknowabouttheAmericanRevolution”Inaccurateorirrelevantstatement(0points)

Generallyaccuratestatement(.5points)

Specificallyaccuratestatement(1point)

Peoplelivedinfear? ItwasawarinAmerica

TheAmericanrevolutioninvolvedGeorgeWashington

Therewerethirtycoloniesduringthewar.

usaarmywasfightingforamerican

becauseofthefamoussaying"notaxationwithoutrepresentation."

Tofreetheslavesinthesouth

Itwasawarinthe1700

Theamericanrevolutionwaswhenourcountryfoughtagainstthebritishrule

Tocalculateinter‐raterreliabilityasecondraterscored20%oftheresponsesrandomly

selected.Firsttheresearcherandtheraterscoredananchorsetoffiveexamples.Thentheyeach

scoredthe20%samplealone.Inter‐raterreliabilitycoefficientsfortotalscoresonthebackground

knowledgemeasurewasr=.82.Theyrangedacrossthethreepromptsfrom.79‐.87.

Criticalevaluation.TheCriticalOnlineInformationLiteracies(COIL)instrumentwas

basedonmeasuresdevelopedbyKiili,Laurinen&Marttunen,(2008),Brem,Russell,&Weems

(2001),andLeuetal.(2010).ThefinalinstrumentwasdeliveredusingSurveyMonkey,anonline

surveytool.Theitemsmeasuredeachofthefollowingconstructs:author,bias,publisher,and

source.

Thereweretwotypesofitems.Thefirstconsistedofscreenshotsofwebsites.Thesewere

eithersinglewebsitesorfourscreenshots.Thesecondtypeofitemrequiredstudentstoopenlinks

tofourdifferentwebsites.Thesewebsitesincludedsecondarylinkstoauthorandpublisher

information.Allofthewebsitesinvolved,eitherstaticordynamic,wererecreationsofactual

websitesthatwerehostedonauniversityownedserver.Anexampleofeachitemtypeisavailable

64

inFigures3.2,3.3,and3.4.Thetaskspresentedineachitemweresituatedinactivitiesthat

adolescentswouldbeengagedinastheysearchedforonlineinformation

Figure3.2.SingleScreenShotExamplefromCOILAssessment

65

Figure3.3FourScreenShotExamplefromCOILAssessment.Note:Thefourscreenshotswere

presentedvertically.

66

Figure3.4FourHyperlinkScreenShotExamplefromCOILAssessment.

Theinstrumentunderwentthreephasesofdevelopment.PhaseIinvolveddefining

constructsandcontentvalidation.PhaseIIinvolvedapilottestforinstrumentvalidation.PhaseIII

usedtheresultsofthevalidationstudytoinformfinalitemdevelopmentbeforetheinstrumentwas

used.Thisthree‐phaseprocessledtothedevelopmentofthefinal14‐itemassessment.

PhaseI.TheconstructionoftheCOILbeganwithaliteraturereviewtodeterminefactorsthat

influencethecredibilityandrelevancyjudgmentofsources(Judd,Farrow&Tims,2006;Kiili,

Laurinen&Marttunen,2008).Previousworkbyotherresearcherswasusedtoidentifysub

constructsthatinfluencedcredibilityandrelevancyjudgments.(SeeTables3.3and3.4).These

subconstructswereusedtodevelopmultiple‐choiceitemsforcredibilityandrelevancy.

67

Table3.3Sub‐ConstructsofCredibility

Sub‐Construct Definition

Evaluateauthor Judgingthecredibilityofawebsitebasedondetailsabouttheauthor(Harris,1997)

Evaluatesourceofclaim

Judgingthecredibilityofatextbasedonthesourceofinformationthatisincluded(Rieh&Belkin,1998;Strømsø&Bråten,2010)

Evaluatebias Judgingthecredibilityofawebsitebasedonaninclinationtowardholdingaparticularperspective(Coiro,2003;Fabos,2008)

Evaluatecontent Judgingthecredibilityofwebsitebasedoncompletenessofinformation(Harris,1997;Kiilietal.,2008)

Evaluateargument

Identifyingandcomparingperspectives(Kiili etal.,2008)

Evaluateaccuracy Comparingclaimswithasecondarytext(Meola,2004)

Table3.4Sub‐constructsofrelevancy

Sub‐Construct Definition

Evaluaterelevanceoftopic Identifyingwebsitesorsearchresultsthatwillhelpansweraquestion(Kiili,etal.,2008)

Evaluaterelevanceofawebsite

Identifyinghyperlinksorheadingsthatwillansweraquestion(Kiili,etal.,2008)

Evaluatepurpose Identifyingthepurposeorintendedaudienceofatext(Harris,1997)

Evaluatecurrency Judgingawebsitebasedonageofpublication(Meola,2004;Kiilietal.,2008)

EvaluateUsability Judgingawebsitebasedoneaseofuse andreadability(Meola,2004;Kiilietal.,2008)

68

Contentvalidationtechniques.Inordertoestablishitemvalidity,theinstrumentunderwent

acontentvalidationphasewithexpertsfamiliarwithcriticalevaluationresearchtodevelop

definitionsfortheconstructs(McKenzie,Wood,Kotecki,Clark,&Brey,1999).Thesixexperts

includedprofessorsandgraduatestudentsfamiliarwithresearchinthecriticalevaluationofonline

information.Theexpertsratedthedimensionalityofeachofthetwentymultiple‐choiceitemsby

indicatingwhichoftheconstructandsubconstructstheitemmeasured.Itemsidentifiedby90%of

participantsasmeasuringthehypothesizedconstructwerekeptforfurtheranalysis(Gable&

Wolfe,1993,McKenzieetal.,1999).

AContentValidityIndex(CVI)(Rubio,Berg‐Weger,Tebb,Lee,&Rauch,2003)wascreated

foreachitemusingthefeedbackprovidedbytheexpertstotestformultidimensionalityofitems.A

CVIiscalculatedbyhavingeachraterrankanitemfromone(irrelevant)tofour(extremely

relevant).TheCVIistheproportionofitemsthatreceivedatleastathreeorfourbytheraters.For

inclusioninthefirstiterationoftheinstrument,theCVIforeachitemneededtoexceedathreshold

of0.70(Rubioetal.,2003).Finally,theexpertswereencouragedtoleavewrittenfeedbackthatwas

usedtoensuretheadequacyandaccuracyofdefinitionsofconstructsanditemsconstructed

(McKenzieetal.,1999).Table3.5liststheresultsfromthecontentvalidationstudy.Itemsarelisted

intheordertheyappearedontheassessment.

69

Table3.5ResultsoftheContentValidationStudy

Item %Correct

%ofExpertsWhoCorrectlyIdentifiedTheConstruct

CVI Comment

1 85 87.5author 2.875 Someconfusionoverauthorandsource.Collapsedsourceintocontentforsecondround.

2 54 90usability 2.33 Expertsfeltreadabilityistoosubjectiveandusabilitytoosituated.Thisitemandsubconstructweredeleted.

3 62.5 12.5argument NA‐ Somereviewerslikedargumentasasubconstructbutitcausedconfusion.Theitemswerecollapsedundercontent.

4 88 100currency 2.67 Somearguedcopyrightofwebsiteisnotclearindicatorofdateofinfo.MovedanswerchoiceawayfromAvalanche.

5 100 62purpose 1.33 Peopleliketheitembutfeelpurposeisreallyarelevancyjudgment.CVIwouldbehigherifpurposewasconsideredarelevancyjudgment.

6 65 64relevancy 2.2 Thesearchresultsneedtobelessrelevantifthisisarelevancyjudgment.Betterdistracterswerepicked.

7 90 34relevancy 1.8 Mostreviewersfeltthatknowingwebsitegenreswasnotatimportanttomeasuringcriticalevaluation.Theitemwasdeleted.

8 87.5 66currency 2.8 Itemkept.Examinedwhypeopledidnotpickcurrency.

9 87.5 12.5argument NA EditeditemsoitiswhichwebsiteusesthebestdetailstosupporttheclaimPlutoisnotaplanet.Collapsedargumentundercontent.

10 100% 83source 2.75 Sourcewascollapsedundercontent

11 85 85.7author 2.75 Nochanges

12 28% 20usability NA Expertswereunsureofthegradelevel,againsomecommentedthatreadinglevelissubjective.Deletedallusabilityitems.

13 83.3 50argument 2.33 Peopleliketheitem.Thiswascollapsedunder

70

content.

14 75 85.7relevanceoftopic

2.8 Nochanges

15 100 57.1purpose 2.25 Againexpertsfeltthatevaluatingpurposewasarelevancyjudgment.

16 50 .25usability NA Deleteditemormakeitarelevancyjudgment

17 85.7 42.8argument 2.33 Argumentwascollapsedintocontent.

18 85.7 75relevanceofhyperlinkonawebsite

2.67 Renamedsubconstruct

19 100 75relevanceofinfo 2.67 Changedthenameofothersubconstruct.

20 50 50source 2.33 Sourcewascollapsedundercontent.

ResultsofphaseI.Asaresultofthecontentvalidation,revisionsweremadetoitemsto

matchsubconstructshypothesizedbythepanelofexperts.ThesechangesarelistedinTable3.6.

Theexpertsfeltthatpurposewasmoreofarelevancyjudgmentratherthanacredibilityjudgment.

Furthermorethepanelcouldnotagreeonwhichitemsmeasuresthesubconstructof

evaluatecontent.Thecontentvalidityindexfortheseitemswastoolow.Giventhedisagreement

overthesubconstructofpurposeandthelowCVIscoresforevaluatingadecisionwasmadeto

revisetheconstructofrelevancy.Thereforethesubconstructofpurposewasmovedfromthe

constructofcredibilitytotherelevanceconstruct,andthesubconstructofevaluatecontentwas

deleted.Thenewlydefinedevaluaterelevanceconstructwasnowdefinedbytwonew

subconstructs.Thisincludedevaluatesearchresultsandevaluateinformationonawebsite.This

ledtotherevisionofitems:6,14,18,and19.

Additionallytheexpertsfeltthatthesubconstructofusabilitywastoosubjective.Theitems

askedstudentstoevaluatethereadinglevelsofwebsites.Thepanelfeltthissubconstructwastoo

dependentonindividualdifferences.Thusthesubconstructofusabilitywasdropped.Theitems

71

werereplacedwithitemstorepresentthesubconstructofbias.Thisledtotherevisionofitemstwo

and12.

Item20wasalsorevisedduetothelownumberofexpertsscoringtheitemcorrectly.Inthe

originalitemtheparticipantswerepresentedwithfourscreenshotsofwebsites.Theywereasked

whichwebsiteusedinformationfromthemostreliablesource.Inthereviseditemspecificclaims

andtheirsourceswerehighlighted.Thesubconstructwasrenamed“evaluatesources”toevaluate

sourcesofinformation.

72

Table3.6RevisionsbasedontheContentValidationoftheCOIL

DraftAssessmentIteminContentValidationStudy

AssessmentItemsIncludedInPilotInstrument

1.Evaluateauthor 1.Evaluateauthor

2.Evaluateusability 2.Evaluatebias

3.Evaluateargument 3.Evaluateargument

4.Evaluatecurrency 4.Evaluatecurrency

5.Evaluatepurpose 5.Evaluatepurpose

6.Evaluaterelevanceoftopic 6.Evaluaterelevancysearchresults

7.Evaluatecontent 7.Evaluateauthorexpertise

8.Evaluatecurrency 8.Evaluatecurrency

9.Evaluateaccuracyofclaims 9.Evaluateaccuracyofclaims

10.Evaluatesourceorclaim 10.Evaluatesourceofclaim

11.Evaluateauthor 11.Evaluateauthor

12.Evaluateusability 12.Evaluatebias

13.Evaluateargument 13.Evaluateargument

14.Evaluaterelevanceoftopic 14.Evaluaterelevanceofsearchresults

15.Evaluatepurpose 15.Evaluatepurpose

16.Evaluatecontent 16.Evaluaterelevanceofsearchresults

17.Evaluateaccuracyofclaims 17.Evaluateaccuracyofclaims

18.Evaluaterelevanceoftopic 18.Evaluaterelevanceofinformationonwebsite

19.Evaluaterelevanceofwebsite 19.Evaluaterelevanceofinformationonwebsite

20.Evaluatesources 20.Evaluatesourceofinformation

73

PhaseII..Totestthereliability,orinternalconsistencyofthescales,acoefficientalphawas

calculated(Pettetal.,2003)usingresultsfromanotherstudy(O’Byrne,2011).Apaperandpencil

versionofthetwentyitemswasadministeredto197seventhgraders.Theachievedcoefficient

alphaforthetwohypothesizedscaleswas.358forrelevancyand.312forcredibility.Thecombined

instrumenthadanoverallcoefficientalphaof.339.Investigationoftheinter‐itemcorrelations

revealedthatthecorrelationsbetweentheitemsinthescalewerelowornegativelycorrelated.

Becausethisassessmentdidnotmeetthethresholdof0.70foracoefficientalphathemeasurewas

showntobenotreliable.

PhaseIII.Giventheinadequatereliabilityofthelastiterationadecisionwasmadetorevise

theinstrument.Itwasdecidedtomakethreechanges:(a)reducethenumberofscalesand

subscales;(b)makedistractorseasiertorecognize;and(c)simplifythetestingformat.

Adecisionwasmadetocreateitemshypothesizedonlytomeasurecredibility.Thisdecision

wasboththeoreticalandpractical.Inboththeoreticalpieces(Coiro,2003;Hartmanetal.,2010)

andinpastinstrumentvalidationstudies(McVerry,O’Byrne,andRobbins,2009)researchersinthe

fieldofnewliteraciesofonlinereadingcomprehensionhavesuggestedthatwhilearelevance

judgmentisanevaluationofwebsites,thecognitiveprocessesinvolveddiffersinimportantways

fromjudgingcredibility.InfactMcVerryetal.(2009)foundthatscoresonrelevancyitems

hypothesizedtoloadwithothercriticalevaluationscoresactuallyloadedonscalescontainingthe

locatingscores.

Reducingthenumberofconstructsalsohasadvantagesforimprovingthepracticalityofthe

instrument.Thetotalnumberofitemsisreducedwhichmakesiteasiertoadminister.Whilefewer

itemsreducetheoverallcoefficientalpha(Cronbach,1972)aone‐factormodelmayhavegreater

internalconsistencythanatwo‐factormodel.Thusitwasdecidedtoonlymeasuretheevaluation

subconstructsofauthorexpertise,publishercredibility,sourcecredibility,andauthorbias.

74

PhaseIII:Contentvalidation.Thisphaseoftheinstrumentwasvalidatedinthreesteps.The

firststepinvolvedanadditionalroundofacontentvalidationsurveybyanexpertpanel.The

secondstepusedstructurallypromptedthinkalouds(Afflerbach,2002)duringcognitivelabs

(Ericsson&Simon,1999)toelicitfeedbackfrom7thgradestudentsonitemformatanddifficulty.

Finallytheinstrumentwasadministeredto1207thgradersfromhigh,low,andmediumSESschool

districtstotestforreliability.

Thefirststepinrevisingtheinstrumentinvolvedanotherrigorouscontentvalidation

process(McKenzieetal.,,1999).First12itemsweredeveloped,threeitemsforeachofthenew

constructs.Then18expertswereidentified.Theexpertsratedtheconstructvalidityofeachofthe

twentymultiple‐choiceitemsbychoosingwhichoftheconstructtheitemmeasured.Allitems

identifiedby90%ofparticipantsasmeasuringthehypothesizedconstructwerekeptforfurther

analysis(Gable&Wolfe,1993,McKenzieetal.,1999).NextaContentValidityIndex(CVI)(Rubioet

al.,2003)wascalculatedforeachitem.TheCVIwascalculatedasaproportionofexpertswho

indicatedifanitemwasextremelyrelevant(4)orveryrelevant(3)onafourpointLikertscale.

FollowingthecalculationofaCVI,aContentValidityRatiowasalsocalculatedtoevaluate

theextenttowhichameasurerepresentsagivenconstruct(McKenzieetal.,1999).Tocalculatea

CVR,youfirstdeterminethedifferencebetweenthenumberofexpertswhomarkedanitemas

essentialandhalfthetotalnumberofexperts.ThentheCVRiscalculatedbydividingthisnumber

byhalfthetotalnumberofexperts.ForinclusioninthefinalversionoftheinstrumenttheCVRfor

eachitemwasrequiredtoexceed0.70(McKenzieetal.,1999).Thus,tobeincludeditemswere

requiredtohaveaCVIof2.67andaCVRofatleast0.70.Table3.7showstheinitialresultsofthis

process.Items2,5,6,8,and11failedtomeeteithertheCVIand/ortheCVRcriteria.Theseitems

wererevised.

75

Table3.7

ContentValidationResultsforRevisedCOIL

Item %OfParticipantCorrectlyRespondingtoItem

ConstructMeasured

%WhoIdentifiedtheConstruct

CVI CVR Comment

1 100 Author 94.1% 3.5 1.0

2 88 Author 40.0% NA NA Confusionbetweenauthor,source,andpublisher.Itemreworded.

3 66.6% Bias 93.3% 3.15 .866

4 93.3% Publisher 100% 3.4 1.0

5 100% Author 0.0% NA NA Hypothesizedpointofviewasundertheconstructofauthorbutallparticipantsrankeditasbias.Theconstructwasrevised.

6 93.3% Source 93.3% 3.08 .858 Neededmoreplausibledistractors

7 85.7% Publisher 92.2% 3.30 .858

8 80.0% Author 80.0% 3.33 .867 Itemwasrevisedtoaddeasierdistractorsandfocusrespondentstothepublisherandnottheauthor.

9 84.6% Source 92.9% 3.23 .857

10 50% Publisher 83.3% 3.16 1.0 Neededmorediscriminantdistractors

11 85.7% Bias 76.7% Neededtochoosewebsiteswithaclearauthorandpublisher.Confusionbetweenthetwoconstructs.

12 72.7 Source 92.9% 3.23 .857

76

PhaseIII:Revisionsbasedoncontentvalidation.Basedonthecontentvalidationresultsthere

appearedtobesomeconfusionoverthedefinitionsforseveralsubconstructs.Specificallytherewas

confusionoverauthor,source,andpublisher.Thiswasevidentinitem2,asonly40%ofexperts

correctlyidentifiedtheconstruct.Torectifythissituationseveralrevisionsweremade.First,

insteadofprovidingstudentswithawebsiteandaskingthemwhichsiteuseddetailsfrommore

reliablesources,aspecificdetailwaschosenfromeachsite.Items6,9,12wererevisedsothata

specificdetailineachwebsitewashighlighted.Eachincludedthesourceoftheinformation.

Second,onquestionsaskingaboutthecredibilityofpublishersthecorrectanswersanddistractors

wererevisedtoincludeonlytheheaderortheaboutuspageaboutapublisher.

PhaseIII:Cognitivelabs.Nextaseriesofcognitivelabs(Ericsson&Simon,1999)were

conductedwitheightstudentsfromanaverageSESschooldistrict.Structurallypromptedthink‐

alouds(Afflerbach,2002)wereused.Thisallowedtheresearchertotestitemformat,wording,and

difficultywiththetargetpopulationforalloftheitems.

Usingthinksaloudprocedures,studentswerefirstaskedtoindicatewhichanswerchoice

theythoughtwascorrectforeachitem.Theythenwereaskedwhytheythoughteachoftheother

answerchoiceswereincorrect.Nextstudentswereaskedtoofferfeedbackoneachitemformat.

PhaseIII:Cognitivelabsresults.Basedonitemdescriptivesandthecognitivelabresultsthe

instrumentneededfurtherrefinement.SeeTable3.8forstudentperformanceonthetwelve‐item

assessment.Aoneindicatesacorrectanswer.Azeroindicatesanincorrectanswer.Themeanscore

isanestimateofitemdiscriminationasthepercentageofstudentswhoscoredcorrectlyonthe

item.Theseresultsindicatedthat3,8,and11hadtoberevised.

77

Table3.8CognitiveLabItemDiscrimination

Item 1 2 3 4 5 6 7 8 9 10 11 12 Total

1 0 0 0 1 1 0 0 1 0 0 1 5

1 1 0 0 1 0 0 0 1 0 0 1 5

1 1 0 1 1 1 0 0 1 1 1 0 8

1 1 0 1 1 0 1 0 0 1 0 1 7

1 0 0 1 1 1 1 0 0 1 0 0 6

1 1 0 1 1 1 1 0 0 0 1 1 8

1 0 1 1 1 1 1 0 0 0 0 1 7

Mean 1 .63 .13 .63 .88 .75 .57 0 .48 .43 .25 .71 6.30

Thethinkalouddataalsosuggesteditemrefinement.Thecognitivelabquestionsthat

examinedbiasby“askingwhichwebsitesincludestrongwordsandimagestoinfluencethe

audience”neededmorerecognizabledistractors.Studentshaddifficultyunderstandinghow

informationwasshapedbyanauthororpublisher.Questionsthree,five,andelevenwererevisedto

makedistractorsmorerecognizable.

Studentresponsesalsoindicatedthatsomeindividualitemsneededtoberevised.No

studentsscoredcorrectlyonquestioneightcorrectly.Thisquestionhadstudentsevaluatean

author’sexpertisegivenherbiographypage.Thedistractorswererevisedtomakethequestion

easier.

Thestudentsfromthethink‐aloudreportedoneaseofusewiththeSurveyMonkey

interface.Theycouldclickonembeddedlinks,andcoulddelineatebetweentheitemstemand

forcedchoiceresponses.Theywerealsoasked,andgiventheoption,tohavetheirwebsitesopenin

newwindowsortabs.Allofthestudentsreportedthattheypreferredwebsitestoopeninnewtabs

ratherthanwindows.Studentsalsosuggestedthewordingonsomeitemsneededrevisions.

78

PhaseIII:Finalinstrumentationandadministration.Therevised12itemswerethen

administeredtothe131studentsinthestudy.Studentstooktheassessmentinoneclassperiod,in

theirclassroomusinglaptops.

Areliabilityanalysiswasrunfollowingtheadministrationofthe12‐itemassessment.A

coefficientalpha(Cronbach,1972)wascalculatedat.43.Thisassessmentdidnotmeetthe

thresholdof0.70foracoefficientalphathemeasurewasshowntobenotreliable.Thelow

reliabilitymayhavebeenduetotheshortassessmentlength.

UsingtheSpearman‐Brownprophecythetestwouldhavetoincreasebyafactorof3.16.

Giventhecomplexityoftheassessmentandthelimitationsofclassroomschedulesa40to50‐item

assessmentwouldnotbepossible.Thusthedecisionwasmadetomorethandoublethe

assessmentlengthbycreating16newitems,foureachfor,publisher,author,bias,andsource.The

itemswerecreatedbymirroringitemsthatwereacceptablefromthecontentvalidationsurvey.

Theseitemswerethenadministeredtothestudentsinthestudybeforetheycompletedanyother

additionalassessmentsandafteraweekoftheinitialbatteryoforiginalitems.

PhaseIII:instrumentationresults.Areliabilityanalysiswasthenconducted.Oneofthe

additionalnewitemsmeasuringauthorexpertisewasremovedfromtheassessmentaprioridueto

anerrorinadministration.Twenty‐sevenitemswereincludedinthefinalanalysiswithanN=110.

Thecoefficientalphawithall27itemswas0.560.Thiswasnotacceptableforresearchgivenitdid

notpassourthresholdforaninternalconsistencyvalueof0.700(Peterson,1994).

Nextanexaminationofthedescriptivestatisticsoftheitemsrevealedthatmanyitemshad

meanscoreoflessthan0.20.Thiswouldmeanlessthan20%ofparticipantsscoredtheitem

correctly.Anyitemwithameanscoreoflessthan0.20wasremovedfromtheanalysis;thisleadto

theremovaloffiveitems.SeeTable3.10foradescriptionofthefinalitemsincludedinthe

assessment.Thecoefficientalphawasthenrecalculatedwithoutthefiveremoveditems.Internal

consistencyfortheremainingitemswascalculatedat.618.

79

Thentheinter‐itemcorrelationmatrixwasreviewed.Anyitemthathadamajorityof

negativeinter‐itemcorrelationswasdeleted;thisleadtotheremovalofsevenadditionalitems.A

reliabilityanalysiswasconductedandthecoefficientalphawasnow0.713.Afterexaminingthe

“scalesifitemdeleted”table,ameasurecalculatedbySPSSsoftware,thedecisionwasmadeto

removethreeadditionalitems.Thecoefficientwasrecalculatedfortheremaining14itemsofthe

assessmentat0.722.Thiscoefficientalphaisacceptable.ThefinalCOILassessmentconsistedofthe

14itemslistedinTable3.9.SeeAppendixCforthecompleteinstrument.Therewerefouritems

measuringauthorexpertise,fouritemsmeasuringpublishercredibility,fouritemsmeasuringbias,

andtwoitemsmeasuringsourcecredibility.Sixoftheitemshadonlyonescreenshot,sixofthe

itemshadfourscreenshots,andtwooftheitemsrequiredparticipantstoclickonfourhyperlinks.

Table 3.9 Final Items in Validated and Reliable COILItems In Final Version Items Deleted1. Evaluate author: Where would you click to learn more about an author?*

3. Evaluate bias: Which website uses strong words or images to persuade the audience?**

2. Evaluate author: Which websites about Chihuahuas and asthma was created by the most knowledgeable author?***

6. Evaluate Source: Which website uses details from the least reliable source?**

4. Evaluate publisher: Where do you click to learn more about a publisher?*

7. Evaluate publisher: Which website was created by a more reliable publisher?***

5. Evaluate bias: Think about the author’s point of view. What may influence the way he thinks about energy drinks?*

8. Evaluate Author: Given the author's profile page what is her expertise?**

10. Evaluate publisher: Which publisher creates a website with the most credible medical information?**

9. Evaluate Source: Which websites uses details that are from the most reliable source about healthy snacks?***

14. Evaluate bias: Think about the authors' point of view. How does the authors' point of view influence the words and images used on the website?***

11. Evaluate Bias: Which website uses strong words, phrases, or images to persuade readers?***

15. Evaluate author: Look at this website. What is the author's expertise?*

12. Evaluate Sources: Which website uses information from the most reliable source?**

80


13. Evaluate publisher: Who is the publisher of this website?#

18: Evaluate bias: Which website uses strong words or images to persuade the audience?**19: Evaluate author: Which author is the best expert on volcanoes?**

17. Evaluate author. Look at these websites. Which website was created by the author with the most expertise on mosquito ringtones?***20: Evaluate author: Where would you click to learn more about an author?*

21: Evaluate bias: Think about the author's point of view. How does her point of view shape the words and images on this website?#

23: Evaluate Source: Where would you click to learn more about the sources an author used?*

22: Evaluate publisher: Which website was created by a more reliable publisher?***

25: Evaluate Publisher: Which website about smoking hazards was created by the most reliable publisher?**26: Evaluate Source: Which source used in the websites is the most reliable source to answer the question, "What killed the dinosaurs?"**

24: Evaluate Source: Which discussion board post uses details from the most reliable source?*

27: Evaluate Publisher: Who is the publisher of this website?*Notes*‐onescreenshot,#‐clickononelink,**‐fourscreenshots,***‐clickonfourlinks

Onlinereadingcomprehensionassessments.Twomeasuresofonlinereading

comprehensionwerecreatedforthisstudy:ORCAlessrestricted(ORCA‐LR)andORCAmore

restricted(ORCA‐MR).Theassessmentsweredraftedandunderwentnumerousvalidityand

reliabilitytests.Intheinitialiteration,eachassessmentconsistedoftwoseparateInternetinquiry

activities(Leuetal,2007).Eachactivityrequiredstudentstolocate,evaluate,synthesize,and

communicateinformation.Onthelessrestrictedtaskstudentscoulduseanywebsiteonagiven

topic.Onthemorerestrictedtasksstudentswereaskedtolocatespecificwebsitesonagiventopic.

EachparticipantthushadascoreforORCA‐LRandORCA‐MR

81

Thedomainfortheonlinereadingcomprehensiontasksusedinthisstudywassocial

studies.Thisdomainwaschosenbecausepreviouswork(Perfetti,Britt,Georgi,&Mason,1994;Van

Sledright,2002)hashighlightedtheimportanceofreadingandevaluatinginformationacross

multiplesources.(Weinberg,1991).Thus,thedisciplinarydemandsofsocialstudiesfitnicelywith

theskillsrequiredtoreadinonlineenvironmentswherereadersoftenhavetolocate,evaluate,and

synthesizeacrossseveralsources.

TopicsfortheInternetinquiryactivitiesrevolvedaroundtheAmericanRevolution.This

topicwaschosenbecauseitiscoveredinfifthgradeinthestateofConnecticut(StateofConnecticut

DepartmentofEducation,2009).Thus,studentsinthestudyarelikelytohavehadexposuretothe

topicandcontent.Alsostudents,bytheseventhgrade,areexpectedtohavecompetencein

accessinginformationfrommultiplesources,evaluatingarguments,andcommunicatinganswers

usingdigitaltexts(StateofConnecticutDepartmentofEducation,2009).Theoriginaltopicsforthe

activitieswere:(a)thetechnologiesoftheAmericanrevolution‐lessrestricted,(b)theaccuracyof

Leutze’spaintingofGeorgeWashingtonCrossingtheDelaware‐lessrestricted,(c)theturningpoint

oftheAmericanrevolution‐morerestricted,(d)thecausesoftheAmericanrevolution‐more

restricted.

Onthelessrestrictedtasksstudentscouldfindanysourcerelevanttotheirtopic.So,for

example,studentswereaskedtolocateanywebsitewithinformationabouttheaccuracyofthe

Leutze’spaintingofGeorgeWashingtonCrossingtheDelaware.Onthemorerestrictedtasks

studentswereaskedtofindaspecificarticleorwebpagelocatedataspecificwebsite.Forexample

studentswereaskedtofindthewebpage“BattleofSaratoga”onthewebsite

AmericanRevolution.orgforthemorerestrictedtask,theturningpointoftheAmericanRevolution.

TheassessmentscreatedforthisstudywerebasedonORCAassessmentsdevelopedfor

previousstudies(Coiro,2012;Leuetal,2012;2008).Thisformathasstudentsfirstlocate

information,synthesizethisinformation,evaluatethesourcesoftheinformation,takeaposition,

82

andthemcommunicatewhatwaslearned.Table3.10showsthatpreviousadministrationsofonline

readingcomprehensionassessmentsusingthisformathavedemonstratedgoodestimatesofboth

validityandreliability

Table3.10.Descriptive,Validity,andReliabilityScoresonORCAassessments.Instrument/# Items Range N * Validity ** ReliabilityORCA-IM 3-36 of 38 12 66.5% .85-.95

ORCA-Blog 0-30 of 32 89 59.2% .84ORCA-I 0-56 of 60 120 51.7% .92ORCA-II 0-56 of 60 120 44.1% .91ORCA-Iditarod 0-33 of 42 220 53.1% .88ORCA-Iditarod, revised

0-20 of 30 373 41.6% .73

*‐Asmeasuredusingproportionoftotalvarianceexplained**‐AsmeasuredusingCoefficientAlpha(Chronbach,1972)

Eachactivityoriginallyconsistedoffourlocatingscorepoints,fourevaluationscorepoints,

foursynthesisscorepoints,andfourcommunicationscorepoints.Theactivitiesbeganwithatask

introduction,eachofwhichcontainedaproblemandadescription.Thetaskintroductionswere

designedtobeasparallelaspossibleacrossthetasks,regardlessoftherestrictednatureofthe

locatingtasks.ThesearepresentedinTable3.11.

ThelocatingitemsdifferedbetweentheORCA‐LRandtheORCA‐MR.Onthelessrestricted

tasksstudentscouldlocateanyfourrelevantsources.Onthemorerestrictedtasksstudentshadto

locatefourspecificsources.

83

Table3.11TaskIntroductionfortheORCAInternetInquiryTasksTask Less Restricted Task Introductions More Restricted Task IntroductionTask A Introduction

The problem: Is the painting of George Washington crossing the Delaware River historically accurate?Mr. Barnes's history class is debating the accuracy of the painting "George Washington Crossing the Delaware" by Emanuel Gottlieb Leutze.

They are debating if the painting is accurate, or truthful, on their class discussion board.

You will do research online and decide if the painting is accurate.

The problem: What was the turning point of the American Revolution?

Mr. Barnes's history class is debating the turning point of the American revolution on the class discussion board.

You will do research to help them answer the question: What was the turning point of the American Revolution?

Task B Introduction

The problem: What role did some women play in the American Revolution?In Mr. Barnes's history class they are discussing women and the American Revolution. His class is posting messages on the class discussion board.You will do research to answer the question: What role did women play during the American Revolution?

The problem: What were the causes of the American Revolution?Mr. Barnes's history class is talking about the causes of the American Revolution on their discussion board.You will do research online to answer the question: What were the main causes of the American Revolution?

84

Foreachinquiryactivity,studentshadtolocatetwowebsitesonthetopic.Ontheless

restrictedtaskthiswastolocateanytwo,relevantsources.Onthemorerestrictedtaskstudents

wereaskedtofindtwo,specificallydefinedsources.Nexttheyhadtocombinethemainideasofthe

twowebsitesintoasynthesisstatement.Thentheylocatedanadditionalsetoftwowebsites,

followingthesamerestrictedparameters.Nexttheywroteanothersynthesisstatementcombining

whattheyreadonthetwowebsites.

Alloftheremainingitems,regardlessoftherestrictednatureoftheinformationspace,were

parallelfromthispointforward.Thestudentswererequiredtolocateanauthorofoneoftheir

websites,evaluatetheauthor’sexpertise,evaluatetheevidenceusedbytheauthor,andevaluate

theauthor’spointofview.Nextstudentstookapositionandbackeduptheirclaimwithevidence

fromwhattheyread.Finallystudentswererequiredtologontoaclassdiscussionboard,makean

originalpost,explainingwhattheylearned,andrespondtooneotherpersononthediscussion

board.Theitemswerescoredwitha1forcorrectanda0forincorrect.Thedichotomousrubric

usedtoscoreeachitemisavailableinAppendixB.

ORCAcontentvalidation.EachORCAassessmentunderwentathree‐stepvalidation

process.Firstapanelofsocialstudiesexpertsevaluatedthemeasuresduringafocusgroup.Next

themeasureswereevaluatedbyafocusgroupofonlinereadingcomprehensionexperts.Finallya

seriesofcognitivelabs(Ericsson&Simon,1999)wereconducted.

Socialstudiesexpertpanelvalidation.Aftertheinitialinquiryactivitiesweredeveloped,

theassessmentsweregiventoafocusgroupthathadexpertiseinthefieldofsocialstudiesand

technologies.ExpertswererecruitedusingTwitter,asocialmediatool.Arecruitmentcallwentout

usingseveralhashtags,amethodforcategorizingmessages:#sschat‐socialstudieschat,#edtech‐

educationaltechnology,and#edchat‐generaleducationdiscussion.Sevenpeoplerespondedtothe

85

call.Thefinalpanelconsistedoffourexperts.Twoweredoctoralstudents,andtwowereclassroom

teachers.

Thepanelmembersthenreviewedthematerials.Theywereaskedtocompleteeachofthe

fourversionsoftheORCA.Theresearcherthenmetwiththepanelmembersinavideo–conference.

Thepanelmemberswereaskedtoassessthevalidityoftheinstrument.Interviewquestions

focusedonthetasksbeingspecifictothedomainofsocialstudies.Theywerealsoaskedifthe

onlinereadingcomprehensionskillsassessedinthetasksmirroredtheskillsusedbyexpertsinthe

field.Finallytheexpertpanelwasaskedifthetopicsincludedinthetasksweresuitedforthetarget

populationofseventhgrade.

Overall,thecontentexpertsfelttheassessmentsadequatelyrepresentedinquirytasks

requiredinthedomainofsocialstudies.Theyremarkedthatevaluatingtheauthorandsourcesof

evidencewereimportantskillsatthislevel.Onthecommunicationtasktheexpertswanted

studentstoalsonotewheretheyfoundtheirinformation.

Theexpertsdidrecommendthatthetopicoftechnologyandtherevolutionarywarwas

unsuitableforthetask.Theyfeltthatwebsitesaboutthistopicwerescarce;andthosethat

studentswouldfindwereabovethenormalreadinglevelofaseventhgrader.Theexperts

recommendedchoosinganewtopic.TechnologyandtheAmericanRevolutionwasreplacedwith

theRoleofWomenDuringtheAmericanRevolution.

Onlinereadingcomprehensionexpertpanel.Thenextstepincontentvalidationinvolved

expertsinthefieldofonlinereadingcomprehension.TwodoctoralstudentsfromtheNew

LiteraciesResearchLabattheUniversityofConnecticutwererecruitedtovalidatetheORCA

activities.Theymetwiththeresearcherandwentthrougheachtask.Thereviewersfeltthatthe

fouractivitiesadequatelycapturedtheelementsoflocating,evaluating,synthesizing,and

communicating.

86

Cognitivelabs.Thefinalstepincontentvalidationwasaseriesofcognitivelabs(Ericsson&

Simon,1999).EightstudentsfromanaverageSESschooldistrictwereselectedtocomplete

structurallypromptedthink‐alouds(Afflerbach,2010).Astructurallypromptedthinkaloud

interruptsstudentsatkeydecisionpoints.Theirscreenactionsandvoiceswerecapturedusing

IshowU.Thisisascreenrecordingsoftwarethatrecordsallactiononthescreenalongwiththe

student’sandresearcher’svoices.Studentsfirstcompletedatrainingtasktofamiliarizethemselves

withcompletingathink‐aloud.Thestudentsthencompletedonemorerestrictedtaskandoneless

restrictedtaskwithastructurallypromptedthinkaloud.Afterthestudentscompletedallofthe

think‐aloudstheywereasked:(a)howtheassessmentscomparedtohowtheyusuallyreadand

writeinsocialstudies;(b)howtheyopenmultiplewindows;(c)howtheynormallytakenotes

whenconductingonlineresearch.

ThethinkalouddatarevealednumerousissueswiththeformatoftheORCAassessment.

Firstonlocatingtasksthestudentsstruggledwithlocatingtherestrictedwebsitesgivenonlythe

titleofthepagewithouttheURLextensions(.com,.org).Forexample,onewebpagetheywere

askedtofindwaslocatedonawebsiteAmericanRevolution.Thereweremultiplewebsiteswiththe

titleAmericanRevolution.Thusitwasdecidedtoaddtheextensiontoallwebsitetitlesonrestricted

tasks.Studentswerenowaskedto,“Findthewebpage‘TheBattleofSaratoga’onthewebsite

AmericanRevolution.org,”insteadof“Findthewebpage‘TheBattleofSaratoga’onthewebsite

AmericanRevolution.”

Theparticipantsalsofeltthatfindingfourwebsiteswasveryredundant.Furthermoremany

studentscouldnotcompletethetaskinfortyminutes.Itwasdecidedtoreducethenumberof

searchtasksfromfindingfourwebsitestofindingthreewebsites.Thisleadtotherevisionofall

fourversionsoftheinstrumentfroma16‐itemassessment(fourlocate,fourevaluate,four

synthesize,andfourcommunicatescorepoints)toa12‐itemassessment(threelocate,three

evaluate,threesynthesize,andthreecommunicatescorepoints).

87

Thefinalrevisionaroundthelocatingtasksinvolvedaddingahelpfeature.Whenstudents

couldnotlocateawebsitetheywereunabletoproceedtoadditionalitems.Inotherwordsthey

couldnotsummarizeawebsitetheycouldnotfind.Thiscouldleadtodependencyissuesasalow

scoreonasynthesispointmightreflectaninabilitytolocateapage.Thus,ahelpfeaturewasadded

toeachlocatingtask.Students,aftersearchingforfiveminutes,couldrespondthattheywere

unabletolocateawebsite.TheycouldsubmitIDK(Idon’tknow)asananswer.Thentheywere

asked,“DidyoutypeIDK:YesorNo?”Ifstudentsindicated,“Yes”theywouldthenbegiventhelink

toanappropriatewebsite.

Thestudentsallfeltthatthesynthesistaskswerealsoredundant.Studentswereaskedto

combineinformationfromthefirsttwowebsitestheyfound,thesecondtwowebsitestheyfound,

andthentakeaposition.Thestudentsfeltthatthefinalsynthesisstatementtheywrotemirrored

thecommunicationtaskthataskedthemtoexplainwhattheylearned.Inordertoreducethe

feelingofredundancyboththesynthesistasksandthecommunicationtaskswererevised.Onthe

synthesistask,studentsnowhadtosummarizethekeyideasonthefirstwebsitetheyfound.Onthe

secondsynthesistaskstudentshadtocombinethemainideasfromthesecondtwowebsitesthey

found.Onthethirdsynthesistaskstudentshadtotakeapositionandprovideevidenceusing

informationthattheyread.

Thecommunicationtaskswerealsorevised.Thestudentsfeltbeingaskedtotakeaposition

onthecommunicationtaskwastoosimilartotakingapositiononthesynthesistask.Thereforethe

assessmentwasrevisedsostudentsnolongerhadtodoaninitialpostonthediscussionboard,

statingtheiropinionandthenreplyingtoanotherpostonthediscussionboard.Insteadthe

studentsnowhadtoagreeordisagreetooneofthetwostudentresponses,usingevidencefromthe

websitestheyread,explainingwheretheyfoundtheinformation.

ThefinalversionsoftheassessmentsareavailableinAppendixA.TheORCAlessrestricted

andtheORCAmorerestrictedeachconsistedoftwotasks.Therewere12possiblescorepointsin

88

eachtask.Thustherewere24possiblescorepointsfortheORCAlessrestricted.Therewerealso24

possiblescorepointsfortheORCAmorerestricted.

Thedispositionsofonlinereadingcomprehensionmeasure.TheDispositionsofOnline

ReadingComprehensioninstrument(DORC)(O'Byrne&McVerry,2008)wasusedtoestimatethe

dispositionsofonlinereadingcomprehensionamongparticipants.Inthisinstrument,dispositions

ofonlinereadingcomprehensionweremeasuredusingaseriesoffive‐pointLikertscalesinaself‐

reportsurvey.Studentsansweredquestionsthatfellonfivescales:reflectivethinking,critical

stance,flexibility,collaboration,andpersistence.Thesefivescaleswerecombinedtoforma

compositescoreofastudent’sdispositiontowardsonlinereadingcomprehension.Previouswork

establishedareliabilityestimateof.72andcontentvaliditywasensuredthroughthreeroundsof

review,followedbyrevisions,usingexpertpanels(O'Byrne&McVerry,2008).Thefullinstrument

isavailableinAppendixD.

QuantitativeProcedures

AdministeringtheIndependentVariableMeasures

Participantsfirstcompletedthebackgroundknowledgemeasure,thedispositionsofonline

readingcomprehensionmeasure,andtheCOIL,whichmeasuredcriticalevaluationskills.Students

weregivenaccesstoalaptopcomputerduringtheirnormallyscheduledclassperiod.Students

wereassignedacomputerandtheirnumberwasrecorded.Screenrecordingsoftware,IShowU,also

capturedallstudentactivitytoprotectagainstdataloss.

Duringthefirstsessionstudentscompletedalloftheindependentmeasuresbeforethe

onlinereadingcomprehensionassessments.Iwouldvisiteachclassandcompletethebackground

knowledgemeasure,thedispositionsofonlinereadingcomprehension,andtheCOIL.The

backgroundknowledgemeasurewasadministeredfirsttoensurethatnolearninggainsfromthe

inquiryactivitiesledtohigherbackgroundknowledgescores.Studentsthencompletedthe

89

dispositionsofonlinereadingcomprehension.Finallystudentsfinishedthefirsttwelveitemsofthe

COIL.

Followingthisfirstsession,theCOILwasanalyzedtocheckforinternalconsistencyasa

measureofadequatereliability.Asoutlinedabove,theinitial,12‐itemCOILdidnothaveadequate

reliability.OnceitwasevidentthatmoreitemswouldbenecessaryIreturnedtotheschoolto

administeranadditional15items.

Theteachersunderwenttrainingsothattheymightadministerthemeasureofverbal

comprehension.Theycompletedtheassessmentwiththeresearcherandwentovertheprocedures.

Theteachersthengavetheassessmenttotheentireclassina10‐minutesession.Ineachclassroom

thecompletionoftheverbalabilitymeasurewasdoneafterallotherassessmentswere

administered.

AdministeringtheOnlineReadingComprehensionAssessments

AllfourORCAtaskswereadministeredtotheparticipants(n=131)followingaprotocol(see

AppendixC).Theorderofthefourtaskswasrandomlyassignedtoparticipantstoprotectagainsta

learningeffect.Thetaskswereassignedtoensurestudentscompletedamorerestrictedorless

restrictedtaskfollowedbyamorerestrictedtaskorlessrestrictedtask,respectively.Therewere

12differentcombinations.Thecombinationswererandomlyassigned.

ForexampleastudentmightbeassignedtheDelawaretask(lessrestricted)followedbythe

Causestask(morerestricted).TheywouldthencompletetheWomentask(lessrestricted)and

finishwiththeTurningPointtask(morerestricted).TheirscoreontheoverallORCAlessrestricted

wouldbeasumofthescorepointsonboththeDelawareandWomentask.TheirscoreontheORCA

morerestrictedwouldbethesumofthescorepointsonboththeTurningPointandCausestask.

Studentscompletedtheassessmentinfour40‐minuteclassperiods.Theycompletedone

taskperclass.Ateachschoolsitestudentscompletedtwoofthetaskswithinthesamefive‐day

academicweek.Thentoreducethepossibilityofalearningeffect,thestudentsdidnotcompletethe

90

remainingtwotasksforthreeweeks.Theresearcherreturnedtotheschoolsandadministeredthe

finaltwotaskswithinone,five‐daycalendarweek.

Theresearcherbrought25laptopstotheclassrooms.Thecomputerswerepre‐distributed

onstudentdesks.Theresearcherpreparedanentranceslipforeachstudent.Ontheslipwasthe

URLfortheirassessment.Theentranceslipalsohadausernameandpassword.Eachstudentwas

assignedauniqueusernameandpasswordforuseonthediscussionboard.Astheinstructorand

researcherpassedouttheactivitycardstheywouldhitrecordonIShowU,ascreenrecording

software.

Onceeverystudentwasgiventheirmachineandactivitycardtheywereaskedtoenter

theirURLandnavigatetotheirassignedtask.Theresearcherthenreadthroughthetwodirection

screenswiththestudents.(SeeAppendixB).Theresearcherthenwalkedaroundtheroomwhile

studentsbegan.

Theprotocolprovidedstudentswithfiveminutesforeachsearchtask.Theresearcher

wouldremindstudentsatthefiveminutemarktotype“IDK,”ifneeded,andthenclickonthelinkto

gotothecorrectwebsite.Whenstudentsfinishedatasktheywouldraisetheirhandandthe

researcherwouldstopthescreen‐recordingsoftware.

ORCAReliabilityAnalysis.AfterthecompletionofthefirsttwoORCAinquirytasks,one

lessrestrictedandonemorerestricted,theresearcherscoredthetasksusingarubric.(See

AppendixB).Toensurereliability,20%(n=40)ofthetaskswererandomlychosenandgivento

anotherdoctoralstudentstudyingonlinereadingcomprehension.Afterscoringfivetogetherthe

ratersseparatedandscoredtheremaining35tasksalone.Agreementwascalculatedusingasimple

percentageanditexceeded93%.Differenceswereresolvedthroughdiscussion.

TheresearcherthenscoredtheORCAtasks.Testsforinternalconsistencywereconducted

usingacoefficientalpha(Cronbach,1972).Reliabilitytestingwasdoneattwolevels..Asstated,the

ORCAlessrestricted(ORCA‐LR)consistedoftheDelawareandWomentasks.TheORCAmore

91

restricted(ORCA‐MR)consistedoftheTurningPointandCausestasks.Firsteachindividualtask

consistingoftwelvescorepointswasevaluated.ThentheORCAlessrestricted,whichconsistedof

theDelawareandWomentasksforacombined24scorepoints,wasevaluated.ThentheORCAmore

restricted,whichconsistedofTurningPointandCausestasksforacombined24scorepointswas

evaluated.Table3.12presentsthereliabilityestimatesfortheORCA‐LR.Table3.13presentsthe

reliabilityestimatesfortheORCA‐MR.

Table3.12InternalConsistencyoftheORCA‐LRORCAInquiryTask Internal

ConsistencyDelaware(lessrestrictedtaskA) .615Causes(lessrestrictedtaskB) .713ORCA Less Restricted Total .722

Table3.13InternalConsistencyoftheORCA‐LRandORCA‐MRORCA Inquiry Task Internal

ConsistencyTurning Point (more restricted task A) .763Causes (more restricted task B) .769ORCA More Restricted Total .804

QuantitativeAnalysis

Thegoalofthequantitativeportionofthisstudywastoexaminetheuniquecontributions

thatbackgroundknowledge,criticalevaluationskills,anddispositionsmadetomodelsofonline

readingcomprehensionoverandaboveverbalability.Thisstudyexaminedtwomodels.Thefirst

involvedalessrestrictedtask(ORCA‐LR).Thesecondinvolvedamorerestrictedtask(ORCA‐MR).

Thequantitativeportionofthestudyexploredtworesearchquestions:



92



comprehension?





comprehension.

Toanswerthetwoquantitativequestions,sequential(hierarchical)regressionanalysis

(Tabachink&Field,2001)wasusedtoestimatethebestfitmodelsbetweentheindependent

variableofonlinereadingcomprehensionscoresandthedependentvariables,aftercontrolling

forverbalintelligence.Hierarchicalregressionanalysisallowstheenteringofvariablesinone

model,andthenadditionalvariablesareaddedinthesecondmodel.Giventhatthegoalofthe

quantitativeportionofthisstudywastoexaminemodelfit,thevariableswereaddedtothe

modelusingthe“enter”method.The“enter”methodentersallofthevariablesthatwereadded

tothemodelatthesametime.Thiswaschosenoverastepwiseregressioninordertoensure

alloftheindependentvariableswereincludedinthefinalmodel.Inthestepwisemethod,

variablesthatdonotmakeasignificantcontributiontothemodel,afteradditionalvariablesare

added,arethenremoved.

Beforetheregressionmodelswerecalculatedallofthedependentandindependent

variableswereexaminedtoensuretheymettheassumptionsnecessaryforaregressionanalysis.

Firsttheskewnessandkurtosisofthevariableswereexaminedtoensuretheassumptionofnormal

distributionwasmet.Thenthedatawasexaminedforoutliers.Testswerealsoruntoensurethe

assumptionsofhomoscedasticityandmulticollinearityweremet.Thesetestsarereportedin

chapterfour.

93

QualitativeProcedures

QualitativeParticipants

Thepurposeofthequalitativephaseofthismixedmethodsstudieswastoexplorethe

strategyusebystudentswhovaryintheironlinereadingcomprehensionability.Twelvestudents,

fourfromeachparticipatingschool,wereselectedforthequalitativeportionofthestudy.

Performanceonthefirstadministrationoftheonlinereadingcomprehensionassessment,including

oneORCA‐LRandoneORCA‐MRtask,wasusedtodetermineparticipantsinthethinkaloud

activity.Theywereselectedasfollows.

1. First all students in their class were ranked based on scores on the first administration of

the ORCA. Students were divided into two groups: the top 10% of scores and the bottom

10% of scores.



3. Then four students from each participating school were randomly selected from the list:

two students who scored in the 10% of their class and two students who scored in the

bottom 10% of their class.


achievement‐levelgroups(high,average,low)basedonORCAscores,independentofschool.They

wererankedusingacombinedscoreofonelessrestrictedtaskandonemorerestrictedtask.The

studentswererankedfrom1(lowest)to12(highest)andbrokenintothreegroupsoffour.Thislist

resultedinamedianscoreof10.5,outof24.Themiddlegroupincludedfourstudentswithintwo

scoresofthemedian.Thelowgroupincludedscoresmorethantwoscoresbelowthemedian.The

highgroupincludedfourstudentswithscoresmorethantwoscoresawayfromthemedian.

Tenstudentswereincludedinthefinalanalysissincetwostudentshadtoberemovedfrom

thestudyduetoerrorsindatacollection.Thisincludedonestudentinthehighgroupandone

94

studentinthemiddlegroup.Table3.14presentsthelistofstudents(usingpseudonyms)andtheir

performancelevelsonthevariablesofinterestbygroup.

Table3.14QualitativeParticipantsPseudonym Verbal PK ORCA1 ORCA2 Total SchoolHigh Isabella 13 1.5 8 9 17 3Olivia 17 2 9 10 19 3Ava 23 2 9 9 18 3 Medium Alexander 8 2 7 8 15 3Sophia 5 0.5 4 7 13 1Jacob 8 2 7 8 15 2Ethan 7 1.5 2 7 9 2 Low Emma 2 0 1 1 2 2 Michael 3 5 5 3 8 1Jaydan 7 .5 4 1 5 2*=ORCAScoreconsistedofonelessrestrictedandonemorerestrictedtask.QualitativeProcedures

Afterparticipantsforthequalitativethinkaloudwereselectedfromeachresearchsite,the

researchermetwithstudentsone‐on‐oneandtheycompletedasingleonlineinquirytask,using

structuredthinkaloudprocedures,overtwoconsecutivedays.First,thestudentsunderwenta

think‐aloudtrainingsession.Theresearchermodeledtoshowstudentshowtothinkaloudwhile

findingthecapitalofFrance.ThenthestudentswereaskedtopracticebyfindingthecapitalofNew

YorkState.

Studentsthencompletedastructurallypromptedthinkaloudactivitywithinanonline

inquirytask,beingpromptedatfixedstructurallocationstothinkaloudbytheresearcher

(Afflerbach,2002)whilereadingonline.Thestudentswereasked,“Whatwereyouthinking?”atkey

decisionpointssuchasclickingonalink,leavingawebsite,orenteringkeywords.Theiractions

wererecordedusingIshowU.

95

QualitativeAnalysis

Thereweretwogoalsforthequalitativeportionofthisstudy.Thefirstgoalwastocompare

strategydifferencesbetweenthelessrestrictedandmorerestrictedconditions.Thesecondgoal

wastoexaminestrategydifferencesbetweengoodandpooronlinereadersastheycompleted

onlinereadingcomprehensiontasks.Thus,atwo‐stageapproach,utilizingthemethods

recommendedbyAfflerbach(2002)andLeuetal.(2009)wereusedtoanswerthethirdandfourth

researchquestions:







Thefirststageofanalysisexamineddifferencesinstrategyuseamongparticipantsbasedon

therestrictednatureoftheinformationspaceandtheironlinereadingcomprehensionability.The

secondstageofanalysisinvolvedamorein‐depthanalysisofpatternsfoundinstageone.Thegoals

ofthissecondstageofanalysisweretwofold:1)toprovideillustrativeexamplesthatmirroredthe

quantitativefindings;2)todeterminetheunderlyingpatternsamongstrategyuseofparticipants

withvaryingdegreesofonlinereadingcomprehensionability.

Stageonedatawereanalyzedfirstthroughthelensofthecomponentsofonlinereading

comprehension:locating,evaluating,synthesizing,andcommunicating.Thisstagebeganby

examiningfrequencytablesofrawcodesforpatterns.Thensuccessivepassesweremadethrough

thedatatoallowfordatareductionandthemestoemerge.Thepurposeofthisstagewastoidentify

patternsinstrategyusethatcouldbeexploredduringthemorein‐depthstage‐twoanalysis.

96

Thesecondstageofanalysisbuiltupontheresultsofthefirststage.Thegoalofthemore

finiteanalysiswastoidentifythemesthatcutacrossthecomponentsofonlinereading

comprehension.Thisstageinvolvedmakingadditionalpassesthroughthedatatocompare

instancesofcodingacrossparticipants.Thepatternsandthemesfromstageonewerethenrefined

andreorganizedataintonewpatterns.Patternidentificationinvolvedaniterativeprocessof

reorganizingthedatafromstageoneandreworkinggroupingssothatthecategorystructures

definedthemesinthedata.

Theanalysisofthethinkalouddatainbothstageswascompletedfollowingbothabductive

(Onwuegbuzie&Leech,2006)methodsandconstant‐comparative(Bogdan&Biklen,2003;

Merriam,1988)methods.Thisdualapproachwasdesignedtoidentifypatternsincognitive

processingwhilealsoallowingforthecodebooktounfoldasthedatawereexamined.

Abductivecoding.Abductivecodingmethods(Onwuegbuzie&Leech,2006)employboth

inductiveanddeductivecodingprocedures.Initialcodingschemeswereinformedbyprevious

work(Leuetal.,2004;Leuetal.,2009).Usingthiswork,alistofexpectedstrategieswascreatedfor

eachofthe12scorepointsineachtask.Theinitialcodebookofexpectedstrategiesappearsin

Table3.15.

ThequalitativedatawascodedusingHyperRESEARCH,asoftwaretoolthatallowsforthe

codingofvideodata.Eachvideocasewasloadedintotheprogram.Thevideoswerethenbroken

intoseparateframes.AseriesofframeswascodedusingthecodebookaspresentedinFigure3.5.

Innumerousinstances,researcherannotationswerealsoaddedtoeachinstanceofthecode.

Table3.15

Initialcodebookforabductiveanalysis

Category CodeLocate K-Locate-Keyword Entry K1-Copy and paste exact words from task

K2-Type exact words from taskK3-Use keywords from taskK4-Revise keywords

97

SR-Locate Read Search Results SR1-Cursor movement indicates reading of results.SR2-Clicks on first linkSR3-Clicks on a link on first pageSR4-Moves to second pageSR5-Returns to search results

LW-Locate Website LW1-Locates website and judges it relevantLW2-Locates website and judges it irrelevantLW3-Skims website

Evaluate CEE-Evaluate Author Expertise CEE1- Finds the author in the text

CEE2- Finds the author on an about us/bio pageCEE3- Infers the author from the text.CEE4- Does a secondary search for the author.CEE5-Does not locate the authorCEE6-Uses an authoritative title to judge the authorCEE7- Uses supporting details from content of the website.CEE8- Uses institutional information to judge author expertiseCEE9-Uses background knowledge to judge author expertise

CES-Evaluate Author1s use of Evidence CES1- Uses the source of claims/evidence to judge use of evidenceCES2-Uses a bibliography or reference to judge use of evidenceCES3-Uses a secondary source to verify information.CES4-Uses overall quantity of content to judge use of evidence

CEP-Evaluate Author Expertise

CEP1- Uses authors perceived level of expertise to describe point of view.CEP2- Uses authors prior experience to describe point of view.CEP3- Uses authors institutional affiliation to describe point of view.CEP4- Uses content of website to describe point of view.

98

SynthesizeSEM-Locate two important details SEM1-Copy and paste important details

SEM2-Copy and paste entire source.SEM3- Copy and paste irrelevant detailsSEM3-Paraphrase important details.SEM4-Paraphrase irrelevant details.

SEC-Combine information from two sources

SEC1- Navigate between multiple tabs/windows.SEC2-Cursor movement provides evidence of reading two sources.SEC3-Copy and paste details from two sourcesSEC4-Copy and paste details from one sourceSEC5-Paraphrase details from two sourcesSEC6-Paraphrase details from one source

SEP-Take a position with evidence SEP1-Student makes a specific claim related to task.SEP2-Student copies and past details in reference to claim.SEP3-Student paraphrases details in reference to claim.SEP4-Student copies and pastes information without making claim.

CommunicateCDB-Correctly use discussion board CDB1-Student can log in to discussion board

CDB2-Student navigates to correct discussion.CDB3- Student responds to another post.CDB4-Student replies to discussion.

CDB-Engage in dialogue CED-Student agreed or disagreed with an initial post.CED-Student did not disagree.

CDE-Provide source of evidence CDE1-Student links to a sourceCDE2-Student refers to a specific sourceCDE3-Student refers to source in general (i.e. websites I read)

99

Figure3.5ScreenShotfromHyperRESEARCH

Followingabductivemethods,additionalcodeswereaddedtothecodebookduring

analysis.Thesecodesdevelopedinductivelyasnewpatternsemergedthatwereeithernot

representedinacodeorcouldnotberepresentedexclusivelybyoneexistingcode.Forexample

manystudentscompletedasearchbyusingtheauto‐fillfeatureofGoogle,whichpredictsasearch

string.Thiscodewasaddedtothecodebook.

Constant‐ComparativeAnalysis.Analyticinductivemethods(Bogdan&Biklen,2003;

Robinson,1951)wereusedrecursivelyacrossthevideodatasources.Afterallofthecaseswere

coded,reportswererunasapreliminaryexaminationofpatternsthatdevelopedusingconstant

comparativeanalyticmethods.First,anoverallreportwasrunonallofthesubareas:locate,

evaluate,synthesize,andcommunicate.Thefrequenciesofthecodeswerematchedwithaspecific

timestampinthevideo.Thisdatawasusedforstageoneanalysis.Ithenwentbacktowatchthe

codedvideosegmentsofeachcodeinstance.Finally,Icomparedtheannotationsofeachcode

100

instance.Aspatternsacrossthefrequencies,videosegments,andannotationsemergedtheywere

noted.

Thenextstepinthestageoneanalysiswastocrosscheckindividualcasesbasedonthe

restrictednatureoftheinformationspaceandthesuccessofstudentsontheORCAinstruments.

Forexample,ifahighperformingstudentexhibitedatendencyto“navigatemultiplewindows”I

examinedifthiswassimilaracrossallparticipantsorjusthighperformingparticipants.Ifthecode

exhibitedstabilityateitherlevelitremainedinthecodebook.

ThetenparticipantsweresplitintothirdsbasedonORCAscores.Theyweredefinedas

highperformingonlinereaders,averageperformingonlinereaders,andlowperformingonline

readers.Areportwasrunforeachsubscale:locate,evaluate,synthesize,andcommunicateforeach

groupofparticipants.Initially,thefrequencyofcodesforeachgroupwascomparedtoother

groups.Then,usingtheinitial,overallpatternsfoundinthefirstpassofthedata,Itriedto

determineifthesamepatternsheldtruebasedonsuccessonthelessrestrictedandmore

restrictedtasks.

Thenextstepinthestageoneanalysiswastoidentifypatternsandthemesfromtheraw

codesandfrequencymatrices.Thefirstlevelofpatternidentificationinvolvedanexaminationof

thefrequencymatrices.Ifdifferencesemergedinthefrequencyofcodesbetweenhigher,average,

andlowerperformingstudentsIreturnedtothevideoforacloserexamination.Eachinstancesof

thecodewasreexaminedandadditionalannotationsmadefocusedonstudentstrategyuse.The

goalofstageoneanalysiswastoidentifypatternsinstrategyusethatledtooverallperformanceon

theORCAassessmentsregardlessoftherestrictednatureoftheinformationspace.Thethemesand

patternsfoundinthestageoneanalysiswerethenusedtoinformthestagetwoanalysis.

Thegoalofthestagetwoanalysiswastodeterminepatternsofstrategyuseamong

participantsbasedontheirabilityandtherestrictednatureoftheinformationspace..Inorderto

condensethepatternsandthemesfromstageoneintomoreexclusivethemesrecursive,analytic

101

inductivemethodswereused(Bogdan&Biklen,2003).Usingthetimestampsfromtheinitialcode

Ireturnedtothevideoandsystematicallyandconsistentlyexaminedthedatatoconfirm,

disconfirm,andgeneratenewpatternsandthemes.

102

ChapterIV

QuantitativeResults

Thisstudyexploredtherelationshipsofbackgroundknowledge,criticalevaluationskills,

anddispositionsofonlinereadingcomprehensioninpredictingonlinereadingcomprehension

whenverbalintelligencewascontrolled.Theserelationshipsweretestedintwodifferentreading

“spaces:”alessrestrictedenvironment,i.e.,wherereaderscouldchoosetheirsources;andamore

restrictedenvironment,i.e.,wherereadershadtolocatespecificwebsitesthatwereprovided.

Thequantitativeportionofthisstudysoughttoanswertwoquestions:





comprehension?





comprehension.

Sequential(hierarchical)regression(Tabachink&Fidell,2001)wasusedtoallowforthe

researchertoenterverbalintelligencetothemodelapriori.Ameasureofverbalintelligencewas

usedasitispresumedtobecausallyrelatedtoreadingcomprehension.Artley(1944)notedthat

verbalintelligenceisakeycomponenttocomprehension.Cain,Oakhill,andBryant(2004)found

thatmeasuresofverbalabilityexplainedthegreatestamountofvarianceinmeasuresofreading

comprehension.Finally,Baddeley,Logie,andNimo‐Smith(1985)alsofoundverbalabilitytobea

strongpredictorofreadingcomprehension.

103

Sincethisstudywantedtoexaminehowbackgroundknowledge,criticalevaluationskills,

anddispositionscontributedtomodelsofonlinereadingcomprehensionitwasdecidedtocontrol

forverbalability.Theremainingindependentvariables:backgroundknowledge,criticalevaluation

skills,anddispositionswereenteredinthesecondsteptotestthebestfitoverandaboveverbal

ability.

Thischapterisorganizedinfoursections.Thefirstsectionreviewsthedatascreening

proceduresforallofthevariables.Thesecondsectionprovidesdescriptivestatisticsfor

independentanddependentvariablesafterthedatascreening.Thethirdsectiondescribesthe

resultsoftheregressionmodelforthelessrestrictedinformationspace.Thefinalsectionprovides

theresultsoftheregressionmodelforthemorerestrictedinformationspace.

DataScreening

Scoresforthepredictorvariableswerecollectedfor131students.However,dueto

absencesandlossofdata33studentswereremovedwithlist‐wisedeletion.Thisleftasampleof98

participants.Thisexceedstheminimumsamplesizeof77forahierarchicalregressionwithan

effectsizeof0.15,adesiredpowerof.80,andoneindependentvariableinthefirststepandthree

independentvariablesinthesecondstep(Sloper,2011).ThisanticipatedCohen’sf2of0.15isa

mediumeffect.Theestimateofpowersizeof.80waschosentoensureanadequateeffect(Sloper,

2010).

Theestimatedeffectsizewaschosenbasedonthereviewoftheresearch.Iexamined

previousstudiesthatusedsimilarregressiontechniques.Iftheresearcherdidnotpresentaneffect

sizeIcalculatedtheeffectsizefromthegivendata.Icalculatedaneffectsize(Cohen’sf2)of.0740

fromthedatapresentedbyBråten,Strømsø,&Britt(2009).Thisstudyusedhierarchicalregression

toexploresourceevaluation.Theyusedintertextual(multipletext)comprehensionasdependent

variableandbackgroundknowledgeasanindependentvariable.Inthesamestudy,Bråten,

104

Strømsø,&Britt(2009)addedtrustindocumenttypetotheirintertextualcomprehensionmodel.

Thismodelhadaneffectsize(Cohen’sf2)of.025316.

IalsousedthedatapresentedbyCoiro(2011).Thisstudyexaminedtheunique

contributionanonlinereadingcomprehensionmeasurehadinpredictingperformanceonan

additionalmeasureofonlinecomprehensionaftercontrollingforbackgroundknowledgeand

offlinereadingability.Whenofflinereadingabilitywasaddedtotheregressionmodelwith

backgroundknowledgeandthescoresonthefirstmeasureofonlinereadingcomprehensionthere

wasaneffectsize(Cohen’sf2)of0.1286whenofflinereadingabilitywasaddedtoaregression

modelwithonlinereadingabilityandbackgroundknowledge.Whenanadditionalmeasureof

onlinereadingcomprehensionwasaddedtothemodeltherewasaneffectsizeof.366.Therefore

myestimatedeffectsizeofCohen’sF2=0.15isaconservativeestimate.

Missingdata.Eachcasewasexaminedforcompletenessofdata.Tobeincludedinthefinal

samplestudentsneededtohavecompletedeightdifferentmeasuresonsevendifferentschooldays.

Themajorityofdatalossinthesamplewasduetostudentabsencesononeofthesevendifferent

schooldays.(SeeTable4.1forasummaryofmissingdata.)Asecondarysourceoflostdatawas

softwaremalfunctionorusererrorthatdidnotresultinscoresbeingrecorded.

Table4.1SummaryofMissingData Measure N Missing School 1 School 2 School 3ORCA 19 9 5 5Background knowledge

9 2 3 4

COIL 13 3 6 8Dispositions 10 3 4 3Total N missing 33 14 8 11

Descriptivestatistics.Means,rangesandstandarddeviationsforthemeasurespriorto

datascreeningarereportedinTable4.2.Afterexaminingthedescriptivestatistics,testswererun

toensurethattheassumptionsforamultipleregressionanalysisweremet.Examinationof

105

skewnessandkurtosisscoresandoutlierswereusedtoassesstheassumptionofnormality,

linearity,andhomoscedascity.Durbin‐Watsonscoreswereusedtotestindependenceofvariables.

Finally,collinearitydiagnosticswereusedtoassessmulticollinearityissues.

Table4.2Range,Means,andStandardDeviationsforDependentandIndependentVariables N Minimum Maximum Mean SDVerbal Intelligence 98 0 24 10.906 5.197Background knowledge 98 0 11 1.67 2.13COIL 98 1.0 11 5.187 2.094Dispositions 98 1.95 5.00 3.419 .4889Less Restricted 98 0 18 8.394 4.117More Restricted 98 0 20 7.15 4.48 Normality.Toassessthenormalityoftheunivariatedistributions,skewnessandkurtosis

statisticswerecalculatedforeachofthepredictoranddependentvariables.Theseresultsare

presentedinTable4.3.Therewasleptokurticdistributionindicatingahighprobabilityforextreme

valuesinthebackgroundknowledgemeasure(3.569).Thelargenumberofstudents(n=58)who

scoredlessthanoneontheirrecalloffactsfromtheAmericanRevolutionmayexplaintheextreme

violationofnormality.

Table4.3SkewnessandKurtosisRatiosforDependentandIndependentVariablesVariable Skewness Ratio Kurtosis RatioVerbal Intelligence 1.37 -.467Background knowledge 1.74 3.59COIL .138 -.559Dispositions .50 1.794Less Restricted ORCA -.578 .136More Restricted ORCA -.438 .576

Inordertomeettheassumptionofnormalityofunivariatedistributionsthebackground

knowledgemeasurewastransformedusinglogarithmictransformation(Osborne,2010).This

resultedinanormaldistributionasindicatedinTable4.4.This,however,didreducemuchofthe

varianceinscores

106

Table4.4TransformedBackgroundknowledgeIndependentVariable Minimum Maximum Mean SD Skewness KurtosisBackground knowledge

1.00 1.32 1.063 .0725 1.269 .425

Linearity,homoscedascity,andindependence.Todetermineiftheassumptionsof

linearityandhomoscedascityweremettheresidualplotswereexamined(seeFigure4.1).The

Durbin‐Watsonstatisticwasusedtotesttheassumptionofindependenceofvariables.TheDurbin‐

Watsonstatisticforboththemorerestricted(1.689)andthelessrestricted(1.706)satisfiedthe

assumptionofindependenceofvariables.

Figure4.1.Scatterplotsandhistogramsofresidualplots.

Outliers.Outliers,whicharepoorfitsfortheregressionmodel,canlowermultiple

correlations.Totestforoutliers,Cook’sdistancemeasurewasused(Tabachink&Fidell,2001).

107

FirstCook’sdistancewascalculatedforthelessrestrictedregressionmodel.Thesescoreswere

thenanalyzedinaboxplotgraphFigure4.2presentstheboxplotofCook’sdistance.

Figure4.2.Boxplotofcook’sdistance.

Anexaminationoftheboxplotrevealedsevenoutliers.Theblacklinerepresentsthe

mediandistance.Theboxrepresentsthefirstandthirdquartilesdistances.Tobeconsideredan

outlierthescoreshadtobeatadistancegreaterfromthemedianscoremorethan1.5timesfrom

theinterquartilerange.Thesesevenscoresareplottedindividuallyascircles.Bothregression

models,lessrestrictedandmorerestrictedwerethentestedwithandwithoutthesevenoutliers.

Theremovaloftheoutliersaffectedthevariablesofinterest.

Nexttwodifferentmodelswererun.Thefirstmodelremovedallsevenoutliers.Thesecond

modelremovedthethreemostextreme.Themodelswiththethreeextremeoutliersremovedwere

thesamewithallsevenoutliersremoved.Thereforeitwasdecidedtorunthemodelswithoutthe

threemostextremeoutliers.Thisleftafinalsampleof95participants.

Multicollinearity.Collinearitydiagnosticsofthebivariatecorrelationsandrelated

statisticsindicatedsomemulticollinearityconcerns.Notolerancevaluewaslessthan0.2andno

varianceinflationfactorsweregreaterthan4.However,inthemorerestrictedmodel,the

108

eigenvalueforthefifthfactor(.002)andtheconditioningindicesexceeded30(47.34).Intheless

restrictedmodeltheeigenvaluealsoapproachedzero(.002)andtheconditioningindicesexceeded

30(47.957).Thisincreaseinmulticollinearitycouldleadtoinflatedstandarderrorsforthe

coefficients.Totesttheeffectofcollinearitythemodelswererunwithandwithoutthedispositions

variables.WithoutdispositionstheR2=.447,andwithdispositionsincludedinthemodeltheR2=

.431.Sincethisdifferencewasnegligibleitwasdecidedtokeepallindependentvariablesinthe

model.

DescriptiveStatistics

Onlinereadingcomprehensionassessment.BasedonthemeansoftheORCA

assessments,themorerestrictedenvironmentwasamoredifficultinformationspaceforstudents

touse.Anexaminationofdescriptivestatisticsrevealedadifferenceinoverallperformancebased

onSESoftheschool.Oneachitemtype(Locate,Evaluate,Synthesize,andCommunicate)students

couldearnamaximumofsixpoints.Anexaminationofmeanscoresfortheitems(seeTable4.5)

indicatedthatinbothtypesofinformationspacescriticalevaluationofwebsiteshadthelowest

meanscore.

Table4.5MeanscoresofL,E,S,Cperformance n Less Restricted More Restricted M SD M SDLocate 95 3.509 1.483 2.775 1.508Evaluate 95 .9554 1.101 .8725 .9917Synthesize 95 2.732 1.483 2.683 1.622Communicate 95 1.554 1.328 1.206 1.269Total 95 8.394 4.117 7.15 4.48

Apairedsamplet‐testwasconductedtocomparethemeanscoresonoftheoftheORCA‐LR

andtheORCA‐MR.Thetestwassignificant(t=4.088,p=.000)withameandifference1.406,SD.33).

ThiswouldindicatethattherewasasignificantdifferenceinstudentscoresontheORCA‐LRand

109

theORCA‐MR.Anexaminationofthehistogramsindicatesthatstudentsdidnotscoreashighonthe

ORCA‐MR.SeeFigure4.1..

Aonewayrepeatedmeasuresanalysisofvariancewasconductedtocomparestudents’

scoresineachofthefourskillareasontheORCA‐LR.Themultivariatestatisticsrevealeda

significanteffectforLESC(Locate,Evaluate,Synthesize,Evaluate)skillareas,Wilks’Lambda=.133,

F(1,93)=408.72p<.005.,multivariatepartialetasquared=.818.Ananalysisofpairwise

comparisonsusingaBonferronicorrectiontocontrolforTypeIerrorfoundasignificantdifference

betweeneachofthefourskillareasandeachoftheothersskills(p<.05forallpairwise

comparisons).Itemsrequiringstudentstolocateinformationweretheeasiest(M=3.509SD=

1.403),followedbysynthesisitems(M=2.732,SD=1.483),communicateitems(M=1.554,SD

=1.328),andfinallyevaluateitems(MD=.9544,SD=1.101),whichwerethehardest.

Aonewayrepeatedmeasuresanalysisofvariancewasconductedtocomparestudents’

scoresineachofthefourskillareasontheORCA‐MR.Themultivariatestatisticsrevealeda

significanteffectforLESCskillareas,Wilks’Lambda=.214,F(1,93)=818.68p<.005.,multivariate

partialetasquared=.214.AnanalysisofpairwisecomparisonsusingaBonferronicorrectionto

controlforTypeIerrorfoundasignificantdifferencebetweeneachofthefourskillsareasandeach

oftheothersskills(p<.05forallpairwisecomparisons).Itemsrequiringstudentstolocate

informationweretheeasiest(M=2.755SD=1.508),followedbysynthesisitems(M=2.683,

SD=1.633),followedbycommunicateitems(M=1.206,SD=1.269),andfinallyevaluateitems

(MD=.8725,SD=.9917),whichwerethehardest.

Thebivariatecorrelationforthetwotestformatswas.658,whichwassignificant,p=.00.

ThebivariatecorrelationspresentedinTable4.6demonstratedthatthemajorityoftheitemson

thelessrestrictedwererelated.Thecommunicationitems,however,didnotsignificantlycorrelate

withtheotheritems.Theitemsonmorerestrictedwerealsorelated.Onceagainthe

communicationitemsdidnotcorrelatewiththeotheritemsexceptfortheevaluationitems.Across

110

theformats(ORCA‐MRandORCA‐LR)alloftheitemscorrelatedsignificantlywiththeir

counterparts.

Table4.6BivariateCorrelationsbyItemType LR-

EvaluateLR-Synthe-size

LR-Comm-unicate

MR-Locate

MR-Evaluate

MR-Synthesize

MR Comm-unicate

LR-Locate .295* .552** -.107 .356** .301** .370** .180LR-Evaluate .556** -.135 .346** .620** .442** .282**LR-Synthesize .064 .441** .565** .607** .262*LR-Communicate .042 .003 -.018 .214*MR-Locate .463** .618** .110MR-Evaluate .618** .246*MR-Synthesize .198

VerbalIntelligence.TheverbalcomprehensionvocabularytestoftheKitofFactor

ReferencedCognitiveTest(Ekstrom,French,Harman&Derman,1976)wasusedasameasureof

verbalintelligence.Themeanforthetestwas10.726withastandarddeviationof5.374.

Backgroundknowledge.Scoresonthebackgroundknowledgemeasureaboutthe

AmericanRevolutionwerelow.Themeanscorewas1.758withastandarddeviationof2.20.Forty‐

threeofthe95finalparticipantsincludedinthesurveycouldnotrecallatleast1relevantideaunit

abouttheAmericanRevolution.Thefrequencyofbackgroundknowledgescoresarepresentedin

Table4.7.

111

Table4.7FrequencyofBackgroundknowledgeScoresScore Frequency0 340.5 81 101.5 52 122.5 13 63.5 44+ 15

CriticalEvaluation.Afourteenitemmultiple‐choiceassessmentwasusedtomeasure

students’abilitytoevaluatewebsites.Themeanscorewas5.1053withastandarddeviationof

2.013.PerformanceontheCOILvariedbyconstructanditemtype.Themeanscoresofeachitemon

theCOILarepresentedinTable4.8.

112

Table4.8DescriptiveStatisticsforCOILItemsItems In Final Version Mean SD1. Evaluate author: Where would you click to learn more about an author?*

.84 .37

2. Evaluate author: Which websites about Chihuahuas and asthma was created by the most knowledgeable author?***

.38 .49

4. Evaluate publisher: Where do you click to learn more about a publisher?*

.62 .49

5. Evaluate bias: Think about the author’s point of view. What may influence the way he thinks about energy drinks?*

.60 .49

10. Evaluate publisher: Which publisher creates a website with the most credible medical information?**

.32 .47

14. Evaluate bias: Think about the authors' point of view. How does the authors' point of view influence the words and images used on the website?***

.48 .50

15. Evaluate author: Look at this website. What is the author's expertise?*

.71 .45


.52 .50

18: Evaluate bias: Which website uses strong words or images to persuade the audience?**

.28 .45

19: Evaluate author: Which author is the best expert on volcanoes?** .44 .5023: Evaluate Source: Where would you click to learn more about the sources an author used?*

.60 .49

25: Evaluate Publisher: Which website about smoking hazards was created by the most reliable publisher?**

.30 .46

26: Evaluate Source: Which source used in the websites is the most reliable source to answer the question, "What killed the dinosaurs?"**

.33 .47

27: Evaluate Publisher: Who is the publisher of this website?* .47 .50Notes*‐onescreenshot,#‐clickononelink,**‐fourscreenshots,***‐clickonfourlinks

DispositionsofOnlineReadingComprehension.Dispositionsofonlinereading

comprehensionweremeasuredusingafive‐pointLikertscaleselfreportsurvey.Students

answeredquestionsthatfellonfivescales:reflectivethinking,criticalstance,flexibility,

collaboration,andpersistence.Thesefivescaleswerecombinedtoformacompositescoreofa

student’sdispositiontowardsonlinereadingcomprehension.MeanscoresontheDORCwere3.382

withastandarddeviationof.471.

113

ResultsForResearchQuestionOne:OnlineReadinginaLessRestrictedInformationSpace

Thefirstresearchquestionexploredthebest‐fitmodelforonlinereadingcomprehensionina

lessrestrictedinformationspace.Itasked:

Whenpredictingonlinereadingcomprehensionofaproblemsolvingtaskwithina less

restricted information space, what is the best fitmodel, after controlling for verbal


variables:backgroundknowledge,criticalevaluation,anddispositionsofonlinereading

comprehension?

Scoresforthefinalanalysisincluded95studentsafterthreeoutliersweredeleted.Means,

rangesandstandarddeviationsforthemeasuresarereportedinTable4.9.Bivariatecorrelations

forthedependentandindependentvariablesarelistedinTable4.10.InordertoanswerResearch

Question1ahierarchicalregressionwasused.TotalscoresonthelessrestrictedORCA(ORCA‐LR)

wereenteredasthedependentvariable.Verbalabilitywasenteredinthefirststep.Thenthe

independentvariables:backgroundknowledgescores,criticalevaluationscores,andthenewly

computeddispositionscoreswereaddedinasecondstep.

Table4.9N,MinimumandMaximumScores,Means,StandardDeviations(SD),forDependentandIndependentVariables N Minimum Maximum Mean SDVerbal Intelligence 95 0 24 10.726 5.374Background knowledge 95 0 11 1.758 2.20COIL 95 1.0 9 5.1053 2.013Dispositions 95 1.88 4.56 3.382 .47114ORCA-LR 95 0 20 6.8421 4.16ORCA-MR 95 0 18 8.3053 3.98

114

Table4.10BivariatecorrelationsofDependentandIndependentVariables ORCA-

MRVerbal Intelligence Background

knowledgeCOIL Dispositions

ORCA-LR .658** .422** 379** .462** .195ORCA-MR .414** .436** .574** .264*Verbal .329** .263** -.088Background knowledge

.314** .128

Critical Evaluation

.122

Note:**=Correlationsignificantatthe.01level,*=significantat.05level

Resultsofthelinearregression,aspresentedinTable4.11,indicated,firstthatverbal

intelligence,asmeasuredbyavocabularytest,explained16.9%ofthevariance,whichwas

significant,F(1,93)=20.530,p<.001.Afterverbalabilitywasaccountedfor,thethreepredictor

variablesofbackgroundknowledge,evaluationskills,andscoresonaself‐reportmeasureof

dispositionsofonlinereadingcomprehensionexplainedanadditional22.4%ofthevarianceonthe

scoresofthelessrestrictedORCA,whichwassignificantF(1,92)=15.415p<.001.Inthefullmodel,

scoresonacriticalevaluationmeasureexplainedasignificantamountofuniquevariance(β=.335

p<.000).Backgroundknowledgealsocontributedsignificantlytothepredictionofscoresonthe

lessrestrictedORCA(β=.211p=.021).Dispositionsofonlinereadingcomprehensiondidnot

contributesignificantlytopredictionsinthemodel(β=.117p=.162).Theadditionofthevariables

inthesecondstepofthelessrestrictedmodelhadalargeeffect,Cohen’sF=.358.

115

Table4.11ResultsofHierarchicalRegressionfortheLessRestrictedORCAModel β Sig ΔR2 ΔFModel 1 .205 21.030*Verbal Comprehension .429 .00 Model 2 .224 11.454*Verbal Comprehension .270 .003Critical Evaluation .335 .000Background knowledge .241 .007Disposition .124 .137Note:*=p<.01 Interactioneffectsrefertothecombinedeffectsofvariablesonthedependentvariableand

theyshouldbeconsideredinmultipleregressions(Pedhazur&Schmelkin,1991).Howeverwhen

testinginteractionsitisimportanttocentertheinteractionstoreducethechanceof

multicollinearity(Tabachink&Fidell,2001).Firstthefourindependentvariableswerecentered

(Aiken&West,1991;Judd&McClelland,1989)bysubtractingthemeanfromeachscore.This

resultedineachvariablehavingameanofzero(Aiken&West,1991).Thenaninteractiontermwas

computedforeachvariable.Finallyahierarchicalregressionwasrunwiththeeachcentered

variableinthefirstmodelandthecenteredvariableandinteractionterminthenextmodel.This

testediftheinteractionvariableshouldbeintroducedtothefullmodel(Aiken&West,1991).

Resultsoftheseanalysesfoundnosignificantinteractionbetweenverbalabilityandcritical

evaluationscorest(93)=1.298,p=.198;nosignificantinteractionbetweenverbalabilityand

backgroundknowledget(93)=‐.368,p=.713;nosignificantinteractionverbalabilityand

dispositionst(93)=‐.408,p=.684;nosignificantinteractionbetweencriticalevaluationscoresand

backgroundknowledget(93)=1.340,p=.184;nosignificantinteractionbetweendispositionsand

backgroundknowledget(93)=1.024,p=.308.

Summaryofresultstoresearchquestionone.Theresultsfromresearchquestionone

indicatethataftercontrollingforverbalintelligence,scoresonameasureofstudents’abilityto

criticallyevaluatewebsitesisthebestpredictorofperformanceonanonlinereading

116

comprehensionassessmentinalessrestrictedspace(β=.334,p=.000).Backgroundknowledge

wasalsoasignificantpredictorofperformanceonanonlinereadingcomprehensionassessmentin

alessrestrictedspace(β=.241,p=.007).Dispositionsdidnotmakeasignificantpredictionin

performanceonanonlinereadingcomprehensionassessmentinalessrestrictedspace(β=.124,

p=.137).

ResultsForResearchQuestionTwo:OnlineReadinginaMoreRestrictedInformationSpace

Thesecondresearchquestionexploredthebestfitmodelforonlinereadingcomprehensionin

amorerestrictedinformationspace.Itasked:

Whenpredictingonlinereadingcomprehensionofaproblemsolvingtaskwithinamore

restrictedinformationspace,whatisthebestfitmodel,aftercontrollingforverbal

intelligence,ofthepercentageofvarianceaccountedforbyeachofthefollowing

variables:backgroundknowledge,criticalevaluation,anddispositionsofonlinereading

comprehension.

ToanswerResearchQuestiontwo,anotherhierarchicalregressionwascalculated.

Verbalabilitywasaddedinthefirststep;followedbytheotherindependentvariablesinthe

secondstep.TotalscoresonthemorerestrictedORCA(ORCA‐MR)wereenteredasthe

dependentvariable.Verbalabilitywasenteredinthefirststep.Thentheindependent

variables:backgroundknowledgescores,criticalevaluationscores,andthenewlycomputed

dispositionscoreswereaddedinasecondstep.

Inthefirststep,verbalabilityexplained15.2%ofthevariance,whichwassignificant,F(1,94)

=17.024,p<.001.Afterverbalabilitywasaccountedfor,thethreepredictorvariablesof

backgroundknowledge,evaluationskills,andscoresonaself‐reportmeasureofdispositionsof

onlinereadingcomprehensionexplainedanadditional38.1%ofthevarianceonthescoresofthe

morerestrictedORCA,whichwassignificantF(1,92)=26.328p<.000.

117

Inthefullmodelscores,aspresentedinTable4.12,acriticalevaluationmeasureexplained

asignificantamountofuniquevariance(β=.499p<.000).Backgroundknowledgealsocontributed

significantlytothepredictionofscoresonthemorerestrictedORCA(β=.189p=.017).Dispositions

alsomadeasignificantpredictiontothemodel(β=.291p=.006).Thisexplanationhadalarge

effect,Cohen’sF=.631.

Table4.12ResultsofHierarchicalRegressionfortheMoreRestrictedORCAModel β Sig ΔR2 ΔFModel 1 .152 17.024*Verbal Intelligence .414 .00 Model 2 .381 25.008*Verbal Intelligence .219 .006Critical Evaluation .499 .000Background knowledge .189 .017Disposition .206 .006Note:*=p<.01 Interactioneffectswereonceagaintestedtoseeiftheyshouldbeaddedtothemodel.

Resultsoftheseanalysesfoundnosignificantinteractionbetweenverbalabilityandcritical

evaluationscorest(96)=.104,p=.917;nosignificantinteractionbetweenverbalabilityand

backgroundknowledget(104)=‐.132,p=.895;nosignificantinteractionverbalabilityand

dispositionst(96)=‐.869,p=.387;nosignificantinteractionbetweencriticalevaluationscoresand

backgroundknowledget(97)=.281,p=.780;nosignificantinteractionbetweendispositionsand

backgroundknowledget(93)=.936,p=.351.

Summaryofresultstoresearchquestiontwo.Onceagain,aftercontrollingforverbal

intelligencescores,ameasureofcriticalevaluationofwebsitesmadethelargestpredictionin

performanceonameasureofonlinereadingcomprehensioninamorerestrictedenvironment(β=

.499p<.000).Backgroundknowledgealsocontributedsignificantlytothepredictionofscoreson

themorerestrictedORCA(β=.189p=.017).Adifferencebetweenthelessrestrictedandthemore

118

restrictedenvironmentwerescoresontheself‐reportmeasureofdispositions.Dispositionsof

onlinereadingcomprehensionmadeasignificantpredictiontothemorerestrictedmodel(β=.291

p=.006).

Exploringbi‐variatecorrelations.Additionalanalyseswereconductedtofurtherexplorethe

relationshipsbetweenthedependentandindependentvariables.Backgroundknowledgewas

significantlycorrelatedwithbothversionsoftheORCA.ThePearsoncorrelationbetween

backgroundknowledgeandtheORCA‐LRwas.370p=.000.Thecorrelationbetweenbackground

knowledgeandtheORCA‐MRwas.433p=.000.Table4.13liststhePearsoncorrelationsforitem

typesoneachformat.Thebackgroundknowledgemeasurewassignificantlycorrelatedwithevery

itemtypeexceptforthecommunicationitems.

Table4.13Pearson’srcorrelationsbetweenVerbalIntelligenceandORCAitems Locate Evaluate Synthesize CommunicateORCA-LR .239* .372** .375** .028ORCA-MR .250* .431** .420** .157Note*=significantat.05level;**=significantat.01level

ScoresonthecriticalevaluationmeasureweresignificantlycorrelatedwiththeORCA.The

PearsoncorrelationfortheORCA‐LRwas.462,p=.000.ThecorrelationfortheORCA‐MRwas.574

p=.000.Table4.14liststhePearsoncorrelationsforitemtypesoneachformat.Thecritical

evaluationmeasurewassignificantlycorrelatedwitheveryitemtypeexceptforthecommunication

items.

119

Table4.14Pearson’srcorrelationsbetweenCOILandORCAitems Locate Evaluate Synthesize CommunicateORCA-LR .235* .414** .397** .065ORCA-MR .432* .534** .575** .109Note*=significantat.05level;**=significantat.01level

Self‐reportscoresontheDORCdidnothaveasstrongacorrelationwiththeORCAasthe

otherindependentvariables.TheDORCandtheORCA‐LRhadaweakandnon‐significant

correlationof.122p=.248.TheDORCandtheORCA‐MRhadaweakbutsignificantcorrelationof

.212p=.042.TheDORCdidnotsignificantlycorrelatewithanyitemtypeontheORCA‐LRbutdid

haveweak,butsignificantcorrelationswiththesynthesisandevaluationitemsontheORCA‐MR.

ThesecorrelationsaredisplayedinTable4.15.

Table4.15Pearson’srcorrelationsbetweenCOILandORCAitems Locate Evaluate Synthesize CommunicateORCA-LR .169 .090 .093 -.014ORCA-MR .213* .225* .131 .065Note*=significantat.05level;

ExaminingDispositionScores.GiventhelowcorrelationbetweentheDORCscoresand

theORCAscoresadecisionwasmadetoexaminethecorrelationsofthesubscalesoftheDORC:

reflectivethinking,criticalstance,collaboration,flexibility,andperformanceontheORCAThedata,

asdisplayedinTable4.16,demonstratedthatscoresonthecollaborationsubscalehadanegative

correlationwithscoresonboththeORCAlessrestrictedbutnottheORCAmorerestricted.

120

Table4.16BivariateCorrelationsofDORCSubscalesandORCA ORCA-LR ORCA-MRReflective Thinking .195 .269**Critical Stance .164 .231*Collaboration -.115 .009Flexibility .035 .122Persistence .213* .236*Note*=significantat.05level;**=significantat.01level

ChapterSummary

Thischapterpresentedtheresultsofastudyseekingtofurtherenrichourunderstandings

aboutmodelsofonlinereadingcomprehension.Tworesearchquestionswereaddressedinthe

quantitativeportionofthisstudy:

1. Whenpredictingonlinereadingcomprehensionofaproblemsolvingtaskwithina

lessrestrictedinformationspace,whatisthebestfitmodel,aftercontrollingfor

verbalintelligence,ofthepercentageofvarianceaccountedforbyeachofthe

followingvariables:backgroundknowledge,criticalevaluation,anddispositionsof

onlinereadingcomprehension?





comprehension?

Specificallythisstudyexaminedtherelativecontributionsofbackgroundknowledge,

criticalevaluationskills,anddispositionsofonlinereadingcomprehensionaftercontrollingfor

verbalintelligence.Thesevariablesweretestedintwodifferentinformationspaces.Thefirst

context,alessrestrictedinformationspace,allowedstudentstosearchforanyrelevantwebsite.

121

Thesecondinformationcontext,amorerestrictedinformationspace,requiredparticipantstofind

specificsources.

Sequential(hierarchical)regression(Tabachink&Fidell,2001)wasusedtoallowthe

researchertoenterverbalintelligencetothemodelapriori.Avocabularytestwasusedtomeasure

verbalintelligence.Ineachmodelthescoresontheverbalintelligencemeasureexplaineda

significantamountofthevariance,andwereenteredinthefirststep.Inthelessrestrictedmodel

verbalintelligenceexplained16.9%ofthevariance,whichwassignificant,F(1,93)=20.530,p<.001.

Inthemorerestrictedmodelverbalintelligenceexplained15.2%ofthevariance,whichwas

significant,F(1,94)=17.024,p<.001.Theadditionofcriticalevaluationscores,background

knowledgescores,anddispositionscoresinthenextstepexplainedanadditional22.4%variancein

thelessrestrictedmodelandanadditional38.9%ofthevarianceinthemorerestrictedmodel.

Inbothmodels,lessrestrictedandmorerestricted,criticalevaluationmadethelargest

contributionafteraccountingforverbalintelligence.Inthelessrestrictedmodelscoresona

measureofcriticalevaluationwereasignificantpredictor(β=.334,p=.000)ofonlinereading

comprehension.Inthemorerestrictedmodelcriticalevaluationwasalsoasignificantpredictorof

performanceonameasureofonlinereadingcomprehension(β=.499p<.000).

Thisresultwasconsistentwithcognitiveflexibilitytheory.Itwaspredictedthatstudents

whoscoredwellonameasureofcriticalevaluationskillsmayhave“advancedwebskills”andthus

wouldscorehigherontheORCAassessments.Furthermorestudentswhoscoredlowonthe

measuremayhavemadeerrorsofoversimplificationorhavehadbiasintheirmental

representationofthesource.Thiswouldleadtoscoresonthecriticalevaluationmeasurebeinga

strongpredictorontheORCAassessment.

Thisresultwasalsoconsistentwithatheoryofnewliteraciesofonlinereading

comprehension.Criticalevaluationofwebsitesisakeycomponentofonlinereading

122

comprehension.Inbothmodelsscoresonthecriticalevaluationinstrumenthadthelargest

predictiveweight.

Inbothmodels,lessrestrictedandmorerestricted,backgroundknowledgescoresalso

madeasignificantprediction.Backgroundknowledgewasasignificantpredictorofperformanceon

anonlinereadingcomprehensionassessmentinalessrestrictedspace(β=.241,p=.007).).

Backgroundknowledgealsocontributedsignificantlytothepredictionofscoresonthemore

restrictedORCA(β=.189p=.017).

Itwaspredicted,basedontheprinciplesofcognitiveflexibilitytheory,thatbackground

knowledgewouldbeasignificantpredictorinbothmodels.Furthermorethepredictionwasthat

backgroundknowledgewouldnotbethestrongestpredictorineitherthelessrestrictedorthe

morerestrictedmodelasstudentswouldhavetorelymoreon“advancedwebskills”ratherthan

topicalknowledge.

Itwaspredicted,basedontheprinciplesofatheoryofnewliteraciesofonlinereading

comprehensionthatbackgroundknowledgewouldhavearoleinpredictingperformanceona

measureofonlinereadingcomprehensionbutitwouldnotbethestrongestpredictorof

performance.Resultsofthisstudyconfirmedthisprediction.Scoresonanassessmentofcritical

evaluationskillswerethestrongestpredictors.

Scoresonaself‐reportmeasureofdispositionsofonlinereadingcomprehensionwereonly

significantpredictorsinthemorerestrictedmodel.Dispositionsdidnotmakeasignificant

predictioninperformanceonanonlinereadingcomprehensionassessmentinalessrestricted

space(β=.124,p=.137).Dispositionsofonlinereadingcomprehensionmadeasignificant

predictiontothemorerestrictedmodel(β=.291p=.006).

Theseresultsdonotalignwiththepredictionsmadebasedoncognitiveflexibilitytheory.It

waspredictedthatdispositionswouldbeasignificantpredictorinthelessrestrictedORCA.This

predictionwasbasedontheideathatstudentswho“haveopenmindsets”woulddobetterinthe

123

unfetteredenvironmentofthelessrestrictedinformationandthatstructuredsearchesofthemore

restrictedORCA,whilebenefiting,”wouldnotallowforserendipitouslearning.Howevertheinverse

wasfoundintheresultsofthisstudy.Scoresonthedispositionsofonlinereadingcomprehension

weresignificantintheORCA‐MRmodelbutnottheORCA‐LRmodel.

Theresultsalsodidnotalignwiththepredictionsmadebasedonatheoryofnewliteracies

ofonlinereadingcomprehension.Basedonthistheorythepredictionwasmadethatdispositions

wouldbeasignificantpredictorinbothmodels.Howeverdispositionswereonlyasignificant

predictorinthemorerestrictedmodel.

124

ChapterV

QualitativeResults

Inadditiontothetwoquantitativeresearchquestions(numbers1and2)thatwere

previouslyexploredinChapterIV,thisstudyalsoexploredtwoqualitativeresearchquestions

(numbers3and4)usingthinkalouddata:

3. What patterns of online reading comprehension strategies appear among high

and low performing online readers during an online reading comprehension task

within a less restricted information space?

4. What patterns of online reading comprehension strategies appear among high

and low performing online readers during an online reading comprehension task

within a more restricted information space?

Iutilizedatwo‐stageanalysis.Firstthedatawereexaminedforthemesandpatterns

betweenhighandlowperformersacrosstheoverallORCAassessments.Thentoanswerquestions

threeandfourthedatawereexaminedforthemesandpatternsbetweenhighandlowperformers:

first,withinalessrestrictedinformationspaceandthenwithinamorerestrictedinformation

space.Thus,thestageoneanalysisconsistedoflookingforpatternsofstrategyusethatledto

betteroverallperformancewhencompletingonlinereadingcomprehensionassessments.Thenin

stagetwothesepatternswereexaminedmorecloselytoseeifthereweredifferencesbasedonthe

natureoftheinformationspace.

Twelvestudents,fourfromeachparticipatingschool,wereselectedforthequalitative

portionofthestudy.Performanceonthefirstadministrationoftheonlinereadingcomprehension

assessmentwasusedtodetermineparticipantsinthethinkaloudactivity.Theywereselectedas

follows.

125

1. First all students in their class were ranked based on scores on the first administration

of the ORCA. Students were divided into two groups: top 10% of scores and the

bottom 10%



3. Then four students from each participating school were randomly selected from the

list: two students who scored in the 10% of their class and two students who scored

in the bottom 10% of their class.


achievement‐levelgroups(highaverage,low)basedonORCAscores,independentofschool.They

wererankedusingacombinedscoreofonelessrestrictedtaskandonemorerestrictedtask.The

studentswererankedfrom1(lowest)to12(highest)andbrokenintothreegroupsoffour.Thislist

resultedinamedianscoreof10.5,outof24.Themiddlegroupincludedfourstudentswithintwo

scoresofthemedian.Thelowgroupincludedscoresmorethantwoscoresbelowthemedian.The

highgroupincludedfourstudentswithscoresmorethantwoscoresawayfromthemedian.

Tenstudentswereincludedinthefinalanalysissincetwostudentshadtoberemovedfrom

thestudyduetoerrorsindatacollection.Thisincludedonestudentinthehighgroupandone

studentinthemiddlegroup.Table5.1liststhescoresofeachparticipant.

Thestudentsthencompletedtwoadditionalthink‐aloudsessionswiththeresearcher.One

taskinvolvedamorerestrictedinformationspace,wherestudentshadtofindspecificsources,and

theothertaskinvolvedalessrestrictedinformationspace,wherestudentscouldselectanysource.

Studentswerescoredusingadichotomousrubric.ThisrubricappearsinAppendixA.

126

Table5.1QualitativeParticipantsPseudonym Verbal PK ORCA1 ORCA2 Total SchoolHigh Isabella 13 1.5 8 9 17 3Olivia 17 2 9 10 19 3Ava 23 2 9 9 18 3 Medium Alexander 8 2 7 8 15 3Sophia 5 0.5 4 7 13 1Jacob 8 2 7 8 15 2Ethan 7 1.5 2 7 9 2 Low Emma 2 0 1 1 2 2 Michael 3 5 5 3 8 1Jaydan 7 .5 4 1 5 2Note:ORCA1andORCA2representsthetwoassessmentsadministeredtoselectstudentsforthe

thinkaloudtask.Itincludedbothamorerestrictedandalessrestrictedtask.

QualitativeAnalysis:ATwo‐StageApproach

Thischapterpresentstheresultsofthequalitativeanalysisorganizedintwoseparate

stages.Stageoneinvolvedlookingatthedataglobally.Thegoalofstageonewastoidentify

patternsofstrategyusethatleadtosuccessontheORCAregardlessofthenatureoftheinformation

space.

Firstfrequencytablesofrawcodeswereexaminedforpatterns.Thecodingscheme

consistedofobservablereadingstrategies.Thensuccessivepassesthroughthedatawere

conductedbyreexaminingthetimestampofcodinginstances.Iwouldreturntothevideoat

specifictimestampsandcomparecodinginstancesacrosscases.Thisallowedfordatareduction

andpatternstoemerge.Thedatawerethenreexaminedinordertogrouppatternsuntilthemes

emerged.Thisapproachleadtothemes,whichconsistedofpatterns,andeachpattern,consistedof

observedbehaviorsfromfrequenciesofcodes.

127

Thesecondstageofanalysisbuiltupontheresultsofthefirststage.Thegoalofthesecond

stagewastoexaminethethemesfromstageonetoanswerresearchquestionsthreeandfour.This

stageinvolvedmakingadditionalpassesthroughthedatatoseeifpatternsemergedthatwere

differentbasedonthenatureoftheinformationspace:morerestrictedorlessrestricted.Pattern

identificationinvolvedaniterativeprocessofreorganizingthedatafromstageoneandreworking

groupingssothatthepatternstructuresdefinedthemesinthedata.

Examiningpatternsinthesecondstageofdataanalysisrevealedadifferenceamong

strategyusebyhighandlowperformers.Itwasevident,byexaminingthedatathattherestricted

natureoftheinformationspaceonlyaffectedpatternsofstrategyuseonthelocatingtasks.

Thereforethesepatternswerethengroupedintothemessurroundingspecificprocessesinvolved

inlocatinginformationduringtheORCAtasks.

StageOneThemes

Thegoalofstageonewastoidentifypatternsofprocessesthatleadtooverallincreased

performanceontheORCAassessment.Recursive,analyticinductivemethods(Angrosino,&Mays

dePerez,2000;Bogdan&Biklen,2003)wereusedtomakeinitialpassesthroughcodedvideosof

studentthinkaloudstoidentifypatterns.Thepatternsthatemergedwereevidentacrossall

elementsofonlinereadingcomprehension:locating,evaluating,synthesizing,andcommunicating.

Thepatternsthatemergedwerealsoevidentinboththelessrestrictedandthemorerestricted

tasks.

Thesepatternswerefurtherdistilledassuccessivepassesweremadetoorganizethe

patternsintothemes.Intheend,fourglobalthemesemergedthatrepresentedcentraltrendsinthe

data.Thesethemesandtheirassociatedpatternsconsistedofthefollowing:

128

Themeone:OverallORCAscoresappearedtoberelatedtostrategictextassembly.

1. Studentswithgreaternavigationalskillswerebetteratreadingmultiple

sources.

2. Studentswhousedcomprehensionmonitoringstrategiesweremore

successfulontheORCAtasks.

3. Studentsmoresuccessfulatsynthesizingonlineinformationreturnedto

theirsourceswithgreaterfrequency.

Themetwo:OverallORCAscoresappearedtoindicatealackofsourceevaluationknowledgeamongparticipants.

1. Fewstudentsweresuccessfulatevaluatingauthorexpertise,evidence

usedbyanauthor,andauthor’spointsofview.

2. Studentsmadeerrorsofoversimplificationduringonlinereading

comprehensiontasks.

Themethree:StudentswhorecalldetailsfrommemoryappeartoperformbetterontheORCAtasks.

1. Students who recalled details from memory when combing multiple sources

may be better at synthesis tasks.

2. Students who recalled details from memory may be better at taking a

position.

Themefour:Studentsappearedunpreparedtoengageinthecommunicationstrategiesnecessaryforonlinereadingcomprehensioninacademicsettings.

1. Studentresponsesfailedtoprovideadequateinformation,especiallyevidence

fromwhattheyhaveread.

2. Studentsdidnotusetheaffordancesofonlinecommunicationspaces.

129

Themeone:OverallORCAscoresappearedtoberelatedtostrategictextassembly.

Acleardifferenceemergedbetweenstudentswithhighlevelsofonlinereading

comprehensionandtheirlessskilledpeersaroundanabilitytostrategicallyassembletextswhile

engaged.inInternetinquiry.Afteralltheconstructionoftextsinsituisanoveldifferenceofonline

readingcomprehension(Leuetal,2004).Thusthisthemewasdefinedastheabilitytoreadfor

meaningwhileflexiblyapplyingbothnavigationstrategiesandcomprehensionmonitoring

strategies.

Threepatternsinformedthistheme:(a)Studentswithgreaternavigationalskillswere

betteratreadingmultiplesources;(b)Studentswhousedcomprehensionmonitoringstrategies

weremoresuccessfulontheORCAtasks;(c)studentsmoresuccessfulattakingapositionwhile

synthesizingonlineinformationusedmorenavigationstrategies.

Patternone:Studentswithgreaternavigationalskillswerebetteratreadingmultiple

sources.Oneofthefirstelementsrequiredforstrategictextassemblyistheabilitytomanipulate

multiplewindowsandtabs.Participantswhoscoredinthehighestperforminggroupontheonline

readingcomprehensionassessmentweremorelikelytoutilizemultipletabsasatooltomove

betweenthetaskandtheirsources.Thisallowedthemtocreateusefulmultiplesourcetexts

throughsefl‐directedtextconstruction(Coiro&Dobler,2008).Inaddition,theyweremorelikely

toclickonadditionallinksattheinitialwebpagetheyfoundbasedonsearchresults.Thisallowed

participantstohaveawiderselectionofsourcestoselectduringtextassemblage.Thesepatterns

aredisplayedinTable5.2.Thispatternincludedtwostrategies:(a)utilizingtabstonavigate

betweentaskandsourceand(b)usingtabstonavigatebetweenmultiplesources.

130

Table 5.2 Frequency of Navigational Strategies ParticipantsgroupedbyORCAscores

Movesbetweentaskandsource

Navigatebetweenmultipletabsorwindows.

High Isabella 20 4Olivia 29 11Ava 14 6TOTAL 64 21 Medium Sophia 16 2Alexander 13 6Jacob 11 3Ethan 11 3TOTAL 51 14 Low Emma 21 2Jaydan 13 3Michael 15 1TOTAL 49 6 Utilizingtabstonavigatebetweentaskandsource.Akeyoveralldifferencebetweenhigh

andlowperformerswasthefrequencyinwhichstudentsmovedbetweenthetaskandthesource

orbetweenthetaskandmultiplesourcesbyusingtabs.Thestudentswhosuccessfullyusedtab

browsingseemedtorelyonthetooltousenavigationalstrategies.Thisstrategysupportedthe

strategictextassemblyofskilledonlinereaders.Successfulstudentsusedwindowsortabbrowsing

tonavigatebetweensourcesandtasksandbetweenmultipleopensources.Thesestrategies

supportedtheirnavigationasstudentsusedthetabstomovebacktothetasktoremembertheir

purposeinreadingasource.

131

Theadventoftabbedbrowsinghasalteredhowmanyreadonline.Theseparticipantswere

infantswhenMozillaintroducedtabbrowsingtoFirefoxin2000.Thusitwasnotasurprisetoseea

relianceontabsacrossallthreegroups.Stillitisclearthatstudentswhoutilizenavigational

strategiessuchastheeffectiveuseoftabbedbrowsingperformedbetter.Forexamplestudentsin

boththehighandmiddlegroupswhoweremoresuccessfulmovedbetweenthetaskandtheir

sourceswithgreaterfrequencyandease.

AvausedtabssuccessfullywhenshehadtoparaphrasetwosourcesontheCausestask.Ava

beganthesecondsynthesisitemontheCausestaskbyreadingtheitem.Shehadfourtabsopen:the

taskandherthreesources.AfterreadingtheitemAvathenclickedonthetabforthesecondsource

thatshefound.Avareadthewebsitefor12seconds,andthenclickedbackonthetabtothetask.

Shethenreadthetaskaskinghertocombinetheinformationsheread.Avanextclickedonthe

secondsourceshefound.Shescrolleddownandreadthewebsiteforafewseconds.Avathen

typed:

Therewasn’tjustoneeventthatcausedtheAmericanrevolution,thereweremany

eventssuchastheFrenchandindianwar,andwhenthebritishstartedtocontrol

howamericansboughtteaand…

IthenaskedherwhereshegotherinformationfromandAvaclickedonherthirdsourceand“said

thisone,andthisone”assheclickedonhersecondsource.Avathenmousedoverlinkstovarious

causesonhersecondsourceandthenclickedbacktoherthirdwebsiteshefound.Shereadthe

pageandthenclickedbackonhertaskandaddedthewords,“stampact.”Byclickingontheactivity

tabtorereadthetaskAvawasengagingincursorcontrol.Thishelpedherincompletingthe

synthesistasks.

Oliviawasaskillednavigatoroftabbedbrowsing.Shewouldconstantlyreferbacktothe

taskinordertoremindherselfofthegoal.Afterre‐readingthetaskOliviawouldmovebacktoher

sourceanddecideifitwasrelevant.

132

Michael,inthelowgroup,ontheotherhand,didnothaveastrongunderstandingoftab

browsing.Heoftenwouldaccidentlycloseouttabshewasusing.Twiceheclosedoutofthetasktab

andthetaskhadtoberestarted.IntheseinstancesIforwardedhimtohislastquestion.Michael

actuallyopenedanadditionalwindowtoconductasearch.Hethenaskedmeifhecould,“Exitoutof

thesetwo,”whilemousingoverthe“Closex”intheupperlefthandcorner.Iexplained,“Youmight

needthoselater,”andMichaelminimizedthewindowwiththetask.Laterheforgotheminimized

thetaskandhadtoberemindedbyme.

Emma,inthelowgroup,seemstohaveusedtabsquiteoften.Howeveranexaminationof

thevideodatarevealedthatEmmawasnotefficientinheruseoftabstoreadmultiplesources.She

wouldoftenclickthroughthetabs,losingherplace,forgettingwhatsourceshewastryingtoread.

Usingtabstonavigatebetweenmultiplesources.Anotherkeydifferencethatledto

higheroverallperformancewasnavigatingbetweensources.ForexampleIsabella,inthehigh

group,usedthetaskinterfaceasanote‐takingtoolassheread.Isabellawouldseamlesslymove

betweenuptofiveopentabsinherbrowser.Ontheturningpointtask,forexample,shehadfive

tabsopen.Shewouldmovethroughthesetabsandfinddetailsoneachpageduringthesynthesis

tasks.Whenaskedaboutmovingbetweentabs,Isabellasaid,“Itiseasiertofindinformationthis

way.”

Alexander,inthemiddlegroup,alsousedtabstoparaphrasetworelevantdetailsfromeach

sourcehefound.Whileworkingonthecausestask,Alexandersaid,“ItsaystotakenotessoIam

goingtolookatallthreeandtakenotesfromeachofthem.”Alexanderthenclickedonthetabfor

thefirstsourcehefound.Heclickedbackontothetaskandtypedintwodetails.Hethenclickedon

atabtothethirdsourcehefound.Alexspentafewsecondsreadingthesource.Hethentabbedback

tothetaskandaddedathirddetail.

Michael,ontheTurningPointtask,wasnotassuccessfulatsynthesizinginformationfrom

twosources.InfactMichaelneverclickedonthetabstomovebetweensources.Onthesecond

133

synthesisitemMichaelquicklyreadthetaskandclickedonthetabforthewebpageaboutthe

TurningPointoftheRevolutiononsocialstudeisforkids.com.Hethenbegantoparaphrasethe

detailsfromthesource.Besidesafewphrases,Michael’sresponseisalmostverbatimfromthe

website.Inordertogetthewording,names,anddatescorrectMichaeltoggledbackandforth

betweenhissourceandtaskatotalofsixtimes.Michaelneverclickedonthesecondsource.

Patterntwo:Studentswhousedcomprehensionmonitoringstrategiesweremore

successfulontheORCAtasks.AnotherkeydifferenceinoverallsuccessontheORCAwastheuse

ofcomprehensionmonitoringstrategiesbyskimmingwebsitesorreadingforsustainedperiods.

Comprehensionmonitoringwascentraltostrategictextassembly.Thispatternisdisplayedin

Table5.3.Twostrategieswereevidentinthispattern:(a)usingastrategyofskimmingwebsitesto

identifykeydetails;(b)andengagedreadingwithinawebsite.

Table5.3FrequencyofComprehensionMonitoringStrategiesUsedtoReadWebsitesParticipantsgroupedbyORCAscores

Skimswebsite

Engagedreadingofrelevantwebsite

Summarizesawebsite

Usesinternalsearchengine

High Isabella 9 1 0 0Olivia 15 4 1 0Ava 10 3 0 0TOTAL 34 9 1 0 Medium Sophia 13 5 0 0Alexander 15 0 1 0Jacob 6 5 0 1Ethan 10 0 0 1TOTAL 44 10 1 2 Low Emma 2 2 0 0Jaydan 8 3 0 0Michael 4 1 0 0TOTAL 14 6 0 0

134

Usingastrategyofskimmingwebsitestoidentifykeydetails.Skimmingandscanning

wasdefinedasquicklyscrollingupanddownawebsite.Overallstudentswhoweremoresuccessful

ontasksskimmedmoresources,lookingforrelevantdetailstointegrateintheirresponses.This

wasevidentinboththehighandmiddlegroups.Studentsinthelowestperforminggroupdidnot

skimwithgreatfrequency.

Participantswhoperformedwellskimmedandscannedinformationmorewithinawebsite.

Inessenceeffectiveonlinereadersquicklyskimmedwebsites,checkedthesourceagainsttheir

purposeandmovedonfromirrelevantsources..Olivia,forexample,wasveryadeptatquickly

readingawebsiteanddisregardingirrelevantinformation.Shewouldoftenskimandscana

websitebeforeshereadthesource.Whenaskedshesaid,“Iliketoquicklylooktoseeifithasany

linksorinformationIcanuse.”

Studentsinthemiddlegroupalsousedastrategyofskimmingwebsites.Themean

frequencyforthemiddlegroup(10.4),however,wasnotquiteashighasthatforthetoptiergroup

(11.3).WhenSophiawasworkingontheTurningPointtasksheclickedonthesite

SocialStudiesforkids.com.Hercursorthenscrolledovertheheadingsandthefirstfewsentencesof

thearticle.Whenaskedwhatshewasdoing,Sophiasaid,“Ithadinformationaboutthe

revolutionarywar.Iamgoingtoreadit.”

Studentswhowerelesssuccessfulalsoskimmedandscannedsources.However,theyused

thestrategywithlessfrequencythantheirmoreaccomplishedpeers.Ethanforexamplelandedon

aWikipediaarticleduringtheTurningPointtask.Hequicklyscrolleddowntothebottomandback

up.Hethenrecordedhisanswerascorrectwithoutactuallyreadingthewebsite.

Jaydandidnotalwaysscanhiswebsitesforrelevantinformation.DuringtheDelawaretask

JaydanlandedontheSocialStudiesForKidswebsite.Hescrolledupanddownandsaid,“Ithasalot

moreinformationaboutit.Soitmighthavemorefacts.”AfterskimmingthepageJaydanwentonto

135

readthesite.YetonthesametaskhelandedonaWikipediaarticle,selectedtheentiretextsand

copiedthatashisresponsetoasynthesisprompt.

Engagedreadingofwebsites.Engagedreadingofsourceswasdefinedasaperiodof

readingbeyondtensecondsthatwasaccompaniedwitheithercursormovementorreadingaloud.

Basicallystudentswhoreliedonstrategictextassemblyhadtheconditionalknowledgeofwhento

slowdownaswellasspeedup.Inadditiontoskimmingandscanning,studentswhoweremore

successfulontheORCAtaskspentmoretimereadingatselectedlocations.

Olivia,amemberofthehighestscoringgroup,heavilyinvestedtimeinthereadingof

sources.OntheWomentaskshelandedonawebpage,AmericanAthenasonthewebsite

AmericanRevolution.org.Oliviaspentoverfiveminutesreadingthesource.Shescrolledthroughthe

sourcemultipletimes.Olivia’sapproachtotheCausestaskwasverysimilar.Sheoftenspent

minutesonasource,wherehercounterpartswouldoftenreadasourceforlessthanaminute.Once

Oliviafinishedreadingasourceshewouldprovideaverbalsummary.

Studentsinthemiddlegroupalsousedastrategyofengagedreading.Theywerealsomore

likelytoreadasourceoutloudwhencomparedtothehighgroup.Thisstrategywasmoreprevalent

withSophiawhospentmoretimeonsourcesthananystudentbesidesOlivia.WhenSophialanded

onthewebpagetitled,“What’sWrongwiththisPainting”duringtheDelawaretasksheexplained

thatshewaslookingforinformationabouttheaccuracyofthepainting.Sophiathenproceededto

readthewebsiteoutloud.Sophiachosetoreadallofherwebsitesverbally.

Alexander,alsoamemberofthemiddlegroup,spentsubstantialtimereadingasource.

WhenAlexanderlandedonasourceandchoseitasrelevanthewouldreadthesourcetohimself.

Alexander,likeOlivia,wouldthenprovideaverbalsummary.Forexample,ontheDelawaretaskhe

wasreadingasite.OncehefinishedAlexanderstoppedandsaid,“Ireadalittlebitofthisandit

says,‘itcontainsanoftendiscussedhistoricalinaccuracy.”

136

Studentsinthelowestperforminggroupalsousedastrategyofengagedreadingon

websites.Howeverthisstrategywasemployedwithlessfrequencythantheirmoresuccessful

peers.Onceagain,likestudentsinthemiddlegroup,thisstrategywasnotedmostoftenwhen

studentswouldreadasourceoutloud.Alsothetimespentonsourcesbythesestudentswasmuch

lessthanthetimespentbystudentsinhigherachievementgroupssuchasOliviaandSophia.

Jaydan,forexample,ontheDelawaretask,foundhimselfonapageonthe

socialstudiesforkids.comwebsite.Hestartedbyreadingthefirstparagraphoutloud.Jaydan,

however,didnotreadanydetailsthatweresignificanttothetask.

Emmadisplayedarudimentaryuseofaengagedreadingonawebsitetoidentifykeyideas.

Shewouldstoptoreadasource,butEmmaoftenmovedquicklyfromthesourcebacktothetask

withoutreadingtheentirepieceandEmmamissedkeydetailsthatwouldanswerthequestion

posedinthetask.ForexampleontheWomenintherevolutiontaskEmmaspentonlyafewsecond

witheachsource.

Patternthree:Studentsmoresuccessfulatsynthesizingonlineinformationreturned

totheirsourceswithgreaterfrequency.Returningtosourceswasanotherkeycomponentof

strategictextassembly.Thisrequiredbothnavigationalskillsandcomprehensionmonitoring.

Thispatterninvolvedtwostrategies:navigatingtoasourcetoreadwhiletakingapositionand

navigatingtothesourcetocopyandpastedetails.Thefrequenciesofthesestrategiesaredisplayed

inTable5.4.

137

Table5.4NavigationStrategiesUsedWhenTakingaPositionParticipants grouped by ORCA scores

Student returns to the source

Student copies and past details in reference to claim.

High

Isabella 1 2Olivia 19 0Ava 1 0TOTAL 21 2

Medium

Sophia 3 1Alexander 4 0Jacob 0 0Ethan 0 0TOTAL 7 1

Low

Emma 1 1Jaydan 0 0Michael 0 0TOTAL 1 1

Navigatingtoasourcetoreadwhiletakingaposition.Participantswhoreferredbackto

thesourcestheyfoundduringthefinalsynthesisitemenhancedtheirperformanceontheORCA

tasks.Thisstrategywasdefinedasclickingonasourceforanygiventimeduringthetakeaposition

items.Itshouldbenotedthateverystudentinthehighestperforminggroupreturnedtothesource

aminimumofonetime.Twooutofthreestudentsinthemiddlegroupusedthisstrategy,andonly

onestudentinthelowestgroupusedthisstrategy.

Studentsinthehighestperforminggroupusedastrategyofreturningtothesource.Olivia

continuedherpatternoflongsustainedreadingonthefinalsynthesistask.Shetookatotaloffour

138

minutestocompletetheitem,thelongestofanyoftheparticipants.Oliviabeganbyreadingthe

item,andtyping,“Thewomen.”IaskedhertoexplainherthinkingandOliviathenclickedontothe

lastsourceshefoundandsaid,“therolethattheyplayed,umtheyfoughtfortheircountrytoo,but

bydoingotherthingslikestickingupfortheirhomes.”Oliviathenclickedbackonthetaskand

typed:

Thewomenfoughtadifferentpartofthewarathome,theyprotectedtheirhome

frominvaders,theyevendecidedtostopdoingsomethingsinordertosavetheir

countrythatwasatbattle.

Oliviathenmovedontothequestionaskinghertosupportherposition.Shereadsilently,ona

singlesource,forapproximatelyoneandahalfminutes.Shethenclickedbackonthetask,and

typed,“Theyendedupsigningadeclarationtostopusingteatohelpthewar,alsotostopthe

Englishimportsthatwerecoming.”

Onlytwostudentsinthemiddlegroupsreturnedtothesource.Alexander,forexample

navigatedtohissourceduringtheCausestask.Hebeganthetakeapositionitembywriting,“Ithink

thattheywerethatAmericanpeoplewereveryindependent...”Alexanderthenclickedonthe

AmericanPersuaderswebsiteandscrolledthroughthepage.Heclickedbackontheaskandadded,

“andtheBostonTeaparty,BostonMassacre,andtheStampAct.Alexanderthenmovedontothe

secondquestion.Hetyped,“Ifoundmyfirstexampleinwordsfrom.”Alexanderthenclickedonthe

tabtohissecondsource,socialstudiesforkids.com.HecopiedandpastedtheURL.Hethenrepeated

thisprocessforthethirdwebsitehefound.

Studentsinthelowestgroupdidnotreturntothesourcewhiletakingaposition.Only

Emmadidonce.JaydanorMichaeldidnotusethisstrategy.

Navigatingtothesourcetocopyandpastedetails.Studentsalsoreliedoncopyingand

pastinginthefinalsynthesistaskasastrategy.Thisstrategywasdefinedascopyingandpastinga

139

detail,andnottheURL,fromthewebsite.Theuseofthisstrategywasnotakeyindicatorofoverall

ORCAperformance.Onestudent,ineachofthegroups,usedthisstrategy.

IsabellacopiedandpasteddetailsfromasourceontheTurningPointtask.Shebeganthe

taskbyansweringthefirstquestionwith,“WhenthebattleswouldhappentheAmericanswere

takingoverandwinningmorethantheBrisith”.Shethenmovedonthesecondquestioninthetask.

Sheclickedononeofhersourcesandcopiedandpasted,“AmericanVictoryatSaratoga.”Isabella

thenadded“…becausethatisoneoftheBattlestheywon.”Shethenclickedonanothersourceand

copiedandpasted“TheBattleofBennington”Isabellathenfinishedthesentencewith“…another

battletheywon.”

Sophiaalsoreliedonacopyandpastestrategy.Whenshestartedthethirdsynthesisitem

ontheTurningPointtaskSophiaimmediatelytyped,“WhentheAmericansandFrenchhad

surroundedtheBritish.”Shethenquicklyclickedononeofhersourcesandcopyandpastedforthe

secondhalfoftheitem.

Emmawastheonlystudentinthelowestperforminggrouptouseastrategyofcopyand

paste.Shedidnotemploythestrategywell.Forexample,onthecausestaskshebeganbyreading

thetask.Emmathenclickedonthetabtooneofhersources.Emmathenproceededtoreadfortwo

minutes.Shethencopiedandpastedalinefromthetaskforherposition.Shethenclickedbackto

hersource,andimmediatelymovedbacktothetaskandtyped,“IDK,”afterthesentenceshecopied.

Themetwo:OverallORCAscoresappearedtoindicatealackofsourceevaluationknowledge

amongparticipants.

Theresultsfromthequalitativeportionofthisstudysupporttheconclusionthatsource

evaluationmaybethemostcriticalcomponenttoonlinereadingcomprehension.Sourcing

strategiesfellalongthetwocontinuumsofrelevancyjudgmentsandcredibilityjudgments.Stage

oneanalysisrevealedthatstudentswhousedmorestrategiestomakecredibilityjudgmentsonthe

140

ORCAtasksoutperformedtheirpeersregardlessofthenatureofthetask.Strategiesforrelevancy

judgmentswereaffectedbythetaskandwereanalyzedinstagetwo.

Twopatternsinformedthistheme:(a)fewstudentsweresuccessfulatevaluatingauthor

expertise,evidenceusedbyanauthor,orauthor’spointsofview;(b)studentsmadeerrorsof

oversimplificationwhenevaluatingonlinesources.

Patternone:Fewstudentsweresuccessfulatevaluatingauthorexpertise,evidence

usedbyanauthor,orauthor’spointsofview.Thejudgingofwebsitecredibilityinvolved

complexsourcingstrategies.Thepatternthatemergedamongtheparticipantswasagenerallackof

effectivestrategiestoevaluateauthorexpertise,evidenceusedbyanauthor,orauthor’spointof

view.

Strategiesusedtoevaluateauthorexpertise.Evaluatingauthorexpertisewasatwo‐step

process.Thefirststepwasidentifyinganauthor.Whilereadingbooksstudentsknowtolookonthe

frontcover,butincomplexonlineenvironmentsthestudentsstrugglewithcorrectlyidentifyingthe

author.ThisisevidentinTable5.5.Inthisstudystudentsusedastrategyofidentifyingtheauthor

inthetext,usedastrategytoidentifytheauthoronanaboutuspage,orhadaninabilitytofindthe

author.

141

Table5.5FrequencyofStrategiesUsedtoIdentifyanAuthorParticipants grouped by ORCA scores

Finds the author in the text

Finds the author on an about us/bio page

Infers the author from the text.

Does a secondary search for the author.

Does not locate the author

High Isabella 1 0 0 0 1Olivia 2 0 0 0 0Ava 2 3 0 0 0TOTAL 6 3 0 0 0 Medium Sophia 2 1 0 0 0Alexander 1 0 0 0 0Jacob 1 1 0 0 0Ethan 1 0 0 0 1TOTAL 7 2 0 0 1 Low Emma 2 0 0 0 2Jaydan 4 1 0 0 0Michael 1 0 0 0 1TOTAL 5 1 0 0 5

Usingastrategyofidentifyingtheauthorinthetext.Themostfrequentstrategydisplayedby

thestudentswastoidentifytheauthorinatext.Thisstrategyinvolvedlookingforinformation

underanarticletitle,theheaderorthefooterofthepage.Allofthestudentsinthehighperforming

groupwereabletoidentifytheauthoronatleastoneoftheirtasks.Therewasalsonodifference

betweenthehigh,medium,andlowgroupsintheuseofastrategytoidentifytheauthorinthetext.

Theuseofthisstrategywasusuallypredicatedmorebythesourcethanbytheabilityofthe

student.Inotherwordsiftheauthorwasreadilyavailableonthepagethestudentswouldnotapply

morecomplexstrategiessuchaslookingforanaboutuspage.Forexamplemanyofthestudents

chosetoevaluatethewebsiteAmericanRevolutionPersuaders.Thewebsiteclearlyliststheauthor

asJeremyJonesinthetitle.Yetifyougotothewebsite’shomepagebyremovingthefileextension

142

intheURLyouarebroughttoabusiness.TheAmericanRevolutionPersuaderswasachild’sessay

hostedonaparent’swebsite.

Usingastrategyofidentifyingtheauthoronanaboutuspage.Acrossthetasksonlyfive

studentsclickedonlinksthatwouldprovideadditionalinformationabouttheauthor.Thisstrategy,

likemostofthecriticalevaluationstrategieswasnotparticulartoanyspecificgroupof

participants.

Avatriedtoutilizethestrategyonbothofhertasks.Shewasnot,however,always

successful.ForexampleontheDelawaretaskAvachosetoreviewthearticle“What’swrongwith

thispainting?”hostedontheDelawareCrossingParkwebsite.WhenlookingfortheauthorAvafirst

clickedonthehomelink.Shethenclickedonalinktositeusage,whichhaddetailsaboutrenting

parkspace.Avathenclickedonthecontactuslinkfollowedbythe“aboutus”link.Once,onthe

“aboutus”pageAvascrolledtothebottomandcopiedinformationaboutthePennsylvania

Governor.Sheenteredthisinformationastheauthor.

SophiaalsotriedtoidentifytheauthorontheDelawareCrossingParkwebsite.Sophia

scrolledupanddownthewebpage,“What’swrongwiththispainting.”Sheexplainedthatshewas

tryingtofindtheauthor.NextSophiaclickedonthe“aboutus”link.Whenshecouldnotfindthe

authorSophialeftthepageandchoseasourcethatlistedtheauthorunderthetitle.

Jacobwasabletousethestrategyoffindinginformationonasecondarylink.Duringthe

CausestaskJacobwasonthewebsiteamericanhistorycentral.com.Whilehelookedfortheauthor

hescrolledtothebottomofthepage,andreadthefooteraloud,“Multieductator,Inc.”“Ohwait,”he

addedasheclickedona“contactus”link.Thisopenedupanemailprogram.Jacobclosedthe

programandclickedonthe“aboutus”link.Hereadthepageoutloudandsaid,“Thereitis.”Jacob

thenentered“multieducatorinc”ashisauthor.

Inabilitytoapplyastrategytofindanauthor.Sixoftheparticipantswereunabletoidentify

theauthorinatleastoneoftheirtasks.Ethancouldnotlocatetheauthoronhistaskaboutthe

143

historicalaccuraciesofGottlieb’spainting.EthanchosetoidentifytheauthorofaWikipediaarticle

hefound.Ethanbeganbyclickingbetweentwowebsites:theWikipediaarticleandanothersource.

Ashewasmovingthroughthepages,scrollingupanddown,theresearcherasked,“Whatareyou

thinking?”Ethan,replies,“Icannotfindtheauthor,theauthortooneofthewebsites.”Hethen

typed“IDK”intheboxfortheauthor.Itmustbenotedhedidnotclickanylinksoneithersite.

Ethanjustscrolledupanddownthepages.

Michaelwhenlookingforanauthorforthe“womenintherevolution”taskquicklytabbed

throughallofhissources.HesettledonawebsitefortheStonyFieldBattlefieldStateHistoricPark.

Thesiteclearlylistedtheauthoratthetopofthewebpageas,GillianCourtney,ParkRanger.

MichaelincorrectlylistedtheauthorsasGillianCourtneyandParkRanger.

Highperformingstudentsdidnotalwaysidentifytheauthor.Oliviaalsostruggledononeof

hertasks.Shereadthetaskandthenclickedonthetabtooneofhersources,historycentral.com.

Shethenscrolledupanddownthepage.TheresearcheraskedOlivia,“Whatareyoulookingfor?”

Sheresponded,“Informationabouttheauthor.”Oliviathenpausedonthecitationlistedatthe

bottomofthepage.Shethenclickedbackovertothetaskandtypedinthename,“Pheobe

Hanafore.”Thiswasnottheauthorofthewebsitebutwasasourcecitedinthewebsite.

IsabellaalsocouldnotidentifytheauthorontheTurningPointtask.Isabellawaslookingfor

theauthorofthewebsiteAmericanRevolution.org.Shespentoneminuteandvisitedthreeseparate

pagesonthewebsitelookingfortheauthor.Isabellaclickedonheroriginalsourceaboutabattlein

therevolution.Shethenclickedonmoreinformationaboutthebattleandfinallythewebsite’s

homepage.Aftershecouldnotlocatetheauthororpublisherofthewebsitesherecordedthe

answer,“websitemakers”astheauthor.

Thesecondstepinevaluatingauthorexpertiserequiredreaderstojudgetheauthor

expertiseusingcommonmarkersofauthoritysuchasoccupation,institutionalaffiliation,or

education.Overalltheparticipantsstruggledtoevaluatetheexpertiseoftheauthor.Thepatternis

144

displayedinTable5.6.Onlythreestudentscorrectlyrespondedtoatleastonepromptabout

expertise.Onlyonestudent,Ava,earnedbothscorepointsforevaluatingauthorexpertise.Thetwo

mostdominantstrategiesusedwerejudgingexpertisebyusingthecontentofthewebsiteorby

focusingonspecificdetails.Studentswhosuccessfullyjudgedauthorexpertisereliedoneffective

markersofexpertise.

Table5.6FrequencyofStrategiesUsedtoJudgeAuthorExpertiseParticipants grouped by ORCA scores

Uses an authoritative title to judge the author

Uses supporting details from content of the website.

Uses institutional information to judge author expertise

Uses background knowledge to judge author expertise

Uses completeness of information to judge author expertise

High Isabella 0 0 2 1 1Olivia 0 1 0 0 3Ava 1 0 2 0 0TOTAL 1 1 2 1 4 Medium Sophia 0 0 0 0 1Alexander 0 1 0 0 2Jacob 0 1 0 0 0Ethan 0 1 0 0 0TOTAL 0 3 0 0 3 Low Emma 0 2 0 0 0Jaydan 2 0 1 0 2Michael 0 1 0 0 1TOTAL 0 2 0 0 3

Usingeffectivemarkersofexpertise.Studentswhocorrectlyevaluatedauthorexpertiserelied

ontwomajorstrategies:institutionalaffiliationandauthor’soccupation.Ava,whocouldnot

identifytheauthorsofawebsitepublishedbytheStateofPennsylvania(shelistedthegovernoras

author)notedthattheauthorswereownersofahistoricalparksotheymustbeexperts.Onher

nexttask,theTurningPointoftheRevolution,Avafocusedontheauthor’soccupation.Shenoted

145

thatDanWhite,creatorofsocialstudiesforkids.comisan“educationalprofessionalwithapassion

forsocialstudies.”JaydanalsonotedthatasasocialstudiesteacherDanWhiteisanexpert.

Sophiausedthepublisheraffiliation,backgroundknowledge,andcontentofsecondary

sourcestoevaluateauthorexpertise.Shewrote:

Ithinkheisbecauseitisa.organdthoseareprettyhelpful.AlsoofwhatIknow

thisallsoundsright.Anotherreasonisthattheyhavealotofinformationthatthe

otherwebsitesdidn't.

SophiafirstfocusedinontheURLextension.Whilenotaguaranteeofexpertiseitdiddemonstrate

ahigherlevelofsourceknowledge.Shethencheckedthedetailsusedbytheauthor.Finallyshe

mentionedthatthewebsitehadinformationfromotherwebsites.Itmustbenotedthatatnotime

didSophiaactuallyevaluatetheauthor.She,likeherpeers,putthemostvalueinthecontentof

websites.

Strategiesusedwhenevaluatingevidenceusedbytheauthor.Anotherimportant

strategy,whenreadingmultiplesourcesonline,isevaluatingthecredibilityofasourcebyjudging

theevidencecitedbytheauthor(Goldman,2010).Thisisalsoanimportantstrategyforthe

disciplineofsocialstudiesasstudentsreadonlinetexts(Manderino,2011).Thisstrategywas

definedasjudgingthecredibilityofthesourcesofclaimsorevidenceusedbyanauthorofa

website.

Overalltherewerenomajordifferencebetweenstudentswhoweremoresuccessfulonthe

ORCAtasksandstudentswhowerelesssuccessfulontheORCAtask.Onlyonestudentsexhibiteda

strategythatevaluatedtheevidenceinawebsite.Exceptforthisstudent,alloftheparticipantsdid

notexhibitstrategiestoevaluatethesourceofevidencewithinawebsite.Thereforetheonly

patternevidentinthedatawasatotallackofstrategyusetojudgetheevidenceusedbyanauthor.

Examiningtheonecaseofsuccessfulstrategyusetojudgeevidenceusedbyanauthor.Sophia

wassuccessfulinevaluatingevidenceusedbytheauthor.Sophia’sstrategywastocheckthe

146

evidenceagainstasecondarysource.Sheresponded,“Becauses|hegivefacts.Andtheother

websitessaythesamething.soHemusthavehisFactsCorrect.”

Strategiesusedwhenevaluatingauthor’spointofview.Participantsinthestudy

displayedthefeweststrategieswhentheywereaskedtoevaluatehowanauthor’spointofview

mayhaveshapedthewordsorimagestheauthorusedonawebsite.Infact,sevenofthestudents

answeredthequestionwithavariationof“Idonotknow.”Furthermoreonlytwosuccessfully

judgedtheauthor’spointofviewforeitherthelessrestrictedorthemorerestrictedtask.Therefore

itisimpossibletodrawadistinctiononstrategyuse.Onceagaintheonlypatternthatemergedin

thedatawasstudents’inabilitytojudgeanauthor’spointofview.

Themostcommonphenomenonnotedwasalackofunderstandingoraninabilitytoanswer

thequestion.Emma,Ava,andMichaelallput“IDK”or“Idonotknow”astheiranswer.Emmabegan

byreadingthequestion.Shethentabbedbackovertohersource,andspentafewsecondsonthe

page.Emmathenclickedonthetabforthetaskandtyped,“IDK.”Emmaalsoput“IDKforher

response.Shesimplyreadthetaskandneverreferredbacktohersourcesformorethanafew

seconds.Infact,onherothertaskEmmacompletedallofthecriticalevaluationquestionswithout

evertabbingbacktohersources.Michaelfinishedthefirsttwoquestionsandpaused.The

researcherasked,“Doyouknowwhatthisquestionisasking?”Michaelresponds,“Ihavenoidea.”

Hethenproceededtotype“IDK.”

Usingauthor’spurposetojudgeauthor’spointofview.Thenextgroupofstudentsreliedon

theauthor’spurposetoinferhisorherpointofview.Whiletheydidnotspecificallyinferhowan

author’pointofviewaffectsherversionofthetruththestudentshadarudimentaryunderstanding

thateverytexthasagoal.Ethan,ontheDelawaretask,forexample,wrote,“toinformreadersthat

GeorgeWashingtondidreallycrosstheDelaware.”ForEthanpointofviewwasconfirmedwitha

genrebasedideaofpurpose.Hereadaninformationalwebsitethereforetheauthor’spointofview

wastoinform.

147

Isabellaalsousedauthor’spurposetodefinetheauthor’spointofview.OntheDelaware

tasksheexclaimedthat,“Ittellsyouthedetailsofthepainting.”OntheTurningPointtaskIsabella

wasevaluatingthewebsiteAmericanRevolution.organdwrote,“Yes,heshowsmanypicturesand

heisnotdefendingonesideheistellingitlikeitis.”IneachcaseIsabellafocusedonauthor’s

purpose.

Examiningthetwocasesofsuccessfulstrategyusetoevaluatingpointofview.OnlyOliviaand

Sophiacorrectlyinferredtheinfluenceofanauthor’spointofview.Sophiacorrectlyinferredthe

author’spointofviewononetask,andOliviaonbothofherassignedtasks.Sophiawasevaluating

aclassroomwebsiteSedivy.tripod.comfortheTurningPointtask.Shewrote,“Theauthorithinkis

tryingtogiveusinformationonthetopic,andshowinguswhathisclasscandoandLearn.”Sophia

focusedontheteacher’sdesiretohighlightwhathisclasswasdoingandhowthisinfluencedthe

designofthewebsite.

Oliviaalsocorrectlyinferredtheauthor’spointofview.Olivia,whenevaluatingawebsite

fortheWomentaskwrote:

Phebetakesthepointofviewshetookbecauseshe’sstickingupforthewomento

showtheycandowaymorethanpeoplethinktheycando.

Oliviaidentifiedtheauthorunderlyingpointofviewinadvancingtherolewomenhaveplayedin

history.Onhernexttask,Oliviadiscussedtheauthor'spointofviewbyidentifyingrevulsionfor

war:

Theauthor'spointofviewdoesinfluencethewordsandimagesusedonthewebsite

becausehispointofviewseemstogoalongthesamepathashiswordsandimages

becauseyoucantellhe'sagainstwhatthe3majoreventsdidtotheUnitedStatesof

Americaandothercountriesinvolvedinthewarbyhiswordchoiceandbythelast

sentenceortwowhenhesaysthathehopesthattheworldwillneverseeanother

AmericanRevolutioneveragain.

148

Patterntwo:Studentsmadeerrorsofoversimplificationwhenevaluatingonline

sources.Inbothlessrestrictedandmorerestrictedtasksandregardlessofabilitylevel,students

madeerrorsofoversimplification(Spiro,2004).Anoversimplificationisamisconceptionlearners

havebasedonpreviousintroductorylearning.(Spiro&Feltovitch,1996).Intermsofonlinereading

comprehension,errorsofoversimplificationoccurwhenusersviewwebsitesasregular,well‐

definedstructures.Instancesofoversimplificationoccurredwhen:(a)studentsoverlyreliedonthe

amountofcontentatwebsites,and(b)studentsoverlyreliedonwebsitefeatures.

Studentsmadeerrorsofoversimplificationbyoverlyrelyingontheamountofcontent

ofwebsites.Thishappenedwhenstudentsusedtheamountinformationatawebsitetoevaluate

authorexpertiseandwhentheyusedtheamountofinformationatawebsitetoevaluatethe

evidenceusedbyanauthor.Themostcommon,yetunsuccessful,strategyforevaluatingexpertise

wasusingthecompletenessofinformationtojudgeanauthor.Thisstrategywasnoted11times

acrossthestudy.Studentsinallgroupsreliedonthisineffectivestrategy(seeTable5.8).

Alexanderreliedontheamountofcontentofthewebsitetojudgeauthorexpertise.Onthe

CausestaskhewasreviewingthewebsiteAmericanRevolutionPersuadersandresponded,“Idon

thinktheauthorisanexpertbutI’msureheknowalotaboutitifheknewthreecausesandcould

supportthem.”

Sophiaalsousedastrategyofrelyingontheamountofcontenttojudgeauthorexpertise.

OntheTurningPointtask,forexample,sheresponded:

No,becausehedidhavesomefactsbutnotallnortheMainfactsontheAmerican

Revolution.Itwasverylittlewritinganddidn'tsayanythingabouttheTurning

Points.

Sophiawasevaluatingtheauthorexpertiseusingtheamountofinformationandnotcommon

markerssuchaseducation,institutionalaffiliation,orexperience.

149

Michaelalsoreliedonthecompletenessofthecontenttojudgeauthorexpertise.Hewas

evaluatingawebsitebyaparkrangerforthewomenoftherevolutiontask.Michaelwroteinhis

response,“Theauthorisbecausehewrotealotanditseamshedidalotofresearch.”Michael’s

responsealsoreflectedhisactions,ashewouldoftenchoosethewebsiteswiththegreatestamount

ofinformation.

Studentsalsomadeerrorsofoversimplificationbyoverlyrelyingontheamountofcontent

atwebsiteswhenjudgingevidenceusedbyanauthor.Onceagainstudentsreliedmoreonthe

quantityandnotthequalityofinformationinthesources.Isabella,forexample,ontheTurning

Pointtask,whenevaluatingawebpageontheAmericanrevolution.orgsitecommentedthatthe

evidenceusedinthewebsitewascrediblebecause,“Hetalksabouteverypartofthebattle.For

exampleduringitbeforeitandafterit.”

Michaelwasalsoillustrativeofstudentswhorelyonthequantityofinformation.Michael

responded:

Yesandno,theauthorusesconvincingevidencebecauseitlookslikeheknowswhat

heistalkingaboutandhedidnotwritealothecouldhavewroteabouttheteaparty

ortheBostonmassacre.

Hewasnotevaluatingtheevidenceintermsofitssource,butinsteadintermsofitcompleteness.

Studentsmadeerrorsofoversimplificationbyrelyingonwebsitefeatures.Thesecond

mostcommonstrategyused,alsounsuccessfully,bytheparticipantstojudgeauthorexpertise

focusedoneitherspecificdetailsortheauthor’sorganizationofthewebsite.Ethan,ontheTurning

Pointtaskforexample,saidtheauthorDavidWhitewasanexpertbecause,“Heusedspecificdetails

andgreatpunctuation.”

Emmanoted,onthecausestask,thattheauthorJeremyJoneswasnotanexpertbecause,

“thereisnopersuasivelanguageused.”Oliviafocusedontheamountoftheinformationandalso

specificdetails:

150

Yes,becausePhebeputalotofinformationintheparagraphsabouteachwoman

whoservedintheAmericanRevolutionalsosheusesspecificdatesofwhen

somethingmajorhappened.

Jaydanalsousedtheinformationtojudgethecredibilityofsources.Hedidnothowever

focusontheoverallquantitybutlikemanyofhispeers,onspecificdetailsinthewebsite.When

evaluatingtheevidenceusedonthewebsiteSocialstudiesforkidsJaydanreplied,“Yesheuses

convincingevidence.HestatedthatthereisnotjustoneTurningPointthattherewasmany.”Then

whenevaluatingasiteforthecrossingtheDelawaretaskJaydanwrote,“Hesaysthatitwasacold

night.Alsohowthecrossingwasasneakattack.”Ineachcaseheneverinvestigatedthesourceof

theevidence.

Otherstudentsunsuccessfullyusedawebsite’sorganizationorfeaturestojudgethe

evidenceusedbyanauthor.EthancommentedthattheauthorofSocialstudiesforkidsusedcredible

evidencebecause,“heusedgreatfactsandeasytofollowwordsandsentences.”Alexanderalso

focusedontheformatofthewebsitebywriting,“Ithinkhedoesuseconvincingevidencebecause

hewritesafullparagraphbackingupwhathethinksthecausesare.”

Themethree:Studentswhorecalldetailsfrommemoryappeartoperformbetteronthe

ORCAtasks.

Thedatafromthisthinkaloudstudyindicatedthatacleardifferencebetweenstudents

whoperformwellandstudentswhodidnotperformwellwastheabilitytorecallinformation

theyhadjustread.Twopatternsinformedthistheme:(a)studentswhorecalleddetailsfrom

memorywhencombingmultiplesourcesmaybebetteratsynthesistasks;and(b)studentswho

recalleddetailsfrommemorymaybebetterattakingaposition.Thesepatternsareexploredin

Table5.7.Whilethesestrategieswereusedbyeverygroupofparticipantsstudentsinthehigher

groupsrecalledmoredetailswithgreateraccuracythantheirpeers.

151

Table 5.7 Frequency of Using a Strategy of Recalling Details From Memory Participants grouped by ORCA scores

Recalls details from memory when combing multiple sources

Recalls details from memory when taking a position

High Isabella 1 2 Olivia 1 2 Ava 2 14 TOTAL 4 18 Medium Sophia 0 2 Alexander 0 3 Jacob 1 4 Ethan 0 3 TOTAL 1 12 Low Emma 0 2 Jaydan 0 3 Michael 0 2 TOTAL 0 7

Patternone:Studentswhorecalleddetailsfrommemorywhencombingmultiple

sourcesmaybebetteratsynthesistasks.Allofthestudentsinthehighgrouprecalledatleast

onedetailfrommemorywhencombiningmultiplesources.However,onlyonestudentinthe

middleandlowgroupsrecalledadetailfrommemorywhencombininginformationfrommultiple

sources.

Ava,inthehighgroup,recalleddetailsfrommemoryonthecrossingtheDelawaretask.Ava

clickedonthetaskandthenonthelastsourcesheread.Shereadquickly;scrolledtothebottomof

thepage,andclickedbackonthetask.Iaskedher,“Whatareyouthinking?”Avaresponds,“Like

thepaintingshowsitwasdaytimelikewithlight,butitwasactuallydoneinthesnowstormat

night.Herresponsecombinedinformationfrombothofthesourcesshereadbutsheonlyreviewed

onesourceduringthesecondsynthesistask.

152

Jacob,inthemiddlegroup,whenhewasonthesecondsynthesisforthecausestaskwas

asked,“Whatareyouthinking?”Jacobrespondedbysaying,“Ihavetocombineallthis,”ashe

tabbedthroughallofhisopenedsources.Jacobrecalledthefirstdetailfrommemorybyalmost

typingasentencefromverbatimthat,“Therewasnosingularcauseofthewar.”

Patterntwo:studentswhorecalleddetailsfrommemorymaybebetterattakinga

position.Therewerenotgreatdifferencesintheuseofthisstrategyacrossallofthe

participants.Therewere,however,differencesinthenumberofdetailsrecalledbysome

students.Ava,forexample,wasabletorecall14separateideasacrosshertwotasks.Sheonly

referredbacktooneofhersourceswhentakingapositionforabrieffewseconds.Duringthe

DelawaretaskAvawrote:

1.no,therearemanythinswrongwiththepainting2.thepaintinghasadaytime

scene,butitwasactuallydoneatnightinasnowstorm.Alsotheboatssizeand

shapeisn’thistoricallycorrect,andifgeorgeactuallystoodupIit,hewouldhave

drowned.

Onhernexttask,Causes,Avarespondedwithoutreturningtohersources:

ithinkthebritishtryingtocontroleverythinganditmadetheamericansfrustrated

becausetheycametoamericatogetfreedom,nottobebossedaround.2.thebritish

increasedtaxesonthethingsthattheamericansneededorboughtalot,likestamps

andtea

IneachcaseAvarecalledallofthesedetailsfrommemory.Avaneverclickedbackontothesources

shefound.Inherresponsesshedirectlyaddressedthequestionposedinthetask.

Studentsinthemiddle‐performinggroupalsoreliedonmemorytorecalldetails.However,

onlySophiaandAlexanderwereaselaborateasstudentsinthehigherperforminggroup.Sophia

beginsthetaskbyaskingme,“Inthisquestiontheyareaskingifitisright?”Ireplied,“Youare

takingaposition.Tellmewhatyoubelieve.”

153

Sophiathenstartedtypingherresponsebyaddingtheword“No.”AssheistypingIasked

heraboutheranswerandSophiarepliedbysaying,“Nobecausetheylikegivegoodpoints.Asin

likeitswinter,soitsnotgoingtobelikewiththesunout.”Sophiathenwrote:

1.No,becauseitswinterandthesunwouldn'tbeout.Thewaterwouldn'tbeas

mellowasitshows.Andtheygivefactthatitdoesn'teventalkplaceintheDelaware

River.2.IfoundthatthesettingthepositionthatGeorgeWashingtonisStandingis

wrong,andnotaccurate.Thethingthepeoplearedoingandevenhowmanypeople

weretherewherewrongto,soIthink.

ItisclearthatSophiakeptthegoalinmindasshetookaposition.Shedirectlyansweredthe

inquiryquestionposedandincludedtwodetailsasevidence.

Alexanderalsorecalleddetailsfrommemory.OntheDelawaretaskherepliedtothe

questionpromptwith:

1.Idon’tthinkit’saccuratebecausethepointtheyraisedaboutitbeingunaccurate

wasgood.2.Ifoundmywebsitesayingtheydidn’tthinktheboatwouldbeabletotay

afloatwithsomnypassengersaboardthesmallboat.

InthisresponseAlexanderisclearlyawareofhisgoal.Hespecificallyaddressesthetask.

Ontheotherhand,somestudentsinthemiddlegroupsuchasJacobonlyimplicitly

addressedthetask.Jacob,forexample,onthecrossingtheDelawaretask,spokeofthelackof

camerastocapturethehistoricalmoment.Jacobwrote:

1.NoIdonotelieveitisbecausehewasn’ttheretoseeithappenohehadtoguess.

2.thattherewasnoonewithcamerastokeeptheimagecorrect

ThefirstsourcethatJacobfoundforthetaskmentionedacamera.Specificallythesite,

AmericanHistory.orgopenedwiththeline:

Wouldn'tithavebeengreatifagroupofnewsreporterswithhightechcamerasand

soundequipmentlinedtheshoresoftheDelawareRiver…

154

Jacobusedthisdetailthroughoutthetaskashismainevidencetosupporttheclaimthatthe

paintingwasnothistoricallyaccurate.

Studentsinthelowestperforminggroupalsousedastrategyofrecallingdetailsfrom

memory.Thesestudentshoweverwerenotassuccessfulatrecallinganumberofrelevantdetails.

Jaydan,forexample,reliedonmemorytodevelopaclaimandprovideevidence.Onbothofhis

ORCAtasksJaydanneverleftthescreenofthetaskthataskedhimtotakeapositiononwhathe

read.Hequicklytypedaresponseandmovedon.OntheTurningPointtaskhewrote:

1.TheTurningPointwasatsaratoga.SomeothersweretheBattleofBennington.2.

itsaysthatthebattleofbenningtonwasgoingtobeasneakattackbutitdidntwork.

OnthecrossingtheDelawaretaskJaydanwrote,“Yesithinkso.‐‐Theysaiditwasarealhappening

andtheysaidwhenitwaspainted.”

Michaelstruggledtousedetailsfrommemoryinhisposts.OntheWomenintheRevolution

taskMichaelwrote,“1.Thaycleanedcookedandweremedic’s.2.IDK”Hedidnottrytouseanother

strategysuchasreturningtohisoriginalsource.

Emmaalsorecalledsparsedetailsfrommemoryinhertakeapositiontask.Onthesame

WomenintheRevolutiontaskEmmawrote,“1.MostBattledwiththemen.2.InonewebsiteI

searchedittoldaboutwomenstayinghomebutthousandsbattled

Themefour:Studentsappearedunpreparedtoengageinthecommunicationstrategies

necessaryforonlinereadingcomprehensioninacademicsettings.

Thedatafromthisthinkaloudportionofthestudyindicatedthatstudentswerenotfully

preparedtoengageinthecommunicationstrategiesnecessaryforonlinereadingcomprehension.

Thisthemewasevidentinstudentresponses:(a)thatfailedtoprovideadequateinformation,

especiallyfromtheevidencethatwasread;and(b)whenstudentsdidnotusetheaffordancesof

onlinecommunicationspaces.

155

Studentresponsesfailedtoprovideadequateinformation,especiallyfromthe

evidencethatwasread.Thestudentresponsestothecommunicationtaskswereshort,

unfocused,andoftendidnotincludeevidencefromwhattheyread.Exceptfortwostudentsno

participantprovidedinformationabouttheirsources.

Whenstudentsfinishedthefinalsynthesistasktheywouldclick“Next”andthenwere

broughttoadirectionscreen(seeFigure5.1).Onthepagewerescreenshotswithdirectionsto

logintothediscussionboard.Atthebottomofthepagewasalinktothediscussionboard.Once

theyloggedonstudentscouldchoosefromthefourtopics(seeFigure5.2).Thenwhentheyclicked

onatopictheysawascreenwithanoriginalpostfromafictitiousteacherandthentworeplies

fromstudents(seeFigure5.3).

Icreatedtworesponsesbyfictionalstudentsundereachtopic.Thesewerebriefstatements

thatcontainedlittleornoinformationthatcouldbeusedbyparticipantsinformulatingtheir

answers.Itisunknown,however,ifthebrevityanddiscourseintheinitialpostsinfluencedthe

lengthandcontentofthepostsbypeers.

156

Figure5.1.ScreenshotofDiscussionBoardDirections

Figure5.2.ScreenshotofDiscussionBoardTopics

157

Figure5.3.ScreenShotofWomenintheRevolutionDiscussion

Thetaskaskedstudentstoagreeordisagreewithamessagepostedbyanotherstudent.The

studentresponsesaredisplayedinTable5.8.Itisclearthatstudentresponses,evenwhentheytake

apositionareshortanddonotuseevidence.Table5.8alsoshowsthatstudentsinthehighest

performinggroupsdidnotengageinanycommunicationstrategiesthatwouldhaveincreasedtheir

performanceontheORCAtask.

158

Table5.8DiscussionBoardResponsesStudent MoreRestricted Lessrestricted Isabella TurningPoint‐Agree Delaware‐disagreeithappenedin

thedeadofnight,andtheystartedinlateafternoon.

Olivia Causes‐TheBostonTeaPartyandFrenchandIndianWarcausedtheAmericanrevolutionbecausetheBritishmadetheAmericanspaytheirshareofthewardebtoftheFrenchandIndianWar.AlsotheBritishputtaxesonteawhentheyknewpeopledidn’thavealotofmoneytospareonuselesstaxeslikentea.

Women‐ActuallythewomendidplayabigpartoftheAmericanRevolution.Theycamtogethertofightathometosavehurtsoilders,keephomessafeabdtostopimportsofgoodsfromGreatBritian

Ava Alsothestampactandsugaractbecauseitincreasedpricesofthingsamericansusedeveryday.

Delaware‐Ithinkthepaintingisn’taccurate,becauseoftheboats,thefactthatGeorgeisstandingup,ifhedidthathewouldhavedrowned.

Sophia TurningPoint‐TheBattleofSaratogawastheTurningPointifthisdidd’thappentherewouldhavebeenachancetheAmericanslost.

Delaware‐Thistomeiswrongbecausethepaintinggivesthewrongfacts.Theplaceandwhathe’sdoingisallwrong.Andthepaintingsendsoutthewrongfacts.Sothisdoesn’tshowwhatitreallylookedlikedwhathappened.Butitdoesshowhimleadinghismen!

Alexander IagreebecauseIfoundtheBritishmadeAmericanspayabigshareofthewardebtfromtheFrenchIndianWarandIagreewithyouransweroftheBostonTeaPartybecausethatwaswenAmericansprotestedtheBritishsothatwasabigcause.

Delaware‐ Ithinkthewarwasreal,butIdonotthinkthepaintingisaccurate.IthoughtthepaintingwasunaccruatebecauseofmanyreasonsIfoundonmywebsitestosupportmythoughts

Jacob Idisagreebecausetherearemanymorecausesthantheses.

Idisagreebecausebeleivthebattleactuallyhappenedbutthepaintingisfakebecausehewastatthebattlepaintingsoitisallcorrect.

Ethan TurningPoint‐youareextremly Delaware‐Thepaintingisrealandif

159

rightaboutthatquotebecauseifyougoonwikipedia.organdtypeintheTurningPointoftherevolutionitwillsaythebatteofsaratogainthatsentence

yougotometmuseum.comandtypeinGeorgeWashingtoncrossestheDelawareriveritwilltellyouthetruefactsaboutthishistoricalevent.

Emma N/A Womendidhavearoleanditwas

major.Somestayedhomeandwerehousewivesotherswenttobattle.

Jaydan TheBattleofBenningtonwasanotherTurningPoint.IttookplaceinNewyork.IthappenedAugust161777

Delaware‐Thepictureshowsthattherewasiceanditwasacoldnight.Sohismenweretiredandcoldsoitwashardtofight.

Michael ThebattleofSaratogawasthe

TurningPointoftheAmericanRevolutionnotthebattleofBennington

Thaywereusedasmedics’theycookedandcleaned

Thetaskaskedstudentstoagreeordisagreewithamessagepostedbyanotherstudentand

thenincludeevidencefromwhattheyread.Studentswerealsoaskedwheretheyfoundtheir

information.

InthetopperforminggroupIsabellatookapositionononetaskbutincludednoevidence.

InhernextresponseIsabellatookapositionandincludedevidence.Olivia’sandAva’sresponses,

whilebrief,tookapositionandprovidedevidence.Nooneinthetopgroupincludedanydetails

aboutthesourceoftheirinformation.

Sophia,inthemiddlegroup,forexample,ontheDelawaretask,respondedtoapostbya

fictitiousstudentBrianBwhowrote,“Thepaintingisfake.Thebattlenevertookplace.”Sophia

repliedback:

Thistomeiswrongbecausethepaintinggivesthewrongfacts.Theplaceandwhat

he’sdoingisallwrong.Andthepaintingsendsoutthewrongfacts.Sothisdoesn’t

160

showwhatitreallylookedlikedwhathappened.Butitdoesshowhimleadinghis

men!

BasedonSophia’sresponseitishardtoinferifshewasstatingthatBrianB’spositionwaswrongor

ifthepaintingwaswrong.

EthantookapositionontheTurningPointtask.Inhisprompthestatedthattheauthorwas

“extremlycorrect.”EthanthenwentontoexplainhowtheBattleofSaratogawastheTurningPoint.

However,thepostthatEthanwasrespondingtotookthepositionthattheBattleofBenningtonwas

theTurningPoint.Therefore,eventhoughEthantookapositionhewasrespondingtotheincorrect

prompt.InotherwordsEthanwasarguingthatthefictitiousstudentwhostartedthethreadwas

correctbutthennegatedthispositionwithhisevidence.

Otherstudentshadamoreimplicitposition.Theymadeaclaim,buttheparticipantsdidnot

explicitlyagreeordisagreewithapreviouspost.Emmaforexample,ontheWomentask,

respondedtoJuliowhooriginallyposted,“WomenplayedamajorroledintheAmerican

Revolution.Theydidlotsofstuff.Emmarespondedwith,“Womendidhavearoleanditwasmajor.”

Michaeltookasimilarapproachandposted,“Thayweremedics’theycookedandcleaned.

Intermsofexplainingwherestudentsfoundtheirinformationtheresponseswereonceagain

inadequate.Onlyonestudent,Ethan,actuallyincludedanyinformationabouthissource.One

student,Alexander,impliedtheuseofasource.Themajorityofthestudents,however,

communicatedevidencefromthesourcestheyreadbutdidnotprovideanydetailson“whereyou

gotyourinformationfrom,”aswasaskedinthetaskinstructions.

OnlyEthanusedastrategyofembeddingacitedasourceinhisdiscussionboardpost.In

eachtaskhementionedawebsite,butdidnotprovideanaddresstoaspecificwebpage.Onthe

crossingtheDelawaretaskEthanfirstloggedin.Hemousedoverallfourtopics.Iaskedhim,“What

doyouneedtodo?”Ethanresponded,“Youneedtoclickonwhichonetheyweretalkingabout.”He

161

thenclickedonthecorrectdiscussion,scrolleddownandreadthetask.Heclickedthequotebutton

andtyped,“Thepaintingisrealandifyougotometmuseum.com….”

Alexanderusedastrategyofimplyingasource.Hemadeavaguereferencetothewebsites

hereadinbothofhisdiscussionboardposts.OnthecausestaskAlexanderreferredtothesources

helocated.Hewrote,“IagreebecauseIfoundtheBritishmadeAmericanspayabigshareofthe

wardebt….”OntheDelawaretaskAlexanderwrote,“Ithoughtthepaintingwasunaccruatebecause

ofmanyreasonsIfoundonmywebsitestosupportmythoughts.”Ineachoftheseexamples

Alexanderdemonstratedsomeknowledgeofhowtousesources.

Studentsdidnotusetheaffordancesofonlinecommunicationspaces.Theparticipants

didnotusethehypertextfeaturesthatwereavailabletothem.Thisisakeystrategytoimproving

communicationinonlinespaces.(Burnett&Meyers,2006).Thediscussionboardtoolincludeda

commontexteditor,whichallowedstudentstobold,underline,anduselists.Itranscribedand

examinedstudentresponses(SeeTable5.8)andnooneusedtheeditortoincludehyperlinksto

sources.OnlyEthanincludedatypedURLwhenhewrote,“Thepaintingisrealandifyougoto

metmuseum.com.”Thislackofusingonlinewritingtoolsindicatesthatstudentusedsurfaceonly

textualfeaturestocommunicate(Burnett&Meyers,2006).

Evidencefromstudentverbalfeedbackalsosupportsthepatternthatstudentsdonotuse

textualfeaturestocommunicateideasinonlinespaces.Nostudentsvoicedanyattemptofusing

hypertextwhenpromptedbytheresearcher.Asstudentsweretypingtheirresponses,Iasked,

“Whydidyouwriteitthisway”or“Whydidyouformatyourresponsethisway.”Eachansweronly

referredtothecontentandnotthedesignoftheresponse.Notonerespondentmentionedtheuse

oftextualandmultimodalelementstoimprovetheirabilitytocommunicateinadigitalage

(Merchant,2007).

162

StageOneSummary

Stageoneanalysishighlightedgeneralthemesthataffectedoverallperformanceonthe

ORCAtaskregardlessoftherestrictednatureoftheinformationspaceandmayhelptoexplain

someofthequantitativefindings.ThefirstthemesuggestedthatoverallORCAscoreswererelated

toanabilitytostrategicallyassembletextsduringonlinereadingcomprehensiontasks..Thishelps

toilluminatethefindingsfromthequantitativeportionofthestudy.Firstmeanscoresonthe

ORCA‐MRwerelowerthanmeanscoresontheORCA‐LR.Thiscouldindicatethatnavigation

requiredwithinawebsitemadetasksmoredifficultforstudentswhocouldnotsuccessfullyuse

textassemblystrategies..Theresultsofthisqualitativestudymayindicatethatparticipantswith

greaternavigationalstrategiesmayhavescoredhigherinamorerestrictedinformationspace.The

observationsofthestudentswhoscoredthehighestontheORCAindicatethattheymademany

morenavigationalmovesthanthestudentswhoscoredinthelowestgroup.Thefirstthememay

alsoillustratewhycriticalevaluationscoreswerethestrongestpredictorsinbothmodels.The

criticalevaluationinstrumentrequiredstudentstomakenavigationalchoices.Thereforestudents

whoreadwithcursorcontrolmayhavedonewellonbothassessments.

Thesecondthemeappearedtoindicateanoveralllackofsourceevaluationknowledge.This

themeindicatedthatthemajorityofstudentswerenotpreparedtoevaluateonlineinformation.

Theresultmayexplainthelowmeanscoreof5.10onthe12‐itemassessmentwith9beingthe

highestscoreontheCriticalOnlineInformationLiteraciesassessment.Thepatternsofstrategyuse

aroundtheevaluationofonlinesourcesmayalsoexplainwhyscoresonthecriticalevaluation

measurewerethestrongestpredictorofonlinereadingcomprehensionscores.Itisclearbasedon

thequalitativeobservationsthatstudentsingenerallackedsourcingskills.Itisalsoevidentfrom

thequalitativedatathatcriticalevaluationiscentraltoonlinereadingcomprehension.Therefore

scoresofthestudentswhodidwellontheCOILmayhavebeencorrelatedwithahighscoreonthe

ORCA.ConverselylowCOILscoresmayhavebeencorrelatedwithalowscoreontheORCA.Finally

163

theillustrativeexamplesincludedinthequalitativeportionofthisstudydemonstratewhythe

evaluateitemswerethemostdifficultforstudentsinboththeORCA‐LRandtheORCA‐MR.Infactin

examiningthetenstudentsincludedinthestudytherewereonlytwoexamplesofstudentswho

successfullyjudgedauthorexpertiseandauthorpointofview.Ifthispatternheldtrueacrossallof

thequantitativeparticipants,theevaluateitemsoftheORCA‐LRandtheORCA‐MRwouldbethe

mostdifficult.

Thethirdthemeindicatedthattheabilitytorecalldetailswhenreadingmultiplesourcesor

takingapositionwasakeyindicatoronsuccessontheORCAtask.Studentsinthehighestgroup

reliedonastrategyofrecallingdetailsmorethanstudentsinthelowestgroup.Thereforethis

studymayprovideevidencethatmoresuccessfulonlinereadersengagein“flexibleassemblage”

(Deschryver&Spiro,2008p.15)ofknowledgebycreating“schemaatthemoment.”Thiswould

indicatethattheroleofbackgroundknowledgemaynotbeasimportantaswetransitionaway

fromaprintbasedworld.ThismaysimplybeduetotheInternetbeingtheworld’slargestexternal

storageofhumanknowledge.Inotherwordswhyremembersomethingifyoucan“GoogleIt”.

Thefourththemeindicatedthatstudentswereunpreparedtoengageinthecommunication

strategiesnecessaryforonlinereadingcomprehensioninacademicsettings.Thisresultisreflected

inthequantitativefindingsofthestudyinthatmeanscoresonthecommunicationitemswerethe

secondlowestonboththeORCA‐LR(1.55)andtheORCA‐MR(1.206).Itwasevidentfromthe

qualitativedatathatstudentsoftenfailedtoprovideevidencefromwhattheyhadread.Ifthis

patternheldtrueacrossallthequantitativeparticipantsitwouldhelpexplainthelow

communicationscores.Similarlyinthisstudystudentsdidnotfullyusetheaffordancesofonline

communicationspaces.Ifthispatternheldtrueinthequantitativedataitmighthelpexplainwhy

manystudentsdidnotreceivepointsforexplainingwheretheyfoundtheirinformation.

Thepurposeofstageoneanalysiswastoidentifydifferencesinstrategyuseacrossthe

ORCAtasksregardlessoftherestrictednatureoftheinformationspace.Thisanalysisexamined

164

differencesinstrategyusebasedonperformanceontheORCAThefollowingstrategieswere

utilizedbythehigherperformingstudents:

utilizing tabs to navigate between task and source;

using tabs to navigate between multiple sources;

using a strategy of skimming websites to identify key details;

engaged reading within a website;

navigating to a source to read while taking a position;

navigating to the source to copy and paste details;

identifying the author on an about us page;

evaluating author using effective markers of expertise;

checking evidence against a secondary source;

inferring an author’s point of view;

recalling details from memory when combining sources;

recalling details from memory when taking a position.

StageTwoThemes

Themesthatemergedfromstageone,highlighteddifferencesinstrategyusethataffected

overallORCAperformance.Thegoalofthestagetwoanalysiswastoexaminethedatainorderto

answerresearchquestionsthreeandfourtoseeifhighandlowperformingstudentsutilize

differentstrategiesinlessrestrictedinformationspacesandmorerestrictedinformationspaces.

Onceagainrecursive,analyticinductivemethods(Angrosino,&MaysdePerez,2000;Bogdan&

Bilken,2003)wereusedtomakeadditionalpassesthroughcodedvideosofstudentthinkaloudsto

identifypatterns.

Itwasevidentfromthisinitialanalysisthatonlyasingledifferenceappearedbetweenhigh

andlowperformingstudents,inrelationtothenatureoftheinformationspace,andthisappeared

onthelocatingitems.Thus,inthestagetwoanalysis,Ireturnedtothedatatolookfordifferent

165

patternsofstrategyuseonthelocatingtasks.Thisanalysis,usingrecursive,analyticinductive

methods(Angrosino,&MaysdePerez,2000;Bogdan&Biklen,2003)wasconductedinrelationto

researchquestionsthreeandfour:

ResearchQuestionThree:Whatpatternsofonlinereadingcomprehensionstrategies

appearamonghighandlowperformingonlinereadersduringanonlinereading

comprehensiontaskwithinalessrestrictedinformationspace?

ResearchQuestionFour:Whatpatternsofonlinereadingcomprehensionstrategiesappear

amonghighandlowperformingonlinereadersduringanonlinereadingcomprehension

taskwithinamorerestrictedinformationspace?

WhatPatternsofOnlineReadingComprehensionStrategiesAppearAmongHighandLow

PerformingOnlineReadersDuringanOnlineReadingComprehensionTaskWithinaLess

RestrictedInformationSpace?

Inthestagetwoanalysisitwasclearthattherestrictednatureoftheinformationspaceonly

affectedstrategyuseonthelocatingtasks.Thereweretwolessrestrictedtasksincludedinthe

ORCAassessment.Studentshadtocompleteoneofthetwoforthequalitativepartofthisstudy.

ThefirsttopicwastheroleofWomenintheAmericanRevolution(Women).Thesecondtopicwas

thehistoricalaccuracyofEmmanuelLeutze’spaintingofGeorgeWashingtoncrossingtheDelaware

(Delaware).Studentswereasked,forexampleto,“Findonewebsiteaboutthehistoricalaccuracy

ofEmmanuelLeutze’spaintingofGeorgeWashingtoncrossingtheDelaware.”Threepatterns

emergedwhenexaminingthedifferencesofstrategyuseamonghighandlowperformersonthe

lessrestrictedORCAtask:

1. StudentswhoperformedbetteronthelessrestrictedORCAusedmoreeffectivestrategies

forenteringkeywords.

2. StudentswhoperformedbetteronthelessrestrictedORCAusedmoreeffectivestrategies

forreadingsearchresults.

166

3. StudentswhoperformedbetteronthelessrestrictedORCAmademoreaccuraterelevancy

judgments.

Patternone:StudentswhoperformedbetteronthelessrestrictedORCAusedmore

effectivestrategiesforenteringkeywords.Acleardifferenceofstrategyuseamongstudents

whoweremoresuccessfulandlesssuccessfulonthelessrestrictedtaskappearedinthestrategies

usedtoenterkeywords.Threestrategieswereevidentinthispattern:(a)moresuccessfulstudents

includedboththetopicandfocusofthetaskwhentheyenteredkeywords;and(b)moresuccessful

studentsincludedboththetopicandfocuswhentheycopiedandpastedkeywords;(c)andsearch

enginefeaturesoftenimpededeffectivesearches.

Moresuccessfulstudentsincludedboththetopicandfocusofthetaskwhenthey

enteredkeywords.ThemajorityofstudentsontheORCAlessrestrictedtaskreliedonastrategyof

usingkeywordsfromthetask.ThepatternisdisplayedinTable5.9.Yetthestudentswhowere

successfulatkeywordentryincludedbothtopic(e.g.paintingofGeorgeWashingtonCrossingthe

Delaware)andfocus(historicallyaccurate)(Eagleton&Guinee,2002).Theinclusionofbothtopic

andfocusposedinthetaskwasamoreeffectivestrategyforlocatingwebsitesonthelessrestricted

task.

167

Table5.9FrequencyofStrategiesUsedtoDevelopKeywordsParticipantsgroupedbyORCAscores

Usekeywordsfromtask

Copyandpastewordsfromtask

UsesSearchEngineTools

Revisekeywords

IncludedTopicandFocus

High Isabella 1 0 2 0 0Olivia 3 0 0 0 3Ava 3 0 1 0 3Total 7 0 3 0 6 Medium Sophia 2 2 1 4 1Alexander 4 0 2 2 1Jacob 0 3 0 2 0Ethan 1 1 1 1 1Total 6 5 4 9 3 Low Emma 3 0 2 2 0Jaydan 3 0 0 1 1Michael 2 0 1 1 1Total 7 1 3 4 1

Successfulkeywordentryvariedsomewhatforparticipantsinthehighestgroupofscores.

Studentswhorememberedboththetopicandthefocus(Eagleton&Guinee,2002)demonstrated

moresuccessfulkeywordentry.Ava,ahighperformingstudent,forexample,begantheDelaware

taskbygoingtoGoogle.Shethentypedinthekeywords,“GeorgeWashingtoncrossingtheDelaware

accuracy.”Avathensaid,“Iaddedaccuracybecauseitmeanstruthful.”

Oliviawasalsosuccessfulonthekeywordtask.ItmustbenotedthatontheWomeninthe

AmericanRevolution,lessrestrictedtask,thereweremoresearchstringsthatwouldreturn

relevantsites.OnherfirstkeywordentryOliviaused,“WomenintheAmericanRevolution.”Onher

nexttwoqueriesOliviausedthephrase,“WomenandtheAmericanRevolutionWar.”

168

IsabelladidnotbeginhersearchstringswitheffectivekeywordsonthePaintingtask.On

herfirstsearchIsabellaused“EmanualeGottliebLeutzepictureonGeorgeWashington.”Then

Isabellaused,EmanuelGottliebLeutzePaintingofGeorgeWashington.Finally,whenlookingforthe

thirdwebsite,Isabella,usedthesearchterm,“PaintingofGeorgeWashingtonbyEmmanuel.”I

askedherwhatshewasthinkingandIsabellasaid,“Iamlookingforinfoonthepainting.”Whileshe

mentionedthetopicIsabelladidnotmakereferencetothefocusofthetask.

Nostudentsinthemiddlegroupbegantheirinitialsearchstringwithboththetopicandthe

focus.AlexandersearchedjustforthetopicontheDelawaretask.Forexample,Alexanderbeganhis

firstsearchlookingfor“GeorgeWashingtonCrossingtheDelaware.”Thisloadedwebsiteswithonly

historicalinformation.Alexanderwasunabletofindrelevantwebsitesusinghiskeywords.

StudentsinthelowestgroupalsostruggledwithkeywordentryontheDelawaretaskbut

wereabletoenterkeywordsontheWomenintheRevolutionTask.Ethanbeganbytyping“George

WashingtonCrossingtheDelaware.”Theauto‐fillsuggestionsgavehimresultsfor“George

WashingtonCrossingDelawareRiverhistoricallyaccurate.”Ethanthendeletedthewords:

historicallyaccurate.Hisnexttwosearcheswereavariationoftheartistsnameandthepainting.

Jaydan,lookedforinformationusingthesearchterms“thepaintingofGeorgeWashington

crossingtheDelaware.”Thisreturnedsearchresultsabouttheactualpainting.Websitesthat

discussedthehistoricalaccuracyofthepaintingwerenotinthetopsearchresults.

OntheWomentaskthelowperformingstudentsfairedbetter.Emmabeganwiththekey

words“therolewomenplayedintheAmericanRevolution.”Sheincludedthetopic“womeninthe

AmericanRevolution”andthefocus“role.”Onhernexttwosearchessheused,“womenandthe

AmericanRevolution.”Michaelalsobeganwith“womenoftheAmericanrevolution”andfollowed

thisupwith“womenintheAmericanRevolution.”

Moresuccessfulstudentsincludedboththetopicandfocuswhentheycopiedand

pastedkeywords.Onlythreestudentsusedastrategyofcopyandpastesoitishardtodraw

169

patternsaboutthisstrategyuse.Noneofthehighestperformingstudentsusedthiskeyword

strategy.However,Sophia,amid‐levelstudent,includedthetopicandfocusinthetask.Sophiaused

acopypastestrategytoentertheauthor’snamebutthentypedtherestofherkeywords.Sherelied

oncopyandpastethroughoutthetasksforhardspellingssuchasnamesandcities.Sheincluded

boththetopicandfocusinherfinaltaskandsucceededonfindingawebsiteforthesecondsearch

item.

Studentswhocopiedandpastedkeywordswithoutthetopicandfocusdidnotsucceed.

Jacobforexample,ontheDelawaretaskonlysearchedfor“EmanuelGottliebLeutze’sGeorge

WashingtonCrossingtheDelaware.”Relevantsearchresultswerefurtherdownthelist.Ethanalso

wasunsuccessfulatusingacopyandpastestrategy.Hecopiedandpasted,EmanuelGottlieb

Leutze’spaintingofGeorgeWashington.

Searchenginefeaturesoftenimpededeffectivesearches.Therewasgreateruseofsearch

toolsbuiltintothesearchengineonthelessrestrictedtask.Yettherelianceonthesearchengine

toolsactuallyhinderedstudentssearchingabilitiesratherthanhelptoscaffoldtheirInternet

inquiries.

Isabellausedtheauto‐fillfeature.Auto‐filltriestopredictsearchterms.Itgivesyouadrop

downlistofoptionsandfillsinthesearchbarwiththe“bestprediction.”Shetyped“George

Washingtoncrossing…”andthenchose“GeorgeWashingtoncrossingdelawareriver”fromthe

suggestedsearchers.Therewerebetterkeywordssuggestedbyauto‐fill.Thesesearchterms

includedboththetopicandthefocus.Isabelladidnotusethese.Shepickedthefirstauto‐fill

suggestion.OnthenextsearchIsabellatyped,“GeorgeWashington…”andselected“George

Washingtonpainting”Shethentyped“de…”andselected“GeorgeWashingtonpaintingdelaware

river.”

Alexanderusedtheauto‐filltofinishhiskeywords.Thesekeywords,however,includedonly

thetopicandnotthefocus.Alexanderbeganbytyping,“GeorgeWashingtoncr…”hethenletthe

170

auto‐fillfinishtherest.Alexandersearchedfor“GeorgeWashingtonCrossingtheDelawareRiver.”

OnhisthirdsearchtaskAlexanderagainreliedontheauto‐fill.Hetyped“GeorgeWashington

crossingDelawareriver.”thenselected“GeorgeWashingtoncrossingtheDelawareriverfacts”from

theauto‐fillsuggestedsearches.

Sophiaalsousedtheautofill.Shetyped,“GeorgeWashingtoncrossing.”Shethenselected

“GeorgeWashingtoncrossingthedelawarepainting.”Thesuggestedsearchtermsdidnotincluded

anyinformationabouttheaccuracyofthepainting.

Michaelalsousedtheauto‐fillfeature.Hebeganbytyping“womenoftheam..”Hethen

selectedthesuggestedsearchtermof,“womenoftheAmericanrevolution.”Onceheclickedonthe

searchtermhewasbroughttotheresultpage.

Asecondsearchenginefeaturethatstudentsreliedonwasthesuggestedspelling.There

werenotenoughinstancestodrawclearpatterns,butonceagainthesuggestedspellingoften

servedasahindranceratherthanasupportivescaffoldsearchingtheInternet.

Emma,asecondlanguagestudent,usedthesuggestedspellingfeature.Sheoriginally

searchedfor,“therolewomenplayedintheAmericanrevolution.Googledisplayedresultsforthe

correctspellingbutgavelinkstosearchesusingtheincorrectandthecorrectspelling.Emmachose

theincorrectspellinglink.Thepagethatloadeddisplayedalinktothesearchresultswiththe

correctspelling.Emmaeventuallypickedthecorrectlink.

Ava,ahighperformingstudent,wastheonlystudenttousethesearchenginefeatures

correctly.AvausedthecorrectedspellingtoolsbuiltintotheGooglesearchengine.Sheforgota

spacebetweenDelawareandpainting.Googledisplayedtheresultsforthecorrectspellingbut

providedalinktothesearchresultsforboththecorrectandincorrectspelling.Avaclickedonthe

linkforthecorrectspelling.

PatternTwo:StudentswhoperformedbetteronthelessrestrictedORCAusedmore

effectivestrategiesforreadingsearchresults.Thesecondpatternofstrategyuseamong

171

studentsmoresuccessfulonthelessrestrictedORCAwastheuseofmoreeffectivestrategiesfor

readingsearchresults.Twopatternsofstrategyusewerenoted:(a)activelyreadingsearchresults;

and(b)navigatingbacktosearchresultsonthelessrestrictedtasks.

Activelyreadingsearchresults.Thequalitativeresultssuggestthathowstudentsread

searchresultsmaybeoneofthemoredefiningindicatorsoftheirperformanceontheonline

readingcomprehensionassessmentsinlessrestrictedinformationspaces.Thesepatternsare

displayedintable5.10,whichshowsthefrequencyofsearchenginereadingstrategiesnoted.

Overallstudentsinthehighestandmiddleperforminggroupsactivelyreadsearchresults,returned

tothesearchresults,andclickedonmorelinks.

Table5.10FrequencyofStrategiestoReadSearchEngineResults

ParticipantsgroupedbyORCAscores

Cursormovementorverbalizationindicatesreadingofresults.

Clicksonfirstlinkwithoutskimming

Movestosecondpage

Clicksonasearchresult

HighIsabella 2 0 0 3Olivia 6 0 0 8Ava 2 0 0 2TOTAL 10 0 0 13

MiddleSophia 5 0 0 7Alexander 5 0 0 5Jacob 1 1 0 7Ethan 3 0 0 0TOTAL 14 1 0 19 LowEmma 3 0 0 3Jaydan 1 1 0 3Michael 2 0 0 3TOTAL 6 1 0 9

172

Theactivereadingofsearchresultswasdefinedasreadingmorethanthefirstoronelink.It

requiredeitheraverbalizationorcursormovement.Thisstrategywasmoreprevalentamongthe

highestperformingparticipants.Olivia,forexample,ontheWomentaskbeganbyenteringinthe

searchterms,“womenandtheAmericanrevolution.”Shethenspent39secondsreadingthesearch

results.Iasked,“Whatareyouthinking?”Oliviahoveredoverasearchresultfor“AmericanAthena”

andsaid,“Ithinkthisisitbecauseitsaysamericanrevolution.org.

OnthenextsearchtaskOliviausedthesamekeywords.IagainaskedOliviawhatshewas

thinking.Oliviareplied,referringtothesearchresults,“Iamjustgoingthrough,like,andreading

thelittledescription,kinda,underneath.”After30secondssheclickedonasearchresult.

Avaalsospentaconsiderableamountoftimethinkingaboutthesearchresultsshewould

clickon.OntheDelawaretaskshebeganbyenteringthekeywords,“georgewashingtonpainting

crossingthedelawareaccuracy.”Hercursorhoveredoverthefirstthreesearchresultsmovingleft

torightassheread.Avathenclickedonasearchresulttoawebpagetitled,“DidGeorge

WashingtonreallystandupinhisboatwhencrossingtheDelaware”onthewebsite“ushistory.org.”

Iasked,“Whydidyouclickonthatone?”Avaexplains,“umm.Itlookedlikethemostrelevantto

whatItypedin.”

Studentsinthelowestperforminggroupsdiddisplaysomerudimentaryreadingofsearch

results,butdidnotthinkaboutthelinkstheyclickedon.Emmaforexampleclickedonasuggested

spellinglinkandexpectedittotakehertoawebsite.Whenaskedaboutwhysheclickedonthelink

Emmasaid,“becauseithadlikeeverythingItyped.”ThenEmmamoveddownthepagewithher

cursor.ShefocusedonanotherlinktoanirrelevantwebsitebecausetheURLendedin.org.Emma

didnotclickthelink.Shethenclickedonthecorrectspellingandsaid,“IamnotsureifIshould

clickonthisbecauseitisWikipedia...wellsometimespeopleputtheirownopinions.Iheardthatthe

173

guywhocreatedWikipediawentthroughandchangedeverything.”Emmathenclickedon

Wikipediaanduseditashersource.

Jacobmovedquicklythroughsearchresultswithoutactivelyreadingthem.Hebeganby

copyingandpasting,“EmanuelGottliebLetutze’spaintingofGeorgeWashingtoncrossingthe

Delaware”fromthetask.Hethenclickedsearchandimmediatelyclickedonthefirstlinkinthe

results,readingonlythewebsitetitleoutloud.Jacobdidrealizethattheresultwasirrelevantand

didnotchooseabettersite.

OnthenextsearchtaskJacobusedthesamekeywords.Onceonthesearchresultspagehe

quicklysaid,“Thisseemslikeagoodone,”andclickedonthefourthsearchresult.Iasked,“Which

didyoupick?”andJacobreplied,“Globalwholesaleart.”Hedidnotmaketheconnectionfromthe

searchresultsthatthiswasacommercialwebsite.Onhisnextclickhealsomadethesamemistake

andjustsaid,“Iwilltrythisone,”andJacobclickedon“allposters.com.”Itisapparentthathewas

onaclickandhuntmission.

Michaelreliedonlyonthetitleofthelinkstochoosethesearchresultstoclickon.Hedid

notreadthedescriptionunderthelinkortheURL.OntheWomentaskMichaelclickedthesecond

linkonthesearchresultspage.Hesaidhechoseit,“becauseitsaidwomenintheAmerican

Revolution.”Heleftthepageandthenextlinkhepickedwasalsosolelybasedonthetitle.

Emmaalsodidnotspendtoomuchtimejudgingtherelevancyofhersources.Onthe

WomentasksheclickedonasearchresultfortheWikipediaarticleonthetopic.Sheskimmedthe

articleanddecided,“Ithasinformationonthetopic.”Emmarepeatedthepatternwhensearching

forhernexttwosources.Sheautomaticallywentwiththefirstsourcesheclickedon.Itmustbe

notedthattheseresultswererelevanttothetask.Emma,however,unlikehermoresuccessful

peers,didnotcomparesourcesbeforechoosingoneshebelievedtoberelevant.

Navigatingbacktosearchresultsonthelessrestrictedtasks.Therewasadifferencein

thenumberoftimesstudentsreturnedtosearchresultsontheORCAlessrestrictedtaskswith

174

cursorcontrol.ThefrequencyofthesenavigationstrategiesaredisplayedinTable5.11.Studentsin

thetwohighestperforminggroupsfrequentlyreturnedtotheirsearchresultstofindabetter

source,usingthecursor.Onlyonestudentinthelowgroupreturnedtosearchresults.

Table5.11FrequencyofNavigationstrategieswhenreadingsearchenginesduringthelessrestrictedtaskParticipantsgroupedbyORCAscores

Cursormovementverbalizationindicatesreadingofresults.

Returnstosearchresults

Isabella 2 0Olivia 6 4Ava 2 0TOTAL 10 4

Sophia 5 4Alexander 5 3Jacob 1 4Ethan 3 0TOTAL 14 11 Emma 3 0Jaydan 1 0Michael 2 1TOTAL 6 1

AvaandIsabella,bothhighperformers,didnotreturntotheirsearchresultsontheless

restrictedtask.This,however,wasaresultoftheirgoodkeywordentryandtheircarefulreadingof

searchresults.Ava,forexample,alwaysincludedtheword“accuracy”inherkeywordsonthe

Delawaretask.Shealsocarefullyreadsearchresultsbeforeselectingalinktoclickon.

Oliviawasverymethodologicalinreturningtosearchresults.Aftershereadawebsiteshe

wouldgobacktothesearchresultstomakesuretherewasnotabetteroption.Forexampleduring

theWomentask,afterspendingafewminutesonasiteOlivialeftbecauseit,“Itdidn’thavethat

175

muchinformation.Othersiteshadlikepicturesandstuff.”OnhernextsiteOlivialeftbecausethe

website,“Didn’treallytalkaboutwomenANDtherevolution.”

Oneofthecleardifferencesbetweenstudentsinthemiddlegroupandstudentsinthe

lowestgroupwasnavigatingbacktosearchresults.Alexanderreturnedtohissearchresultsonhis

thirdsearchtask.Thefirstwebsitehevisitedwasaprintcompany.AsheleftAlexandersaid,“This

oneismostlylikeselling.Sobackoutofthat”Hereturnedtothesearchresultsandsaid,“Clickon

theoneaboveitbecauseitsaysWashingtoncrossingtheDelawarebythatname”(referringtothe

artist).

Sophiaalsoreturnedtohersearchresults.AfterenteringherkeywordsontheDelaware

task(missinganyclaimsaboutaccuracy)Sophiafirstclickedonalinktoamuseumsite.Asshehit

thebackbuttonSophiasaid,“Itjusttalksabouthowprettythepictureis.”

Jacobreturnedtohissearchresultsmorethanonce.AsJacobprogressedhissearchterms

improved.BythethirdsearchtaskJacobwasincludingthewordaccuracyinhiskeywords.After

readingamuseumwebsiteJacobleftaftersaying,“Itdoesn’thavemuchonaccuracy.”

InthelowestgrouponlyMichaelreturnedtohissearchresultsonthelessrestrictedtask.

Hedidnotverbalizehisdecision.Hejustwentbacktothesearchresultsaftervisitingarelevant

pagethatwouldhaveworkedforthetask.

Patternthree:StudentswhoperformedbetterontheORCAlessrestrictedtaskmore

frequentlywereabletodeterminethatwebsiteswereirrelevant.Successfulstudentswere

abletodeterminethatwebsiteswereirrelevant.ThispatternisdetailedinTable5.12.Twomajor

strategieswerenotedinthelocatingofrelevantwebsites:(a)Studentswhosuccessfullymade

relevancyjudgmentsduringthelessrestrictedORCAskimmedwebsites;and(b)Studentswho

choseirrelevantwebsitesreliedonastrategyofchoosingthefirstlink.

176

Table5.12FrequencyofRelevancyJudgmentsonlessrestrictedtasksParticipantsgroupedbyORCAscores

Locateswebsiteandjudgesitrelevant

Locateswebsiteandjudgesitirrelevant

Isabella 1 0Ava 3 0Olivia 4 4TOTAL 8 4 Sophia 2 4Alexander 1 5Jacob 3 5Ethan 2 1TOTAL 8 15 Emma 3 0Jaydan 3 0Michael 2 2TOTAL 8 2

StudentswhosuccessfullymaderelevancyjudgmentsduringthelessrestrictedORCA

skimmedwebsites.Oliviaisagreatexampleofastudentwhospenttimeconsideringtherelevancy

ofeachwebsitesheread.ShebegantheWomentaskbyfirstclickingonawebsiteentitled

“AmericanRevolution.”WhensheleftthesiteIaskedher,“Whatareyouthinking?”Sherepliedthat

thesite“…didn’thavealotofinformation,likeotherwebsites.Thathad,like,picturesandfacts.”

Oliviathenclickedonanotherwebsitethatlistedlinkstospecificbiographicalinformationabout

famouswomenoftherevolution.SheleftthewebsiteandIasked,“Whydidn’tyoupickthat

website?”Oliviarespondedthatitdidn’treallytalkabout,“…womenANDtheAmericanrevolution.

Itjusttalksaboutwomenwhofoughtinit.”Oliviathenwentbackandrereadthetask.Shethen

lookedattwomoresources,anddecidedafterallthatthewebsiteshefirstclickedonwasthemost

relevant.

177

Sophiaalsospentconsiderabletimeconsideringtherelevancyofhersourcesonthe

Delawaretask.ShebeganbyfirstclickingontheWikipediaarticleaboutthebattle.Sophiaquickly

returnedtohersearchresultsandchangedthekeywords.IaskedSophiawhysheleftthepageand

shestated,“Idon’tthinkithadtherightinformation.”Sophiathenskimmedoverthesearchresult

andclickedonthefirstlinkbecauseshesaid,“itcamefromamuseum.”Sophiathensaid,“Itjust

talksabouthowbigthepictureis,”andshereturnedbacktothesearchresults.Sophiarejectedthe

museumwebpageasirrelevantonceshereadit.Sophiathenclickedonawebsitetitled,“Whatis

wrongwiththispainting?”Sophiapickedthissourceasbeingrelevanttothetopic.Byexamining

threedifferentsourcesSophiamaderelevancyjudgmentsuntilshelocatedausefulsource.

Studentswhochoseirrelevantwebsitesreliedonastrategyofchoosingthefirstlink.On

theotherhandJaydanrepresentsthetypeofreaderwhodidnotjudgetherelevancyofsourcesand

quicklychoseawebsitefromtheirsearchresults.Thispatternwascommoninthelowestgroupof

performers.Jaydanatfirstdidnotlikethesearchresultshegotfor“GeorgeWashingtonCrossing

theDelaware”andaddedtothekeywords“wasitaccurate.”Jaydanthenclickedonafirstlinktoa

Wikipediaarticleaboutthecrossing.BeforeJaydanevenskimmedthearticlehecopiedandpasted

thelinksintothetask.Jaydanrepeatedthispatternonthenexttwosearchtasks.Healways

acceptedthefirstwebsiteheclickedonasrelevant.

Emma,alsointhelowestgroup,neverjudgedanywebsiteasirrelevantontheless

restrictedtask.OntheWomentaskEmmaalwayswentwithherfirstclick.Sheneverreturnedto

thesearchresultsafterreadingawebsite.

WhatPatternsofOnlineReadingComprehensionStrategiesAppearAmongHighandLow

PerformingOnlineReadersDuringAnOnlineReadingComprehensionTaskwithinaMore

RestrictedInformationSpace?

Stagetwoanalysisrevealedthattheonlydifferencebetweenstrategypatternsappearedin

thelocatingtasks.Thereweretwo,morerestrictedtasksincludedintheORCA.Studentsinthe

178

qualitativeportioncompletedoneofthetwo.Thetwotopicswere:a)thecausesoftheAmerican

Revolution(Causes)andb)theTurningPointoftheAmericanRevolution(TurningPoint).

OntheCausestaskstudentshadtofindthreedifferentwebsites.Thefirsttaskasked

studentsto,“FindthewebpageCausesofRevolutionaryWaronthewebsite

SocialStudiesforKids.com.”Thesecondtaskasked,“FindthewebpageListofCausesoftheWaronthe

websitehistorycentral.com.”Thefinallocatingtaskaskedstudentsto,“Findthewebsitetitled

AmericanRevolutionPersuaders.”

StudentsalsohadtofindthreewebsitesfortheTurningPointrestrictedtask.Thefirsttask

said,“CandacepostedthequestiontothewebsiteYahooAnswers,‘WhatistheTurningPointofthe

AmericanRevolution?’Locatethisdiscussion.”Thedirectionsforfindingthesecondwebsiteasked,

“FindthearticleabouttheBattleofBenningtononthewebsiteTheAmericanRevolution.org.”Onthe

finalsearchtaskstudentswereaskedtofindthearticlesabouttheBattleofSaratogaonthewebsite

SocialStudiesForKids.com.”

Stagetwoanalysisfoundthatthepatternofstrategyusetolocatewebsitesdifferedonthe

morerestrictedtasks.Therestrictednatureofthetask,thatishavingstudentslookforaspecific

source,reducedthenumberofinstancesofkeyworduse.Insteadmanyofthestudentswent

directlytoawebsiteandsearchedwithinthesitefortherequiredinformation.Threepatterns

emergedintheanalysisofthemorerestrictedtask:

1. StudentswhoperformedbetteronthemorerestrictedORCAusedmoreeffectivestrategies

tosearchwithinawebsite.

2. StudentswhoperformedbetteronthemorerestrictedORCAusedkeywordsincludingboth

thetopicandthesource.

3. StudentswhoperformedbetteronthemorerestrictedORCAmademoreaccuraterelevancy

judgments.

179

Patternone:StudentswhoperformedbetteronthemorerestrictedORCAusedmore

effectivestrategiestosearchwithinawebsite.Studentswhoweresuccessfulonlocating

informationduringthemorerestrictedtaskdemonstratedtheabilitytosearchwithinawebpage.

Oftenthisrequiredbetterforwardinferencing(Coiro&Dobler,2007),orpredictingwherelinks

willtakeyou,whenstudentssearchedwithinawebsite.Thestudentsinthehighestperforming

groupsweremoreadeptatlocatingapagewithinawebsite.Studentsinthelowestgroupoften

struggledtofindawebpagewithinasite.ThesepatternsareillustratedinTable5.13.

Table5.13FrequencyofLocatingStrategiesontheMoreRestrictedORCA

Navigatesdirectlytothesource

Searchessuccessfullywithinawebsiteforasource

Usesbothtopicandsourceinkeywords

Copiesandpastesexactwordsfromtask

UsesSearchEngineTools

Reviseskeywords

High Olivia 2 3 4 0 1 0Ava 2 2 1 0 0 0Isabella 2 1 1 4 1 2TOTAL 6 6 6 4 2 2 Medium Sophia 2 2 1 0 1 0Alexander 1 1 0 1 0 0Jacob 0 2 1 0 1 0Ethan 0 1 0 0 1 0Total 3 6 2 1 3 0 Low Emma 0 0 0 0 0 2Jaydan 0 0 0 0 1 0Michael 0 0 1 0 2 0TOTAL 0 0 1 0 3 2

180

Studentsinthehighestperforminggrouptypicallywentdirectlytothewebsitebytyping

theURLintothenavigationbarofthebrowser.Theythensearchedwithinthewebsite.Oliviabegan

bytypingtheURLtosocialstudiesforkids.cominthenavigationbar.Shethenclickedbackonthe

task.Oliviaclickedbackonhersourceandscrolledupanddownthepageandsaid,“Iamtryingto

thinkofwhereitwouldbe.”Sheclickedon“USatWar”intheleftmenubar.NextOliviaclickedon

“RevolutionaryWar.”Shescrolledupanddownthepage,andclickedonthe“2”atthebottomofthe

pagetogotothenextpage.NextOliviaclickedbackandsaid,“Ireadallthedescriptionsandthisis

best.”SheclickedonalinkabouttheAmericanRevolutionandfoundthelinktothecorrectpage

thatexplainedthecausesoftheRevolutionaryWar.

Avaalsosearchedwithinapagebymakingforwardinferencesaboutlinks.Shestartedby

firstsearchingjustforthesourceusingGoogle.OnceshefoundthesourceAvasearchedwithinthe

source.Avasearchedforthespecifiedwebsite,SocialStudiesForKids,inGoogleratherthanthe

webpagethatlistedcausesoftheAmericanRevolution.OnceonthewebsiteAvascrolleddownthe

homepageandclickedonalinkintheleftnavigationbartitled“U.S.atWar.”Shethenscrolleddown

andclickedonthe“AmericanRevolution.”Shethenreadthepageanddidnotseealinkto“causes

ofthewar”inthelefttoolbar.Shedidnotfindtheexactlinkbutwasabletoinferthecausesofthe

waronthewebsite.

OnhernextsearchtaskAvaenteredintheURLaddressdirectlyintothenavigationbar.She

waslookingforthewebpageonthewebsiteAmericanRevolution.org.OnceshewasonthepageAva

movedhercursorthroughthesidenavigationbar.Shefoundthecorrectlinkandsaid,“Thisisit.”

Studentsinthemiddlegroupweresomewhatsuccessfulatsearchingwithinawebsite.They

performedinasimilarfashiontotheirpeerswhohadhigheronlinereadingcomprehensionability.

AlexanderforexamplemadeatotalofnineclicksbeforehelocatedthecorrectpageontheCauses

task.HeenteredtheURLtosocialstudiesforkids.cominthenavigationbar.Alexanderthensaid,“I

amgoingtolookaroundonthistab.ItsayslinksandstuffsoIamgoingtogotohistory.”Hethen

181

scrolledupanddownandsaid,“Iamgoingtolookforcausesoftherevolutionorcausesofthe

AmericanRevolution.”Alexanderthenclickedon,“USGovernment”andscrolledupanddownthe

page.Hethenclickedbackonthetaskandsaid,“Iamlookingforcausesoftherevolutionarywar.

Hethenclickedon“warsaroundtheworld.”Alexanderscrolledupanddown;heclickedonalink

forasecondpageofresults.ThelinktotheAmericanRevolutionwasatthetopofthepage.

Alexanderclickedonit.Hescrolledupanddownthepageandsaid,“MaybeifIclickontimeline.”

Heclickedonthelink,scrolledupanddownthepage,anddidnotfindthewebpage.Hethensaid,

“I’llgoback…,”andheclickedthebackarrow.AshehoveredoveralinkAlexanderadded,“…and

maybethisandfindinformationinit.”HeclickedonalinktotheAmericanRevolution.Thisbrought

Alexandertoapageoffofthewebsite.Hesaid,“Notit,”andclickedthebackarrow.Hemousedover

anadditionallinkandsaid,“Thatonelookslikeaspecificperson.SoIamgoingtodothatone.”He

clickedonalinkabouttheRevolutionandthenclickedonthelinktotherequiredwebpagethatwas

listedontherightofthepageinamenuframe.

Sophiawasabletolocatethespecifiedwebsite.Shewaslookingforthewebpage“Causesof

War”onthewebsitehistorycentral.com.Shecopiedandpastedhistorycentral.comintothe

navigationbar.Shesaid,“OnthewebsiteIgotIamjustgoingtolookforListofCauses.”Sophia

scrolledupanddownthepageandthenfoundthelinktoAmericanRevolutioninthelefttoolbar.A

popupwindowofferingafreeiPadopenedup.Sophiawasabletocloseitwithoutclickingonanyof

thehiddenlinks.SheskimmedthepageandclickedonthelinktoCausesoftheWar.

Jacobfoundtwoofhisthreesourcesbysearchingwithinapage.OntheCausestaskhe

foundthewebpageonsocialstudiesforkids.combyusingtheinternalsearchengine.Hewasthe

onlystudentinthestudytoutilizethisstrategy.Ashewasskimmingthepagehesaid,“Iamgoing

totrysearchinginthisthing.”Hethenmovedhiscursortothesearchbar.

Thestudentsinthelowestgroupdidnotperforminasimilarfashionasstudentsinthe

middleandthehighgroup.Theirsearchstrategiesweremorelimitedandlesssuccessful.No

182

studentfoundthepageswithinawebsite.Twostudentsacceptedthehomepageofawebsiteasthe

correctpagetheywerelookingfor.Onestudenthadtobedirectedtothesourcesbyme.

Jaydanwasunabletofindthecorrectwebsites.Hesubmittedthehomepagesasthecorrect

sites.Whenhegottothesynthesissitehesaid,“Ican’tfindit.Maybehistory.”AfterafewminutesI

foundbothwebsitesforhimusingtheinternalsearchengine.

Emmawasunabletofindanyofthetargetedwebsites.Shesearchedfortwowebpages

withinawebsite.OntheCausestasksheenteredtheURLforsocialstudiesforkids.comandthe

submittedthehomepageURLasananswer.ThenshemovedontotheAmericanRevolution.orgtask

andonceagainsubmittedthehomepageasheranswertothelocatetask.WhenEmmamovedonto

thesynthesistasksheclickedonsocialstudiesforkids.com.Shethenclickedonalinkto“How

Presidentsgetelected”andsaid,“UmmIdon’tknowwhattodo.”Ifoundthetworequiredpagesfor

her.

Patterntwo:StudentswhoperformedbetteronthemorerestrictedORCAused

keywordsincludingboththetopicandthesource.Higherperformingandmiddlelevelstudents

typicallyusedtheURLtolocateinformationonthemorerestrictedtask.Studentsinthelowest

performinggroupsmoreoftenusedasearchengine.Whentheydid,thosewhosearchedforboth

thetitleofthewebpageandthesourceweresuccessfulatthetask.Isabella,forexample,onthe

TurningPointtask,putthekeywords“AmericanRevolution.orgbattleofbennington”inGoogle.She

wasabletofindtherequiredsource.

Middleandlowperformingstudentswhosearchedforthetitleofwebsiteswithoutthe

sourcenamewerelesssuccessfulonthetask.Sophia,forexample,usedthewords,“TurningPoint

oftheAmericanRevolution.”However,shedidnotmentionthespecificsource,YahooAnswersthat

studentsweretaskedtofind.Whilethecorrectwebsiteappearedonthefirstpageoftheresultsit

wasfurtherdowninthesearchresults.

183

Emmaalsoneverincludedanyinformationaboutthesourceinheranswer.Sheenteredin

thekeywords“listofcauses”whenlookingforawebsitefortheCausestask.Shethenspenttime

readingtheresultssaying,“Iamjusttryingtofigureoutwhichitwouldbe.Emmaneverincluded

anyinformationaboutthesource,andneverfoundthecorrectwebsite.

Patternthree:StudentswhoperformedbetteronthemorerestrictedORCAmade

moreaccuraterelevancyjudgments.Studentswhoweremoresuccessfulontherestrictedtask

weremoreabletojudgewebsitestheyvisitedasirrelevant.Thisstrategywasdefinedasnavigating

toawebsiteorwebpageandthenleavingafterdeterminingitwasnotuseful.Studentsinboththe

highandmediumgroupsutilizedthisstrategy.Nostudentsinthelowestthirdjudgedawebsite

theyvisitedasirrelevant.ThepatternisexploredinTable5.14.

184

Table5.14FrequencyofStrategyUseonMoreRestrictedLocatingTasksParticipantsgroupedbyORCAscores

Locateswebsiteandjudgesitirrelevant

LocateswebsitebytypingURLinNavigationbar

Locatesspecifiedwebpagewithinawebsite

Isabella 2 0 1Olivia 0 2 2Ava 1 0 1TOTAL 3 2 4 Sophia 0 0 2Alexander 1 4 2Jacob 0 4 2Ethan 0 0 0TOTAL 1 8 6 Emma 0 2 1Jaydan 0 3 0Michael 0 1 1TOTAL 0 6 2

IsabellaontheTurningPointtaskcouldjudgeawebsiteirrelevant.Whileshewas

searchingfortheYahooanswersdiscussionsheclickedthrougheachwebsiteandjudgedthemto

beirrelevant.Atonepointtheresearchstoppedandaskedher,“Whydidyoupickthatone”when

sheclickedonanirrelevantwebsite,“Itlookedokay,”sheresponded,referringtothesearchresults,

“butitwasashop.”ThewebsitesoldhistoryDVDs.EventhoughIsabelladidnotfindthespecified

websiteshedidexhibitbetterstrategicreadingintermsofjudgingwebsiterelevancy.

OntheotherhandstudentswhowerenotassuccessfulontheORCAtaskshadatendencyto

acceptthefirstlinktheychosefromthesearchresultsortocopyandpasteaURLintothe

navigationbarasthelegitimatesourcethetaskaskthemtofind.ForexampleEthan,whenlooking

fortheYahoodiscussionboard,firstclickedonanarticleonWikipediaabouttheAmerican

185

Revolution.Hesays,“ThishassomestuffIamlookingfor,”andhecopiedthelinktothe

SurveyMonkeytask.LaterinthetaskwhenEthanwaslookingfortheAmericanPersuaderswebsite

heclickedonawebsitesellingstudentessays.Seeingthetitleofthearticlehecopiedandpasted

andsubmittedtheURLascorrect.

StageTwoSummary

Thepurposeofstagetwoanalysiswastoexaminethedatatoseeifthenatureofthe

informationspaceaffectedpatternsofstrategyuseamonghighandlowperformersontheORCA

assessment.Initialanalysisdeterminedthatthenatureoftheinformationspaceonlyaffected

strategyuseonthelocatingtasks.Thereforestagetwoanalysisexaminedifferencesinstrategyuse

onthelocatingtasksinthelessrestrictedspace.Thenstagetwoanalysisexamineddifferencesin

locatingstrategyuseamorerestrictedinformationspace.Thesedifferenceswereanalyzedby

comparingstudentsontheirperformancelevel.

Onthelessrestrictedtasksstudentswhoperformedbetterthantheirpeersusedspecific

strategies.Firsttheyweremoreaptatdevelopingkeywords.Thesekeywordsincludedboththe

topicandthefocus.Theyalsoreadsearchresults.Finallymoreproficientstudentsmademore

accuraterelevancyjudgmentsinthelessrestrictedenvironment.Thefollowingstrategieswere

notedamongbetterperformingstudentsinthelessrestrictedlocatingtasks:

using keyword that included both the topic and focus;

copying and pasting keywords that include both the topic and focus;

actively reading search results;

navigating back to search results;

skimming websites to make accurate relevancy judgment.

OnthemorerestrictedORCAtasksstudentswhoperformedbetterthantheirpeersused

specificstrategies.Moresuccessfulstudentsgenerallysearchedwithinawebsiteratherthanuseda

searchengine.Whenstudentsdiduseasearchengine,thosewhoincludedatopicandsource

186

performedwell.Similartothelessrestrictedtask,studentswhomademorerelevancyjudgments

onthemorerestrictedtaskoutscoredtheirpeersontheORCAassessments.Thefollowing

strategieswerenotedamongbetterperformingstudentsinthemorerestrictedlocatingtasks:

navigating directly to a source;

searching for a specific source;

searching within a website;

using keywords with topic and source;

making accurate relevancy judgments.

Thestagetwofindingsalsoilluminatedthequantitativefindingsofthisstudy.Specifically

thestagetwothemesmayexplaindifferencesinscoresontheontheORCA‐LR(8.394)andthe

ORCA‐MR(7.15).Stagetwoanalysisrevealedthatmoreproficientusersutilizedverydifferent

strategiestosearchwithinasourceonthemorerestrictedtask.Onthelessrestrictedtaskboth

studentsinthehighandmiddlegroupsearchedforwebsitesusingeffectivekeywords.Thefactthat

thelocatingtasksonthemorerestrictedORCArequiredspecificskillsonlyobservedinthemost

proficientreadercouldhavecontributedtodifferencesinthemeanscoresforthesixlocating

scorepointsontheORCA‐LRwas3.599.Themeanscoreforthesixlocatingscorepointsonthe

ORCA‐MRwas2.755.

Furthermorethestagetworesultsmayexplainwhyscoresontheself‐reportdispositions

measureweresignificantintheORCA‐MRmodelbutnotintheORCA‐LRmodel.Basedonthethink

alouddatasearchingwithinasourcewasastrategyprevalentamongproficientusersduringORCA‐

MR.Theinstancesofthisstrategyuseindicatethattheabilitytosearchwithinawebsiterequired

morereflectivethinking,persistenceandflexibility.Thesearethreesubscalesofthedispositionsof

onlinereadingcomprehensioninstrument.ThestudentswhoweremoresuccessfulontheORCA‐

MRtaskdemonstratedanabilitybeyondthatofcognitiveskills.

187

ChapterSummary

Thegoalofthequalitativeportionofthisstudywastoexaminedifferencesofonlinereading

comprehensionstrategyuseamonghighandlowperformersonanassessmentofonlinereading

comprehensioninalessrestrictedinformationandalsoinamorerestrictedinformationspace.

ThisanalysiswasdesignedtoexplainthequantitativedifferencesfoundinChapterFour.Atwo‐

stagequalitativeanalysiswasutilizedtoexaminethink‐alouddata.

Analysisinstageoneidentifiedactionsthataffectedperformanceregardlessofthe

informationspace.Theseincludedgreatersourceknowledgeandcursorcontrol.Furthermorestage

oneanalysishowalackofcriticalevaluationandcommunicationskillslimitedstudent

performance.

Thestageoneresultshelpedtoillustratesomeofthequantitativefindingsofthisstudy.

Moststrikinglythethinkalouddataexploredthedifficultystudentshadwithboththeevaluation

andcommunicationitems.Statisticalevidencefoundthesetobethehardestitemclustersonboth

theORCA‐LRandtheORCA‐MR.Stageoneresultsrevealedthatstudentsseldomhadthestrategies

toevaluateonlineinformationandwereunpreparedtocommunicateinonlinespaces.

Thestageoneresultsmayalsohelptoexplainwhybackgroundknowledge,while

significant,wasnotthebestpredictorofperformanceineithertheORCA‐LRortheORCA‐MR.

Studentswhodidwellinbothformatscouldquickly“reassemble”informationtheyreadwhile

synthesizingsources.Thisabilityrequiredgreatercursorcontrolratherthanbackground

knowledge.

Instagetwothequalitativeanalysisidentifieddifferentpatternsofstrategyusebasedon

therestrictedinformationspace.Theonlydifferenceinstrategyuseamonghighperforming

studentsandlowperformingstudentswasontheitemsmeasuringlocatingskills.Ontheless

restrictedtasksstudentswhosearchedforatopicandfocus,readsearchresults,andmade

relevancyjudgmentsoutperformedtheirpeers.Onthemorerestrictedtasksstudentswho

188

successfullysearchedwithinwebsitesoutperformedtheirpeersintherestrictedinformationspace.

InfactthemostproficientusersnavigateddirectlytothesourcebyenteringtheURLintothe

navigationbar.Whenstudentsdidusekeywordsthosestudentswhoincludedthetopicandsource

outperformedtheirpeers.Finallystudentswhomademorerelevancyjudgmentsdidwellonthe

ORCAmorerestrictedlocatingitems.

Thestagetworesultshelptohighlightmanyofthequantitativefindingsofthisstudy.Most

importantlytheymayshedlightonthedifferencesintheoverallmeanscoresoftheORCA‐LR

(M=8.394SD=4.117)andscoresoftheORCA‐MR(M=7.15SD=4.48).Theresultsofthethinkaloud

studyindicatethatitwasmoredifficultforstudentstosearchforaspecificsourceratherthan

informationonagiventopic.

Thisstudyalsohelpedtoidentifystrategiesthatareusedbymoreproficientstudentsas

theyusetheInternettolearn.Thisstudytookplaceinthedisciplineofsocialstudies.Thestrategies

thatwereobservedinthemoreproficientstudentsinclude:

Locating strategies on a less restricted task

o using keyword that included both the topic and focus;

o copying and pasting keywords that include both the topic and focus;

o actively reading search results;

o navigating back to search results;

o skimming websites to make accurate relevancy judgment.

Locating strategies on a more restricted task

navigating directly to a source;

searching for a specific source;

searching within a website;

using keywords with topic and source;

making accurate relevancy judgments. utilizing tabs to navigate between task and source;

189

Strategies regardless of the restricted nature of the information space

o using tabs to navigate between multiple sources;

o using a strategy of skimming websites to identify key details;

o engaged reading within a website;

o navigating to a source to read while taking a position;

o navigating to the source to copy and paste details;

o identifying the author on an about us page;

o evaluating author using effective markers of expertise;

o checking evidence against a secondary source;

o inferring an author’s point of view;

o recalling details from memory when combining sources;

o recalling details from memory when taking a position.

190

ChapterVI

DISCUSSIONOFTHEFINDINGS

Introduction




restrictedormorerestrictedinformationspaces.Specificallythisstudysoughttoanswerfour

questions:

1. When predicting online reading comprehension during a problem solving task within

a less restricted information space, what is the best fit model, after controlling for



online reading comprehension?


more restricted information space, what is the best fit model, after controlling for



online reading comprehension.



less restricted information space?



more restricted information space?

191

Quantitativemethodswereusedtoaddressthefirsttworesearchquestions.Hierarchical

regression,controllingforverbalintelligence,wasusedtotesttwodifferentmodels.Thefirstmodel

includedalessrestrictedinformationspace.Thesecondmodelinvolvedamorerestrictedspace.In

bothcasesscoresonabackgroundknowledgemeasure,acriticalevaluationmeasure,anda

dispositionmeasurewereusedtopredictscoresoneitheranonlinereadingcomprehension

assessmentinalessrestrictedinformationspaceandanonlinereadingcomprehensionassessment

inamorerestrictedinformationspace.

Qualitativemethodswereusedtoanswerthesecondtworesearchquestions.Verbal

protocolanalysisusingabductive(Onwuegbuzie&Leech,2006)codingmethodsfollowedbya

constant‐comparative(Bogdan&Biklen,2003;Merriam,1988)analysiswereused.Thegoalofthe

qualitativecomponentwastoanalyzepatternsofprocessingamongstudentswithvaryingdegrees

ofonlinereadingcomprehensionability.Tenparticipantseachcompletedtwoonlinereading

comprehensionassessmenttasks:alessrestrictedtaskandamorerestrictedtask.Screencasts

weremadeoftheactivitiesandpatternsofstrategyusewereidentifiedusingconstantcomparative

methods.

Theuseofthismixedmethodapproachallowedmetounderstandnotonlytheunique

contributionsthevariablesofinterestmadetothemodelbutalsohowthesevariablesmightlookin

classroomcontexts.Byexaminingboththequantitativeandqualitativefindingstheimplications

forresearchandclassroompracticeemerge.

DiscussionoftheQuantitativeResults

OnlineReadingComprehensionAssessment:DescriptiveStatistics

Thedescriptivestatisticsrevealimportantinsightsintotheresultsofthisstudy.Asnoted

statisticaltestingfoundsignificantdifferencesbetweenthemeanscoresofthetwoORCAformats

(t=4.088,p=.000).Thuswouldindicatethattherewasasignificantdifferenceinstudentscoreson

192

theORCA‐LRandtheORCA‐MRwithscoresonORCA‐MRbeingsignificantlylower.Aoneway

repeatedmeasuresanalysisofvariancewasconductedtocomparestudents’scoresineachofthe

fourskillareasontheORCA‐LRandtheORCA‐MRrespectively.Inbothmodels,themeanscores

weresignificantlydifferent.Followuppost‐hoctestsfound,forboththeORCA‐LRandtheORCA‐

MR,thatlocationitemsweretheeasiestitems,followedbysynthesisitems,thencommunicate

items,andfinallyevaluateitems.

Theseresultssomewhatreplicaterecentstudiesthathaveevaluatedpatternsinthescores

ofLocate,Evaluate,Synthesize,andCommunicateitems.ForzaniandBurlingame(2012),for

example,whenanalyzingarelatedORCAassessmentfoundthatsynthesiswastheeasiestskillarea,

followedbylocate,thencommunicateitemsand,finally,byevaluateitems.Bothofthesestudies

foundthatcommunicateitemsandevaluateitemswerethehardestandhadtheloweststudent

scores.

Thesefindingshaveimportantimplicationsforinstructionandresearchinonlinereading

comprehension.Intermsofclassroompracticeitappearsthatstudentsstruggletocommunicate

whattheylearnandtoevaluatethesourcestheyreadonline.Thisisacriticalissueforschoolsas

thesearebothareasemphasizedintheCommonCoreStateStandards(CCSS,2010).Futurestudies

shouldbeconductedtoseeifsimilarpatternsofstudentscoresarereplicated.Iftheyare,

instructionalstudiesshouldbeconductedtoevaluatehowbesttoteachtheseareaswherestudents

performlowest.

SummaryoftheLessRestrictedModel:ResearchQuestionOne

Researchquestiononeasked:

Whenpredictingonlinereadingcomprehensionduringaproblemsolving

taskwithinalessrestrictedinformationspace,whatisthebestfitmodel,

aftercontrollingforverbalintelligence,ofthepercentageofvariance

193

accountedforbyeachofthefollowingvariables:backgroundknowledge,

criticalevaluation,anddispositionsofonlinereadingcomprehension?

Resultsofthelinearregressionindicatedfirst,thatverbalability,asmeasuredbya

vocabularytest,explained16.9%ofthevariance,whichwassignificant,F(1,93)=20.530,p<.001.In

thefullmodel,scoresonacriticalevaluationmeasureexplainedasignificantamountofunique

variance(β=.335p<.000).Backgroundknowledgealsocontributedsignificantlytotheprediction

ofscoresontheunrestrictedORCA(β=.211p=.021).Dispositionsofonlinereading

comprehensiondidnotcontributesignificantlytopredictionsinthemodel(β=.117p=.162).The

totalexplanationhadalargeeffect,Cohen’sF=.358.

Inthelessrestrictedmodelscoresonthecriticalevaluationmeasurehadthelargest

predictiveweight.ThisfindingisconsistentwithbothCognitiveFlexibilityTheory(CFT)thenew

literaciesofonlinereadingcomprehension.ItwaspredictedbasedonCFTthatscoresonthe

criticalevaluationassessmentwouldbeasignificantpredictorinthelessrestrictedmodel.Thisis

duetostudentswith“advancedwebskills”performingbetterinanopenInternetspace.

ThepredictivepowerofCOILscoresalsoalignedwithatheoryofnewliteraciesofonline

readingcomprehension.TheprinciplesofthebroadercapitalletterNewLiteracieshasrecognized

thecentralrolethatcriticalliteraciesplayindigitalenvironments(Leu,Kinzer,Coiro,Castek,&

Henry,2013).Morespecificallythenewliteraciesofonlinereadingcomprehensiontheoristshave

alsolongarguedthatthecriticalevaluationofonlinesourcesisakeydifference(Coiro,2003;Coiro

etal,2008;Leuetal.,2004).

Inthelessrestrictedmodelbackgroundknowledgewasasignificant,butnotthelargest,

predictor.ThisresultisalsocongruentwiththepredictionsmadebasedonbothCognitive

FlexibilityTheoryandatheoryofnewliteraciesofonlinereadingcomprehension.Thesetheories

havesuggestedthattheroleofbackgroundknowledgeinpredictingreadingcomprehension

changesinonlineinformationcontexts.Whilebackgroundknowledgehasbeenoneofthemost

194

stablefindingsofcomprehensionresearchthisstudymayaddcredencetotheclaimthatwemust

rethinktheroleofbackgroundknowledgewhendevelopingtheoreticalmodelsofonlinereading

comprehension.

Inthelessrestrictedmodelscoresonaself‐reportmeasureofdispositionsofonlinereading

comprehensionwerenotasignificantpredictor.Thisresultdidnotalignwiththepredictionsmade

fromeithertheoreticalperspective.CognitiveFlexibilityTheoryaguesthatstudentswith“open

mindsets”arebetterpreparedtoreadintheill‐structuredspaceoftheInternet.Newliteraciesof

onlinereadingcomprehensionarguesthatnewdispositionsarecentraltomakingmeaningonline.

Yetinthelessrestrictedmodelascoreonadispositionsmeasurewasnotasignificantpredictor.

SummaryoftheMoreRestrictedModeltoAddressResearchQuestionTwo

Researchquestiontwoasked:

Whenpredictingonlinereadingcomprehensionofaproblemsolvingtask

withinamorerestrictedinformationspace,whatisthebestfitmodel,after

controllingforverbalintelligence,ofthepercentageofvarianceaccounted

forbyeachofthefollowingvariables:backgroundknowledge,critical

evaluation,anddispositionsofonlinereadingcomprehension.

Resultsforthelinearregressionindicatethataftercontrollingforverbalintelligence,the

threepredictorvariablesofbackgroundknowledge,evaluationskills,andscoresonaself‐report

measureofdispositionsofonlinereadingcomprehensionexplainedanadditional38.1%ofthe

varianceonthescoresofthemorerestrictedORCA,whichwassignificantF(1,92)=26.328p<.000.

Inthefullmodelscoresonacriticalevaluationmeasureexplainedasignificantamountofunique

variance(β=.499p<.000).Backgroundknowledgealsocontributedsignificantlytotheprediction

ofscoresontheunrestrictedORCA(β=.189p=.017).Dispositionsalsomadeasignificant

predictiontothemodel(β=.291p=.006).Thistotalexplanationhadalargeeffect,Cohen’sF=.631.

195

Inthemorerestrictedmodel,similartothelessrestrictedmodel,scoresontheCOIL,a

measureofcriticalevaluationskills,wasasignificantpredictorwiththelargestbetaweightinthe

model.ThisfindingmirroredthepredictionsthatweremadebasedonbothCFTandatheoryof

newliteraciesofonlinereadingcomprehension.Thiswouldonceagainsuggesttheabilityto

flexiblyapplynewstrategiesinavarietyofsituationsisacentralrequirementforsuccesswhen

makingmeaninginonlinespaces.

Inthemorerestrictedmodelbackgroundknowledgewasasignificant,butnotthelargest,

predictorofperformanceontheORCA‐MR.Thisresultdidnotalignwiththepredictionmadebased

ontheprinciplesofCognitiveFlexibilityTheory.Itwaspredictedthatbackgroundknowledge

wouldbesignificantinbothmodels.Howeveritwasbelievedthatbackgroundknowledgewouldbe

thestrongestpredictorinthemorerestrictedspace,aslearnerswouldnotbenefitfrom“flexible

schemaassembly”and“serendipitous”learningwhilelookingforspecificsources.Inotherwords

becausestudentscouldnotbuilduptheirknowledgewhileconductingopenInternetsearches

thosewithhigherbackgroundknowledgewouldhaveagreaterchanceofsuccesslookingfor

specificsources.Yetinthemorerestrictedmodelitwascriticalevaluationandnotbackground

knowledgethatwasthestrongestpredictor.

Inthemorerestrictedmodel,scoresonaself‐reportmeasureofdispositionsofonline

readingcomprehensionwereasignificantpredictorofperformanceontheORCA‐MR.Thisdiffered

fromtheresultsoftheORCA‐LRmodel.Thisfindingalignedwithpredictionsmadebasedon

CognitiveFlexibilityTheoryandatheoryofnewliteraciesofonlinereadingcomprehension.InCFT

itisarguedthatreaderswith“openmindsets”willfairbetterinonlinespaces(Spiro&Deschryver,

2010).Newliteraciesofonlinereadingcomprehensiontheoristsalsoarguethatnewdispositions

arerequiredtomakemeaningwhenreadingonline(Leuetal.,2004).

ExploringTheImplicationofBothModels

196

Theoreticalmodelsofreadingcomprehension.Theresultsofthesetwomodelsindicate

theneedtoexpandourmodelsofreadingcomprehensiontoaccountfortheriseofonlinetexts.

Mosttheoreticalmodelsofreadingaredrawnonprinciplesofstaticandconsistentconstructs.For

exampletheheuristicofreader,text,andactivity(Snow,2002)reliesonwell‐foundedresults

establishedthroughdecadesofcomprehensionresearch.Yetthisstudy,andsimilarwork,callsinto

questionstaticmodelsofreadingcomprehension.

Overthepastthreedecadesresearchershaveidentifiedstablefactorstoincludeinmodels

ofreadingcomprehension(Snow,2002).Yetinadigitalagewhennewtextsandtoolsforliteracy

emergeeveryday(Coiroetal.,2008)researcherscannolonersolelyrelyonstablefindings.Asnew

textandliteracypracticesemergewiththeshiftfrompagetopixel(Hartmanetal.,2010)new

variablesofinterestwillemergeandconfoundourpreviousmodels.Thesefactors,forexample,

couldincludethecentralityofevaluationandnavigationasnotedinthisstudy.Theywillalso

includevariablesyettobeidentifiedastheInternetcontinuestoevolve.Wecannolongerrelyon

staticmodelsofreadingcomprehension.

Insteadweneedtocontinuetodevelopdynamicmodelsofreadingcomprehension

(Hartmanetal.,2010)thatcanaccountforconstantchange(Leu,2000).Thisstudyhelpsto

illuminatesomepossibleelementsofdynamicmodelsofreadingcomprehension.Firstinthese

modelstheevaluationofmultiplesourcesasafluidtextmustbeatthecenterofreading.Alsoin

thesenewtheoreticalmodelsthereadermayrelyonactivelyconstructingknowledgeandtextsin

themoment(DeschryverandSpiro)ratherthansolelyrelyingonbackgroundknowledge.Finally

thesemodelsmayhavetoaccountforepistemologicalprocessesandhabitsofmindinorderto

accountfornewdispositions.

Criticalevaluation.Themostcompellingfindingfromthequantitativefindingsofthis

studyisthecentralroleofcriticalevaluation.Aftercontrollingforverbalcomprehension,scoreson

theCOILassessmentwerethebestpredictors(β=.429)ofscoresonthelessrestrictedinformation

197

spaceandalsoonthemorerestrictedmodel(β=.499).Thefactthatcriticalevaluation,not

backgroundknowledge,wasthebestpredictorofonlinereadingcomprehensionmaylend

additionalsupporttostudiessuggestingthatonlinereadingcomprehensionisdifferentfromoffline

readingcomprehension.Thisfindingaddsweighttorecentresearch(Braten,Stromso,&

Samuelstuen,200;Coiro,2011;Goldman,Braasch,Wiley,Graesser,&Brodowinska,

2012).Goldmanetal.,(2012)usingathinkaloudmethodamongadultsfoundthatexpertreaders

usedmoreevaluativestrategieswhencomparedtonovices.Coiro(2011)foundthatafter

controllingforbackgroundknowledgeandofflinereadingabilitythatknowingastudent’sonline

readingcomprhensionskillpredictedadditionalvarianceinscoresonanonlinereading

comprehensionassessement.Herresultsindicatethatnewknowledge,skills,anddispositionsare

involvedinonlinereadingcomprehensionassessment.Thisstudysuggeststhat,inaddition,the

relativecontributionofcriticalevaluationskillsmaybegreaterinonlinereadingcomprehension,

comparedtoofflinereadingcomprehension.

Braten,Stromso,andSamuelstuen(2008)foundthatknowingastudents’sourcingskills

bestpredictedtheirabilitywithmultiplesourcecomprehensioninofflinereadingcontexts.

Similarlythisstudyfoundthatknowingstudents’criticalevaluationskillsbestpredictstheironline

readingcomprehensionability.Sinceonlinereadingcomprehensiontypicallyinvolvesmultiple

sourcereading,thismayindicatethatmultiplesourcereadingisanimportantdifferencebetween

offlineandonlinereadingcomprehension.Furthermoreitmayindicatethatevaluatingsourcesis

centraltobothmultiplesourcereadingofflineandonlinereadingcomprehension.However,given

thatmultiplesourcereadingofflineusuallyinvolvespre‐vettedsourcessuchasnewspapersand

journals,additionalevaluationskillsmayberequiredinonlineenvironments.

AssessingCriticalEvaluation.Perhapsoneofthegreatestchallengesthatliteracy

researchersfaceisthefurtherrefinementanddevelopmentofmeasurestoevaluatecritical

evaluationskills.Thefinal12itemCriticalOnlineInformationLiteraciesassessment(COIL)hada

198

coefficientalphaof.71.Whilethislevelofareliabilityestimateisacceptableonemighthavegreater

confidenceinthemeasurehaditbeenhigher.

Researchmustalsobeconductedtounderstandhowitemformateffectsitemdifficulty.The

COILcontainedforcedresponseanswersthatincludedentirescreenshotsofwebsites.Thisviolates

basicprinciplesofcomprehensionassessmentthatcallforshortdistractors(Fuchs,Fuchs,&

Maxwell,1988;Keenan&Betjemann,2008).ItemsintheCOILwerealsopresentedindifferent

formats.Someitemscontainedsinglescreenshots,otheritemscontainedfourscreenshots,and

finallyitemscouldhavehadfourhyperlinkstorecreationsofactualwebsites.Thisitemformatmay

haveinfluenceditemloadings.Futurestudiesshouldbeconductedusingitemresponsetheory

(Baker,2001)totestifdistractorlengthanditemformatinfluencesitemdifficulty.

Backgroundknowledge.Basedonthelinearregressionmodelsinthisstudy,background

knowledgewasasignificantpredictorofscoresonthelessrestrictedORCA(β=.241).Itwasalsoa

significantpredictorofscoresonthemorerestrictedORCA(β=.189).Backgroundknowledgeisone

ofthemostwidelyacceptedpredictorsofreadingcomprehension.Ithasbeendemonstratedthat

backgroundknowledgeaffectscomprehension(Anderson&Pearson,1984;Kintsch,Patel,&

Ericson,1999;Voss,Fincher,Keifer,Green,Post,1985).Theoverarchingtheoryisthatgoodreaders

relatetheirbackgroundknowledgetothetextandthetask.

Thisstudymaycallintoquestionthelong‐standingdominanceofbackgroundknowledgein

comprehension.Ithasbeenargued,whenexaminingprintbasedcomprehensionthatknowledgeis

comprehension(Willingham,2010).Yetwhenexaminingtheresultsofthisstudyitisclearthat

newknowledge,skills,andstrategiesmaybejustas,orevenmoreimportant,thanbackground

knowledge.Thisfindingcallsintoquestionmuchofourlong‐standingbeliefsabouttheroleor


Infactthereisongoingdebateastotherolebackgroundknowledgewillplayinonline

readingcomprehension.TheresultsofthisstudyparallelthosesuchasHillandHannifin(1997)

199

andCoiro(2012)whothatfoundbackgroundknowledgepredictedperformanceinonlinereading

comprehensiontasks.However,Bilal(2000;2001)foundthatbackgroundknowledgedidnotplaya

significantroleindeterminingsuccesswhenconductingInternetinquires.Thisstudyadds

evidencetothedebate.

Mostoftheresearchontheroleofbackgroundknowledgeassumesacognitivestructure

basedontheprinciplesofinformationprocessingtheories(Kirschner,Sweller,&Clark,2006).This

includesacognitivearchitecturewithelementsoflong‐termmemory,short‐termmemory,and

workingmemory.Basedonthistheorybackgroundknowledgeisconsideredtobeanelementof

long‐termmemory.YettheresultsofthisstudyandrecentfindingsbyBilal(2000;2001)andCoiro

(2012)maycallintoquestionthesetheoreticalmodels.Insteadofrecallingdetailsfromlong‐term

memorystorageonlinereadersmaybecreatingwhatCFTtheoristshavelabeled“schemaatthe

moment”whichiscreatedthrough“flexibleassemblage”ofinformationencounteredonlineinreal‐

time.

Theseobservationsmayalignbetterwithsituatedcognitionmodels,whichsuggestthat

knowledgeexists“insitu,inseparablefromcontext,activity,people,culture,andlanguage.”

(McVerry,2010,para.1)Theliteracycommunitymaybenefitfromalineofresearchintotheroleof

backgroundknowledgewithresearchdesignsthatdrawonprinciplesofsituatedcognition

(Greeno,1989)aswellastraditionalinformationprocessingtheories(Kirschner,Sweller,&Clark,

2006;LaBerge,&Samuels,1974).

Futurestudiesshouldalsoincludemeasuresofsystemsknowledgeandnotsimplycontent

knowledge.Oneofthekeydifferencesbetweenstudiesthatfoundaroleforbackgroundknowledge

(Coiro,2012;Hill&Hannafin,1997)andthosethatdidnot(Bilal,2000;2001)wastheinclusionof

itemstomeasuresystemsknowledgeabouttheInternet.Thisstudyonlyincludedameasureof

topicalknowledgeontheAmericanRevolution.Followupstudiesmightincludebackground

knowledgescoresfromabasicnavigationtest.

200

Dispositionsofonlinereadingcomprehension.Thefactthatscoresontheself‐report

measureofdispositionsofonlinereadingcomprehensionwerenotasignificantpredictorinthe

ORCAlessrestrictedmodel(β=.124)butwereasignificantpredictorinthemorerestrictedmodel

(β=.206)isaveryinterestingoutcome.Thesefindingsmayindicatethattheinformationspacehas

aninfluenceduringonlinereading.Furthermore,theresultsmaysuggestthatmoretargetedand

specializedsearches,whichutilizemorerestrictedspaces,requirereaderswhocanadapttonew

situations.

ThebivariatecorrelationsoftheDORCwereveryinformativeinexploringthisfinding.Of

mostinterestwasthenegativecorrelationoftheitemsonthecollaborationsubscale.Collaboration

hadnocorrelationwiththelessrestrictedORCAandanalmostnon‐existentandinsignificant

correlationwiththemorerestrictedORCA.

Yetatthesametimerecentworkhashighlightedhowimportantcollaborationisduringin

onlinereadingcomprehension(Coiro,Castek,andGuzniczak,2011;Kiili,Laurinen,Marttunen,&

Leu,2011;O’Byrne,2011;Zawilinski2011).Ineachofthesestudies,collaborationimproved

studentperformancewhenconductingInternetinquiryactivities.Kiilietal.(2011)foundthat

collaborativegroupsreadingonlinecomparedmultipleperspectivesandsoughtdeepermeaning

thanindividualsreadingonline.O’Byrnefoundthatstudentsworkingingroupstocreatespoof

websitesimprovedtheirevaluationskills.FinallyZawilinskifoundthatfirstgradersandfifth

gradersbloggingtogetherimprovedcommunicationskills.Theseresultshaveleadresearchers

(Coiro,Castek,&Guzniczak,2011;Killietal.,2011)toarguethatwemustviewonlinereading

comprehensionasacollaborativesocialpractice.

Itisclearbasedonthesestudiesthatopennesstocollaborationisacentraldispositionto

onlinereadingcomprehension.YetinthisstudythecollaborationelementsoftheDORCwerenot

significantlycorrelatedwithperformanceontheORCA.Thismaybeanartifactofthesolitary

201

natureofthetestingenvironment.Inotherwordstheassessmentdesigndidnotallowstudentsto

takeadvantageoftheaffordancesthatawillingnesstocollaborateprovides.

Futureworkshouldalsobeconductedtofurtherrefinemeasuresofdispositionsforonline

readingcomprehension.Thisstudyusedapreviouslyvalidatedinstrumentthatincludedfive

subscales:reflectivethinking,criticalstance,persistence,flexibilityandcollaboration.Otherwork

beingconducted(Putman,inpress)hasexaminedmotivation,self‐efficacy,valueandanxiety.

Furthermoresomeresearchershavecritiquedself‐reportmeasuresofdispositions(Carr&

Claxton,2002).Themostpromisingdirectionforthefieldmaybenotininstrumentationbutin

facialrecognitionsoftware(D’Mello&Graesser,2010).Thisresearchcancapturerealtime

indicatorsofdispositionsinplaybycapturingsubtledifferenceinfacialexpression,whichcan

recognizepersistence,frustration,andsuccess.

ResearchshouldalsobeconductedthatmeasuresthepredictivevalueoftheDORCwhen

studentsarecompletingonlinereadingcomprehensiontasksinpairsoringroups.Giventhat

recentworkhasfoundsuchastrongroleforcollaborationisassistingonlinereading

comprehensionitwouldbebeneficialtothereadingresearchcommunitytounderstandhow

collaborationchangescomprehension.Weneedtoidentifytheaffectivevariablesthatpredict

successincollaborativeenvironments.

Summaryofthequantitativeimplications.Thisstudyexaminedmodelsofonline

readingcomprehension.Specificallyitexaminedtherelativecontributionsthatcriticalevaluation,

backgroundknowledge,anddispositionsmaketoonlinereadingcomprehensionaftercontrolling

forverbalintelligence.Thismodelwastestedinbothalessrestrictedinformationspace,where

studentscouldsearchforanywebsite,andamorerestrictedinformation,wherestudentshadto

findspecificwebsites.Inbothmodelscriticalevaluationscoreswerethelargestpredictorsof

performance.Backgroundknowledgewasalsoasignificantpredictorinbothmodels.Thescoreon

thedispositionmeasurewasonlyasignificantpredictorinthemorerestrictedmodel.

202

Theseresultsindicatethatresearchersandteachersmightmorecarefullyconsiderthe

differentspacesinwhichonlinereadingcomprehensiontakesplace.Wewillneedanincreased

effortinunderstandinghowtheinformationfieldinfluencestheperformanceofstudentsduring

onlinereadingcomprehension.Wealsoneedtodevelopmodelsofinstructionthatprepare

studentsforthe“openmindsets”and“advancedwebskills”necessaryforonlinereading

comprehension(Deschryver&Spiro,2010,p.4).

Afterall,theCommonCoreStateStandardscallforonlineresearchandmediaskillstobe

embeddedinallcontentareasandacrossallclassrooms.Theresultsofthisstudyindicatethata

large‐scaleeffortmustbetakentopreparestudentsforonlinereadingcomprehension.Basedon

themeanORCAscoresandtheCOILscoresitisclearthatstudentsareillpreparedtomake

meaninginonlinespaces.Theresearchcommunityandeducatorsalikemustworktogetherto

developnewinstructionalroutinestoaddressthischallenge.

DiscussionoftheQualitativeResults

Thepurposeofthequalitativestudywastoinvestigatedifferencesofstrategyuseamong

studentswithvaryingdegreesofonlinereadingcomprehensionproficiency.Tenstudentswere

includedinthefinalanalysis.EachparticipantcompletedalessrestrictedORCAandamore

restrictedORCAtask.Usingverbalprotocolanalysis(Afflerbach,2002)andabductivecoding

methodsfollowedbyconstantcomparativemethods(Bogdan&Biklen,2003;Merriam,1988),this

studysoughttoanswerthethirdandfourthresearchquestions:







203

Toanswerthesequestionsatwo‐stageanalysiswasused.Inthefirststage,patternsof

processingwereidentifiedthatimprovedoverallperformanceontheORCAassessmentsregardless

ofthenatureoftheinformationspace.Inthesecondstageofanalysis,patternsofprocessingwere

identifiedthatchangedbasedonthenatureoftheinformationspace.Bothstagesofanalysishelped

toilluminatethequantitativefindings.

QualitativeResults:DiscussionofStageOneFindings

Stageoneanalysisidentifiedprocessingpatternsthatledtoincreasesinoverall

performanceregardlessoftheinformationspace.Fourthemeswereidentifiedinthisstageof

analysis:(a)overallORCAscoresappearedtoberelatedtostrategictextassembly;(b)overall

ORCAscoresindicatedalackofsourceevaluationknowledgeamongparticipants;(c)studentswho

recalleddetailsfrommemoryappearedtoperformbetterontheORCAtasks;and(d)students

appearedunpreparedtoengageinthecommunicationstrategiesnecessaryforonlinereading

comprehensioninacademicsettings.

OverallORCAscoresappearedtoberelatedtostrategictextassembly.Thefirsttheme

instageoneanalysishighlightedhowimportantnavigationalandcomprehensionmonitoringskills

wereindeterminingsuccessatonlinereadingcomprehension.Students,whonavigatedbetween

sources,returnedtothetaskandalteredtheircursorspeedwhilereadingoutperformedtheirpeers

whodotheselessfrequently.

Thisfindingisconsistentwithpreviousresearchthathasfoundreadingonlinerequires

similarbutmorecomplexreadingskills(Coiro&Dobler,2007;Goldmanetal.,2012).Oneofthese

newcomplexitiesmaybeintheabilitytonavigateinformationalspaces.InfactGoldmanetal.,

usingthinkaloudprocedurestocomparecollegeageexpertandnovicereadersfoundnavigationto

beacrucialdifference.Similartothisstudy,Goldmanetal.foundthatexpertonlinereadersmade

significantlymorenavigationalchoicesbacktoreliablesourcesandawayfromunreliablesources.

McWilliamsandClinton(2010)arguedthat“readingwithmouseinhand”isafundamental

204

differenceinonlinespaces.O’Hanlon(2002)arguesthatstudentsentercollegewithoutbasic

navigationalabilitiestoconductonlineresearch.Ifstudentsaretograduatehighschoolwiththe

onlineresearchandmediaskillstobecollegeandcareerready(CCSS,2010)theneffortsmustbe

undertakentoincreasethefullrangeofnavigationalskills.

Comprehensionmonitoringwasdefinedinthisstudyasalsoknowingwhentoskimand

scansourcesandwhentoslowdownforengagedreading.Inthisstudytheexpertreadersused

strategiestoskimsourcesandusedstrategiesofengagedreadingwithagreaterfrequencythan

theirpeers.ThisissimilartotheresultsfoundGoldmanetal.(2012)whereexpertundergraduate

readersspentconsiderablemoretimereadingreliablesourcesandlesstimereadingunreliable

sourcesthantheirpeers.

Thisfirsttheme,thatperformanceontheORCArequiresstrategictextassembly,mayalso

informthequantitativefindingsofthisstudy.Inboththelessrestrictedandthemorerestricted

modelstudents’scoresonthecriticalevaluationmeasurewerethestrongestpredictorof

performanceontheORCA‐LRandORCA‐MR.Thiscouldbeduetothenumberofnavigational

choicesrequiredintheCOILinstrument.TheCOILconsistedoffourdifferentitemtypes:astatic

screenshotofasinglewebpage,astaticimageoffourscreenshots,awebpagewithhyperlinks,and

listoffourhyperlinks.Thusstudentswhohaveahigherabilitytoreadwithcursorcontrolmay

havesucceededontheCOILaswellastheORCA.

Animportantcontributionthatemergedfromthemeoneishighlightingtheimportanceof

navigationalskillsacrossallelementsofonlinereadingcomprehension.Thisstudyfoundthat

studentswhomovebetweenthetaskandthesource,especiallytheparticipantswhousedtabbed

browsing,outperformedtheirpeersontheORCAtasks.Thisstudyalsofoundthatexpertstudents

movedbetweensourceswithagreaterfrequencythantheirnovicepeers.Thesestudentsmadeuse

ofnavigationalstrategiesthroughouttheORCAassessmentbutpatternsespeciallyemergedduring

thesynthesisandcommunicationtasks.

205

Manypreviousstudieshaveoperationalizednavigationalstrategiesaspartofsearchengine

use(Bilal,2000;2001;HillandHannifin,1997).Yetthisstudy,inalignmentwiththeworkby

Goldmanandherpeers,foundthatnavigationalstrategiesarejustasimportantacrossallelements

ofonlinereadingcomprehension.

Implications.Theroleofnavigationalabilityfoundinthisstudymayhaveimportant

consequencesforclassroompractice.Firstofallthenextgenerationofhighstakestestingwillbe

deliveredelectronically.Thetestsaretobecompatiblewithdesktop,laptop,andtabletcomputing

(SBAC,2012;PARCC,2012).Giventhatthisstudyfoundthattheabilitytonavigateaninformation

spaceiscriticaltoperformanceteacherswillneedtoensurethatstudentsarepreparedforthis

typeoftestingenvironment.Itisalsoclearthateducatorsneedtoincreasetheirinstructioninthe

useofmultipleonlinesourcesinclassroom.Basedonthefindingsofthisthememuchofthis

instructionshouldfocusongoaldirectednavigationalstrategies.

OverallORCAscoresappearedtoindicatealackofsourceevaluationknowledge

amongparticipants.Thesecondthemeindicatedthatthemajorityofstudentssimplydonot

evaluatesourcestheyencounteronlineandiftheydo,theyrelyonverysuperficialmarkersof

relevancyandreliability.Thefactthatveryfewdifferencesinpatternsofprocessingwereidentified

amongstudentsbasedontheirproficiencylevelisstriking.Inrealityfewstudentshadtheabilityto

evaluateauthor’sexpertise,evidenceusedbytheauthor,andauthor’spointofview.Whenstudents

didtrytodothese,theyoftenmadeerrorsofoversimplification.

Thisthemereflectsmanyoftherecentfindingsofresearchinonlinereading

comprehension.Braaschetal,(2009)foundthatmiddleschoolstudentsrarelyevaluatedthe

usefulnessofsources.Leuetal.(2007)foundstudentshadaninabilitytoidentifyspoofsites.

Goldmanetal.(2012)alsofoundnosignificantdifferenceinthewebsiterankingsofreliability

amongexpertandnoviceundergraduatereaders.Coiro(2012)usingacasestudyanalysisofthree

206

students,foundthatthestudentseitherlackedcriticalevaluationskillsorreliedonsurfacelevel

informationtojudgesources.

Thisthemealsohelpedtoinformthequantitativefindingsofthisstudy.Itisconsistentwith

outcomesonboththeCOILassessmentandtheORCAassessments.IntermsoftheCOILassessment

thethinkalouddataillustrateswhymeanscoreswerelow.Itisclearfromthesetenparticipants

thatsofewcoulduseprocessesforsourceevaluationcorrectly.Infactonlytwostudentswas

successfullyabletojudgeanauthor’spointofview.This,addsadditionalevidenceforthevalidity

ofthismeasure.Thequalitativestudyalsohelpstoexplainwhytheevaluateitemswerethemost

difficultforstudentsinboththeORCA‐LRandtheORCA‐MR.

Implications.Giventhatstudentsrarelyutilizedcriticalevaluationstrategies,andwhen

studentsdidusethesestrategiestheyoftenmadeerrorsofoversimplification,itisclearthat

currentclassroompracticestoteachsourceevaluationmaybeinsufficient.Thequalitativeportion

ofthisstudyindicatedthattheseerrorsofoversimplificationmayhelptoexplaincommonresults

instudiesthatusecriticalevaluationmeasures.Inthisstudyandinsimilarresearchinvestigating

criticalevaluation,studentsoftenreliedonsurfacelevelfeaturestojudgesources.

Studies(Goldmanetal.,2012;Sanchezetal.,2006;Zhang&Duke,2007),forexample,that

usewebsiterankingstocomparenoviceandexpertreadersfindnosignificantdifferenceintheir

evaluationability(Goldmanetal.,2012).FurthermoreZhang&Duke(2007)andSanchezetal.

(2006)foundthatstudents’performanceatrankingwebsitesdidnotsignificantlyimprove,

followinganintervention,butstudentwrittenjustificationsdidsignificantlyimprove.Theseresults

mirrorthelackofevaluationskillsnotedinthisstudy.

Acrossallofthestudiesthateitherusemeasuresofcriticalevaluation(Goldmanetal,

2012;Sanchezetal.,2006;Wiley,2009;Zhang&Duke,2007;)orverbalprotocolstudies(Coiro,

2011;Coiro&Dobler,2007)studentsmadeerrorsofoversimplificationbyrelyingonsurfacelevel

207

textfeatures.Thepatternsinthisthemesupportthisconclusion.Inthisstudystudentsalsooverly

reliedonwebsitecontentandwebsitedesigntojudgeonlinesources.

Giventherelativestabilityofstudentsoverlyrelyingontextualfeaturestojudgewebsites

therehavebeencallstomoveawayfromatop‐downtaxonomyapproachofteachingcritical

evaluationskills(Goldmanetal.,2012).Insteadresearcherscallfortheoreticalandinstructional

modelsthatviewsourceevaluationasmoreintegraltothemeaningmakingprocess(Goldmanet

al.,2012).Onealternativetotopdowntaxonomieswouldbetodrawontheprinciplesof

instructionoutlinedinCognitiveFlexibilityTheory(Spiro,2004).Thiscallsformultiple

representationsofmaterialindiversecases.Suchanapproachwouldalignwellwiththetheoretical

modeloutlinedbyGoldmanetal.(2012).

StudentswhorecalldetailsfrommemoryappeartoperformbetterontheORCA

tasks.Thethirdthemeinstageoneanalysisindicatedthatstudentswhorecalleddetailsofwhat

theyreadduringtheORCAtasksoutperformedtheirpeers.Thesepatternswereespeciallyevident

whenstudentscombinedinformationfromwhattheyread.Thisthemewasalsoobservedwhen

studentstookapositiononthefinalinquirytask.

Thisresultreflectsrecentworkexaminingtheroleofshorttermandworkingmemory

duringInternetinquiry.LabergeandScafalia(2013)foundworkingmemorycapacitytobeastrong

predictorofperformanceinadultsengagedinreadingasinglewebsite.Herder&Juvina,(2004)

foundthatweaknavigationalchoiceswereassociatedwithlowerscoresonmeasuresofworking

memory.Withsimilarfindingsthisstudyfoundthatstudentswhoareabletorecallmoredetails

aboutthesourcestheyreadmayperformbetteratonlinereadingcomprehensiontasks.

Implications.Thisfindingsupportstheideathatresearchshouldcontinuetoinvestigate

therolememoryplaysinonlinereadingcomprehension.Thisstudyexaminedtherolebackground

knowledge,asoperationalizedascontentknowledge,playedinpredictingonlinereading

comprehensionscores.Howevergiventhatthethinkalouddatafoundstudentswhorecalled

208

detailsfrommemoryoutperformedstudentswhodidnot,futureresearchisneededintotheroleof

memory.Specifically,studiesshouldbecreatedthatexaminethecombinedpredictivepowerof

multiplesourcesofbackgroundknowledgesuchascontentknowledgeandsystemsknowledgeof

searchengines.Studiescouldalsoexaminethepredictivecapacityofshort‐termmemorycapacity,

andworkingmemorycapacity.Thiswouldprovideaclearerpictureoftheroleofvariouselements

thatmightcontributetobackgroundknowledgeuseduringonlinereadingcomprehension.

Studentsappearedunpreparedtoengageinthecommunicationstrategiesnecessary

foronlinereadingcomprehensioninacademicsettings.Thefourththemefromthestageone

analysissuggeststhatstudentsareillpreparedtoconveymeaninginformalonlineacademic

settings.Thiswasespeciallytruewhenstudentsdidnotincludetheinformationtheyreadintheir

finalpostsonasimulateddiscussionboard.

Theparticipantsinthisstudyrarelyusedevidencetobackupclaimsandonlyonestudent

includedanyinformationidentifyingasource.Theparticipantsinthisstudylackedthebasic

discoursemovesandargumentative(Tillman,1956)structuresnecessaryforacademicwriting.

Thiswouldindicatethatstudentsneedadditionalinstructionintheuseofargumentativetext

structureinonlinespaces.

Studentsalsodidnotfullyusetheaffordancesavailableinanonlinecommunicationtoolto

improvetheirwriting.Nostudentembeddedhyperlinkstosourcesorimages.Infacttheonly

multimodalfeatureusedwereemoticons,orsmallsmileyfaces.Thisstudycouldindicatethat

students,withoutformalinstruction,maybemorebiasedtocasualwritinginonlinespaces.

Theresultsofthisstudyreflectsimilarresearchintoacademicwritinginonlinespaces.

Whilethereissignificantevidenceofhighqualitycommunicationstrategiesininformalspaces

(Black,2007;Stone,2007)thereisagrowingbodyofevidencetosupportthatstudentsarenot

preparedforacademicdiscourseinonlinecontexts.ForzaniandBurlingame(2012),inevaluating

itemdifficultyonanORCAassessment,foundthatscoresoncommunicationitemswerethesecond

209

lowest,behindevaluationscores.Theirenvironmenttestedbothwikiandemailcommunication

tools.Thisstudyreachedthesameconclusionusingadiscussionboardformat.

Thistheme,alackofcommunicationstrategies,alsoinformsthequantitativefindingsofthis

study.MeanscoresonboththeORCA‐LRandtheORCA‐MRwerelowforthecommunicationitems.

Onlyevaluationscoreswerelower.Thequalitativeinstancesandanexaminationofthestudent

discussionboardpostsputthesescoresintofocus.Studentsaresimplyillpreparedforacademic

writinginonlinespaces.

Implications.Weneedadditionalefforttoincreasetheamountofopportunitiesstudents

havetoutilizeonlinecommunicationtoolsintheirclassrooms.Everystudentshouldhaveaccessto

schoolassigned,child‐safe,email.Everyteacherevaluationplanshouldmeasureateacher’sability

toutilizehybrid‐teachingenvironments.Giventhat70%ofcurrentcollegestudentswilltakean

onlineclassduringtheirsecondaryeducationalcareer(Sloan,2010)itishardtoarguethathigh

schoolscangraduatestudentswhoarecollegeandcareerreadyiftheycannotuseonlinespacesfor

academic,argumentativewriting.

QualitativeResults:DiscussionofStageTwoFindings

Thepurposeofthestagetwoanalysiswastoexamineifthereweredifferencesinpatterns

ofprocessingthatwereaffectedbytheinformationspace.Whereasstageoneanalysisidentified

patternsofprocessingthataffectedoverallperformance,stagetwoanalysisfocusedonresearch

questionsthreeandfour:







210

Ananalysisofthedataindicatedthattheonlydifferencesinthepatternsofprocessing,basedon

theinformationspace,appearedonthelocatingitems.

Whatpatternsofprocessingappearamongreaderswithvaryinglevelsofonline

readingcomprehensionabilityduringanonlinereadingcomprehensiontaskwithinaless

restrictedinformationspace?Threepatternswereidentifiedinthelessrestrictedinformation

space.FirststudentswhoperformedbetterontheORCA‐LRusedmoreeffectivestrategiesfor

enteringkeywords.AlsostudentswhoperformedbetterontheORCA‐LRusedmoreeffective

strategiesforreadingsearchresults.FinallystudentswhoperformedbetterontheORCA‐LRmade

morerelevancyjudgments.

Thesethreepatternsreflectmuchofthecurrentworkinvestigatingstudentsearchhabits

(Coiro&Dobler2007;Reih,2004;Tabatabai&Shore,2005).Whilekeywordselectionoftenbegins

withgeneralterms,researchhasfoundmoreskillfulreadersrevisekeywordduringthesearch

process(Rieh,2004).FurthermoreTabatabai&Shore(2005)foundthatexpertsusedmore

keywordsthannovices.Coiro&Dobler(2007)foundthatlocatingskillsmightaddtothe

complexityofonlinereading.Thisstudybuildsonthesefindingsbysuggestingskilledusersmay

adapttheirsearchingbehaviortotheinformationspace.

Theexaminationofprocessingpatternsonthelessrestrictedlocatingtasksmayalso

illuminatethequantitativefindingsofthisstudy.Specificallytheresultsofthethinkaloudindicated

thatstudentswhomademorerelevancyjudgmentsoutperformedtheirpeers.Giventhatthere

weremoreacceptableanswersontheORCA‐LRtaskwhencomparedtotheORCA‐MRthismay

haveleadtothedifferencesinmeanscoresonthelocatingitemsontheORCA‐LR(3.599)andthe

ORCA‐MR(2.755).Itmayhavecontributedtotheoverall,andsignificant,differencesinscoreson

theORCA‐LRandtheORCA‐MR.OntheORCA‐LRitemsstudentsoftenchosetoevaluatewebsites

wheretheauthorwaseasilyidentifiable.ThiswasnotanoptionontheORCA‐MRitems.Thismay

haveloweredevaluationitems.OntheORCA‐MRsynthesisitemstheinabilitytofindtherequired

211

sourcemayhaveimpactedthewebsitesthatwereusedtocombinedetails.Thesefactorsmayhave

leadtosignificantlylowerscoresontheORCA‐MR.

Whatpatternsofprocessingappearamongreaderswithvaryinglevelsofonline

readingcomprehensionabilityduringanonlinereadingcomprehensiontaskwithinamore

restrictedinformationspace?Thereweremajordifferencesinthepatternsofprocessingonthe

ORCA‐MRwhencomparedtotheORCA‐LR.OnthemorerestrictedORCAstudentswhoused

strategiestosearchwithinawebpageoutperformedtheirpeerswhodidnot.Whenstudentsdid

usesearchenginesthosewhoincludedboththetopicandthesourcescoredbetterthantheirpeers.

Finally,andcomparabletotheORCA‐LR,studentswhomademorerelevancyjudgments

outperformedtheirpeers.

Theseresultsalsoreflectfindingsinpreviousresearch.Successfulreaders infer which link

may be most useful on a webpage (Coiro & Dobler, 2008; Henry, 2006) during online reading

comprehension. Furthermore researchers (McDonald & Stevenson, 1996; Rouet, 2006) have found that

more skilled online readers efficiently scan for relevant information within websites. Goldman et al,

(2012) found that expert readers could navigate within a website when compared to their novice peers.

Similarly, in this study there was a clear difference between the proficient online readers and their peers.

More successful online readers could navigate directly to a website and search within that site. Less

proficient online readers often searched for a website using a search engine and accepted their first click

on the homepage as a relevant source.

The patterns of processing on the locating tasks of the ORCA-MR may also help to explain the

quantitative findings of this study. Specifically the ability of proficient readers to keep digging within a

source until they found the specified website may explain why dispositions scores were significant in the

ORCA-MR model but not the ORCA-LR model. Based on the think aloud data searching within a source

required more reflective thinking, persistence, and flexibility. These variables were measured by the

dispositions of online reading comprehension assessment.

212

ExploringTheImplicationsOfDifferentPatternsOfProcessingDueToTheInformation

Space

Implicationsforresearch.Inthisstudyonlytheinformationspace,specificallythe

locatingtasks,wererestricted.Thisinfluencedhowthevariablesofinterestbehavedinthemodels.

Ifotherelementsoftheinformationspace,beyondthelocatingtasks,wererestrictedthemodels

mayhavebeendifferent.Therearetwoadditionalmethodsforrestrictingtheinformationspace

thatmightbeattemptedinfuturework:restrictingtheonlinereadingcomprehensionassessment

itemsandrestrictingtheoverallinquirytask.

Futureworkmustcontinuetoseewhattherelativecontributionsofbackground

knowledge,criticalevaluationskills,anddispositionscoresplayinpredictingscoresononline

readingcomprehensionassessmentwhentherestrictednatureofevaluation,synthesisand

communicationitemsismanipulated.Forexamplestudentscouldbegiventhesourcetoevaluate

ratherthanchoosingasourcetoevaluate.Givingstudentsanotebooktoolwithscaffoldedprompts

couldrestrictthesynthesisitems.Finallyproviding“sentencestarters”couldrestrict

communicationitems.Eachoftheseelementswouldfurtherrestricttheinformationspacesbeing

usedbystudentsandcouldinfluencethecontributionsthatbackgroundknowledge,critical

evaluationskills,anddispositionscoresplayinpredictingscoresduringonlinereading

comprehension.

Futureworkalsoneedstocontinuetoexplorehowtaskdifferencesaffectonlinereading

comprehension.Itiswellknownthatthenatureofthetaskinfluencescomprehension(Snow,

2002).Thereforestudiescouldbeconductedthatrestricttheinquirytaskratherthantheelements

ofonlinereadingcomprehension.ForexampleThisstudywouldbeinformedbyfutureworkthat

thatteststhesameinquirytask,suchastheCrossingtheDelawaretask,inbothalessrestrictedand

morerestrictedinformationspace.Thiswouldhelpusdrawadistinctionbetweentaskdifferences

andtheinformationspace.

213

Infutureworktheinquirypromptitselfshouldbeevaluatedbasedonhowitaffectsthe

contributionthatbackgroundknowledge,criticalevaluationskills,anddispositionscoresplayin

predictingscoresononlinereadingcomprehensionassessment.Inthesestudiestheinquiry

promptandtaskcouldbethevariedasafactorindeterminingthenatureoftheinformationspace.

Forexampleonlinereadingcomprehensionassessmentscouldbebuiltthataskstudentstofinda

specificanswertoamorerestrictedquestionsuchas,“Whatisthedistancetothemoon?”Other

assessmentscouldasklessrestrictedquestionssuchas,“Whatisthebestwaytofightglobal

hunger?”Researchmayfindthatrestrictingtheinformationspacebyalteringtheinquiryquestion

mayaffectthecontributionthatbackgroundknowledge,criticalevaluationskills,anddisposition

scoresplayinpredictingonlinereadingcomprehensionperformance

Implicationsforpractice.Thestagetwothemesalsoaddtothegrowingevidencethat

teachingstudentstolocatesourcesontheInternetshouldbecomeacentralpracticeintoday’s

classroom.Thepatternsfoundinthisstudyaddtoresearchwhichconcludedthatgoodonline

readersgenerateeffectivekeywordsearchstrategies(Bilal,2000;Kuiper&Volman,2008),make

forwardinferences(Coiro,2011)inordertojudgetherelevancyoflinks(Henry,2006),and

effectivelyskimandscansourcestofinddetailswithinasource(Rouet,2006;Rouetetal.,2011).

Thefindingsfromthisstudysuggestthatweneedanincreasedinstructionalfocusonteaching

theseskillswithindifferenttypesofinformationspaces.Thepatternsofprocessingonthelocating

taskswerequitestarkbetweenthemoreproficientonlinereadersandtheirpeers.

Limitations

Thisstudyofseventhgradestudentsinthreediverseschooldistrictswasdesignedtotest

thecontributionsthatbackgroundknowledge,criticalevaluationskills,anddispositionsofonline

readingcomprehensionmadeonscoresofameasureofonlineaftercontrollingforverbal

intelligence.Twodifferentmodelsweretested,alessrestrictedinformationspaceandamore

restrictedinformationspace.Furthermorequalitativeanalyseswereconductedtoexaminethe

214

differencesinstrategyuseamongstudentswhovariedontheironlinereadingcomprehension

ability.Potentiallimitationstothisstudyneedtoberecognizedinanyinterpretation.

Instrumentation

Thetheoreticalmodelsinthisstudyareasonlygoodastheinstrumentsusedtocalculate

studentscores.Thisstudyrequiredthecreationandvalidationoffourseparateinstruments.

Thereforeresultsfromthisstudyshouldnotbegeneralizeduntiltheinstrumentscanundergo

furthertestingandthemodelsverifiedwithconfirmatorystudies.Therearelimitationstothe

ORCAassessments,thebackgroundknowledgemeasure,andthedispositionsofonlinereading

comprehensionmeasurethatmaylimittheinterpretivepowerofthisstudy.

Onlinereadingcomprehensionassessments.ThereliabilitiesfortheORCA‐LRand

ORCIA‐MRwereacceptablebutlow.ReliabilityforthelessrestrictedORCAwasestimatedbya

coefficientalphaof.72(Cronbach,192).ThecoefficientalphaforthemorerestrictedORCAwas

.804.Theseareacceptableforresearch(Nunnally,1978).Howeverreliabilityestimatesinthis

rangecansometimesleadtotheoverestimationofeffectsizesinmultipleregressionanalyses

(Osborne,2002).Infactinmultipleregressionmodels,“Withtheadditionofoneindependent

variablewithlessthanperfectreliability,eachsucceedingvariableenteredhastheopportunityto

claimpartoftheerrorvarianceleftoverbytheunreliablevariable”(Osborne&Waters,2002,para.

12).Giventheexploratorynatureofthisstudy,andthefactthatreliabilityestimatesexceedthose

requiredforresearch(Nunnally,1978)theinternalconsistencyofthequantitativemeasureswas

deemedadequate.

Backgroundknowledgemeasure.Thebackgroundknowledgemeasureusedinthisstudy

haslimitationsassociatedwiththevariable’suseintheregressionmodel.Thebiggestthreattothe

studywastheoverallnumberofstudentswhoreceivedazeroonthemeasureindicatingthatthey

couldnotrecallanyinformationabouttheAmericanRevolution.Thispresentsanumberofissues.

215

Firstitishighlyunlikelythatthestudentsdidnothaveanybackgroundknowledgeofthe

AmericanRevolution.Thisstudy,byusingthebackgroundknowledgemeasure,mayhave

underestimatedparticipants’backgroundknowledge.Thebackgroundknowledgemeasureusedan

onlineform,inagroupsetting,toaskstudentstorecallasmanydetailsaspossibleaboutthe

AmericanRevolution.Itmaybethatanothermethodsuchasaconstructedresponsetestor

administeringthesameinstrumentorallyandoneononewouldhavebeenabettermethodto

estimatebackgroundknowledge.

Thefrequencyoflowscoresontheassessmentalsoviolatedtheassumptionofnormality

requiredforregressionmodels.Thescoreshadtobetransformedduetoexcessivelyhighkurtosis.

Theuseofthislogarithmictransformationalsolimitstheinterpretationsofthisstudy.Thevariable

usedinthemodelwasnottheobservedbutrathertheresultofatransformationandmaycontain

newbiases(Beauchamp&Olson,1973).Thusanyinterpretationofthemodel,especiallyofthe

backgroundknowledgevariableshouldbeundertakenwithcaution.

Thelackofvariabilityofthetransformedbackgroundknowledgevariablealsopresentsa

limitationtointerpretation.Afterthetransformationthestandarddeviationforthebackground

knowledgescorewas.0725.Thislackofvariancethreatenstheoverallregressionmodels.Lower

variancecanleadtolargerstandarderrors(Osborne&Waters,2002).Inflatedstandarderrorscan

leadtoincreasedTypeIerrorrate.

Theformatofthebackgroundknowledgeassessmentmayhaveledtothelowmeanscore

andoveralllowvarianceinthescores.Thetaskhadstudentsansweraprompttolistallthatthey

knewabouttheAmericanRevolution.Theywerethengiventwomorechancestoanswerthesame

prompt.Thisformatmaynothaveelicitedallofthebackgroundknowledgestudentshadaboutthe

AmericanRevolution.Futureworkshouldtrytoelicitadditionalbackgroundknowledgeconnected

tothedisciplinaryliteraciesofsocialstudies(Shanahan&Shanahan,2008).Onepossiblemethod

wouldbetoprovidestudentswithblankconceptmapswithconceptsassociatedwiththeAmerican

216

Revolution.TherecouldbedifferentareasforleadersoftheRevolution,battles,causes,dates,etc.

Thismethodofassessmentmaybebetteralignedwiththetypesofknowledgeneededinthe

discipline.

DispositionsofOnlineReadingComprehensionMeasure.Thebiggestlimitationofthe

dispositionsofonlinereadingcomprehensionmeasure(DORC)wasthetreatmentofordinaldataas

intervaldataintheregressionmodels.TheDORCusedafivepointLikertscale.Labowitz(1967;

1970)arguesthatthevalueoftreatingordinaldata,suchastheDORCscores,asintervaldatain

multivariateanalysisoutweighstheriskofincludingordinaldatainregressionmodels.

Theoperationaldefinitionsoftheconstructsmeasuredbythedispositionsinstrumentalso

presentalimitationtothestudy.ItmaybethattheDORCfailedtocapturetheepistemologicaland

ontologicalstancesof“openmindsets”(Deschryver&Spiro,2010)necessaryforonlinereading

comprehension.Intermsoftherestrictednatureoftheinformationspace,forexample,itmaybe

thatdifferentmindsetsareneededtosucceedinarestrictedspacewhencomparedtoaless

restrictedspace.Thereforeareaderwhohastheabilitytoadoptmultipleperspectivesmaysucceed

atonlinereadingcomprehensiontasks.TheitemsincludedintheDORCdonotattempttomeasure

thesetypesofepistemologicalshiftsinthereader.

Futuremeasuresofdispositionsshouldtrytocaptureboththe“openmindsets”andthe

abilityofreaderstoshiftworldviewsandmindsetsbasedontheinformationspace.Thiswould

requirethecreationofatwo‐scaleinstrumentwithitemsindicatinganopenmindsetoraclosed

mindset.Thisinstrumentcouldthenbeappliedtomultiplemodels.

MissingDataandSampleSize

Anotherlimitationtothisstudyisthehighrateofmissingdata.Thesamplestartedwith

131students.Afterthreeoutlierswereremovedthefinalnequaled95students.Thishighrateof

attritioncouldleadtoimportantstudentsbeingleftoutofthemodel.Theattritionratewassimilar

acrossthethreeresearchsites.Thelargenumberofmissingdatawasduemainlytooneresearcher

217

beingresponsiblefortheadministrationofsomanyassessments.Itwasimpossibletoensureall

thestudentscompletedeachtaskandallabsenteeswereaccountedforinamanageabletimeframe.

Asmallernumberofstudentswerelosttocorrupteddata.

Themissingdataledtoasmallersamplesize.Inordertoensuretherewasenoughpowerin

themodelthetenstudentswhoconductedtheverbalprotocolportionofthestudywereincludedin

thefinalsample.Themeanscores,ontheORCA‐LRandtheORCA‐MRwerehigherforthe

qualitativeparticipantsthantheoverallmeanscoresontheORCA‐LRandtheORCA‐MRofstudents

whodidnotparticipateintheverbalprotocols.Itcouldbethatelicitingstudentstothinkaloud

aboutthestrategiestheyusedresultedinraisingtheirperformance(Gerjets,Kammerer,&Werner,

2010).

ResearcherBiasinQualitativeAnalysis

Astheonlypersoncodingandanalyzingdatathequalitativeportionofthisstudycould

havebeenthreatenedbymyinherentepistemologicalandontologicalviews.Mydesiretofind

differencesamongstudentstrategyusecouldhaveinfluencedmyresults.Inordertocontrolfor

thisacodebookwasdevelopedandanotherresearcherutilizedtoensureinter‐raterreliability.I

wasalsoabletoprovideself‐checksthroughoutthestudytoensurepatternsIfoundwere

consistentwiththedata.DuringtheanalysisprocessIwasabletorepeatedlyreturntothevideo

dataforself‐checks.Iwouldensurethatideaunitsonlyfitaspecificcode.Oncethecodeswere

collapsedintopatternsIreturnedtothevideodatatoensurethatpatternswereuniqueanddid

notoverlap.OncethepatternswerecollapsedintothemesIreturnedtothevideodatatoensure

thethemesencompassedallofthepatternstheycaptured.Theuseofaninter‐raterreliabilitycheck

andtherecursiveselfcheckshelpedtominimizethethreatofresearcherbias.

AddressingtheGrowingChallengesofOnlineReadingComprehension

Theresultsofthisstudycontributetobothresearchandpractice.Forresearch,theresults

informricherandmorecomplexmodelsofonlinereadingcomprehension.Thiswasoneofthefew

218

studiestoconsiderhowtherestrictednatureofonlinereadingcomprehensiontasksinfluences

studentperformance.Itsuggeststhatthenatureoftheinformationspaceforonlinereading

comprehensiondoes,indeed,makeadifference.

Recentcallstoaddressadolescentliteracyhaveemergedacrossthenation(CCAAL,2010;

CouncilofStateGovernments,2010;Snow&Biancarossa,2003).Infactthesecallshaveculminated

intotheCommonCoreStateStandardsplacingaheavyemphasisonpreparingstudentswhoare

“collegeandcareerready”(CCSS,2010,p.5).Whilethisisanoblecause,researchershavenoted

thatnewstatestandardsplacetooheavyanemphasisonofflinedefinitionsoftext(Draper,2012;

Leuetal.,2011).Infactthisstudyhelpstodemonstratethatanycalltoactiontoimprove

adolescentliteracymustplaceastrongemphasisononlinereadingcomprehensionandthenature

oftheinformationspace.

AsthesecallsforimprovedadolescentliteracyhaveemergedtheInternethasgrowninits

dominanceasthetextofchoiceforourstudents.Adolescentsnowspendmoretimereadingonline

thanoffline(KaiserFamilyFoundation,2005).FurthermoretheInternetistheprimarysource

whendoinghomeworktheInternetforhomework.Thereforeanyeffortstoimproveadolescent

literacymustalsoincludeeffortstoimprovemeaningmakinginonlinespaces,andtheliteracy

researchcommunityneedsmorerobustmodelsofcomprehensioninonlinecontexts.

Theseonlinespacesareverydiverse.Infactoneofthegreatestdemandsplacedonthe

readerwhenreadingonlineistheshiftinginformationspace(Leu,2002)Afterallwhenreading

onlinethesizeoftheinformationfieldconstantlyshiftsbasedonthetaskofthereader.Thisstudy

soughttoenrichthetheoreticalmodelsofonlinereadingcomprehensionbyexaminingshifting

informationspaces.

Inthisstudyitwasevidentthattherestrictednatureoftheinformationspaceaffectsonline

readingcomprehension.Usingbothquantitativeandqualitativemethodsthisstudydemonstrated

thatreadercharacteristicsbehavedifferentlywhentheinformationspaceisalteredduringonline

219

readingcomprehensiontasks.Innaturalsettingstheinformationspaceusedinonlinereading

comprehensiontaskswillalwayschange.Thusresearchersneedtodeveloptheoreticalmodelsof

readingthataccountforthisconstantflux(Hartmanetal.,2010;Leuetal.,2004).

Itisalsoclearfromthisstudythatlongheldassumptionsaboutreadingcomprehension

mayhavetobereconsidered.Specificallytherolebackgroundknowledgeplaysindetermining

comprehensionmaybelessenedinonlinespaces.Inthisstudyscoresonacriticalevaluation

measure,notbackgroundknowledge,werethestrongestpredictorofperformanceonboththeless

restrictedmodelofonlinereadingcomprehensionandthemorerestrictedmodelofonlinereading

comprehension.Thusitisclearthatcriticalevaluationskillsarecentraltosuccessfulonlinereading

comprehension(Coiro,2003).

Unfortunatelystudieshaveconsistentlyshownthatstudentsareillequippedtoevaluate

onlineinformation(Bennett,Maton,&Kervin,2008;Goldmanetal.,2012;Jewitt,2008;Johnson&

Kaye,1998;Livingstone,2004;Leuetal.,2007bRieh&Belkin,1998;).Boththequantitativeand

qualitativedataofthisdutysupportthisconclusion.Ifwearetoaddressadolescentliteracyitis

clearthataconcertedeffortisneededtoteachstudentscriticalevaluationskills.Thisissuewillbe

paramountforbotheducatorsandresearchers.

Forpractice,theresultsinformteacherschargedwithteachingliteracyinaconstantly

evolvingworld,oneinwhichtheInternetisincreasinglyimportanttobothreadingandlearning

(Snow,2002).Thisstudyhighlightsthatteachersneedtoconsiderfourcriticalareasforonline

readingcomprehension:searchstrategies,navigationalstrategies,evaluationstrategiesand

communicationstrategies.Whilebuildinginstructionalroutinesthatfocusonthesestrategiesthis

studyalsosuggeststhatteachersbuildinopportunitiestomeasureanddevelopdispositions

centraltoonlinereadingcomprehension.

Ifwearetomeetthecalltoimproveadolescentliteracyonlinereadingcomprehension

cannotbeignore.Itisacriticalissuefacingliteracyresearchersandeducators.AstheInternet’s

220

globalreachcontinuestospreadwemusthavestudentspreparedtomakemeaningwithdigital

texts.

221

References

Afflerbach, P. P. (1990). The influence of prior knowledge on expert readers’ main idea

construction strategies. Reading Research Quarterly, 25, 31-46.

Afflerbach, P. (2002). The use of think-aloud protocols and verbal reports as research

methodology. In M. Kamil, P. Mosenthal, P.D. Reason, & R. Barr, (Eds.), Methods of

literacy research (pp. 87-103). Mahwah, NJ: Erlbaum.

Agosto, D. (2002). Bounded rationality and satisficing in young people’s web-based decision

making. Journal of the American Society for Information Science and Technology. 53,

16-27.

Alexander, P. A., & Fox, E. (2004). A historical perspective on reading research and practice. In

R. B. Ruddell, N. J. Unrau (Eds.), Theoretical models and processes in reading (5th ed.,

pp. 33-68). Newark, NJ: International Reading Association.

Alexander, P. A., & Fox, E. (2011). Adolescents as readers. In M. Kamil, P. D. Pearson, E.B.

Moje, & P. Afflerbach (Eds.), Handbook of reading research (Vol. 4, pp. 157-176).

New York, NY: Routlage.

Allen, B. (1992). Cognitive differences in end user searching of a CD-ROM index. In N. J.

Belkin, P. Ingwersen, & A. M. Pejtersen (Eds.), Proceedings of the 15th Annual

International ACM SIGIR Conference on Research and Development in Information

Retrieval (pp. 298-309). ACM, New York, NY, USA, DOI=10.1145/133160.133212

http://doi.acm.org/10.1145/133160.133212

Allen, I.E., & Seaman, J. (2010). Class differences: Online education in the United States, 2010.

Retrieved from The Sloan Consortium website:

http://sloanconsortium.org/sites/default/files/class_differences.pdf

222

Alvermann, D. E. (2004). Media, information communication technologies, and youth literacies:

A cultural studies perspective. American Behavioral Scientist, 48, 78-83. doi:

10.1177/0002764204267271.

Anderson, R. C., & Pearson, P. D. (1984). A schema-theoretic view of basic processes in reading

comprehension. In P. D. Pearson (Ed.), Handbook of Reading Research (pp. 255-291).

New York: Longman.

Anderson, R. C., Reynolds, R. E., Schallert, D. L., & Goetz, E. T. (1977). Frameworks for

comprehending discourse. American Educational Research Journal, 14, 367-381.

Avons, S. E., Wragg, C. A., Cupples, L., & Lovegrove, W. J. (1998). Measures of phonological

short-term memory and their relationship to vocabulary development. Applied

Psycholinguistics, 19, 583-602.

Baker, F. B. (1985). The basics of item response theory. Portsmouth, NH: Heinemann.

Baker, L., & Wigfield, A. (1999). Dimensions of children’s motivation for reading and their

relations to reading activity and reading achievement. Reading Research Quarterly, 34,

452-477. doi: 10.1598/RRQ.34.4.4

Beauchamp, J. J., & Olson, J. S. (1973). Corrections for bias in regression estimates after

logarithmic transformation. Ecology, 54, 1403-1407.

Berners-Lee, T. (1989). Information Management: A Proposal, CERN, March 1989. Retrieved

from http://www.w3.org/History/1989/proposal.html.

Biancarosa, C., & Snow, C. E. (2006). Reading next—A vision for action and research in middle

and high school literacy: A report to Carnegie Corporation of New York (2nd ed.).

Washington, DC: Alliance for Excellent Education.. Retrieved from

http://www.all4ed.org/files/ReadingNext.pdf

223

Bilal, D. (2000). Children's use of the Yahooligans! Web search engine: I. Cognitive, physical,

and affective behaviors on fact-based search tasks. Journal of the American Society for

Information Science, 51, 646-665.

Bilal, D. (2001). Children’s use of the Yahooligans! Web search engine: II. Cognitive and

physical behaviors on research tasks. Journal of the American Society for Information

Science, 52, 118-136.

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy

of educational objectives: Handbook I: Cognitive domain. New York, NY: Longman.

Bogdan, R. C., & Biklen, S. K. (2006). Qualitative research for education: An introduction to

theories and methods (5th ed.). Boston, MA: Allyn & Bacon.

Braasch, J. L., Lawless, K. A., Goldman, S. R., Manning, F. H., Gomez, K. W., & MacLeod, S.

M. (2009). Evaluating search results: An empirical analysis of middle school students'

use of source attributes to select useful sources. Journal of Educational Computing

Research, 41, 63-82.

Bråten, I., Strømsø, H.I., & Britt, M.A. (2009). Trust matters: Examining the role of

source evaluation in students' construction of meaning within and across multiple

texts. Reading Research Quarterly, 44, 6-28. doi: 10.1598/RRQ.44.1.1

Bråten, I., Strømsø, H. I., & Samuelstuen, M. (2008). Are sophisticated students always better?

The role of topic-specific personal epistemology in the understanding of multiple

expository texts. Contemporary Educational Psychology, 33, 814-840. doi:

10.1016/j.cedpsych.2008.02.001.

Brem, S., Russell, J., & Weems, L. (2001). Science on the Web: Student evaluations of scientific

arguments. Discourse Processes, 32, 191-213. doi: 10.1207/S15326950DP3202&3_06.

224

Britt, M. A., & Gabrys G. (2002) Implications of document-level literacy skills for Web site

design. Behavior Research Methods, Instruments, & Computers, 34, 170-176.

Britt, M. A., Perfetti, C. A., Sandak, R., & Rouet, J.-F. (1999). Content integration and source

separation in learning from multiple texts. In S. R. Goldman, A. C. Graesser, & P. v. d.

Broek (Eds.), Narrative comprehension, causality, and coherence: Essays in honor of

Tom Trabasso. Mahwah, NJ: Erlbaum.

Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning.

Educational Researcher, 18(1), 32-42. doi: 10.3102/0013189X018001032

Campbell, J. R. (2005). Single instrument, multiple measures: Considering the use of multiple

item formats to assess. In S. G. Paris & S. S. Stahl (Eds.), Children's reading

comprehension and assessment (pp. 347-368). Mahwah, NJ: Erlbaum.

Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for

research. Belmont, CA: Wadsworth.

Campione, J. C., Shapiro, A. M., & Brown, A. L. (1995). Forms of transfer in a community of

learners: Flexible learning and understanding. In A. McKeough, J. Lupart, & A. Marini

(Eds.), Teaching for transfer: Fostering generalization in learning (pp. 35-68). Mahwah,

NJ: Erlbaum.

Carnegie Council on Advancing Adolescent Literacy. (2010). Time to act: An agenda for

advancing adolescent literacy for college and career success. New York: Carnegie

Corporation of New York.

Carr, M. (1999). Being a learner: Five learning dispositions for early childhood. Early Childhood

Practice, 1, (1). 81-99.

225

Carr, M., & Claxton, G. (2002). Tracking the development of learning dispositions. Assessment

in Education: Principles, Policy & Practice, 9, 9-37. doi: 10.1080/09695940220119148

Carroll, J. B. Psychometric tests as cognitive tasks: A new "structure of intellect." In L. B.

Resnick (Ed.), The nature of intelligence. Hillsdale, NJ : Erlbaum, 1976.

Cervetti, G., Pardales, M. J., & Damico, J. S. (2001, April). A tale of differences: Comparing the

traditions, perspectives, and educational goals of critical reading and critical literacy.

Reading Online, 4(9). Retrieved from

http://www.readingonline.org/articles/art_index.asp?HREF=/articles/cervetti/index.html

Chiesi, H.L., Spilich, G.J., & Voss, J.F. (1979). Acquisition of domain-related knowledge in

relation to high and low domain knowledge. Journal of Verbal Learning and Verbal

Behavior, 18, 257-273.

Coiro, J. (2003). Rethinking comprehension strategies to better prepare students for critically

evaluating content on the Internet. [Electronic Version]. New England Reading

Association Journal. Retrieved from http://ctell1.uconn.edu/coiro/nera2003.pdf

Coiro, J. (2011). Predicting reading comprehension on the Internet: Contributions of offline

reading skills, online reading skills, and prior knowledge. Journal of Literacy Research,

43, 352-392. doi: 10.1177/1086296X11421979

Coiro, J., Castek, J., & Guzniczak, L. (2011). Uncovering online reading comprehension

processes: Two adolescents reading independently and collaboratively on the Internet. In

P.J. Dunston, L.B. Gambrell, K. Headley, S.K. Fullerton, & P.M. Stecker (Eds.), Sixtieth

yearbook of the Literacy Research Association (pp. 354–369).

226

Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by

sixth-grade skilled readers to search for and locate information on the Internet.. Reading

Research Quarterly, 42, 214–257.

Coiro, J., Knobel, M., Lankshear, C., & Leu, D. J. (2008). Central issues in new literacies and

new literacies research. Lawrence Erlbaum Associates, Taylor & Francis Group.

Retrieved from http://eprints.jcu.edu.au/7369/

Coiro, J., Knobel, M., Lankshear, C. & Leu, D. (2008). Central issues in new literacies and new

literacies research. In J. Coiro, M. Knobel, C. Lankshear, & D. Leu (Eds.), Handbook of

research on new literacies (pp. 25-32). Mahwah, NJ: Erlbaum.

Collins, K. M. T., Onwuegbuzie, A. J., & Sutton, I. L. (2006). A model incorporating the

rationale and purpose for conducting mixed methods research in special education and

beyond. Learning Disabilities: A Contemporary Journal, 4(1), 67-100.

Conley, M.W. (2008). Cognitive strategy instruction for adolescents: What we know about the

promise, what we don't know about the potential. Harvard Educational Review, 78, 84-

106.

Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for

field settings. Chicago, IL: Rand McNally.

Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed

methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed

methods in social and behavioral research (pp. 209-240). Thousand Oaks, CA: Sage.

Cronbach, L. J. , Gleser, C.G., Nanda, H., & Rajaratnam, N. (1972). The dependability of

behavioral measurements: Theory of generalizability for scores and profiles. New York,

NY: John Wiley.

227

Damico, J., & Baildon, M. (2007). Examining ways readers engage with websites during think-

aloud sessions. Journal of Adolescent & Adult Literacy. 51, 254-263. doi:

10.1598/JAAL.51.3.5

Davis, E. A. (2000). Scaffolding students' knowledge integration: Prompts for reflection in KIE.

International Journal of Science Education, 22, 819-837.

DeSchryver, M. & Spiro, R. (2008). New forms of deep learning on the Web: Meeting the

challenge of cognitive load in conditions of unfettered exploration. In R. Zheng (Ed.),

Cognitive effects of multimedia learning (pp. 134-152). Hershey, PA: IGI Global.

DeSchryver, M. & Spiro, R. (in preparation). CFT on the Web: Illustrated theoretical cognitive

and metacognitive processes for advanced nonlinear knowledge acquisition of ill-

structured topics.

D'Mello, S., & Graesser, A. (2009). Automatic detection of learner's affect from gross body

language. Applied Artificial Intelligence, 23, 123-150.

Dwyer, B. (2010). Scaffolding Internet reading: A study of a disadvantaged school community in

Ireland (Doctoral dissertation, University of Nottingham, Nottingham, UK). Retrieved

from http://etheses.nottingham.ac.uk/2426/1/523046.pdf.

Eagleton, M. B. & Dobler, E. (2007). Reading the Web: Strategies for Internet inquiry. New

York, NY: Guilford.

Ekstrom, R. B., French, J. W., & Harman, H. H. (1979). Cognitive factors: Their identification

and replication. Multivariate Behavioral Research Monographs, 79(2), 3-84.

Ennis, RH. (1962). A concept of critical thinking. Harvard Educational Review, 32, 81-111.

228

Ennis, R.H. (1987). A taxonomy of critical thinking dispositions and abilities. In J. B. Baron &

R.J. Sternberg (Eds.), Teaching thinking skills: Theory and practice (pp. 9–26), New

York, NY: W.H. Freeman.

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.)

Cambridge, MA: MIT Press.

Fincher-Kiefer, R., Post, T. A., Greene, T.R., & Voss, J.E. (1988). On the role prior knowledge

and task demands in the processing of text. Journal of Memory and Language, 27, 416-

428.

Fitzgerald, M. A. (1999) Evaluating information: An information literacy challenge. School

Library Media Research (Vol. 2). Retrieved from

http://www.ala.org/ala/mgrps/divs/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume2

1999/vol2fitzgerald.cfm

Flanagin, A. J., & Metzger, M. J. (2000). Perceptions of Internet information credibility.

Journalism and Mass Communication Quarterly, 77, 515–540.

doi: 10.1177/107769900007700304

Fogg, B.J., Marshall, J., Laraki, O., Osipovich, A., Varma, .Fang, N. , . . . Treinen, M. (2001).

What makes web sites credible? A report on a large quantitative study. Proceedings of

the SIGCHI Conference on Human Factors in Computing Systems, Seattle, WA. 61-68,

doi: 10.1145/365024.365037

Fogg, B. J., Soohoo, C., Danielson, D. R., Marable, L., Stanford, J., & Tauber, E. R. (2003).

How do users evaluate the credibility of Web sites? A study with over 2,500 participants.

Proceedings of the Conference on Designing for User Experiences. Retrieved from

http://dl.acm.org/citation.cfm?id=997097

229

Fox, S., & Rainie, L. (2002). Vital decisions: How Internet users decide what information to trust

when they or their loved ones are sick. Washington, DC: Pew Internet & American Life

Project. Retrieved from

http://www.pewinternet.org/pdfs/PIP_Vital_decisions_May2002.pdf

Freebody, P., & Luke, A. (1990). Literacies programs: Debates and demands in cultural context.

Prospect: Australian Journal of TESOL, 5(7), 7-16.

Freire, P. (1970). The adult literacy process as cultural action for freedom. Harvard Educational

Review, 40, 205-225.

Fritch, J. W., & Cromwell, R. L. (2001). Evaluating Internet resources: Identity, affiliation, and

cognitive authority in a networked world. Journal of the American Society for

Information Science and Technology, 52,, 499–507.

Fritch, J. W., & Cromwell, R.L. (2002). Delving deeper into evaluation: Exploring cognitive

authority on the Internet. Reference Services Review, 30, 242–254.

Gable, R. K., & Wolf, M. B. (1993). Instrument development in the affective domain: Measuring

attitudes and values in corporate and school settings. (2nd ed.). Norwell, MA: Kluwer

Academic Publishers.

Gates, A. I. (1931). What do we know about optimum lengths of lines in reading?. The Journal

of Educational Research, 23, 1-7.

Gathercole, S. E., & Baddeley, A. D. (1993). Working memory and language. East Sussex, UK:

Psychology Press.

Gathercole, S. E., Service, E., Hitch, G. J., Adams, A -M., & Martin, A. J. (1999). Phonological

short‐term memory and vocabulary development: Further evidence on the nature of the

relationship. Applied Cognitive Psychology, 13, 65-77.

230

Gee, J. P. (2004). Situated Language and Learning: A Critique of Traditional Schooling.

London: Routledge.

Gee, J. P., & Green, J. L. (1998). Discourse analysis, learning, and social practice: A

methodological study. Review of Research in Education, 23(1), 119-169.

Goldman, S. R., Braasch, J. L., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012).

Comprehending and learning from Internet sources: Processing patterns of better and

poorer learners. Reading Research Quarterly, 47, 356-381. doi: 10.1002/RRQ.027

Goldstein, W. M., & Hogarth, R. M. (Eds.). (1997). Research on judgment and decision making:

Currents, connections, and controversies. New York: Cambridge University Press.

Graesser, A. C., Wiley, J., Goldman, S. R., O’Reilly, T., Jeon, M., & McDaniel, B. (2007).

SEEK Web tutor: Fostering a critical stance while exploring the causes of volcanic

eruption. Metacognition and Learning, 2, 89-105.

Gross, E. (2004). Adolescent Internet use: What we expect, what teens report. Journal of Applied

Developmental Psychology, 25, 633-649. doi: 10.1016/j.appdev.2004.09.005

Guthrie, J. T., & Wigfield, A. (Eds.). (1997). Reading engagement: Motivating readers through

integrated instruction. Newark, DE: International Reading Association.

Guthrie, J. T., Wigfield, A., & Perencevich, K. C. (Eds.). (2004). Motivating reading

comprehension: Concept-Oriented reading instruction. Mahwah, NJ: Erlbaum.

Harris, F. J. (2008). Challenges to teaching credibility assessment in contemporary schooling.. In

M. J. Metzger & A. J. Flanigan (Eds.), Digital Media, Youth, and Credibility, (pp. 155-

180). Cambridge, MA: MIT Press.. doi: 10.1162/dmal.9780262562324.155

Hartman, D.K., Morsink, P.M., & Zheng, J. (2010). From print to pixels: The evolution of

cognitive conceptions of reading comprehension. In E.A. Baker (Ed.), Multiple

231

Perspectives on New Literacies Research and Instruction (pp. 131-164). New York:

Guilford.

Herder, E., & Juvina, I. (2004). Discovery of individual user navigation styles. Proceedings of

the AH 2004 Workshop Eindhoven, The Netherlands. pp. 40-49. Retrieved from

http://www.dcs.bbk.ac.uk/%7Egmagoulas/AH2004_Workshop/Proceedings.htm

Henry, L. A. (2006). SEARCHing for an Answer: The critical role of new literacies while

reading on the Internet. The Reading Teacher, 59, 614-627. doi: 10.1598/RT.59.7.1

Hill, J.R., & Hannafin, M. J. (1997). Cognitive strategies and learning from the World Wide

Web. Educational Technology Research & Development, 45(4), 37-64.

Hirsch, E. D. (2003). Reading comprehension requires knowledge – of words and the world:

Scientific insights into the fourth-grade slump and stagnant reading comprehension.

American Educator, 27(1), 10-29.

Hoffman, J. L., Wu, H-K., Krajcik, J. S., & Soloway, E. (2003). The nature of middle school

learners' science content understandings with the use of on‐line resources. Journal of

Research in Science Teaching, 40, 323-346.

Hogarth, R. M. (1987). Judgment and choice: The psychology of decision. New York, NY: John

Wiley.

Johnston, P.H. (1984). Prior knowledge and reading comprehension test bias. Reading Research

Quarterly, 19, 219-239.

Judd, V. C., Farrow, L. I., & Tims, B. J. (2006). Evaluating public web site information: A

process and an instrument. Reference Services Review, 34, 12-32. doi:

10.1108/00907320510631571

232

Katz, L.G. (1993). Dispositions: Definitions and implications for early childhood practices.

Perspectives from ERIC/EECE: A monograph series, no. 4. Urbana, IL: ERIC

Clearinghouse on ECCE. Retrieved from http://www.eric.ed.gov/PDFS/ED360104.pdf

Katz, J., & Rice, R. (2002). Social consequences of Internet use: Access, involvement, and

interaction. Cambridge MA: MIT Retrieved from

http://www.citeulike.org/user/maxsenges/article/228698

Kay, K. (2009) Middle schools preparing young people for 21st century life and work. Middle

School Journal, 40(5), 41-54. Retrieved from

http://cesa7ita2009.pbworks.com/f/middle%20schools%20preparing%20young%20peopl

e%20for%20work%20in%20the%2021st%20century.pdf

Kiili, C., Laurinen, L., & Marttunen, M. (2008). Students evaluating Internet sources: From

versatile evaluators to uncritical readers. Journal of Educational Computing Research,

39, 75-95. doi: 10.2190/EC.39.1.e

Kiili, C., Laurinen, L., Marttunen, M., & Leu, D. (2011). Working on understanding:

collaborative reading on the Web. Manuscript submitted for publication.

Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction-

integration model. Psychological Review, 95, 163-182.

Kintsch, W. (1998) Comprehension: A paradigm for cognition. New York: Cambridge

University Press.

Kintsch, W., & van Dijk, T. A. (1978). Toward a model of text comprehension and production.

Psychological Review, 85, 363-394.

Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into

practice, 41, 212-218.

233

Kress, G. (2000). Multimodality. In B. Cope & M. Kalantzis (Eds.). Multiliteracies: Literacy

learning and the design of social futures (pp. 179-200). New York, NY: Routlage.

Kuiper, E. & Volman, M. (2008). The Web as a source of information for students in K-12

education. In J. Coiro, M. Knobel, D. Leu, & C. Lankshear (Eds.). Handbook of Research

on New Literacies (pp. 241-266). Mahwah, NJ: Erlbaum.

Kuiper, E., Volman, M., & Terwel, J. (2005). The Web as an information resource in K-12

education: Strategies for supporting students in searching and processing information.

Review of Educational Research, 75, 285-328.

Labbo, L. D., & Reinking, D. (1999). Negotiating the multiple realities of technology in literacy

research and instruction. Reading Research Quarterly, 34, 478-492. doi:

10.1598/RRQ.34.4.5.

Lankshear, C. & Knobel, M. (1998, April). Critical literacies and new technologies. Paper

presented at the Annual Meeting of the American Educational Research Association, San

Diego, CA.

Lankshear, C. & Knobel, M. (2006). New literacies: Everyday practices and classroom learning

(2nd ed.). Maidenhead, UK: Open University Press.

Lankshear, C., & Knobel, M. (2007a). Researching New Literacies : Web 2 .0 Practices and

Insider Perspectives, 1-31.

Lankshear, C., & Knobel, M. (2007b) Researching new literacies: Web 2.0 practices and insider

perspectives. E-Learning and Digital Media, 4, 224-240. Retrieved from

http://dx.doi.org/10.2304/elea.2007.4.3.224

Lawless, K. A., & Schrader, P. G. (2008). Where do we go now? Understanding research on

navigation in complex digital environments. In J. Coiro, M. Knobel, D. Leu, & C.

234

Lankshear (Eds.). Handbook of Research on New Literacies (pp. 267-296). Mahwah, NJ:

Erlbaum.

Lee, C. D., Spratley, A. (2010). Reading in the disciplines: The challenges of adolescent literacy.

New York, NY: Carnegie Corporation of New York.

Lemke, E. A., Klausmeier, H. J., & Harris, C. W. (1967). Relationship of selected cognitive

abilities to concept attainment and information processing. Journal of Educational

Psychology, 58, 27-35.

Lenhart, A., Horrigan, J., & Fallows, D. (2004). Content Creation Online: 44% of US Internet

users have contributed their thoughts and their files to the online world. Pew Internet &

American Life Project. Washington, DC.

Leslie, L. & Caldwell, J. (1995). Qualitative Reading Inventory – II. Glenview, IL: Harper

Collins.

Leu, D. J. (2000). Literacy and technology: Deictic consequences for literacy education in an

information age. In M.L. Kamil, P. Mosenthal, P.D. Pearson, & R. Barr (Eds.),

Handbook of reading research: Volume III (pp. 743-770). Mahwah, NJ:

Erlbaum.

Leu, D. J., Castek, J., Hartman, D., Coiro, J., Henry, L., Kulikowich, J., & Lyver, S. (2005).

Evaluating the development of scientific knowledge and new forms of reading

comprehension during online learning. Final report presented to the North Central

Regional Educational Laboratory/Learning Point Associates. Retrieved from

http://www.newliteracies.uconn.edu/ncrel_files/FinalNCRELReport.pdf

Leu, D. J., Kinzer, C. K., Coiro, J., & Cammack, D. (2004). Towards a theory of new literacies

emerging from the Internet and other ICT. In R. B. Ruddell & N. J. Unrau (Eds.),

235

Theoretical models and processes of reading, (5th ed., pp. 1570-1613). Newark, NJ:

International Reading Association.

Leu, D. J., Reinking, D., Hutchinson, A., McVerry, J. G., Robbins, K., Rogers, A., Malloy, J.,

O’Byrne, W. I., & Zawilinski, L. (2008, December). The TICA project: Teaching the

new literacies of online reading comprehension to adolescents: An alternative symposium

conducted at the National Reading Conference, Orlando, FL.

Leu, D. J., O’Byrne, W. I., Zawilinski, L., McVerry, J. G., & Everett-Cacopardo, H. (2009).

Expanding the new literacies conversation. Educational Researcher, 38, 264-269. doi:

10.3102/0013189X09336676

Leu, D. J., Zawilinski, L., Castek, J., Banerjee, M., Housand, B., Liu, Y., et al. (2007). What is

new about the new literacies of online reading comprehension? In L. Rush, J. Eakle, & A.

Berger, (Eds.). Secondary school literacy: What research reveals for

classroom practices. (37-68). Urbana, IL: National Council of Teachers of English.

Library of Congress. (2008). About the library. Retrieved from http://www.loc.gov/about/

Luke, A. (2000). Critical literacy in Australia: A matter of context and standpoint. Journal of

Adolescent & Adult Literacy, 43, 448-461.

Manderino, M. L. (2011). Reading across multiple multimodal sources in historical

inquiry (Doctoral dissertation, University of Michigan). Retrieved from

http://indigo.uic.edu/bitstream/handle/10027/9605/Manderino_Michael.pdf?sequence=1

Masoura, E. V., & Gathercole, S. E. (1999). Phonological short-term memory and foreign

language learning. International Journal of Psychology, 34, 383-388.

doi:10.1080/002075999399738

236

McKenzie, J. F., Wood, M. L., Kotecki, J. E., Clark, J. K., & Brey, R. A. (1999). Establishing

content validity: Using qualitative and quantitative steps. American Journal of Health

Behavior, 23, 311-318.

McWilliams, J., & Clinton, C. (2013). Reimagining and Reinventing the English classroom in

the digital age. In H. Jenkins & W. Kelley, with K. Clinton, J. McWilliams, R. Pitts-

Wiley, & E. B. Reilly (Eds). Reading in a participatory culture: Remixing Moby Dick in

the English classroom (pp. 185-196). New York, NY: Teachers College Press.

McVerry, J. G., O’Byrne, W. I., & Robbins, K. (2009, April). Validating instruments used in the

TICA Project. Presented at the Annual Meeting of the American Educational Research

Association Annual Meeting, San Diego, CA.

Mellor, B., & Patterson, A. (2004). Poststructuralism in English classrooms: Critical literacy and

after. International Journal of Qualitative Studies in Education, 17, 85–102.

Merriam, S.B. (1988). Case study research in education: A qualitative approach. San Francisco,

CA: Jossey-Bass.

Metzger, M. (2007). Making sense of credibility on the Web: Models for evaluating online

information and recommendations for future research. Journal of the American Society for

Information Science and Technology, 58, 2078-2091.

Metzger, J. M., Flanagin, J. A., & Zwarun, L. (2003). College student Web use, perceptions of

information credibility, and verification behavior. Computers & Education, 41, 271-290.

doi: 10.1016/S0360-1315(03)00049-6

Mills, E. (2005, October 8). Google ETA? 300 years to index the world's info. CNET. Retrieved

from http://news.cnet.com/Google-ETA-300-years-to-index-the-worlds-info/2100-

1024_3-5891779.html

237

Miniwatts Marketing Group. (2012). Internet World Stats. Retrieved from

http://www.internetworldstats.com/

Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A

framework for integrating technology in teacher knowledge. Teachers College Record,

108, 1017-1054.

Moje, E. B. (2009). A call for new research on new and multi-literacies. Research in the

Teaching of English, 43, 348-362.

Murray, D. E., & McPherson, P. (2006). Scaffolding instruction for reading the Web. Language

Teaching Research, 10, 131-156.

New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard

Educational Review, 66, 60-92.

Norris, P. (2001). Digital divide?: Civic engagement, information poverty, and the Internet

worldwide. Cambridge University Press. Retrieved from

http://portal.acm.org/citation.cfm?id=51619.

O’Byrne, W.I. (2011). Facilitating critical evaluation skills through content creation:

Empowering adolescents as readers and writers of online information. Unpublished

doctoral dissertation, University of Connecticut, Storrs.

O'Byrne, W. I., & McVerry, J. G. (2009). Measuring the dispositions of online reading

comprehension: A preliminary validation study. In K. Leander, D. Rowe, D. Dickson, M.

Hundley, R. Jiménez, & V. Risko (Eds.), The 57th National Reading Conference

Yearbook (pp.58-84). Oak Creek, WI: National Reading Conference.

O'Hanlon, N. (2002). Net knowledge: Performance of new college students on an Internet skills

proficiency test. The Internet and Higher Education, 5, 55-66.

238

Onwuegbuzie, A. J., & Leech, N. L. (2006). Linking Research Questions to Mixed Methods Data

Analysis procedures, The Qualitative Report, 11, 474-498.

Osborne, J. W., Christensen, W. R., & Gunter, J. (April, 2001). Educational psychology from a

statistician’s perspective: A review of the power and goodness of educational

psychology research. Paper presented at the national meeting of the American

Education Research Association, Seattle, WA.

Osborne, J., & Waters, J. (2002). Four assumptions of multiple regression that researchers

should always test. Practical Assessment, Research & Evaluation, 8(2). Retrieved

from http://PAREonline.net/getvn.asp?v=8&n=2

Palincsar, A. S., Magnusson, S. J., Pesko, E., & Hamlin, M. (2005). Attending to the nature of

subject matter in text comprehension Children's Reading Comprehension and

Assessment, 257. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension

and assessment (pp. 257–278). Mahwah, NJ: Erlbaum.

Paris, S. G., & Stahl, S. A. (2005). Children's reading comprehension and assessment..

Routledge. Mahwah, NJ: Erlbaum.

http://books.google.com/books?hl=en&lr=&id=hhLS9ax8IocC&pgis=1

Paris, S. G., Wasik, B. A., & Turner, J. C. (1991). The development of strategic readers. In R.

Barr, M. L. Kamil, P. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading

research:Volume II (pp. 609-640). White Plains, NY: Longman.

Partnership for 21st Century Skills (2008). 21st century skills, education and competitiveness: A

resource and policy guide. Tuczon, AZ: Author. Retrieved from

http://www.p21.org/storage/documents/21st_century_skills_education_and_competitiven

ess_guide.pdf

239

Pearson, P. D. (1982). A context for instructional research on reading comprehension (Technical

Report No. 230) Cambridge, MA: Bolt, Beranek and Newman.

Pearson, P. D., Hansen, J., & Gordon, C. J. (1979). The effect of prior knowledge on young

children's comprehension of explicit and implicit information. Journal of Reading

Behavior, 11, 201-209.

Perfetti, C. A., Britt, M. A., Rouet, J.-F., Georgi, M. C., & Mason, R. A. (1994). How students

use texts to learn and reason about historical Uncertainty. In J. F. Voss & M. Carretero

(Eds.), Cognitive and instructionalpProcesses in history and social sciences (pp. 257-

284). Mahwah, N.J.: Lawrence Erlbaum Associates.

Peterson, R. A. (1994). A meta-analysis of Cronbach's coefficient alpha. Journal of Consumer

Research, 381-391.

Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis. The use of

factor analysis for instrument development in health care research. Thousand Oaks, CA:

Sage.

Pressley, M., & Afflerbach, P. (1995). Verbal protocols for reading: The nature of

constructively responsive reading. Hillsdale, NJ: Erlbaum.

Princeton Survey Research Associates International. (2005). Leap of faith: Using the Internet

despite the dangers. Princeton, NJ: Author. Retrieved from

http://www.consumerwebwatch.org/pdfs/princeton.pdf

Quintana, C., Zhang, M., & Krajcik, J. (2005). A framework for supporting metacognitive

aspects of online inquiry through software-based scaffolding. Educational Psychologist,

40, 235–244.

240

Rachlin, H. (1989). Judgment, decision, and choice: A cognitive/behavioral synthesis. New

York, NY: W. J. Freeman.

Rideout, V. J., Foehr, U. G., & Roberts, D. F. (2010). Generation M2: Media in the lives of 8- to

18-year-olds: A Kaiser family foundation study. Retrieved from the Henry J. Kaiser

Family Foundation website: http://www.kff.org/entmedia/upload/8010.pdf

Rieh,S.Y. (2002). Judgment of information quality and cognitive authority in the Web. Journal

of the American Society for Information Science and Technology, 53, 145–161.

Rieh, S. Y., & Belkin, N. J. (1998). Understanding judgment of information quality and

cognitive authority in the WWW. In C. M. Preston (Ed.), Proceedings of the 61st Annual

Meeting of the ASIS (pp. 279-289). Silver Spring, MD: American Society for Information

Science.

Robinson, H. M. (1964). Developing critical readers. In R. G. Stauffer (Ed.), Dimensions of

critical reading: Proceedings of the Annual Education and Reading Conferences, (pp. 1-

12). Newark, DE: University of Delaware.

Rouet, J., Britt, M. A., Mason, R. A., & Perfetti, C. A. (1996). Using Multiple Sources of

Evidence to Reason About History. Journal of Educational Psychology, 88, 478-493.

Rouet, J. F., Ros, C., Goumi, A., Macedo-Rouet, M., & Dinet, J. (2011). The influence of surface

and deep cues on primary and secondary school students' assessment of relevance in Web

menus. Learning and Instruction, 21, 205-219.

Rubio, D. M., Berg-Weger, M., Tebb, S. S., Lee, E. S., & Rauch, S. (2003). Objectifying content

validity: Conducting a content validity study in social work research. Social Work

Research, 27, 94-104.

Russell, D. (1961). Children learn to read (2nd ed.). Boston, MA: Ginn.

241

Sadler, D. R. (2002). Learning dispositions: Can we really assess them? Assessment in

Education: Principles, Policy & Practice, 9, 45-51.

Sanchez, C.A., Wiley, J., & Goldman, S. R. (2006). Teaching students to evaluate source

reliability during Internet research tasks. In S. A. Barab, K.E. Hay, & D.T. Hickey (Eds.).

Proceedings of the seventh international conference on the learning sciences (pp. 662-

666). Mahwah, NJ: Erlbaum.

Schmar-Dobler, E. (2003). Reading on the Internet: The link between literacy and technology.

Journal of Adolescent & Adult Literacy. [Online Serial]. Retrieved from

http://www.readingonline.org/newliteracies/jaal/9-03_column

Shanahan, T., & Shanahan, C. (2008). Teaching disciplinary literacy to adolescents: Rethinking

content-area literacy. Harvard Education Review. 78, 40-61.

Shirky, C. (1995). Voices from the Net. Emeryville, CA: Ziff-Davis Press.

Snow, C. (2002). Reading for understanding: Toward an R&D program in reading

comprehension. New York: Rand Corporation.

Snow, C. E., & Biancarosa, G. (2003). Adolescent literacy and the achievement gap: What do we

know and where do we go from here?. New York, NY: Carnegie Corporation.

Soper, D. S. (2010). The Free Statistics Calculators Website. Retrieved from

http://www.danielsoper.com/statcalc/

Spache, G. D. (1964). Reading in the elementary school. Boston, MA: Allyn & Bacon.

Spilich, G. J., Vesonder, G. T., Chiesi, H. L., & Voss, J. F. (1979). Text processing of domain-

related information for individuals with high and low domain knowledge. Journal of

Verbal Learning and Verbal Behavior 18, 275-290.

242

Spiro, R. (2004). Principled pluralism for adaptive flexibility in teaching and learning to read. In

R. B. Ruddell & N. Unrau (Eds.), Theoretical Models and Processes of Reading (pp. 654-

659). Newark, DE: International Reading Association.

Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson, D. K. (1988). Cognitive flexibility

theory: Advanced knowledge acquisition in ill-structured domains (Tech. Rep. No. 441).

Urbana-Champaign, IL: University of Illinois, Center for the Study of Reading. Retrieved

from http://eric.ed.gov

Spiro, R. J., Feltovich, P. J., Jacobson, M. I., & Coulson, R. L. (1991). Cognitive flexibility,

constructivism, and hypertext: Random access instruction for advanced knowledge

acquisition in ill-structured domains. Educational Technology, 35, 24-33.

Spiro, R. J., Feltovich, P. L., Jacobson, M. J., & Coulson, R. L. (1992). Cognitive flexibility,

constructivism, and hypertext: Random access instruction for advanced knowledge

acquisition in ill-structured domains. In T. Duffy & D. Jonassesn (Eds.), Constructivism

and the technology of instruction (pp. 57-76). Hillsdale, NJ: Erlbaum.

Spiro, R. J. & Jehng, J-C. (1990). Cognitive flexibility and hypertext: Theory and technology for

the non-linear and multidimensional traversal of complex subject matter. In D. Nix & R.

Spiro (Eds.), Cognition, education, and multimedia: Exploring ideas in high technology

(pp. 163-206). Hillsdale, NJ: Erlbaum.

Spiro, R., Vispoel, W., Schmitz, J., Samarapungavan, A., & Boerger, A. (1987). Knowledge

acquisition for application: Cognitive flexibility and transfer in complex content domains.

[Electronic Version] In B. Britton & S. Glynn (Eds.), Executive control processes in

reading (pp. 177–199). Hillsdale, NJ: Erlbaum.

243

Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the WWW: The role of

metacognition in the formation of documents models. International Journal of Computer-

Supported Collaborative Learning, 2, 191-210.

Stadtler, M., & Bromme, R. (2008). Effects of the metacognitive computer-tool met.a.ware on

the web search of laypersons. Computers in Human Behavior, 24, 716-737.

Stahl, S. A., Hynd, C. R., Britton, B.K., McNish, M. M., & Bosquet, D. (1996). What happens

when students read multiple source documents in history?. Reading Research Quarterly,

31, 430–456.

State of Connecticut Department of Education. (2009). Connecticut's Social Studies

Frameworks. Hartford, CT: Author.

Street, B. (2003). What's 'new' in New Literacy studies? Critical approaches to literacy in theory

and practice. Current Issues in Comparative Education. 5(2), 77- 91.

Strømsø, H. I., & Bråten, I. (2009). Beliefs about knowledge and knowing and multiple‐text

comprehension among upper secondary students. Educational Psychology, 29, 425-445.

Stone, J. C. (2007). Popular websites in adolescents‘ out-of-school lives: Critical lessons on

literacy. In M. Knobel & C. Lankshear (Eds.), A new literacies sampler (pp. 49-65). New

York, NY: Peter Lang.

Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston, MA:

Allyn & Bacon.

Tashakkori, A., & Teddlie, C. (2008). Foundations of mixed methods research: Integrating

quantitative and qualitative approaches in the social and behavioral sciences. Thousand

Oaks, CA: Sage.

244

Tate, M., & Alexander, J. (1996). Teaching critical evaluation skills for World Wide Web

resources. Computers in Libraries, 16(10), 49-52.

Taylor, R. S. (1986). Value-added processes in information systems. Norwood, NJ: Ablex.

Tillman, H. (2003). Evaluating quality on the net. The selected works of Hope Tillman.

(Unpublished manuscript) Retrieved from http://www.hopetillman.com/findqual.html

Tillotson, J. (2002). Web site evaluation: A survey of undergraduates. Online Information

Review, 26, 392-403.

Tsai, M. J., & Tsai, C. C. (2010). Junior high school students’ Internet usage and self-efficacy: A

re-examination of the gender gap. Computers & Education, 54, 1182-1192.

Tuckman, B. W. (1993). The coded elaborative outline as a strategy to help students learn from

text. The Journal of Experimental Education, 62, 5-13.

VanSledright, B. (2002). Fifth graders investigating history in the classroom: Results from a

researcher-practitioner design experiment. The Elementary School Journal, 103, 131-160.

Wallen, E., Plass, J. L., & Brünken, R. (2005). The function of annotations in the comprehension

of scientific texts: Cognitive load effects and the impact of verbal ability. Educational

Technology Research and Development, 53(3), 59-71.

Wilson, B. G., & Myers, K. M. (2000). Situated cognition in theoretical and practical context. In

D. H. Jonassen & S. M. Lund (Eds.), Theoretical foundations of learning environments

(pp. 57-88). Mahwah, NJ: Erlbaum.

Wilson, P. (1983). Second-hand knowledge: An inquiry into cognitive authority. Westport, CT:

Greenwood Press.

245

Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in

the evaluation of documentary and pictorial evidence. Journal of Educational

Psychology, 83, 73-87.

Wolfe, M. B. & Goldman, S. R. (2005). Relationships between adolescents’ text processing and

reasoning. Cognition & Instruction, 23, 467-502.

Wolf, W., King, M. L., Huck, C. S. (1968). Teaching critical reading to elementary school

children Reading Research Quarterly, 3, 435-498.

Zawilinski, L. (2011). An exploration of a collaborative blogging approach to literacy and

learning: A mixed method study. (Unpublished doctoral dissertation). University of

Connecticut, Storrs.

Zawilinski. L., Carter, A., O’Byrne, I. W., McVerry, G., Nierlich, T., & Leu, D. (2007,

December). Toward a taxonomy of online reading comprehension strategies. Paper

presented at the 57th Annual National Reading Conference. Austin, TX.

246

AppendixA:OnlineReadingComprehensionAssessment

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

AppendixBORCAProtocol

311

312

313

AppendixC:CriticalOnlineInformationLiteraciesAssessmentItems

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

AppendixD:DispositionsofOnlineReadingComprehensionAssessment

341

AppendixE:OnlineReadingComprehensionAssessment

342

Documents

The Internet and Adolescent Readers: Exploring ... - CORE