16
Productive use of learning resources in an online problem-based learning environment Heisawn Jeong a, * , Cindy E. Hmelo-Silver b a Department of Psychology, Hallym University, 39 Hallymdaehak-gil, Chuncheon, Gangwon-do 200-702, Republic of Korea b Graduate School of Education, Rutgers University, 10 Seminary Place, New Brunswick, NJ 08901, USA article info Keywords: Learning resources Problem-based learning (PBL) Online hypermedia environments Contrasting cases analysis Knowledge resources Video resources abstract This study examined students’ use of learning resources in a technologically-mediated online learning environment. Undergraduate student groups were engaged in an online problem-based learning (PBL) environment, rich with pre-selected video and knowledge resources. Quantitative and qualitative analy- ses showed that students accessed resources fairly frequently and benefited from them. Resources helped students construct a rich understanding of the problem and provided ideas for problem solutions. Detailed analyses of resource exploration along with contrasting case analyses between high-achieving and low-achieving student groups suggested that for learning to be effective in resource-rich environ- ments, students first need to develop an understanding of the resources and learn how to access them efficiently. Second, students need to learn to process the contents of resources in meaningful ways so that they can integrate diverse resources to form a coherent understanding and apply them to solve problems. Finally, students need to develop knowledge and skills to use resources collaboratively, such as sharing and relating to each other’s resources. The results indicated that students, especially low-achieving stu- dents, need guidance to use resources effectively in resource-rich learning environments. Ó 2009 Elsevier Ltd. All rights reserved. 1. Use of learning resources in a technologically-mediated online problem-based learning environment Resources are an important part of human learning (Pea, 1993). Textbooks, encyclopedias, pictures, calculators, and even pencils and papers have all assisted learners in locating, recording, and fur- ther processing of the information. With the advent of information technology, the availability of resources and cognitive tools has ex- ploded. Students are increasingly exposed to an array of sophisti- cated learning resources and technology tools such as hypertexts, streaming video, and visualization tools. The way learners interact with resources is changing qualitatively, and the success of their learning is increasingly dependent on how effectively they utilize diverse resources (Hill & Hannafin, 2001; Nesbit & Winne, 2003). In addition to these changes, theoretical concerns also demand that we pay more attention to the role of resources in learning. Learning is now increasingly considered to be an acquisition of competen- cies needed to function successfully in a given domain (Collins, Brown, & Newman, 1989; Engle & Conant, 2002; Greeno, 2006; Greeno, Collins, & Resnick, 1996; Sfard, 1998). Building a compe- tency in a given domain such as physics or medicine requires, among many things, an understanding of available resources in the domain. The notion of resource is also critically related to self-directed learning and the notion of agency. To use resources effectively, learners must be self-directed and exert agency in choosing and using resources (Hmelo-Silver, 2004; Hoffman & Ritchie, 1997; Nesbit & Winne, 2003). 1.1. Definition of learning resources We define learning resources as information or tools that can be used to assist learners in the process of locating, recording, and fur- ther processing of the learning materials. A variety of learning re- sources exist that differ in several dimensions. First, resources can differ in terms of where they reside. We typically think of re- sources that exist external to the learner such as teachers, books, or calculators, but it is also possible to have internal resources such as prior knowledge (Arvaja, Salovaara, Häkkinen, & Järvelä, 2007; Fischer & Mandl, 2005). Second, learning resources can differ in the functions they provide. Some resources serve as repositories of information (e.g., books, Internet, videos), whereas others serve as cognitive tools that assist learners in processing information (e.g., calculators, visualization tools; Kim & Reeves, 2007; Lajoie, 1993). Within information resources, further distinctions are pos- sible such as primary (e.g., journal article) versus secondary re- sources (e.g., textbook) or static (e.g., printed resources) versus dynamic resources (e.g., Wikipedia; Hill & Hannafin, 2001). Finally, 0747-5632/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.chb.2009.08.001 * Corresponding author. Tel.: +82 33 248 1725; fax: +82 33 256 3424. E-mail address: [email protected] (H. Jeong). Computers in Human Behavior 26 (2010) 84–99 Contents lists available at ScienceDirect Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh

Productive use of learning resources in an online problem-based learning environment

Embed Size (px)

Citation preview

  • non-NJ 0

    Video resources

    ntste-se

    Detailed analyses of resource exploration along with contrasting case analyses between high-achieving

    chnoloironm

    f hum

    we pay more attention to the role of resources in learning. Learningis now increasingly considered to be an acquisition of competen-cies needed to function successfully in a given domain (Collins,Brown, & Newman, 1989; Engle & Conant, 2002; Greeno, 2006;Greeno, Collins, & Resnick, 1996; Sfard, 1998). Building a compe-tency in a given domain such as physics or medicine requires,among many things, an understanding of available resources in

    or calculators, but it is also possible to have internal resources suchas prior knowledge (Arvaja, Salovaara, Hkkinen, & Jrvel, 2007;Fischer & Mandl, 2005). Second, learning resources can differ inthe functions they provide. Some resources serve as repositoriesof information (e.g., books, Internet, videos), whereas others serveas cognitive tools that assist learners in processing information(e.g., calculators, visualization tools; Kim & Reeves, 2007; Lajoie,1993). Within information resources, further distinctions are pos-sible such as primary (e.g., journal article) versus secondary re-sources (e.g., textbook) or static (e.g., printed resources) versusdynamic resources (e.g., Wikipedia; Hill & Hannan, 2001). Finally,

    * Corresponding author. Tel.: +82 33 248 1725; fax: +82 33 256 3424.

    Computers in Human Behavior 26 (2010) 8499

    Contents lists availab

    u

    eviE-mail address: [email protected] (H. Jeong).Textbooks, encyclopedias, pictures, calculators, and even pencilsand papers have all assisted learners in locating, recording, and fur-ther processing of the information. With the advent of informationtechnology, the availability of resources and cognitive tools has ex-ploded. Students are increasingly exposed to an array of sophisti-cated learning resources and technology tools such as hypertexts,streaming video, and visualization tools. The way learners interactwith resources is changing qualitatively, and the success of theirlearning is increasingly dependent on how effectively they utilizediverse resources (Hill & Hannan, 2001; Nesbit & Winne, 2003).In addition to these changes, theoretical concerns also demand that

    Ritchie, 1997; Nesbit & Winne, 2003).

    1.1. Denition of learning resources

    We dene learning resources as information or tools that can beused to assist learners in the process of locating, recording, and fur-ther processing of the learning materials. A variety of learning re-sources exist that differ in several dimensions. First, resourcescan differ in terms of where they reside. We typically think of re-sources that exist external to the learner such as teachers, books,1. Use of learning resources in a teonline problem-based learning env

    Resources are an important part o0747-5632/$ - see front matter 2009 Elsevier Ltd. Adoi:10.1016/j.chb.2009.08.001and low-achieving student groups suggested that for learning to be effective in resource-rich environ-ments, students rst need to develop an understanding of the resources and learn how to access themefciently. Second, students need to learn to process the contents of resources in meaningful ways so thatthey can integrate diverse resources to form a coherent understanding and apply them to solve problems.Finally, students need to develop knowledge and skills to use resources collaboratively, such as sharingand relating to each others resources. The results indicated that students, especially low-achieving stu-dents, need guidance to use resources effectively in resource-rich learning environments.

    2009 Elsevier Ltd. All rights reserved.

    gically-mediatedent

    an learning (Pea, 1993).

    the domain. The notion of resource is also critically related toself-directed learning and the notion of agency. To use resourceseffectively, learners must be self-directed and exert agency inchoosing and using resources (Hmelo-Silver, 2004; Hoffman &Online hypermedia environmentsContrasting cases analysisKnowledge resources

    ses showed that students accessed resources fairly frequently and beneted from them. Resources helpedstudents construct a rich understanding of the problem and provided ideas for problem solutions.Productive use of learning resources in aenvironment

    Heisawn Jeong a,*, Cindy E. Hmelo-Silver b

    aDepartment of Psychology, Hallym University, 39 Hallymdaehak-gil, Chuncheon, GangwbGraduate School of Education, Rutgers University, 10 Seminary Place, New Brunswick,

    a r t i c l e i n f o

    Keywords:Learning resourcesProblem-based learning (PBL)

    a b s t r a c t

    This study examined studeenvironment. Undergraduaenvironment, rich with pre

    Computers in H

    journal homepage: www.elsll rights reserved. use of learning resources in a technologically-mediated online learningstudent groups were engaged in an online problem-based learning (PBL)lected video and knowledge resources. Quantitative and qualitative analy-online problem-based learning

    do 200-702, Republic of Korea8901, USA

    le at ScienceDirect

    man Behavior

    er .com/locate /comphumbeh

  • rs inresources can differ by the medium in which they are presented.Information, for example, can be presented either in paper or dig-ital book forms. Likewise, computation can be supported eitherwith calculators, abacus, or computer programs. In this paper, wefocus on external information resources provided in technologi-cally-mediated learning environments. These resources are oftenin the form of hypertexts that consist of non-linear and dynamicdocuments, images, and videos.

    1.2. Challenges for effective use of learning resources

    Learning resources provide unique opportunities for construct-ing new and rich understanding. They represent a collection of cul-tural and scientic knowledge accumulated over the years (Hill &Hannan, 2001; Yeo & Tan, 2008). They can provide a wealth ofauthentic and up-to-date information not necessarily available intextbooks. They also provide rich contextual information anddiverse perspectives on how to interpret the information. As such,resources can be used to help students to anchor their learning,examine their understanding from diverse perspectives, make con-nections across related concepts, and bridge the gap between theirtheoretical understanding and practical knowledge (CTGV, 1997;Davies, Ramsay, Lindeld, & Couperthwaite, 2005; Hoffman &Ritchie, 1997; Nesbit & Winne, 2003; Palmer, 2007; Ruthven, Hen-nessy, & Deaney, 2005; Schrader et al., 2003). In spite of thesepotentials, however, the presence of resources does not automati-cally improve learning; productive use of resources can be difcultto achieve (Bera & Liu, 2006; Bowler, Large, & Rejskind, 2001; Dil-lon & Gabbard, 1998; Hoffman, Wu, Krajcik, & Soloway, 2003; Nes-bit & Winne, 2003; Oliver & Hannan, 2000; Recker, Walker, &Lawless, 2003; Wallace, Kupperman, Krajcik, & Soloway, 2000).

    Several challenges need to be addressed before students canproductively use resources to enhance their learning. One chal-lenge is that despite easy availability and accessibility of a richvariety of resources, students are often unwilling or disinclinedto access them. Cramer, Collins, Snider, and Fawcett (2007) exam-ined how many students used online resources (e.g., lecture notes)and found that although students who used the resources foundthem helpful and performed better on the exams, only 20% of thestudents accessed the available resources. Kirkwood (2006) sur-veyed more than 16,000 students taking 122 courses on how theyaccessed two major online resources that the university provided(an online index to electronic texts and a customized databasewith links to quality external websites relevant to a particularcourse). Less than 30% of the students made some use (i.e., at leastonce a month) of the online resources. He additionally found thatstudents use of online resources varied enormously betweencourses. Courses that had project components and/or requiredthe use of online resources elicited more resource use. These re-sults suggest that simply making resources available is insufcientto ensure resource uptake. The pedagogical design of the courseneeds to encourage resource use.

    Another challenge is that students may experience difcultiesin searching, navigating, and locating online information (Hsieh-Yee, 2001; Large & Beheshti, 2000; MaKinster, Beghetto, & Plucker,2002; Wallace et al., 2000). This is partly due to the sheer magni-tude of information, especially when the Internet is involved. TheInternet, while opening a door to a rich array of informationincluding primary resources, can also be a challenge because stu-dents need to search for and sift through huge amounts of informa-tion before they can nd what they are looking for. Also, theinformation on the Internet is neither well-organized nor quality-controlled. Web pages are also not necessarily constructed with

    H. Jeong, C.E. Hmelo-Silver / Computelearning in mind. Much of the information on the Internet maybe too difcult to comprehend and ill-matched to students knowl-edge and skill level (Bowler et al., 2001; Ng & Gunstone, 2002;Ruthven et al., 2005). These factors contribute to the difcultiesstudents experience with online resources, especially when theyhave limited prior knowledge and/or experience.

    One solution to this problem has been to scaffold the searchprocess by providing students with pre-selected resources and/ortechnical tools to assist the handling of information. Providingpre-selected resources can free up students cognitive resourcesso that they can focus their attention more on building theirknowledge rather than searching of information (e.g., Hoffmanet al., 2003; Oliver & Hannan, 2000; Ruthven et al., 2005). Techni-cal tools can also assist students with the process of collecting,organizing, and evaluating complex information. Oliver and Han-nan (2000) provided middle-school students with a set of techno-logical tools that assist lower-order functions (e.g., informationcollection) and higher-order functions (e.g., reasoning). When stu-dents worked with a large set of resources without any conceptualguidance (e.g., advance organizers), they simply skimmed the re-sources: they had difculty with extracting even basic informationfrom the resources. When students worked with a smaller set ofpre-selected resources with conceptual guidance, they were betterat extracting information from resources. Students resource use,however, still remained at a rudimentary level. They identiedonly basic information from these resources. Tools provided toscaffold higher-order thinking were rarely used by students, andwhen used, they were used mostly to assist lower-order thinking.Few students used tools to organize information, seek patterns ofrelationships in the resources, specify solution path, for example.Pre-selected resources and tools freed up cognitive resources forstudents, but it did not necessarily lead to sophisticated processingof resources.

    Yet another challenge is thus students shallow engagementwith the contents of the resources. Wallace et al. (2000) examinedsixth grade students using the Web as an information resource inscientic inquiry. Students navigated hypermedia resources quitewell and experienced few problems with the basics of technology.A closer look, however, revealed that students did not explore theresources extensively or engage deeply with the contents of the re-sources. They were busy submitting searches and were able to ac-cess a great deal of information very rapidly, but they stayed closeto their initial search queries and did not search deeply. In addi-tion, only a small portion of their exploration was related to thecontents of their inquiry (i.e., only 31% of their visits were to con-tent pages). Even when students visited a content page, they didnot spend enough time to nd out what kinds of information wereavailable in the page. They only took a cursory look at a contentpage in most cases and spent an average of 35 s per page. Bowleret al. (2001) also reported that students resource explorationwas shallow and poor except for images with action and topicswith popular gures. In most cases, the extent of students engage-ment with online resources was limited to cut and pasting chunksof information. They rarely put information into their own words.

    Given such usage, it would be surprising if students achieveddeep levels of understanding while learning in resource-rich envi-ronments. Hoffman et al. (2003) examined how well students usedresources during inquiry. They provided considerable support tostudents to help with their search and inquiry processes. They pro-vided pre-selected, high-quality resources that were well-orga-nized and age-appropriate. Students were able to learn fromtheir resource-driven inquiry in this study. In addition, studentswho engaged in deep search and access strategies tended toachieve deeper understanding. However, students still failed to de-velop a level of content understanding that might be expectedfrom the rich array of available resources. Although a majority of

    Human Behavior 26 (2010) 8499 85the students offered explanations, these were only partially accu-rate and lacked depth, often being limited to recalling informationor simple explanations.

  • learning resources in STELLAR? This question was addressed byexamining how actively students accessed the resources and therole of STELLAR in this process. Second, was resource use helpfulto learning, and if so, how did resources support students learn-ing? This question was addressed by examining how resources en-riched and anchored students understanding and supportedproblem solving processes. Third, what might be the features ofproductive resource use? This question was addressed by examin-ing how high-achieving and low-achieving students differed intheir use of learning resources. Quantitative and qualitative analy-ses were carried out both at macro- and micro-levels in order toaddress these questions.

    2. Methods

    2.1. Participants

    Thirty-four students were enrolled in an online PBL course

    Step 1 Study video case Individual,

    rs in Human Behavior 26 (2010) 8499These results suggest that developing a deep understandingfrom resources is a complex and challenging process. Merely pro-viding resources to students is not enough. Attention needs to bepaid to the design of the learning tasks and learning environmentsso that active resource use can be promoted. It is also important toensure that the size and the quality of the resources are appropri-ately matched for learners knowledge and skill levels. Above all,we need to think carefully about the kinds of learning activitiesthat can promote deep engagement with resources. Resourcesare learning objects themselves around which meaning-makingactivities should occur. For learning resources to engender the kindof deep learning expected from rich and complex resources, learn-ers need to engage in additional cognitive processing of that infor-mation (Yeo & Tan, 2008). Unlike in traditional learningenvironments where learners are more or less restricted a fewauthoritative resources such as texts, learners are increasinglyasked to deal with a large amount of diverse resources in re-source-rich online learning environments. Such environments posemore cognitive demands and challenges for students as studentsneed to identify and evaluate core resources and compare and inte-grate them into a coherent understanding in a self-directed man-ner. We are beginning to identify some of the challengesstudents face in such resource-rich learning environment, but theprocesses involved in productive resource use are not well under-stood yet (Nesbit &Winne, 2003). The goal of this paper is to exam-ine how students use resources in a resource-rich online learningenvironment and identify some of the conditions and processesof productive resource use.

    1.3. Context for current study

    This study examined the use of learning resources in a resource-rich online learning environment called STELLAR (Socio-TechnicalEnvironment for Learning and Learning Activities Research; Derry,Hmelo-Silver, Nagarajan, Chernobilsky, & Beitzel, 2006; Hmelo-Sil-ver & Derry, 2007). STELLAR is an online problem-based learning(PBL) environment for preservice teachers. It provides various on-line resources such as video clips and hypertexts along with a set oftools to assist their processing (Derry et al., 2006). PBL is aninstructional approach that structures learning activities aroundsolving authentic problems such as medical diagnosis or instruc-tional planning (Hmelo-Silver, 2004). PBL is one of the instruc-tional contexts that can benet greatly from the availability ofdiverse online resources (Hoffman & Ritchie, 1997). In PBL, stu-dents work in a group and identify knowledge gaps blocking theirprogress toward problem solutions (the learning issues) at theoutset. Students are not told what to learn or study. There are notextbooks or any other mandatory study materials. Students iden-tify their own learning goals in the context of a given problem andthen research these learning issues more or less in a self-directedmanner. Understanding of important concepts is developed inthe process of solving the problems. Although students do not al-ways research the learning issues they generate in their self-direc-ted learning (Dolmans, Schmidt, & Gijselaers, 1995), students inPBL programs use learning resources more actively, visiting li-braries more frequently for longer period of time than studentsin traditional programs (Marshall, Fitzgerald, Busby, & Heaton,1993).

    STELLAR has been shown to be effective in promoting studentslearning. Derry et al. (2006) compared the learning outcomes fromSTELLAR courses and traditionally taught courses and found thatstudents in STELLAR courses showed reliable pre- to post-test gainscompared with students in traditional educational psychology

    86 H. Jeong, C.E. Hmelo-Silver / Computecourses. In this study, we examined how students used the re-sources to support their learning and problem solving process.Three questions were examined. First, how did students exploreonlineStep 2 Record observations and initial proposals in

    online personal notebook that guides studentstowards relevant lesson features

    Individual,online

    Step 3 View other students proposals Collaborative,online

    Step 4 Identify concepts to explore for re-design Collaborative,face-to-face

    Step 5 Conduct and share research Collaborative,online

    Step 6 Collaborative lesson design Collaborative,online

    Groups present project to class Collaborative,face-to-face

    Step 7 Explanation and justication of group product Individual,using STELLAR in educational psychology at a large northeasternUS university. Students were divided into six groups. Four groupsconsisted of six students and two groups consisted of ve students.Grouping was done to assure a range of content expertise in thegroup. The majority of the students were undergraduate students(50% juniors and 18% sophomores and seniors with the rest beinggraduate students) who majored in social studies (29%), psychol-ogy (18%), music education and English (5% each), and other (e.g.,math, art history; 24%). The course was a prerequisite for entranceinto the teacher education programs. Faculties who have taughtthe course more than 10 years estimated that more than 90% ofthe students have taken this course with an intention to teach inelementary or secondary education.

    2.2. Stellar

    Learning activities in STELLAR were organized into eight PBLsteps in order to make the complexity of the learning process man-ageable. The steps involved either individual or collaborativephases (Table 1). Students began each problem with Step 1 by indi-vidually studying one or two video cases. In Step 2, students gen-erated brief individual proposals for instructional re-design. Theindividual notebooks provided guidance for the students regardingthe lesson features that were relevant for re-design. This work wasthen shared with group members in Step 3. In Step 4, students con-tributed their best ideas to the group proposal and researched edu-

    Table 1Structure of STELLAR activities (Hmelo-Silver, Nagarajan, & Derry, 2006).

    Activity Description ModalityonlineStep 8 Reection Individual,

    online

  • rs incational psychology concepts to further the re-design as well ascommenting on and evaluating others proposals. In Step 5, theywere to critically discuss each others proposals and vote for thebest proposal to be included in the nal group product. The stu-dents then viewed their nal group product in Step 6. In Step 7,students were asked to provide individual explanations of the nalgroup product from an educational psychology perspective. Theycould disagree with the groups decision here as long as they pro-vided a principled reason. Step 8 was designed for students to re-ect on their learning. Reecting on learning provided anopportunity to think about their performance, assess problemsand adjust the learning process in order to improve it (Collinset al., 1989).

    Students solved three PBL problems with STELLAR in this study.In Problem 1, students watched an inquiry-oriented classroom inwhich children were learning science through design activities.Students were asked to design an approach to assess learning fromsuch learning activities. In Problem 2, students watched two videocases, one showing a traditional physics teacher who used lecturesand demonstrations and a contrasting one showing a teacher whoadopted constructivist instructional approach. Constructivistinstructional approaches postulate that learning is more effectivewhen students are more actively engaged in the construction ofknowledge rather than passively receiving it. Instead of transmit-ting knowledge, a constructivist teacher guides students to developtheir own understanding on the subject (Bransford, Brown, & Cock-ing, 2000; Chi, 2009; Hmelo-Silver & Barrows, 2008). The construc-tivist teacher in this problem, for example, demonstrated thephenomena of static electricity and asked her students to describewhat they observed, explain why it happened, and come up withan experiment to prove or disprove their explanations. Studentstask was to help the rst teacher improve the lesson on static elec-tricity by adapting some of the techniques of the second teacher. InProblem 3, students saw a video case of a foreign language teacherwho wanted to re-design her lesson to meet new foreign languageteaching standards. In this problem, the objective was to help herwith the re-design of her lesson.

    STELLAR provided a set of tools to support students learningand problem solving processes: an Individual Notebook, ThreadedDiscussion Board, and Group Whiteboard. The Individual Notebookwas provided for students to record their observations regardingthe video and their research on related educational psychologyconcepts. The Threaded Discussion Board and Group Whiteboardwere available as a means for exchanging ideas and serving as ashared workspace for developing group proposals. The GroupWhiteboard provided different tabs corresponding to the three as-pects of the re-design for which students were asked to generateproposals: (a) instructional objectives, (b) evidence of understand-ing, and (c) instructional activities. When students proposedinstructional objectives, they were prompted to list what kinds ofeducational psychology research were needed. When students pro-posed evidence of understanding and instructional activities, theywere expected to list relevant research ndings. The Whiteboardalso provided a space where students could comment on eachothers proposals (see Derry et al. (2006) and Hmelo-Silver, Derry,Woods, DelMarcelle, and Chernobilsky (2005), for details).

    STELLAR provided a diverse array of resources: the Video CaseLibrary, Knowledge Web (KW), Research Library, and PBL help.The Video Case Library consists of a set of video cases divided into516 video clips. The KW provides hypertext explanations on edu-cational psychology concepts. The Research Library provided linksto outside resources (e.g., ERIC data base). PBL help provided assis-tance with how to use the system and worked examples of how to

    H. Jeong, C.E. Hmelo-Silver / Computeuse the online tools. The resources were all pre-selected and editedto be appropriate for college undergraduates taking an introduc-tory course in educational psychology. There were a nite set of77 web pages and 128 video clips, but they provided ample re-sources students could use in their PBL activities. Of the four typesof resources, the Research Library only provided links to outside re-sources, and once students left STELLAR, the system did not trackstudents activities. The PBL help was rarely accessed by students.This study thus focused on the use of the Video Case Library andthe Knowledge Web, which are described below in more detail.

    2.2.1. Video Case LibraryThe Video Case Library consisted of 12 cases which consisted of

    128 clips. The length of the individual clips varied, but they were ingeneral a few minutes long. Parts of the Video Case Library wereintegrated into the PBL activities in that students were explicitlyasked to view them at the beginning of each online PBL problem.In Problem 1, students were asked to view Video Case 5 (consistingof eight clips). In Problem 2, students watched two video cases,Case 1 (consisting of nine clips) and Case 4 (ten clips), one showinga traditional physics teacher and a contrasting one showing a con-structivist instructional approach. In Problem 3, students wereasked to view Video Case 9 (consisting of ten clips). Required videoresources refer to the video cases or clips that students were askedto view as part of the PBL activities, whereas other video resourcesrefers to the video cases or clips that were not integrated as part ofthe activities, but were accessible to students whenever theywanted to. There were four required and eight other video cases(i.e., 37 required and 91 other clips).

    2.2.2. Knowledge WebThe KW consisted of hypertext for 77 educational psychology

    concepts such as knowledge construction or metacognition.In Step 4, Students were instructed to Research the Learning Sci-ences (using the Research Library tool as a place to start) to furtherdevelop your proposals and to comment on and evaluate othersproposals. STELLAR did not specify which concepts studentsshould research, but provided a list of Related Concepts hyperlinkedto each video clip (see example in Fig. 1) to help students better lo-cate relevant concepts. Educational psychology concepts in the KWwere called either recommended or other concepts based onwhether the concepts were linked to specic video clips. Recom-mended concepts were concepts that were linked with specic vi-deo clips. Other concepts were not linked to the video clips used,but were linked to each other as well as to other video clips andcould be accessed as needed. There were 56 recommended and21 other concepts.

    2.3. Instructional context

    The course lasted 14 weeks, during which student groupsworked on ve problems dealing with various educational psy-chology concepts. The class was conducted with a combinationof face-to-face and online PBL. Students worked rst on twoface-to-face problems in the beginning of the course and com-pleted the three remaining problems online using STELLAR. Theanalysis of this study focused on the STELLAR portion of thecourse. Note that even during the online PBL problems, the classmet as a group in Steps 4 and 6. For example, at the culminationof Step 6 the re-design was presented at a poster session, wherethe groups shared their re-design proposals in-class. Studentswork on the problems was self-paced, but a target date was pro-vided for each activity step. A STELLAR problem typically lasted 23 weeks.

    As part of the normal grading process, several assessments werecarried out. First, the solution to each of the three online PBL prob-

    Human Behavior 26 (2010) 8499 87lems was assessed using a complex rubric that took into consider-ation various parts of the solutions the students came up with suchas objectives for the re-design, evidence of understanding and

  • quantitative data (i.e., system report of resource use and rating

    ed t

    rs in Human Behavior 26 (2010) 8499instructional activities based on the principles of backward design(Wiggins & McTighe, 1998) as well as the use of educational psy-chology principles to justify their designs. The total possible scorewas 12 for each online problem. Second, the nal exam, consistingof both take-home and in-class parts, was given. The in-class por-tion of the nal exam included a video case analysis task wherestudents analyzed a video and provided an explanation of a videocase presenting a learning and instruction dilemma. This videoanalysis task was scored for the appropriate use of educationalpsychology concepts. The nal grade was determined based on stu-dents PBL activities and exam performance.

    2.4. Data sources and analyses

    Fig. 1. Video cases link

    88 H. Jeong, C.E. Hmelo-Silver / ComputeData sources for analyses were: (a) system report on resourceuse, (b) log data, (c) postings in the Discussion Board and GroupWhiteboard, and (d) student reections and ratings. In the systemreport on resource use, the STELLAR system automatically gener-ates a resource use report for the Video Case Library and the KW.The reports contain information about (a) frequency of visit to re-source and (b) the coverage of visits (i.e., which video clips andconcepts were visited). The system report was made at the grouplevel, but because group size varied, the frequency of visits was di-vided by the number of the students in the group. No correction forgroup size was made for the coverage of visits measure, becausethis does not necessarily change as a function of the group size.A larger group may cover fewer resources if members explorationis concentrated on a small set of resources. The log data includedinformation about date and times of students logins and the pagesvisited within STELLAR. The postings in the Threaded DiscussionBoard and group Whiteboard were of two kinds: (a) proposals forinstructional re-design problem (instructional objective, evidenceof understanding, and instructional activities) and (b) commentsfor the proposals. Lastly, at the end of each online PBL session, stu-dents answered open-ended reection questions designed to pro-mote knowledge abstraction, as well as reections on theproduct they created and the processes they engaged in (e.g.,What enduring understanding did you acquire during this activ-ity? How will you use what you have learned in your futureteaching practice?). They also rated the STELLAR tools. For exam-ple, students were asked questions such as How did your under-data). The micro-level analyses aimed at understanding the de-tailed pattern of resource use by individual groups and relied onboth quantitative (i.e., log data) and qualitative data (i.e., postingsstanding of the case change as you worked on the activity? Theyalso rated how well the Video Case Library, the Related Conceptlist, and the KW worked on a ve-point scale.

    2.4.1. Macro- and micro-level analysesAnalyses were carried out at two levels: macro- and micro-level

    (see Table 2). The macro-level analyses examined the overall pat-tern of resource use across all six student groups over the three on-line PBL problems. The macro-level analyses were carried out using

    o the Knowledge Web.in the Discussion Board and the Whiteboard, and reection data). Asingle group produced around 4000 logged activities per online PBLproblem as well as a sizable amount of qualitative data from theonline discourse. As a result, while macro-level analyses examineddata from all six groups of students in the class over the three on-line PBL problems, micro-level analyses were carried out with twoindividual groups, Group H and Group L, which are described inmore detail in the next section. Micro-level analyses focused onthe second PBL problem, Problem 2. Problem 2 was selected for mi-cro-level analysis because it represented the most typical STELLARproblem. In Problem 1, students were learning about the environ-ment itself as well as solving a PBL problem. In Problem 3, studentswere taking an off-line take-home exam concurrently, which be-came an important motivation for students extensive explorationof both resources during this period even though it was not part ofthe online PBL activities.

    Table 2Characteristics of macro- and micro-level analyses.

    Macro-level Micro-level

    Data System report on resourceusage and students ratings

    Log data, postings in the DB andWB, students reection data

    Groups 6 groups 2 groupsProblems Problem 13 Problem 2Contrasts High-achieving versus low-

    achieving groups (N = 3 each)Group H versus Group L

  • pects of the instructional re-design but some did not. Some stu-

    3.1. Resource exploration in STELLAR

    Students as a group visited the video resources an average of85.09 times while solving the three online PBL problems (28.36times per problem). During these visits, student groups on averageaccessed 54.67 clips (43% of the available clips). Groups also ac-cessed the KW concepts 58.42 times on average while solvingthe three online PBL problems (19.47 times per problem). Duringthese visits, they explored 56.50 concepts (73% of the availableconcepts). Students ratings of the two resources were positive.The average students rating was 4.15 for the Video Case Libraryand 4.40 for the Knowledge Web. In sum, students accessed the re-sources fairly frequently and also felt that the two resources wereeffective.

    Pedagogical features of STELLAR inuenced students explora-tion of resources. First, the PBL activity structure of STELLAR inu-

    rs in2.4.2. Contrasting case analysesContrasting groups were analyzed in order to identify potential

    mechanisms important to students learning success. Analysis ofdistinctly dissimilar cases can exploit the variability among casesand thus facilitate discovery of appropriate explanations, in thiscase, of different strategies for using the learning resources (Fire-stone, 1993). Based on the groups average course grade, we com-pared three high-achieving and three low-achieving groups formacro-level contrasts. Micro-level contrasts were based on onehigh-achieving group, Group H, and one low-achieving group,Group L. Group H was the group whose average nal score wasthe highest in the class. Group L was the group whose average nalscore was next to the last in the class. The group whose averagescore was the lowest included a member who was absent for muchof the term, so the next lowest average group was selected to makea fair comparison among groups in which all members were ac-tively engaged.

    2.4.3. Content analysesQuantitative data were analyzed by rst computing descriptive

    statistics. Although the sample size was small, inferential statisticswere carried out in some cases. There were three types of qualita-tive data: postings in the Discussion Board, proposals and com-ments in the Group Whiteboard, and students reections.Qualitative data were analyzed differently depending on the datatypes and analysis objectives. First, the data in the DiscussionBoard consisted of students research posting and their interactionover them. Excluding erased postings and postings made by facili-tators, there were a total of 17 and 30 postings in Group H andGroup L, respectively, in Problem 2. A coding scheme was devel-oped inductively to examine how student groups shared and com-municated over their research. Posting in the Discussion Boardswere rst coded in terms of reporting types into whether theywere (a) research reports that described the contents of the re-search (R), (b) content-related responses to research reports (R-con-tent) such as questions about how to apply the research to theirgroup proposal, (c) other responses to research reports (R-other)such as request to provide a source for the research, and (d) proce-dural postings (P) such as questions about when to vote for theproposal).

    The research reports (i.e., postings that described the contentsof the research) were further coded in terms of whether and howthey incorporated information contained in the Knowledge Web.For each KW concept whose contents were posted in the Discus-sion Board, the KW page was segmented into paragraph units. Eachparagraph was basically segmented into a separate unit, but titlesand paragraphs with fewer than 30 words were collapsed into thenext paragraph depending on the contexts. Paragraph units werethen coded in terms of whether the segment was reported (re-ported segment coding) and how they were reported (reportingtype coding). Three different types of reporting were coded: (a)Copy and pasting (C), (b) Selective copying (SC), and (c) Paraphrasing(PH) (see Fig. 2). Copy and pasting refers to the case when studentsimply copied the entire or most of the segments. Selective copyingrefers to cases when students basically copied the information inthe segment, but were selective in that they copied only a subsetof information that they deemed important. Paraphrasing refersto the case where students restated the information from the seg-ment in their own words and/or added new information. Examplesof these codes are presented in Table 3. To verify reliability of cod-ing, two coders independently coded all the postings and Cohenskappa was computed. The kappa was .70 for posting type, .78 forreported segment, and .75 for reporting type.

    H. Jeong, C.E. Hmelo-Silver / ComputeSecond, the Whiteboard data consisted of proposals and com-ments on the proposals. All students generated at least one pro-posal during Problem 2 except one student in Group L who diddents also generated more than one proposal for a single aspectof the re-design. Different proposals generated by the same stu-dents were thus collapsed together into a composite proposal foreach individual student and examined as a whole. These data wereexamined qualitatively to understand how resources inuencedstudents proposal generation and negotiation processes. We rstidentied instances where students explicitly mentioned video orknowledge resources in their proposals. These ranged from a sim-ple mention (e.g., name of teacher in the video clips or KW con-cepts) to a more detailed acknowledgement of how the resourceswere related to the proposal (e.g., This idea of rubric came frommy research on self-directed learning and its assessment). We alsoidentied more subtle, implicit inuences of resource use by com-paring students proposals against their research for possible inu-ences of resources.

    Finally, students reection data from the second online prob-lem were also analyzed qualitatively to gather insights about stu-dents subjective experience and perspectives on resource use. Thereection data were examined to complement and validate theinterpretation of other analyses and also to identify other emer-gent resource-related themes. Due to the exploratory nature ofthese analyses, inter-rater reliability was not checked for analyseson the Whiteboard and students reections. When verbatimexamples were quoted from the data, pseudonyms were used.

    3. Resultsnot generate any proposals. Students generated different numbersof proposals. Some students generated proposals for all three as-

    Fig. 2. Coding categories for the Discussion Board postings.

    Human Behavior 26 (2010) 8499 89enced students exploration of resources. Fig. 3 shows how GroupHs access to video and knowledge resources changed during the17 days that they worked on Problem 2. Students generally

  • Table 3Examples of three types of research postings in the Discussion Board.

    Codes Knowledge Web description

    Copy and pasting Hands-on thinkingAssigning authentic tasks to students is not a guarantee thatthey will learn more than they would have learned under moretraditional instruction, however. Teachers need to incorporateactivities that will challenge students to use higher-levelthinking skills and apply their prior knowledge to the problemat hand in the process of constructing new knowledge

    Selective copying ValidityDoes the test measure what was taught in a specic unit?Content validity is a measure of the relationship between thecontent of the assessment and the purpose of the test. Thequestions should have the same emphasis as the teacher gavethem during the teaching of the unit, and the test should reect

    it.han

    lass

    erlys

    90 H. Jeong, C.E. Hmelo-Silver / Computers inexplored resources in ways that were consistent with the PBLactivity structure. The majority of the access to the video clips oc-curred at the beginning of the PBL session when students wereasked to view the video clips after reading the problem description(Step 1). KW access was more dispersed throughout the problem,but it occurred typically after students viewed the clips and whilethey searched for ideas for proposals and supporting evidence(Steps 46).

    Second, the way the system encouraged resource use also inu-enced students resource exploration. Students were encouraged to

    the same objectives as were emphasized in teaching the unFor example, embedded assessments tend to be more valid ttraditional tests because they are embedded andimplemented within the actual learning activity

    Paraphrasing An orderly workplace with a set of routines helps keep the crunning smoothly. However, the unpredictable is sure tohappen, and teachers need to keep exible. Besides the ordworkplace, the physical setting of the classroom contributeequally to the feeling of warmth and enhances cooperativelearning, small group activities, and knowledge acquisitionuse both the video and knowledge resources, but in somewhat dif-ferent manners. As for the video clips, STELLAR explicitly askedstudents to watch specic video clips in Step 1 of each problem(e.g., View the video with Ms. Bakers teaching the design lesson).Not all students followed this instruction faithfully, but this expli-cit and specic instruction had the effect of constraining studentsto concentrate on required clips while neglecting other clips: stu-dent groups visited required clips 78.81 times (94% of total visits),but visited other clips only 6.28 times (6% of total visits). In termsof coverage, student groups viewed 100% of the required clips butonly 19% of the other clips, as Table 4 shows. The encouragementwas more subtle for the knowledge resources. Although they were

    Fig. 3. Access to video and knowledge resources by day.asked to explore the KW, there were no mandatory concepts to ex-plore. Instead, the system scaffolded students exploration with Re-lated Concepts lists hyperlinked to each video clip. These listsinuenced students resource exploration, but their inuence wasnot as strong as the inuence of the explicit instruction on the re-quired video clips. Student groups visited recommended concepts51.56 times (88% of the total visits), but visited other concepts6.86 times (12% of the total visits). In terms of coverage, groups ex-plored 48 recommended concepts (86%) but only 8.50 other con-cepts (40%). Explicit encouragement to use specic resources

    Examples of posting

    Hands-on thinkingAssigning authentic tasks to students isnt a guarantee that they will learnmore than they would have learned under more traditional instruction,however. Teachers need to incorporate activities that will challenge studentsto use higher-level thinking skills and apply their prior knowledge to theproblem at hand in the process of constructing new knowledge

    ValidityDoes the test measure whatwas taught in a specic unit?

    Teachers need to keep exible when theunpredictable happens!!!

    Human Behavior 26 (2010) 8499resulted in heavy exploration on the targeted resources, whereassubtle prompts resulted in somewhat diffuse and wider resourceexploration.

    3.2. Resource use and learning at the macro-level

    High-achieving groups in general visited resources more fre-quently than low-achieving groups (see Fig. 4). Correlations werecomputed between learning outcomes (i.e., average nal coursegrade of the group and average PBL activity score) and the follow-ing measures of resource exploration: (a) total visits to video andknowledge resources and (b) the proportion of video and knowl-edge resources visited. The sample size was too small (N = 6) tohave much statistical power, but the percentage of other conceptsexplored was signicantly correlated with the average nal gradeof the group, r = .82, p < .05, and marginally correlated with theaverage PBL activity scores the groups received for their group pro-posals, r = .79, p < .07.

    High-achieving groups and low-achieving groups differed littlein their exploration of required or recommended resources, butthey differed in their exploration of other resources, especiallyother knowledge resources. Both groups visited 100% of the re-quired clips and extensively explored recommended concepts, vis-iting 89% and 82%, respectively. However, high-achieving groupsvisited more other video clips than low-achieving groups (22%versus 16%) and also more other KW concepts (51% versus 30%;see Fig. 5). In addition, high-achieving groups differed from low-achieving groups in the number of visits they made to eachresource (see Fig. 6). Although revisits were not frequent,

  • high-achieving groups made more revisits to required or recom-

    but also engaged in more focused exploration of the resources that

    not always differ much in their resource use, and even when they

    their current problem. A micro-level contrast between Group Hand L revealed that the adoption of this wider exploration strategyby high-achieving group was not uniform across problems (see Ta-ble 5). Students in Group H explored the KW widely during Prob-lems 1 and 3, but not in Problem 2. They accessed 39% ofrecommended concepts and 16% of other concepts in Problem 1,but only 10% of recommended and 3% of other concepts in Problem2. Unlike Group H, Group Ls extent of concept exploration wassimilar in both Problems 1 and 2 with 13% of the recommendedand 2% of the other concepts explored in Problem 1 and 15% ofthe recommended and 4% of the other concepts explored in

    Table 4Access to the video and knowledge resources by all groups across all three problems.

    Group Total visits to resources No. of resources visiteda Visits per resource

    Video Knowledge Video Knowledge Video Knowledge

    R Other R Other R Other R Other R Other R Other

    1 129.00 22.33 65.17 12.67 37 (100%) 50.00 (55%) 50 (89%) 10 (48%) 3.49 .45 1.30 1.272 60.50 .33 47.67 9.00 37 (100%) 2.00 (2%) 46 (82%) 11 (52%) 1.64 .17 1.04 .823 70.20 .80 34.00 2.80 37 (100%) 4 .00 (4%) 42 (75%) 5 (24%) 1.90 .20 .81 .564 72.00 10.20 37.20 6.00 37 (100%) 31.00 (34%) 45 (80%) 6 (29%) 1.95 .33 .83 1.005 79.50 1.83 74.33 6.00 37 (100%) 9.00 (10%) 54 (96%) 11 (52%) 2.15 .20 1.38 .556 61.67 2.17 51.00 4.67 37 (100%) 10.00 (11%) 51 (91%) 8 (38%) 1.67 .22 1.00 .58

    Mean 78.81 6.28 51.56 6.86 37 (100%) 17.67 (19%) 48 (86%) 8.50 (40%) 2.13 .26 1.06 .80

    Note. R, required or recommend resources.a Group-level visits. Percentages were out of the 37 required and 91 other video resources and 56 recommended and 21 other knowledge resources.

    H. Jeong, C.E. Hmelo-Silver / Computers in Human Behavior 26 (2010) 8499 91did, the results should be taken only as suggestive.

    3.3.1. Learning about the systemThe macro-level analysis indicated that high-achieving students

    explored knowledge resources more widely than low-achievinggroups, especially the concepts that were not directly relevant towere most relevant to their learning objectives.

    3.3. Resource use and learning at the micro-level

    To gain a more detailed understanding of how resources con-tributed to students learning, we carried out micro-level analysesof resource use regarding the following aspects of learning in STEL-LAR: (a) learning about the system, (b) using video resources tocontextualize and enrich problem understanding, (c) researchingeducational psychology concepts, (d) proposal generation, (e)negotiation of the group proposal, and (f) students reection. Wealso contrasted Group H and L for each aspect of learning. Thesecontrasting cases analyses were done in order to identify speciccharacteristics of resource use, but note that the goal of the analy-ses was to explore potential mechanisms rather than to test den-itive hypotheses. The results were based on two groups who didmended resources. Taken together, it appears that high-achievinggroups adopted a more exible strategy for exploring resourcesas compared to low-achieving students. They explored resourceswidely even the ones not directly needed to the problem at hand,Fig. 4. Frequency of visits to resources by high-achieving and low-achievingstudent groups. Note. V_R, required video resources; V_other, other video resources;KW_R, recommended knowledge resources; KW_other, other knowledge resources.Fig. 5. Percentage of resources visited by high-achieving and low-achieving studentgroups.Fig. 6. Number of visits per resources by high-achieving and low-achieving studentgroups.

  • Problem 2. As a result, visits to concepts in Problem 2 were morelikely to be a revisit in Group H with 33% of the visits to conceptsin Problem 2 being revisits, while only 8% of the visits were revisitsin Group L.

    3.3.2. Using video resources to contextualize the problemIn a typical online PBL session, students rst read the problem

    description and were asked to watch video clips about relatedclassroom situations. By watching them, students could contextu-alize the problem and enrich their problem understanding. Analy-ses of the log data showed that Group H and L differed in howmuch time they spent on the video resources. Students in GroupH spent on average 4055 s (about an hour and 8 min), roughly212 s (3.5 min) per clip. In contrast, students in Group L spent only

    the students occupied and engaged will keep them focused

    visiteda Visits per resources

    Knowledge Video Knowledge

    Other R Other R Otherb R Otherc

    1.17 (1%) 10.50 (39%) 7.83 (16%) 1.84 1.00 1.52 1.330.17 (0%) 3.83 (10%) 1.17 (3%) 1.16 0.17 2.08 1.50

    3 20.50 22.83 24.17 19.33 10.00 (100%) 15.33 (13%) 13.33 (44%) 11.50 (25%) 2.05 1.49 1.79 1.59

    0.80 (1%) 3.40 (13%) 1.20 (2%) 1.58 1.00 1.34 1.222.00 (2%) 5.60 (15%) 1.60 (4%) 1.19 0.20 1.22 2.3812.00 (10%) 7.80 (27%) 6.80 (14%) 1.70 1.28 1.28 1.27

    92 H. Jeong, C.E. Hmelo-Silver / Computers in Human Behavior 26 (2010) 8499L 1 12.60 0.80 5.60 1.60 7.80 (98%)2 22.60 2.00 6.80 3.60 17.00 (89%)3 17.00 16.00 10.60 8.60 8.40 (84%)

    Note. R, required or recommended resource.2826 s (47 min), roughly 138 s (2.3 min) per clip. A closer look atthe data showed that while Group H faithfully watched all the re-quired clips, Group L did not. Viewings of each required clip wascoded into four categories depending on the length of the visits:(a) complete if viewing time was equal to or greater than the dura-tion of the clips at least in one of the visits, (b) incomplete if viewingtime was shorter than the duration of the clips in all the visits tothe clip, (c) miss if the clips were never watched, and (d) unknownif viewing time could not be determined, which occurred when thevisit was the last visit of the session or lasted longer than twohours. Group H completed viewing 96% of required clips, whereasGroup L completed only 65% of them (see Fig. 7). Instead of com-pleting the clips, Group L simply skimmed (incomplete category,21%) or skipped (miss category, 9%) them. Even when studentscompleted the viewing, the two groups differed in the amount oftime they spent on the clips. There were four video clips that allthe students in both Group H and L completed viewing. For thoseclips, students in Group L spent on average 94.5 s (about1.5 min) per clip, whereas students in Group H spent 123.5 s (about2 min), suggesting that they were watching some of the clips mul-tiple times. In sum, Group L did not put in enough time to processthe video resources, whereas Group H spent more time watchingthe required clips and were more likely to process the contentsof the video resources.

    Even when students did not complete all 19 required clips,however, video resources exerted a strong inuence on studentsunderstanding of the problem. Students often later talked aboutthe teachers who appeared in the video clips in their proposals.One role of the video resources was to help student construct a ri-cher understanding of the problem as can be seen in the followingexample:

    Blairs [sic] class might have trouble with this because I do notbelieve he gave them enough to go on, even though the students

    Table 5Access to the video and knowledge resources by Group H and L by the problems.

    Group Problem Total visits to resources No. of resources

    Video Knowledge Video

    R Other R Other R

    H 1 14.67 1.17 16.17 11.33 6.17 (77%)2 22.00 0.17 7.83 1.83 19.00 (100%)a Individual-level visits. Percentages were based on the required/recommended and otProblem 2, and 10 in Problem 3. There were 27 recommended concepts in Problem 1, 3

    b The average is based on students who visited other concepts.are suppose to form most of the ideas themselves [sic], theyneed something to base the models upon.

    Failure to provide students with enough materials was notmentioned in the problem description. By watching the video clips,the student identied a specic aspect of the instruction that needto be addressed in the re-design.

    Another role of the video resources was to help students under-stand educational psychology concepts at a more concrete level asthe following example shows:

    Blaire is clearly demonstrating IRE discourse [Initiate, Respond,Evaluate]. He lectures, demonstrates, and expects his studentsto comprehend. I propose a discourse more toward the recipro-cal [sic] teaching method in that students actively engage inconstructing meaning while exerting the conscious use ofeffective comprehension of the material (1).

    In this example, the student was making a connection betweenthe concept of IRE and the classroom situation portrayed in the vi-deo resources. It seems that video resources provided a context tosituate the concept of IRE. Conversely, it might be that the conceptof IRE provided a conceptual tool, a vocabulary to talk about whatthe student observed in the video clips.

    This enriched problem understanding became quite useful laterwhen students evaluated proposals. In the example below, the stu-dent had just proposed instructional activities, discussion andactive learning such as jigsaw method, and was projecting out-comes of the proposed activity.

    Overall, having the students engage in such activities entailsmore interest in the subject of static electricity. I think having

    Fig. 7. Viewing of the required video resources.her resources for each problem. There were eight required clips in Problem 1, 19 in8 concepts in Problem 2, and 29 concepts in Problem 3.

  • portion of the information covered in the KW. Group Ls reports,in comparison, were narrower in scope, covering a limited por-tion of information mentioned in the KW. Second, the twogroups differed in how much they processed the contents ofthe KW before sharing it with other group members. Althoughboth groups processed the resources rather shallowly, exten-sively engaging in copy and pasting strategy, Group L was morelikely to do so. Group H, although its members engaged in copyand pasting, also paraphrased the contents of the knowledge re-sources (see Fig. 9).

    The difference in the two groups research reporting can be seenclearly in their report on the concept of Discussion Method, whichboth groups researched. The KW page for this concept contained14 paragraph segments, of which Group H reported on three seg-ments, whereas Group L reported on one segment. The two groupsalso differed in how they reported the information. Both groups re-ported on the paragraph segment about IRE, which was describedas follows in the KW.

    IRE

    Initiate, respond, evaluate is used frequently in what many belabeled the traditional classroom. It has been called the defaultpattern in classroom discourse. The teacher asks a questionand the student answers, but its goal seems to be a playbackof course content rather than a window into deep-learning.Teachers may feel more comfortable with this technique when

    Fig. 9. Research postings in the Discussion Board by Group H and Group L. Note.Included S, KW segments included in the postings; C, copy and pasting; SC, selectivecopying; PH, paraphrasing.

    rs inand let Blaire maintain better control [management] over hisclass than he had in the lm clips we saw.

    In this example, the student engaged in a thought experimenttesting out her proposed activities. The concrete contexts createdby video resources helped the student elaborate on what kinds ofeffect they would have in the specic classroom.

    Interestingly, although Group H spent more time on the videoresources, it was Group L which mentioned them more frequentlyin the proposals. One student in Group H mentioned the video re-sources in the proposal and did so only once. On the other hand,four students, that is, all the students who generated proposalsin Group L mentioned video resources and did so multiple times(three times per student). Considering that Group L watched fewervideo resources and spent less time than Group H, it was puzzlingthat Group L mentioned video resource more frequently. It is un-clear yet why this discrepancy arose, but it might be that video re-sources grab attention easily and are thus easy to process up to apoint. In addition, the classroom situation portrayed in the videoclips closely may have resembled their prior experience as stu-dents. These characteristics of the video resources might have al-lowed Group Ls students to remember and benet from thevideo resources with ease in spite of their limited viewing of thisresource.

    3.3.3. Researching educational psychology conceptsAs students developed an understanding of the problem and its

    contexts, they set out to research relevant educational psychologyconcepts. During a face-to-face meeting, student groups brain-stormed their initial ideas about the problem and generated a setof learning issues. Each group generated their own learning issues,and the nal list of learning issues varied from group to group. Inthe case of Group H and L, the topics they researched overlappedby only 28%. The difference in the composition of their researchtopics could reect the differences in their prior knowledge, butthere was a slight tendency for Group L to focus more on the re-sources outside the system or systems recommendation. Of theKW concepts explored, 81% of Group Hs research was on recom-mended concepts, while 73% of Group Ls research was on recom-mended concepts. As for the outside research, although the systemdid not track students activities outside the STELLAR, it could beinferred from external research citations. Group H reported vepieces of outside research, whereas Group L reported seven piecesin the Discussion Board. It might be the case that Group L was moreeasily distracted and less likely to engage in task-relevant research,but the difference was small and we cannot make any strong infer-ences about these differences.

    As was the case in the video resources, the two groups differedin the amount of time they spent on the knowledge resources. Stu-dents in Group H, on average, spent 3,261 s (about 54.3 min),roughly 9.9 min per concept. On the other hand, students in GroupL spent 1871 s (31 min), roughly 4.9 min per concept. The visits toeach KW concept were categorized in terms of the total length ofthe visits to concepts. As Fig. 8 shows, Group L students spent lessthan 30 s on one third of the concepts they visited, hardly enoughtime to process the contents of a page. In contrast, Group H stu-dents spent more than 30 s in the majority of the concepts they vis-ited. Group H spent more time on the knowledge resources andwas more likely to process the contents than Group L.

    The research postings in the Discussion Board revealed thatthe groups were also differentially engaged with the contentsof the knowledge resources. First, they differed in the scope oftheir research reports. On average, Group H included 55% of units

    H. Jeong, C.E. Hmelo-Silver / Computefrom a given KW concept in the reports, while Group L includedonly 23% of the units. This means that Group Hs reports weremore comprehensive than Group Ls reports, including a largerFig. 8. Length of the visits to knowledge resources.Human Behavior 26 (2010) 8499 93they seek more control or want to probe comprehension whilekeeping students more attentive to what they are saying. (SeeChinn & Waggoner, 1992.)

  • Group H was more likely to share their research than Group L; stu-

    students research. Even when students did not explicitly mentionhow their research helped them, the contents of the proposalsshowed a clear inuence of their research. For example, one stu-dent posted the following report on self-directed learning (self-di-rected learning was incorrectly abbreviated as SDAL by thestudent in the example below):

    How to assess SDALteachers can create a rubric which students could have someimput [sic] in example of SDAL rubric

    1. Topic selection-reasons for selecting topic and its scope.2. Connection to prior knowledge-relevancy of the elicited prior

    knowledge.3. Questions and their classication-relevancy to the context and

    exhaustiveness [sic] and inclusiveness of the classication.4. Key question-type of question and its implications.5. Studying plan-relevancy and variety of resources and tools,

    sequence of stages in plan.6. Knowledge construction-evidence regarding organization and

    rs indents in Group H, on average, shared 28% of their individual re-search, whereas students in Group L shared 17% of their research.In addition to sharing their research more, Group H was also morelikely to interact over the substance of the research, reading andcommenting on the research others posted. Interaction in the Dis-cussion Board was modest in both groups, but the majority of thepostings (88%) were related to the substance of research such as re-search reports or responses to the reports in Group H, whereas onlyhalf of the postings (50%) were related to the substance of researchin Group L (see Fig. 10). In sum, it appears that Group H was moreGroup Ls posting on IRE was an exact copy of the KW descrip-tion down to the citation. There was no summarizing or paraphras-ing. On the other hand, Group Hs report on IRE was:

    IRE (Initiate, respond, evaluate)this is considered the tradi-tional way of teaching. First the teacher asks a question thenthe student responds and the teacher then either rewards thestudent if its a correct answer by appraisal or corrects themwith the correct information.

    Note that Group Hs report summarized the information inthe KW instead of merely copy and pasting. It also includednew pieces of information. It described what IRE stands for byputting initiate, respond, and evaluate inside the parentheses.The last part of the report, the teacher then either rewardsthe students if its a correct answer by appraisal or corrects themwith the correct information, was also new. It is not clearwhether the student acquired these pieces of information fromsome other source or added them based on prior knowledge,but regardless of where the information came from, it showedan effort on the students part to process the contents of the re-sources meaningfully and integrate them with their priorknowledge.

    Resource use in STELLAR was a collaborative activity. Studentsresearched resources together and shared their research. Collabo-rative research involved division of labor in this study. Once indi-vidual groups decided on their learning issues, they divided thelearning issues among themselves and each researched differentconcepts. Such division of labor allowed groups to research moretopics potentially needed to solve the problem. One important fac-tor in this arrangement is the amount of redundancy among indi-vidual members tasks (Hutchins, 1995). If there is zeroredundancy, the group can cover a lot of topics, but cannot recoverin case of a member loss or failure. If there is too much redun-dancy, it means that individual members conduct identical re-search and there is little need for collaborative research. It isunknown yet what level of redundancy is ideal in collective prob-lem solving situations such as PBL, but Group H and L differedsomewhat in the redundancy of their research. For Group H, 29%of concepts that they researched was redundant, that is, researchedby more than one member, whereas 35% of Group Ls research wasredundant.

    For collaborative research to be effective, students also neededto share the results of their research with other group members.This means two things, sharing their research with the rest of thegroup members and processing the research that other membersshared with them. Unless these two things happen, the individualresearch cannot make its way to the collective information pool ofthe group, and the group cannot really benet from collaborativeresearch. In this study, students shared their research by postingit in the Discussion Board. Analysis of these postings showed that

    94 H. Jeong, C.E. Hmelo-Silver / Computeeffective in collaborative research than Group L, being more activein sharing their individual research and engaging in more substan-tive interaction over their research.3.3.4. Proposal generationAs students carried out their research, they generated solu-

    tion proposals. Generating a solution to instructional re-designproblems was a complex and intricate process inuenced by anumber of factors such as students prior knowledge, inferencegeneration, social interaction as well as resources use. It wasdifcult to isolate the inuences of resources from other inu-ences, but we still attempted to identify whether and how re-sources inuenced students proposals. We rst examinedexplicit references to resources in the proposals. Resources werefrequently mentioned in the proposals in part due to theprompts by STELLAR to identify related research, but these men-tions were often vague (e.g., The research for this proposal canbe found in the discussion board). In addition, explicit refer-ences to resources can be somewhat misleading as we haveseen Group Ls frequent mentions of video resources. Still, someof these references clearly indicated that students proposal gen-eration was inuenced by resources, as in the followingexample:

    The idea of using a rubric came from my research on self-direc-ted learning and its [sic] assessments. Assessment in Education,Vol. 9, No. 1, 2002.

    We also examined more subtle inuences of resources by com-paring the contents of the nal proposals against the contents of

    Fig. 10. Postings in the Discussion Board. Note. R, research reports; R-content,content-related responses to research reports; R-other, other responses to researchreports; P, procedural postings.Human Behavior 26 (2010) 8499integration of info collected.7. Conclusions-exent [sic] of generalizations of ndings, ideas for

    application.

  • knowledge construction, (7) conclusions, (8) reection and

    rs ininuenced their proposals in one way or another in most cases inboth Group H and L.

    Ideas for proposals came mostly from knowledge resources, butother resources also played a role. In the example below, the ideafor the proposal came from the video resources:

    This is an adaptation of Etkinas manner of teaching (remem-ber that last video where she said that if her students wereasked why they did something, shed want them to sayBecause we came up with an explanation, and we needed totest it.

    Etkinas manner of teaching referred to the demonstration oftesting experiments in the second video case used for this prob-lem. Another source for proposal ideas was other students re-search as the following example shows:

    I decided to use the activities of experiments and keeping aportfolio because of the research that was done on the impor-tance of external representation and open learning environ-ments which will occur by having the students do theirexperiments independent of the teacher. . . This also would8. Reection and self-assessment-evidence of metacognitive [sic]awareness, accuracy of self-assessment of performance.

    9. Knowledge dissemination-awareness of audience in the presen-tation plan

    as a group we could possible create our own rubric. . .Im not quitesure how this would differ from the portfolio though. . .maybe thisis the way that we could assess the portfolio. . .Ill put this in myproposal.

    It is notable that the last part of this posting demonstrated somemetacognitive awareness as the student realized that she was notcompletely sure of how creating a rubric was relevant and startedengaging in some planning. She later gured out how to use whatshe learned from the resource and generated the following pro-posal about assessment:

    assessing these instructional objectives can be accomplishedwith the use of the portfolio. The portfolio can be graded by set-ting up a rubric. The students can also have a say in some of themajor key points of the rubric. My suggestion for the rubricwould be:

    1. Initial thoughts-how thoughtful they are at their approach insolving the experiment.

    2. Connection to prior knowledge-relevancy of the elicited priorknowledge.

    3. Questions and their classication-relevancy to the context andexhaustiveness [sic] and inclusiveness of the classication.

    4. Key question-type of question and its implications.5. Studying plan-relevancy and variety of resources and tools,

    sequence of stages in plan.6. Knowledge construction-evidence regarding organization and

    integration of info collected.7. Conclusions-exent [sic] of generalizations of ndings, ideas for

    application.8. Reection and self-assessment-evidence of metacognitive [sic]

    awareness, accuracy of self-assessment of performance.9. Knowledge dissemination-awareness of audience in the presen-

    tation plan.

    Although she did not explicitly mention where her ideas camefrom, it is clear from the above two excerpts that her researchstrongly inuenced her proposal. Students proposals were not al-ways mapped onto their research this closely, but their research

    H. Jeong, C.E. Hmelo-Silver / Computeinclude the research that Gladys did on the importance ofhands-on-learning.self-assessment, and (9) knowledge dissemination.

    In this example, the student tried to incorporate another stu-dents idea of using a rubric into her own proposal. This reectedan attempt to integrate her proposal with the other students.Group Hs proposals included eight such mentions of other stu-dents proposal/ideas; in contrast, Group Ls proposal includedonly three mentions. The following excerpt from one of GroupHs proposals shows how they tried to integrate different individ-ual proposals as widely as possible (indicated in italicizedportion).

    Problem-based learning will be the model for how our acti-vites [sic] will be taught. Like PBL our unit plan will try touse many real-life situations to personalize the lesson to thechildrens [sic] scope of thinking. PBL will include group/classdiscussion (Peggy), Hands-on learning(Gladys) in the form ofexperiments, and the progressive learning will be monitoredThe student explained that one of the reasons that she decidedto use activities of experiments was another students research(Gladys). Paying attention to others research helped the studentconstruct a proposal in more principled way. By sharing their re-search and actively incorporating others research in their propos-als, students engaged in collaborative knowledge building (Bereiter& Scardamalia, 2006).

    Interestingly, although both Group H and L were equally likelyto include their own or others research, they differed somewhat inhow likely they were to acknowledge them because some studentswould just list concepts or research pieces without acknowledgingwhose research it was. Both Group H and L mentioned ve piecesof research by other students. While all of them were acknowl-edged in Group H, only one was acknowledged in Group L. In addi-tion to being more active in sharing and interacting over resources,Group H was more active in acknowledging the specic inuenceof others research.

    3.3.5. Negotiation of the group proposalNegotiation of the group proposal occurred in the process of

    sharing and viewing of one anothers proposals, commenting onand revising proposals, and voting for the best proposal. The rawinformation from the video and knowledge resources played a lesssignicant role in this process, but examination of the contents ofthe proposals and comments showed that students individual pro-posals and comments served as yet another resource, a secondaryresource socially created in the process of negotiating the groupproposals. Consistent with the overall differences that Group Hand L showed with respect to collaborative research, Group Hand L differed again in how actively they created and utilized thesesocially-created resources.

    First, Group H was more active in commenting on other stu-dents proposals. Group H generated 4.89 comments/student, andGroup L generated 4.00 comments (see Fig. 11). In Group L, themajority of the comments were generated by the facilitator,whereas in Group H, students generated most of the commentsand their comments were directed toward others proposals. Sec-ond, Group H was more active in integrating proposals by differentmembers. Students often mentioned other students ideas in theirproposals:

    We could include as part of the grading rubric some of the sug-gestions put forth in Cates proposal #5, such as (1) initialthoughts, (2) connection to prior knowledge, (3) questionsand their classications, (4) key question, (5)studying plan, (6)

    Human Behavior 26 (2010) 8499 95[sic] in the portfolio (Cate & Becky)as part of assessment ofunderstanding.

  • rs in. . . [Details of the proposal deleted]. . .

    Research can be found on the discussion board. Gladys reached[sic] Hands-on learning, Peggy researched discussion, Cate andBecky researched portfolio, and I researched PBL.

    In this example, the student integrated and acknowledged dif-ferent individual proposals and research. Group L also attemptedto integrate different individual proposals, but not to the extentas in the above example. One of Group Ls best attempts to incor-porate other students ideas is shown in the next example (noteitalicized portion).

    I think a good way to incorporate the theories of self-regulatedlearning, discovery learning, and reciprocal teaching would beto have the students do a jigsaw.

    . . . [Details of the proposal deleted]. . .

    I think Faunas proposal for an overall assessment of the unit wouldwork well with this jigsaw idea incorporated.

    Consistent with their treatment of system-provided re-sources, Group L showed little engagement with the socially-created resources. Taken together, it seems that students inGroup H were more oriented towards collaborative researchthan the students in Group L and actively used their group

    Fig. 11. Comments on the proposals in the Whiteboard. Note. self, commentsgenerated for ones own proposals; other, comments generated for othersproposals; facilitator, comments generated by the facilitators.

    96 H. Jeong, C.E. Hmelo-Silver / Computemembers as resources.

    3.3.6. Students reections on learningStudents reection data conrmed that resources played a sig-

    nicant role in students learning. Students often listed what theylearned from resources as one of the enduring understanding theyacquired during the PBL problem. In the following example, a stu-dent wrote about the educational psychology concepts that hergroup researched as part of her enduring understanding fromthe PBL problem (italics indicate concepts researched by thegroup):

    The enduring understandings that I acquired would be theimportance of both problem-based learning and hands-on-learn-ing. It is important for students to see how what they arelearning is used in real life. In addition for students to betterunderstand new information it is important that they be moreactive in their learning. By having the students create theirown experiments it forces the students to ask themselves[sic] questions in order to better understand the mainconcepts.We decided with the holiday to try the whole online thing withonly meeting once outside of class. We had to be more self-directed and disciplined. I feel I was very productive I wasalways checking on everyone elses research and making suremy proposals were changed each time I acquired new informa-tion. I also began to learn to make the connections between notonly my research but the rest of the groups research. It felt reallygood nally understanding a problem.

    Interestingly, the reection data also hinted at why Group L wasnot effective at collaborative research.

    I think this time I really got the hang of the PBL website. I alsomananged [sic] my timemuch better. Our group overall had bet-ter communications this time around as well. Again we dividedup the research topics and not everyone abided by the guidelineswe set. For the last problem I hope we all can go by what we agreeon in lecture and not go off and do individual tasks for the problem.The group needs to stay more focused on completing the grouptask and not just on getting the points for their own individual grade.

    Individual members of Group L appeared to be more interestedin pursuing their own agendas without aligning them with thegroups agenda. This failure to manage the tension between indi-In addition, students often talked about applying what theylearned from resources in classroom setting when they become ateacher (italics added):

    As always, explanation served to be one of the key facetsthroughout this activity (Im seeing a connection here, becauseexplanation is involved with each problem). I also had to useapplication in that I was made to use my knowledge in newand diverse situations, such as relating both Johnsons andEtkina teaching in such a manner to create a better learningenvironment for Johnsons students. I also achieved perspectivedue to the fact that I had to place myself in Johnsons shoes in orderto solve this problem.

    The above example also shows that resources helped studentsto take the perspective of the teacher in this process.

    In addition to contributing to the development of contentunderstanding and professional perspectives, the reections dataalso showed that students were also learning how to use, man-age, and process resources more effectively. The following notesare from two students as they reected on their learning process.

    In doing my research as well as in posting it for the group Imade more of an effort to make clear connections to how myresearch could be used in actually solving the problem.

    When doing the research, I found myself asking myself ques-tions. I did the research, and then I did more research basedon the questions I had asked myself from the original researchthat I found. I think that by nally asking myself questions[sic] like Why? and How come?, I was able to get more outof what I was trying to accomplish.

    Students were learning to focus on relevant resources, ask spe-cic questions about their research, and make connections be-tween research and problem solving objectives. Some of thelessons about how to manage and use resources included theimportance of sharing research and paying attention to others re-search (italics are added):

    Human Behavior 26 (2010) 8499vidual and collective learning goals may be a possible explanationfor why Group L was less strategic in dividing research topics andless likely to interact over their research.

  • rs in4. Discussion

    In this study, student as a group visited close to half of the avail-able video resources and close to three quarters of knowledge re-sources. Although the exploration was by no means extensive orexhaustive, resources still enriched students learning in a numberof ways in this study. Resources introduced students to a set ofnew educational psychology concepts and provided concrete con-texts to anchor the concepts. Resources also helped students gen-erate ideas for problem solutions. The enriched problemrepresentation served as a situation model, based on which stu-dents generated and projected potential solutions. Ideas for futurepractice and professional perspectives as a teacher were con-structed in this process as well.

    This success, however modest it might be, did not happenmerely by providing resources to students. One of the key factorsresponsible for the success is the use of PBL. In PBL, students needto build their own understanding from the available resources.What they research determines what they learn in PBL. The specicfeatures of STELLAR also played a role. STELLAR simplied the pro-cess of searching and evaluating online resources by providing pre-selected resources that were manageable and quality-controlledalong with a set of links to outside resources. Students were ableto concentrate on researching and processing resources rather thanspend a large amount of time searching and selecting informationin a purely open-ended fashion. STELLAR also provided tailored re-sources for different learning activities. Video resources assistedstudents with the construction of problem representation. Knowl-edge resources assisted students with proposal development. Re-source use was also guided by explicit and subtle promptsembedded in STELLAR. Features of PBL and STELLAR played animportant role in prompting students to pay more attention to re-sources and relate the contents of the resources more closely totheir learning activities.

    Although resources were accessed fairly frequently and in turncontributed to students learning and problem solving in thisstudy, this study also suggested that much is desired with respectto student management and processing of resources. Althoughmore evidence is needed, we identied a number of the areas aspotentially important for productive resource use in this study.First, there is a need to know the resources when students workin learning environments rich with resources. They need to learnwhat kinds of resources are available out there and what theirmain characteristics are (e.g., how the Encyclopedia Britannicamight differ from Wikipedia). Learners do not necessarily need toknow the contents of all the resources, but they should know theresources enough to determine what kinds of resources might beconsulted when specic learning needs arise. Students also needto have the knowledge and skills needed to locate the resourcessuccessfully, which may include computer or information literacy(Wecker, Kohnle, & Fischer, 2007) or search expertise (MaKinsteret al., 2002).

    When students are initiated into a new learning environmentfor the rst time, they need to know the environment itself. In thisstudy, Group H explored the educational psychology conceptswidely in Problem 1 but not in Problem 2. Group H appeared tobe mindful of the need to learn about the new learning environ-ment, in this case, what kinds of resources were out there andhow to navigate the environment. By exploring what kinds of con-cepts were covered in the KW and how to navigate them, Group Hwas exploring the resource space of STELLAR in Problem 1. Oncethey achieved this goal to some extent in Problem 1, they engagedin more targeted exploration of resources in Problem 2. Unlike

    H. Jeong, C.E. Hmelo-Silver / ComputeGroup H, students in Group L did not seem to realize the need toexplore the affordances of the new environment and were focusedon exploring resources directly relevant to their current problemsfrom the beginning. It is not clear whether students in Group Lwere just unmotivated or whether they did not recognize the needto explore the resource space in their new learning environment. Itmight be that developing an understanding of available resourcesmay pose excessive cognitive demands for lower achieving stu-dents, and additional scaffolding may be needed (Hmelo-Silver,Duncan, & Chinn, 2007; Kirschner, Sweller, & Clark, 2006).

    Second, students need to understand that resources need to beprocessed more carefully. For learning resources to be useful inbuilding content understanding, learners need to use them con-structively, summarizing, interpreting, and reecting upon them(Chi, DeLeeuw, Chiu, & LaVancher, 1994). Engaging in constructivelearning activities is useful in all learning situations, but the needfor such activities is greater when learning depends on self-direc-ted resource use. Learning resources, even when they are appropri-ate to the level of students understanding, are often generic anduntailored to the specic learning tasks or objectives. It is alsomore likely that there are some unresolved disagreements amongdifferent resources. In the absence of teachers and textbookauthors, the burden of contextualizing, evaluating, extracting,and integrating information is with the learner. In spite of thegreater need for deeper processing of resources, however, studentsdo not necessarily engage in such processing. In this study, stu-dents approached resources rather shallowly. Students, especiallylow-achieving students, often skimmed and skipped resourcesand rarely went beyond copy and pasting. High-achieving studentsdid a better job, but not by much. Copy and pasting is useful at cer-tain stages of gathering and processing information (Macdonald,Heap, & Mason, 2001). However, it should be a preparation for laterreective activities rather than the nal destination. Students haveto understand the need to process resources in a more careful man-ner, thinking deeply about the relevance of the resources to thecurrent learning activities and asking why and how. Students donot need to process every resource in depth, but they need to doso with a set of core resources that they deem important.

    Finally, students need to learn about collaborative resource use.Collaborative group research and/or group problem solving arequite common in resource-rich hypermedia learning environ-ments. Students typically have relevant but partial informationabout the problem, which needs to be pooled for successful groupproblem solving. It is also often impossible for any individual stu-dents to cover the entire resource space of a given problem. Forcollaborative problem solving to be successful, students need toshare and integrate their individual resources and researchendeavors effectively. This can mean a number of things rangingfrom eliciting the unshared information and perspective fromother members to dividing up the research task in such a way thatmaximum research space is covered and yet individual learningneeds are still met. In addition, upon completing individual re-search, students not only need to share and post their individualresearch, but also to pay attention to and use other students re-search. Unless individual resources are shared and incorporatedinto the collective problem solving process, they are unlikely tomake signicant inuence on the collective learning outcomes. Inthis study, although Group H was somewhat better than Group L,the overall level of interaction over resources was low in bothgroups. Students might share summaries of their research, but theyinfrequently interacted over their individual research. They seldomdiscussed the relevance of research to the given problems or howdifferent pieces of research t together. Students reection sug-gested that some of them learned these skills as they went along,but efforts are needed to better facilitate collaborative resource

    Human Behavior 26 (2010) 8499 97use so that resources can be better researched, shared, and utilizedin collaborative problem solving contexts.

  • eno et al., 1996). Increasingly, however, this idea is being chal-

    sources, how to locate and nd them, and how to better make use

    rs inof them is increasingly becoming an essential competency in manydomains.

    Notwithstanding the insights gained from this study, it isimportant to note that the