Personalizing the Selection of Digital LibraryResources to Support Intentional Learning

Qianyi Gu1, Sebastian de la Chica1, Faisal Ahmad1, Huda Khan1,Tamara Sumner1, James H. Martin1, and Kirsten Butcher2

1 Department of Computer Science, Institute of Cognitive ScienceUniversity of Colorado at Boulder, Boulder CO 80309, USA

{qianyi.gu,sebastian.delachica,faisal.ahmad,huda.khan,tamara.sumner,james.martin}@colorado.edu

2 Learning Research and Development CenterUniversity of Pittsburgh, Pittsburgh,PA 15260, USA

kbutcher@pitt.edu

Abstract. This paper describes a personalization approach for usingonline resources in digital libraries to support intentional learning. Per-sonalized resource recommendations are made based on what learnerscurrently know and what they should know within a targeted domain tosupport their learning process. We use natural language processing andgraph based algorithms to automatically select online resources to ad-dress students’ specific conceptual learning needs. An evaluation of thegraph based algorithm indicates that the majority of recommended re-sources are highly relevant or relevant for addressing students’ individualknowledge gaps and prior conceptions.

Keywords: Personalization, Information Retrieval, Intentional Learn-ing, Knowledge Map.

1 Introduction

Educational digital libraries have made available a vast amount of educationalresources for educators and learners. However, with this great number of re-sources, learners still face the challenge of effectively accessing and using thesedigital resources to address their specific learning goals or conceptions. Cognitiveresearch has shown that maximum learning benefits can be achieved when learn-ing is personalized using learners’ prior knowledge, individual differences andlearning styles [7]. Personalized information retrieval can support ”intentionallearning” by providing educational resources based on individual students’ learn-ing requirements. Intentional learning consists of ”cognitive processes that havelearning as a goal rather than an incidental outcome” [4]. Intentional learningoccurs when learners choose to empower and transform themselves by settinggoals and using strategies and processes to ensure learning [16].

Existing approaches for providing personalized learning support, such as adap-tive learning environments, provide individualized content and pedagogy [21].

B. Christensen-Dalsgaard et al. (Eds.): ECDL 2008, LNCS 5173, pp. 244–255, 2008.c© Springer-Verlag Berlin Heidelberg 2008

Personalizing the Selection of Digital Library Resources 245

Although these environments have demonstrated learning impacts, they are notscalable across different domains or different learner models. These systems re-quire substantial human effort in the initial creation of learner profiles and do-main knowledge. This paper described an automated mechanism for supportingpersonalized learning through the suggestion of educational resources target-ing learners’ conceptions and gaps in knowledge within a particular domain. Wepresent a digital library personalization service which: (1) is based on an individ-ual learner’s prior knowledge, (2) is domain-independent, (3) requires no humaneffort to build the learner profile, and (4) supports intentional learning. Oursystem automates resource recommendation based on dynamically constructedmodels of students’ misconceptions and current knowledge.

We first describe related work on adaptive learning environments. An exampleis used to illustrate personalized resource recommendations within a functioninglearning environment. The technical approach used to automatically recommendonline resources based on students’ prior knowledge is discussed in detail. Fi-nally, we present the results of our evaluation of the automatic resource recom-mendation service, and we outline our future work for supporting personalizedlearning.

2 Related Work

Constructivism can inform the design and evaluation of personalized learning en-vironments. This learning theory sheds light on individual learners’ knowledgeconstruction and integration processes [14]. According to this theory, learning isan active process in which new knowledge is continuously integrated with exist-ing knowledge [9]. As such, learners’ prior conceptions and current knowledgeprofoundly influence how they understand new concepts.

The design of our personalized resource recommendation service both borrowsand differs from other personalized learning environments. Adaptive learning en-vironments, such as intelligent tutoring systems and adaptive hypermedia learn-ing environments, personalize content or pedagogy based on profiles of learnersand models of domain knowledge [17]. Cognitive tutors, such as the PracticalAlgebra Tutor [2] [15] and Auto Tutor [11], compare representations of studentunderstanding with a knowledge model depicting an ideal representation of whatstudents ought to know about a domain. This formalized domain model guidesthe selection and presentation of appropriate materials to the learner. Adaptivehypermedia learning environments, such as INSPIRE [12] and AHA [5], providevarying levels of adaptive presentation and adaptive navigation based on modelsof learners’ knowledge states and learning style preferences [19].

Many adaptive learning environments have been shown to improve learn-ing. However they also highlight the major disadvantage of detailed knowledgemodeling: there is significant initial cost and human-intensive effort required todevelop accurate domain models. These models typically are expensive, difficultand impractical to implement for a wide variety of topics. As a general rule, themore detailed the conceptual feedback offered by technology, the less able the

246 Q. Gu et al.

technology is to scale quickly to new tasks, domains, and disciplines. Our per-sonalized resource recommendation service does not require intense human effortfor the construction of a domain model. Rather, as our example demonstrates,this service utilizes dynamically constructed representations of user knowledgeand domain knowledge.

Our resource recommendation service employs knowledge maps to representstudent and domain knowledge and graphs to represent concepts covered in aresource and concepts which students should know. Knowledge maps are a semi-formal knowledge representation that uses a network layout containing richly de-scriptive statements in nodes to capture concepts and ideas related to a domain,and a limited number of link types to depict important relationships betweenconcepts [13], [20]. Graphs are an effective mathematical construct for modelingrelationships and structural information. Many applications in information re-trieval and machine learning model data as graphs since graphs can retain moreinformation than vectors of simple atomic features.

3 An Example

Here we illustrate how personalization using online resources in digital librariescan support intentional learning. Our approach is based on a research projectcalled ”customized learning service for concept knowledge” (CLICK). The goalof CLICK is to design and evaluate an end-to-end prototype which will enablestudent-centered customizations by comparing students’ conceptual understand-ing, depicted as knowledge maps, with reference domain knowledge maps gener-ated by analyzing digital library resources [10].

In this example, an undergraduate student from the University of Coloradois assigned the task of writing an online essay on the cause of earthquakes usingour CLICK personalized learning environment. As shown in Fig. 1, the studentis in the process of writing her essay. The left pane contains the essay editor andthe right pane provides personalized resource recommendations to inform heressay writing. In her essay, the student wrote ”These are so small and insignifi-cant because the plates are so thin and not big enough to be capable of a largescale earthquake.” The system analyzes the student essay and identifies this sen-tence as a misconception in her scientific understanding. The student appearsto believe that the magnitude of earthquakes is related to the size of continentalplates. The system highlights this sentence in the essay editor and automati-cally recommends a small selection of digital library resources to address thisspecific misconception. Currently, the system presents the student with up tofive possible misconceptions in the right pane. For each potential knowledge gapor conceptual need, the system suggests three digital library resources, includ-ing specific pages within the larger resource, and it provides a general cognitiveprompt to encourage the student to reflect on the relationship between her essayand the suggested resources.

The CLICK personalized learning environment uses natural language process-ing techniques to identify misconceptions by comparing automatically constructed

Personalizing the Selection of Digital Library Resources 247

Fig. 1. CLICK Personalized Learning Environment

domain knowledge maps with the learner’s knowledge map constructed from heronline essay. The components for constructing knowledge maps and identifyingmisconceptions are reported elsewhere [1], [10]. As shown in Fig. 2, these com-ponents provide the personalized resource recommendation algorithm with threeinformation elements: (1) a list of misconceptions for a specific student essay, (2)a selected portion of the learner knowledge map associated with each identifiedmisconception, and (3) a selected portion of the domain knowledge model associ-ated with each identified misconception. In the following section, we will describehow these information elements are used by the personalized resource recommen-dation algorithm to recommend digital library resources targeting specific learnerknowledge gaps and prior conceptions.

4 Technical Approach

The algorithm employs a four-step process:

1. It first transforms the domain and student knowledge map representationsinto a computational data model called a concept matrix.

2. Then it processes the concept matrix and the student’s misconceptions toconstruct a concept graph representing the student’s prior knowledge andlearning needs. This graph offers an efficient mechanism for representingboth what the student knows and what the student should know.

3. The algorithm then identifies up to twenty digital library resources as likelycandidates for recommending. For each identified resource, it constructs aresource concept graphs representing the concepts and the relationships be-tween them as described in that particular resource.

248 Q. Gu et al.

Fig. 2. Major elements of the Personalized Resource Recommendation Algorithm

4. Using graph similarity measurement techniques, the algorithm selects themost promising resources to address that student’s specific knowledge gapsand learning needs.

4.1 Concept Matrix Construction

The concept matrix provides a computational data model for efficiently repre-senting concepts and their relationships at a fine-grained level. It transformsour domain and learner knowledge maps into a single data structure represent-ing both the student’s current and desired knowledge. Both dimensions of theconcept matrices are composed of key concepts extracted from nodes in theknowledge maps; the values in the matrices are the conceptual distance betweenthe corresponding pair of key concepts as represented by the links (distances) inthe knowledge maps.

The knowledge maps are successful at representing domain knowledge andstudent’s prior knowledge by generating nodes composed of sentences represent-ing a concept unit and by revealing relationships between different knowledgenodes [10]. However, this sentence-based representation does not provide thefine-grained granularity necessary to support personalized information retrievalsince online resources are indexed at the level of terms. The concept matrix notonly represents the key terms, but also represents the relative position and struc-ture of those terms within the knowledge maps. Compared to plain text, this isan important and unique feature that the knowledge map brings in. The struc-ture and positions of key concepts represent semantic structure of the students’prior knowledge. The structure and positions of key concepts are representedthrough the conceptual distance value in the concept matrix.

4.2 Concept Graph Generation

The concept matrix is then used to construct the concept graph representing thestudent’s knowledge status and learning needs. For instance, one student wrote in

Personalizing the Selection of Digital Library Resources 249

her essay ”Seismographs measure earthquakes occurring along lines where platetectonics are located.” Our personalized learning environment detects there is amisconception for the student’s scientific understanding based on this sentencebecause: ”this is an incomplete understanding of what seismographs do. Theydon’t measure earthquakes, but rather measure ground motion and time.” Giventhis identified student misconception, the graph construction algorithm processesthe concept matrix to represent key concepts as graph vertices and uses a labelingfunction to assign weights to each vertex to represent the importance of eachconcept. The automatically generated concept graph based on this misconceptionis shown at Fig. 3.

Fig. 3. Student Concept Graph

The formal definition of such graphs and labeling functions are below:

Definition 1. A graph g is defined as: g = (V, E, α, β) where:V is the finite set of verticesE is the finite set of edges: E = V × Vα : V → Lv is the labeling function for vertices. Lv is set of labels which areappear on vertices of the graph.β : E → Le is the labeling function for edges. Le is set of labels which are appearon edges of the graph.|g| is the size of graph g, where: |g| = number of elements in V

Definition 2. Terms and weights computation functions from labels of graphvertices are defined as:δ : Lv → T . The set T is set of concepts represented at each vertex of the graph.λ : Lv → W . The set W is set of numiric weights represented at each vertex ofthe graph.

It also constructs a set of online resource concept graphs from digital librariesaligned to that student concept graph. The on line resource concept graph is therepresentation which consists of resource text, metadata description and hyper-link web pages from the online digital library resource. This representation isconstructed through this process: The online resource pages and metadata in the

250 Q. Gu et al.

digital libraries are crawled and indexed in the system. From this index, a set ofresource concept graphs is dynamic generated aligning to the target student con-cept graph. The method for representing such web document content as graphsis based on examining the terms on each web page and their adjacency. Whenwe create a concept graph of online resource content, key terms are extractedby looking for the aligned vertex labels (δ) in student concept graph and eachkey term becomes a vertex in the resource concept graph. We connect the pairof vertices which is within 20 words distance in the original document with anedge that is labeled with the distance between them.

4.3 Resource Selection

The resource selection is based on the similarity between student concept graphand resource concept graph. Much research has been performed in the area ofgraph similarity in order to exploit the additional information allowed by graphrepresentations to introduce mathematical frameworks for dealing with graphs[8]. We use graph distance to compute graph similarity between student con-cept graph and resource concept graph. Graph distance is a numeric measure ofdissimilarity between graphs, with larger distances implying more dissimilarity.By graph similarity, we are interested in some measurement that tells us howsimilar the pair of graphs is.

We have generated a student concept graph for each particular learning task torepresent the student’s prior knowledge and misconceptions and learning needs.Then we use graph distance to measure the distance between this particularstudent concept graph against each resource concept graph we generated fromonline resources in digital libraries. The best matched resource graph in graphdistance computation represents the appropriate online resource to address thestudent’s particular learning needs in this current learning task.

We measure the similarity of a pair of graphs based on the computation ofthe maximum common subgraph. Previous research work [6] have shown thatthe size of the maximum common subgraph related to the similarity between apair of graphs. We define maximum common subgraph in Definition 3,4.

Definition 3. Let two graphs g1 and g2 where:g1 = (V1, E1, α1, β1) and g2 = (V2, E2, α2, β2)g2 is defined as subgraph of g1 as: g2 ⊆ g1 if:V2 ⊆ V1 and E2 ⊆ E1α2(x) = α1(x) for all x ∈ V2β2(y) = β1(y) for all y ∈ E2

Definition 4. Let two graphs g1 and g2 where:g1 = (V1, E1, α1, β1) and g2 = (V2, E2, α2, β2)g is defined as Maximum Common Subgraph of g1 and g2 if:g ⊆ g1 and g ⊆ g2For all the g′ that satisfy: g′ ⊆ g1 and g′ ⊆ g2, we have: |g| ≥ |g′|

Personalizing the Selection of Digital Library Resources 251

We measure the similarity of a pair of graphs G1 = (V1, E1, α1, β1) and G2 =(V2, E2, α2, β2) based on the size of maximum common subgraph:

Similarity(G1, G2) = 1 − |MaximumCommonSubgraph(G1,G2)|max(|G1|,|G2|)

We compute the maximum common subgraph of a pair of graphs, where bothgraphs (G = (V, E, α, β)) satisfy the condition that:∀(v1, v2 ∈ V ), we have : δ(α(v1)) �= δ(α(v2))

For any pair of graphs, we preprocess them to assure that they satisfy thecondition above. The preprocessing steps for any graph G = (V, E, α, β) are:

∀(v1 ∈ V, v2 ∈ V ) if δ(α(v1)) = δ(α(v2)), DELETE v2; ASSIGN λ(α(v1)) =λ(α(v1)) + λ(α(v2))

After this preprocessing, both graphs satisfy the condition above. Then theconstruction of the Maximum Common Subgraph G = (V, E, α, β) procedure forGraph G1 = (V1, E1, α1, β1) and G2 = (V2, E2, α2, β2) are:

∀(v1 ∈ V1, v2 ∈ V2) if δ(α1(v1)) = δ(α2(v2)), CREATE v ∈ V and ASSIGNδ(α(v)) = δ(α1(v1))

∀((vs ∈ V, ve ∈ V, v1s ∈ V1, v1e ∈ V1, v2s ∈ V2, v2e ∈ V2) ∧ (δ(α(vs)) =δ(α1(v1s)) = δ(α2(v2s))) ∧ (δ(α(ve)) = δ(α1(v1e)) = δ(α2(v2e)))) IF ∃((e1 =(v1s, v1e) ∈ E1) ∧ (e2 = (v2s, v2e) ∈ E2)) CREATE e ∈ E, e = (vs, ve)

5 Preliminary Evaluation and Iterative Design

We conducted an evaluation where 23 University of Colorado first-year studentswere asked to write essays about earthquakes and plate tectonics. Four geol-ogy and instructional design experts analyzed these essays to identify potentialscientific misconceptions. For each essay, three individual misconceptions wereidentified by the experts. We used a Wizard of Oz setup in which a human en-acted the resource selection algorithms based these identified misconceptions toprovide personalized online resource recommendations from the Digital Libraryfor Earth System Education (DLESE - www.DLESE.org). The algorithm rec-ommended three digital library resources for each student misconception. Eachrecommendation was presented as two URLs: the top level resource URL and aspecific recommended page within the resource.

To evaluate the performance of the algorithm, we gave an Earth science educa-tion expert a recommendation report containing a list of specific misconceptionsand their associated resource recommendations. The expert was asked to assessthe quality of each recommended resource based on how well it addressed a spe-cific student misconception. The expert evaluated how well the recommendedonline resource and particular web page addressed the student misconception byranking them as ”Highly Relevant”, ”Somewhat Relevant”, or ”Not Relevant.”Along with the scales, the expert reported reasons for each ranking. The expertalso identified what specific portion of a recommended page within a resourcewas most appropriate for addressing the student misconception.

252 Q. Gu et al.

We analyzed the data provided by the expert and identified several improve-ments for our algorithm design. Two examples of problems identified and reme-died include:

– Some resources were ranked ”Highly Relevant” because they cover core con-cepts from the domain knowledge map which are not covered in studentknowledge maps. This implies that the algorithm needs to identify core con-cepts in the domain knowledge map that are missing in student maps andlabel such concepts with high weight to give them more priority.

– Some resources were ranked ”Not Relevant” because they were for teachers,not for students. This implies that the algorithm needs to use the resourcetype field in the digital library metadata, where it is available, to focusresource recommendations on student-oriented resources.

Based on these findings, we redesigned our algorithms and conducted a secondround of evaluation with the same expert. The results of the second round arecompared with the results from the first round in Fig. 4. For both top levelresources and specific web pages, it shows what percentages of them are rankedas ”Relevant” (combining both highly and somewhat relevant). As indicated bythis comparison, after our redesign process, we have achieved better performancewith resources being ranked as relevant improving by about 20% for both toplevel resource URLs and specific pages within a resource.

Fig. 4. Evaluation Results Comparison

6 Discussion

The results from our evaluation indicate that the graph based algorithms un-derpinning the personalized resource recommendations demonstrate strong po-tential for identifying resources to address student’s individual knowledge gapsand learning needs.

Personalizing the Selection of Digital Library Resources 253

The concept matrix appears to represent mainly those concepts that are trulysalient for representing learners’ current and desired knowledge (what they doknow and should know as depicted in the knowledge maps). A deeper analy-sis of the evaluation data examined the reasons resources were rated as ”NotRelevant.” As previously mentioned, sometimes this rating was assigned dueto audience differences (teachers versus students). We also analyzed whetherresources ranked as ”Not Relevant” were being recommended because criticalconcepts and relationship were missing in the concept matrices. We found thatonly 13% of the recommendations were due to this reason; this promising levelof performance suggests that the concept matrices are representing the majorityof concepts necessary to perform effective personalization.

However, when the concept matrices are transformed to concept graphs andare assigned weights through the labeling function λ, the weights are not alwaysable to adequately represent the importance of a particular concept. Furtheranalysis of the expert’s evaluation highlighted mismatches between the miscon-ceptions prioritized by the human expert and those prioritized by the algorithm.Even though these prioritized concepts were included in the concept matrix,they were not assigned high enough weights through the labeling function, andthus the algorithms were not able to retrieve corresponding resources to addressthese important concepts.

7 Conclusion and Future Work

The research reported in this paper introduces our personalization approach forusing online resources in digital libraries to support intentional learning. We pro-vide personalized resource recommendations based on what learners know andwhat they should know in their learning process. The results from our evaluationindicate that our use of graph based algorithms to make personalized recom-mendations are effective for addressing student’s individual knowledge gaps andlearning needs. We believe this approach shows great promise for creating person-alized learning environments that adapt to individual learners. We use naturallanguage processing and graph similarity techniques to automate the genera-tion of the learner knowledge model and resource selection processes. Thus, weminimize the initial cost and human-intensive effort to build such systems andimprove on prior efforts by demonstrating an approach that is potentially scal-able to new domains and disciplines. We have just completed a controlled learn-ing study assessing the impact of personalized resource suggestions on studentlearning. Preliminary results are very encouraging: students using the CLICKenvironment appear to be engaging in deeper knowledge processing and scientificreasoning than their counterparts in the controlled condition. We will completethe data analyses and report on this study’s outcomes in the future. We are alsoinvestigating how knowledge maps can be used as user interface components inlearning environments to assist learners in synthesizing ideas from the multipleonline sources recommended by our system.

254 Q. Gu et al.

References

1. Ahmad, F., de la Chica, S., Butcher, K., Sumner, T., Martin, J.H.: Towards auto-matic conceptual per-sonalization tools. In: Proc. of the 7th ACM/IEEE-CS JointConference on Digital Libraries, Vancouver, British Columbia, Canada, pp. 452–461 (2007)

2. Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors:Lessons learned. Journal of the Learning Sciences 4(2), 167–207 (1995)

3. Bandura, A.: Perceived self-efficacy in the exercise of personal agency. The Psy-chologist: Bulletin of the British Psychological Society 2, 411–424 (1989)

4. Bereiter, C., Scardamalia, M.: Intentional learning as a goal of instruction. In:Knowing, learning, and instruction: Essays in honor of Robert Glase. ErlbaumAssociates, Hillsdale, New Jersey (1989)

5. Bra, P.D., Smits, D., Stash, N.: Creating and delivering adaptive courses with aha!In: First european conference on technology enhanced learning, Crete, Greece, pp.21–33. Springer, Heidelberg (2006)

6. Bunke, H., Shearer, K.: A graph distance metric based on the maximal commonsubgraph. Pattern Recognition Letters 19, 255–259 (1998)

7. Chi, M.T.H.: Three types of conceptual change: Belief revision, mental model trans-formation, and categorical shift. In: Handbook of research on conceptual change.Erlbaum, Hillsdale (in press)

8. Cormen, T.H., Leiserson, C.E., Rivest, R.L.: Introduction to Algorithms. The MITPress, Cambridge (1997)

9. Dalgarno, B.: Interpretations of constructivism and consequences for computerassisted learning. British Journal of Educational Technology 32(2), 183–194 (2001)

10. de la Chica, S., Ahmad, F., Sumner, T., Martin, J.H., Butcher, K.: Computationalfoundations for personalizing instruction with digital libraries. International Jour-nal of Digital Libraries, Special Issue on Digital Libraries and Education (2007)

11. Graesser, A.C., Lu, S., Jackson, G.T., Mitchell, H.H., Ventura, M., Olney, A.:Autotutor: A tutor with dialogue in natural language. Behavior Research Methods,Instruments, Computers 36(2), 180–192 (2004)

12. Grigoriadou, M., Papanikolaou, K., Kornilakis, H., Magoulas, G.: Inspire: An intel-ligent system for personalized instruction in a remote environment. In: Proceedingsof third workshop on adaptive hypertext and hypermedia, Sonthofen, Germany(2001)

13. Holley, C.D., Dansereau, D.F.: Spatial learning strategies: Techniques, applications,and related issues. Academic Press, Orlando (1984)

14. Hung, D., Nichani, M.: Constructivism and e-learning: balancing between the indi-vidual and social levels of cognition. Educational Technology 41(2), 40–44 (2001)

15. Koedinger, K.R., Anderson, J.R.: Intelligent tutoring goes to school in the big city.International Journal of Artificial Intelligence in Education 8, 30–43 (1997)

16. Martinez, M.: Designing intentional learning environments. In: Proceedings of the15th annual international conference on Computer documentation, October 19-22,1997, pp. 173–180, Salt Lake City, Utah (1997)

17. Park, O.-C., Lee, J.: Adaptive instructional systems. In: Handbook of research oneducational communications and technology, 2nd edn., pp. 651–684 (2004)

18. Shen, R., Goncalves, M., Fan, W., Fox, E.: Requirements Gathering and Modelingof Domain-Specific Digital Libraries with the 5S Framework: An ArchaeologicalCase Study with ETANA. In: Rauber, A., Christodoulakis, S., Tjoa, A.M. (eds.)ECDL 2005. LNCS, vol. 3652, pp. 1–12. Springer, Heidelberg (2005)

Personalizing the Selection of Digital Library Resources 255

19. Sugiyama, K.: Studies on Improving Retrieval Accuracy in Web Information Re-trieval. PhD thesis, Nara Institute of Science and Technology (2004)

20. Sumner, T., Ahmad, F., Bhushan, S., Gu, Q., Molina, F., Willard, S.: LinkingLearning Goals and Educational Resources through Interactive Concept Map Vi-sualizations. International Journal on Digital Libraries 5(1), 18–24 (2005) (SpecialIssue on Information Visualization Interfaces for Retrieval and Analysis)

21. Wade-Stein, D., Kintsch, E.: Summary street: Interactive computer support forwriting. Cognition and Instruction 22(3), 333–362 (2004)