[IEEE 2013 Fourth International Conference on E-Learning and E-Teaching (ICELET) - Shiraz, Iran (2013.02.13-2013.02.14)] 4th International Conference on e-Learning and e-Teaching (ICELET

The 4th International Conference on e-Learning and e-Teaching, ICELET 2013

Iraq E Lea ... "I"g Anoelallon

� '/",/ " '..i �O-I'

Shiraz University

A Supporting tool in Online Learning Forums based on

Multi-Documents Summarization

Mohammad Reza Fani Sani

Advanced e-Learning Technologies Lab

Ahmad A. Kardan


Arman Cohan


Amirkabir University of Technology Tehran, Iran

[email protected]


[email protected]


[email protected]

Abstract-Online discussion groups and forums now hosts

millions of users that join them with different purposes, to

discuss various matters or ask their questions or providing

answers to others' questions. Web 2.0 and online forums provide

an environment that supports the following important

functionalities: content generation, content and knowledge

sharing, synchronous and asynchronous interaction and so on.

With increasing usage of collaborative learning these forums and

communities are known as a suitable platform that facilitates

collaborative activities. Many learners refer to these forums to

find answers for their questions. Unfortunately due to huge

amount of information on these forums, finding the appropriate

answers is becoming more time consuming. In addition, there is

no suitable mechanism to measure the reliability of answers

being provided. To overcome these challenges a new supporting

tool for e-Learning in online communities and forums is

proposed. This supporting tool consists of three main

components. At first relevant questions to learners query is

found. The next component finds the best answers for these

relevant questions. Finally, selected answers are summarized and

an abstract view of these answers that contains the most

important information is returned to the learner. To evaluate the

proposed tool the contents of 30 discussions are collected. The

results of applying this supporting tool show that it can help

learners to find their answers in learning forums more easily.

Collaborative Learning; e-Learning; Online Forums; Learning Forums, Text Summarization

I. INTRODUCTION

Internet and web allowed all the users to access information openly. In the beginning, structure of the web was in a way that regular users were only able to use the available information on the web. Although with invent of web 2 technology and social web, users could generate their own content and share their content with other users. It can be said that the social web, is based on user collaboration and content sharing. Thanks to the social web technologies, the speed of content generation and the total volume of data on the web, increased substantially. However, as the data were increasing, the task of finding some specific information was becoming harder. As a

978-964-462-445-2/13/$31.00 © 2013 IEEE

30

result the role of search engines has become more and more significant. Search engines help users to find their desired information by searching the web, based on the user's query. But users sometimes demand answers to a particular questions. Search engines are still unable to process the queries that are questions and their question answering capabilities are still limited. One group of popular applications in social web, is online commullltIes and forums in which many users share information. These forums are also used extensively in collaborative learning environments in which learners can share their experiences, find answers to their questions or answer other learners' questions [1]. One important requirement in effective e-Learning environments is interaction between learners [2]. This interaction can be synchronous or asynchronous [3]. As it was mentioned before, discussion forums are one of the most important supporting tools in e-Learning environments to support asynchronous collaboration [4]. Learners refer to these forums to discuss course contents, ask questions about a specific topic from instructor or other students, provide more information about the course topic, etc. Another advantage of forums is that questions and answers are categorized based on the topic of discussions [5]. It means that all the questions that relate to particular topic are grouped into one topic discussion. It is highly probable that a question be common among many learners. So if a learner is searching for an answer to his/her question, the probability that he/she finds his own answer among the answers relating to a similar question, is significantly high. However, usually answers to the questions are numerous and browsing the answers for the best one, is highly time consuming. Furthermore, not all answers have the same credibility. One important factor in determining the credibility of an answer is the expertise of its author. For example an answer that is posted by an expert in a particular knowledge domain or an instructor is not as reliable as an answer that is posted by a novice learner in that domain [6]. In this paper a supporting tool for e-Iearning is proposed that facilitates learners to find their desired information and


Iraq E Lea ... nlng Anoelallon

potential answers to their questions. This system recommends a number of relevant questions along with some selected answers in response to a learner's query. This makes it possible for the learner to know what questions have been previously asked that are similar to their own question. Moreover, by providing the abstract of important existing answers, it helps the learner to know whether the existing answers are sufficient to answer their question or not. The rest of the paper is organized as follows. In section 2 the related works are briefly reviewed. Section 3 discusses existing important approaches to text summarization that can be used in online learning forums. In section 4 the proposed supporting tool for online learning forums is described and the evaluation results of this system are discussed in section 5. Finally the paper is concluded in section 6.

II. RELATED WORK

With regard to the increasing volume of information on the web, one important research domain information management has been organization and optimal usage of the information. Text summarization as a method to organize the existing text resources, is of utter importance [7]. The goal in text summarization is to convey the main concept of a document, or a part of a document, by a limited number of sentences [8][9]. Generally there are two methods for text summarization: Methods that are based on extracting sentences and methods that are based on generating sentences [10 ]. Each method has its own usage considering the context. In recent decade, discussion groups as one of the most important tools for collaborative learning [11] have been discussed. In discussion groups, generating a summary of each discussion can play an important role in knowledge elicitation and group evaluation. It also can help the new users to understand the conclusions of each discussion without the necessity to read all the discussion. There have been some text summarization methods that can be employed in this domain. In [9] and [12] a method is proposed that extracts a summery from a number of documents with relevant titles. In [8] a language independent approach for multi document summarization that is based on graph rating. In [13] a centroid based text summarization method is proposed that selects a subset of sentences from all sentences in a document as a summery for that document. None of these methods are suitable to be applied to discussion groups and forums due to collaborative aspects of these groups in comparison with multi document environments that are used in these methods for evaluation. General text summarization differs from text summarization in discussion groups and forums. In discussion groups each post do not correspond directly to the previous posts. Also the reliability and credibility of the posts are not equal and they must not be considered in the same way. In order to build a better conversational model, questions and answers can be used. That means the posts in a topic must correspond to topic question. This helps to reduce the semantic distance between the posts and converge them.

Figure 1. The overview of the topic based mutli-document text summarization method

31

� ;t;(..!.1,

Shiraz University

III. TOPIC-BASED MUTLl-DoCUMENT TEXT

SUMMARIZA nON

This method is the best text summarization method to be used in online learning forums because in these forums, various responses to a question can be considered in one broader concept. In this method sentences which are related to the desired topic are extracted, scored and sorted according to their scores. Then unique sentences with highest scores are introduced as the summary of the documents. Fig. 1 shows the overall view of this method. The fIrst part of this method relates to the analyzing the question, so that the retrieved summary be in line with the question. The question is compartmentalized to the most important elements and also the entities in the question are extracted. In the next phase, the documents are analyzed. This process relates to the information that should appear in the summary from the documents. Before this step there is a need to preprocess the documents to separated paragraphs and with timestamps. In the next phase documents are classifIed based on topic to determine the dependency of the sentences and their relation to the topic. The role of this phase in discussion groups and forums is not as signifIcant as other phases because in these communities the answers to a question are highly relevant to the topic of the question. The next phase which is the most important phase, involves scoring all the sentences based on their relation with the topic of question. This is done by linear combination of some parameters that their value is between 0 and 1. These parameters are: 1- The number of words that are present in both the question and answer. To increase the functionality of this parameter sematic tools like Wordnet dictionary can be used to capture similar words. 2- The position of sentences; usually sentences that are in the beginning of the document carry more semantic weight. This rule is observed in many languages including English. 3- The semantic weight of the words is another effective parameter that can be obtained by TF-IDF method. 4- Cosine similarity is another parameter that is used in many document similarity methods. This measure is calculated by making use of the document vectors that are obtained by their words and the amount of similarity is calculated by equation 1.

In which Sim(Q,DJ denotes the amount of similarity between the question and the document i. Also dlj denotes the weight of the word Ti in document i and Wq/ shows the weight of the word � in the question. This parameter analyses the similarity of the sentences by words. The maximum similarity is obtained when the high weighted words in one sentence carry a high weight in another sentence as well. To obtain the weight of the words TF-IDF is used. TF-IDF is a popular method in information retrieval which is extensively described in [14] and cosine similarity is discussed in [12].


Iraq E Lea ... nl"g Anoelallon

The next step includes the summarization of sentences; in this step sentences that are used to support another sentence are omitted. These sentences include the ones that are between {}, o and also sentences that come after explanatory words like "that", "which", "where", etc.

Analysing the Qeusti o n J Analysing the Answe r "I

�=====_. J ��== =s =c =o =r i=n =g=th= e==

s e= n= t=e =n =s e= s==========l

Summarizati on of the sente nses

Combining sentense s J Figure I. The overview of the multi-document text summarization

method

IV. THE PROPOSED FRAMEWORK FOR OPTIMAL TEXT

SUMMARIZATION IN LEARNING FORUMS

The proposed framework for optima text summarization in forums and online communities is given in Fig. 2. As it can be seen in the figure, this framework consists of three main components. The first component receives the query of the learner and adjusts it to the contents of the forum and extracts the questions in the forum that are relevant to the learner's query. The second component finds the experts and important answers and the last component summarizes the important answers and recommends it to the learner. In the first part, the keywords of the learner query are extracted and then the query is expanded to match more relevant questions using a semantic aware query expansion algorithm. The reason for using the query expansion algorithm to generalize learner's query is that in most information retrieval methods, the assumption is that users know what they are looking for, whereas in e-Learning systems learners my not know their needs explicitly. Hence, their query needs to be generalized to cover broader range of matching contents. One well performing query expansion method in e-Learning systems that can be utilized in online learning forums is proposed in [15]. Using this algorithm the learner's query is expanded by adding similar keywords and hyponyms to the learner's keywords. Next, PageRank and HITS algorithms are used [16] to find the most relevant questions to the learner's query in the learning forum. To find the experts, the context based expert finding algorithm proposed in [17] was used. In this algorithm first the community expertise network of the forum is constructed that shows the expertise of each user in the network as well as overall distribution of the expertise among users. Then the edge weight of the edges for this network is computed using the equation 2. In second part for each questions that are extracted in part 1, the most important answers are extracted. Important answers are usually posted by the experts. The more expert is an author

32

� '/",/ " '..i �O-I'

Shiraz University

in a particular knowledge domain, the higher the probability that his/her answer is helpful or important. To find the experts, the context based expert finding algorithm proposed in [17] was used. In this algorithm first the community expertise network of the forum is constructed that shows the expertise of each user in the network as well as II

Keyword Extraction

Finding Relevant Questions

Web Information Retrieval and Query Expansion

Answers retrieval S Zr--��"'-'-"'-� Finding Best Answers

Finding the experts Finding Best Answers

Sentence Scoring

Summarizing the answers

Sentence Comparis on Sentence combination

Relevant questions with the summary of best answers

Figure 2. The proposed supporting tool consists of three main components: a) The component for finding relevant questions, b) the component for finding best answers and c) the component for summarizing the answers

distribution of the expertise among users. Then the edge weight of the edges for this network is computed using the equation 2.

In this equation WAB refers to the weight of the edge between users A and B, NAB denotes the aggregate number of B's answers to A's questions. Np is the number of tags of post P,

C is the related context, T shows the tag and d is the ass distance function between context C and tag T. Then using the equation 2, the transition probability matrix for the network is built and using this matrix the experts in each context are determined. For more details about context based expert finding algorithm refer to [17]. After determining the experts and high rated answers, the best answers to each question are known. In the third component of the proposed system, the most important answers for each question are sununarized. For this goal, multi document text compression methods are used. The only difference is that to find the relevant answers, the candidate answers that are retrieved in the part 2, are used. In this component firstly a score is allocated to each sentence of the answers using the linear summation of the parameters that had been discussed in section 3. As it was previously mentioned in section 3, after scoring, sentences are compressed and auxiliary parts are omitted. Finally the first n sentences are selected based on a threshold that can be optimized later and are sorted based on the timestamps.


Iraq E Lea ... nl"g Anoelallon

At last in answer to the learner's query, some relevant questions along with the best answers to each one are returned. The learner is able to see an abstract of key answers which might answer his/her question. If the answers are not sufficient he/she can refer to the complete discussion. This helps the learner not to read all the relevant questions with all of answers and therefore save a considerable amount of time.

V. EVALUATION

To evaluate the proposed tool the contents of 30 discussions of metafilter I forum is collected. In this forum learners could answer questions in different domains and then others answer their questions. After gathering these data, for each question three best answers are selected. Then using these answers and a developed summarizer an abstract of answers is built. The statistical information of this gathered data is shown in table l. As it is illustrated in table 1 the average length of all answers to a question is S03 words in each discussion. It is highly time consuming if a learner wants to browse all these answerers. As it was explained in previous section, 3 answers for each question are selected using the context based expert finding algorithm. The average length of selected answers for each question at this point was 291 words. Finally, to summarize the selected answers the "open text summarizer" was used [IS].The average length of summarized answers was S9 words which is very fewer than total number of answers before summarizing. It is concluded that the summarized answers demand significantly less time to be read by learners. As the proposed tool summarizes the best answers, it is expected that most information in the summarized abstract is valuable and can support the learners to find their desired answers in fewer amount of time. A sample of this summarization is shown in Fig. 4.

In the rest of the experiment, best answers with their summarization are given to five fluent English learners. After reading each summarization, learners were requested to answer some questions about the quality of the summarization. For each question learners were asked to give a score between 1 and 5. The questions of the survey are given in table 2. The results of this experiment are shown in Fig. 3.

As it can be seen in Fig. 3 the average score that learners assigned for question 2 is remarkably high. This is mainly because the summarizer selects full sentences from the best answers. Also the average score of question 3 is acceptable too. This happens because the summarizer tries to select sentences which contain the most meaningful words. The score for question 3 is not good enough because the summarizer selects the sentences from different parts of some best answers and consequently the coherence of summarized answer is not much acceptable. The question 1 requests the general view of learners about the summarization. As it is

1 http://ask.metafilter.com

33

� )'yl" ,/ " '..i /M ,o-I,

Shiraz University

shown in Fig. 3 this score is 3.S1 in average that is fairly acceptable.

VI. CONCLUSION AND FUTURE WORK

Online communities and forums are one of the best resources that users can refer to find answers for their questions. These communities are also important in E-Learning applications. Many learners engage in collaborative activities

TABLET. THE STA TlsnCAL INFORMA nON OF GATHERED DATA

N umber of Discussions 30

The average number of answers 13

The number of selected answers 3

The average length of all answer of each question 803

The average length of all selected answers for each question 291

The average length of summarized answer 89

TABLEl!. THE SURVEY QUESTIONS THAT ARE USERS WERE REQUESTED TO ANSWER

Question Question's Context

number

I How do you evaluate the quality of summarization?

2 How do you rate the linguistic structure of the summarization?

3 How do you rate the integrity of the summarization?

4 How do you rate the completeness of the summarization?

5 4.73

4

3

2

o QI Q2 Q3 Q4

Figure 3. The average score given to each question by users.

To share knowledge and find information. In this paper a new supporting tool for e-Learning in forums is proposed to help learners save a significant amount of time to fmd their desired information. The learner's query is initially expanded and then relevant questions to the learner's query are extracted. Then for each extracted question the most important answers are selected based on the expertise of the author. Finally these answers are summarized using a topic based multi document summarization method. To do so all sentences in the answers are scored and then sentences with high scores are summarized and the result is combined. To evaluate the proposed tool the contents of 30 discussions in the metafilter


Iraq E Lea ... nlng Anoelallon

online forum were collected. Then for each questions 3 best answers are selected. At last these best answers are summarized. 5 English fluent learners were asked to review these summaries to evaluate the quality of them. The result of this evaluation shows that, this system is able to provide support for learners in online forwns; by enabling easier and faster access to their desired information and guiding them directly to the most relevant questions along with the most important answers.

AKNOWLEDGMENT

The authors of paper would like to thanks the members of AEL T Lab for their helps and comments.

[I]

[2]

[3]

[4]

[5]

[6]

REFERENCES

M. 1. W. Thomas, "Leaming within incoherent structures: the space of online discussion forums," Journal of Computer Assisted Learning, vol. 18, no. 3, pp. 351-366, Dec. 2002. P. C. Abrami, R. M. Bernard, E. M. Bures, E. Borokhovski, and R. M Tamim "Interaction in distance education and online learning: using evidence and theory to improve practice," Journal of Computing in Higher Education, vol. 23, no. 2-3, pp. 82-103, Mar. 2011. M. Oztok, D. Zingaro, C. Brett, and J. Hewitt, "Exploring asynchronous and synchronous tool use in online courses," Computers & Education, vol. 60, no. I, pp. 87-94, Jan. 2013. C. S. L. Ng, W. S. Cheung, and K. F. Hew, "Interaction in asynchronous discussion forums: peer facilitation techniques," Journal of Computer Assisted Learning, vol. 28, no. 3, pp. 280-294, Jun. 2012. S. Jain, Y. Chen, and D. Parkes, "Designing incentives for online question and answer forums," Proceedings of the 10th ACM conference on Electronic Commerce, 2009. J. Bian, Y. Liu, D. Zhou, E. Agichtein, and H. Zha, "Learning to recognize reliable users and content in social media with coupled mutual reinforcement," Proceedings of the 18th international conference on Worldwide web, pp. 51-60,2009.

[7]

[8]

[9]

[10]

[II]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

34

� '/",/ " '..i �O-I'

Shiraz University G. Carenini, 1. C. K. Cheung, and A. Pauls, "MULTI-DOCUMENT SUMMARIZATION OF EVALUATIVE TEXT," Computational Intelligence, Apr. 2012. R. Mihalcea and P. Tarau, "A language independent algorithm for single and multiple document summarization," Proceedings of IJCNLP, 2005. A. Farzindar, F. Rozon, and G. Lapalme, "CATS a topic-oriented multi-document summarization system," DUC2005 Workshop, 2005. A. Nenkova, K. McKeown, C. C. Aggarwal, and C. Zhai, "A Survey of Text Summarization Techniques," in Mining Text Data, C. C. Aggarwal and C. Zhai, Eds. Boston, MA: Springer US, 2012, pp. 43-76. T. Schellens and M. Valcke, "Collaborative learning in asynchronous discussion groups: What about the impact on cognitive processing?," Computers in Human Behavior, vol. 21, no. 6, pp. 957-975, Nov. 2005. R. M. Aliguliyev, "A new sentence similarity measure and sentence based extractive technique for automatic text summarization," Expert Systems with Applications, vol. 36, no. 4, pp. 7764-7772, May 2009. D. R. Radev, H. Jing, M. Stys, and D. Tam, "Centroid-based summarization of multiple documents," information Processing & Management, vol. 40, no. 6, pp. 919-938, Nov. 2004. L. Hennig and D. Labor, "Topic-based multi-document summarization with probabilistic latent semantic analysis," Recent Advances in Natural ... , pp. Il4-149, 2009. M.-C. Lee, K. H. Tsai, and T. I. Wang, "A practical ontology query expansion algorithm for semantic-aware learning objects retrieval," Computers & Education, vol. 50, no. 4, pp. 1240-1257, May 2008. 1. Bian, Y. Liu, D. Zhou, E. Agichtein, and H. Zha, "Learning to recognize reliable users and content in social media with coupled mutual reinforcement," 2009. A. A. Kardan, A. Omidvar, and M. Behzadi, "Context based Expert Finding in Online Communities using Social Network Analysis," international Journal of Computer Science Research and

Application(JJCSRA), vol. 2, pp. 79-88,2012. Rotem, N. "Open text summarizer (ots)." Retrieved July 3.2006 (2003): 2006.

The 4th International Conference on e-Learning and e-Teaching, ICELET 2013 �

'/ " ,/" '..i �O-I'

All answers

Best Answers

Iran E Learning AlISQdalion Shiraz University

• Q: e-Iearning for math- What e-Iearning systems provide students with the math-related symbols they need and lets them "show their work," while still letting them submit all answers online?

• AI: The University of Texas has a browser plug-in called the Mathpad that allows students to type in equations and make graphs. They can submit the material to the graders and the graders can scribble on it and return it. You can contact me if you want to see it. I designed and programmed it.

• A2: I'm taking a calculus class online right now through San Francisco State University. I wouldn't recommend it to anyone but my problems with it aren't stemming from the submission of homework. The other students and I have to show our work and most either scan their work or take a digital picture of the pages and crop them to allow for viewing. Obviously the quality of the photograph has to be pretty good to allow for cropping out nearly 75% of the image without becoming pixelated. This has a lot to do with minimum focusing distance for the camera and not just megabites. Anyway, I digress.

• A22 : Of course, it depends on what level courses you are talking about. For math majors, especially when they are learning to write proofs, I have no problem with asking them to learn LaTeX and submit typed .pdfs of their work (to me, in paper, but there would be no reason they couldn't e-mail it). These could be annotated using some .pdf annotation procedure, when comments needed to be made. But this would be inappropriate for College Algebra or Calculus courses.

• LaTeX is beautiful, I started using it my freshman year in college, and never went back to paper (although, granted, my handwriting is so bad even I can't read it). Mastering Physics has an interface you could look at, if you're interested in developing something like this. Their site is craptacular - slow and unreliable and all that - and the interface could be improved, but at least it's easy to use.

• Of course, it depends on what level courses you are talking about. For math majors, especially when they are learning to write proofs, I

have no problem with asking them to learn LaTeX and submit typed .pdfs of their work (to me, in paper, but there would be no reason they couldn't e-mail it). These could be annotated using some .pdf annotation procedure, when comments needed to be made. But this would be inappropriate for College Algebra or Calculus courses.

• The University of Texas has a browser plug-in called the Mathpad that allows students to type in equations and make graphs. They can submit the material to the graders and the graders can scribble on it and return it. You can contact me if you want to see it. I designed and programmed it.

• LaTeX is beautiful, I started using it my freshman year in college, and never went back to paper. Mastering Physics has an interface you could look at, if you're interested in developing something like this. The University of Texas has a browser plug-in called the Mathpad that allows students to type in equations and make graphs. Mastering Physics has an interface you could look at, if you're interested in developing something like this.

Figure 4. Summarization for a sample question in metafilter online forum, there are 22 answers to the question from which 3 best answers were selected and finally a summary of these 3 answers is returned as the result of user's query.

35