WP5: Validation of developed functionality Report on the final Experiments Anne de Roeck Diane Evans James Gray The Open University - UK

WP5: Validation of developed functionality

Report on the final Experiments

Anne de Roeck

Diane Evans

James Gray

The Open University - UK

OutlineIntroductionBusiness hypothesesStudent Scenario (QUIZ)Results Desktop ActivitiyResultsTutor ScenarioResultsFinal Conclusions

Overall HypothesisThe augmentation of e-learning systems (in this

case ILIAS) with NLP and semantic web technologies

increases the effectiveness of learning and teaching,

and in particular,

Increases the effectiveness of teachers and learners in locating relevant learning objects in the context of learning related tasks.’

IntroductionChanges were made to the methodology after the

previous tests and review.

Measuring learning

Performing Quantitative tests across the scenario activities

Defined Hypotheses and formulated tests to measureIntroduced Mitigation controlsAlso included value judgements as supporting analysis

Restriction to 2 scenarios (1 student & 1 tutor)Scenarios built on experiences of previous trials

Data captured electronically

Student Hypotheses

Retrieval of Content (Student)1.1 The addition of functionalities based on

NLP increases the retrievability of learning objects in terms of the relevance of the content. [Quant test]

1.2 The addition of functionalities based on NLP increases the effectiveness of learners in locating relevant learning objects in the context of answering Quiz questions. [Quant test]

Support for Learning

2.1 The developed tools and ontology help to support learners to better, more effectively grasp the terminological and conceptual space which defines a certain domain of knowledge. [Quant test]

Multi-lingual study

3.1 The ability to easily retrieve content in more than one language supports learning activities of students who are studying in a Multi-lingual situation e.g. the language of study is not their native language.

[Quant test]

Support for Learning Paths and Personalised Learning

4.1 Learners are able to build individual learning paths by entering key terms of concepts they need to learn. [Qual. test]

4.2 Learners are able to classify and order learning material (in the form of documents) they have placed within their personal desktop by using concepts and keywords linked to each document

[Qual. test]

Cont..

4.3 The facility for a learner to classify and order learning material supports self-guided learning.

[Qual. test]

4.4 The facility for a learner to classify and order learning material supports self-guided learning by enabling the user to create a meaningful / linked path through the content. [not tested]

Basis of Student scenarioThe student scenario focussed on:

Demonstrating improvement in learning (as requested by the Reviewers)

The LT4eL search functions i.e. Semantic, Definition, Keyword and Ontology Browsing.

Multi-lingual search and retrieval associated with these search functions

The Use of the Personal Desktop to support individual learning

The scenario centred around a multiple choice QUIZ

Structure and rationale of the student scenario

• Target - 1st year undergraduate humanities students• Topics - Information exchange: protocols and markup languages

You will shortly be studying a short course covering 'Information Exchange on the Internet'.

This is a subject on which you have little or no previous knowledge.

Your tutor has prepared some research questions for you in the form of a quiz to enable you to gain a basic grounding in key aspects.

Depending on your group you will either use internet facilities such as Google and Wikipaedia etc or you will have access to different ILIAS search facilities to assist your search, including the LT4el searches and the structured concept browser.

Experiment design - students

To assess whether technologies "increase the effectiveness of learning", students needed to do some learning:

Learning experience -

Quiz to encourage students to engage with learning objects

Learning experience -

Quiz to encourage students to engage with learning objects

Assessing the learning

Assessment Activity

Pre-Test

Assessment Activity

Pre-Test

Learning experience(quiz)


Assessment Activity

Post-Test

Assessment Activity

Post-Test

The answers provided by the student at each of the 3

stages were marked

Measuring the effect of our technology

We introduced a control group with a different learning experience:


ILLIAS / LT4eL


ILLIAS / LT4eL


Internet / Google


Internet / Google

TARGET CONTROL

To compare NLP technologies with plain search in the same repository

PreTestPreTest

Quiz Part 1 - ILIASQuiz Part 1 - ILIAS

Quiz Part 2 - LT4eLQuiz Part 2 - LT4eL

PostTestPostTest

PreTestPreTest

Internet/Google Internet/Google

PostTestPostTest

TARGET CONTROL

Test ATest A

Question Set AILIAS

Question Set AILIAS

Controlling for question differences (eg is the post-test easier than the pre-test?):

Question Set BLT4eL

Question Set BLT4eL

Test BTest B

Test BTest B

Question Set B

ILIAS

Question Set B

ILIAS

Question Set A

LT4eL

Question Set A

LT4eL

Test ATest A

Quiz Part 1

Quiz Part 2

PreTest

Post-Test

Test ATest A

Question Set AWWW

Question Set AWWW

Question Set BWWW

Question Set BWWW

Test BTest B

Test BTest B

Question Set BWWW

Question Set BWWW

Question Set A

WWW

Question Set A

WWW

Test ATest A

Group 2 Group 3 Group 4Group 1

TARGET CONTROL

Pre & Post-TestsFive questions testing understanding of relationships between concepts

Pre-test and Post-test questions were paired, to test the same area

Pre-test and Post-test questions were alternated in different groups to ensure fairness

QuizSeven questions with learning materials in the same topics as the test questions designed to encourage engagement.Each question followed by supplementary questions to ascertain:

whether student already knew the answersource document (including language)which tool (if any) helped find the answer

Quiz is time-limitedQuestion groups alternate between student groups

Results of Student Experiments

Main Student Scenario

Searching

Learning

Multilinguality

Student Scenario:Searching

Hypothesis 1.1: The addition of functionalities using / based on NLP increases the retrievability of learning objects in terms of the relevance of their content.

Hypothesis 1.2: The addition of functionalities using / based on NLP increases the effectiveness of learners in locating relevant learning objects in the context of answering Quiz questions.

Student: Searching

Test that mean scores in Part 2 of the quiz (LT4eL) are greater than mean scores in Part 1 of quiz (plain LMS) for the target group.

Student: Searching

Searching - Test 1

Test 1

Quiz 1 Quiz 20

1

2

3

4

5

3.44

4.07

Improvement in quiz scores

Students scored higher in the quiz when they had access to LT4eL technologies to help them.

Student: Searching Test 1 Results

Was it the technology that helped?

Question differences?

no - rotating banks of questions

Order effect?

might expect students to be learning, thus achieve better results in second half of quiz.

Test 1a: test by looking for improvement in control group

Student: Searching Test 1a

Hypothesis 1.1 -

Gains achieved in both groups

Control group scored better in both quizzes

Target Control0

1

2

3

4

5

3.44

4.074.07

4.57

Improvement in quiz scores Quiz 1

Quiz 2

Sco

re

Student: Searching Test 1a Results

Searching - Conclusions so farStudents scored higher in the quiz when they had access to LT4eL technologies to help them.

Partly explained by order effect - students do better in second half of quiz anyway

Students using resources of World Wide Web score higher than those restricted to learning objects within an SMS

LMS score improvement appears slightly better than WWW score improvement

Student: Searching Conclusions 1

Searching - test 2Test that for any given question in Part B of the quiz, students who choose to use LT4eL functionality score more highly for that question than those who do not use LT4eL functionality.

Note: In Part 2 of the quiz, students are free to use whichever search method they feel will be most effective. This test is to assess whether those who use LT4eL methods score more highly than those who do not.

Student: Searching Test 2

Only three questions answered better using LT4eL functionality

Q1 Q2 Q3 Q4 Q5 Q6 Q70%

20%

40%

60%

80%

100%

Score per question by functionality

Full text

LT4eL

Question

Sco

re

Student: Searching Test 2 Results

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Full text search

Keyword search

Semantic search

Concept browser

Definition finder

Success in answering quiz questions by functionalityF

un

cti

on

ali

ty

Success

Student: Searching Test 2a Results

(target groups only)

Student: Searching Opinions 2

What did they dislike about Semantic Search?It didn't return relevant results.

because it doesn't find what I am searching for

its too vague

i didn't use it a lot as the results were chaotic

i find it not much to the point for the types of research i usually do

It is a bit too much to offer this much search methods

the name semantic is confusing

i liked this type best.. it was the easier to find the relevant information

Student: Searching Comments: Semantic

What did they dislike about the Concept Browser?

It didn't return relevant results

It was too slow for my part and did not give any additional value

it's a roundabout way of searching

was not eay to use it. maybe this was because i did not fully understand how the concept browser worked

I am not sure what the concept browser is

don't know what it is.

for content questions this might be a relevant search method. However, less relevant when studying a language

I like that method the most but it wouldn't be useful in my studies - English philology

it helps a lot to understand given topic/term

Student: Searching Comments: Concept Br

0% 10% 20% 30% 40% 50%

full text search

keyword search

semantic search

concept browser

definition finder

Most useful functionality - students' opinionsF

un

ctio

nal

ity

Percentage finding this the most useful

Student: Searching Opinions 1

Searching - Test 3 Test that for any given question in Part B of the quiz, students who choose to use LT4eL functionality and answer correctly, do so more quickly than those who answer the question correctly using Full Text Search

Notes:

In Part 2 of the quiz, students are free to use whichever search method they feel will be most effective. This test is to assess whether those who use LT4eL methods are faster than those who do not.

Only consider students who searched for an answer, and answered correctly

Student: Searching Test 3

Students answered faster using LT4eL technologies than using full text search.

Student: Searching Test 3 result

Q1 Q2 Q3 Q4 Q5 Q6 Q70

30

60

90

120

150

180

210Average time to answer questions

Full text

LT4eL

Question

Sec

on

ds

Test 3 resultStudent: Searching

Q1 Q2 Q3 Q4 Q5 Q6 Q70

30

60

90

120

150

180

210Average time to answer questions

Internet

Full text

LT4eL

Question

Sec

on

ds

Searching - conclusions

Using Semantic Search, Concept Browser and Definition Finder to find answers, students score better than using plain text search

Students appreciate all functionality, especially the Definition Finder;

Students' ratings of Semantic Search and Concept Browser, though favourable, showed that these techniques were less familiar to them.

Students answered faster using LT4eL technologies than using full text search; speed of searching seems to improve with practice.

ConclusionsStudent: Searching

Hypothesis 2.1: The developed tools and ontology help / support learners to better / more effectively grasp the terminological and conceptual space which shape a certain domain of knowledge.

Student: Learning Test 1

Test: Assess learning by testing that the mean score in the post-test is higher than the mean score in the pre-test

Assessment Activity

Pre-Test


Assessment Activity

Post-Test

Learning has taken place. This validates an important part of the methodology.

Pretest Post test0

1

2

3

4

5

2.53

3.58

Improvement in test scores

Student: Learning Test 1 - result

Learning - test 2Test that mean improvement in test scores for target sample (using ILIAS and LT4eL) is higher than mean improvement in test scores for control group (using Internet to find content).

Note: This test is intended to show whether learning is more effective in the target sample than in the control sample.


Learning appears slightly greater in Control group; but they started from slightly lower base. But not statistically significant

Target Group Control Group0

1

2

3

4

5Improvement in test scores

Pretest

Posttest

Sco

re


Learning - Test 3Test whether students who use the LT4eL functionality in the LMS learn more than those who do not.


Did not use LT4eL functional-ity

Used LT4eL func-tionality some-times (1-4 times)

Used LT4eL func-tionality frequent-ly (5-7 times)

0%

20%

40%

60%

80%

100%

Test improvement by Quiz functionality

No of students dropping marks

No of students keeping same marks

No of students gaining marks


Learning - Conclusions

Learning took place in target and control group

Students in the target group who used LT4eL functionality extensively, learned more than those who stuck to plain text search.

Student: Learning Conclusions

MultilingualityHypothesis: The ability to easily retrieve content in more than one language supports learning activities of students who are studying in a Multi-lingual situation.

Student: Multilinguality

Notes:• We considered test persons who are capable to read and

understand documents in other languages than their native. • In normal situation we expect also students who are studying in

other language that their native (exchange students) to make use of the multilingual facilities and find documents in their language

• For our tests this was however not replicable as we would not have been able to ensure a relevant sample size

For the quiz:

Allow students to select most appropriate documents to answer questions, irrespective of language

Students will retrieve greater linguistic diversity of documents

Students who find documents in other languages will score highly

FindingDocuments

Getting correct

answers

Student: Multilinguality Tests

Multilinguality - Test 1Searching

Test: Allow students in a quiz to select the most appropriate document to answer the questions, irrespective of language.

We expect those using NLP-based functionalities to select a higher proportion of documents in a non-course language

Test 1Student: Multilinguality

Target Quiz 1

Target Quiz 2

Control Quiz 1

Control Quiz 2

0%

20%

40%

60%

80%

100%

Language of document retrieval

Course language

Non-course language

Student: Multilinguality Test 1 result

Conclusions so farTarget group retrieved more non-course language documents when they had access to LT4eL functionality, showing that multilingual access supported the learning activity

No significant increase in non-course language retrieval for control group, suggesting this is not an order effect


Test: test that, of students who are studying in a non-native language, those who retrieve second-language documents in Quiz Part 2 (either target or control) score more highly for those questions than those who do not.

Note: This is intended to show that multilingual retrieval is helpful in the context of answering quiz questions

Student: Multilinguality Test 2

Multilinguality - Test 2Getting the right answer

Document in main course language

Document in other language

0%

20%

40%

60%

80%

100%

74% 71%

Percentage of correct answers


Those retrieving documents in a non-course language score statistically the same for those questions as those who retrieved documents in their main course language.

These results depend on the quantity and quality of information in different languages available within the repository being searched (or across the Internet for control group students).

It is possible that students retrieving in a second-language document are less fluent in that language, hence have more difficulty assimilating the information within the document. In 328 cases (48%) the language of retrieval was English. It is also possible that students only searched in a second language when they had failed to find the answer in their course language.


Multilinguality - test 3Does our functionality aid multilingual retrieval?

Student: Multilinguality Test 3

Full text Keyword Semantic Concept browser

Definition finder

0

20

40

60

80

100

Language of documents by functionalityDocuments in other language

Documents in course language

Search method

Nu

mb

er o

f d

ocu

men

ts


Student ScenarioQualitative evaluation of multilingual Search

Multilingual search would be useful for my studies?

71% (157 Students) STRONGLY AGREED

12% (25 Students) DISAGREED

Lower results for students in countries where large collections of materials in their native language are available (EN, DE) or in countries were usually materials (at least for the particular domain of the test) are in English (NL)

Student: Multilinguality Opinions

Multilinguality conclusions

Students retrieved more non-course language documents when they had access to LT4eL functionality, but this did not result in higher quiz scores.

LT4eL functionality helped retrieval of multilingual content - particularly Concept Browser, Definition Finder and Semantic Search

Student: Multilinguality Conclusions

Overall conclusions - StudentsUsing Semantic Search, Concept Browser and Definition Finder to find answers, students score better than using plain text search

Students most appreciate Definition Finder; they found it harder to see the benefit to them of Semantic Search and Concept Browser

Students answered faster using LT4eL technologies than using full text search

Learning took place in target and control group

Students in the target group who used LT4eL functionality extensively, learned more than those who stuck to plain text search.

Students retrieved more non-course language documents when they had access to LT4eL functionality, but this did not result in higher quiz scores.

LT4eL functionality helped retrieval of multilingual content - particularly Concept Browser, Definition Finder and Semantic Search

Students Overall conclusions

Personal Desktop experiments

Create classifications and organise their content within these sections

Use LT4eL functionality to locate other resources in own and course languages.

Organise these into structure

Student tasks required them to:

Evaluate the usefulness of these features for their study

Students were able to classify documents into topic sections within the personal desktop.

Using the concepts and keywords linked to each document helped them do the classification

Students were able to order documents within the Sections

RESULTS

Student ScenarioMultilingual support with the Personal Desktop

Finding, classifying and ordering of multilingual documents within the personal desktop is possible using the NLP based functionality

?

62% (10% strongly) AGREED that they were able to classify and order documents in more than one language based on key concepts

75% (24% strongly) AGREED that they found documents in more than one language using concept and topics

Student: Multilinguality Opinions

Students agreed that:Features would help students organise their study

Multiple course modulesIndividual directed study

RevisionResearchHaving classified and ordered documents,

they have a clearer understanding of the relationships between the topics

Tutor Hypotheses

Support for Teaching in a multi-lingual context - finding resources

The addition of functionalities based on NLP increases the effectiveness of tutors in identifying supplementary resources to support students who are studying in a multi-lingual context. [Quant test]

The addition of functionalities based on NLP increases the effectiveness of tutors in identifying documents that are relevant, reliable, and appropriate for level..

[Quant test]

Content Preparation

Using the Keyword Extractor based on NLP increases the quality and consistency of keywords assigned to repository content and helps tutors who are expected to perform this task. [Quant test]

Using the Keyword Extractor based on NLP enables tutors to produce a set of keywords faster than those tutors who perform the task unsupported

[Quant test]

Basis of Tutor Scenario

The tutor scenario focussed on:Multi-lingual search the LT4eL functions of Semantic search and ontology browsingThe Keyword Generator for new content

Both in English and in native language

Tutor scenario design

The tutor scenario was based on the need to

find content to support students’ learning

and to add additional content to the repository

Tutor tasks

Internet Resources Internet Resources

LMS resourcesLMS resources

Locate Content

Group1 - ILIASGroup1 - ILIAS Group2 - LT4eLGroup2 - LT4eL

Search Topics

Students will learn which markup languages are appropriate for holding data.

Data formats on the Web

Students will be able to identify ways to construct links to internal and external resources.

links in markup languages

Students will be able to identify which protocols are required to transfer files over the internet

file transfer and internet protocols

DescriptionTopic

Session 1 - Set A topics

Students will learn some of the different picture formats & how to select an appropriate one for different purposes.

picture formats on the Web

Students will be able to identify how to present data in tabular form on web pages

tables in markup language

Students will be able to identify which protocols are used for sending & receiving emails

email and internet protocols

DescriptionTopic

Session 2 -Set B topics

Add New ContentAdd New Content

Group1 - ILIASGroup1 - ILIAS Group2 – LT4eLGroup2 – LT4eL

Tutor Scenario - Results

Searching

Multilinguality

Keyword addition

Hypothesis: The addition of functionalities based on NLP increases the effectiveness of tutors in identifying documents that are relevant, reliable, and appropriate for level.

Test within the same repository

Use tutors' own assessments of relevance, reliability and appropriateness for level.

Tutor: Searching Test

Rele-vance Native

Rele-vance English

Reliabili-ty Native

Reliabili-ty Eng-lish

Level Na-tive

Level English

0

1

2

3

4

5

Quality of documents

ILIAS score

LT4eL score

Attribute

Sco

re

Tutor: Searching Results

Searching - conclusion

LT4eL functionality makes no difference to the tutors' assessments of the Relevance, Reliability or Level of the documents found when searching within the repository.

Tutor: Searching Conclusion

Hypothesis: The addition of functionalities based on NLP increases the effectiveness of tutors in identifying supplementary resources to support students who are studying in a multi-lingual context.

i.e. tutors can find content in two languages

Tutor: Multilingual

In more detail...

Tutors are better able to find second language documents using LT4eL than using Internet search / plain ILIAS search in the same repository.

Tutors find location of second language documents using LT4eL is relatively faster than using Internet searches / standard ILIAS.

Tutors find location of second language documents using LT4eL is relatively easier than using Internet searches / standard ILIAS in the same repository.

Tutor: Multilingual Tests

Tutors are better able to find second language documents using LT4eL

Internet searches

ILIAS searches

LT4eL searches

0%

20%

40%

60%

80%

100%

Success of multilingual searching

Two Documents Not Found

Two Documents Found

Tutor: Multilingual Test 1

Tutors find location of second language documents using LT4eL is relatively faster than using Internet searches / standard ILIAS


Internet Ilias LT4eL0%

20%

40%

60%

Fastest document to find

First Language

Both Same

English Language

Tutors find location of second language documents using LT4eL is relatively easier than using Internet searches / standard ILIAS in the same repository.


Internet ILIAS LT4eL0%

20%

40%

60%

Easiest document to find

1st language docs easier

Both docs the same

English docs easier

Multilingual searches are more successful on the Internet than within repository.

Within the repository, LT4eL functionality does not improve ability to find multilingual content

Within the repository, when two documents are found, tutors using are more likely to find both documents together (same speed, same ease).

Tutor: Multilingual Conclusions

KeywordsUsing the Keyword Extractor based on NLP increases the quality and consistency of keywords assigned to repository content and helps tutors who are expected to perform this task

Test: Tutors using the keyword generator produce a more consistent set of keywords than those who do not

Tutor: Keywords Test 1

Number of distinct keywords Number of words unique to one tutor

0

20

40

60

80

100

Consistency of keywords

Ilias

LT4eL

Tutor: Keywords Test 1 results

Keywords - test 2Using the Keyword Extractor based on NLP enables tutors to produce a set of keywords faster than those tutors who perform the task unsupported.

Tutors assessed their own time to complete the exercise:


ILIAS LT4eL 0

1

2

3

2.04

1.41

Time taken to assign keywordsA

vera

ge

co

de

d t

ime


Keywords - test 3Using keyword extractor gives:

more consistent set of keywords

faster results

But are tutors using the extractor just assigning fewer keywords?


ILIAS LT4eL0

1

2

3

4

5

6

7

8

6.94 6.97

Average words per tutor per document


English Own language0%

20%

40%

60%

80%

100%

Did you think the keyword generator helped you to assign keywords to the document?

Helped a lot

Helped a little

No help

Tutor: Keywords Opinions

base = 23 base = 27

Keywords - Conclusions

Tutor: Keywords

Using the keyword extractor yields a much more consistent set of keywords than manual

Using the keyword extractor, tutors assign keywords 30% faster than not using it.

About 90% of tutors said the keyword generator helped them

Conclusions

Overall conclusions - Tutors

Within the repository, LT4eL appears to make finding multilingual documents together slightly more likely (same speed, same ease).

LT4eL functionality makes no difference to the Relevance, Reliability or Level of the documents found

Using the keyword extractor yields a much more consistent set of keywords than manual

Using the keyword extractor, tutors assign keywords 30% faster than not using it.

Tutor Overall Conclusions

Evaluation - Conclusions

WP5 Participation in Experiments

Participants in Student Scenario

22637387675Total per group

36661311Romania

244389Portugal

31551011Poland

275499Netherlands

41771413Malta

244488Germany

244488Czechoslovakia

19 (23) 2566 (*10)Bulgaria

TotalGroup 4Group3Group 2Group1

8 languages and 226 students

Participants in Student Personal Desktop Activity

12092016182713116

TotalRomaniaPortugalPolandNetherlandsMaltaGermanCzechBulgaria

8 Languages and 120 students

Participants in Tutor Scenario

71Total

11Netherlands

12Romania11German

10Portugal10Czech

11Poland6Bulgaria

7 Languages and 71 tutors

Not a trivial task!8 partners

Pre-test, post-test, Quiz-A, Quiz-B, Desktop questionnaire, Tutor questionnaires - all translated into host languages (and verbatim answers translated back to English)

152 data files to download, clean, and load into database

Analysis database contains: 18 tables, 585 different fields, 2483 records. The total number of non-blank data items is 66,725.

Over 6,500 lines of code to manipulate, condition, load and analyse that data.

Overall HypothesisThe augmentation of e-learning systems (in this case

ILIAS) with NLP and semantic web technologiesincreases the effectiveness of learning and teaching,

and in particular, Increases the effectiveness of teachers and learners in locating relevant learning objects in the context of learning related tasks.’