Kypros-net Web Site:Learn Greek! On-line Course

SI 622

Evaluation of Systems and Services

M. Elledge

P. Zaphiris

4/15/00

The Project

• Evaluate Learn Greek! language course– Usability– Functionality– Quality

• Client: Kypros-net Originator/Administrator

The Product

• Kypros-net description– Formation– Purpose

Kypros-net Web Site

Learn Greek! Course

• Course Description– Independent study course to learn Greek

• 105 audio lessons, transcriptions, vocabulary, spell checker, dictionary

– Objectives• Assist people in learning Greek language

• Develop site using User Participation philosophy

• Minimize expense (tapes from Cyprus Broadcasting Corporation (CyBC) radio; volunteers; MIT server)

Learn Greek! Home Page

Methods Used

• Interviews with site creator, content contributors (2), director and administrator of U-M

• Forty-three user surveys, 371 discussion board postings

• Literature search of PsychInfo and Wilson databases

• Comparative Evaluation with 16 web language sites

• User testingUser testing of 8 tasks with 5 subjects

• GTN for first two levels

• Metaphor, Vocabulary analyses

• Checklist/Cognitive walk-through on course

• GOMS on 2 tasks

Key Client Questions

• How useful is the course?

• How functional is the course?

• How well does it meet on-line course standards?

• What can be done to improve the course?

Question #1

• How useful is the course?– Does it meet client objectives?– Does it meet user expectations?

• Findings– Client objectives exceeded– User expectations met– Still, there are complaints

Findings

• Client Objectives– Participatory Design: 105 lessons transcribed

by 8 users; dictionary, spell checker installed by users; on-going help provided by users

– Steady increase in site visits over timeGreek Online Access Statistics

0

1000

2000

3000

4000

5000

6000

Week Ending

Findings

• User Expectations– High user satisfaction

1.51.5 1.5

1.4

1.11.1

0.9 0.9

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Overall Ease ofuse

Overallsatisfaction

Simple to use Easy to Learn Info Effectiveto Complete

Tasks

CompleteWork

Efficiently

Easy to FindInfo

InformationClear

Findings

– Positive Comments:

• EASY TO UNDERSTAND

• Fun

• It helped refresh my Greek

• Finally after surfing for a long time I found that you could learn Greek from the internet

• It is a very progressive course, builds up confidence as it goes

• Very good method of learning

• It falls into place after listening a couple of times.

• Greek is the most kick-butt language on earth!!!!

Findings

• Some User Complaints– Can’t download lessons– No pronunciation guide– Can’t compare text with lessons– Few instructions or information about course

content

Discussion Board PostingsCategory/Subject # % Total %TechnicalProblems with needing/installing keyboard fonts 56 15.1%Problems with/wanting to download lessons 49 13.2%Problems with installing/using RealPlayer 40 10.8% 39.1%

ContentPostings of on-line notes 73 19.7%Questions about availability of text, CDs or

including links to web sites 37 10.0% 29.7%

ComprehensionQuestions about vocabulary 28 7.5%Questions about grammar/spelling 17 4.6%Questions about dialect 2 5.4% 17.5%

Miscellaneous 69 18.6% 18.6%

Question #2

• How functional is the course?– Do the operations work?– Is the architecture successful?

• Findings– Site is subject to Internet vagaries– Learn Greek! architecture can be improved

Findings

• Operations– ON the site generally work

• Translations, lessons, etc. function

• Waiting can become tedious

– Web operations less dependable• Server downtime

• Slow lesson downloads

• Intermittent transmission

Findings

• Architecture– Layout

• Violates requirement for visual integrity

• Inconsistent formatting

– Labeling• Confusing (Greek to English in Greek?)

• Incorrect (Vocabulary/notes = transcripts?)

• Out-of-order (Instructions after the task?)

Findings

• Architecture (cont.)– Content

• Few instructions (RealPlayer, Greek fonts)

• No “Help” or FAQ

• Multiple translations

– Navigation• Hard to find things (transcripts)

• Search engine can’t target course

Question #3

• Does it meet language course standards?– Is the content appropriate?– Is the methodology appropriate?

• Findings– Fairly unusual: hybrid of independent study +

web-based learning– Pedagogy okay, course content can be

enhanced

Findings

• Lacks some features

Findings

• Meets basic teaching methodologies

Conclusions

• Yin/Yang: Strengths are also weaknesses– Accessibility vs. dependability– Participation vs. consistency– Frugality vs. thoroughness

Recommendations

• Technical

• Architecture

• Content

Questions?

Other Findings

• Findings

• Recommendations

Conclusions

Next Steps

Learn Greek! Site Visits

Greek Online Access Statistics

0

1000

2000

3000

4000

5000

6000

Week Ending