Project 1Pre-Final Presentation

Quiz Preparation in an online management system

Kenneth CompassSanjay Jauhar

Camille MercadoKristen Wakamoto

Methods SummaryPhase 1• Contact and set up either a face to face, telephone or Skype

meetings with past instructors who have used Moodle v1.97• Assess Moodle 1.97 using HCI Guidelines

Primaryo  Gain insight of the quiz preparation system through a

walkthrough with the users Think aloud procedure Done on Moodle 1.97 Questionnaires are also given out; Unstructured

Interviews Secondary

o  For users that were not able to have scheduled meetings with us Sent out questionnaires/surveys about their usage;

questions based on interviews with primary interviewees

Methods SummaryPhase 2• Formulate structured interviews• Re-administer structured interviews to new respondents 

o Based on Moodle 2.0

Primaryo Look into differences between 1.97 and 2.0o Ask test subjects to try out 2.0 and ask them for

feedbacko Structured Interviews and re-evaluation

Secondaryo  Look into differences between 1.97 and 2.0o Ask secondary users about differences found through

questionnaires and see what they think about it• Analyze data and make recommendations

Methods Summary

Phase 3 • Contacted users who did not respond to the first

attempt• Determined their availability.

o Sent out surveys to those who were under time constraints

o For others, we set up either face to face, Skype, or phone interviews

o Think out loud walk through's of Moodle 1.97• Ask users to look at Moodle 2.0 and then send

questionnaires regarding their thoughts

Preliminary Results

Initial Interviews (done with Moodle 1.97): Phone, Skype, and  In-Person Think Out Loud Walkthroughs

•     Major usability issues identified and confirmed with   pre-assessment of system by team.

•     Turn editing on/off  •     Many options - confusing and daunting•     Adding questions does not go directly to quiz

o Clunky interactions to correctly add a question to a quiz

Preliminary Results

Demographic Interviewed:• Got in touch with 15 instructors• Most were in their 40s and 50s• Taught mostly Management, Engineering, or

Computer Science classes at the University Extension

• Have used the Moodle system for at least 1.5 years• Create their own quizzes• Have not really had to use any other learning

management systems aside from Moodle

Preliminary ResultsIn Person and Skype Interviews (1.97)• Screen recording with walkthroughs of the Moodle

System• Found the initial set up of the quiz system to be easy

o e.g. Creating the title, description, etc.o Did have issues concerning the help buttonso Still had concerns about arbitrary processes

•  Main roadblock came with the question bank featureo "Why is it that you have to set it up with the

question bank?"o How do you edit questions? In the question bank

or the quiz itself?• Interface is found to be too "crunched up"

Preliminary ResultsPhone Interviews (1.97):• One concern brought up was how to prevent cheating• Issues with the question bank feature as well• Keeping the students updated (if he/she misses a

quiz or if the instructor modifies the quiz)Surveys (1.97)• In general, instructors say that the interface is good

for conducting online classes• Most brought up the point of functionality vs.

usability• "Yes there was a steep learning curve, but

eventually, each time I use it I discover more"• Compare it to Blackboard (has a better quiz/exam

feature)

Compiled ResultsCommon Usability Issues:• Turn editing on - Why? If needed - Move to a more

noticeable area• Drop down Menu's contain too much information

o Need more commonly used default values• Question bank Interface - Completely confusing

o Needs directions or more intuitive flow• All around - needs more on screen instructions, not

just help actions

Some users liked the system as is, but still felt it was hard to learn

Compiled ResultsMoodle 2.0 Addresses most usability issues in 1.97

• Turn editing on - Moved to more "eye popping" position

• Drop down menu content re-vamped (slightly better)

• Adding questions to quizzes improved

• Quiz question bank improvedo Can add questions directly to quizzes and use in a

bank

Adding questions in Moodle 1.97

Adding a question in Moodle 2.0

Moodle 2.0 Question bank

Compiled Results• Many options - needed for high user customization -

but they appear to complicate the system• Still compiling results• Almost all tests indicate an upgrade is needed• Demo runs of Moodle 2.0 were positive and

enthusiastic

Preliminary Recommendations:

• Leaning towards recommending quiz system be upgraded to newer version of Moodle

• Do not feel like customization is needed 

Updated Schedule1. Week Five (04/24 - 04/30)

o Meet up with users; run usability tests (all members)o First Phase of the usability tests with the primary group

of users (all members)o Create relevant features "list" Moodle 2.0 that might

address usability issues. (Kenny)  – Week Six (05/01 - 05/07)

o Analyze data received from usability tests (Camille and Kristen)

o Proceed with second phase of testing (all members)o Publish web survey and notify all participants to

participate(Sanjay).– Week Seven (05/08 - 05/14)

o Analyze data received from usability tests (Sanjay and Kenny)

o Re-evaluate system(all members).

Updated Schedule

1.Week Eight (05/15 - 05/21)o Contact a user set for suggestions and opinions on

redesign (Sanjay, Camille)o Conduct Final remaining interviews (Sanjay,

Kenny, Camille)2. Weeks Nine and Ten (05/22 - 06/04)

o Conduct Final remaining interviews (Sanjay, Camille)

o Refine and render collected data (Sanjay, Kristen)o Produce Final Report (Team)o Final Presentation (Sanjay, Camille, Kenny)

Thank you

Questions ?