Ar3 X0 Final Research Paper G Compris Usability Research Report

GCompris Usability Testing 1

GCompris Usability Testing Research Report

Shakarah McCrae - 06004107

AR3X0 – Research Paper

May 15, 2009

Supervisor: David Bain

Course Moderator: Dr. Ramsay


Table of Contents

Executive Summary………………………………………………………………… ...2

Introduction…………………………………………………………………………… 3

Research Question…………………………………………………………......3

Null Hypothesis………………………………………………………………..3

Background…………………………………………………………………….3

Worth of Study………………………………………………………………....6

Objectives………………………………………………………………………6

Literature Review………………………………………………………………………8

Methodology…………………………………………………………………………...11

Procedure………………………………………………………………………12

Participant Summary…………………………………………………………..13

Gcompris Set-up……………………………………………………………….14

Data Presentation………………………………………………………………………16

Analysis………………………………………………………………………………..17

Conclusions and Recommendations ………………………………………………….23

Limitations……………………………………………………………………

References……………………………………………………………………………..25

Appendices………………………………………………………………………….....27


Executive Summary

This study is in line with the requirements of the University of the West Indies where a research

project has to be done. This topic was chosen in accordance with my field of study and to

discover how influential or effective this medium is in the classroom.

Gcompris was chosen because it is free and can be used as a tool for teaching basic math to

students. The research is aimed at showing how user friendly the software is and how the

students will react to using it.

Usability testing is used to measure the appropriateness of software for its user. It is aimed at

assessing the effectiveness, efficiency and satisfaction with which specified users can achieve

specified goals in particular environments or contexts of use.

The chosen open source software for this purpose is GCompris, which is an educational software

suite comprising of numerous activities for children, aged 2 to 10. Some of the activities are

game orientated, but nonetheless educational. More specifically, I will be looking at the extent to

which G-Compris, as a tool for teaching basic mathematics skills can improve or aid learning in

mathematics for grade one students at August Town Primary School.


Introduction

Research Question:

How effective is Gcompris- an open source software- for teaching basic mathematics skills to

grade one students at August Town Primary school?

Null Hypotheses:

Gcompris as an educational tool is not beneficial in improving the performance of grade one (1)

students in school.

Background

The disappointing performance of Jamaican students in the mathematics subject area

continues to be an increasing concern to the wider Jamaican society. According to Francis

(2006):

“New statistics from the CXC show that of the 35,428 students who sat the 2006

mathematics exam, only 32 per cent managed to achieve grade one to three. This

represents a decrease over last year's 36 per cent. This figure is also three per cent below

the regional average” (para. 2).

In fact, Wesley Barrett Pro-Registrar for the Caribbean Examination Council states "These

results are disappointing and it is still a very worrying problem," ” (Francis, 2006, para. 2). Poor

mathematics performance is not only at the Caribbean Secondary Education Level but also at the

Primary Level.


With this in mind and out of the need to stem the problem from a tender age, I have

decided to carry out a usability study of an open source software for teaching basic mathematics

skills to grade one students in Jamaican primary schools. Usability testing “is a technique for

ensuring that the intended users of a system can carry out the intended tasks efficiently,

effectively and satisfactorily.” (Gaffney, 1999)

The chosen open source software for this purpose is GCompris, which is an educational

software suite comprising of numerous activities for children, aged 2 to 10. Some of the

activities are game orientated, but nonetheless educational. (http://gcompris.net/-en- )More

specifically, I will be looking at the extent to which G-Compris, as a tool for teaching basic

mathematics skills can improve or aid learning in mathematics for grade one students at August

Town Primary school.

Worth of Study

The worth of this research is multi-facet in nature. If we can identify software that can aid

and improve the performance of children from a tender age, we will not only be facilitating

improvement in the grades but also a greater appreciation and a more fun way of learning the

subject. It is of academic value because it provides information to educators and their

stakeholders as to the value of educational software to students and the implications it can have.

In addition, it provides valuable information to be taken into consideration by developers of

educational software for students.

http://gcompris.net/-en-


Objectives of the study include:

1. To determine whether or not Gcompris is effective and appropriate for children six (6) and seven (7) years of age

2. To use the findings of this research to make recommendations to Educators as to one way of improving students performance in school

3. To prove whether or not Gcompris as an educational tool is beneficial to children

4. To evaluate students learning potential based on the activities available on GCompris

5. To document findings that are as a result of the usability research done.

Dominant terms that will be used throughout this research are Usability, usability testing,

educational, multimedia, multimedia cognitive theory, activity theory and children. There is an

Independent Variable and a Dependent Variable. The independent variable is GCompris and the

dependent variable is the August Town Primary Grade one (1) students. The main theories that

will be guiding this research are the Multimedia Cognitive theory, Cognitive Load theory, The

Multimedia Principle and the activity theory.

For this testing, eighteen grade one primary student with little or no computer experience will

be chosen randomly. The eighteen students will be further divided into two groups of nine

and exposed to the software for 20 minutes on two separate occasion. The students will be

observed, recorded, and a post-test evaluation will be done.

This research report provides background, findings and recommendations of the

Gcompris Usability testing. It documents and discusses theories and methodologies of the testing

between March 17 and April 21, 2009. The report provides participant demographics, test


evaluation methods and detail additional information that is necessary for this report. Our

recommendations are based on the data-driven findings and the user-centered theory.


Literature Review

According to Sashoi Green, teacher at Jamaica College (Clarke, 2009, para 6) "There is a

phobia of mathematics, but when students relate it to things around them, they connect more." In

a report in the Jamaica Gleaner, She said students develop a dislike for mathematics at an early

age. (Clarke, 2009, para. 8). This line of thought is in keeping with the train of thought that

exists. In fact, with the current low performance in mathematics within Jamaican Schools, new

policies have been designed and researches are being conducted in an effort to stem the problem.

Another perception of the cause of this low performance is provided by the Government of

Jamaica who has admitted in a statement recently by Opposition spokesman on education,

Senator Basil Waite that “Jamaican students are performing poorly in the area of mathematics

because many primary school teachers did not pass the subject at high school level.” (Douglas,

2009, para. 1)

This problem has led to developments, research and policies that have surfaced. One

example of such is the New Horizons for Primary Schools (NHP), which was implemented by

the United States Agency for International Development (USAID) in 1999 and lasted until 2004.

“The objective of the program was to improve the language arts and mathematics performance of

72 of Jamaica’s poorer performing schools, through a school-based model of intervention.”

(Marlaine, Harris, Gammill, & Barrow, 2006, p.1 ). “The NHP employs a holistic, student centre

approach to improvement of instruction and learning in mathematics and reading (n.d.)


Another current example is the National Mathematics Strategy being undertaken by the

Education Transformation Team under the Ministry of Education. National Mathematics

coordinator, Tamika Benjamin, says the launch of this strategy was the “beginning of a "long and

difficult journey". (Anonymous, 2008, para. 9)

“Through this strategy, direction will be given to stem the continued decline in student

performance and to increase national awareness about the importance of

numeracy.”(Anonymous, 2008 ,Para. 2) the establishment of a National Mathematics and

Numeracy Policy, which is aimed at improving the performance of students’ island wide.

While there are many theories that speak to learning and the use of multimedia

application to aid the learning, the theories I will be focusing on for the purpose of this research

are The Cognitive Theory of Multimedia, Cognitive Load theory, The Multimedia Principle and

the activity theory. According to Mayer & Moreno (2002) The Cognitive Theory of Multimedia

states that multimedia narration and graphical images produce verbal and visual mental

representations, which integrate with prior knowledge to construct new knowledge.(to cite).

According to Mayer and Moreno (1998) and Mayer (2003), the Cognitive

Theory of Multimedia Learning is based on several assumptions. First, working memory

includes auditory and visual channels, which are equivocal to the phonological loop and the visu

CognitiveTheory & Multimedia Instruction spatial sketch pad in Baddeley’s (1986) theory of

working memory. Second, each subsystem of working memory has a limited capacity, consistent

with Cognitive Load Theory (Sweller, 1988, 1994). Third, humans are knowledge-constructing

processors who produce meaningful learning when they attend to relevant incoming information,


organize the information in coherent representational structures, and then integrate it with other

existing knowledge (Mayer, 1996, 1999).

Fourth, connections can be made only if corresponding visual and verbal representations are in

working memory at the same time, which is similar to Paivio’s (1986; Clark and Paivio, 1991)

Dual Coding Theory.

“Cognitive Learning Theory is concerned with the way in which a learner’s cognitive resources

are focused and

used during learning and problem solving, suggesting that for instruction to be effective, care

must be taken to design instruction in a way as to not overload the mind’s capacity for

processing information.”(to cite)

Multimedia Principle

The multimedia principle states that better transfer occurs from animation/pictures and narration/

words than from words alone. When words and pictures are both presented, learners have the

chance to construct verbal and visual cognitive representations and integrate them.

Limitation to this study includes the fact that there have not been much recorded accounts of

similar studies conducted. This limits the resources available to the researcher and the

information for which to base the research on.

The Activity Theory based usability testing methodology offers evaluators an insight into the

natural context of use in an artificial laboratory setting which offers a high degree of control. (to

cite)


Methodology

For the purpose of this research, the group that I have decided to study is grade one (1) students

at the August Town Primary School. Eighteen students were randomly chosen, Nine from each

of the two grade one students classes that were present.

In an effort to effectively choose methods that would reflect the required data needed, two main

usability evaluation methods based on Preece (1993), were considered and chosen. These are

observational evaluation and survey evaluation. “Observational evaluation implies collecting

data that provide information about what users do when interacting with educational software.”

Preece (1993) stated that two broad categories of data may be obtained: how users tackled the

tasks given, where the major difficulties lie and what can be done; and performance measures

like frequency of correct task completion, task timing, frequency of participant errors.

“Surveys are employed to know users' opinions or to understand their preferences about an

existing or potential product through the use of interviews or questionnaires.”

The techniques that were used include direct observation, video recording, software logging,

verbal protocols and Questionnaire administered by interviews.

The Gcompris usability evaluation was conducted as monitored test sessions. Eighteen (18)

students were tested. While this number is not representative of the general grade one population

in Jamaica for a true experimental design, a minimum of 10 to 12 participants per condition must

be used (Spyridakis, 1992). On the other hand, new research has shown that four to five

participants can reveal up to 80% of usability insufficiencies in products, which often times tend

to be most of the major problems (Virzi, 1990).

The test sessions were held in a computer lab on the computer science department of the


University of the West Indies, Mona campus.

Procedure

The eighteen students were divided into two groups of nine and the tests carried out in four

sessions, each group being exposed to two. Each testing lasted for twenty minutes per session.

From henceforth on, the two groups will be referred to as Group A and Group B.

For the first session, Group A had three persons monitoring the test and Group B had five

persons. This was due to the availability of persons to participate as monitors. The students were

transported from their school to the lab. On arrival, the participants were lead to the computer lab

where there were each placed in front of a computer which had the software on screen. The

students were then given one minute without instructions to see if they would know what to do.

After the initial minute, the participants were first introduced to the computer keyboard aspect of

the software where they were allowed to do one (1) activity of falling letters to allow them to

familiarize themselves with the keyboard. After the basic introduction with the computer, the

students were then exposed to basic addition and subtraction mathematics activities. All 9

students were exposed to the mouse where they were allowed to do an activity that required them

using the mouse to click the correct answer.

Two activities were chosen for this.

1. where they were allowed to connect the numbers to draw an object. This not only aided

their mouse using skill but also tested their skills in counting.

2. Others were tested based on their ability to read and identify objects where they were

shown different objects and they were required to click the letter that was missing.


This first session was used as an introductory session. The students were monitored and

comments and expressions were monitored.

The second session however was of a different format, as the students were first exposed to

mathematics addition activities for the first 10 minutes then subtraction activities for the other

ten minutes. Their individual score was documented, after which a simple questionnaire was

administered to the participants individually.

Participant Summary

Eighteen participants participated in the Gcompris usability study. Participants were randomly

selected based on their availability to participate in the study from the two grade one classes at

August Town Primary. The selected participants were mixed in their performance in

mathematics. That means some participants were exceptional performers, some needed extra

help and the others were rather slow. The participants were each given permission letters to give

to their parents/guardians, which were signed and returned to the researcher.

Participant Age Group Sex Previous Computer Usage(1-Expert, 5-Novice)

Used other educational software

Previous Gcompris Usage

1 6 1 F 5 None None

2 6 1 M 5 None None

3 6 1 M 5 None None

4 7 1 F 5 None None

5 7 1 F 5 None None

6 6 1 F 5 None None


7 7 1 M 4 None None

8 7 1 M 5 None None

9 6 1 F 4 None None

10 7 2 F 5 None None

11 7 2 M 5 None None








Gcompris Set-up

Gcompris was pre-installed for each participant on the linux operating system at the computer

science computer lab. During the pretesting familiarizing section of the test, the participants were

introduced to the welcome screen where no instruction was given at first.

The participatory observation as an instrument of data collection is more appropriate in this case

than other instruments that exist. In that , while dealing with the children it is much better to be

able to interact with them while observation their reaction and behavior towards the software.

Furthermore, this is best suited as a usability study consists of and generally involves measuring


how well test subjects respond in four areas: efficiency, accuracy, recall, and emotional response,

which will be useful in what we are trying to achieve.

The aim of the evaluation was to validate the usability of the activities specifically the addition

and subtraction mathematical activities which, are features of Gcompris. The testers were given

activities to complete and their effectiveness, efficiency, reaction and satisfaction of the software

were noted.


Data Presentation

Of the eighteen participant respondents, nine were female and the remaining nine male. Giving a

fifty-fifty percentage. However, group one had five females and four males while group two had

four females and five females. The respondents are all between the age of six and seven. In fact

of the ten of the participants are six years of age while the remaining eight are seven years of

age.

Usability Problems Number of Participants affected

Time given for activities

18

Needs Supervision 14

Navigation 14

Participants were unsure what to do

18

Figure 1: Major Usability Problems and participants affected.

Figure one shows the major usability problems and the number of participants affected by it.

The Time given to complete the activities is at the top because it is the greatest concern among

the participants. However with time and with practice, the respondents will be allowed to

complete the activities within the given time.


Analysis

From the test conducted and from general observations and posttest evaluation, various

findings from which generalizations can be made were found. With reference to the effectiveness

of the GCompris it was found out that although the majority of the participants enjoyed playing

the other activities they did not like the math activities that were available for more than one

reasons, time availability and their ability perform well at the activities. Although the math

questions being asked in the activities were appropriate, 80% of the participants felt that they

needed to be guided when participating in the activities. This is due to their ability to calculate

and function on their own as these participants were not as academically inclined as the others.

While 50% of the participants did not have a problem with the mathematical activities,

the time issues was of a major concern to these participants as they used their fingers to help

them calculate. However based on the researcher’s observations, the participants will eventually

perform better once they are exposed to the software and general computer usage skills on a

weekly basis.

Twenty percent of the participants managed the mathematical program very well because

they were able to apply the skills taught in basic computation to those that were being tested on

the software. For example, one student was given a computation of 10-6=___, immediately she

held up her ten fingers and subtracted 6 and was able to reel the answer (4) in seconds. Another

option was the reading skills which asked them to fill in the missing letters. This was done with


ease as well because the students already had a grasp of the words and so identifying the missing

letters was pretty easy. Owing to the fact that they were using the computers for the first time the

difficulty came when they had to identify the keys on the keyboard in order to get the right

answer.

It is also important to note that the age group used for this exercise was 6-7 so one might

say that the level of work is very easy, but the ability of the students was mixed and so there

were those who were performing at a lower level and those who were excelling. Nevertheless,

the students of both mental capacities were able to use the software with relative ease and

explanation at times.

Overall, the participants were enthusiastic about the software and the activities available on the

software. They enjoyed playing the various games and it for example (insert comments here)

The list below represents some findings based on our participant responses to the feedback survey:

• Of our 18 participants, 14 indicated that they had no previous computer exposure.

• Two of the four with previous experience said they had access to a computer at home..

• Of the 18 participants, none had previous exposure to Gcompris

10 out of the 18 participants suggested they would only use the software again if they had

someone to help them as these students stated that the math activities were too hard. These 10

students happen to be the ones that scored the lowest on the activities


1. Participant 11 Boy is confused when it comes it comes to math, you have to completely

break it down for him or work with him. He is easily distracted and works better on other

activities

2. Participant #12 is unable to do basic addition, she is very slow and loses focus easily

when she gets the answers incorrect. She s unable to recognize basic mathematics signs

on her own.

3. Participant number 10 is very fast, she uses her fingers to assist her in solving problems.

She works off her own initiative and is twice as fast as the others

4. Participant #1 likes math

Likes to do it on the computer

Likes the “thing” that she is using to do the math

5. Participant #7 likes subtraction

-father encourages him to do math

-likes the software

-the math was too hard

-wanted someone to help him because he did not know what to do

-would not use it again because it was too hard.

6. Participant #5 – likes it because its nice

Likes the game with the fish

It was too hard because he can not do the math

He would use it again if he had somebody to help him

6. Participant # 3 it was fun


Fish game is her favourite

The math was not hard

Yes would use it again

Has a computer at home.

7. Participant #2 thought some of the math were hard, wanted someone to help with the

ones that were hard, could do the math

When asked if they would like the teacher to use it in class 90% said yes

Activities the children liked

1. Letters that fall from the sky - 2

2. One participant response was her stomach hurts her when she does not know what to do.

3. Another one likes the addition

Disapproval

- The time was too short

- They were not familiar with the layout of the computer keyboard

It is also important to note that the age group used for this exercise was 6-7 so one might

say that the level of work is very easy, but the ability of the students was mixed and so there

were those who were performing at a lower level and those who were excelling. Nevertheless,

the students of both mental capacities were able to use the software with relative ease and

explanation at times.


Conclusions and Recommendations

From this usability testing research on the extent to which G-Compris, as a tool for teaching

basic mathematics skills can improve or aid learning in mathematics for grade one students at

August Town Primary school, numerous recommendations can be made. Some of these

recommendations have been put forward by monitors and participants themselves and the others

will be put forward by the researcher based on the process and findings of the research.

Recommendations put forward by the monitors/evaluators/participants include;

o The length of the time on the software needs to be slower which will seek to

accommodate students that have learning disabilities and are not so quick.

o Students on initial exposure to the software have to be closely monitored in order to fully

utilise and captivate the benefits that can be had from the software.

Recommendations put forward by the researcher include:

A comparative study to be done between a preparatory and a primary school.

More recorded research including that of usability testing should be done in the

Caribbean so that Caribbean researchers including students may have readily available

literature to them when conducting research of this nature and others.

The developers of children software needs to take into consideration children who are not

tech-savvy hence the researcher is recommending that the developers of GCompris allot


more time to the completion of the initial activities in order to allow users to get familiar

and comfortable with the product.

A computer lab should be made available to the students with academic software such as

Gcompris installed so as to exposure them to alternative methods of learning which will

be beneficial to the teachers and students alike.

The Ministry of Education should re-visit the syllabus

In conclusion, the researcher discovered that while GCompris can indeed improve or aid the

performance of students in mathematics, the timing available for completion of the activities

needs to be revisited for students that are not that academically inclined and needs more time to

work on the activities. The fact that the most of the participants who were all unfamiliar with

GCompris could learn to use it fairly with just two exposures suggests that GCompris is

generally usable for the intended target population. However the developers of the software

needs to look in lengthening the time given for completion of each activity as this present timing

is not conducive or receptive of slow learners.


References

Clarke, O (2009, April 18) Curing the math phobia - Math competition shows problem solving to

be a good tool. Retrieved January 27, 2009 from

http://www.jamaica-gleaner.com/gleaner/20090418/lead/lead4.html

Douglas, L (2009, April 5) 'WE HAVE A MATH PROBLEM'. Retrieved January 27, 2009 from

http://www.jamaicaobserver.com/magazines/Career/html/20090404T220000-0500_148820_OBS__WE_HAVE_A_MATH_PROBLEM__.asp

Francis, P (2006, September 15) Poor CXC results - Jamaican students lag behind region in

Maths, English. Retrieved January 26, 2009 from


Gaffney, G (1999) Information & Design: Usability Testing Series. Retrieved January 26, 2009

from http://www.infodesign.com.au/ftp/UsabilityTesting.pdf

Marlaine, L., Harris, A, Gammill, P & Barrow, K (2006) IMPACT OF NEW HORIZONS FOR

PRIMARY SCHOOLS ON LITERACY AND NUMERACY IN JAMAICA 1999-2004.

Retrieved on February 5, 2008 From

http://www.equip123.net/JEID/articles/2/NewHorizons.pdf

Preece, J. (1993). A Guide to Usability: human factors in computing. Addison Wesley, the Open

University.

http://www.equip123.net/JEID/articles/2/NewHorizons.pdf

http://www.infodesign.com.au/ftp/UsabilityTesting.pdf






Rubin, Jeffrey. (1994). Handbook of Usability Testing: How to Plan, Design and Conduct

Effective Tests. New York: Wileyl

Spyridakis, J. H. (1992). Conducting Research in Technical Communication: the application of

true experimental designs. Technical Communications, Fourth Quarter, 607-624.

Virzi, R. A. (1990). Streamling in the Design Process: Running Fewer Subjects. Proceedings of

the Human Factors Society, 291-294.

Educators count on new maths strategy (2008, June 4) Retrieved on January 27, 2009 from

http://www.jamaica-gleaner.com/gleaner/20080604/news/news3.htmlv

Mayer, R. E. (1996). Learning strategies for making sense out of expository text: The

SOI model for guiding three cognitive processes in knowledge construction. Educational

Psychology Review, 8, 357-371.

Multimedia in Jamaican Schools retrieved on from

http://www.iicd.org/articles/IICDnews.import42/

http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/36/25/

e2.pdf

http://expanding-educational-horizons.com/Numeracy/NumeracyPolicy.pdf

Appendices

http://expanding-educational-horizons.com/Numeracy/NumeracyPolicy.pdf

http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/36/25/e2.pdf

http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/36/25/e2.pdf

http://www.iicd.org/articles/IICDnews.import42/

http://www.jamaica-gleaner.com/gleaner/20080604/news/news3.htmlv


Caribbean Institute of MEDIA and COMMUNICATIONThe University of the West Indies3 Sherlock Drive, Mona Campus

Kingston 7Jamaica, W.I

Tel. (876) 927-1481, 977-2111, 977-0898,977-1398, 977-9990, 702-3353,

Fax (876) 977-1597E-mail [email protected]

Web www.mona.uwi.edu/carimac

Dear Parent/Guardian,

Re: Permission for your Child/Ward to participate in a computer study

This letter seeks permission for your child/ward to participate in a multimedia computer study to be conducted at the University of the West Indies Mona Campus. Although we would have preferred to conduct the study on the premises of the school, the computer lab is currently unavailable for use, hence the need for the study to be done on UWI’s campus. The project will run for 2 to 3 weeks commencing on Tuesday March 17, 2009. It will require that your child visits the UWI computer lab on two occasions within one of the following sessions (Tuesdays and Thursdays 9:00 am – 10:30 am) during duration of the study.

Shakarah McCrae, a final year student at the Caribbean Institute of Media and Communication (CARIMAC), will conduct the study, which is intended to fulfil her research requirement to investigate multimedia software and children in the classroom.

Your child will be required to use a computer with Mathematics learning software for a period of 20 to 30 minutes.

Sincerely,

----------------------------------------

David Bain

Caribbean Institute of MEDIA and COMMUNICATION

http://www.mona.uwi.edu/carimac


Multimedia Co-ordinator CARIMAC, UWI Mona

-----------------------------------------

Shakarah McCrae

Final Year Student CARIMAC UWI Mona

-----------------------------------------

Morlton Wilson

Principal, August Town Primary School

---------------------------------------------------------------------------------------------------------------------

I ____________________________________ have given permission for

Parent/Guardian

__________________________ my child/ward to participate in the multimedia computer

study to be held between March 17, 2009 and March 26, 2009 at the UWI campus.

Documents

Ar3 X0 Final Research Paper G Compris Usability Research Report