IETI36,4 327

The Cautious Student - A User-centred Evaluation ofWeb-supported LearningMark Gillham, Kathy Buckner and Richard Butt, Queen Margaret University College, Edinburgh, UK

SUMMARY

A user-centred evaluation was carried out on a Web site developed to support a traditionally taughtcourse. The students came from an arts course and tended not to be enthusiastic computer users. Thesite was regarded as highly successful in introducing students to Web-based educational material.However, with this application there has been little 'trade off in reducing traditional tutor-studentcontact time. The students that were surveyed seemed resistant to moving further along the path tocomputer-assisted learning and appeared to be content with what they were familiar with.

INTRODUCTION

With the publication of the 'Dealing Report' (1997) onHigher Education in the Learning Society in the UK,there is much pressure on academics to examine waysin which communication and information technologiescan be used to enhance the learning process to improvethe 'quality, flexibility and effectiveness of highereducation' (Dearing, 1997). Providing flexible learningenvironments that can be used in a variety of ways tosuit individual students is likely to become increasinglyimportant.

Many course producers are now developing materialswhich students can access over the World WideWeb - see, for example, the World Lecture Hallat the University of Texas (http://www.utexas.edu/world/lecture/index.htm). While some of thesematerials are presented within an integrated learningenvironment, which enables students to use thecomputer to communicate with each other andwith relevant academics across the network, manyare primarily repositories of administrative data andcourse materials. It has to be acknowledged thatthe support and management issues for developingand maintaining integrated learning environmentscan be problematic (Buckner and Davenport, 1996).Much has been done by the UK funding councils tosupport academics in their professional developmentthrough the provision of funding to projects suchas NetLinks (Levy et al., 1996; Nasser and Ashton,1998) and NetSkills (http://www.netskills.ac.uk/).Many academics can now feel confident about

developing their own materials for publication on theWeb but the extent to which students benefit from suchresources (particularly on courses which require littleprior knowledge of computing) requires furtherinvestigation.

Integrated learning environments or online courseswhich require students to interact with each otheror with their tutor through the computer havebeen investigated by several researchers (Fowell andLevy, 1995; Naidu etal., 1995; Newman etal., 1997).However, in many cases there is a significant learningcurve for students involved in systems requiring thistype of interaction. We believe there is a role forthe provision of support materials on the Web fortraditionally taught courses. This type of provision isless demanding for students in terms of their ability tointeract with a computer but nevertheless providesadjunct materials at a relatively low 'learning cost' forthose students who use them. This may be a precursorto moving more material onto the Web in order toreplace all or more likely some of the traditionaldelivery mechanisms. It is especially likely as an initialstep where students are often not already familiar withthe Web and in disciplines that are not the first toembrace new technologies.

328 IETI36,4

AIM OF PROJECT

The purpose of the research project was to perform auser-orientated evaluation of a Web site providingsupport material for a traditionally taught course andto ascertain students' views of this and alternativeteaching delivery methods.

THE WEB SITE AND ITS EDUCATIONALSETTING

The site under study was introduced in autumn 1997 tosupport a second-year undergraduate degree modulecalled Narrative and Genre (NAR). This module istaken primarily by students on the integrated Honoursdegree programme in Communication Studies atQueen Margaret University College, Edinburgh. TheNAR module introduces students to the economics ofthe film industry and to various theoretical approachesto the study of narrative and genre.

The site was developed by 4th year students whohad previously studied NAR, in collaboration withthe course tutor. The students who helped developthe site were on a level four Computer MediatedCommunication module, and were required, as part oftheir assignment, to design a Web site for a client. Thestudents had themselves taken the Narrative and Genremodule two years previously, and therefore selectedthe module as one which they felt would lend itself wellto Web site development.

The NAR module was taken by 90 students in semester1, academic year 1997/98. These are the first cohort touse the Web site. The module consists of 12 weeks ofteaching where there is a one-hour lecture to the wholegroup and a one-hour seminar delivered to groups of 15students. The assessment procedure requires eachstudent to produce one 1,500 word essay. The Web site(http://jimmy.qmced.ac.uk/usr/cibutt/narrgen/) has asits core a small text-based set of seven pages, mostof which were written specially for the site (see Table1). The content of the site, including the pages ofadditional reading, replicated material given to studentsbefore the availability of the Web site. The choice oftexts is revised each year as part of the ongoingdevelopment of the module, but there was no overallincrease in the number of texts recommended.

EVALUATION METHODOLOGY

The aim of the research was to assess the users'interactions with and views of the site as a supportmechanism for their study. Of particular interest wasthe fact that the users of the site were not on atechnology course and were not generally found to betechnology enthusiasts.

A number of techniques for evaluating onlineand hypermedia courses were examined (Alavi, 1994;Draper et al, 1994; Hutchings et al, 1993; Reitererand Oppermann, 1993). It was then decided toadminister a questionnaire containing mostly closed

Table 1. Description of the contents of the NAR Web site pages

Page title Contents

Menu home page outlining the week by week content of the module and a menu of options. (4290 bytes)

Descriptor a copy of the full module descriptor for NAR. This document is the formal and approved outline ofthe module briefly covering aims, learning outcomes, content, method of study, assessmentmechanism and key reading. (2649 bytes)

Staff a link to the NAR lecturer's home page. (1343 bytes)

Reading (part 1) a list of additional reading texts for each week for the first part of the module with links to weeklylecture overheads. (6809 bytes)

Reading (part 2) a list of additional reading texts for each week for the second part of the module with links to weeklylecture overheads. (8441 bytes)

Links a page of links to external sites of relevance to the module, including academic and commercial sites(film companies and organizations) and fan-produced pages about film stars. (14292 bytes)

Assignment examples of previous assessment essay questions for the module. (5065 bytes)

The Cautious Student - A User-centred Evaluation of Web-supported Teaching 329

questions in order to elicit the following informationfrom the users:

• access and use of the site;• evaluation of key criteria;• general views about Web-based education;• use of other computer facilities and feelings about

computing;• demographic details.

Evaluation criteria were derived after referenceto previous evaluation work by the authors (Gillhamet al., 1996) and to papers specifically looking atWeb review services criteria (Rettig, 1996). Thequestionnaire was administered to the studentsattending the NAR lecture in week 11 of the course.Completion of the questionnaire was voluntary.

RESULTS AND ANALYSIS

The sample

The questionnaire was completed by 38 respondentswhich is 42% of the students registered for the course.In the following analysis it should be rememberedthat 58% did not give their views and considerationshould be given to any bias this may introduce.The respondents were studying one of three under-graduate programmes, Communication Studies (74%),Corporate Communications (18%) and CombinedStudies (8%). Sixty-nine per cent studied year one oftheir course while the rest entered directly into yeartwo after studying elsewhere. The sample consistedof 38% international students (mostly from Norway)and the distribution of gender was 60% female and40% male. Eighty-one per cent of the respondents werebetween 18 and 25 years old.

Access to the site

The NAR Web site was used by 90% of therespondents with only four admitting to not using it.The reasons they gave were that they did not knowabout it or that they had not yet started their assignment(implying that they would use it later). The results inthis section (on access) and the following section (onevaluation) refer to the 34 respondents who used thesite. Most users accessed the site once every two weeksor once a month (see Table 2). Access at less than oncea month (25%) suggests a single visit. It may beexpected that the site would be visited weekly to matchthe weekly pattern of classes on the course. However,only about 9% did this. The information on the site was

Table 2. Frequency of access to the NAR site (n = 32)

How often do you usethe site?

most daysonce a weekonce every 2 weeksonce a monthless than once a month

Percentage

0.09.4

31.234.425.0

Frequency

03

10118

not added to during the semester, so it may be thatstudents accessed information in a single visit ofrelevance for several weeks' work.

The time spent during a visit ranged from 1 to 90minutes. The average times of a short, typical and longvisit were 11 minutes, 21 minutes and 43 minutesrespectively. These figures must be treated cautiouslyas there are likely to be inaccuracies in recall of thisinformation especially as several respondents did notattempt to answer this question.

The users were asked how frequently they visited thedifferent parts of the site (as listed in Table 1). It isinteresting to note that the module descriptor was inmost cases visited just once as might be expected, butthe readings were visited once, occasionally or oftenby significant numbers with the middle frequency(occasional) with the highest score (see Table 3). Thisvariation in access frequency as well as similarvariation seen in access to the assignment suggests thatthere are distinctly different strategies of use inevidence.

It seems that some students were accessinginfrequently, in some cases just once, while others weremaking several visits to these areas of importantinformation. The former may be printing out the keyinformation in bulk (the weekly reading and lectureoverheads) to reduce the need for further visits or mayhave looked at the information and decided that it isnot of further interest.

It would be expected that the staff page was visitedinfrequently as it does not contain informationnecessary for the NAR module, and that the externallinks page would be visited often. Estimated timingsshow that longer is spent looking at external sites(mean 23 minutes per visit) than at the other part of thesite (eg mean 10 minutes per visit for both readingspages). Given the often low frequency of visits it maybe expected that information is gathered in bulk. Over

330 IETI36,4

Table 3. Frequency of access to the NAR site

Menu Descriptor Staff Reading parti Reading part2 Links External Assignment

neveronceoccasionallyoften

07

177

316111

131611

08

158

08

149

546

15

1109

10

90% of the respondents said they printed outinformation from the site while about 24% and 28%stated that they downloaded data or wrote out from thescreen, respectively.

The Web site enabled the students to print off theoverheads before the lecture and annotate them duringthe lecture. This allowed more time in the lectures forthe students to translate the content of the lecture intotheir own words, rather than slavishly copying downthe contents on the overheads, thus facilitating moreeffective learning. Further student time was saved bythe provision of an extensive structured list of linksto relevant Web sites, which alleviated the need forstudents to spend lengthy periods of their time in theInformation Technology Centre using search engines.

Evaluation of the site

The site was highly regarded overall by the users witha large majority stating that it was essential or usefuland every respondent stating that it had assisted themin their studies (see Figure 1). A small majority ofrespondents considered that the site should beexpanded (59%) while 41 % considered it to be the rightsize. No one thought that it was too large.

The respondents were required to score variousevaluation criteria on a scale of 1 (excellent) to 5

(poor). All factors scored favourably with 'usefulnessof the content' scoring highest. The suggestion that thesite was 'fun to use' scored lowest at 2.53 followed bythe 'aesthetics of the design' at 2.22 (see Table 4).

Despite the students spending more time accessingexternal sites than other areas and visiting theseareas most frequently, they assessed the value ofthese external sites to be quite low (see Table 4). Thisrelatively low rating given by the students to theexternal sites is probably explained by the types of siterelevant to this academic area that are available. Themajority of these fall into four categories:

• the Professional sites of the major studios, such asDisney and Paramount, which are rich in graphicalcontent and promotional information, but light onthe type of information, such as the financial detailsof film production, which would be useful forstudents on a module such as this;

• sites of professional related organizations, suchas the British Film Institute, which are only justbeginning the immense task of digitizing theirarchives;

• academic sites such as online journals, which tend tobe pitched at post-graduate level; and

• sites constructed by fans of individual stars ordirectors, which are useful for their filmographies

Value of the site(none answered 'of no use')

of limiteduse (9.1%)

essential,(39.4%)

useful(51.5%)

How the site has assisted study(none answered 'not at all')

a lot(54.5%)

Figure 1. Users' assessment of the value of the site to their study and assistance it has given

The Cautious Student - A User-centred Evaluation of Web-supported Teaching 331

Table 4. Users' scores for evaluation criteria

Criteria

Usefulness of contentEase of useQuality of the site overallQuality of the informationEase of navigationAmount of informationValue of the external linked sitesAesthetics of designFun to use the site

Mean

1.751.811.911.972.002.002.142.222.53

S.D.

0.720.820.780.650.880.800.760.941.05

n

323232323232283232

and interview transcripts, but unreliable in terms ofthe accuracy of their information.

Views of Web-based educational material

All of the respondents said that they would like similarsites available for other modules. Respondents werealso asked a series of questions to see what balancebetween traditional teaching and Web-based deliverythey favoured (see Figure 2).

It is interesting to see a marked conservatism towardsmoving away from the traditional forms of tutorcontact. All the respondents were of course in

attendance at a lecture which may immediately biasthe sample. Also the questions stressed the aspect oftutor contact rather than flexibility in learning as wouldoccur with more of a Web-based programme. Thepopularity of the particular course tutor and the natureof the lectures, which usually include appropriate filmextracts to illustrate points, may be influential here.

MEDIA STUDIES TEACHING AND THE WEB

Perhaps the most educationally useful sites for teachingin the area of Media and Cultural Studies are thevarious commercially produced databases, such asthe Internet Movie Database (http://uk.imdb.com/),the Internet Underground Music Archive (http://www.iuma.com/) and ShowBIZ data (http://www.showbizdata.com/). However, it might be suggestedthat the fact that these sites are the most useful raises abroader issue concerning the impact of computertechnology on knowledge itself, for it is the nature ofthese databases, and of information technologies ingeneral, to privilege particular types of knowledge overothers. As Lyotard (1987, p 4) has argued:

The nature of knowledge cannot survive unchangedwithin the context of general transformation. It can fitinto the new channels, and become operational, onlyif learning is translated into quantities of information.

100 -i

agree not sure disagree

| | I prefer 'traditional' lectures and seminars and see no need for online material like this site.

^ J I like lectures and seminars supported by online material like this site.

I would like fewer lectures and more material online.

\y\ I would like lectures replaced by online material, but still have seminars/tutorials.

f ^ | I would like online teaching with only occasional contact with tutors.

Figure 2. Students' views about various types of teaching approach

332 IETI36.4

We can predict that anything in the constituted body ofknowledge that is not translatable in this way will beabandoned and that the direction of new research willbe dictated by the possibility of its eventual results beingtranslatable into computer language.

The databases referred to above contain informationgermane to Media and Cultural Studies that is easilytranslated. However, as vast as these databases are,what they cannot provide is the type of knowledgewhich makes up the majority of Media and Culturalstudies, if not the Arts and Humanities disciplinesin general. Consequently, as educational practitionersare being encouraged to use communication andinformation technologies to facilitate and enhance thelearning process, practitioners in some disciplines mayfind that only parts of their curriculum can survive inthis new learning environment.

CONCLUSION

It is clear that the students involved in this investigationfound the Web site a useful additional resource withrespect to their study of the NAR module. The siteseemed to be used by many as a rather static source ofinformation with over half of the respondents accessingit only once a month or less. The material was notchanged during the semester, so the batch approachof printing out material and presumably keeping it onfile is appropriate. There is currently no 'latest news'area in the site, but given the weekly meeting of lecturerand students in the classes this may be consideredunnecessary. Given that the site is not as dynamic asmany Web sites, it could be reasonably argued that theinformation could be satisfactorily provided on paperto the students. The area giving real added value inhaving a Web site is the links to external sites. Indeed,it is this facility that was used most frequently althoughnot highly valued by the students. In the delivery of themodule the following year, the students were requiredto use the external sites linked to in the module Website to retrieve information which they were thenrequired to discuss in the seminars as part of an attemptto integrate the site more fully into the teaching andlearning strategies of the module.

The site was highly successful in introducing studentsto Web-based educational material. However, withthis application there has been little 'trade off inreducing traditional contact situations, although whenthe lecturer was absent one week, he was able toprovide an exercise requiring use of the Web site inpreparation for seminar discussion. The students that

were surveyed seemed resistant to moving furtheralong the path to computer-assisted learning; theyappeared to be content with what they were familiarwith. Much of the contentment may be due to factorsother than that of the teaching delivery methods.

Further in-depth study is planned including focusgroups in an attempt to get a deeper understanding ofboth the students' understandings, hopes and concernsabout computer assisted learning based on theirexperiences with the NAR site, and the effect of thesite on the methods of study adopted by the students.

REFERENCES

Alavi, M (1994) 'Computer Mediated CollaborativeLearning - an empirical evaluation', MIS QuarterlyJune, 159-74.

Buckner, K and Davenport E (1996) 'Support for case-based learning in an undergraduate Human Factorsclass', Education for Information, 14, 4, 331–42.

Dearing, R (1997) Higher Education in the LearningSociety, National Committee of Inquiry into HigherEducation, [online]http://www.leeds.ac.uk/educol/ncihe/ (accessed 27/1/98)

Draper, S W, Brown, M I, Edgerton, E, Henderson, F P,McAteer, E, Smith, E D and Watt, H D (1994) Observingand Measuring the Performance of EducationalTechnology, TILT, University of Glasgow.

Fowell, S P and Levy, P (1995) 'Computer-mediatedcommunication in the information curriculum: an initia-tive in computer-supported collaborative learning',Education for Information, 13, 193-210.

Gillham, M, Kemp, R and Buckner, K (1996) 'Evaluatinginteractive multimedia products for the home', The NewReview of Hypermedia and Multimedia, 1, 199-212.

Hutchings, G A, Hall, W and Colbourn, C J (1993)'Patterns of students' interaction with a hypermediasystem', Interacting with Computers, 5, 3, 295-313.

Levy, P, Fowell, S P, Bowskill, N and Worsfold, E(1996) 'NetLinks: a national professional developmentproject for networked learner support', Education forInformation, 14, 4, 261-78.

Lyotard, JF(1987) The Postmodern Condition: A Report onKnowledge, Manchester University Press, Manchester.

Naidu, S, Barrett, J and Olsen, P (1995) 'Improvinginstructional effectiveness with computer mediatedcommunication', Alt-7,3,2, 63-76.

Nasser, N and Ashton, S (1998) 'NetLinks: collaborativeprofessional development for networked learnersupport', [online http://netways.shef.ac.uk/index.htm](update 15/1/98, accessed 27/1/98).

Newman, D R, Johnson, C, Webb, B and Cochrane, C(1997) 'Evaluating the quality of learning in computersupported co-operative learning', Journal of theAmerican Society for Information Science, 48, 6,484-95.

The Cautious Student - A User-centred Evaluation of Web-supported Teaching 333

Reiterer, H and Oppermann, R (1993) 'Evaluation of userinterfaces: EVADIS II a comprehensive evaluationapproach', Behaviour and Information Technology, 12,3,137–48.

Rettig, J (1996) 'Beyond "Cool": analog models forreviewing digital resources', Online, 20, 5, 52-64.

BIOGRAPHICAL NOTES

Dr Mark Gillham is course leader of the BA(Hons)Information Management at Queen MargaretUniversity College, Edinburgh. His current research isin the areas of networked learning, the evaluation ofmultimedia and Internet information systems, and thesocial use and effects of IT in domestic settings.

Kathy Buckner is a lecturer in InformationManagement and has just completed a three yearsecondment to the University College's Centre forTeaching and Learning. Her research interests are inthe areas of networked learning and human-computerinteraction, in particular human factors which affectusers in domestic and educational environments.

Dr Richard Butt lectures in Media and Cultural Studiesand has developed Web-based teaching and resourcematerial for Communication Studies students. Hismain areas of research are in documentary film andcultural history.

Address for correspondence: Dr Mark Gillham,Department of Information Management, QueenMargaret University College, Clerwood Terrace,Edinburgh, EH12 8TS.Tel: +44(0)-131-517-3623;Fax:+44(0)-131-316-4165e-mail: m.gillham@mail.qmced.ac.uk;Web site: http://jimmy.qmced.ac.uk/usr/cigill/