ICT in higher education: evaluating outcomes for health education

Journal of Computer Assisted Learning (2001) 17, 275-283

2001 Blackwell Science Ltd 275

ICT in higher education: evaluating outcomes for health education

L. Lockyer, J. Patterson & B. Harper University of Wollongong, Australia

Abstract This paper presents an investigation that examined and compared the effectiveness of collaborative tutorial activities carried out in both web-based and face-to-face learning environments within an undergraduate health education subject. Effectiveness of the different learning environments was measured in terms of observed learning outcomes, analysis of learner interactions and reported perceptions of the learners regarding their experience. Results demonstrated that web-based environments, with embedded collaborative activities, can effectively foster rich learning experiences that result in attaining positive learning outcomes.

Keywords: Collaboration; Health education; Interaction; Learning outcome; Tutorial; Undergraduate; World-wide web

Introduction

It has been argued that higher education should incorporate proven pedagogical strategies such as small group work, cooperative learning, peer teaching, open-ended questioning, role plays, idea sharing and reflection (Ramsden, 1992). However, traditional university teaching has been criticised for being largely devoid of these cognitively powerful strategies because they are often difficult to integrate and implement within traditional structural frameworks of instruction (Ramsden, 1992; Cuban, 1993). The issue has been exacerbated in recent years with increasing student enrolment coupled with declining proportions of tenured academic staff reinforcing the reality of didactic tutorials where the instructor spends the majority of the time lecturing to students (Ainley et al., 1997; Webb, 1983; Brown & Atkins, 1991; OECD, 1997; Welch, 1998).

On-line technologies, specifically web-based applications, are increasingly being utilised as a delivery mechanism in higher education to address these and other issues. Specifically, it is the capability of the web to facilitate communication and collaboration between/among students and instructors that could overcome the increasing barriers to effective teaching and learning in higher education. Research in computer-mediated communication – specifically in asynchronous modes — has demonstrated outcomes such as active participation, equity of participation, peer interaction, multiple perspectives, research, analysis, reflection and divergent thinking (Laurillard, 1993; Romiszowski & Mason, 1996; Harasim et al., 1997).

Accepted 25 May 2001

Correspondence: Dr. Lori Lockyer, Faculty of Education, University of Wollongong, Wollongong, NSW 2522, Australia Email: [email protected]

276 L. Lockyer et al.

2001 Blackwell Science Ltd, Journal of Computer Assisted Learning, 17, 275-283

Thus, indicators suggest that web-based learning environments that include communication-based activities can effectively enhance the educational experience.

However, as Reeves & Reeves (1997) point out, ‘despite all the interest, little research evidence exists to support claims for the effectiveness of Web-based instruction’ (p. 59). Furthermore, and importantly, the thorough examination of the adaptation and integration of those dynamic pedagogical strategies from face-to-face situations to web-based environments has been largely ignored. This paper discusses the adaptation of pedagogical strategies employed in an undergraduate health education subject and their integration in both web-based and face-to-face learning environments. The paper also details the research method and implementation process used and discusses the findings of the investigation.

Subject re-design: focusing on pedagogy before technology

The aforementioned learner-centred strategies hold particular place in the field of health education where emphasis is placed on exploration of health-related values, attitudes and beliefs (Greene & Simons-Morton, 1984; Hendry et al., 1995; O’Connor & Parker, 1995). As such, phase 1 of this investigation involved the exploration of how such pedagogical strategies could be effectively adapted from proven face-to-face situations to web-based delivery. The underlying premise was that it was not possible to move directly from the face-to-face teaching and learning environment to an on-line situation without careful re-conceptualisation of how the pedagogy can be adapted and implemented using the most appropriate technology.

Health and Health Behaviour, an introductory subject within a Bachelor of Education (Physical and Health Education) degree programme, was chosen to be the focus of an investigation of the use of web technologies for health education. The subject served as the precursor for a sequence of subjects that examined the major issues inherent in health and society. The desired learning outcomes in this subject were for students to have: identified and described aspects of specific chronic, degenerative, communicable, and infectious diseases; identified and described issues related to nutrition; described themes of health behaviour and health behaviour theory; and discussed issues of personal choice, attitude and behaviour related to

specific diseases and health issues and practices. The subject had traditionally been delivered face-to-face with a weekly two-hour lecture and a one-hour tutorial that involved a classroom-based seminar format where groups of two or three students worked on one specific topic related to the subject content for presentation during the scheduled tutorial.

The pedagogical strategy for the tutorial component of the subject was re-designed for implementation in both face-to-face and web-based learning environments (see Fig. 1). In the revised version the strategy consisted of task-based collaborations undertaken by learning groups which consisted of four or five students. For each

Lecture

Tutorial Task

Learning Group Collaboration

Learning Group Response/Solution

Lecture

Assigned Textbook Reading

Independent Reserch

Pre existing knowldege,

attitudes, behaviours

Fig. 1. Pedagogical strategy employed in health education

Evaluating outcomes for health education 277


tutorial activity, the learning groups were presented with material designed to stimulate discussion and collaboration (e.g. a survey to be answered, questions to be considered, etc.). The objective for the learning groups was to develop a consensus or shared understanding that was then communicated to the larger class during the subsequent lecture period.

Once the pedagogical model was adapted, a web learning environment was designed and developed. A formative evaluation of the prototype website, with embedded activities, was undertaken. This involved usability testing and expert review of interface issues; clarity and quality of information and external links; and, perceived pedagogical effectiveness of the learning activities.

The structure of the implementation version of the website included four main components: the subject outline; the subject schedule; resources and weblinks; and, most importantly, the Activity Centre that facilitated the learning activities for the students who were participating in web-based tutorials. Attention was given to the structure of the Activity Centre so that it would adequately support each aspect of the tutorial strategy: interactive introductory questions, sharing responses among group members, presentation of group task, a group-specific discussion forum, and a

mechanism for groups to submit their final responses. The navigation was structured so that all areas of activities were available to the student at all times (i.e. students were not forced to follow a particular page sequence when engaging in the activity). The site map for the activities is illustrated in Fig. 2.

A structure similar to that of the web environment was required for implementation of the face-to-face tutorials. Students attended a brief in-class session with the tutor to enable an orientation to the task and the distribution of worksheets that included the introductory questions and group tasks. Groups then had a one-week period in which to develop a group product based on the task and submit it to the tutor in their paper-based group folder.

Implementation: experimental design

Health and Health Behaviour was implemented with the revised pedagogical model and supporting website; 62 students were enrolled in the subject and all participated in the study. An experimental design (phase 2 of the investigation) was implemented in conjunction with delivery of the subject in order to measure the effectiveness of this teaching and learning innovation. Specifically, the focus of the study was to determine: • How effective was the use of a web-based learning environment in delivering

health education relative to that of a face-to-face environment in terms of knowledge, attitude and behaviour change?

Learning Activity

This Week’s Task Intro

Individual Responses

Disucssion Area

GroupTasks

Submit Page Scaffold

or Completed Response Submit

Page Received

CommentReceived

Fig. 2. Site map of web-based tutorial activities



• How did learners participate in and contribute to collaborative health education learning activities within web-based and face-to-face learning environments?

• What were learners’ perceptions of the effectiveness of web-based and face-to-face learning environments for health education?

Students were randomly assigned into two tutorial groups and both tutorial groups were further, randomly, divided into working groups consisting of four or five students; there were seven working groups in each of the two tutorial groups. For the first three weeks of session both tutorial groups (thus all working groups) engaged in face-to-face tutorial activities. For the purposes of the intervention which commenced in week four, each tutorial group was randomly assigned to one of two learning environments (face-to-face or web) for three weeks in the first half of session. In the second half of session, groups exchanged modes, to engage in their tutorials within the alternate learning environment for another three week period.

Both quantitative and qualitative data were collected in order to assess the effectiveness of the collaborative group activities within the web-based and face-to-face learning environments (Fig. 3).

Participants = 62

General Information QuestionnaireComputer Attitude Questionnaire (pre)

HIV/AIDS-related Questionnaire (pre)

Tutorial Class 1:Web LearningEnvironment

Tutorial Class 2Face-to-face Learning

Environment

HIV/AIDS-related Questionnaire (post)

In-depthInterview

Nutrition-related Questionnaire (pre)

Nutrition-related Questionnaire (post)

Tutorial Class 1:Face-to-face Learning

Environment

Tutorial Class 2Web LearningEnvironment

Web Use QuestionnaireComputer Attitude Questionnaire (post)

Experiment 1HIV/AIDSLearning Activities

Experiment 2NutritionLearning Activities

LearningInteractionsRecorded

In-depthInterview

LearningInteractionsRecorded

Fig. 3. Techniques to measure effectiveness of pedagogical model and learning environments



Pre-tests and post-tests for knowledge, attitude, and behaviour related to the health topics covered by the learning activities were administered to all students. These were aspects of HIV/AIDS during the first half of session and aspects of nutrition and body image in the second half. Given the focus for learning outcomes of the investigation, the knowledge, attitude and behaviour test format involved commonly used tools for assessing these domains. The format for knowledge items was true/false/not sure while attitude and behaviour was assessed using Likert-type scales. The questionnaires were developed utilising the content within the knowledge domain of the subject and established instruments developed for the purposes of examining knowledge, attitudes and behaviours (such as Alteneder et al., 1992; Goh et al., 1996; Imperato, 1996; for HIV/AIDS; IOX Assessment Associates, 1988 for nutrition). Questions related to the constructs of the knowledge domain, along with attitudinal and behaviour questions taken from the established questionnaires, were provided to a panel of experts, in the area of health education; these experts were satisfied that criteria for construct and face validity were met. The instruments were then trialed to establish test retest reliability (0.713 for HIV/AIDS; 0.676 for nutrition) and internal reliability of the knowledge scale using Cronbach’s Alpha.

To investigate the nature of the group interactions, the collaborative learning activities were recorded via audio tape and electronic web logs. Once students had engaged in the learning activities, a representative of each working group was chosen, at random, and asked to engage in an in-depth interview regarding his or her experience and perceptions of the face-to-face and web learning environments. Specifically, the interview sessions gathered information regarding: how the group process was managed within both learning environments; perceptions of engaging in the group-based tutorials within the web learning environment; perceptions of the effectiveness of the two environments in terms of facilitating group interaction, and their own understanding of the subject matter.

During the final lecture, all students were asked to complete a survey which included items related to their use of the subject website in general, and specifically their experience of engaging in the web-based learning activities and their perception of their effectiveness.

Findings and discussion

Knowledge, attitude and behaviour change Data associated with the HIV/AIDS and nutrition instruments were analysed to examine the impact of the use of the web-based and face-to-face learning environments on knowledge, attitude and behaviour change. For each topic, HIV/AIDS and nutrition, a score for each domain of knowledge, attitude and behaviour before and after intervention was calculated and means and standard deviations computed. An ANCOVA was performed to assess changes, before and after, in each domain (i.e, knowledge, attitude and behaviour) with the pre-test score as the covariate. Finally, a two-tailed paired t-test was performed to determine whether a significant difference was found between pre-test and post-test scores for each domain. Table 1 provides a summary of the analysis related to knowledge, attitude and behaviour change.

Both web-based and face-to-face learning environments facilitated favourable health-related knowledge construction and attitude and behaviour change. When comparing the two learning environments, one did not prove to be consistently and



significantly more effective than the other in terms of facilitating knowledge constru-ction and attitude and behaviour change related to health education. However, descriptive indices suggested web-based environments may be more effective.

Results associated with attitude and behaviour must be considered in light of the life stage of the study population — first year undergraduate students in their second session of study. Changes in attitude and behaviour associated with HIV/AIDS and nutrition may be related more to changes in sexual practices and eating behaviour, which were associated with the age and lifestyle changes of sample, rather than the health education intervention specifically. Furthermore, these findings are parti-cularly important given the limitations of the investigation. The real world situation of subject delivery necessitated that each intervention consisted of only three weeks. Thus, such knowledge, attitude and behaviour change that was realised was notable.

Participation and contributions to collaborative activities To understand how learners participate and contribute to collaborative learning activities within web-based and face-to-face learning environments, transcripts of group interactions were analysed using a framework adapted from Jara & Mellar (1999), focusing on the learners’ references to the task; group process comments and off-task discussion. In-depth interviews with group representatives provided explanatory data.

Learners contributed fewer individual inputs to group discussions related to the collaborative tutorial activities within the web-based learning environment compared to those in the face-to-face situation. However, learners’ individual contributions within web-based discussion spaces were far more detailed than those that were provided when learners engaged face-to-face.

Learners tended to elaborate on ideas and challenged their group members with requests for clarification within the face-to-face situation more so than via the web. However, learners tended to refer to sources (whether personal experience, subject material or external resources) within the web-based environment more than in face-to-face.

The web-based learning environment seemed to provide nonconfronting learning experiences, as evidenced by learners’ tendency to contribute more references to personal experiences in web-based tutorials than in face-to-face tutorials. Also learners tended to contribute more socio-affective comments toward their learning group members within web-based tutorials than occurred in face-to-face situations.

Table 1. Summary of knowledge, attitude and behaviour change

Web class perform better than Web Face-to-face face-to-face class pre- post- pre- post- pre- to post-test

Experiment 1 (HIV/AIDS) Knowledge 12.03 13.19* 13.37 13.8 ✔ Attitude 16.9 17.43 17.77 17.9 ✔ Behaviour 19.21 20.07 19.3 19.12 ✔

Experiment 2 (Nutrition) Knowledge 10.32 11.04 9.4 11.03* Attitude 22.11 23.54* 22.4 23.07 ✔* Behaviour 16.68 17.39 16.1 15.97 ✔ * Change pre- to post- statistically significant at p < 0.05



These findings support the literature that has suggested that collaborations within computer-mediated communications environments are as powerful as, or even more so than face-to-face encounters (Romiszowski & Mason, 1996).

Perceptions of learners To determine learners’ perceptions of the effectiveness of web-based learning environments the post-intervention survey and in-depth interview data were analysed. Of the 59 students who responded to the survey question related to facil-itating understanding of the subject matter covered, 83% (n = 49) felt that the web-based tutorials were ‘as effective’ or ‘more effective’ than the face-to-face tutorials.

On the assumption that ‘understanding of the subject matter’ was related to subject-related knowledge, this information was compared with gain scores on the knowledge scale of the both HIV/AIDS and nutrition questionnaires for those students who engaged in the HIV/AIDS web tutorials and nutrition web tutorials. Gain scores were dichotomised into ‘increased’ or ‘decreased’ with those students undergoing no change being collapsed into the decreased category. Table 2 summarises the relationship between the students’ reported perceptions of effectiveness and their knowledge scores.

For both HIV/AIDS and nutrition, more students who engaged in the web-based learning environment improved than did not improve. And, of those students who improved, more felt the web tutorials were more effective or as effective than the face-to-face tutorials in facilitating their understanding of the subject matter. Even for those students who did not improve pre to post, more felt that the web tutorials were more effective or as effective than the face-to-face tutorials in facilitating their understanding of the subject matter. Although these differences in percentages did not achieve statistical significance (χ = 0.096, p > 0.05 for HIV/AIDS and χ = 0.82, p > 0.05 for nutrition), they were of practical importance as web-based tutorials were perceived as achieving results in learning and indices pointed, although not significantly, in this direction.

Students also reported that they enjoyed the experience of engaging in the web-based learning environment. Eighty-three percent of the students agreed with the survey statement ‘I enjoyed using the Web site for the EDUP144 tutorials’. Further, during the in-depth interviews, all representatives of the learning groups expressed positive opinions on their experience of engaging in the web-based tutorials. The interview data provided explanatory support to the students’ perception of effectiveness. Many suggested the flexibility afforded by the asynchronous web-based environment contributed to its efficacy for learning as it supported reflection and individual inquiry. This theme of reflection was continuously mentioned by

Table 2. Student perception of Web-based tutorial effectiveness related to HIV/AIDS knowledge scale gain scores and nutrition knowledge gain scores.

Web more effective or as Web less effective Total N effective as face-to-face than face-to-face

Experiment 1 (HIV/AIDS) Knowledge score increased 17 2 19 Knowledge score decreased 10 1 11

Experiment 2 (Nutrition) Knowledge score increased 12 3 15 Knowledge score decreased 8 4 12



interviewees. The opportunity to retain a record of the discussion was considered important.

Regardless of their perception of effectiveness of the web-based or face-to-face learning environments, the issue of interaction with the lecturer was an important one for the students. The component considered to be missing from the web-based tutorials was the ability to ask questions and receive feedback from the lecturer ‘there and then.’ Similar to findings by Schutte (1997) and Oliver & Omari (1999), learners felt that they were afforded more access to the lecturer in the face-to-face tutorials and that this was an important factor in facilitating their understanding of the topics covered within the health education subject.

While group learning methods have been argued for and proven to be effective (Slavin, 1989), participants in this research study perceived difficulties in working on collaborative activities. Interviewees who were members of groups that experienced consistent cohesion in terms of motivation and participation reported positive learning experiences regardless of the learning environment. Similarly, those who experienced group process difficulties reported that these difficulties arose in both face-to-face and web-based learning environments.

Concluding comments and recommendations

The web-based learning environment was effective in facilitating the intended learning outcomes for students undertaking this health education subject and thus provided evidence to support the potential of the web as a medium for collaborative learning activities.

The researchers’ experiences in the study led to the development of recommend-ations for the use of on-line and web-based learning for higher education: • instructional designers should consider the selection of appropriate pedagogical

strategies and design of learning activities as essential components in the development of effective web-based learning environments;

• formative evaluation of learning activities and learning environments prior to implementation (or re-implementation) should be undertaken where possible;

• learners should be supported in developing group management and process skills when collaborative strategies are employed within both face-to-face and web-based learning environments;

• learners should be supported in the development of studentship skills that are specific to engaging in technology-supported learning environments;

• developing structure around the group process and task may facilitate the development of shared group decision-making;

• web-based learning environments should ensure adequate involvement of and access to the lecturer/instructor.

References

Ainley, J., Malley, J. & Lamb, S. (1997) Thematic review of the transition from initial education to working life: Australia background report. Australian Council for Educational Research, Canberra.

Alteneder, R.R., Price, J.H., Telljohan, S.K., Didion, J. & Locher, A. (1992) Using the PRECEDE Model to determine junior high school students’ knowledge, attitudes, and beliefs about AIDS. Journal of School Health, 62, 10, 464–470.

Brown, G. & Atkins, M. (1991) Effective Teaching in Higher Education. Routledge, London.



Cuban, L. (1993) How teachers taught (2nd edn.). Teachers College Press, New York. Goh, D.S., Primavera, C. & Bartalini, G. (1996) Risk behaviours, self-efficacy, and AIDS

prevention among adolescents. The Journal of Psychology, 130, 5, 537–546. Greene, W.H. & Simons-Morton, B.G. (1984) Introduction to Health Education. Macmillian,

New York. Harasim, L., Calvert, T. & Groeneboer, C. (1997) Virtual-U: A Web-based system to support

collaborative learning. In Web-Based Instruction (ed. B.H. Khan) pp. 149–158. Educational Technology Publications, Englewood Cliffs, NJ.

Hendry, L., Shucksmith, J. & Philip, K. (1995) Educating for Health. Cassell, London. Imperato, A.M. (1996) Acquired immunodeficiency syndrome and suburban adolescents:

Knowledge, Attitudes, Behaviours, and Risks. Journal of Community Health, 21, 5, 329–347.

IOX Assessment Associates. (1988) Program evaluation handbook. Center for Health Promotion and Education, Washington, DC.

Jara, M. & Mellar, H. (1999) A methodology for the analysis of collaborative learning in CMC courses: a proposal. Paper presented at the CAL99, London.

Laurillard, D. (1993) Rethinking University teaching. A Framework for the Effective Use of Educational Technology. Routledge, London.

O’Connor, M.L. & Parker, E. (1995) Health Promotion: Principles and Practice in the Australian Context. Allen & Unwin, St. Leonards, NSW.

OECD. (1997) Thematic review of the first years of tertiary education – Country Note: AUSTRALIA. OECD, Paris.

Oliver, R. & Omari, A. (1999) Using on-line technologies to support problem-based learning. Learners’ responses and perceptions. Australian Journal of Educational Technology, 15, 1, 58–79.

Ramsden, P. (1992) Learning to Teach in Higher Education. Routledge, London. Reeves, T.C. & Reeves, P.M. (1997) Effective Dimensions of Interactive Learning on the

World Wide Web. In Web-Based Instruction (ed. B.H. Khan) pp. 59–66. Educational Technology Publications, Englewood Cliffs, NJ.

Romiszowski, A.J. & Mason, R. (1996) Computer-mediated communication. In Handbook of Research for Educational Communications and Technology (ed. D.H. Jonassen) pp. 438–456. Macmillian LIBRARY Reference USA, New York.

Schutte, J.G. (1997) Virtual Teaching in Higher Education: The New Intellectual Superhighway or Another Traffic Jam? [http://www.csun.edu/sociology/virexp.htm] [Accessed 1998, 14 March].

Slavin, R.E. (1989) Research on cooperative learning: an international perspective. Scandinavian Journal of Educational Research, 33, 231–243.

Webb, G. (1983) The tutorial method, learning strategies and student participation in tutorials: some problems and suggested solutions. Programmed Learning and Educational Technology, 20, 2, 117–121.

Welch, A. (1998) The end of certainty? The academic profession and the challenge of change (editorial). Comparative Education Review, 42, 1, 1–7.

Documents

ICT in higher education: evaluating outcomes for health education