Embed Size (px)
Citation preview
This article was downloaded by: [University of California Santa Cruz]On: 19 November 2014, At: 21:42Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Open Learning: The Journal of Open,Distance and e-LearningPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/copl20
Course evaluation in open and distancelearning: a case study from IndiraGandhi National Open UniversityAshok K. Gaba a & Nirod K. Dash aa Indira Gandhi National Open University , IndiaPublished online: 23 Jan 2007.
To cite this article: Ashok K. Gaba & Nirod K. Dash (2004) Course evaluation in open and distancelearning: a case study from Indira Gandhi National Open University, Open Learning: The Journal ofOpen, Distance and e-Learning, 19:2, 213-221
To link to this article: http://dx.doi.org/10.1080/0268051042000224806
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Open Learning, Vol. 19, No. 2, June 2004
Course evaluation in open and distancelearning: a case study from Indira GandhiNational Open UniversityAshok K. Gaba* & Nirod K. DashIndira Gandhi National Open University, India
The success of any open and distance learning course depends on how well it is designed,executed, and evaluated. Evaluation of a course not only demonstrates its strengths, but alsopoints out any inherent shortcomings in the course. This is why course evaluation constitutes animportant function in an open and distance learning system. The present paper aims to evaluatea distance education course ‘Instruction in higher education’ from the Postgraduate Diploma inHigher Education (PGDHE) offered by the Indira Gandhi National Open University (IGNOU),India. Feedback on this course was obtained from 230 respondents with the help of a structuredquestionnaire. The data collected from students were analysed using percentages. Findings fromthe study provide an understanding of the operation of the different components of the course.While the majority of the respondents were satisfied with self-instructional materials, assignments,the extended contact programme, and delivery mechanisms, a sizeable number of respondentswere dissatisfied with academic counselling, and the use of audio-video programmes. Generally,the respondents provided useful feedback and suggestions which will help the faculty in courserevision.
Keywords: Course evaluation; Open University; Distance education; Students’ reactions; Printedmaterial; Electronic media
The context of the course
Traditionally, professional development of higher education teachers has beendirected towards academic excellence in one’s subject area and the ability to extendknowledge of one’s discipline in various fields of life. But with growing understand-ing of the fact that professional development of a teacher concerns not onlyexcellence in one’s discipline but also the development of a wide range of skillsrelating to competency as a teacher, professional development of teachers at highereducation level has become a distinct area of study, with a well documented bodyof knowledge, generated out of both theory and praxis.
Several attempts have been made by various countries to evolve short-term andlong-term programmes for the professional development of higher education teach-ers. In India, efforts have also been made to provide teacher training and orientation
*Corresponding author: Staff Training and Research Institute of Distance Education, IGNOU,Maidan Garhi, New Delhi 110068, India. Email: [email protected]
ISSN 0268-0513 (print)/ISSN 1469-9958 (online)/04/020213-09 2004 The Open UniversityDOI: 10.1080/0268051042000224806
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 21:
42 1
9 N
ovem
ber
2014
214 A. K. Gaba & N. K. Dash
programmes for the professional development of higher education teachers. At anational level, serious consideration was given to professional development of highereducation teachers in 1986, when the National Policy on Education (NPE) 1986,proposed that professional development of higher education teachers should takeplace through both face-to-face and distance modes. In response to the policydirectives, Academic Staff Colleges were established to provide face-to-face pro-fessional development programmes for teachers of higher education, while the IndiraGandhi National Open University was charged with this responsibility throughdistance mode. In response to the recommendation of NPE-1986, the School ofEducation, IGNOU, designed and developed a Postgraduate Diploma in HigherEducation (PGDHE) to cater to the professional development needs of teachers ofhigher education.
The PGDHE comprises four theoretical courses, a course on project work and acourse delivered by an extended contact programme (ECP) (School of Education,2001). The theoretical courses are:
• Course 1. Higher education: its context and linkages.• Course 2. Instruction in higher education.• Course 3. Higher education: the social-psychological field.• Course 4. Planning and management of higher education.
Course II: instruction in higher education
Course 2, ‘Instruction in higher education’, deals with classroom instruction inhigher education. The course covers various areas relating to teachers’ competen-cies, such as learners’ background, etc, and discusses their potential in the contextof different instructional objectives. The course also discusses the use of technologyand communication skills, instructional procedures, utilization of infrastructurefacilities and curriculum. It also covers evaluation in higher education in India andthe mechanics of educational evaluation. The course consists of 4 blocks and 17units and carries a weighting of 6 credits, which is equivalent to 180 hours ofstudents’ learning time.
Instructional system of the course
The instructional system of the course, like other courses in the PGDHE, comprisesmultimedia packages, which include printed material, audio-video programmes,project work, assignments, the extended contact programme, counselling, andteleconferencing. The printed material of the course consists of four blocks in theform of printed booklets. Each block contains four or five units. The units arewritten in self-instructional style with access devices, namely, the structure of theunit, introduction, objectives, main contents, sub-contents, self-check exercises,summaries, unit-end activities, points for discussion, keywords and suggested read-ings. Printed materials are supplemented with audio-video programmes, whichstrengthen the theoretical knowledge of the students by providing direct experiences
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 21:
42 1
9 N
ovem
ber
2014
Course evaluation: a case study 215
on themes related to higher education. Students are expected to work on three tutormarked assignments and submit them to the Study Centre for assessment andfeedback by academic counsellors. Academic counselling provides a platform wherestudents interact with the academic counsellor (teacher) and clarify their doubts andthe difficulties they face while going through the course. Students also get theopportunity to interact with their peers. Although project work is an independentcourse within the PGDHE, students identify project research topics/problems whichcut across the four theoretical courses. Many students choose to identify researchproblems from Course 2 and develop project proposals for submission to the Schoolof Education. The relevant faculty member approves or rejects the proposals withappropriate feedback and sends them back to the students. The completed projectreports are submitted to the School of Education for evaluation.
The Extended Contact Programme (ECP), a ten-day face-to-face programme,constitutes the practical component of the PGDHE and aims to build the pro-fessional competencies of the students. It covers skill development components fromCourse 2, such as lecture practice, organisation of small group interaction tech-niques, preparation of question papers, etc. The ECP has been designed in such away that it could be organised entirely as a face-to-face programme of ten days indifferent regional centres; or it could also be organised partly through teleconferenc-ing, whereby some of the sessions are presented through teleconference mode andothers through face-to-face mode in each region (Menon & Dash, 1997). Theface-to-face sessions in each regional centre are organised by local resource persons.The teleconferencing mode which is adopted by IGNOU is a two-way audio andone-way video teleconferencing mode. Experts present lessons from the studio (i.e.,the teaching end) and learners receive the messages on a TV monitor with the helpof a satellite link at the regional centre (i.e., the learning end). Students putquestions to the experts and clarify their doubts with the experts as well as withstudents at other receiving centres through long line STD calls.
Delivery system
Instruction in PGDHE is delivered to students through a variety of means. Theprogramme guide, course materials, assignments, and handbooks on project work,are delivered by post. Audio-video programmes, academic counselling services andlibrary facilities are made available to students at the study centres. Audio-videoprogrammes are also delivered to students through broadcast and telecast. The ECPis organised at regional centres, whereas ECP through teleconferencing is organisedfor students by both the headquarters and regional centres.
Issues about design and delivery features of the course
The PGDHE was launched in January 1992. The particular course under consider-ation was already ten years old. Hence, it urgently required evaluation and revision.An initial analysis was undertaken to gain insight into the overall operation of thecourse—both design and delivery—so that key issues could be pulled out in order to
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 21:
42 1
9 N
ovem
ber
2014
216 A. K. Gaba & N. K. Dash
form the bases for course evaluation, and subsequently for course revision. Whiledesigning and developing the course, all possible care was taken with regard to theappropriateness of content, instructional design, andragogic considerations, etc. Itwas also assumed that the course inputs would match the learning requirements ofthe students. However, there was hardly any empirical evidence to support such anassumption. Similarly, it was assumed that other support services like academiccounselling, audio/video programmes, the extended contact programme, assign-ments and project work would also fulfil additional learning requirements of thestudents. Although some feedback was obtained by faculty members informallyduring their face-to-face interaction with students, there was no documentation ofsuch feedback. The same was the case with the delivery system of the course. Thisanalysis of the operation of the course certainly raised some questions/issues whichneeded an empirical examination. The issues arising from the foregoing analysis are:
• How do students of the PGDHE react to various aspects of the printed materialssuch as content, language, format, style, etc.?
• How do they perceive academic counselling?• What is their reaction to the extended contact programme?• How useful do they find the audio/video programmes?• What is their reaction to assignments?• What is their opinion of the delivery system?
In order to address the above issues/questions, a research study was undertaken.
Evaluation method
For the present research, a survey method was adopted. A structured questionnairewas developed by the researchers which consisted of 73 items and was validated withthe help of distance education experts. Although the population of the study couldbroadly be described as all the students of the PGDHE who had enrolled andcompleted the programme successfully since its launch in 1992, a sample of 800students was taken for the purpose of the study, out of which 271 students hadcompleted the PGDHE from December 1996-December 2000, and 529 studentswere registered for the PGDHE during 2001. All eight hundred students were sentthe questionnaires by ordinary post during October 2001. The researchers waitedfor four months. At the end of four months—January 2002—a total of 230 (29%)completed questionnaires were received. Of the 230 respondents, 136 were studentswho had completed the PGDHE from December 1996 to December 2000 and 94respondents were on roll during 2001. The data collected from students wereanalysed using percentages.
Findings and discussions
The major findings of the study, based on analysis and interpretation of theresponses, are presented below.
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 21:
42 1
9 N
ovem
ber
2014
Course evaluation: a case study 217
Printed materials
• The majority of the respondents (about 64%) found very few or no printedmistakes in the self-instructional materials, which shows that adequate care wastaken while printing the material.
• The majority of the respondents (more than 80%) were satisfied with the lan-guage, style, content, and structure of the course materials. However, an in-significant percentage reported content, language, and structure to be poor.
• The majority of the respondents (about 80%) reported that they gained most fromstudying printed materials alone. However, a few of the respondents reported thatthey benefited from attending tutorials, watching video, listening to the audio, andattending teleconferencing. This indicates that the majority of the respondentsdepended on the print materials for their learning. The lower percentage ofresponses with regard to the benefits from other learning resources might be dueto difficulties experienced by students in accessing these learning resources.
• The majority of the respondents (more than 78%) reported that the course helpedthem to improve their teaching skills, to become competent in their teachingprofession; and also met their expectations; but an insignificant percentage of therespondents (2 to 9%) reported in the negative.
• The majority of the respondents (about 78%) found the instructional material tobe fully self-instructional and the content relevant from an instructional point ofview.
• The majority of the respondents (about 87%) attempted self-check questions/ex-ercises/activities and found them very helpful for understanding the material.They preferred objective type questions to descriptive type in these exercises andactivities.
• The majority of the respondents (more than 80%) found that the ‘objectives’, and‘summaries’ in the units were ‘very useful’. They also found that the illustrations,figures, cartoons and diagrams explained the theoretical concepts in self instruc-tional material (SIM).
• The majority of the respondents (about 63%) found that the course was relevantto their professional practice.
Respondents’ suggestions
• Language should be simple.• Progressive methodologies should be included and case studies could be incorpor-
ated.• Very old data are used, and this should be updated regularly.• Objectives and introductions should be clarified.• More references should be provided.• Information on the higher education curriculum, teaching techniques/technology,
instructional strategies, evaluation techniques, how to make a project report,communication skills, research design, statistical techniques—such as standarddeviation, mean, etc—should be included.
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 21:
42 1
9 N
ovem
ber
2014
218 A. K. Gaba & N. K. Dash
• The response to term-end questions should not be 1500 words and it should beto the point and relevant.
• Repetition of topics and lengthy sentences should be avoided.• Content, structure, language, and objectives should be relevant to modern tech-
nology and the environment.• Objective type questions should be incorporated in self-check exercises.• Materials should follow a clear sequence instead of there being scattered para-
graphs and lengthy sentences.• More diagrams and colourful cartoons should be incorporated.• The size of the booklet should be reduced.• Stress should be placed on teaching skills.• Teachers should not allow personal views to influence their evaluation of students’
performance.
In the present study, the majority of the respondents depended on printed materialsrather than other components for their learning. This supports the findings of Sahooet al. (1998), Sahoo (1985) and Koul (1988). The majority of the respondentsreported that the study materials were appropriate and relevant, which supportsKamau (1995). Respondents in the present study reported that they found theintroduction and objectives meaningful and useful. Anne Clyde et al. (1983) alsoreported similarly that students read the module introduction, objectives and studyguides.
Respondents in the present study reported that they attempted self-check exer-cises and found these helpful: in developing a thorough understanding of the contentmaterial; in undertaking self-evaluation; and in paying attention to self-instructionalmaterial. In contrast, Sahoo (1985) and Koul (1988) reported that self-checkexercises at the end of course units were stereotyped and hardly engaged the interestof students. While the majority of the students found the present course materials tobe fully self-instructional, Sahoo (1985) and Koul (1988) found in their studies thatcourse materials were mostly written in essay-type format lacking the pedagogy ofself-learning.
Most of the respondents in the present study were satisfied with the examples,illustrations and cartoons used in the material. However, some of them recom-mended more diagrams. In contrast, the studies by Sahoo (1985) and Koul (1988)found that materials were sketchy; contents were not fully elaborated or explainedwith sufficient examples/illustrations.
The majority of the respondents were satisfied with the language, style, content,and structure of the course materials. Jheengut (1999) also found that 90% of thelearners found the course books easy to read and understand. In contrast, a largemajority of students reported difficulties in comprehension and language of coursematerials in earlier studies (Sahoo, 1985; Koul, 1988).
Academic counselling
Half of the students could not attend counselling sessions due to the followingreasons:
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 21:
42 1
9 N
ovem
ber
2014
Course evaluation: a case study 219
• The great distance of study centres from their residences.• Lack of awareness of the time schedule of counselling sessions.• Counselling was not organised at their respective study centres.• They could not attend counselling sessions because of their busy schedule.
However, those who attended were not completely satisfied and suggested makingcounselling compulsory and increasing the number of counselling sessions. Similardissatisfaction with counselling was earlier reported by Nagaraju (1982) and Sahoo(1985). In order to strengthen counselling activities, respondents suggested thatsteps should be taken for regular monitoring by IGNOU staff. Other suggestionsincluded: advance indication of the counselling schedule, training counsellors inteaching technology, opening counselling centres in district headquarters, deliveringcounselling in workshop mode, using PGDHE students as counsellors and supply-ing the counsellor’s address to students.
Extended contact programme
The majority (about 75%) attended the ECPs and reported that resource personswere helpful. They also reported that they achieved competence in teaching skillsand evaluation processes. Some of the respondents suggested that the quality of theECP should be increased; resource persons should be more knowledgeable; basicinfrastructure should be provided; topics on the Internet and IT should be includedin discussions; and the ECP should be conducted after receiving relevant material.
Media
The majority of the respondents (about 68%) did not benefit from audio/videoprogrammes because of the lack of availability of the basic infrastructure at theirrespective regional centres, lack of awareness of the time schedule and unsuitabletiming for the showing of the audio/video programmes. They suggested that audio/video cassettes should be made available for sale and that more video programmesshould be available and issued to students. However, those who could watch videoprogrammes and listen to audio programmes found them good. This finding isconsistent with Sahoo et al. (1998).
Assignments
The majority of the respondents (about 88%) found that assignments were appro-priate and they were satisfied with the grading. Most of them reported thatassignments helped them in their learning and term-end examinations. They weresatisfied with the types of questions asked in the assignments. However, a fewrespondents reported that assignments were difficult to write. The majority of therespondents reported that assignments were sent back after evaluation.
The above findings are consistent with earlier studies which found that studentswanted detailed comments on their assignments and appropriate evaluation of their
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 21:
42 1
9 N
ovem
ber
2014
220 A. K. Gaba & N. K. Dash
performances (Biswal, 1979; Sahoo, 1985). Respondents suggested that assign-ments should be practically oriented and case based.
Delivery of printed material
The majority of the respondents (about 79%) received material in time and 84%reported that the present system of distribution of material suited them. Thefindings are consistent with Jheengut (1990) who reported that the learners ex-pressed their satisfaction regarding the course and its method of delivery. Thefindings of the present study contradict the earlier findings. For example, Kamau(1995) reported that support services did not assist the learners. The despatch ofmaterials was irregular and time-consuming (Nagaraju, 1982; Sahoo, 1985; Balsub-ramaniam, 1986; Rathore, 1991). However, some students suggested that materialshould be distributed through Regional Centres. The size of all blocks should bereduced and it should be provided in one booklet.
Conclusion
The present paper is the outcome of a research study conducted to evaluatestudents’ perceptions of the effectiveness of the open and distance learning course‘Instruction in higher education’ from the Postgraduate Diploma of Higher Edu-cation, IGNOU. The findings of the study have immediate implications for therevision of the course. However, the findings of the study also provide certaininsights into the operation of distance education courses in the context of developingcountries like India.
In spite of the attempts made by ODL institutions to provide students withmultiple media, the majority of students depend primarily upon the self-instruc-tional printed materials as the major instructional input. However, attention shouldbe paid to the following to make distance education courses more effective:
• The self-instructional printed material should be updated with the latest infor-mation, sound instructional technology, and inbuilt variations in the contenttreatment to meet the learning requirements of students.
• Counselling sessions should be monitored from time to time by Headquarters andorientation of the counsellors should be undertaken.
• The basic infrastructure for the display of audio/video programmes should beprovided at the receiving end. The time schedule for the programme should beprovided in advance to the learners.
• Adequate care must be taken to provide detailed comments on the learners’assignments.
• Self-instructional material should be provided at the time of registration.
Generally, the study has, apart from providing a plethora of feedback on the variousaspects of the course, illustrated certain points which are of great importance for thesystemic development of open and distance learning institutions in the context ofdeveloping countries like India.
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 21:
42 1
9 N
ovem
ber
2014
Course evaluation: a case study 221
Acknowledgements
The present paper is the outcome of a collaborative research study between StaffTraining and Research Institute of Distance Education (STRIDE) and School ofEducation, IGNOU, funded by STRIDE (IGNOU).
References
Balasubramaniam, S. (1986) The status of correspondence courses in India, University News,24(42), 13–15.
Biswal, B. N. (1979) A study of correspondence education in India. Ph.D. thesis, Baroda, TheMaharaja Sayajiro University of Baroda.
Clyde, A., Crowther, H., Patching, W., Putt, I. & Store, R. (1983) How students use distanceteaching material: an institutional study, Distance Education, 4(1).
Government of India (1986) National policy on education (New Delhi, Ministry of HumanResource Development).
Jheengut, I. (1999) Basic English course evaluation. Available online at: http://saide.org.za/world-bank/management/benefits/m42amau.html.
Kamau, J. (1995) Distance learners’ perceptions of the quality of course materials and studentsupport in distance learning programmes in Kenya, paper presented at the InternationalCouncil for Distance Education XVII world conference, Birmingham, UK, 26–30 June.
Koul, B. N., Singh, B. & Ansrai, A. A. (1988) Tutor comments: a distance teaching techniques,in: B. N. Koul et al. (Eds) Studies in distance education (New Delhi, AIU & IGNOU).
Menon, M. B. & Dash, N. K. (1997) Higher education staff development at a distance, in: P.Santosh (Ed.) Staff development in higher and distance education (New Delhi, Aravalli BooksInternational).
Nagaraju, C. S. (1982) Evaluation of correspondence-cum-contact in-service teacher-training pro-gramme in Kerala (Bangalore, Institute for Social and Economic Change).
NCERT (1998) Fifth educational research survey (New Delhi, NCERT).Sahoo, P. K. (1985) A study of correspondence education in an Indian university, Ph.D. thesis,
Baroda, The Maharaja Sayajiro University of Baroda.Sahoo, P. K. & Patri, K. L. (1998) A comparative study of courses from IGNOU and University
correspondence courses in Orissa State, India, paper presented at the Asian Association ofOpen Universities XII annual conference, Hong Kong, 4–6 November.
School of Education (2001) Programme guide (New Delhi, IGNOU).
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 21:
42 1
9 N
ovem
ber
2014
Dow
nloa
ded
by [
Uni
vers
ity o
f C
alif
orni
a Sa
nta
Cru
z] a
t 21:
42 1
9 N
ovem
ber
2014
