Evaluating Distance Education (or ICT) Programs, Clayton R. Wright

8/8/2019 Evaluating Distance Education (or ICT) Programs, Clayton R. Wright

http://slidepdf.com/reader/full/evaluating-distance-education-or-ict-programs-clayton-r-wright 1/4

Evaluating Distance Education ProgramsItems for Consideration

Clayton R. Wright, PhDcrwr77[@]gmail.com

Purpose: To assess whether a distance education or training program is accessible,

appropriate, effective, affordable, and accepted. This series of questions can be used toevaluate distance education programs. If you substitute distance education with the term³information and communication technologies (ICTs)´, you could use the questions to guide ICTimplementation and/or evaluation.

In order to complete this assessment, we need to determine answers to these basic questions:

y What rationale was established for the program?

y What has the program achieved? What worked and did not work?

y Are the achievements congruent with the rationale?

y What changes should be made to improve the program? What recommendations arefeasible given the current environment? What conditions need to change before other recommendations can be implemented?

Institutional, Ministry, or Business Priorities and Policy Directionsy Rationale for the program and its fit with current provincial, national, and/or business

priorities

y Impact the program has or will have on local communities or business and/or theprovince, or nation

y Fit with institutional or business goals, vision, and policies, e.g., openness, accessibility,equality, and internationalization or global reach

y

Existing opportunities for individuals to upgrade themselves and/or obtain accreditation inthe community, business, province, or country. What is the competition?

y Rationale for selecting a distance delivery mode for this course or program. Is thisrationale congruent with the mission of the institution or business?

y Cost effectiveness of the distance learning program

y Commitment of the institution or business to the program

y Concerns the institution or business may have with the current program. What are thesources of this concern? Are the sources reliable? Are the concerns widespread? Whatindications are present that the concerns are valid?

y Extent to which the institution, at all the appropriate levels, is committed to change

Administrative Structures

y Centralized vs. decentralized structures, pros and cons for the given situationy Administrative structure of the program and the interconnections among academic and

service departments. Are the linkages set out in policy or are they primarily dependent onthe willingness of those in the linked positions to collaborate?

y Administrative capabilities regarding planning, communication, support, and monitoring.Do they have the expertise and resources required to conduct their administrative workeffectively?



y Financial capabilities of the administration. Are adequate financial resources available toadminister the distance education program and do administrators have control over theresources that they require?

y Willingness of administration to operate in a transparent manner; to listen to and respondto faculty, staff, and student concerns, especially the concerns raised by students in ruralareas; to provide needed resources; and to adjust to changing circumstances

y Capability of the administration to make decisions and to act expedientlyy Presence of job descriptions for each type of personnel involved in the administration,

development, delivery, and support of the program. Do the job descriptions contain themission of the institution or business, tasks to be performed, the percentage of time to bespent on each task, the level to which each task is to be performed, the number and typeof positions that report to this position, how individuals will be evaluated, the immediatesupervisor, and the conditions under which an individual may be reprimanded or dismissed?

y Collaborative initiatives administration has fostered in an effort to involve variousstakeholders and to use limited resources wisely

y The accreditation procedures that are accepted by students and the community. Thesemay involve provincial, national, or international bodies.

y Faculty, full-time vs. part-time/contract personnel and their connection with the academicdepartment or business unit and service departments

y Student recruitment process. Emphasis placed on how it is administered to attract those inrural areas or small centres of a company¶s operation and to provide students withaccurate information that they can use to make a decision about their educational or training pursuits.

y Registration procedures that are accessible, simple, and flexible to meet the needs of distance or independent learners

y Procedures employed to give recognition for prior learning. An indication that potentialstudents have been informed of the existence of a prior learning program.

y Fee payment methods and scheduling that are accessible, flexible, and affordable

y

Library and learning resources ± accessibility, variety, depth, and currencyy Counselling and guidance ± accessibility, variety of services including job placement

y Help desk services including availability and type of services offered; training for help deskstaff; the existence of problem-solving rubrics

y Possible re-configurations of the existing administrative structure in order to gain anincrease in effectiveness and efficiencies

Learner, Graduate, and Non-Graduate Profilesy Profile of the learners including past, current, and future students. Particularly, their age,

gender, current educational level, where they live, and competencies or currentemployment or position within the company. The primary focus is to gather student datathat will enable administrators to track trends and predict future possibilities rather than to

focus on one individual.y Completeness of the information program personnel have regarding the attributes and

circumstances of the learners and the availability of this information

y Perceptions students have about the current program ± both positive and negative

y Challenges learners face with the existing program, such as accessibility and the feelingof isolation



y How the challenges faced by learners can be addressed. If technology is employed aspart of the solution, will the learners have the knowledge/experience and technicalcompetencies to use it effectively? Will the learners have access to the technology?

y An indication of whether the graduates would recommend this program to their colleagues

y Privacy policies and care taken to limit access to sensitive information.

Employers of Program Graduatesy Profile of those who hire graduates, e.g., position, type of institution or company, size of

company

y The attributes employers are seeking in graduates

y Shortcomings of the existing program as perceived by those who hire graduates

y The induction or training programs employers must implement in order to address theshortcomings of the program

y Suggestions regarding how the existing program could be improved by those who hiregraduates

y An indication of whether the employers will continue to hire program graduates

Program Design and Developmenty Structure of the academic program and its congruency with local or national needs as well

as its f lexibility to achieve learning outcomes

y Clarity of the learning objectives and learning outcome statements as well as whether theyare suitable for the intended audience and the content to be studied

y A qualitative determination of whether the content of the program meets the needs of thelearners and those who hire graduates

y A quality assessment of the learning materials to ensure that the objectives are congruentwith the instructional/learning strategies and assessment procedures and that theyengage the learners

y Contextualization of the material to ensure that it meets the needs of all learnersparticularly those in rural areas or those who live in foreign countries and may have

different life experiences than those who live in urban areas or in one nationy Packaging of the materials for delivery to a rural or international audience

y Use of media or technology, particularly how it facilitates learning and supportsinteractivity, and its appropriateness for use in a particular social, economic, andgeographical context

y Adherence to intellectual property laws and the treatment of the work of others

y The assessment component of the learning program including its congruency with thelearning objectives and the number and feasibility of formative and summativeassessments

y Quality measures that are used in course design and development including the reviewprocess that ensures that the materials are accurate and current, e.g., the use of editorsand external content reviewers

y Suggestions for improving the quality of the learning materials in order to address theneeds of the students and ensure the stated outcomes of the program are met

Program Deliveryy Program delivery process and procedures including the quality assurance systems that

are in place prior to the distribution of materials

y Delivery mode of the materials and its appropriateness for the audience and congruencywith the instructional intent



y Record keeping procedures that track the distribution of resources and the progress of thelearners

y Distribution of materials in a timely manner and the arrival of the resources in a useableform at its destination

y Emphasis program administrators place on learning support

y Role and effectiveness of the learning or study centres and their staff

y Distribution of learning centres and their accessibility to learners

y Availability of learning resources

y Extent and quality of learning support including communication options available tostudents who need to contact faculty, tutors, counsellors, registration personnel, libraryresource staff, help desk staff, and their peers

y The identification of students who fall behind on their studies and the timely and effectivesupport that is provided to these students

y Turnaround time for student feedback

y The type of feedback provided to learners. Is it constructive, positive, and nurturing? Isdetailed feedback provided?

y Administration of assessments with due regard to security and timeliness

Human Resourcesy An assessment as to whether current staff are experienced and qualified for their

positions

y Hiring procedures that are fair and transparent

y Faculty and tutor issues regarding workload, professional development, and supportreceived from administration

y Recognition and reward systems available to faculty, tutors, and staff

y Recruitment and training of faculty, tutors, and staff

y The support provided to faculty, tutors, and staff who work in rural areas or overseas

Quality Assurancey Quality assurance methods or systems that are in place and their relationship to future

planning

y Mechanisms used to collect information from various stakeholders and to providefeedback to stakeholders

y Visibility of the quality assurance system and its associated criteria or standards

y Faculty and staff buy-in to quality

y Affiliation and accreditation issues as they relate to quality assurance

y Improving the existing quality assurance methods with particular emphasis on thecollection, analysis, and use of data to improve operations especially in rural areas and/or overseas

Overall Recommendationsy Is the program accessible and appropriate for the intended learners?

y Does the program meet its overall objectives? Is it effective?

y Is the current program affordable and sustainable?

y Does the program have credibility with the learners and prospective employers?

y What, if anything, needs to be done to improve the current administration, development,delivery, and assessment of the program?

y Are there suitable benchmarks which could be employed?

Documents

Evaluating Distance Education (or ICT) Programs, Clayton R. Wright