How to Discover our Quality

Handbook for Quality Monitoring in

Technical Education in Tanzania

2

Contents

Background ...................................................................................................................... 4 How to use it ........................................................................................................................................................................ 6

Chapter 1 .......................................................................................................................... 7

NACTE and the Technical Institutions ................................................................................ 7 1.1. Introduction ...................................................................................................................................................... 7 1.2. NACTE and the National Qualification framework ............................................................................... 8 1.3 NACTE and the Job profiles .......................................................................................................................... 9 1.3.1 The Content of a Job Profile ..................................................................................................................... 10 1.3.2 How to Develop a Job Profile? ................................................................................................................. 13 1.3.3 A Benchmark Program ................................................................................................................................ 13 1.3.4 The Job Profiles on the Website NACTE ............................................................................................... 14 1.4. Quality monitoring in a new setting ....................................................................................................... 15

Chapter 2 ........................................................................................................................ 17

Internal Quality Assurance (IQA) and External Quality Assurance (EQA), two sides of the same coin ....................................................................................................................... 17 2.1 Responsibilities of the Institution ........................................................................................................... 17 2.1.1. The QA-unit and the QA –officer ............................................................................................................. 17 2.1.2. The Instrument of Self Assessment ........................................................................................................ 20 2.2. Responsibilities of NACTE .......................................................................................................................... 22 2.2.1 . Support of Quality and Quality Assurance by NACTE ....................................................................... 22 2.2.2 Quality monitoring by NACTE ................................................................................................................... 23

Chapter 3 ........................................................................................................................ 26

What is Quality? Some Theory and Background Information .......................................... 26 3. 1 What is Quality? ........................................................................................................................................... 26 3.2 The Stakeholders and Quality .................................................................................................................. 27 3.3 The use of performance indicators and quantitative data ............................................................. 31

Chapter 4 ........................................................................................................................ 34

Quality Monitoring by NACTE ......................................................................................... 34 4.1 Registration .................................................................................................................................................... 34 4.1.1. Registration of a new institution ............................................................................................................. 34 4.1.2 Registration new programs ....................................................................................................................... 35 4.2 The Quality Audit.......................................................................................................................................... 35 4.3 Institutional accreditation ......................................................................................................................... 36 4.4 Program Accreditation ............................................................................................................................... 36

Chapter 5 ........................................................................................................................ 37

Internal Quality Assurance and the Quality Audit ............................................................ 37 5.1 What do we mean by Quality Assurance? ........................................................................................... 37 5.2 Towards an Internal Quality Assurance (IQA) System ...................................................................... 38 5.4 Conducting the Self-assessment ............................................................................................................. 42 5.5. Strengths/Weaknesses Analysis .............................................................................................................. 52 5.6 The Self-assessment Report (SAR) .......................................................................................................... 53 5.7 Preparation for the Quality Audit ........................................................................................................... 54 5.8 The Quality Audit.......................................................................................................................................... 54 5.8.1. he Audit Team ............................................................................................................................................... 55 5.8.2. Preparatory Work of the Audit Team .................................................................................................... 55 5.8.3. The Site Visit .................................................................................................................................................. 55

3

5.8.4 Formulating the Findings ........................................................................................................................... 56 5.8.5 The Exit Meeting ........................................................................................................................................... 57 5. 9 NACTE and the Quality Audit ................................................................................................................... 57

CHAPTER 6 ...................................................................................................................... 59

Self-assessment of the Institution as Preparation for Accreditation ................................. 59 6.1 An analysis Model for the Self-Assessment of the Institution ....................................................... 59 6.2 The self-assessment: the quality aspects to be assessed................................................................ 61

The institution actively encourages the development of all staff. ............................. 67 6.3 Strengths/weaknesses Analysis ............................................................................................................... 71 6.4 The Self-assessment report (SAR) ........................................................................................................... 72 6.5 Preparation for the Institutional Accreditation .................................................................................. 73 6.6 The Institutional Accreditation ................................................................................................................ 74 6.6.1. The Expert Team ........................................................................................................................................... 74 6.6.2 Preparatory Work of The Expert Team ................................................................................................. 75 6.6.3. The Site Visit .................................................................................................................................................. 75 6.6.4 Formulating the Findings ........................................................................................................................... 77 6 .6.5 The Exit Meeting ........................................................................................................................................... 78 6.6.6 The Expert Team's Report ......................................................................................................................... 78 6.7 NACTE and the Institutional Accreditation .......................................................................................... 79

CHAPTER 7 ...................................................................................................................... 81

7. How to Discover the Quality of the Training Programme? ............................................ 81 7.1 An Analysis Model for The Self-Assessment of The Department and Its Programs ............... 81 7.2. The Self-Assessment: The Quality Aspects to Be Assessed ............................................................ 83 7.3 Strengths/Weaknesses Analysis .............................................................................................................. 99 7.4 The Self-Assessment Report (SAR) ...................................................................................................... 101 7.5 The Follow Up After the Self-Assessment ......................................................................................... 102 7.6 Preparation for Program Accreditation ............................................................................................. 102 7.7 The organisation of the Program Accreditation .............................................................................. 102 7.7.1 The expert team ........................................................................................................................................ 102 7.7.2 Preparatory work of the expert team ................................................................................................ 103 7.7.3 The Site Visit Program ............................................................................................................................. 105 7.7.4 Formulating the Findings ........................................................................................................................ 106 7.7.5 The Exiting Meeting .................................................................................................................................. 107 7.7.6 The Expert Team's Report ...................................................................................................................... 107 7.8 NACTE and the Program Accreditation .............................................................................................. 108

Appendix 3: Checklist Quality Aspects of a Programme ................................................. 113

Appendix 4: Independence of Team Members .............................................................. 116 Abbreviations ................................................................................................................................................................. 119

4

Background

NACTE has been installed in 1997 by the National Council for Technical Education Act to oversee and coordinate the provision of technical education and training in Tanzania. The Council became operational in 2001. In the period 2001 – 2016, NACTE had developed processes and procedures to meet the requirements as set in the Act and to comply with the given task. It has undertaken a lot of activities and published many documents. To date, the majority of the Technical Institutions are registered 580 institutions. The phase 2001-2016 is characterized by controlling and guidance of the Technical Institutions by NACTE as the main actor. It was the time, where there was a need to control both the existing and new coming institutions. However, after 15 years of activities the question is if NACTE still is on the right track? What was good and necessary in 1997 might not longer be necessary. Regional and international developments too have their influence on functioning of NACTE.

In the Act from 1997, the following tasks are given to NACTE: a) to register and accredit technical institutions capable of delivering courses; b) to register technical teachers and other qualified technicians; c) to assist technical institutions in the transmission of knowledge, principles and training in the

field of technical education and training for the benefit of the people of Tanzania; d) to assist technical institutions in the overall development of the quality of education they

provide and to promote and to maintain approved academic standards; e) to establish and make awards in technical education which are consistent in standard and

comparable to related awards in Tanzania and internationally; f) to ensure that the quality of education required for the awards is met and maintained

throughout the duration of the delivery of the course; g) to assist technical institutions in their development by introducing and developing policies and

procedures that will allow institutions attain greater autonomy in the delivery of courses; h) to advise the government on the planning and development of technical education including

matters relating to manpower planning, staffing, budgetary and capital provision, the efficiency of the sector and the development of the curriculum;

i) to review technical education and training policies from time to time in the light of changing technologies and economic development;

j) to accept in partial fulfillment of a course of the study leading to the Council's awards such periods of learning and training as it may recognize as being equivalent in outcome to that award;

k) to co-operate with other bodies and organizations for such purposes as the Council may from time to time deem to be appropriate to achieve its objectives;

l) to advise the Minister on the grant of autonomy to any accredited technical college m) to appoint moderators for all validated courses leading to the council's award; n) to appoint the members of its Constituent Boards and Committee o) to receive income and undertake expenditure as maybe necessary or appropriate for the

discharge of its functions; p) to introduce any measures and procedures which may be necessary for the furtherance of the

objectives of the Council. Looking to the Act the following clusters of tasks can be made: i) Formulating standards for the Awards (the National Technical Awards)/Consider Coordinating

Prior Learning Recognition ii) Registration and Accreditation of TI’s iii) Support and guidance of the TI’s iv) Advising the Minister v) Registration of Technical Teachers

5

The most important task in the beginning was the formulation of the National Technical Awards (NTA). The Council has defined and established a range of National Technical Awards (NTA) to be conferred on graduates of technical education and training upon successful completion of their respective studies in various technical fields. The NTA are competence/outcomes based defined according to specific levels of achievement and designed to testify that the holder of the award possesses the requisite knowledge necessary to apply competently the knowledge and skills described in the relevant occupational sector.

After the document analysis and the interviews with stakeholders, the following was concluded: a) The assessment system is seen as too bureaucratic and too complicated. b) Looking at the processes and procedures for registration and accreditation, one can only

conclude that the process is bureaucratic and complicated. The system is not based on real self-assessment. The role of the expert team is more “box ticking” instead of acting as accountant, consultant and adviser. The whole system should be more simplified.

c) The role of NACTE and the institutions are not balanced. d) The role NACTE is playing in the process of registration and accreditation is not in balance with

the role of the institutions. The impression exist that the TI’s are working for NACTE and NACTE is prescribing what the TI’s has to do. There is not always a good balance between the responsibilities of the TI’s and the responsibilities of NACTE.

e) The system is not user-friendly. f) The whole procedure is not user-friendly, nor for NACTE and its experts nor for the institutions. g) The burden for the institution and NACTE is too high h) Although more than 450 institutions are registered, still many others are to come. Furthermore,

many programs have to be validated. Looking at what NACTE is requiring from the institutions, the only conclusion is that for the institutions the burden is too high. In addition, the burden for NACTE is too high as well. The question is: is it possible to develop an assessment system “light” that is efficient and effective, but saving time and money?

i) The documents have to be revised. j) Analysing the documents; it became clear that some of the documents were out-dated and

should be rewritten. It also became clear that the documents were sometimes difficult to understand. Also the assessment tools needed to be reviewed to give the assessment items reasonable weights.

k) The tasks for registration/accreditation and support/guiding are not clearly distinct and separated.

l) There must be a clear separation between support activities and assessment; a Chinese wall should be build. It is not acceptable to have support and assessment in the same hand. When one has supported an institution to develop a curriculum, one should not be involved in the assessment of the program. It will be difficult to call a program weak and not satisfactory, when one has helped to build the program.

For streamlining the processes and procedures for Quality Assurance (QA) and Quality Control (QC), the decision was made to develop a handbook both for the Technical institutions and for NACTE. The handbook for the Technical Institutions is an instrument to develop a quality assurance framework inside the institution. It will contribute to harmonization of the quality assurance at institutional level and it will bring the QA in the institutions in line with regional and international developments. Furthermore, the handbook is a powerful instrument to support the change in Quality Assurance from control orientation towards an improvement orientation. The handbook is a toolkit to prepare the Technical Institutions for external assessment.

Parallel to the development of the handbook for the Technical Institutions, a handbook for NACTE is developed. This handbook replaces1 the current approach of NACTE concerning the activities of

1The handbook is at this moment not yet officially endorsed. This means that in the text replace(s) has to be read as Intended to replace

6

registration, accreditation, validation and approval. The handbook will bring NACTE in line with the regional and international developments. Just as for the Technical Institutions, the handbook will support the change from a bureaucratic, control approach to an improvement oriented approach. And last but not least, the handbook will lighten the burden of NACTE.

The handbooks are based on the IUCEA handbook A Roadmap to quality2. This handbook is adapted to the Tanzanian context of Technical Education. The academic standards, as formulated by NACTE and the specific topics for the Technical Institutions are included in the handbook.

The handbook of the Technical Institutions and the handbook of NACTE are each other’s mirror image. The content of the one is mirrored in the content of the other. The handbook of the institutions is the basis for the internal Quality Assurance inside the institution, but is also the instrument to prepare for the external assessment by NACTE. The handbook for NACTE is the basis for the institutional accreditation, program accreditation and the quality audit.

How to use it The Handbook “Quality in Technical Education” is divided in 7 chapters.

Chapter 1describes the relation between NACTE and the TI’s. It offers the rationale for the way of monitoring the quality in the Technical Education

Chapter 2 shows the relation between internal quality assurance (responsibility of the Technical Institutions) and external assessment (responsibility of NACTE).

Chapter 3 raises the question what is quality and provides some theory and background. This chapter can be used by the QA-officer to promote quality awareness in the institution.

Chapter 4 describes the quality monitoring activities of NACTE

Chapter 5 aims at the Implementation of a Quality Assurance system and is especially useful for the Quality Assurance Coordinators for the development and installation of an Internal Quality Assurance (IQA) system. It shows how to conduct a self-assessment of the IQA-system. This self-assessment is also a preparation for a Quality Audit by NACTE.

Chapter 6 offers an instrument to discover the quality of the institution by means of a structured self-assessment. This self-assessment is also a preparation for an Institutional accreditation by NACTE.

Chapter 7 offers the department the instrument of self-assessment of the programme(s). This self-assessment is a preparation for the Programme accreditation by NACTE.

The handbook How to Discover our Quality aims to:

stimulate an effective and efficient quality monitoring system for NACTE

streamline the QA processes and procedures of NACTE

make the system more user-friendly

develop an adequate External Quality Assessment system that fits international developments.

support the Technical Institutions in Tanzania in:

Implementing good practices for quality assurance.

Applying the standards and criteria, as formulated by NACTE.

Developing an adequate Internal Quality Assurance system that fits international developments.

Discovering their own quality by offering self-assessment instruments for IQA, the program, and for the institutions as a whole.

2Inter-university Council for East Africa, A ROADMAP TO QUALITY, Handbook for Quality Assurance in Higher Education (volume 1,2,3 and 4)

7

Chapter 1

NACTE and the Technical Institutions

1.1. Introduction The National Council for Technical Education (NACTE) was established by Act of Parliament Cap. 129 to oversee and coordinate the provision of technical education and training in all non-university institutions in Tanzania. Technical education and training in this context is defined as: “education and training undertaken by students to equip them to play roles requiring higher levels of skills, knowledge, understanding and attitudes/ethics and in which they take responsibility for their areas of specialization.” This scope of NACTE covers all tertiary education and training institutions other than universities and their affiliated colleges, delivering courses at technician, semi professional and professional levels leading to awards of certificates, diplomas degrees and other related awards. NACTE has under its ambit 600 institutions and has registered all of them. The mandate of the National Council for Technical Education as derived from Cap 129 is threefold and may be summarized as follows: Regulatory Function: to establish the regulatory framework for technical education and training,

leading to quality assured qualifications; Quality Assurance Function: to assist technical institutions to improve and maintain the quality

of the education they provide and to ensure that their programs meet labour market demand, by guiding and monitoring their adherence to the regulatory framework; and

Advisory Function: to advise both Government and technical institutions on the strategic development of technical education and training.

These three aspects are interrelated and together make up the core functions of NACTE. The key issues that NACTE is required to address include: To register and accredit both public and private technical education and training institutions

capable of delivering courses; To register technical teachers (teaching staff in technical institutions); Ensure that the quality of education required for the NACTE awards is met and maintained

throughout the duration of the delivery of the course (monitoring and evaluation); To ensure the relevance of technical education and training to labor market demand; To institute systems of quality control and quality assurance in technical education and training;

and To maintain databases on technical education and training, which will serve as a source of

information needed by the Government and other stakeholders for strategic development of technical education and training in Tanzania.

From the above functions it is clear that NACTE’s core business is to establish, sustain, regulate and uphold standards in technical education in Tanzania. As a regulator, NACTE has established a framework that sets the quality standards for technical education and training. All developed procedures leading to registration and accreditation of institutions as well as the adoption of a Technical Education Qualifications Framework are intended to sustain and uphold standards in technical education in Tanzania. NACTE has formulated its vision as “To establish a well organized efficient and effective system of national qualifications and excellence in the delivery of technical education and training, and the resulting output and formulated its mission as “To establish and operationalize policies, regulations and procedures for setting and maintaining standards and quality of technical education and training and advising the Government on the strategic development of the sector.” The mission statement is translated into the following three basic objectives: To establish and operationalize policies, regulations and procedures for setting and maintaining

standards and quality of technical education and training; To assist technical institutions to achieve and maintain high quality of education and training;

8

and To advise the Government on the strategic development of technical education and training. The guiding theme for the National Council for Technical Education is “Striving for world-class excellence in technical education and training”. NACTE’s employees are guided by the following core values: • Upholding professional standards and moral ethics; • Ensuring stakeholders participation in NACTE operations; • Commitment and self motivation towards achieving NACTE’s goals, • Objectivity, transparency and public accountability; • Social responsibility, and • Judicious application of ICT.

NACTE is cognizant of and fully accepts its responsibility of contributing significantly towards the realization of the Tanzania vision 2025 which envisages Tanzania to be a nation with high quality livelihood; a nation which produces the quantity and quality of well educated people sufficiently equipped with requisite knowledge and skills to solve the society’s problems, meet the challenges of development and attain a strong and competitive economy. These aspirations of the country can be realized, to a greater extent, with the availability of a technical education and training system, which the Council is endeavouring to establish, capable of producing a critical mass of high quality technicians and professionals required to effectively respond to and manage development challenges of our nation at all levels. Since the start of the activities, NACTE has done what was necessary to assure the quality of technical education in Tanzania. In this handbook, NACTE shows the way forward and the new approaches it has adopted.

1.2. NACTE and the National Qualification framework To create a critical mass of intermediate and full professional human resources that can meet the economic and social needs of the country NACTE has defined and established a range and levels of awards in technical education (emphasis on theory) and training (emphasis on skills and technology). The NACTE NTA system has seven levels linked to a three level National Vocational Training Awards (NVTA) system under the Vocational Education and Training Authority (VETA). This provides a ten level framework of Technical and Vocational Education and Training (TVET) qualifications. Each qualification on the NTA system has a broad competence level descriptor as shown in table 1:

Table 1: TVET Qualifications Framework

S/N Qualification Level Qualification Award Competence Level Descriptors

1 NVA Level 1 Transcript The holder of the qualification will be able to apply “basic vocational knowledge and skills”

2 NVA Level 2 Basic Vocational Certificate The holder of the qualification will be able to apply “intermediate vocational knowledge and skills”

3 NVA Level 3 Vocational Certificate The holder of the qualification will be able to apply “full / higher vocational knowledge and skills”

4 NTA Level 4 Basic Technician Certificate The holder of the qualification will be able to apply skills and knowledge at routine level

5 NTA Level 5 Technician Certificate The holder of the qualification will be able to apply skills and knowledge in a range of activities, some of which are non-routine and be able to assume operational responsibilities.

6 NTA Level 6 Ordinary Diploma The holder of the qualification will be able to apply skills and knowledge in a broad range of work activities, most of which are non-routine.

7 NTA Level 7 Higher Diploma The holder of the qualification will be able to apply knowledge, skills and understanding in a broad range of complex technical activities, a high degree of personal responsibility and some responsibility for work of others

9

8 NTA Level 8 Bachelors Degree The holder of the qualification will be able to apply knowledge, skills and understanding in a wide and unpredictable variety of contexts with substantial personal responsibility, responsibility for the work of others and responsibility for the allocation of resources, policy, planning, execution and evaluation.

9 NTA Level 9 Masters Degree The holder of the qualification will be able to display mastery of a complex and specialised area of knowledge and skills, employing knowledge and understanding to conduct research or advanced technical or professional activity, able to work autonomously and in complex and unpredictable situations

10 NTA Level 10 Doctor of Philosophy The holder of the qualification will be able to apply knowledge and understanding and do advanced research resulting into significant and original contributions to a specialised field, demonstrate a command of methodological issues and engaging in critical dialogue with peers, able to work autonomously and in complex and unpredictable situations.

1.3 NACTE and the Job profiles So far, NACTE has published 36 qualification standards. The qualification standards provided the following information: i) The occupational profile; ii) Name of the qualification; iii) Purpose of the qualification; iv) Competence level descriptors; v) Total credits; vi) Principal learning outcomes; vii) Assessment criteria; viii) Essential knowledge; ix) Essential skills; and x) Essential tools. For an example of a Qualification standard see the example of the Basic Technician Certificate in Accountancy in appendix 4. Instead of using Qualification standards, which might lead to some misunderstanding, NACTE will use the term Job profile. This is strongly connected to the idea of Competence-based education and training. Competence-based education and training is an approach to instruction based on the philosophy that “given appropriate instruction, time and conditions, almost all learners can and will learn most of what they are supposed to learn (includes what they are taught and what they achieve from self learning.)” In Competence based learning the emphasis is on outcomes of learning based on performance standards, which have to meet the skills required by an employee at the workplace. An outcome is a result of the learning process. It is anything that one can demonstrate after undergoing a learning process. For instance one can demonstrate the ability/competence to repair a car or construct a brick house. One can show also that he/she understands how to solve problems, plan, communicate well or collect and sort out information. Measurable learning outcomes, including their assessment criteria are the key features of Competence-based education and training. Therefore, a job profile, containing all-important elements is a prerequisite for the design of a curriculum. A job profile provides the institution with a description of the profession and contains the expectation of the employers concerning the graduate. A Job profile contains The name of the qualification Description of the profession

10

The competence level descriptor NQF, The Expected Learning Outcomes The Assessment Criteria. Total number of credits

1.3.1 The Content of a Job Profile 1. Name of the Qualification The job profile starts with the name of the qualification, for example Basic Technician Certificate in Accountancy, NTA level 4 2. The description of the profession: The description at least contains the following:

A statement on typical context where the graduate with the particular qualification will work, for example, in farm estates, hospital laboratories, manufacturing concerns, retail outlets, and so on;

A statement on the level at which graduates is expected to function, by indicating the level of degree of independence in the workplace, as per NTA Level Descriptors.

3. The Competence Level Descriptor NQF The job description reflects the clusters of attributes as indicated in the specification of NTA Competence Level Descriptors, which include:

Knowledge and Understanding;

Practical Skills;

Communication Skills; and

Wider abilities for executing workplace tasks / functions effectively.

4. The Number of Credits

The job profile contains the number of credits. At least 120 for each level and 240 for NTA level 7.

5. Expected Learning Outcomes

In the Job profile the expected learning outcomes are formulated. Expected Learning Outcomes are statements of the knowledge, skills and attitude that a learner is able to demonstrate on completion of a learning process. Learning Outcomes can be separated in three domains

Cognitive learning (Knowledge)

Psychomotor learning (Skills)

Affective learning (Attitude),

In the taxonomy of Bloom, the teaching and learning hierarchy is important for the correct and consistent building of the knowledge side of the Learning Outcomes. The cognitive domain comprises six levels starting with the easiest level remembering and ending in the top with creating as the most complex level of the taxonomy (see figure 1). Formulating Expected Learning Outcomes one has to formulate actions, starting at the lowest level of the taxonomy. See, for examples, figure 2.

11

Figure 1: Revised Taxonomy of Bloom (Anderson and Krathwohl 2001)3

Figure 2: Action words for the cognitive domain (Anderson and Krathwohl 2001) There are two kinds of competencies: competencies that can be taught (and are part of education)

3Krathwohl, D.R., Bloom, B.S. and Masia, B.B. Taxonomy of Educational Objectives: Handbook II. The

Affective Domain.N.Y., David McKay Company, Inc. 1964. In: Van der Klip, Cees: Profession based education

and training. ,A Teachers guide, Draft 2015

12

and competencies that are personal characteristics and cannot be taught. One can teach Novel Writing, but one cannot teach the talent for novel writing. The Competences that can be taught are called Expected Learning Outcomes. Expected Learning Outcomes can be divided into: • Knowledge

Knowledge means the outcome of the assimilation of information through learning. Knowledge is the body of facts, principles, theories and practices that is related to a field of work or study. Knowledge is described as theoretical and/or factual;

• Skills Skills mean the ability to apply knowledge and use know-how to complete tasks and solve problems. Skills are categorized as: Cognitive skills (involving the use of logical, intuitive and creative thinking); Practical skills (involving manual dexterity and the use of methods, materials, tools and

instruments); Interpersonal skills (the way of communication, cooperation, etc.).

Attitude A predisposition or a tendency to respond positively or negatively towards a certain idea, object, person, or situation. Attitude influences an individual's choice of action, and responses to challenges, incentives, and rewards (together called stimuli). Four major components of attitude are (1) Affective: emotions or feelings. (2) Cognitive: belief or opinions held consciously. (3) Conative: inclination (natural tendency) for action. (4) Evaluative: positive or negative response to stimuli.

Figure 3 shows the relationships between knowledge, skills and attitude. Figure 3: Categorisation of Learning Outcomes

Figure 3: Categorisation of Learning Outcomes In formulating learning outcomes, a distinction has to be made between generic learning outcomes and module specific learning outcomes. Generic learning outcomes are those outcomes expected from all academic trained graduates. Examples of generic learning outcomes are: problem solving, communication skills, and ability to cooperate. A key characteristic of a generic learning outcome is that you have to practice it in a specific field. Module specific learning outcomes are those that are typical to that discipline.

Competencies

A tudeSkillsKnowledge

LearningoutcomesIndividualcharacteris cs

cogni veskills

prac calskills

Inter-personalskills

13

6. Assessment Criteria Assessment criteria are statements that clearly indicate what learners are expected to do in order to demonstrate that they have successfully realized an Expected Learning Outcome. They are required mainly to assist the institutions in developing corresponding assessment.

1.3.2 How to Develop a Job Profile?

Because a job profile describes the content of the job/profession and the expected competencies, it is self-evident that the job profile only can be developed in consultation of the labour market. For the formulation of a job profile, the following steps will be taken:

1. NACTE will install a Task Force to develop the job profile. The development of the job profile is seen as a national activity and therefore a task of NACTE. If NACTE has not developed or is not developing a specific job profile, a TI or a group of TI’s may take the initiative. In that case, the same procedure will be followed.

2. Members of the Task Force are representatives of: a. NACTE b. The TI’s, offering programmes in that specific field c. The labour market d. The professional body (if any) e. MOET

3. The Task Force decides about the job(s)/profession(s) that will be included in its activity. For example nurse/nursing or financial manager/financial management.

4. The Task Force will conduct a task analyse of the job/profession, among others by analysing advertisements

5. Based on the task analysis, the Task Force will formulate the expected competencies and translate them in Expected Learning Outcomes (see section 1.3.1)

6. After formulation of the ELO’s, the Task Force will formulate a benchmark program (see section 1.3.3)

7. Finally, the Task Force will formulate the text for publication on the website (see section 1.3.4)

1.3.3 A Benchmark Program

After formulation of the Expected Learning outcomes, the Task Force will develop a benchmark program. Such a program is not a straitjacket and is not a prescription how a program should look like, but is meant as a benchmark: a TI can check how far its own program is in line with what is expected. The next step in the process of the job profile is to identify what modules are needed in the program to achieve the ELO’s. Thereafter, the Task Force will examine the core modules and the supporting modules. To check if the planned courses cover the learning outcomes, it is important to develop a curriculum alignment matrix as shown in Table 2. For each course one has to formulate the specific learning outcomes for that course and to check how far this course contributes to the program-learning outcomes.

Table 2: Curriculum alignment matrix Expected Learning Outcome Module 1 Module 2 Module 3 Module 4 Module 5

Communication skills x x

Critical thinking x x x

Problem solving x x

Cooperate/working together x x x

etc

14

When the Task Force has completed the Curriculum alignment matrix, it will summarize the modules for a specific program. Distinction should be made between

Core modules. These are the essential modules offering a thorough foundation of the profession. The core modules are the backbone of the program. They are typical job-related modules a student may never miss. This means that the modules are compulsory.

Supporting modules. These are modules for backing up the core modules. Without these modules it will be difficult to understand the core modules. For example “Business Mathematics”, “Basics of Computer Studies”. Those modules are also compulsory for all students.

Elective modules. These are modules that can be taken by a student, to deepen or to broaden the knowledge, but they are not compulsory. However, a student has to make a choice.

1.3.4 The Job Profiles on the Website NACTE The last task of the Task Force is the formulation of a job profile document to be published on the website of NACTE. This document should contain the following information: 1. Name of the Qualification The job profile starts with the name of the qualification, for example Basic Technician Certificate in Accountancy, NTA level 4 2. The Description of the Profession: The description at least contains the following:

A statement on typical context where the graduate with the particular qualification will work, for example, in farm estates, hospital laboratories, manufacturing concerns, retail outlets, and so on;

A statement on the level at which graduates is expected to function, by indicating the level of degree of independence in the workplace, as per NTA Level Descriptors.

3. Expected Learning Outcomes The job description reflects the clusters of attributes as indicated in the specification of NTA Competence Level Descriptors, which include:

Knowledge and Understanding;

Practical Skills;

Communication Skills; and

Wider abilities for executing workplace tasks / functions effectively.

4. The number of credits

The job profile contains the number of credits. At least 120 for each level and 240 for NTA level 7. The information will be presented as shown in table 3.

Table 3: example of a job profile

Job Profile Accountancy

Name of the Qualification

Basic Technician Certificate in Accountancy NTA level 4

Description of the profession

The employee will - record accounting data, - process receipts/payments using book-keeping and ICT skills, - handle documents and provide customer care on day to day basis. - Etc.

Competence level descriptor

The holder of the qualification will be able to apply skills and knowledge at routine level

Number of credits 120

Expected Learning Outcomes

Will be able to: 6. Apply bookkeeping and arithmetic skills to record accounting data and process receipts

and payments 7. Apply communication and customer care skills to handle clients 8. Operate ICT instruments to capture data and communicate information

15

9. Apply record keeping skills to handle documents Etc.

Assessment Criteria

ELO 1

Arithmetic principles are applied in handling accounting data;

Bookkeeping principles are applied in recording accounting data; and

Relevant Regulations and procedures are used in processing receipts and payments. ELO 2

Communication skills are applied in carrying out daily operations;

English language skills are applied in daily operations; and

Customer care techniques are used in handling internal and external customers. ELO 3

ICT knowledge and skills are used in operating computers and other office equipment;

Word processor, spread sheets and accounting packages are applied to capture data; and

ICT tools are applied in communicating information. ELO 4

Information handling techniques are demonstrated in carrying out daily operations;

Record keeping skills and procedures are applied to classify, store and retrieve documents;

Record keeping procedures are used to maintain records movement register; and

Safety and security procedures are observed in handling documents.

Benchmark program Core modules Supporting modules 1 1 2 2 3 3 4 4 5 5 6 6

The formulated job profiles are a starting point for the institution in curriculum design, might be the design of new programs, and might be for the revision of an old program.

1.4. Quality monitoring in a new setting

Quality monitoring in the new setting aims at making the system user-friendlier and less bureaucratic. It is less control oriented and more improvement oriented. Also the role of the TI’s in the monitoring process is a more active one by putting more emphasis on the self-assessment by the institution. NACTE will monitor the quality of the institution and its programs by organizing external quality assessments. The process of quality monitoring contains the following sub-processes: 1. Registration 2. Quality Audit 3. Institutional accreditation 4. Program Accreditation In table 4 the old and the new system are compared. The different activities in the field of quality monitoring are discussed in chapter 4. Table 4: Old and new system of quality monitoring

current New

Procedure Based upon Procedure Based upon

Registration:

Procedures for Registering and Accreditation Technical Institutions in Tanzania

Registration: (new institutions; new programs

Information and documents provided by TI

Accreditation (of the TI)

Full

Provisional

Procedures for Registering and Accreditation Technical Institutions in Tanzania

Guidelines for Preparation of Quality Management Plan for Institutions

Institutional Accreditation

Self assessment according guidelines in chapter 6 of the handbook

16

Accredited by NACTE

Self Evaluation Study Guide

NACTE Academic Quality Standards

Performance Indicators for Assessment of Institutions

for TI’s

External assessment according chapter 6 of the handbook

Departmental recognition Procedures for Curriculum Development and Review

Procedures for Curriculum Approval and Validation

Program accreditation

Self assessment according guidelines in chapter 7 of the handbook

External assessment according chapter 7 of the handbook

Program approval

Program validation

Quality assurance system Is included in Institutional Accreditation. IQA system based on:

Guidelines for Preparation of Quality Management Plan for Institutions Accredited by NACTE, June 2004

Framework for Institutional Quality Assurance, August 2004

Guidelines for Establishing Institutional Policies and Procedures on Quality Control and Quality Assurance, June 2010

Quality Audit Self assessment according guidelines in chapter 5 of the handbook

Quality audit according chapter 5 of the handbook.

In the following chapters, the elements of the Quality Monitoring System are discussed: 1. Institutional Registration (chapter 4) 2. Internal Quality Assurance (chapter 5) 3. Institutional Accreditation (chapter 6) 4. Programme Accreditation (chapter 7)

17

Chapter 2

Internal Quality Assurance (IQA) and External Quality Assurance (EQA), two sides of the same coin Quality Assurance in Higher Education, including the Technical Institutions has two elements: - Internal Quality Assurance - External Monitoring and Assessment. In the case of Tanzania it concerns, on one side the responsibility of the Technical Institutions to assure the quality of the institution, the quality of the program with the final aim the quality of the graduate. On the other side the responsibility of the outside regulatory body NACTE to set standards for the quality and to check if the institutions meet the requirements set by NACTE in consultation with the key stakeholders.

2.1 Responsibilities of the Institution It is the responsibility of the institution to develop processes and procedures in such a way that it provides confidences to the stakeholders that it delivers competent graduates in an efficient and effective way. To achieve this, the institution must: - Appoint a Quality-officer and set up a quality unit; - Implement a well functioning Internal Quality assurance system; - Conduct a self-assessment of its IQA-system, every 5 years; - Conduct a self-assessment of the institution, every 5 years; - Conduct self–assessment of its programs on a regular base The above-mentioned activities will show the strengths and the weakness of the institution and its programmes. The outcomes of the self-assessment will lead to a quality improvement plan to tackle the shortcomings. The institution will report yearly to NACTE (via the QA-officer) about the state of the art of the Quality improvement plan.

2.1.1. The QA-unit and the QA –officer According to NACTE, an institution must have a Quality Assurance Committee to ensure quality policies and objectives are set, implemented and evaluated. NACTE describes a recommended Quality Management Structure for the institution.4 Organisational structure Of course, there is no one system that fits all. A large institution has other needs than a small one. But in general we can look at the following structure for Quality Assurance at the TI (figure 4): 1. At central level, the institution has installed or will install a Quality Assurance Committee. The

Principal/Rector of the institution will chair the Committee. Members of the QA-committee are the headheads of d the departments. The role of the QA-committee is a decision making one. In the QA-Committee, all decisions concerning quality of the institutions are made.

2. At central level, the institution must have a Quality Assurance centre or Quality Assurance unit, headed by a QA-officer. The QA-centre or unit is established for the purpose of leading, supporting and coordinating quality assurance processes in all units of the institution. It should be acknowledged that administrative procedures vary from one Technical Education institution to another. A high-ranking faculty member should run the QA-unit. He/she should have the expertise, knowledge and experience of quality assurance in Tertiary Education, and the ability to take over the reins of effective leadership. The manager of the quality assurance centre should also be administratively accountable to the Principal/Rector of the institution.

4NACTE, Guidelines for preparation of Quality Management Plan for institutions accredited by NACTE, August 2004.

18

As for the staff required by the QA- unit, this depends on the size of the institution and multiplicity of its facilities, and the extent of centralization of the responsibilities of the centre or their distribution to all departments and units of the institution. In any case, the procedures used in the quality assurance centres focus on the small size of the centre and its effectiveness, so the staff usually consist of two to five members, including the director of the centre, the administrative support team, secretary, and one or two specialists to whom certain responsibilities are assigned. The QA-unit must act independently and autonomous, but is accountable to the QA-committee. The role of the QA-unit is facilitating Quality Assurance activity in the institution at all levels. At departmental level, the institution should set up a Quality Assurance committee. The head of the department chairs the QA-committee. The role of the QA-committee is to take care of quality assurance at departmental level with support of the central QA-unit. Activities will be: self-assessment of the programs, preparation for accreditation, curriculum design, quality assurance of the programs, staff, student support, facilities etc.

Figure 4: QA at the institution

Task of the QA-officer and the QA-unit A central QA-unit has been installed with the aim to enhance the quality of the institution. Although the responsibility for quality belongs to the academic community as a whole (managements, staff and students), the QA-unit plays a facilitating role. The tasks of a QA-unit concern the following fields:

Promoting quality awareness and a quality culture in technical institution Not all academics (management and staff) are aware of the need of quality assurance. There is resistance to innovations. The QA-unit has the mission to make clear that an institution will not survive without attention to quality. The QA-officer can use chapter 3 of the handbook of the TI’s as basic information for promoting quality awareness.

Governing/AdvisoryBoard

QA-Council• Chair:Headoftheins tu on• Members:deansofthedepartments

• Role:decisionsaboutQA-ma ers

QA-commi ee:• Members:dean+staff+

student

• Role:QualityAssuranceatdepartmentallevelwith

supportofQA-unit

QA-unitHead:QA-officerTask:facilita ngdepartments

AccountabletotheQA-council

Department Department Department

QA-commi ee:• Members:dean+staff+

student

• Role:QualityAssuranceatdepartmentallevelwith

supportofQA-unit

QA-commi ee:• Members:dean+staff+

student

• Role:QualityAssuranceatdepartmentallevelwith

supportofQA-unit

19

Development of a clear quality policy in the institution Although there will be some attention to quality (often on individual basis), a structured quality policy is missing in many cases. It is a task of the QA-unit to support management in the development of a quality policy plan, supported by the academic community. A Quality policy and quality strategy covers all activities the institution will use to improve quality, to enhance quality and to assure the quality of all core activities and the context of the core activities.

Implementation and maintenance of a robust Internal Quality Assurance system In general, an institution will pay special attention to some topics. However, a system with structured and continuous attention to quality assurance is often missing. It is the task of the QA-unit to analyse the current situation and to implement a robust system of IQA. In chapter 5 of this handbook, the QA-officer will find an instrument to promote the implementation.

Supporting general management to find out the quality of the institution and institutional management. While NACTE plans Institutional accreditation, it is important for the institution to prepare itself for the accreditation. However, when there is no external assessment, it is important to know the quality of the institution. Chapter 6 of the handbook for TI’s offers the QA-officer the instrument to organise a self-assessment at institutional level.

Supporting departments in the self assessment process for discovering the quality of the programs A self-assessment at program level is a powerful instrument to improve the quality and to assure the quality. It is an important task of a QA-unit to support the departments in the process of the self-assessment of the programs. Chapter 7 of the handbook for TI’s offers the instrument for the support.

Organizing student evaluations An important instrument for getting feedback on the quality of a program is student evaluation. Because student-evaluation should be compulsory through the whole institution, the QA-unit will play a role in the development of the student questionnaire. In general, there will be one basic questionnaire for all courses, although individual teachers may add additional questions. It will be a task of the QA-unit to process the questionnaires and provide the department with feedback

Organising tracer studies In the framework of an Internal Quality Assurance system, it is important to know where our graduates find a job. Therefore, tracer studies are necessary. It will be a task of the QA-unit to support faculties and/or departments in designing tracer studies.

Staff developments activities There is a strong relation between quality assurance and staff development. Carrying out self-assessments at program level will shows weaknesses. Therefore it is important that the QA-unit develop staff development activities. The QA-unit may offer workshops/training in:

o Formulation of learning outcomes o Curriculum design o Writing examination questions o Working with small groups o Lecturing in big groups o Etc.

Support faculties/departments in designing new curricula. Of course, a QA-unit will never be responsible for a new curriculum. However, the QA-unit may share its experiences with the department. It will be important to ask the QA-unit for advise on the new curriculum.

NACTE plays an important role in quality assurance of the Technical Education by monitor the quality of the TI’s (registration, accreditation and audit and by supporting the TI’s in the development of a quality culture.) Because NACTE embraces the principle of autonomy of the institutions and the principle that quality assurance is in the first instance the responsibility of the institution, it will be

20

clear that NACTE only can be successful with a well functioning QA-unit. The QA-unit plays a double role:

Facilitating QA activities in the institution and to contribute to quality improvement and quality enhancement

To act as contact address for NACTE when NACTE organises an external assessment, might be of the institution as a whole, might be at program level.

The QA-unit will provide NACTE information on the state-of-the-art of quality assurance in the institution. The QA-officer is expected to report yearly about its activities. This can be done through the Chairperson of the Quality Committee, because the QA-director has to report to the Quality Committee too.

2.1.2. The Instrument of Self Assessment A powerful instrument for assuring the quality is the instrument of self-assessment. It is a good way to discover the quality. A critical self-assessment is important because we are sometimes too eager to accept that everything is good: “I have been teaching this way for years and my course has never caused problems. My students have always been content and employers have never complained about the graduates.” This may be true, in general. In an educational organisation, which is a professional organisation, the players always aim to deliver quality. The demand for self-evaluation is not inspired by lack of quality. It means that the quality has to be examined in a structured way, within a well-defined framework. Normally, a self-assessment serves as a preparation for a site visit by external experts organised by NACTE. The self-assessment report (SAR) provides the external experts with basic information. However, a self-assessment has specific value for the institution itself too. It provides an opportunity for discovering quality. In the self assessment, the following questions are important: • Why do we do what we are doing? Do we indeed do the right things? • Do we do the right things in the right way? • Do we have a thorough command of the process to actually realize what we want? • Do we really achieve what we want to? An effective self-assessment is time-consuming. It requires effort by staff and students. Often, it will require an investment of time that has to be taken away from other activities. However, the returns and the profits of a good self-assessment are high. The self-assessment will provide information not known to everyone; Information often exists, but only a small group of people knows it. Sometimes facts will have another dimension when they are connected to one another. Normally, a self assessment will be conducted once in the 5 years, assuming that the self assessment will lead to a quality action plan that every year will be checked on progress. The self- assessment involves co-workers and students in the discussion on the quality of education: the discussion will be raised beyond the level of the individual who is active in the curriculum committee or administration; and the views on quality of individual co-workers and students will be examined together in order to establish a policy for the institution. It shows on which considerations choices need to be made (choices are often made implicitly or postponed) and the information gathered is brought to bear on earlier formulated principles. A decision is reached as to whether a policy should remain unchanged or an explicit choice made. In organizing an effective self-assessment, one has to take into account some basic principles: • Primarily, a self-assessment should never be felt as threatening. A self-assessment should not be

used to assess an individual, should never be used for punishment or reward and should never be used to blame someone.

• A self assessment aims at improvement and enhancement of the quality. • It is necessary to create a broad basis for the self-assessment and to sensitize staff and students.

21

The whole organization has to prepare itself for it. • Looking at quality is more than testing the performance. It also means organizational

development and shaping the institution. Everybody has to be responsible and involved for real self-assessment.

• The management of the institution must fully support the self-assessment. Relevant information is needed for an effective policy and good management. The self-assessment serves to acquire structural insight in performance of the university;

• Conducting a critical self-evaluation demands a good organization5. Primarily someone has to coordinate the self-assessment process. It would be good to designate someone specifically with the self-evaluation project.

The coordinator has to meet some requirements:

In order to obtain the required information, it is important that the coordinator has good entry at all levels of the institution. Therefore, it is very important that the coordinator has good contacts within the institution, with the central management as well as with the faculties and the staff members.

The coordinator must have the authority to make appointments with management and staff about conducting the self-assessment and about the information that is needed.

It is desirable to constitute a substantive team of staff in-charge of the self-assessment.

It is important that the team is structured in such a way that the involvement of all sections is assured. The working group is in charge of the self-assessment, gathering data, analyzing materials and drawing conclusions.

It is assumed that self-assessment is an analysis supported by the whole faculty/department. Therefore, it is important that everyone should be at least acquainted with the contents of the self-assessment report and should recognize it as a document from his or her own institution. The working group may organize a workshop or seminar to discuss the draft SAR.

Not everyone has to agree with all the points in the self-assessment report. There may be disagreement as to what are seen as weaknesses and strengths and what is to be considered as the causes of the weaknesses. Should there be very big differences of opinion between certain groups or bodies, then the SAR should report on it.

The institution determines how self-assessment is carried out. However, it is good to use experiences gained elsewhere. On the basis of experiences with self-assessment in other institutions some suggestions may be made that can facilitate the process (the tentative organization of the process is given in Table 5): • Self-assessment should never be the work of a single person. • Make a group responsible for the self-assessment. • This group should consist of some three to five people, chaired by a coordinator. Students should be involved in the self-assessment. • A clear timetable should be set up, assuming a total amount of time available of about five to

six months between the moment of the formal announcement by NACTE and the actual site visit.

• The topics that have to be considered in the self-evaluation should be distributed among the committee members and each member made responsible for collecting information, and for analyzing and evaluating the data from the self assessment.

• The draft results should be discussed on the largest scale possible. It is not necessary to have consensus concerning the report; it is, however, necessary for as many people as possible to be aware of its contents.

Table 5: The process of organizing self assessment

5Additional interesting suggestions about the organization of the self-assessment can be found in the publication NACTE, Self-evaluation study Guide, June 2010 (page 1-10). Concerning the planning, the content of the SAR and the criteria (standards) to be evaluated see chapter 5, 6 and 7 of the handbook.

22

Date Activity

Eight months before the planned end of the self-assessment

Appoint the leader of the assessment process

Compose the assessment team, including students

The following 6 months Divide the cells to be dealt with

Each person responsible for collecting information and

data, collects that information

Write draft information of the cells

4 months after the start Discussion on the drafts in the group

Second draft

About 5 months after the start Discussion of the second draft with all faculty staff and

students during an open hearing

6 months after the start Edit the comments of the hearing for the final draft

Send the SAR to NACTE

8 months after the start External assessment

The TI’s will conduct the self-assessments against NACTE academic quality standards, using the guidelines in this handbook (chapter 5, 6 and 7).

2.2. Responsibilities of NACTE The responsibility of NACTE can be summarized as monitoring that the graduates from the Technical Institutions fulfil the expectations of the employers and the profession. By formulating the standards of the awards (NTA), NACTE has set the basic condition for the competent, qualified graduate. By monitoring the quality of the institutions NACTE contribute on its way to the quality assurance of Tanzanian Technical Education. Monitoring of the quality of the institution is necessary, because the internal quality assurance, belonging to the responsibility of the institution, need an external check to provide confidence to the stakeholders in the graduates of the Technical Institutions.

2.2.1 . Support of Quality and Quality Assurance by NACTE

NACTE will support the quality and quality assurance of the TI’s in the following way (see figure 5): a) Checking the quality by

i) Institutional accreditation ii) Program accreditation iii) Quality audit iv) Registration

a) Supporting quality by:

i) Keeping up to date the database Job profiles ii) Training seminars and staff development activities (e.g. How to formulate learning

outcomes; curriculum design; formulating assessments) iii) Supporting Quality Assurance officers in implementing IQA systems iv) Supporting institutions in conducting self assessments

b) Advisory functions: i) Advise the government on planning and development of Technical Education ii) Advise Institutions on new developments iii) Clearinghouse function concerning information about Technical Education

23

Figure 5: Quality Monitoring by NACTE

2.2.2 Quality monitoring by NACTE NACTE will monitor the quality of the institution and its programs by organizing external quality assessments. The process of monitoring contains the following sub-processes:

Registration of a new institution or new program;

Compulsory report of the institution about the IQA-system, based on self assessment by the TI, within 18 months after recognition

Institutional accreditation within 2 or 3 years after recognition

Program accreditation While it is the responsibility of the institution to conduct on regular base self-assessments of the IQA-system, the institution and the programs, it is the responsibility of NACTE to organise the external assessments. This might be a Quality Audit, Institutional Accreditation or Program Accreditation. In all cases, external experts are playing an important role. The role of the experts is not an easy one. The expert team has to combine various functions. The team will:

Check the outcomes of the self-assessment

Reflect on the self-assessment

Be engaged in dialogue and discussion with the institution

Act as an auditor/inspector. The expert team is expected to combine two missions:

The team members should listen to the staff of the institution and act as colleagues, using their expertise and experience to offer advice and recommendations.

Q u a l i t y M o n i t o r i n g b y N A C T E

P r o g r a m a c c r e d i t a o n

D a t a b a s e J o b

p r o f i l

e

s

C h e c k i n g q u a l i t y

R e g i s t r a o n

I n s t u o n a l a c c r e d i t a o n

Q u a l i t y A u d i t

T r a i n i n g S e m i n a r s

S t a f f d e v e l o p m e n t

S u p p o r t Q A - o f f i c e r s c o n c e r n i n g

I m p l e m e n t a o n I Q A s y s t e m

S u p p o r n g q u a l i t y

S u p p o r t I n s t u o n s c o n d u c n g S e l f

a s s e s s m e n t

A d v i s e

A d v i s e G o v e r n m e n t

A d v i s e I n s t u o n s

C l e a r i n g h o u s e f u n c o n

24

At the same time, the team has to write a report with its independent verdict on the module of assessment.

In one way the team of experts has to act collegially and in the other way it has to remain independent. It will not always be easy to combine the divergent roles.

In section 5.8, the approach of the Quality audit is described. Section 6.6 contains the process of an institutional accreditation, while section 7.7 contains the process of program accreditation.. Just like other agencies, NACTE has to choose where it will put emphasis. The emphasis will be on institutional accreditation. This will always be a site visit to check the self-assessment report (SAR). The assessment of the IQA system will be done by NACTE, based on the compulsory SAR of the institution. Programme accreditation will have different modalities too (see chapter 7). When a site visit is planned, NACTE has to appoint a team of external experts. Those teams will differ for the Institutional accreditation, program accreditation and a quality audit. However some aspects are important to take care of:

Selection and appointment NACTE will appoint a team of experts. The experts to be nominated must be of high esteem.

Independence and confidentiality It is important that the expert team and each member individually, act independently and without any conflict of interest. If a member has any connection with the institution to be audited, he or she cannot be appointed. To assure independence and eliminate conflict of interest, all members of the team will sign the declaration of independence (appendix 4. The members are also bound to confidentiality about everything they will hear or read about the institution under assessment. The Self-Assessment Report (SAR) and all interviews are confidential.

Training of the experts team Assessing quality is a specific skill. Normally, experts in a team are specialists in institutional management or in a discipline and do not have much experience in evaluation or quality assessment. Therefore, the experts must be trained beforehand. Therefore, NACTE will organise on a regular base, training of the experts. All members should have knowledge of the basic ideas of quality and quality assurance; they all need to be aware of the dos and don'ts. The basic elements of the training are:

What is quality and how can quality be measured?;

How to use the quality model;

How to cope with criteria and standards set by NACTE?;

How to read the self-assessment report;

How to organise the interviews;

How to behave during the assessment; and

How to write the external assessment report.

The secretary Because all external assessments should be done in equivalent /similar way, it is important that NACTE provides the secretary of each expert team. In addition to his or her duties as secretary, described below, he or she also acts as project leader during the assessment. The secretary:

Checks the SAR for completeness and compliance with the requirements made on it;

Maintains contact with the institution about the planning of the external assessment

Makes preparations for the team’s visit;

Files the documents referring to the assessment process.

Once the team has been installed, the secretary has the following three-fold task:

25

To monitor the team’s working procedure and compliance with the assessment protocol. The secretary is the connecting link between NACTE and the team. His or her primary responsibility is to monitor the assessment process. Is the expert team following the guidelines laid down for it? Is it maintaining its independence? Are agreed procedures followed? To support the team with specific expertise The secretary supports the team in the fulfilment of its duties. As the chairman’s right-hand person, he or she plays an active role in drafting the assessment report. Although not formally a member of the team, the secretary does contribute specific skills in the fields of quality assessment and policy development in tertiary education.

To archive the audit trail. The secretary is responsible for keeping the documents relating to the assessment at least until the end of the assessment procedure.

26

Chapter 3

What is Quality? Some Theory and Background Information

3. 1 What is Quality? The word quality is already used several times without explanation what quality is. However, everybody who thinks about quality and quality assurance is faced with the question: "What is quality?" In the time that the call for quality became louder and louder, many discussions on quality started with a quote from the book Zen and the Art of Motorcycle Maintenance:

"Quality...you know what it is, yet you don't know what it is. But that's self-contradictory. But some things are better than others, that is, they have more quality. But when you try to say what the quality is, apart from the things that have it, it all goes poof! There's nothing to talk about. But if you can't say what Quality is, how do we know what it is, or how do you know that it even exists? If no one knows what it is, then for all practical purposes it doesn't exist at all. But for practical purposes it really does exist. What else are the grades based on? Why else would people pay fortunes for some things and throw others in the trash pile? Obviously some things are better than others... but what's the 'betterness'? So round and round you go, spinning mental wheels and nowhere finding any place to get traction. What the hell is Quality? What is it?" (Pirsig, 1974)6

In spite of these reflections by Pirsig, many books and articles have been written trying to discover the nature of quality. But quality is like love. Everybody talks about it and everybody knows what he/she is talking about. Everybody knows and feels when there is love. Everybody recognises it. But when we try to give a definition of love we are left standing empty-handed.

The quote from Pirsig shows how desperately the writer is thinking about quality and reveals the problem that relates to quality: There is no general consensus on the concept of quality. A comprehensive, by everyone endorsed definition of quality does not exist, because quality is, just like beauty, in the eyes of the beholder. Whoever asks whether something has quality has already a certain concept in mind and certain expectations. When we talk about the quality of a product or the quality of a service, the definition often used is the satisfaction of the client. The client has certain expectations about the product or service and wants “value for money".

While quality, in general, is already a difficult concept in itself, quality in higher education is much more confusing, because it is not always clear what the "product" and who the "client" is. Is the "graduate" the "product" that we offer society and the labour market? Or is the graduate-to-be, the student, our "client" and the program that we offer the "product"? Technical Institution has a multiple product system and a multi-client system. In the discussion on quality in higher education Green (1994)7 is often quoted. In this article he describes the different views on quality:

Quality as excellence

Quality as fitness for purpose

Quality as threshold

Quality as added value

Quality as value for money

Quality as satisfaction of the client

6Pirsig,Zen and the Art of Motorcycle Maintenance, 1974. 7Green, D (1994), What is quality in Higher Education? Concepts, Policy and Practice. In: Green (ed)(1994)What is Quality in Higher Education? London: SRHE/Open University Press

27

Quality as excellence. In this concept, the emphasis is on high-level standards. Being the best, being excellent. We may say that something has quality and something has more quality. People talking about promoting quality frequently mean promoting excellence. However, quality is not the same as being excellent. Of course, everybody likes to do his/her best to deliver quality, but not every institution can be a Yale or MIT universities. A country with only excellent institutions does not exist. An institution can choose not to aim for excellence, because it likes to educate a broad range of graduates and not only the brightest ones. A typical regional institution with a mission to develop its country will choose differently to an institution like Berkley.

Quality as fitness for purpose. With this concept of quality, the basic question is if the institution is able to achieve its formulated goals. It concerns the quality of the processes. This quality concept is improvement oriented. But, will this quality approach assure achievement of the threshold quality because goals and aims are not the issue? An institution might have set its goals too low, through which it can easily achieve them. This means that we not only have to discuss the fitness for purpose, but also the fitness of purpose.

Quality as a threshold. In this view, quality is seen as meeting threshold requirements. This quality concept often forms the basis for accreditation decisions. The problem is that it is not always clear what basic quality is. Setting threshold standards might also hinder innovations. Compliance with the threshold standards does not stimulate innovations.

Quality as added value. This concept emphasises what happens to the students. Education is about doing something to the student. Quality means the value added to the student during education and training. It is the method of formulating learning outcomes and realising the outcomes in the graduates. The basic quality question is: "What has he/she learnt?

Quality as value for money. This quality concept has its focus on efficiency. It measures outputs against inputs. It is often a concept supported by governments. The concept is connected with accountability.

Satisfaction of the client. With the rise of the concept of the "student as a consumer", quality is described as: "something has quality when it meets the expectations of the consumer; quality is the satisfaction of the client". Figure 6: different views on quality

3.2 The Stakeholders and Quality

Quality assurance in Technical Education is much more complicated than quality assurance in industry, because there are so many players in the field. Technical Education has many stakeholders

QUALITY

Fitness for purpose.

The view of external

assessors aiming for

improvement

Excellence. The

view of academia

Threshold. The

view of accreditors Added value. The

view of students

Value for money.

The taxpayer’s

view and

government’s

view

Client satisfaction. The

view of students and

employers

28

and each stakeholder has its own ideas about quality. The following stakeholders can be distinguished:

The government or the state

The employers

The academic world

The students

Parents

Society at large The view on quality differs from stakeholder to stakeholder:

When the government considers quality, it will look first at the pass/fail ratio, the dropouts and enrolment time. Quality in the eyes of governments can be described thus: "As many students as possible finishing the program within the scheduled time with an international level degree at reduced costs."

Employers talking about quality will refer to the knowledge, skills and attitudes obtained during the studies: the "product" that is tested is the graduate.

Quality of education has a different meaning in the eyes of the students and their parents. For them, quality is connected with the contribution to their individual development and preparation for a position in society. Education must link up with the personal interests of the student. But the educational process also has to be organised in such a way that students can finish their studies in the given time.

An academician will define quality as "A good academic training based on good knowledge transfer and a good learning environment and a good relationship between teaching and research."

We must conclude that quality is a very complex concept. We cannot speak of "The Quality"; we have to speak about "qualities". We have to distinguish between quality requirements set by the different stakeholders: by the student, by the academic world, by the labour market (employers), by society, and by the government. Each stakeholder will appreciate different aspects of quality, as can bee seen in Table 6. Table 6: Stakeholder appreciation of the quality aspects Stakeholders Students Employers Government HEI Staff

Aspect of quality

Input (for example):

student intake * * *

selection * * *

budget * *

academic staff * *

Process (for example):

aims/goals * (*) * *

educational process * * *

educational organisation * * *

Content

*

*

*

advice * *

Output (for example):

pass/fail rate *

*

*

the graduate * * * * *

Quality is not a simple one-dimensional notion. Quality is multi-dimensional. So there is quality of input, process quality and quality of output. All these dimensions have to be taken into account when discussing quality and judging quality. The different views on quality and multi-dimensional notion of quality mean that it is a waste of time to try to define quality precisely. Context free, objective quality does not exist. However, if we seriously try to assure our quality, we have to agree on a workable concept of quality. Taking into account that each player has his or her own ideas about quality, we can

29

agree that we should try to find a concept of quality that fits most of the ideas and that covers most of the expectations of the stakeholders.

With so many stakeholders and players in the field, we may say "Quality is a matter of satisfying the stakeholders in an adequate way". In this process, each stakeholder needs to formulate, as clearly as possible, his/her requirements. The institution or department, as ultimate supplier, must try to reconcile all these different wishes and requirements. Sometimes the expectations will run parallel, but they can just as well end up in conflict. As far as possible, the requirements of all stakeholders should be translated into the mission and goals of an institution and into the objectives of a faculty and of the educational program and as far as this concerns research, the research programs. The challenge is to achieve the goals and objectives. If this is the case, then we can say that the institution, the faculty has "quality" (see Figure 7). Although it remains necessary to strive for a good description of the different requirements and aspects of quality, the lack of a definition may never be an excuse for not paying attention to quality or for not working for quality enhancement.

Figure 7: From requirements of stakeholders to quality

A definition of quality that satisfies everybody does not exist. However, for the sake of mutual understanding, the following description of quality might be adopted:

Quality is achieving the formulated goals and aims in an efficient and effective way (fitness for purpose), assuming that the goals and aims reflect the requirements of all our stakeholders in an adequate way (fitness of purpose).

Talking about quality, we must take into account the following remarks:

Quality is not always the same as efficiency! The discussion on quality assessment is often connected with the concept of "efficiency" (saving money, making more rational use of public resources). In assessing quality, an important question will be: "Do we achieve the required level of quality at acceptable cost?" An efficiency-oriented approach as such is a good starting point, but the problem is that efficiency is not always defined as "at acceptable cost", but often as "at minimal cost", and this may threaten quality. It may be very efficient to have lectures for a thousand students, but it is not effective. It may be considered efficient to have a very structured degree program with student assessments every four weeks, forcing students to work and to keep up with the program. However, does this method lead to the creation of an independent, and critically thinking graduate? It may be considered efficient to use only multiple-choice questions for student assessment, but does it enhance verbal and written communication skills?

Stakeholders and Quality

30

Quality is context bound When striving for quality, the main question is: "Do we offer the stakeholders what we promise to offer." This means that starting point for assessing the quality will be our promises (i.e. goals) and that the verdict “quality or no quality” will be based on the promises. Therefore, we have to look at our quality in the given context. McDonald's, for example, will strive for quality, and when we eat a fast food meal, we will probably get quality. However, this is not the same quality, as we will get when we have dinner in a restaurant with one or two stars in the Guide Rouge of the Best Restaurants. So, we cannot assess the quality of McDonald's with the same standards as those used to assess a star restaurant. This also means that we may never assess a regional Technical Institution, e.g. in South America with the same standards that we apply to more sophisticated institutions like MIT, Berkley or the TH Zurich. For an institution claiming excellence, other standards count than for an institution striving at contributing to the development of the country and the region. We cannot assess the quality of the University of the Amazon as with the standards applied to Berkley. Each level of quality has its price. The only common feature is that we may ask: "Will we get what we expect?"

Quality is context bound that is true. Nevertheless, most institutions also like to play a role on the international stage and strives to send its graduates abroad. This means that an institution has to meet at least the basic standards that are applied to Technical Institutions in general. There is at least a bottom line for the threshold quality, although it is not clear what that bottom line is. This is something that the international community has to decide.

Having accepted a workable definition of quality, there is another hot topic: how do I asses the quality? How to measure quality? What are the criteria measuring quality? What are the standards against which quality is assessed? Talking about quality assessment, it is important to make a clear distinction between criteria and standards.

A criterion can be defined as a specific aspect taken into consideration to make a judgement about quality. For example, the criterion (the aspect) we will look for might be formulation of expected learning outcomes or computer facilities.

Standard is the level that a criterion must reach. Are there learning outcomes formulated and if so, is this done in an adequate way or more than adequate? Sometimes a criterion might be quantified: talking about computer facilities, the standard might be 1 computer per 5 students.

Criteria are valid in all circumstances and in all places. In fact all accrediting bodies, all over the world are looking at more or less the same aspects/criteria assessing the quality of a program, e.g.:

• Goals and objectives; expected learning outcomes • Program content • Program specification or description • Program organisation • Didactic concept/teaching/learning strategy • Student assessment • Staff quality • Quality of the support staff • Student profile • Student advice/support • Facilities & infrastructure • Student evaluation • Curriculum design & evaluation • Staff development activities • Benchmarking • Achievements /graduates

31

• Satisfaction stakeholders

To know if a criterion is adequate/satisfactory, fulfilled, depends on the standard set for that criterion. Standards are context bound and may differ from situation to situation. As said, the standards to be applied to Berkley will be different from the standards applied to the University of the Amazonas.

3.3 The use of performance indicators and quantitative data As already been stated, with a simplification of quality assessment we may say: "define quality and look for a set of performance indicators to measure the quality." Nowadays, we are living in an evaluative society which very much likes to measure everything: The performance of the public health system (number of patients treated, length of the waiting list for surgery), the performance of the police officer (number of fines, number of solved cases) and, of course, the performance of Technical Education (number of graduates, pass rates, average time spent in the institution). Managers (politicians in particular) are fond of such quantitative performance indicators (PI). They look for more and more hard, statistical data, because this is considered to be more objective. But the question is whether there is a real link between so-called performance indicators and quality. Opinions are differing. It is evident that differences of interpretation arise, when people try to assess quality directly from quantitative data. Consider, for example, the measurement of the quality of research. Is the total number of publications a true measure of quality? The analysis of information and experience gained elsewhere indicates that this is not always the case. Such performance indicators, like the number of articles, reveal the danger of using performance indicators. Once set, the indicator will be corrupted as soon as possible. Instead of publishing one good article, we see now that the article is split into several articles, because each counts for the record. Another example from the field of education: the interpretation of success rates. One faculty has a pass rate of 80%, another one has a rate of 60%. But does the figure tell us anything about the performance of the faculty? Does it tell us anything about the quality of education? Is the performance of institution Y with a pass rate of 80% superior to the achievement of institution X with a rate of 60%? Or has institution lowered its level? Or is institution X more selective in the first year? There is a considerable amount of literature on performance indicators (PIs). A striking factor in the discussion on these is that there are two opposing parties. It is mostly governments who lay a strong emphasis on the importance of using performance indicators: they are optimistic about the possibility of determining the right indicators. Technical Education Institutions, on the other hand, are generally very reserved and sceptical about them. Many governments are trying to formulate performance indi-cators that would be useful in quality assessment, but so far without any success. The following reasons can be put forward:

The term "performance indicator' is very confusing, despite many attempts to explain the meaning and functions of performance indicators. The problem is that a performance indicator does not always relate directly to the performance of an institution, but should rather be considered as statistical data. For example, one of the PIs used in the student population is the male-female ratio. However, this indicator says nothing about the quality of an institution. It is more a government indicator that shows how far the objective of "gender balance" has been achieved.

People attach different functions to performance indicators. Without using the term 'PI', it will be clear that the use of certain data (enrolment figures, student numbers, number of graduates, unemployment figures) is important. These data play an important role in monitoring and evaluation. Governments, on the other hand, often look to performance indicators as instruments for governing financial allocation. In fact, we can see attempts to establish a system of performance-related funding in several countries.

Transforming indicators into standards. Looking at the various functions attributed to performance indicators, it is not unreasonable to fear that indicators will be transformed into standards. The success rates may be an indicator of achieving the goal of "enabling as many students as possible to graduate". A pass rate of 70% would appear to be more successful than a pass rate of 60%. But the figure says nothing about the quality of teaching. However, there may be a tendency to specify that a success rate of at least 70% should be achieved.

32

The conclusion should be that all attempts and all discussions so far have failed to produce a generally accepted set of performance indicators. Putting emphasis on the quantitative performance indicators, we run the risk to enforce reality by over-simplifying quality. Quality in Technical Education is more than a collection of figures and data. Although qualitative aspects are often more difficult to assess, we should not flee into quantitative aspects with fake objectivity. We have to learn to live with the idea that the judgement of quality in Technical Education is not an objective activity, but rather an activity with a human factor. Is there a role for Performance indicators (quantitative aspects) in quality assessment? What is the value of performance indicators as opposed to peer review or in combination with peer review? Looking at the sets of performance indicators that are often used, we see that quantitative indicators are often basic data, but are immediately decorated with the notion of a "PI": They cover numbers of students, numbers of staff, dropout rates, student-staff ratios, etc. When these data are used properly, a performance indicator will raise questions but never give answers. The so-called qualitative performance indicators may be seen as elements that have an influence on quality aspects to be taken into account when looking at quality. The question is whether we can (or will) rely more on performance indicators than on the subjective judgements by peers. The role of performance indicators in quality assessment is a limited one, as can be illustrated by the following example: I have a bottle of wine and want to assess its quality. Which aspects are important? First, I have to decide on which aspects I will assess the wine: acidity, tannin, alcohol percentage, and sediment. Of course, I can measure the wine on these aspects, but still I do not know whether the wine is good or not. Someone has to tell which figure is good and which not. But there are other aspects more important for the assessment: taste and smell. These aspects are not quantifiable. We need a panel that can judge the taste and smell as fine or not. (Vroeijenstijn 19958); see Figure 8).

Figure 8: Relation between performance indicators and expert assessment

Working with performance indicators seems so attractive because it looks like they might provide a clear picture of strengths and weaknesses. But these analyses must be handled very carefully and must be complemented with other information. PIs play a role in quality assessment, but only a minor one. Performance indicators as the set of quantitative indicators play a role in supporting the opinion of experts. But these performance indicators can never have the last say or take the place of

8Vroeijenstijn, A.I (1995), Improvement and Accountability: Navigating between Scylla and Charybdis, London, Jessica Kingsley Publishers

% alcohol

Acidity

tannin

sediment

performance

indicatorssmell

taste

expert team

33

expert/peer review. The opinion of the experts must be based on facts and figures, but can never be replaced by performance indicators. Performance indicators should be used not as an end in themselves to draw definitive conclusions, but to trigger areas of concern and provide a catalyst for further investigation. It will be clear that performance indicators can never speak for themselves, but must be interpreted by experts. Where they seem to be objective, they are not really performance indicators, but only statistical data or management information. Just as is the case for the concept of "quality", it is also a waste of time searching for the philosopher's stone: a set of performance indicators to measure quality.

34

Chapter 4

Quality Monitoring by NACTE

4.1 Registration

Section 5 (1) (a) of the National Council for Technical Education Act, 1997 empowers the Council to register technical institutions capable of delivering courses. Registration is mandatory [Registration Regulations (Government Notice No. 279 published on 26/10/2001). At the moment of writing NACTE has registered 600 Technical Institutions. Registration of an institution means license the institution to admit students and to offer programs, leading to one of the level awards of the framework of Technical and Vocational Education and Training (TVET) qualifications. Registration (Licensing) is a process in which NACTE satisfies itself that an institution has been legally established and is viable for conducting training programmes sustainably. Registration can be:

Registration of a new institution (section 4.1.1)

Registration of a new program (section 4.1.2)

4.1.1. Registration of a new institution An institution, looking for registration, sends a letter to NACTE expressing interest to establish a technical institution. To be registered and get a license, the institution will provide NACTE with the following information: 1) The statement of the vision, mission, the objectives and strategic and academic plans to achieve

its goals;

2) Intended academic programmes with curricula and rationale to the labour market and

competition with other local providers;

3) Administrative and academic regulations;

4) Academic and support staff information, contracts and CVs;

5) Student admission requirements and 5-year enrolment projections;

6) Information regarding physical infrastructure, financial and learning resources;

7) Legal documents related to occupancy;

8) Institutional governance and organization structure

9) Institutional layout map showing its location and size;

10) Fee structure and rationale.

The registration process

NACTE verify the document(s) submitted by applicant institution.

NACTE may appoint experts from the relevant occupational field as verification team for physical verification of the applicant institution. NACTE may also send a NACTE official to check the physical conditions.

The verification team/NACTE official submits a report to the relevant module board for recommending the registration stage;

The module board submits a report to the Council for decision;

NACTE informs the applicant institution on the decision of the Council by official letter The decision might be:

Provisional Registration: Granted to institutions that either have deficiencies that they can redress in a short duration before they are considered for full registration or have just started to admit students, and have enough resources for full or considerable part of the programme.

Full Registration: Granted to institutions that are fully operational and have acquired enough

35

experience, human, physical and financial resources to ensure sustainability for running its programmes.

The registration will be valid for 3 years. Within those 3 years, the institutions have to be accredited. Depending on the lengths of the programs at offer (1 year programs or 2 year programs, the accreditation will take place 2 years after registration or 3 years after graduation. To gain some insight into the achievement of the on paper promised quality (registration) the institution will undergo a quality audit within 18 months after registration. The Quality Audit will be based on a self-evaluation according to Chapter 5 of the handbook.

4.1.2 Registration new programs

An institution, that has been registered, but not yet accredited, has to ask registration of a new program. To be registered and get a license to offer the program, the institution will provide NACTE with the following information: 1. Name of the program

2. Competence level descriptor in the NQF

3. Rationale to start the new program

4. 5-year enrolment projections;

5. Student admission requirements

6. The resources to run the program

7. The job profile, the program is based on

8. The expected learning outcomes

9. The content of the program

10. The staff offering the program

11. The facilities

The registration process

NACTE verify the document submitted by applicant institution.

NACTE may appoint experts from the relevant occupational field as verification team for physical verification of the applicant institution. NACTE may also send a NACTE official to check the physical conditions)

The verification team/ NACTE consultant submits a report to the relevant module board with advise about registration.

The module board submits an advice to the Council for decision;

NACTE informs the applicant institution on the decision of the Council by official letter The decision might be registration or non-registration. Registration means: license to offer the program. It might be expected that the program will deliver qualified graduates. The registration of the program is valid for 3 years. The license to offer the program will be withdrawn, if the institution is not accredited within those 3 years. The process under 4.1.2 is also applicable when an accredited institution likes to start a new program outside the module areas already covered by the accreditation.

4.2 The Quality Audit A newly registered Institution has to conduct a self-assessment of the Internal Quality Assurance system within 18 months. The NACTE still organise a quality audit according to the guidelines in chapter 6.

36

It might also be possible that NACTE will organise a Quality Audit on request of an institution. This will done according to the same guidelines for the self assessment by the institution (chapter 6 handbook TI’s) and Quality audit by NACTE (Chapter 6 handbook NACTE)

4.3 Institutional accreditation NACTE has the task to accredit Technical Institutions (section 5(1) (a) of the National Council for Technical Education Act, 1997. Accreditation means that NACTE will check if the institution indeed is offering quality based on past performance and provide the institution with a quality label. Because accreditation is based on past performance it is necessary for the institution to have at least one cohort of graduates. Institutional Accreditation will take place within 2/ 3 years after registration, depending on the length of programs at offer. When NACTE conducts an institutional accreditation, the institution will prepare itself by a Self-assessment of the institution according to chapter 6 of the handbook. The assessment of the quality assurance system is included in the Institutional accreditation. If a TI offers three or fewer programs, the programs will be involved in the Institutional assessment. The programs concerned conduct at the same time a self-assessment according to chapter 7 of the handbook. The assessment team must include expertise in the field of the programs. Provides a TI more than three programs, then all the programs conduct a self-assessment, but for assessment during the site visit only two will be selected.

4.4 Program Accreditation

NACTE has the task to validate and approve the programs of an institution. An instrument for

assessing the quality of the programs is program accreditation. The input for the Program

Accreditation is the self-assessment report (SAR) of the department and the program involved

(chapter 7 of the handbook).

As far as possible, NACTE will include the program assessment in the Institutional assessment. When

the Institution has 3 or fewer programs at offer, all programs will be assessed. If more than 3

programs are offered, all programs will conduct a self-assessment according the guidelines in

chapter 7 of the handbook. The expert team will make a selection of 2 programs

For a separate program assessment with site visit, NACTE will use the following

modalities:

1 NACTE has some doubts about the program(s) in an institution. NACTE will verify the quality by an individual Program Assessment.

2 NACTE takes the initiative and announces a nationwide Program Accreditation for a certain type

of programs (e.g. accountancy or nursing). An advantage of the nationwide approach is that there is a possibility of benchmarking of the programs in the different institutions. This comparative character of the Program Accreditation makes it possible to have a good overview of the state-of-the-art in that discipline/module. The programmes involved are expected to conduct a self-assessment. An expert team, installed by NACTE, will assess the quality based on the SAR. If there is any doubt, NACTE will organize a restricted site visit

3 The institution always can ask NACTE to organise a program accreditation

37

Chapter 5

Internal Quality Assurance and the Quality Audit

A well functioning Internal Quality Assurance system is seen as a basic condition for institutional accreditation. The QA-officer and the QA Unit play an important role in implementing a quality assurance system in the institution. One of the first activities of the QA Unit is to explore what is already done in the field of quality assurance both at institutional level and at staff level. What elements are already in place? What should be strengthened? In this chapter the QA Officer will find guidelines for the implementations of a quality assurance system. The following activities are expected:

Use the model in figure 5 to analyse the state-of-the-art of quality assurance in the institution

Formulate the strengths and weaknesses

Develop a strategic plan to implement a robust Internal Quality Assurance system An institution, that aspires to be registered, has to be accredited first. Registered be registered an institution has to submit to NACTE a Self-assessment report of its Internal Quality Assurance system. NACTE will assess the IQA-system based on the SAR and makes recommendations for improvement.

In the Guidelines for Preparation of Quality Management Plan for Institutions Accredited by NACTE, June 2004 is stated: All institutions under the auspices of NACTE are required to have quality control systems in place to ensure that respective institutions meet the accreditation requirements. This requires availability of clear institutional policies and procedures on quality control and quality assurance. The key purpose is to guarantee quality of outputs from technical institutions and win confidence of stakeholders in the technical education provided. Quality control involves operational techniques and activities that are aimed both at monitoring process(es) and at eliminating causes of unsatisfactory performance in all stages of the quality loop. The ultimate goal is to achieve economic or desired effectiveness.(page iii). In this chapter, the outline of a robust IQA system commonly to be used in the institutions will be discussed. It will be based on general accepted criteria. The criteria and standards from NACTE are included. 5.1 What do we mean by Quality Assurance? Nowadays, so much attention is paid to quality so that people might think that quality is an invention of the last decades. But of course this is not true. Attention to quality is not new; it has always been part of the academic tradition. It is the outside world that now emphasizes the need for explicit attention to quality. Several reasons can be given for Quality Assurance:

All Technical Institutions want to train graduates who meet the needs of society. We all like to deliver a "product" that is wanted. We all like to be proud of our graduates.

The labour market expects institution’s to provide the students with adequate knowledge, skills and attitude, important for the right job fulfilment (Competence based learning).

Internationalisation of the profession and a world that is becoming a small village brings us greater competition than before. The graduates of the institutions not only have to compete inside the country, but also with other countries. Globalisation not only has negative aspects, but also positive ones. It offers our graduates the opportunity to enter the world market, but under the condition that the qualifications they posses have the required quality.

38

There is need for "consumer protection": our students and their parents are spending a lot of time and money on their education. Therefore, they have the right to receive a quality education,

The relationship between Technical Education and society has changed. Society has become increasingly interested in Technical Education. Also the relationship between Technical Education and the labour market has become a topic for discussion.

Quality has become increasingly important for Technical Education institutions, because of the question as to whether it is still possible to deliver the same quality within the given frameworks.

One can talk of a 'quality gap': on the one hand, governments are striving to increase the numbers of students enrolled (Technical Education for as many as possible); on the other hand, we see a continuous decrease in investments. Technical Education institutions have to do more with less money. But at the same time quality is expected to be maintained or to improve.

Student exchange and international cooperation require insight into quality. There has always been exchange of students between countries, but with the world becoming a global village; it has become increasingly clear that it is very important to know about the quality at the other institutions. Questions have to be asked, such as: 'Can I recognise the program?' or 'Is it good enough?'

5.2 Towards an Internal Quality Assurance (IQA) System

If we like to assure our quality, it is necessary to establish a structured quality assurance system that makes it possible to monitor our quality, to improve the quality and to evaluate our quality. There is no single approach or system that is applicable to all institutions. Each institution has to build its own system. However, when developing an IQA system, there are some basic conditions that have to be taken into account. These are as follows:

It should be kept as simple as possible;

It should not be a bureaucratic process;

It should have the support of management and staff;

There must be a right balance between a centralised and decentralised approach;

It should use effective instruments; and

The internal quality assurance system must be tuned to national and international developments.

An Internal Quality Assurance system (IQA system) is a system aiming at setting up, maintaining and improving the quality and standards of teaching and learning and service to community. The overall objective is to continuously promote and improve the quality of the programmes, their mode of delivery, and their support facilities, etc.

There is no Internal Quality Assurance system that fits all institutions. However, Internal Quality assurance has a pivotal position in the framework of accreditation and therefore, in some cases there are requirements formulated for an IQA-system, like what is done by the European Association for Quality Assurance (ENQA) (ENQA, 2005, revised in 20149). The requirements can be summarized as follows: 1. Policy and Procedures for IQA An Institution has a clear policy and associated procedures for the assurance of the quality and standards of their programmes and awards. The institution commits itself explicitly to the development of quality culture and quality awareness. To achieve this, the institution develops and implements a strategy for the continuous enhancement of quality. The strategy, policy and

9

Standards and guidelines for quality assurance in the European Higher Education Area (DRAFT endorsed by the Bologna

Follow-Up Group on 19 September 2014. Module to approval by the Ministerial Conference in Yerevan, 14-15 May 2015), September 2014

39

procedures should have a formal status and be publicly available. They also include a role for students and other stakeholders.

2. A Monitoring and Evaluation System An Institution has a structured monitoring and evaluation system to collect information about the quality of its activities. At least the monitoring and evaluation system should include: - Students evaluations - Students progress system - Structural feedback from the labour market - Structural feedback from the alumni

3. Periodic review of the core activities (training, research and community services) An institution has formal mechanisms for periodic review or evaluation of the core activities: The programmes and awards, the research activities (if applicable) and community service.

4. Quality Assurance of the student assessment An institution has clear procedures to assure the assessment of students. Students are assessed using published criteria, regulations and procedures, which are applied consistently. There are clear procedures to assure the quality of the examinations

5. Quality Assurance of teaching staff An institution has ways of satisfying itself that staff is qualified and competent to conduct the core activities of the institution: training, research and community services.

6. Quality Assurance of facilities An institution should have clear procedures to ensure that the quality of the facilities needed for student learning are adequate and appropriate for each program offered.

7. Quality Assurance of the student support An institution has clear procedures to assure the quality of the student support services and student counselling.

8. Self Assessment An institution conducts regularly, but at least every 5 years a self-assessment of its core activities and of the institution as a whole to learn about the strengths and weakness. This self-assessment will lead to a quality plan.

9. Internal Audit A self-assessment might be part of the external quality assessment/accreditation process and the self-assessment report as an input for the external review team. If the self-assessment is not connected to the EQA, the institution is expected to organise an audit, based on the self-evaluation report.

10. Information Systems An institution should ensure that it collects, analyzes and uses relevant information for the effective management of their core activities.

11. Public Information An institution should regularly publish up to date, impartial and objective information, both quantitative and qualitative, about the programmes and awards that it is offering.

40

12. A Quality Handbook An institution should have a QA handbook, where all regulations, processes and procedures concerning Quality Assurance are documented. All people concerned (i.e. stakeholders) should publicly know the existence and contents of this handbook.

According to NACTE, the minimum requirements for Internal Quality Assurance system are:

1. The establishment of a quality assurance committee

2. Preparation and adaptation of an institutional quality assurance policy

3. Planning for quality assurance policy implementation

4. Execution and evaluation of this implementation10 In Figure 9 the requirements as set among others by ENQAA, INQAAHE and NACTE are visualized in the model for an Internal Quality Assurance system. The model contains all the elements of an Internal Quality Assurance system, which are:

The monitoring instruments;

The evaluation instruments,

The QA processes for specific activities

Specific QA instruments.

The institution can use the model to see:

What elements of IQA are already in place and what should still be done?

How far it achieves the criteria set by NACTE

10See Timothy Manyaga(2008) , “Standards to assure quality in tertiary education: the case of Tanzania”, Quality Assurance in Education, Vol. 16 Iss2 pp 164-180

41

Figure 9: Model for an IQA System

5.3 Self-assessment of the IQA System at the Institution

An efficient Internal Quality Assurance (IQA) system is necessary to assure quality. To learn about the quality of the IQA system and to see how far the institution is meeting the NACTE criteria, the QA unit will organise a self-assessment of the quality assurance approach in the institution. This section shows how to conduct such self-assessment. The guidelines given for the self-assessment of the IQA system should not be seen as a straight jacket, because the IQA system may differ from institution to institution. Therefore, it should be seen as a benchmark: to what extent:

do we reach the NACTE standards?

do we reach internationally accepted standards for IQA? If we do not reach the NACTE standards or the internationally accepted standards then we should ask ourselves why not and what can we do to change it? The model in figure 9 will be used for the critical evaluation. The following aspects will be treated:

Internal Quality Assurance: general aspects

Monitoring instruments

Evaluation instruments

QA procedures to safeguard specific activities

Specific QA instruments For each aspect, the following format is used:

The name of the cell in the model is given, just as the title of the aspect that will be treated, e.g. Internal Quality Assurance, general aspects.

QA-0rganisa onandprocedures1.QA-managementplan2.QA-commi ee3.QA-center

20.Followupac vi es

Monitoringinstruments

Evalua oninstruments

SpecialQAprocesses

SpecificQAinstruments

8Student

evalua on

9Course/curriculum

evalua on

12

AssuranceQualitystudent

assessment

13AssuranceQualitystaff

6FeedbackLabour

market&

Alumni

7Research

Performance

11CommunityService

evalua on

10Researchevalua on

14

AssuranceQuality

Facili es&infrastructure

15AssuranceQuality

Studentsupport

19Qualityhandbook

18Informa onsystems

17Inter-collegialaudit

orPeerreview

16SWOT-analysisSelfassessment

4StudentProgress

5Passrates

Dropoutrates

42

The criterion concerning the aspect is given in a box. The criterion shows what commonly is expected from an institution. When NACTE has formulated a criterion, different from the commonly accepted criteria, the NACTE criterion is added.

Looking for evidence A set of questions and statements is drawn up to help the QA unit to find evidence if the criteria are being met. Please be aware of the following as far as these questions are concerned:

The questions set are not meant as a compulsory list that has to be completed. It is not a questionnaire to be answered point by point. It must be seen as a tool to collect information and evidence. The questions are to be seen as reminders.

The model and the questions have been developed for general use. This means that the list has to be adapted to the institutions own situation and to its specific identity.

Section 5.4 will lead the institution through all criteria that have to be considered.

5.4 Conducting the Self-assessment

Go, topic by topic and:

Give a description of the situation at the moment

analyse the situation (What do we think about it? Are we satisfied with the situation?)

If not, describe how you think the situation can be changed and improved.

What evidence do you have that the formulated criterion (criteria) has (have) been met? (Documentation, effects, outcomes)

QA ORGANISATION (CELL 1-3) It is accepted worldwide that an institution has a clear policy and associated procedures for the assurance of the quality and standards of its programmes and awards. The institution commits itself explicitly to the development of quality culture and quality awareness. To achieve this, the institution develops and implements a strategy for the continuous enhancement of quality. The strategy, policy and procedures should have a formal status and be publicly available. They also include a role for students and other stakeholders. A clearly formulated policy and clearly formulated procedures for quality assurance provide a framework for developing and monitoring the effectiveness of the quality assurance system. They also help to generate public confidence in institutional autonomy. The formulated policy contains the statements of intent and the principal means by which these will be achieved. 1. Quality Management Plan

Criterion

The institution has a quality management plan containing clear and specific policies and procedures that provide a framework for the quality assurance activities. The institution should expressively and clearly commit to disseminating the culture of quality among all its members.

Looking for evidence

Does the institution have a quality management plan? It includes strategies, policies and procedures for continuous improvement of performance. The quality management plan, covering strategies, policies and procedures for continuous improvement of performance should be approved by the management at the institution and then announced to all stakeholders.

The quality management plan includes the following: o The vision, mission and quality policy of the institution;

43

o The specific roles, authorities, and responsibilities of management and staff of the institution with respect to QA activities;

o The means by which effective communications with personnel actually performing the work are assured;

o The processes used to plan, implement, and assess the work performed; o The process by which measures of effectiveness for QA activities will be established and

how frequently effectiveness will be measured; and o The continued improvement based on lessons learned from previous experience.

2. Quality Assurance Committee

Criterion The institution has a Quality Assurance Committee to ensure quality policies and objectives are set, implemented and evaluated

Looking for evidence

Does the institution have a Quality assurance committee chaired by a high level official from the higher management of the institution? Tasks of the quality assurance committee include: - Offering consultation, advice and guidance to the quality centre/ unit/ department

regarding the improvement of quality system in the institution and the improvement plan of the institution.

- Reviewing and approving assessment tools that are used in the quality assurance activities in the institution.

- Monitoring the performance quality in the institution and submitting the reports required for that.

3. Quality Centre / Unit /Department Criterion

The institution has a centre/ unit/ department linked to the higher management of the institution with the task is to execute, coordinate and monitor the quality assurance. The size of the quality centre/ unit/ department suits the size of the institution.

Looking for evidence

Does the institution have a centre/ unit/ department for managing the activities of quality assurance in the institution?

Are the required equipment, sources, resources and budget adequately provided for it.

It should be managed by a high-level staff member who has knowledge and experience on quality assurance and who has the ability to lead quality assurance activities.

He/she is also to be accountable to the CEO of the institution or any high level official and be assisted by a qualified team, preferably small.

THE MONITORING SYSTEM (CELL 4-7) 4. Monitoring Student Progress Criterion

Student progress is systematically recorded and monitored, feed back to students and corrective actions are made where necessary

44

Explanation How students are monitored and supported by staff is essential to a good student career. An institution must ensure that a good physical, material, social and psychological environment is in place. Looking for evidence

Is attention paid to study progress? Is student progress recorded? Does the recording lead to problems being pointed out in time? When is first contact made with problem cases? Does this result in remedial and/or preventive actions being introduced for the individual student or programme development?

Is special attention paid to coaching first-year students? If so, how does it work?

5. Monitoring Pass Rates and Drop-outs Criterion

An institution has a structured monitoring system to collect information about the success rates and the drop out among the students.

Explanation It is important that an institution monitors the success rates and the drop out of the students. (Pass rates or success rate: number of students, successfully finishing the program; Dropout rate: number of students that do not finish the program). Looking for evidence

What is the opinion of the department about the pass rate? If not satisfactory, what measures have been taken to improve the pass rate?

Have any fluctuations in the success rate been seen over the last years?

How high is the dropout rate? Are there explanations for the drop-out rate?

Does the department know where the drop-out students are going?

6. Feedback from Labour Market and Alumni Criterion

The institution has a structured method to obtain feedback from all stakeholders for the measurement of their satisfaction. The monitoring system includes at least: - Structural feedback from the labour market

- Structural feedback from alumni

Explanation An important element of the internal Quality Assurance system is to collect the feedback from the stakeholders in a structured way, especially the labour market and the alumni. How do we know what they think about our performance?

45

Looking for evidence Opinions from Labour Market

Do structured contacts exist with employers and the labour market for getting feedback on graduates?

How do employers appreciate the graduates? Are there any specific complaints? Do the employers appreciate specific strengths?

How do we cope with complaints from the labour market Opinions from Alumni (graduates)

Does the institution interview alumni on a regular basis?

What is the opinion and feedback of graduates when they are employed?

Is the feedback of the alumni used to adapt the programme? 7. Monitoring Research Performance (if applicable) Criterion

An institution has a structured monitoring system to collect information on the quality of its core activities. This includes monitoring the research output (number of publications), the number of grants of the staff etc.(if applicable)

Explanation As far as the institution also is involved in research, it is important to keep track of the research performance. Looking for evidence Does the TI have an efficient monitoring system keeping:

records concerning the number of publications registered by staff

records on the number of research grants

citation index? EVALUATION INSTRUMENTS (CELL 8-11 ) 8. Students Evaluation Criterion

The institution makes use of student evaluation on a regular basis. The outcomes of the student evaluation are used for quality improvement. The institution provides the students with feedback on what is done with the outcomes

Explanation Students are the first to judge the quality of teaching and learning. They experience the delivery methods. They have opinions about the facilities. Of course, the information given by students has to be counter-balanced by other opinions. Nevertheless, the institution is expected to carry out students evaluations and to use the outcomes for improvement. Looking for evidence

Does the institution use student evaluation in a structured manner?

Who is responsible for the evaluations?

What is done with the outcome of the evaluations? Are there any examples of this contributing to improvements?

46

What is the input of the students who are members of the committees involved in the internal quality assurance process?

9. Course and Curriculum Evaluation Criterion

The institution has specific mechanisms to design, approve, monitor and review the programmes within the frame of quality assurance system in the institution.

Explanation The confidence of students and other stakeholders in Technical Education is more likely to be established and maintained through effective quality assurance activities which ensure that programmes are well designed, regularly monitored and periodically reviewed, thereby securing their continuing relevance and currency. The quality assurance of programmes and the credentials awarded is expected to include:

Development and publication of explicit intended learning outcomes;

Careful attention to curriculum and programme design and content;

Specific needs for different modes of delivery (e.g. full-time, part-time, distance-learning, e-learning) and types of Technical Education (e.g. academic, vocational, professional);

Availability of appropriate learning resources;

Formal program approval procedures by a body other than that teaching the programme;

Regular periodic reviews of programmes (including external panel members).

Looking for evidence Does the institution have a mechanism to ensure the quality of its programmes? Does the institution formulate learning outcomes for each programme in consistency with the

job profiles offered by NACTE? Does the institution have a mechanism to ensure monitoring the implementation and periodic

review of the programmes to ensure their modernity and connection to the needs of the market and society?

10. Research Evaluation (if applicable) Criterion

The institution, with a task in research, has a system for regular review of research outcomes.

Explanation Research assessment is important to learn about the quality of the research efforts of an institution Looking for evidence

Is the institution involved in research assessment by an outside body? Is the research of the institution assessed when applying for grants? Does the institution organise research assessment at a regular basis?

47

11. Community Service Evaluations Criterion The institution has a system for regular review of the community outreach.

Explanation An institution is not only responsible for teaching and sometimes research. It is also responsible for serving society. Consultancy involves a broad range of activities. In general, the term consultancy covers the provision of professional advice or services to an external party for a fee or other non-monetary consideration. It is important to evaluate regularly if the institution is achieving what it wants to achieve with the community outreach. Therefore an evaluation system is important. Looking for evidence

Does the institution evaluate the role it is playing in the local, national and international community?

Are the non-profit activities of the institution evaluated? SPECIFIC QA PROCESSES (CELL 12 -15) 12. Quality Assurance of the Students’ Assessment Criteria

An institution has clear procedures to assure the assessment of students.

Students are assessed on the basis of published criteria, regulations and procedures that are applied consistently.

There are clear procedures to assure the quality of the examinations.

There is an appeal procedure.

Explanation Students’ assessment is one of the most important elements of Technical Education. The outcomes of assessment have a profound effect on student’s future careers. It is therefore important that assessment is carried out professionally at all times and takes account of the extensive knowledge that exists on testing and examination processes. Assessment also provides valuable information for institutions about the efficiency of teaching and learner support. Student assessment procedures are expected to:

be designed to measure the achievement of the intended learning outcomes and other programme objectives;

be fit for purpose, whether diagnostic, formative or summative;

have clear and published grading/marking criteria;

where possible, the assessment is not relying on the verdicts of single examiners;

take account of all the possible consequences of examinations regulations;

have clear regulations covering student absence, illness and other mitigating circumstances;

ensure that assessments are conducted securely in accordance with the institution's stated procedures;

be module to administrative verification checks to ensure the accuracy of the procedures;

inform students clearly about the assessment strategy being used for their programme, what examinations or other assessment methods they will be subjected to, what will be

48

expected of them, and the criteria that will be applied to the assessment of their performance.

Looking for evidence

Does the assessment method foster open, flexible, reflective and outcome-based assessment?

Are the criteria made explicit?

Are the assessment strategies in line with clearly defined learning outcomes?

Is a range of assessment methods used?

Is the scope and weighting of the assessment known to all concerned?

Are the standards applied in assessment explicit and consistent across the curriculum?

Are procedures regularly applied to ensure that, as far as possible, assessment schemes are valid, reliable and fairly administered?

Do students have ready access to reasonable appeals procedures?

Is the reliability and validity of the assessment methods documented as required and regularly evaluated?

Are new assessment methods developed and tested? The formulated questions have to be looked at in a general way, taking into account the general approach across all programmes, not just in a specific programme. What is the general practice at the institution? 13. Assurance of the Quality of the Staff Criteria An institution has means to satisfy itself that its staff are qualified and competent to conduct the core activities of the institution: training, (research) and the community outreach: - Adequate staff appointment procedures - An adequate staff appraisal system

- Staff development activities and plans

Explanation Teaching staff is the single most important learning resource available to most students. It is important that those who teach have full knowledge and understanding of the modules they are teaching, have the necessary skills and experience to communicate their knowledge and understanding effectively to students in a range of teaching contexts, and can access feedback on their own performance. Institutions should ensure that their staff recruitment and appointment procedures include a means of making certain that all new staff have at least the minimum necessary level of competence. Teaching staff should be given opportunities to develop and extend their teaching ability and should be encouraged to value their skills. Institutions should provide inexperienced teachers with opportunities to improve their skills to an acceptable level and should have the means to remove them from their teaching duties if they continue to be demonstrably ineffective. Looking for evidence

How is the staff recruitment system organised?

How is the promotion system organised? What criteria are important for promotion?

Are staff appraisals carried out? How are these done? What are the consequences?

Has a clear Human Resource policy been put in place?

Are staff development activities and plans in place and carried out?

49

14. Quality Assurance of the Facilities

Criteria An institution has clear procedures to ensure that the quality and quantity of its facilities needed for student learning are adequate and appropriate for each programme offered: - Adequate checks on the computer facilities - Adequate checks on the library

- Adequate checks on the laboratories and the related equipment - Adequate checks on the audio-visual aids - Adequate checks on the physical resources -

Explanation In addition to their teachers, students rely on a range of resources to assist their learning. These vary from physical resources such as libraries or computing facilities to human support in the form of tutors and other advisers. Learning resources and other support mechanisms should be readily accessible to students, designed with their needs in mind and responsive to feedback from users of these services. Institutions should routinely monitor, review and improve the effectiveness of the support services available to their students. Looking for evidence What procedures do you have to assure the quality of the:

Lecture halls, etc.?

Libraries?

Laboratories?

Learning resources

Research resources? 15. Quality Assurance of Student Support Criteria

An institution has clear procedures to assure the quality of the student support and student advice. In establishing a learning environment to support the achievement of quality student learning, teachers must do everything in their power to provide not only a physical and material environment that is supportive of learning and is appropriate to the activities involved, but also a social or psychological environment.

Explanation It is important that the quality of student support is assured. This may include student advice and/or counselling plus the physical and material environment. Looking for evidence What procedures to you have to assure the quality of the student support activities?

A tutoring system?

Students’ advice and/or counselling?

Accommodation facilities?

Recreational facilities?

Health services?

Spiritual services

Transport services?

Students’ government?

50

SPECIFIC QA INSTRUMENTS (CELL 14-17) 16. The SWOT Analysis or Self-assessment Criterion

The institution regularly (but at least once every 5 years), conducts a self-assessment of its core activities and of the institution as a whole to learn about its strengths and weaknesses. This self-assessment must lead to a quality plan.

Explanation A self-assessment or a SWOT analysis is a powerful instrument for learning more about the quality of the core activities and the quality of the institution as a whole. It will answer the basic questions whether the institution is doing the right things right and whether it is able to achieve its goals. Often a self-assessment is connected with external assessment or accreditation, because the accrediting body or external assessors ask for a self-assessment report as input. Even when there is no connection with an external assessment, it will be productive for the institution to conduct self-assessment on a regular basis. Looking for evidence

Does the institution already have experience with the instrument of self-assessment?

Is there any connection with external assessment/accreditation?

If not yet done, is the institution planning to conduct self-assessments on regular basis?

If the institution has or has not yet conducted self-assessments, how does it know about its quality?

17. The Inter-collegial Audit/Peer Review Criterion A self-assessment might be part of an External Quality Assessment (EQA) or accreditation process where the self-assessment report acts as input for the external review team. If the self-assessment is not connected to the EQA, the institution will be expected to organise an audit itself based on the self-assessment report.

Explanation The self-assessment gives us a good idea about our quality. However, this is not enough. We have to check our own view against the views of the outside world. Therefore, it is advisable to organise an inter-collegial audit. This means that experts/colleagues from another institution will check our SAR Looking for evidence

Does the institution have an inter-collegial audit system?

How often is the system used?

Does the institution have trained auditors? Where were they trained?

What does the institution do with the outcomes of an audit? Give some examples. 18 Information Systems Criterion

The institution should have an information system that documents its performance on the key performance indicators approved in the quality improvement plan. Updated, fair and objective information on programmes should be published

as drawn from the outcomes of its information system or any other comparative studies.

51

Explanation Institutional self-knowledge is the starting point for effective quality assurance. It is important that institutions have the means to collect and analyse information about their own activities. Without this they will not know what is working well and what needs attention, or the results of innovative practices. The quality-related information systems required by an individual institution will depend to some extent on local circumstances, but are at least expected to cover:

Student progression and success rates;

Employability of graduates;

Student satisfaction with their programs; effectiveness of teachers;

Profile of the student population;

Available learning resources and their costs;

The institution’s own key performance indicators. An efficient information system is also important for benchmarking the institution in question with other institutions in the region. In fulfilling their public roles, Technical Institutions have a responsibility for providing information about the programmes they offer, the intended learning outcomes, the qualifications they award, the teaching, learning and assessment procedures used, and the learning opportunities available to their students. Published information might also include the views and employment destinations of past students and the profile of the current student population. This information should be accurate, impartial, objective and readily accessible and should not be used simply as a marketing opportunity. The institution should verify that it meets its own expectations in respect of impartiality and objectivity. Looking for evidence

Does the institution have an information system revolving on the key performance indicators?

Does the institution collect data on the key performance indicators and feed the same to the information system?

Does the institution analyses the data and uses it to improve its activities and academic programs?

Does the institution have an office assigned with linking the institution with the society?

Does the information system cover the key performance indicators?

Are the results used to compare the performance of the institution with other institutions similar in type and size?.

19. The QA Handbook Criterion

An institution has a QA handbook that documents all regulations, processes and procedures concerning Quality

Assurance. This handbook is public and known to all the people concerned.

Explanation A QA handbook contains all the documents (or references to the documents), processes and procedures concerning activities in the Quality Assurance process. Looking for evidence

Does the institution already have a QA handbook?

What is the content of the QA handbook? Sum up the chapters.

What documents, processes and procedures are already available?

What documents, processes and practices need to be developed?

Who will collect and compile all the information?

52

20. The Follow-up Criterion There is clear evidence that the institution used the information from the monitoring system and the outcomes of evaluations to improve the quality.

Explanation Quality assurance and the activities in the framework of Internal Quality Assurance only make sense if the outcomes and information are used for the enhancement and improvement of the quality. Without follow up actions, evaluation is a waste of time. Especially a self-assessment must lead to a quality plan for the near future. Looking for evidence

The institution has a clear policy for enhancement and improvement of the quality

The institution uses the information collected by the monitoring system and the outcomes of evaluation to develop a yearly quality plan

There are examples, showing the results of Quality Assurance?

5.5. Strengths/Weaknesses Analysis The self-assessment is followed by a strengths-weaknesses analysis. At the same time, this serves as a check to see how far the institution is in compliance with the given criteria. This is best done, using the checklist (see appendix 1) and summarizing the outcomes in table 7. The different aspects of the program will be assessed on a scale 1 – 7. The marks have the following meaning:

1 = absolutely inadequate; immediate improvements must be made 2 = inadequate, improvements necessary 3 = inadequate, but minor improvements will make it adequate 4 = adequate as expected 5 = better than adequate 6 = example of good practice 7 = excellent

The overall assessment of the different aspects is based on the scores given to each sub-aspect in the category. But, of course not all sub-aspects have the same weight. This means that you cannot calculate mathematically an average. You have to balance the various sub-aspects and judge the importance of each of them. Aspects with positive outcomes may compensate for some negative ones. Summarizing the outcomes per category is not a mathematical enterprise, nor ticking boxes. One has to balance the importance of each criterion. The same counts for the overall judgment of the IQA-system. One cannot calculate the average of the categories, but look at the importance of the category. Summarizing the outcomes for each category shows the strengths and weakness of the IQA system. Use table 7. Table 7: Summary of the Assessment of the IQA system

1 2 3 4 5 6 7

1 QA Organisation

2 Monitoring

3 Evaluation Instruments

53

4 Special QA processes

5 Specific QA instruments

6 Follow up

Overall judgment about the QA -system

Summary of strengths Summary of the points that the institution considers to be its strengths and mark the points that the institution is proud of. Summary of weaknesses Indicate which points the institution considers to be weak and in need for improvement. Also indicate what the institution is going to do about this. For the improvement of the weaknesses, the institution must include planned actions in the Quality improvement plan. Only with a clear follow up and a quality action plan, the investment in the self-assessment and the SAR makes sense.

5.6 The Self-assessment Report (SAR) After finishing the self-assessment of the Internal Quality Assurance system, the institution will write down the outcomes of the assessment in a Self Assessment Report (SAR). The SAR is an important document. It contains the basic information to be provided to NACTE for the Quality Audit to be organised by NACTE. On the other hand the SAR is the basic document for the institution for the formulation of a quality improvement plan for the coming years. The content of the SAR follows the lines of the aspects, discussed during the self-assessment process. For each aspect to be treated one should:

Describe clearly the state-of-the art. An outsider must understand the situation.

Analyse the situation. What is your opinion about it? Satisfied or not? If not, why not?

Describe how far you meet the formulated criteria. What evidence can you provide?

Describe the weakness and the strengths. Content of the Self-assessment Report Table 8 defines the content of the self-assessment report. Be sure to discuss the report within the institution and ensure that everybody is able to recognise himself/herself in this picture. Table 8: Content of a Self-assessment Report of IQA-system

Introduction

How was the self-assessment carried out?

Short description of the institution

Chapter 1: Policies and Procedures for Internal Quality Assurance

Chapter 2: The Monitoring Instruments 2.1 Student progress 2.2 Pass rates and drop out 2.4 Feedback from labour market and alumni 2.5 Research performance

Chapter 3 Evaluation Instruments 3.1 Student evaluation 3.2 Course /curriculum evaluation 3.3. Research evaluation 3.4. Service evaluation

Chapter 4: Specific Quality Assurance Processes 4.1 Assurance of student assessment 4.2 Assurance of quality of the staff 4.3.Assurance of quality of facilities and infrastructure 4.4. Assurance of quality of student support

Chapter 5: Specific Instruments

54

5.1 Self-assessment/SWOT analysis 5.2. Inter-collegial audit/peer review 5.3. Information system 5.4 Quality handbook

Chapter 6: Follow-up Activities

Chapter 7 Strengths-weaknesses Analysis 7.1 Summary of strengths 7.2 Summary of weaknesses 7.3 Quality plan for the coming years

5.7 Preparation for the Quality Audit All institutions intending to be registered must submit to NACTE a SAR of the evaluation of the IQA system. The self-assessment of the IQA system will be followed by a Quality Audit, organised by NACTE. An external audit team will visit the institution. To prepare the institution for the Quality Audit, the following is important:

Make sure that the SAR is submitted timely to NACTE

Make sure that the SAR is informative, analytical and honest. The SAR is the basic information for the audit team

Make sure that all staff (and students if applicable) know about the SAR

Make sure that during the site visits the most important documents, showing evidence that the institution meets the criteria are displayed for the audit team

Make sure that everybody is available for interviews during the site visit.

5.8 The Quality Audit A Quality Audit is the process of checking that the TI has a robust Internal Quality Assurance system. An IQA system is aiming at setting up, maintaining and improving the quality and standards of teaching, and service to community. The overall objective is to promote and improve the quality of the programs continuously, their mode of delivery, their support facilities, and finally the outcome; the graduates. The use of the instrument of Quality Audit assumes that a Quality Audit will reveal the potential of the TI to offer quality programmes and to deliver competent graduates. The existence of a well functioning IQA system is a sine qua non for institutional accreditation. There are three possibilities for the organisation of the Quality Audit:

An institution, that aspires to be registered, has to be accredited first. In order to be registered, it is compulsory for the institution sends a Self-assessment report of its Internal Quality Assurance system. NACTE will assess the IQA-system based on the SAR and makes recommendations for improvement;

The Quality audit is included in the process for Institutional accreditation;

The TI takes the initiative and invites NACTE to organize a Quality Audit visit; or

NACTE takes the initiative and informs the institution that a Quality Audit will be organized for the same

In all cases, the Quality Audit process starts with a self-assessment of the IQA system by the institution. NACTE will check the SAR and organise a site visit. The basis for the site visit is the outcome of the self-assessment process, the SAR. The time given for the self-assessment is 6 months. Because the members of the Audit team need time for preparation and for reading the SAR, a Quality Audit can only take place 7 months after the formal announcement.

55

After the formal announcement, NACTE start the preparations for the Quality Audit. This concerns:

If needed, supporting the QA-officer of the institution in conducting the self assessment

Appointment of the members of the audit team

Training of the members of the audit team

Organization of the site visit

5.8.1. The Audit Team An effective audit team, commissioned to conduct the Quality Audit, will have at least 3 members. The expert team should include:

A chairperson, totally independent and unconnected with the Technical Institution to be assessed. The chair should have experience with management structures in Technical Institutions and have experiences with the developments that have taken place over the last few years

An expert in quality assurance

An expert from the labour market area or from a professional body, covering the nature of the Technical Institution

There are some conditions members of the audit team have to meet:

Members should act independently

There should be no conflicts of interest. Members should stand to gain nothing from their verdict

5.8.2. Preparatory Work of the Audit Team As soon as the institution has sent the self-assessment report to the Audit Team, the members will study the report carefully before the team comes together in a preliminary meeting. The expert team will meet before the planned site visit for training and to prepare itself for the audit. The meeting will be held on the day the site visit begins. The issues for discussion in the meeting are:

Discussion on the frame of reference, used for making judgments ( the model for IQA) It is important that all members of the audit team understand the requirements for an IQA-system. Therefore it is important to discuss the model for an IQA-system (figure 9).

Discussion on the self-assessment report During the meeting, the team will discuss the SAR and formulate questions to be asked during the site visit

5.8.3. The Site Visit After the preparatory meeting, the expert team will start the site visit. The SAR is the basic source of information for the audit team and should provide the basic information. But other sources should also be used:

Interviews with management, QA officer, staff and students

Review of documents providing evidence of meeting the criteria The site visit, on average will take 2 days, but this depends on the size of the institution. NACTE should have confirmed the programme for the site visit in consultation with the institution according to a given format (see Table 9). Beforehand, appointments will have been made with whom the audit team would like to talk to. The interviews start with a discussion with the writers of the self-assessment report. In this interview, the team can ask for clarification of any obscurities and explanation of any topics that are not totally clear. Table 9: Draft Program for a Quality Audit

Time Activity

Day 1 09.00-12.00

Audit Team meet for information about their task and discussion on:

The SAR

Specific questions

The program of the site visit

12.00 – 13.00 Welcome by management of the TI

13.00 - 18:00

Interviews with:

the writers of the self-assessment report

management

56

QA-office

staff members

students

Review of documents

19:00-20:00 20:00

Short meeting for discussion on the findings of the day and for setting the programme for the next day

Day 2 9:00 - 11:00

Additional interviews, if needed/ consultation and review of documents

11:00 - 13:00 Formulation of the findings

13:00 - 13:30 Feedback

13:30 End of the visit

5.8.4 Formulating the Findings The the second day of the visit is normally used for drawing up the findings. There are about two hours available for this difficult task. The best method is to start with completion of the checklist (see appendix 1) by each member individually. The committee members are requested to give a mark between 1 and 7 for the various aspects. To have some idea of the value of the figures, bear the following ideas in mind:

Score 1-2 when you believe this aspect should be considered as critical. The TI has to take action. Something has to be done and cannot wait.

Score 3 when you believe this aspect is unsatisfactory. It must be improved, but does not directly threaten the quality.

Score 4 when you believe the situation is satisfactory. The TI may be satisfied, but there is no reason to be proud.

Score 5 when you believe this aspect can be assessed as more than satisfactory, but not excellent.

Score 6 when you believe this aspect can be assessed as more than satisfactory and can be seen as an example of good practice.

Score 7 when you believe this aspect can be assessed as excellent. The TI can be proud of it and it is certainly a strong point.

The overall assessment of the different aspects is based on the scores given to each sub-aspect in the category. But, of course not all sub-aspects have the same weight. This means that you cannot calculate mathematically an average. You have to balance the various sub-aspects and to judge the weighting of each of them. Positive aspects may compensate for some negative ones. Summarizing the outcomes per category is not a mathematical enterprise, nor ticking boxes. One has to balance and weight the importance of each criterion. The same counts for the overall judgment of the IQA system. One cannot calculate the average of the categories, but weight the importance.

After completing the checklist by the members, the chairperson will discuss the situation and fix the judgement of the team and complete the general overview (table 10). Summarizing the outcomes for each category shows the strengths and weakness of the IQA system.

Table 10: Summary of the Assessment of the IQA System

1 2 3 4 5 6 7

1 QA Organisation

2 Monitoring

3 Evaluation Instruments

4 Special QA processes

5 Specific QA instruments

6 Follow up

Overall judgment about the QA -system

57

After completing the list, the resting time will be used to draw up an inventory of the issues to be discussed during the exit meeting.

5.8.5 The Exit Meeting The oral presentation of the issues noted to the Governing Board/Management of the institution at the end of the visit holds a special position in the process. Sometimes, findings and conclusions are not really suitable for the report, but the team would like to make a critical statement about them. In that case, the oral presentation can be used to formulate strongly worded recommendations. In order to do justice to the institution, the oral presentation is not public; the team reports to the Management or the Governing Board. The chairperson should stress that this is an interim report; some conclusions may change during the final discussion on the report. It is advisable not only to mention weaknesses, but also strengths. After the site visit, the chairperson and secretary will write a first draft of the report, using the completed checklists and the minutes of the oral presentation. Table 11 gives an outline of the content of the assessment report. Table 11: Content of the Audit Report IQA-system

Introduction

Background of the Quality Audit: Why is the assessment done?

Composition of the audit team

Short description of the TI

Chapter 1: QA Organization

Chapter 2: The Monitoring Instruments 2.1 Student progress 2.2 Pass rates and drop-out 2.3 Feedback from labour market and alumni 2.4 Research performance

Chapter 3 Evaluation Instruments 3.1 Student evaluation 3.2 Course /curriculum evaluation 3.3. Research evaluation (if applicable) 3.4. Community service evaluation

Chapter 4: Specific Quality Assurance Processes 4.1 Assurance of student assessment 4.2 Assurance of quality of the staff 4.3. Quality assurance of facilities 4.4. Quality assurance of student support services

Chapter 5: Specific Instruments 5.1 Self-assessment/SWOT analysis 5.2. Inter-collegial audit/peer review 5.3. Information system 5.4 Quality handbook

Chapter 6: Follow up activities

Chapter 7 Strengths-weaknesses Analysis; Recommendations 7.1 Summary of strengths 7.2 Summary of weaknesses 7.3 Summary of the recommendations

The 1st draft of the report will be discussed with the team members. The 2nd draft will be sent to the TI for comment. The comments should concern only factual errors and inaccuracies, not the differences in opinion. The audit team will decide what to do with the comments.

5. 9 NACTE and the Quality Audit The final report of the audit team will be sent to the Council. The Council will check if the Quality Audit has been done according the rules and see if the report contains all information needed. If the audit report is accepted, the Council will follow the advice of the audit team. The decision might be:

To provide the TI with a specific Quality Label for the IQA system as it is considered to meet the requirements in a satisfactory way

58

To formulate recommendations the TI has to meet for the improvement of the IQA system. The TI will report yearly to NACTE about the state-of-the-art.

If the institution disagrees about the way the assessment is done, disagrees about the findings of the audit team or disagree about the decisions it may contact NACTE within 30 days from the day of evaluation. NACTE will install an independent committee that will investigate the complaints. The verdict of the independent committee will be binding both for NACTE as the institution.

59

CHAPTER 6

Self-assessment of the Institution as Preparation for Accreditation

NACTE has the task to accredit Technical Institutions (section 5(1) (a) of the National Council for Technical Education Act, Cap. 129. Accreditation means that NACTE will check if the institution has put in place a robust quality assurance and quality control system which will ensure the product of the TI will meet the requirements of the labour market There are three possibilities:

Accreditation for registration. This is compulsory for newly established TIs

NACTE takes the initiative for re-accreditation and announces that an Institutional Accreditation will be organized for the institution.

The institution takes the initiative and invites NACTE to organize an institutional accreditation. When NACTE conducts an institutional accreditation, the institution will be required to prepare itself by conducting a Self-assessment. However, in the absence of a call from NACTE for accreditation, the QA-officer may still initiate Self-evaluation for the improvement of the institution. This chapter offers the QA-officer an instrument to conduct the self-assessment at institutional level.

6.1 An analysis Model for the Self-Assessment of the Institution In order to map the quality in a self-assessment we need a clear model to prevent looking at some aspects and ignoring others. Figure 6 (next page) shows a model for the analysis of quality of the institution. For the critical self-assessment, this analysis model will be used Concerning criteria we have to keep in mind that there are no absolute and objective criteria and standards. The criteria for assessing the quality given in the handbook are based on the criteria as formulated by external quality assessment agencies, e.g. European, American, Asian, Australian and South African accrediting bodies, among others. After studying many sets of standards and criteria, a common denominator has been formulated. To verify compliancy of its own criteria, the TI can use the regional criteria as benchmark. In general, one may say that the formulated criteria can be seen as the minimum criteria. The Academic standards, formulated by NACTE are included in the model. Table 12 shows the number of the cells of the analysis model, where the NACTE standards are discussed. Table 12: NACTE standards and the analyze model of the quality of an institution

NACTE Standard Cell nr. Analysis model

Standard 1: Institutional Vision and Mission 2

Standard 2: Governance and Administration 4

Standard 3: Institutional Integrity 2

Standard 4: Institutional Effectiveness 4/8

Standard 5: Educational Programmes; 8/11

Standard 6: Student Information and Admission to Programmes 8

Standard 7: Student Guidance and Support 8

Standard 8: Staff Selection, Appraisal and Development 5/8

Standard 9: Physical Resources 7

Standard 10: Financial Resources. 7

The self-assessment aims at finding evidence that the institution is meeting the criteria. Therefore, one has to look at the criteria and try to find indications of meeting the criteria:

give a description of the state-of-the-art of the aspect

60

make a critical analysis of the state-of-the-art. (Is one satisfied with it or not?)

describes the strengths and weaknesses concerning the mentioned aspect

what evidence is there that you are meeting the criteria?

if there are problems or if you are not satisfied, what actions are planned to overcome the shortcomings?

Set up a time frame for the action plan Figure 10 shows a model for the analysis of quality of the institution. For the SWOT-analysis, this model will be used. In section 6.2, under the heading of the aspects (such as Mission, Vision, Goals and Aims, Quality Assurance, Facilities and Infrastructure) the criteria to be met are given. An explanation and interpretation of the criteria is given where necessary. For each aspect, the following format is used:

The name of the cell in the model is given, just as the title of the aspect that will be treated, e.g. Vision and Mission of the Institutions

The Criterion concerning that aspect is given in a box. The criterion shows what commonly expected form an institution is. The Academic standards as formulated by NACTE has priority above the general accepted criteria

If NACTE is not mentioning a criterion, but the criterion is seen internationally as important, the criterion is mentioned.

Under the heading “Evidence” a set of questions and/or statements is drawn up to help to find evidence if the criteria are being met. Please be aware of the following as far as these questions are concerned:

The questions-set are not meant as a compulsory list that has to be completed. It is not a questionnaire to be answered point by point. It must be seen as a tool to collect information and evidence. The questions are to be seen as reminders.

Figure 10: Assessment model for the quality of the institution

2.1MissionVision

AimsObjec ves

2.2Ins tu onalintegrity

1.Requirementsstakeholders

13.Sa sfac onstakeholders

12A

CH

IEV

EM

ENT

S

3Policyplan

8Educa onalEffec veness

8.1Theprogram

8.2Studentassessment8.3fielda achment8.4programdelivering

8.5QualityStaff8.6Studentadmission

8.7Facili es

9Communitycontribu on

10benchmarking

11Quality

assurance

4Governance

&

administra on

5Humanresource

6Funding

&

Financialmanagement

7Facili es

&

Infrastructure

2

61

6.2 The Self-assessment: The Quality Aspects to be Assessed 1. Requirements of Stakeholders Criterion

The institution has a clear idea about the relevant demands and needs of all stakeholders

Explanation Technical Education has many stakeholders and each stakeholder has its own ideas about quality: the government or the state, the employers, the academic world, the students and parents, and society at large. Each stakeholder will appreciate different aspects of quality. Because each stakeholder has its own ideas and expectations, each stakeholder needs to formulate, as clearly as possible, his/her requirements. The TI, as an ultimate supplier, must try to reconcile all these different wishes and requirements. As far as possible, the requirements of all stakeholders should be translated into the expected goals and objectives/outcomes of the institution. This regards the three core activities: teaching/learning, research and community outreach. Evidence

Does the institution have a clear idea about the requirements set by the government?

Does the institution know clearly the requirements of academia?

Does the institution know the needs and requirements of the labor market?

Does the institution analyze the needs and requirements of the students/parents?

Does the institution analyze the needs and requirements of the society?

How does the institution balance the requirements of the different stakeholders? 2. Vision, Mission, Aims and Objectives 2.1 Vision Mission, , Aims and Objectives Criterion The institution has a statement of vision and mission that defines the institution, its educational purposes, its student constituency, and its place in the technical education and training community.

Explanation The mission and objectives of the TI should be drafted at the level of the TI and suit its nature and the group it serves to clearly determine its identity, academic activities and its current and future role in Technical Education in a brief, realistic and comprehensive manner. Evidence

Existence of a statement of vision and mission, adopted by the governing board or advisory board or council identifying the broad-based educational purposes it seeks to achieve;

A clear vision and mission statement consistent with the original objectives and defining the students the institution intends to serve as well as the parameters under which programs can be offered and resources allocated;

Institutional planning and decision-making guided by the vision and mission statement;

Regular evaluation and revision of the institution’s vision and mission statement.

62

2.2 Institutional Integrity Criterion The institution demonstrates honesty and truthfulness in presentations to its constituencies and the public; in pursuit of truth and the dissemination of knowledge; in its treatment of and respect for administration, academic staff, other staff, and students; in the management of its affairs and in relationships with NACTE and other external agencies.

Evidence

Clear, accurate, and consistent statements and publications to its constituencies, the public, and prospective students through its prospectus including those presented in electronic formats concerning: o Educational purposes; o Qualifications, curricular offerings, educational resources, and course offerings; o Student fees and other financial obligations, student financial aid; o Requirements for admission and for achievement of awards, including the academic calendar

and information regarding program length; and o The names and academic qualification and achievements of administrators, academic staff,

and governing or advisory board.

Readily available governing or advisory board-adopted policy stating the institutional commitment to the pursuit and dissemination of knowledge and which fosters the integrity of the teaching-learning process;

Academic staff and other staff distinguish between personal conviction and proven conclusions and present relevant data fairly and objectively to students and others;

Clear policies concerning the principles of academic honesty and the sanctions for violation;

Policies and practices, of appropriate understanding of and concern for issues of equity and diversity;

Honesty and integrity in its relationships with NACTE and compliance with NACTE standards, policies, guidelines, public disclosure, and self-evaluation requirements;

Regular evaluation and revision of institutional policies, practices, and publications to ensure integrity in all representations about its vision and mission, programs and services.

3. The Policy Plan

Criteria

The institution plans constantly to achieve its mission and objectives and evaluate the extent, way and quality of achieving them. Results of the annual assessment are used in the processes of continuous comprehensive planning and improvement. It also works on self-assessment and criticism and reviews its objectives, policies and procedures accordingly.

The institution ensures the participation of all councils, committees, administrations, faculty members and students in planning and assessment.

The institution periodically reviews the assessment processes and planning activities to ensure their effectiveness.

Evidence

The institution adopts strategic planning based on complete analysis of the current situation, identifying the gap between it and its desired measurable objectives, and prepares executive plans to fill the gap and links them to its budget so they are implementable.

The institution commits to having assessment as one of the main components in the process of strategic planning. It evaluates its activities including teaching, learning, research and community service in light of its mission and objectives. Results of the assessment are used to improve the quality of its academic programs, services and activities.

The institution provides the necessary resources for the processes of planning and assessment.

63

4. Governance 4.1 The Governing Board, Advisory Council

Criterion

The institution has a Governing Board or Advisory Board or Council responsible for the quality and integrity of the institution.

Explanation The leadership and administration of the TI should be faithful to the TI and work to achieve its interests and the interests of the beneficiaries of its services. Managers and leaders should perform their tasks within an accurate organizational structure, set clear administrative systems that are consistent with the mission and objectives of the TI, and prepare clear policies for managing it and an accurate accountability system. Evidence

Existence of a Governing Board/Advisory Board/Council, which is an independent policy-making board capable of reflecting the public interest in board/council activities and decisions;

Putting in place a mechanism that provides for continuity of board/council membership and staggered terms of office;

Having in place operational policies and procedures for ensuring that all inputs, work processes, and products or services of the institution are in line with its vision and mission.

4.2. Administration Criterion

The Institution has effective Administrative structure and system which facilitates the achievement of set goals and objectives

Evidence

Existence of qualified administrative staff of appropriate number to enable the institution to achieve its goals and is organized to provide appropriate administrative services;

A lean administrative structures and systems which ensure appropriate roles for the board, administration, academic staff, supporting staff, and students, and facilitate effective decision making among the institution's constituencies.

5. Human Resources (HR)

Criteria

The institution attracts and retain high-quality faculty staff and support staff by clearly defining their responsibility, and by evaluating their performance on a regular basis by means of an adequate staff appraisal system

The institution provides for: - a system of staff development to enhance the knowledge and skills of faculty and supporting staff in conducting

activities that have a direct influence on the quality of teaching-learning. This should include the formulation of a concrete personnel development plan;

- evaluation of the effectiveness of the provided training - compilation of records of education, experience, training, and other essential qualifications required of lecturers

and supporting staff.

The institution establishes an activity plan and evaluates activities to encourage students, faculty members and other personnel to be conscientious in their thoughts and speech.

The institution enhances the professional ethics of its students, faculty members and other personnel

64

Evidence • How does the institution select and appoint its academic staff? • Is an adequate staff appraisal system in place for use in evaluating performance and promotion? • How is staff performance evaluated?

How does the institute use performance evaluation results to improve staff performance. What mechanism does the institute employ to retain competent employee • What opportunities are given for staff/HR development and training? • How does the institution evaluate the efficiency of its staff/HR development activities? • How does the institution stimulate the ethics of its students, academics and other staff? 6. Funding and Financial Management Criterion

The provision of financial resources to meet academic requirements is effectively planned and reviewed.

Explanation The institution should ensure provision of sufficient financial resources to implement its academic programs and the services it offers and use these resources with high efficiency. The TI adopts long term planning while preparing the budget. It should have effective systems for delegating financial authorities as to provide administrators in different levels with flexibility to make financial decisions governed by an accurate accountability system. Evidence

Effective planning, provision and management of financial resources, which are accurately assessed and integrated with academic priorities and programme requirements;

Availability of adequate financial resources to achieve, maintain, and enhance institutional programmes and services to enable students to attain the required programme outcomes;

Adequate level of financial resources that provides a reasonable expectation of financial viability and institutional improvement;

Good management of institutional financial affairs that is subjected to regular external audit.

7. Facilities and infrastructure Criterion

The provision of physical resources to meet academic requirements is effectively planned and reviewed.

Explanation The institution should design education buildings and facilities or modify them to fulfil the needs of teaching and learning required by the academic programs and its activities and services. It should ensure that education buildings and facilities are safe and healthy in order to provide high quality service. Evidence

Effective planning and provision of physical resources, which are accurately assessed and integrated with academic priorities and programme requirements;

Availability of sufficient and appropriate physical resources to enable students to attain the required programme outcomes; and

Physical resources that are reviewed and maintained to ensure that they are fit for the purpose.

65

Facilities of the institution are clean and attractive and are being maintained.

The educational facilities meet the standards of safety and are adequate for the use of persons of special needs.

The institution provides administration, maintenance and operation of the educational facilities in a sufficient manner that ensures achieving the requirements of necessary quality and safety whether while implementing the academic programs or while providing the rest of services of the institution.

8. Institutional/Educational Effectiveness Educating competent graduates is one of the core activities of an institution. To determine the quality of the teaching/learning process and the quality of curricula, faculties/departments one have to evaluate the programs individually. The outcomes must be used to get a general overview of the quality of the educational provisions. For self-analysis at program level and the criteria on teaching/learning, see chapter 7 of the handbook. The outcomes of the self-assessment at program level will be used for assessing the average quality of the core activity, teaching and learning of an institution. At the institutional level an analysis need to be made from the following aspects: 8.1 The Programs 8.2 Student Assessment 8.3 Field attachment/workplace training components 8.4 Program Delivery 8.5 Quality of the Staff 8.6 Students Admission 8.7 Facilities and Infrastructure

8.1. The Programs Criteria

The programs at offer in the institution: - are meeting the expectations of the stakeholders - are meeting the current competence requirements - have clearly formulated expected learning outcomes - are coherent - are up-to-date

Stakeholders have appropriate opportunities to be involved in formulation and review of program outcomes.

Explanation The programs at offer in the institution should be in line with the expectations of the stakeholders and should be in line with the mission and vision of the institution. The objectives and the expected learning outcomes11 must make this clear. The programs should be in line with the current expected competence requirements. Evidence

Qualifications are guided by current labour market demand. Formulation of the job profile is based on occupational analysis to incorporate current competence requirements;

Curriculum (programme) development process is guided by job profile

11Learning outcome: A learning outcome is the specification of what a student should learn as the result of a period of specified and supported study (INQAAHE,: Harvey, L., 2004–6, Analytic Quality Glossary, Quality Research International). See also section 1.3.1 of this handbook.

66

Clear and concise programme regulations, which are consistent with the level of the qualification and with the institution’s academic policies and procedures;

Curriculum development process includes inputs from internal and external stakeholders;

The completion of curriculum approval processes which include appropriate input by external and internal stakeholders. This includes the development and approval of awards and qualifications associated with the programme.

Development and approval processes for revised or extended programmes, which are consistent with the requirements for establishing a new programme at that level.

8.2. Student Assessment Criteria

The institution has well functioning student assessment systems through all programs at offer and clear rules to assure the quality of the assessments.

The institution I has a clear policy to promote that the examinations are objective, equivalent and trustworthy

The institution ensures consistency of the examinations; consistency between the programs and in time

The institution has a policy to promote a variety of assessments methods

The institution takes care that examination committees function adequately and performs the statutory task.

Students have an opportunity to appeal the results of assessment in a manner that is fair and equitable.

Explanation Student assessment is an important element in Technical Education. The outcomes of the assessment have a profound effect on students' future careers. It is therefore important that assessment is carried out professionally at all times and takes into account the extensive knowledge that exists on testing and examination processes. Assessment also provides valuable information for institutions about the efficiency of teaching and learner support. It is the responsibility of the faculty/department to assure the quality of the student assessment. The central management must have a good policy and good control mechanism to monitor the decentralized activities. Student assessment procedures are expected to:

be designed to measure the achievement of the intended learning outcomes and other program objectives;

be fit for purpose, whether diagnostic, formative or summative;

have clear and published grading/marking criteria;

take account of all the possible consequences of examination regulations;

have clear regulations covering student absence, illness and other mitigating circumstances;

ensure that assessments are conducted securely in accordance with the institution's stated procedures;

be module to administrative verification checks to ensure the accuracy of the procedures;

Inform students clearly about the assessment strategy being used for their program, what examinations or other assessment methods they will be module to, what will be expected of them, and the criteria that will be applied to the assessment of their performance.

Evidence

Pegging the learning outcomes prescribed for a particular qualification with corresponding assessment criteria and credit values;

Implementation of procedures for the moderation of assessment for all programs and a process for ensuring that the results are used as feedback for ongoing improvement of the program;

Mechanisms to ensure consistency of assessment of learning outcomes where different modes of delivery are utilized for students enrolled on different programs leading to the same qualification, or where the same program is delivered on more than one site.

67

Student appeals procedures, which are made known to students and present no unreasonable barriers to access i.e. students should understand the details of appeal procedures and should have access to them without prejudice; and

Records of student appeals and outcomes, which demonstrate the fairness and equity of the process.

8.3 Field attachment/workplace training components Criterion

Field attachment/workplace training components are effective and integrated into curricula

Explanation Field attachment, practical work and workplace training are essential elements in a competence oriented program. Evidence

Consistency between the learning outcomes and assessment strategies of the field attachment/workplace training components of programs and the aims and philosophy of those programs;

Collaboration of training institution with stakeholders during the development of the rationale for the field/work-based training components of programs;

The use of clear criteria that address both academic and safety issues for the selection and monitoring of suitable training sites; and

A field attachment policy which clearly defines the responsibilities of staff, students and industry personnel, and mechanisms in place to ensure that these responsibilities are fulfilled.

8.4 Program Delivery Criterion

The institution fosters good teaching-learning process

Evidence

Established policies and procedures to foster and evaluate good practice;

Delivery modes and teaching-learning methods appropriate to the level and contents of the program as well as to the class size of learners;

Modular delivery of courses to enhance flexibility to learners and promotion of transfer of credits;

Delivery modes designed to address a diversity of student learning styles and needs;

Appropriate access for students to learning resources; and

Systematic and valid student evaluation of teaching practice linked to ongoing quality improvement.

8.5. Quality of Staff Criterion

Appropriately qualified staff are employed and retained to enable quality provision of programs.

The institution actively encourages the development of all staff.

A well integrated performance reward system to foster productivity,

Explanation

68

The quality of the academic staff is the key to the quality of the institution. There will be no quality without qualified and competent staff. Looking at the criteria, we have to look at:

The size of the staff and their qualifications

The staff/HR management. Evidence

The institution has sufficient academic and support staff with appropriate qualifications and experience to meet program development and delivery requirements.

Appropriate and effective staff induction processes;

Systematic performance review of all staff, which is used to recognize excellent practice;

Effective planning and monitoring of outcomes of appropriate staff development opportunities that have been agreed and negotiated with each staff member. Planning for professional development should have the overall objective of enhancing student learning;

The availability of sufficient resources to enable planned staff development to take place; and

Availability of induction programs, performance review and professional development opportunities to part-time and temporary staff.

8.6. Students information and admission Criteria

Prospective and continuing students have effective guidance to assist them in making informed decisions on their programme of study

Entry and selection criteria are appropriate for the level of each programme, are well publicised, and are applied consistently.

The institution has implemented effective credit transfer policies and procedures

The institution makes appropriate provision for the recognition of prior learning and current competency for its enrolled students.

Explanation It is important to attract the right students and to select the students that can finish the training. Evidence

The provision to students of timely and accurate information on credit transfer, cross crediting and opportunities for the recognition of prior learning; and

The provision of assistance to students in making informed and appropriate choices in the selection of programs.

Clear and appropriate admission and selection criteria, which are published, accessible to students and include no unreasonable barriers;

Records of student selection decisions that demonstrate consistent application of the criteria.

Absence of unreasonable barriers to credit transfer and cross crediting;

The provision to students of timely advice and decisions on credit transfer and cross credits.

the provision and promotion to students of effective services for recognition of prior learning and recognition of current competency assessment appropriate to the program of study.

8.7. Facilities and Infrastructure Criterion

The provision of physical resources to meet academic requirements is effectively planned and reviewed.

Explanation

69

Facilities and resources should be in line with the formulated goals and aims and with the designed program. Facilities are also connected to the teaching/learning strategy. For example, if the philosophy is to teach in small working groups, small rooms must be available. Computer-aided instruction can only be realised with enough computers for the students. The main learning resources consist of books, brochures, magazines, journals, posters, information sheets, Internet and intranet, CD-ROMs, maps, aerial photographs, satellite imagery and others. Evidence

Effective planning and provision of physical resources, which are accurately assessed and integrated with academic priorities and program requirements;

Availability of sufficient and appropriate physical resources to enable students to attain the required program outcomes; and

Physical resources that are reviewed and maintained to ensure that they are fit for the purpose. Teaching rooms

Are enough lecture halls, seminar rooms, laboratories, reading rooms, and computer rooms available? Do these meet the relevant requirements?

Is the library sufficiently equipped for education?

Is the library within easy reach (location, opening hours)?

Are laboratory facilities and support staff sufficient?

Do the laboratories meet the relevant requirements?

Didactic aids and tools

Are sufficient audio-visual aids available?

Are there enough computers? Appropriate and enough computer programs (computer-aided education, maths programs, design programs, etc)?

To what extent do the facilities/infrastructure promote or obstruct delivery of the program?

Is the total budget for aids and tools sufficient? 9. Community Contribution Criterion

The institution has clear guidelines for consultancy and community outreach

Explanation The institution is not only responsible for teaching and (if applicable) doing research. It is also responsible for serving society. Consultancy involves a broad range of activities. In general, the term consultancy covers the provision of professional advice or services to an external party for a fee or other non-monetary consideration. Among other things, guidelines on providing consultancy cover the following:

Policy objectives

Policy on key policy principles, compliance, accountability framework, legal and financial protection, conflicts of interest

Procedures for the contribution to society and the community

Procedures for institutional/academic consultancy

Procedures for private consultancy

Evidence

What role does the institution play in the local, national and international community

What are the key activities, which of these lay outside normal teaching or research? How do they relate to the institution’s mission?

70

What are the non-profit activities of the institution?

Is there a clear policy on consultancy and the contribution to society and the community?

How is the income from consultancy regulated? 10. Benchmarking

Criterion

The institution uses the instrument of benchmarking for analyzing the quality of its core activities and its management

Explanation The UNESCO definition of benchmark is: A standard, a reference point, or a criterion against which the quality of something can be measured, judged, and evaluated, and against which outcomes of a specified activity can be measured. The term, benchmark, means a measure of best practice performance. The existence of a benchmark is one necessary step in the overall process of benchmarking. NACTE defines benchmarks as: A systematic and continuous measurement process; a process of continuously measuring and comparing an organization business process against business leaders anywhere in the world to gain information which help the organization to take action to improve its performance12 Benchmarking is a process that enables comparison of inputs, processes or outputs between institutions (or parts of institutions) or within a single institution over time. It is important for a n institution to compare its functioning with equivalent institutions in the country, the region and internationally. Evidence • Is the institution using the instrument of benchmarking? How is it using the instrument? • Does the executive management use the collected information? • What is done with the benchmarking?

11. Quality Assurance

Criterion

The institution has an efficient internal quality assurance system

Explanation A robust and well functioning system of internal quality assurance is necessary to deliver quality and to provide “consumer” protection. Evidence To find evidence that the IQA system is working well, one has to evaluate the IQA system in the framework of the institutional self-assessment. For the self-assessment of the IQA system, see chapter 5 of the handbook. When the self-assessment will be followed by institutional accreditation and there has not yet been a Quality Audit, the institution must conduct a self-assessment of the QA-system too.

12NACTE, Benchmarks for Comparing Performance Across Courses and Across Technical Institutions, January

2004. The publication offers a lot of information about the benchmark process.

71

12. Achievements Criterion

An institution has the means and the opportunity to check whether the achievements are in line with the expected

outcomes.

Evidence • Are the achieved outcomes (our graduates; the research output; services to society) in line with

the formulated goals and aims? • How does the institution check that it achieves what it wants to achieve? • If the achievement is not satisfactory, what remedial action does the institution take? 13. Stakeholders Satisfaction Criterion

An institution has a structured method for obtaining feedback from the stakeholders

Explanation

After analyzing the mission of the institution, the management structure, policy and strategic planning, human resource management and the core activities, the institution has to analyse the satisfaction of all stakeholders. What do they think about the performance? How do we know that? Evidence

Is regular student evaluation carried out? How is it done? Is it adequate?

What is done with the results of student evaluations?

Does the TI have an insight into the opinion and feedback of graduates when they are employed?

Are the complaints or positive feedback received from alumni used to adapt the programs?

Are there any structured contacts with employers and the labour market for obtaining feedback?

How do the employers appreciate graduates?

Are there any specific complaints?

Are specific strengths appreciated by employers?

Does the TI have any tools to obtain feedback from society?

6.3 Strengths/weaknesses Analysis

The self-assessment is followed by a strengths-weaknesses analysis. At the same time, this serves as a check to see how far the institution is in compliance with the given criteria. This is best done, using the checklist (see Appendix 2) and summarizing the outcomes in table 6.The different aspects of the quality of the institution will be assessed on a scale 1-7. The marks have the following meaning: 1 = absolutely inadequate; immediate improvements must be made 2 = inadequate, improvements necessary 3 = inadequate, but minor improvements will make it adequate 4 = adequate as expected 5 = better than adequate 6 = example of good practice 7 = excellent

72

The overall assessment of the different aspects is based on the scores given to each sub-aspect in the category. But, of course not all sub-aspects have the same weight. This means that you cannot calculate mathematically an average. You have to balance the various sub-aspects and judge the importance of each of them. Aspects with positive outcomes may compensate for some negative ones. Summarizing the outcomes per category is not a mathematical enterprise, nor ticking boxes. One has to balance the importance of each criterion. The same counts for the overall judgment of the quality of the institution. One cannot calculate the average of the categories, but look at the importance of the category. Community contribution and benchmarking are less important than Educational effectiveness or Quality Assurance. Summarizing the outcomes for each category shows the strengths and weakness of the institution. See table 13 Table 13: summary of quality aspects in institutional analysis

no 1 2 3 4 5 6 7

1 Requirements stakeholders

2 Vision, Mission, , aims, objectives; integrity

3 Policy plan

4 Governance & administration

5 Human resource

6 Funding & Financial management

7 Facilities & infrastructure

8 Educational effectiveness

9 Community contribution

10 Benchmarking

11 Quality assurance

12 Achievements

13 Satisfaction stakeholders

Overall judgment

Summary of strengths Summaries the points that the institution considers to be its strengths and mark the points that you are proud of. Summary of weaknesses Indicate which points the institution considers to be weak and in need for improvement. Also indicate what you are going to do about this. For the improvement of the weaknesses, the institution must include actions in the Quality improvement plan. Only with a clear follow up and a quality and action plan, the investment in the self-assessment and the SAR make sense.

6.4 The Self Assessment Report (SAR) After finishing the SWOT-analysis of the institution, one will write down the outcomes of the assessment in a Self Assessment Report (SAR). The SAR is an important document. At the one hand it contains the basic information for the Institutional Accreditation by the NACTE expert team that will come and assess the institution. On the other hand it is the basic document for the institution for the formulation of a quality improvement plan for the coming years. The content of the SAR follows the lines of the aspects, discussed during the self-assessment process. For each aspect to be treated one should:

Describe clearly the state-of-the art. An outsider must understand the situation.

Analyse the situation. What is your opinion about it? Satisfied or not? If not, why not?

Describe how far you meet the formulated criteria. What evidence can you provide?

Describe the weakness and the strengths.

73

Content of the self-assessment report Table 14 defines the content of the self-assessment report. Be sure to discuss the report within the institution and ensure that everybody is able to recognise himself/herself in this picture. Table 14: Content of a Self-Assessment Report at Institutional Level.

Introduction

How was the self- assessment carried out?

Composition of the SAR team

Short description of the institution and its departments

The specific profile of the institution

Chapter 1: Requirements and expectations of stakeholders

Chapter 2: The organisation 2.1 Vision, Mission, Aims and Objectives 2.2 Institutional Integrity 2.3 The Policy Plan 2.4 Governance and Administration 2.5 Human Resource 2.6 Funding and Financial Management 2.7 Facilities and Infrastructure

Chapter 3 Educational Effectiveness 3.1 The Programs 3.2 Student assessment 3.3 Field Attachment/Workplace Training Components 3.4 Program delivery 3.5 Quality of the staff 3.6 Students admission 3.7 Facilities and Infrastructure

Chapter 4: Community Service

Chapter 5: Quality Assurance 5.1 Internal Quality Assurance 5.2 Benchmark

Chapter 6:Achievements of the Institution 6.1 Graduates/Graduate Profile 6.2 Research output (as far as applicable) 6.3 Community services

Chapter 7 Strengths-weaknesses analysis 7.1 Summary of Strengths 7.2 Summary of Weaknesses

Quality Plan for the Coming Years

6.5 Preparation for the Institutional Accreditation The self-assessment of the institution will be followed by a institutional accreditation by NACTE. To prepare the institution for the accreditation, the following is important:

Ensure that the SAR is on time at NACTE;

Ensure that the SAR is informative, analytical and honest. The SAR is the basic information for the audit team;

Ensure that all staff and students knows about the SAR;

Ensure that during the site the most important documents, showing evidence that you are meeting the criteria are displayed for the audit team; and

Ensure that everybody is available for interviews during the site visit.

74

6.6 The Institutional Accreditation Institutional accreditation will establish stakeholder confidence that provisions (input, process and outcomes) fulfils expectations or measures up to threshold minimum requirements There are three possibilities for Institutional accreditation:

Accreditation for registration. This is compulsory for newly established TIs.

NACTE takes the initiative (or for accreditation or re-accreditation) and announce that an Institutional Accreditation will be organized for the institution.

The institution takes the initiative and invites NACTE to organize an institutional accreditation.

In all cases, the Institutional Accreditation process starts with an institutional self-assessment by the institution and by the programs. The institutional self-evaluation will be done according the guidelines, given in this chapter. The basis for the Institutional/program assessment is the outcome of the self-assessment process, the SAR. The time given for the self-assessment is 6 months. Because the members of the expert team need time for preparation and for reading the SAR, the Institutional Accreditation can only take place 7 months after the formal announcement. If there is no separate quality audit planned, the quality audit is included in the institutional accreditation. The institution will analyse the internal quality assessment, using the guidelines for the Internal Quality assessment system (chapter 5 of the handbook). The institutional accreditation will also be used for program accreditation. If a TI offers three or fewer programs, the programs will be involved in the Institutional assessment. The programs concerned conduct at the same time a self-assessment according to chapter 7 of the handbook. The assessment team must include expertise in the field of the programs. Provides a TI more than three programs, then all the programs conduct a self-assessment, but for assessment during the site visit only two will be selected. After the formal announcement, NACTE start the preparations for the Institutional Accreditation. These concerns:

If needed, training of the TI in conducting the self assessment(s)

Appointment of the members of the expert team

Training of the members of the expert team

Organization of the site visit

6.6.1. The Expert Team An assessment team, commissioned to conduct the Institutional assessment, combined with a program assessment, will have at least 6 members + secretary. Membership of an expert team should include:

A chairperson, totally independent and unconnected with the institution to be assessed. The chair should have experience with management structures in Technical Institutions and have experiences with the developments that have taken place over the last few years

An expert in organisation and management.

3 experts in the specific field of the institution. One coming from the academic world, one from the labour market area or from a professional body

An expert in quality assurance There are some conditions that members of the expert team have to meet:

Members should act independently

There should be no conflicts of interest. Members should stand to gain nothing from their verdict

75

Members must be accepted by the institutions to be assessed. This mean that NACTE has to check if the institution has any serious objection against one of the members. It is up to NACTE to decide if the objection is justified.

What is the expert team looking for? The expert team assesses the quality of the institution (including the internal quality assurance system and the quality of the chosen programs). The SAR (S) will provide detailed information. During the site visit, the team will look for evidence with the following questions in mind:

Provide the organisation and management satisfactory confidence in the institution?

Is there enough evidence for meeting the threshold standards?

Are the departments of the institutions able to deliver the programs at offer?

Is the Quality Assurance system satisfactory

Is the quality of the programs at offer satisfactory? For the assessment of the quality of the institution, the expert team will use the model for analysis of the quality of the institution (see figure 10).For the assessment of the programs, the expert team will use the model for analysis of the quality of programs (see figure 11 in chapter 7) Especially the 3 experts in the discipline will be responsible for the reports about the quality of the programs. They will use the guidelines in section 7.7, but adapted it to the specific situation the expert team is working.

6.6.2 Preparatory Work of the Expert Team As soon as the institution has sent the self-assessment report(s) to the expert team, the members will study the report(s) carefully before the team comes together in a preliminary meeting. The expert team will meet before the planned site visit for training and to prepare itself for the assessment. The meeting will be held on the day before the site visit begins. The topics in the meeting are:

Discussion on the frame of reference, used for making judgments about the institution (= the analysis model for the quality of the institution, figure 10) and discussion on the frame of reference, used for making judgments about the programs (= the analysis model for the quality of a program, figure 11 in chapter 7) It is important that all members of the expert team understand the threshold requirements set for an institution. Therefore it is important to discuss the model for the analysis of the quality of an institution (figure 10). Concerning criteria we have to keep in mind that there are no absolute and objective criteria and standards. The criteria for assessing the quality given in the handbook of the TI’s are based on the criteria as formulated by external quality assessment agencies, e.g. European, American, Asian, Australian and South African accrediting bodies, among others. After studying many sets of standards and criteria, a common denominator has been formulated. In general, one may say that the formulated criteria can be seen as the minimum criteria. The Academic standards, formulated by NACTE are included in the model. Table 12 shows the number of the cells of the analysis model, where the NACTE standards are discussed.

Discussion on the self-assessment report(s) During the meeting, the team will discuss the SAR(S) and formulate questions to be asked during the site visit.

6.6.3. The Site Visit After the preparatory meeting, the expert team will start the site visit. The SARS are the basic source of information for the expert team and should provide the basic information. But other sources should also be used:

Interviews with management, QA officers, staff and students

Investigation of documents providing evidence of meeting the criteria

76

The site visit will, at average, take 2.5 days. The time depends on the size of the institution. NACTE had confirmed the program for the site visit in consultation with the institution according to a given format (see table 15). Table 15: Draft program for institutional accreditation

Time Activity

Day 1 15.00 18.00

Expert Team meets in the hotel for information about their task and discussion on:

The SAR

Formulation specific questions

The program of the site visit Welcome by management of the institution; joined diner

Day 2 09.00– 17.00

Institutional experts Interviews with:

the writers of the self-assessment report

management

QA-office

staff members

students

Studying documents

The 3 discipline exerts Interviews with:

the writers of the self-assessment report

management

QA-office

staff members

students

Studying documents

19:00-20.00

Dinner for the expert team Short meeting for discussion on the finding of the day and for setting the program for the next day

Day 2 9:00 - 11:00

Interviews with head of department(s) Interview management

11:00 - 13:00

Visit facilities & infrastructure

13.00 – 16.30

Formulation findings

16.30 - 17.00

Feedback to the institution

17.00

End of the visit

Beforehand, appointments will have been made with whom the expert team like to talk. The interviews start with a discussion with the writers of the self-assessment report. In this interview, the team can ask for clarification of any obscurities and explanation of any topics that are not totally clear. The interviews with the students are purposely planned to take place before the interviews with the staff members. The students are a very rich source of information, but the information needs to be checked and tested against the ideas of the staff members. Student interviews are important to get an insight into the study load, the didactic qualifications of the staff, the coherency of the program, to find out if they are acquainted with the learning outcomes/objectives, the organisation of the curricula and the facilities. These student interviews should be held in the absence of staff members, so that they can speak freely. The size of the student groups is ideally about ten each time. The composition of the student panels requires special attention. It is important that the group is as far as possible representative of the whole student population in that field, i.e. that it not only includes the good students, but also the less gifted ones. It is better not to leave the invitation of students to the faculty or the staff. The best way is to ask a student organisation (if there is any) to nominate the students. If there is no such organisation, the expert team will invite students at random. Interviews with staff

77

members will be used for discussion on the content of the curriculum, the goals and objectives/expected learning outcomes. "Why and how did you choose this program?". Other topics to be discussed include continuous assessment, end of semester examinations, field attachment etc. Other interviews will be held with members of a curriculum committee and with members of the committee responsible for examinations. During the interview with the curriculum committee, the question of how the curriculum is kept up to date will be discussed as will the question on how innovations are planned and realised, etc. The interview with the examination committee must clearly show how the quality of the examinations and degrees is assured. In the program time is allocated for looking at the facilities: Lecture halls, working group rooms, laboratories, practical rooms, libraries, etc. When it concerns an Institutional assessment combined with program assessment, the team will be split in two groups during the day of interviews (Day 2). The 3 experts with the institutional expertise will held interviews to find information to check the Institutional SAR, the 3 discipline experts will held interviews to find evidence for assessing the program SARS. When the institution is a really small one, the team may decide not to split.

6.6.4 Formulating the Findings

The afternoon of the second day is used for drawing up the findings. There are about 3.5 hours available for this difficult task. The best method is to start with completion of the checklist (see appendix 2) by each member individually. The team members are requested to give a mark between 1 and 7 for the various aspects. To have some idea of the value of the figures, bear the following ideas in mind:

Score 1-2 when you believe this aspect should be considered as critical. The institution has to take action. Something has to be done and cannot wait.

Score 3 when you believe this aspect is unsatisfactory. It must be improved, but does not directly threaten the quality.

Score 4 when you believe the situation is satisfactory. The TI may be satisfied, but there is no reason to be proud.

Score 5 when you believe this topic can be assessed as more than satisfactory, but not excellent.

Score 6 when you believe this topic can be assessed as more than satisfactory and can be seen as an example of good practice.

Score 7 when you believe this topic can be assessed as excellent. The institutions can be proud of it and it is certainly a strong point.

The overall assessment of the different aspects is based on the scores given to each sub-aspect in the category. But, of course not all sub-aspects have the same weight. This means that you cannot calculate mathematically an average. You have to balance the various sub-aspects and judge the importance of each of them. Aspects with positive outcomes may compensate for some negative ones. Summarizing the outcomes per category is not a mathematical enterprise, nor ticking boxes. One has to balance the importance of each criterion. The same counts for the overall judgment of the quality of the institution. One cannot calculate the average of the categories, but look at the importance of the category. Community contribution and benchmarking are less important than Educational effectiveness or Quality Assurance. Summarizing the outcomes for each of the 13 categories shows the strengths and weakness of the institution (table 16)

78

Table 16: summary quality aspects institutional analysis

SN 1 2 3 4 5 6 7

1 Requirements stakeholders

2 Mission, vision, aims, objectives; integrity

3 Policy plan

4 Governance & administration

5 Human resource

6 Funding & Financial management

7 Facilities & infrastructure

8 Educational effectiveness

9 Community contribution

10 Benchmarking

11 Quality assurance

12 Achievements

13 Satisfaction stakeholders

Overall judgment of the institution

After completing the list, the resting time will be used to draw up an inventory of the topics to be treated in the oral presentation.

6 .6.5 The Exit Meeting The oral presentation to the management of the institution at the end of the visit holds a special position in the process. Sometimes, findings and conclusions are not really suitable for the report, but the team would like to make a critical statement about them. In that case, the oral presentation can be used to formulate strongly worded recommendations. In order to do justice to this principle, the oral presentation is not public; the team reports to the management board. The chairperson should stress that this is an interim report; some conclusions may change during the final discussion on the report. It is advisable not only to mention weaknesses, but also strengths.

6.6.6 The Expert Team's Report After the site visit, the chairperson and secretary will write a first draft of the report, using the completed checklists and the minutes of the oral presentation. Table 17 gives an outline of the content of the assessment report. For each topic the following format will be followed:

What is said in the SAR about this topic?

What are the findings of the expert team?

Conclusion

Recommendations Making recommendations, the expert team must keep in mind:

To summarize the recommendations at the end Prioritize the recommendations as far as possible Ask itself if it is a realistic recommendation The expert team must make clear for whom the recommendations is made The team should not ride hobbyhorses If there are many recommendations, the team has to ask itself if they all are important

Table 17: Content of an Assessment Report of an Institutional Accreditation

Introduction

Background of the Institutional Accreditation: Why is the assessment done?

Composition of the Expert Team

79

Short Description of the Institution

Chapter 1: Requirements and Expectation Stakeholders

Chapter 2: The organisation 2.1 Vision Mission, , Aims and Objectives 2.2 Institutional Integrity 2.3 The Policy Plan 2.4 Governance and Administration 2.5 Human Resource 2.6 Funding and Financial Management 2.7 Facilities & Infrastructure

Chapter 3 Educational effectiveness13 3.1 The Programs 3.2 Student Assessment 3.3 Field Attachment/Workplace Training Components 3.4 Program Delivery 3.5 Quality of the Staff 3.6 Students Admission 3.7 Facilities and Infrastructure

Chapter 4: Community Service

Chapter 5: Quality Assurance 5.1 Internal Quality Assurance 5.2 Benchmark

Chapter 6: Achievements of the institution 6.1 Graduates/Graduate Profile 6.2 Research Output (as far as applicable) 6.3 Community Services

Chapter 7 Strengths-Weaknesses Analysis 7.1 Summary of Strengths 7.2 Summary of Weaknesses 7.3 Summary of the Recommendations

The 1st draft of the report will be discussed with the team members. The 2nd draft will be sent to the institution for comment. The comments should concern only factual errors and inaccuracies, not the differences in opinion. The audit team will decide what to do with the comments. If the institution disagrees with the way the assessment is done or disagree with the findings of the audit team, it may contact NACTE.

6.7 NACTE and the Institutional Accreditation The final report of the audit team will be sent to the Council. The expert team gives its advise about accreditation or not accreditation in a side letter. The Council will check if the Institutional Accreditation has been done according the rules and see if the report contains all information needed. If the assessment report is accepted, the Council will follow the advice of the expert team. The decision might be:

Conditional Accreditation: The Council grants conditional accreditation to an institution where the Council is of the opinion that there are certain requirements to be fulfilled by the institution. The term, in which the condition has to be fulfilled, is given. NACTE will check if the conditions are fulfilled. Where the institution fails to fulfill the requirements within the given time, then the Council may:

Cancel the conditional Accreditation granted to the institution (and so the registration); or

After being satisfied that, there was reasonable cause for failure to comply, extend the time for conditional Accreditation to such other time, as it may be determined.

Full Accreditation: The Council grants full accreditation to institutions, which meet adequately the accreditation requirements.

13As far as specific programs are involved in the Institutional accreditation, this chapter will provide the request information for each program involved.

80

Accreditation is valid for 5 years. The institution is expected to send after 3 years a mid term report to NACTE When the institutional assessment included the assessment of specific programs, the Council grant accreditation (or conditional accreditation) for the programs, based on the findings and advise of the expert team. If the institution disagrees with the way the assessment is done, disagree with the findings of the expert team or the decisions it may contact NACTE. NACTE will install an independent committee that will investigate the complaints. The verdict of the independent committee is binding both for NACTE as the institution.

81

CHAPTER 7

7. How to Discover the Quality of the Training Programme?

NACTE has the task to validate and approve the programmes of an institution. An instrument for assessing the quality of the training programmes is Programme Accreditation. The input for the Program Accreditation is the self-assessment report (SAR) of the department and the program involved. Looking at the huge amount of programs, it will not always be possible to organise a program assessment with site visits for all individual programs, therefore NACTE might use the following modalities:

Program assessment is included in the institutional assessment;

The Institutional Accreditation showed some doubts about the program(s) in the institution and NACTE will verify the quality by an individual Program Accreditation.

NACTE takes the initiative and announces a nationwide Program Accreditation for a certain type of programs ( e.g. Accountancy or Nursing). An advantage of the nationwide approach is that there is a possibility of benchmarking of the programs in the different institutions. This comparative character of the Program Accreditation makes it possible to have a good overview of the state-of-the-art in that discipline/module. The programmes involved are expected to conduct a self assessment. An expert team, installed by NACTE, will assess the quality without site visit

The institution ask NACTE to organise a programme accreditation The basic information will always be the SAR. Even when no Program Accreditation is foreseen, it is worthwhile for the institution to conduct a self-assessment of the department and its program(s) to find out the weakness and strengths of the department. This chapter offers the instrument of self assessment of the programme(s)

7.1 An Analysis Model for the Self-Assessment of The Department and Its Programs In order to map the quality in a self-assessment we need a clear model to prevent looking at some aspects and ignoring others. Figure 11 shows a model for the analysis of quality of our programs. For the critical self-assessment, this analysis model will be used Concerning criteria we have to keep in mind that there are no absolute and objective criteria and standards. The criteria for assessing the quality given in the handbook are based on the criteria as formulated by external quality assessment agencies, e.g. European, American, Asian, Australian and South African accrediting bodies, among others. After studying many sets of standards and criteria, a common denominator has been formulated. The criteria for validation of Curricula, formulated by NACTE 14are included in the model. For each aspect, the following format is used:

The name of the cell in the model is given, just as the title of the aspect that will be treated, e.g. Learning Outcomes

The Criterion concerning that aspect is given in a box. The criterion shows what commonly is expected form an institution.

If NACTE is not mentioning a specific criterion, but the criterion is seen internationally as important, the criterion is mentioned.

14 NACTE, Procedures for Curriculum Approval and Validation, August 2004.

82

Looking for evidence A set of questions and/or statements is drawn up to help the department to find evidence if the criteria are being met. Please be aware of the following as far as these questions are concerned:

The questions-set are not meant as a compulsory list that has to be completed. It is not a questionnaire to be answered point by point. It must be seen as a tool to collect information and evidence. The questions are to be seen as reminders.

The statements will help you to demonstrate that you are meeting the criterion. The basic rules to apply in self-assessment are: • All aspects (segments of the model) need to be discussed. It is not possible to make a selection. • For each aspect the following steps are to be taken:

• Description • Analysis • Evidence for meeting the criteria • Formulation of strengths and weaknesses • Action plan for improvement

Figure 11: An analysis model for the self-assessment of Teaching and Learning

Departmental

capacity

1.

Mission

&

objectives

2

Leadership

&

Administration

3.

Financial

Resources

4.Requirementsstakeholders

5E

XPECTE

DLEA

RNING

OUT

COMES

6Programmespecifica on

7Contentof

theprogramme

8Organiza on

theprogramme

9Didac cconcept

10

StudentAssessment

11Quality

Academicstaff

12QualitySupportstaff

13ProfileOf

students

14StudentAdvice

&support

15Facili es

&Infrastructure

16Student

evalua on

17Curriculumdesign

18Staffdevelopment

ac vi es

19Bench-marking

20

Achievements:

The graduates

21.Sa sfac onstakeholders

22Communityservice

Curriculum

Process

Input

Quality Assurance

83

7.2. The Self-Assessment: The Quality Aspects to Be Assessed A: THE DEPARTMENT 1. Mission and Objectives Criteria

The mission of the department is consistent with the mission of the institution. It is applied based on specific

objectives and requirements associated with the nature of specialization. It should be clearly stated and be influential in guiding, planning and implementing the program.

The mission of the department is the guide for the processes of planning and decision-making. It is carefully employed while preparing and reviewing the academic programme.

The programme and its objectives are periodically reviewed in light of performance assessment and the interaction of implementers of programs with their surroundings.

Explanation The program should have a mission, objectives and learning outcomes that are consistent with the mission and objectives of the department and the training institution, and that reflect what is planned for students to gain from studying the program. The mission and objectives should guide the processes of planning and decision-making. Also the mission is employed while preparing and reviewing the program and in the entire program's activities and it is considered a reference when assessing its effectiveness. Evidence

The department has a mission and objectives prepared through participation of all concerned persons. It is publically known for beneficiaries inside and outside the department.

The mission and objectives of the department reflect its nature, educational role, social responsibility and future requirements.

The mission and objectives of the department are consistent with the mission and objectives of institution.

2. Leadership and Administration The administration of the department should reflect appropriate balance between commitment to accountability before the senior management of the institution and the flexibility needed for achieving the specific requirements of the academic program. The concerned parties should be involved in the processes of planning (such as students, specialized bodies, representatives of industry or commerce or the external entity benefitting from the graduates, and staff members) in developing the objectives and purposes and reviewing the achieved results and responding to them. 2.1 Strategic Planning Criteria

The department plans to achieve its mission and objectives and assess the extent, way and quality of achieving its mission and objectives. Results of the annual assessment are used in the processes of continuous comprehensive planning and improvement.

The department ensures the participation of all councils, committees, administrations, faculty members and students in planning and assessment.

The department periodically reviews the assessment processes and planning activities to ensure their effectiveness.

84

Evidence

The department pursues strategic planning based on complete analysis of current situation, identifying the gap between it and its desired measurable objectives, and prepares plans to fill the gap. It also searches for the suitable funding so they become achievable.

The department evaluates its activities including teaching, learning, research and community service in light of its mission and objectives. Results of the assessment are used to improve the quality of the academic programs, services and activities.

The department provides the necessary resources for the processes of planning and assessment. 2.2 Head of the Department/Program Criterion

The head of the department decides on the program's priorities, prepares development plans, supports administrators and staff members and the environment of teaching and learning as to achieve the mission and objectives of the program. There is also clear specification of the responsibility of the academic program.

Evidence

The head of department is appointed for each program according to the laid down laws/regulations/procedures. He/she is responsible of managing the academic and administrative affairs of the department and matters of research (if applicable) and community service.

The head of the department provides all the required documents to the departmental meeting when presenting the academic and administrative matters that need the approval of the department. He/she also ensures that the recommendations offered are presented in a way that shows the different options and the results for each of them.

2.3 Administration

Criterion

The department is responsible in the first place for quality and credibility in the implementation of the program. It should give priority for the effective development of the program in order to ensure the interests of the students and community it serves.

Evidence

The tasks, responsibilities and key requirements for governing the department and program and the administrative procedures are clearly set and published.

The department/ program periodically assesses its performance and reviews its policies whenever is necessary to ensure the effective undertaking of responsibilities.

3. Financial Resources

3.1 Financial Planning and Budgeting Criterion

The process of financial planning is directed to achieve the objectives and priorities of the program.

Explanation The financial resources should be sufficient for the adequate implementation of the academic program. Requirements of the program should be introduced sufficiently and in advance to prepare

85

the budget according to scientific bases. The budget should allow for long-term planning at least for three years. Sufficient amount of the flexibility needed for the effective administration and responding to unexpected events should be available. This flexibility should be associated with suitable mechanisms for accountability, transparency and report submission. Evidence

Requirements of the program are inserted into the annual budget of the institution in consultation with the procurement administration, and they are reviewed carefully before approving them.

The institution has a policy for linking the financial planning with its strategies and objectives. 3.2 Financial Management Criteria Financial affairs should be effectively managed in order to ensure balance between the required flexibility in units

and the central accountability and responsibility.

The budget and accounts of the department are managed by a specialized financial manager.

Evidence

The program has a sufficient budget.

The head of the department or the program coordinator participates in preparing the budget and is responsible for expenditure within the frame of the approved budget.

Revenues and expenditure are monitored by the budget committee of the department.

B: The Quality of the Program/Curriculum 4. Requirements of Stakeholders Criterion

The department, responsible for the program has a clear idea about the relevant demands and needs of all stakeholders, based on situation analysis

Explanation . Tertiary Education has many stakeholders and all stakeholders have their own ideas about quality:

The government or the state

The employers

The academic world

The students

The parents

The Society at large

Professional bodies Professional Association Trade Unions

Each stakeholder will appreciate different aspects of quality and because all stakeholders have their own ideas and expectations, we may say that Quality is a matter of negotiating between the institution and the stakeholders. In this negotiation process, each stakeholder needs to formulate, as clearly as possible, his/her requirements. The organization (department) as supplier of the academic training must try to reconcile all these different wishes and requirements. As far as possible, the requirements of all stakeholders should be translated into the expected learning outcomes of the program. Evidence

Does the department have a clear idea about the requirements set by the government?

86

How does the department know the needs and requirements of the labour market?

Did the department conducted situation analysis or is the knowledge about requirements of stakeholders based on situation analysis, done by others (e.g. NACTE)?

How does the department analyse the needs and requirements of the students/parents

How does the department analyse the needs and requirements of the society?

How does the department n balance the requirements of the different stakeholders? 5. Expected Learning Outcomes Criteria

The program/curriculum has clearly formulated learning outcomes (knowledge, skills, attitude) reflecting the relevant demands and needs of all stakeholders, especially the labour market

The expected learning outcomes are competence based

The program has formulated learning outcomes on entrepreneurial skills

The program has a learning outcome, enabling the student to get a gender sensitive attitude

Explanation Before we can assess the quality of our program, we need to know clearly what we expect that students will learn. We must formulate very clearly the learning outcomes. Students come to the TI to learn something. Therefore, we have to formulate very clearly what we expect the student to learn and what we expect our graduates have learned in terms of knowledge, skills and attitude. The expected learning outcomes form the starting point for the self-assessment. We have to distinguish between generic academic skills and discipline specific skills. Special attention must be paid to the formulation of learning outcomes, promoting entrepreneurial skills and learning outcomes promoting a gender sensitive attitude. Some definitions:

Aim: An aim is an overall specification of the intention or purpose of a programme of study or institutional mission or policy.

Learning outcome: A learning outcome is the specification of what a student should learn as the result of a period of specified and supported study (The graduate will be able to…..)

Program Objective: An objective is a specific statement about what we expect from the program (the program will contribute to a better understanding…)

Evidence

What are the expected learning outcomes (ELO) of the program?

Are the ELO’s based on a Job profile/qualification standard, formulated by NACTE?

Are the ELO’s in line with the NTA level?

How does the ELO fit into the mission of the institution as a whole?

Does the labour market express specific requirements for graduates to meet? Is there a well-defined job profile for the graduates of this program?

Do the formulated learning outcomes offer the student the possibility to acquire entrepreneurial skills?

Do the learning outcomes enable the students to acquire a gender sensitive attitude?

How are the ELO’s made known to the staff and the students?

To what extent do we think the ELO’s have been realised?

Do we have any plans to adjust the ELO’s? Why?

87

THE PROCESS (CELL 6-10 ) 6. The Program Specifications (Or Program Description) Criteria

The department publishes, for each program they offer, a program specification/description which gives the intended learning outcomes of the program in terms of:

knowledge and understanding that the students will have acquired upon completion of the program

cognitive skills, such as an understanding of methodologies or ability in critical analysis

module specific skills, such as laboratory skills, clinical skills, etc.

Explanation The formulated learning outcomes must be translated into the program. It is important that the objectives are well known to everybody. Therefore, the department will publish a program specification or description for each program they offer. The program specification is:

As a source of information for students

As a source of information for employers, particularly about the skills and other transferable intellectual abilities developed by the program.

As a source for professional and statutory regulatory bodies, accrediting technical training programmes, leading to entry into a profession or other regulated occupations.

Evidence

Does the department have a clear program/curriculum specification/description?

What is the content of the description?

Is the description known to staff and students? 7. The Content of the Program/Curriculum Criteria

The program shows a balance between specialist contents and general knowledge and skills.

The program takes into account and reflects the vision, mission, aims and objectives of the institution.

The objectives and expected learning outcomes of the program are explicit and are known to staff and students.

The program shows the expected learning outcomes of the graduate. Each course should clearly be designed to show the expected learning outcomes of the course. To obtain this, a curriculum map must be constructed. A program map must be available.

The program shows evidence of attention to develop entrepreneurial skills

The program shows evidence of attention to develop a gender sensitive attitude

Explanation The content of a program is closely linked to the translated goals and aims. The formulated learning outcomes decide the content of the program. Furthermore, the program must be coherent and up-to-date. For each course it should be clear how it contribute to the achievement of the overall learning outcomes. Evidence

Do the contents of the program reflect the expected learning outcomes?

Can the program be considered as adequate for achieving the expected outcomes?

Are the modules in the program clearly interrelated?

Is the program coherent?

Is there a balance between specific and general modules?

Do the modules demonstrate a growing complexity?

88

Is the program content up-to-date? 8. The Organisation of the Program Criteria

The programme is designed in such a way that the module matter is integrated and also strengthens other courses in the programme

The programme shows range, depth and coherence of the courses

The programme is organised in a modular way

Evidence

Why is this program structure chosen?

How are the modules organised?

Has the program been changed structurally over recent years? If so, why?

Were any requirements specified on the internal coherence of the courses? Who set these requirements?

Are bottlenecks experienced within the program?

Is the instruction/teaching provided by other departments satisfactory?

Is the chosen academic year structure (semester) adequate? What is the opinion of those involved?

9. Didactic Concept and Teaching/Learning Strategy Criteria The department has a clear didactic concept

The didactic concept is student oriented. Hence, the conception of teaching is the facilitation of learning.

In promoting responsibility in learning, teachers should:

create a teaching-learning environment that enables individuals to participate responsibly in the learning process

provide curricula that are flexible and enable learners to make meaningful choices in terms of module content, program routes, approaches to assessment and modes and duration of study

Explanation Didactic concept means the strategy developed by the institution/department with regard to the didactic and pedagogical approach in the program. What didactic and pedagogic approaches are practised? Of course there is no a single didactic concept that is valid for all. However, at least one has to think about the didactic model behind the program. Evidence

Is there an explicit didactic concept and teaching learning strategy shared by all staff members? Is this adequate?

Are the instructional methods used (organisation of self-instruction for students, size of classes, organisation of seminars, practical courses/internships etc.) satisfactory?

The role of the computer in the program?

Is there sufficient variety in the teaching/learning methods?

Circumstances that prevent the use of desired instructional methods (number of students, material infrastructure, lecturer skills)?

If research is a done by the department:

When do students come into contact with research for the first time?

How is the interrelationship between education and research expressed in the program?

How are the research findings included in into the program?

89

The practical training of students (trainees) is a specific aspect in the didactic concept. Describe the position given to practical training in the program:

Is practical training a compulsory part? Size in credit points.

Have any criteria been formulated for the practical training to comply with?

Preparation of practical training in the program (concerning content, method and skills).

Is the level of the practical training satisfactory?

Are there any bottlenecks in the practical training? If so, what causes them?

How are students coached?

How is the assessment done? 10. Student assessment Criterion

The system of assessments and examination provides an effective indication whether the students have reached the expected learning outcomes of the program or its components.

The tests, evaluations and examinations are in line with the content and learning objectives of the various parts of the program.

The program provides individual students with adequate feedback concerning the extent to which the various learning objectives have been achieved.

The program ensures adequate consistency of the student assessments.

The assessment is adequately organized (as regards e.g. announcement of the results, opportunities to re-sit tests or examinations, compensation arrangements etc.).

The examination committee functions adequately and performs its statutory tasks

Explanation Student assessment is one of the most important elements of education. The outcomes of such assessment have a profound effect on students' future careers. It is therefore important that assessment is carried out professionally at all times and takes into account the extensive knowledge that exists on testing and examination processes. Assessment also provides valuable information for institutions about the efficiency of teaching and learner support. Student assessment procedures are expected to:

be designed to measure the achievement of the intended learning outcomes and other program objectives;

be fit for purpose, whether diagnostic, formative or summative;

have clear and published grading/marking criteria;

be undertaken by people who understand the role of assessment in the students' progression towards achieving the knowledge and skills associated with their intended qualification; where possible, not relying on the verdicts of single examiners;

take into account all the possible consequences of examination regulations;

have clear regulations covering student absence, illness and other mitigating circumstances;

ensure that assessments are conducted securely in accordance with the institution's stated procedures;

be module to administrative verification checks to ensure the accuracy of the procedures;

inform students clearly about the assessment strategy being used for their program, what examinations or other assessment methods they will be module to, what will be expected of them, and the criteria that will be applied to the assessment of their performance.

Evidence

To what extent do the assessments and examinations cover the content of the courses and program? To what extent do the assessments and examinations cover the objectives of the courses and of the program as a whole?

90

Do the assessments have clear and published grading/marking criteria? Are the pass/fail criteria clear?

Are a variety of assessment methods used? What are they?

Are the assessment/examination regulations clear?

Are the procedures clear? Are they well known? Well followed?

Are any safeguards in place to ensure objectivity?

Are the students satisfied with the procedures? What about complaints from students?

Do clear rules exist for re-assessments and are students satisfied with these? A special form of student assessment is the final project (essay, thesis or assignment). This requires students to demonstrate their knowledge and skills and their ability to manipulate the knowledge in a new situation.

Do clear regulations exist for the final project/final essay?

Are the criteria for the final project clear?

Is the level of the final project/final essay satisfactory?

Do any bottlenecks exist for producing the final project? If so, why?

Describe how students are coached. INPUT VARIABLES (CELL 11-15) A department's quality not only depends on the program itself. We also have to look at the input:

The quality of the program will be near impossible to achieve without qualified and competent teaching staff and support staff

The quality of the entering student will influence the quality of our process and the quality of the output.

Not only are the human resources important; so too are the financial resources, i.e. the programs funding and financing for the facilities.

11. Quality of the teaching staff Criterion

The staff is competent and qualified

The size of the teaching staff is sufficient to deliver the curriculum and suitable in terms of the mix of qualifications, experience, aptitudes, age, etc.

Recruitment and promotion of academic staff are based on merit system, which includes teaching, (research) and services

Duties allocated are appropriate to qualifications, experience, and aptitude.

Time management and incentive system are directed to support quality of teaching and learning.

There are provisions for review, consultation, and redeployment.

Termination, retirement and social benefits are planned and well implemented.

There is a well-planned staff appraisal system based on fair and objective measures in the spirit of enhancement which are carried out regularly

Explanation The quality of a program depends on the interaction between the staff and the student. We expect that the staff is competent and qualified. Competent teaching staff is able to:

design and deliver a coherent teaching and learning program

apply a range of teaching and learning methods and select methods most appropriate to desired learning outcomes

employ a range of techniques to assess students’ work and match these to intended learning outcomes

monitor and evaluate their own teaching performance and evaluate programs they deliver

91

reflect upon their own teaching practices Size of the staff and their qualifications Use table 18 to specify the number of staff. Mention possible vacancies separately, and specify the reference date for the data. Specify the staff/student ratio and the staff/graduate ratio as per table19. Table 18: Teaching staff in the program

Rank Full time Part time Total

Assistant Tutor II

Assistant Tutor I

Tutor/instructor II

Tutor/instructor I

Senior Tutor/instructor II

Senior Tutor/instructor I

Principal Tutor/instructor II

Principal Tutor/instructor I

Chief Tutor/instructor II

Chief Tutor/instructor I

Tutorial assistant

Assistant Lecturer

Lecturer

Senior Lecturer

Associate professor

Professor

Guest Lectures from industry

Table 19: Staff/student ratio and staff/graduate ratio (please specify the year)

Number of staff Number of Students

Number of graduates Year:

Number of students Staff member

Number of graduates staff member

Evidence

Is the staff competent and qualified for their job? Are the competencies and expertise of the staff adequate for delivering this program?

Are there any problems with the human resources? Age structure? Vacancies difficult to fill? What difficulties are there in attracting qualified staff?

What policy is pursued with regard to the employment of staff, both in teaching and research?

What about teaching load? The staff/student ratio? The staff/graduate ratio?

What about the involvement of instructors/lectures from the industry/labour market? Staff management

Does the department have a clearly formulated staff management structure?

Is staff recruitment based on experience in teaching or practical experiences?

Is there a system of staff appraisal?

What role do teaching qualifications and teaching activities play in the career of the staff members?

What does the department think of its HR policy so far?

What future developments are there?

How are teachers prepared for the teaching task?

Is the teaching delivered by the staff supervised and assessed?

92

12. Quality of support staff

There is adequate support in term of staffing at the libraries, laboratories, administration and student services.

Explanation Program quality depends mostly on interaction between staff and students. However, teaching staff cannot perform well without the quality of the support staff. These might be staff members who support the library, laboratories, computer facilities etc. Evidence

Are the library support staff members competent and sufficient?

Are the laboratories support staff members competent and sufficient?

Are the computer facilities support staff members competent and sufficient?

Are the administrative support staff members competent and sufficient? 13. The profile of the Student Criterion

There are clearly formulated admission criteria for the programs?

If there is selection, the procedure and criteria are clear, adequate and transparent

The planned study load is in line with the real study load

Explanation The quality of the output depends a lot on the quality of the input. This concerns also the entering students. The intake

Give a summary of the intake of first year students using Table 12.

Give a summary of the total number of students enrolled in the program using Table 13.

Table 20: Intake of first-year students (last 5 years) Full-time Part-time

Year M F Total M F Total

Table 21: Total number of students (last 5 years)

Full-time Part-time

Year M F Total M F Total

Evidence

How do you analyse the development of the student intake? Reasons to worry? Causes of problems? Prospects for the future?

What are the admission procedures? Are students selected? If so, how are they selected? What are the requirements?

What policy is pursued with regard to the intake of students? Does it aim to increase the intake or to stabilise it? Why?

What measures are taken to affect the quality and size of the intake? What effect do these measures have?

93

Study load

Does the department use a credit points system? How are credits calculated?

Does the program's actual study load correspond with the prescribed study load?

Is the study load divided equally over and within the years?

What measures are taken in the field of program development and/or student advice when parts of the program deviate from the prescribed study load (too difficult/heavy or too easy)? Are these measures effective?

Can an average student complete the program in the planned time? 14. Student advice and support Criteria

Student progress is systematically recorded and monitored, feed back to students and corrective actions are made

where necessary.

In establishing a learning environment to support the achievement of quality student learning, teachers do all in their power to provide not only a physical and material environment which is supportive of learning and which is appropriate for the activities involved, but also a social or psychological one.

Explanation How students are monitored and supported by staff is essential to a good student career. A department must ensure that a good physical, material, social and psychological environment is in place. Evidence

What role do staff members play in informing and coaching students?

What role do they play in integrating students into the department?

How is the information flow to potential students organised? Is sufficient attention paid to the requirements of their educational background?

Does the future student get a good impression of the education offered? Is the information evaluated? If so, what happens with the results?

How are students informed about the study program?

Is attention paid to study progress? Is student progress recorded? Does the recording lead to problems being pointed out in time? When is first contact made with problem cases? Does this result in remedial and/or preventive actions being introduced for the individual student or program development?

Is special attention paid to coaching freshmen? If so, how does it work?

Are there specific facilities to provide study skills for students with problems? Are these available within the department, the faculty or centrally? How is information on these matters organised?

Is separate attention paid to coaching advanced students?

Is assistance given in completing the final project? Where can students who get stuck with their practical training or final project get help?

How are students advised on problems concerning course options, change of options, interruption or termination of studies?

Is information provided on career prospects? Do students have the opportunity to familiarise themselves with the labour market by means of practical training, application courses and the like?

If students wish to extend their course of study, are the reasons examined? If yes, what are usually the findings and what measures do they result in?

To what extent do the structure and organisation of the program contribute to students taking on an active study approach?

To what extent does the program challenge student to make a satisfactory investment in their studies/program?

94

Are you satisfied with the tools available to improve study progress?

15. Facilities and infrastructure Criteria

The physical resources to deliver the program, including equipment, materials and information technology are

sufficient

Equipment is up-to-date, readily available and effectively deployed

Information technology systems are set up or upgraded

The computer centre continuously provides a highly accessible computer and network infrastructure that enables the campus community to fully exploit information technology for teaching, research and development, services and administration.

Explanation Facilities and resources should be in line with the formulated goals and aims and with the designed program. Facilities are also connected to the teaching/learning strategy. For example, if the philosophy is to teach in small working groups, small rooms must be available. Computer-aided instruction can only be realised with enough computers for the students. The main learning resources consist of books, brochures, magazines, journals, posters, information sheets, internet and intranet, CD-ROMs, maps, aerial photographs, satellite imagery and others. Evidence Teaching rooms

Are enough lecture halls, seminar rooms, laboratories, reading rooms, and computer rooms available? Do these meet the relevant requirements?

Is the library sufficiently equipped for education?

Is the library within easy reach (location, opening hours)?

Are laboratory facilities and support staff sufficient?

Do the laboratories meet the relevant requirements? Didactic aids and tools

Are sufficient audio-visual aids available?

Are there enough computers? Appropriate and enough computer programs (computer-aided education, maths programs, design programs, etc)?

To what extent do the facilities/infrastructure promote or obstruct delivery of the program?

Is the total budget for aids and tools sufficient? QUALITY ASSURANCE (CELL 16-19) The confidence of students and other stakeholders in Technical Education is more likely to be established and maintained through effective and efficient quality assurance activities which ensure that programs are well-designed, regularly monitored and periodically reviewed, thereby securing their continuing relevance and currency. A well functioning quality assurance system has at least the following elements

Student evaluation (16)

Curriculum design (17)

Staff development activities (18)

Benchmarking (19)

95

16. Student evaluation Criterion

• The department makes use of student evaluation on a regular base • The outcomes of the student evaluation are used for quality improvement

• The department provides the students with feedback on what is done with the outcomes of the evaluation.

Explanation Students are the first to judge the quality of teaching and learning. They experience the delivery method. They have an opinion about the facilities. Of course, the information given by students has to be counterbalanced by other opinions. Nevertheless, the institution/department is expected to carry out student evaluations and to use the outcomes for improvement. Evidence

Do the department use student evaluations in a structured manner?

Who is responsible for the evaluations?

What is done with the outcome of the evaluations? Are there any examples of this contributing to improvements?

What is the input of the students who sit on the committees involved in the internal quality assurance process?

17. Curriculum design & evaluation

Criterion

The curriculum design (or redesign) is done in a structured way, involving all stakeholders, especially employers.

There is a well functioning program or curriculum committee

The curriculum is regularly evaluated

Revision of the curriculum takes place at reasonable time periods

Quality assurance of the curriculum is adequate

Explanation Developing or designing a curriculum is a special activity. Too often, a curriculum is seen as a number of courses provided by the present teachers. They sometimes act like small shopkeepers, selling their own product, but not knowing what others offer. Curriculum design should start with the formulation of the expected learning outcomes. The next question will be what courses are needed to achieve the objectives and finally who will teach the courses? It is important that a curriculum is seen as a joint enterprise. Evidence

Who is responsible for designing the curriculum?

How is the labour market involved in the curriculum design?

How do curriculum innovations come about? Who takes the initiative? On the basis of what signals?

Who is responsible for implementation?

When designing curricula, is there any benchmarking with other institutions?

In which international networks does the department participate?

With which institution abroad does exchange take place?

Has the program been recognised abroad?

96

Course and curriculum evaluation

How is the program (curriculum) evaluated? At course level? At curriculum level?

Is the evaluation done systematically?

How are the students involved in evaluating the education and training?

How and to whom are the results made known?

Is anything done with the results? How is this made transparent? 18. Staff development activities Criteria

Staff development needs are systematically identified, in relation to individual aspirations, the curriculum and institutional requirements.

Teaching and supporting staff undertake appropriate staff development programs related to identified needs

Explanation It is important that the teaching staff have full knowledge and understanding of the module they are teaching, have the necessary skills and experience to communicate their knowledge and understanding effectively to students in a range of teaching contexts, and can access feedback on their own performance. Institutions should ensure that their staff recruitment and appointment procedures include a means of making certain that all new staff have at least the minimum necessary level of competence. Teaching staff should be given opportunities to develop and extend their teaching ability and should be encouraged to value their skills. Institutions should provide poor teachers with opportunities to improve their skills to an acceptable level and should have the means to remove them from their teaching duties if they continue to be demonstrably ineffective. Evidence

Does the department have a training program for the staff about: • Curriculum design • Test construction • Teaching skills • Computers in the class room

Does the HEI offer the staff possibilities to develop and extend their teaching abilities by participation in conferences etc?

19. Benchmarking

The faculty/department uses the instrument of benchmarking for analysing the quality of its program and its performance.

The UNESCO definition of benchmark is: A standard, a reference point, or a criterion against which the quality of something can be measured, judged, and evaluated, and against which outcomes of a specified activity can be measured. The term, benchmark, means a measure of best practice performance. The existence of a benchmark is one necessary step in the overall process of benchmarking. NACTE defines benchmarks as: A systematic and continuous measurement process; a process of continuously measuring and comparing an organization business process against business leaders anywhere in the world to gain

97

information which help the organization to take action to improve its performance 15 Benchmarking is a process that enables comparison of inputs, processes or outputs between institutions (or parts of institutions) or within a single institution over time. It is important for a n institution to compare its functioning with equivalent institutions in the country, the region and internationally.

Evidence

Is the HEI using the instrument of benchmarking? How is it using the instrument?

Does the executive management use the collected information?

What is done with the benchmarking? 20. The achievements Proof of the pudding is in the eating. In assessing our quality we have to look not only at our process quality, but also have to take into account the output. First of all, we must look at our graduates. Did they achieve the expected standards? Are the achieved outcomes equal to the expected outcomes? Have the graduates acquired the expected knowledge, skills and attitudes? 20.1 The profile of the graduates Criteria

The final qualifications achieved by the graduates are in line with the formulated NTA level

The content and level of the graduation projects are in line with the NTA degree and the NQF.

Graduates are able to operate adequately in the field for which they have been trained.

Explanation Quality has been formulated as achieving our objectives in an efficient and effective way, assuming that the goals and aims reflect the requirements of all our stakeholders in an adequate way. The final test of our quality is the graduate. Did he or she really achieve the expected learning outcomes? This is not easy to measure and can only be known by means of feedback from the labour market and feedback from alumni. Evidence • Is the average standard of our graduate satisfactory? • Do the achieved standards match the expected standards? • Do our graduates easily get jobs? Are the jobs that the graduates get in accordance with the

level of graduation? • Have any changes been signalled in the labour market prospects of graduates over the last few

years? What are the prospects? 20.2 Pass rates and dropout rates Criteria

The department responsible for the program has set targets for the student success rate (i.e number of graduates per year) and the duration of studies comparable with those for other relevant programs.

The actual student success rate is in line with these targets. Explanation Because the output quality has to be evaluated within the framework of the process, we have also to look at the efficiency of our provisions, among others we have to look at the pass rates and the

15NACTE, Benchmarks for Comparing Performance Across Courses and Across Technical Institutions, January

2004. The publication offers a lot of information about the benchmark process.

98

dropout rate, the average time to complete a degree program (graduation time), and the employability of graduates. Pass rates or success rate: number of students, successfully finishing the program Dropout rate: number of students that do not finish the program. The dropout may be enrolled in another academic program in or outside of the department, but for the program he or she left, it is counted as drop out. Provide information on the pass rate and dropout rates. Evidence • What is the opinion of the department about the pass rate? If not satisfactory, what measures

have been taken to improve the pass rate? • Have any fluctuations in the success rate been seen over the last five years? • How high is the dropout rate? Are there explanations for the dropout rate? • Does the department know where the dropout students are going? 20.3 Average time to graduation

The average time for graduation is in line with the planned time for finishing the program

Indicate the average time a student spends on a program. If necessary, categorise the students in groups. • What does the department think of the average time to graduation? • What measures have been taken to promote graduation and to shorten the average time to

graduation? • What effect have these measures had? 20.4 Employability of the graduates Criterion

The employment/unemployment rate of the graduate is in line with the target set by the faculty.

Evidence • What percentage of graduates found a job within six months of graduation over the past five

years? How many within a year? • What percentage of graduates are still unemployed 2 years after graduation? 21. Stakeholders satisfaction Criterion

The department must have a structured method to obtain feedback from all stakeholders for the measurement of their satisfaction.

Explanation After analysing the input, the process and the output, we have to analyse the satisfaction of all stakeholders. What do they think about our performance? How do we know that?

99

Evidence Opinion - Students • Does the department know what students think about the courses, the program? The teaching?

The examinations? • Is student evaluation carried out regularly? Is it done adequately? • What is done with the outcomes of student evaluations? • How does the department cope with complaints by students? Opinion - Alumni (graduates) • Does the department interview graduates on a regular basis? • What is the opinion and feedback of graduates when they are employed? • Is the feedback of the alumni used to adapt the program? Opinion-Labour market • Do structured contacts exist with employers and the labour market for getting feedback on

graduates? • How do employers appreciate the graduates? Are there any specific complaints? Do the

employers appreciate specific strengths? • How do we cope with complaints from the labour market?

22. Community contribution Criterion

The department has clear guidelines for consultancy and community outreach

Explanation The department is not only responsible for teaching and (if applicable) doing research. It is also responsible for serving society. Consultancy involves a broad range of activities. In general, the term consultancy covers the provision of professional advice or services to an external party for a fee or other non-monetary consideration. Among other things, guidelines on providing consultancy cover the following:

Policy objectives

Policy on key policy principles, compliance, accountability framework, legal and financial protection, conflicts of interest

Procedures for the contribution to society and the community

Procedures for institutional/academic consultancy

Procedures for private consultancy Evidence

What role does the department play in the local, national and international community

What are the key activities, which of these lay outside normal teaching or research? How do they relate to the department’s mission?

What are the non-profit activities of the department?

Is there a clear policy on consultancy and the contribution to society and the community?

How is the income from consultancy regulated?

7.3 Strengths/Weaknesses Analysis

The self-assessment is followed by a strengths-weaknesses analysis. At the same time, this serves as a check to see how far the institution is in compliance with the given criteria. This is best done, using the checklist (see Appendix 3) and summarizing the outcomes in table 22. The different aspects of the program will be assessed on a scale 1-7. The marks have the following meaning:

1 = absolutely inadequate; immediate improvements must be made 2 = inadequate, improvements necessary

100

3 = inadequate, but minor improvements will make it adequate 4 = adequate as expected 5 = better than adequate 6 = example of good practice 7 = excellent

The overall assessment of the different aspects is based on the scores given to each sub-aspect in the category. But, of course not all sub-aspects have the same weight. This means that you cannot calculate mathematically an average. You have to balance the various sub-aspects and judge the importance of each of them. Aspects with positive outcomes may compensate for some negative ones. Summarizing the outcomes per category is not a mathematical enterprise, nor ticking boxes. One has to balance the importance of each criterion. The same counts for the overall judgment of the quality of the institution. One cannot calculate the average of the categories, but look at the importance of the category. Programme specification and benchmarking are less important than Learning Outcomes or content of the program. Summarizing the outcomes for each of the 22 categories shows the strengths and weakness of the institution. Table 22: summary quality aspects program analysis

SN 1 2 3 4 5 6 7 1 Mission & objectives 2 Leadership& administration 3 Financial resources 4 Requirements stakeholders 5 Expected learning outcomes 6 Programme specifications 7 Content of the programme 8 Organization of the program 9 Didactic concept 10 Student assessment 11 Quality teaching staff 12 Quality support Staff 13 Profile of students 14 Student advice & support 15 Facilities & Infrastructure 16 Student evaluation 17 Curriculum design 18 Staff development activities 19 Benchmarking 20 Achievements 21 Satisfaction stakeholders 22 Community service Overall judgment

Summary of strengths Summaries the points that the department considers to be its strengths and mark the points that you are proud of. Summary of weaknesses Indicate which points the department considers to be weak and in need for improvement. Also indicate what you are going to do about this.

101

For the improvement of the weaknesses, the department must include actions in the Quality improvement plan. Only with a clear follow up and a quality and action plan, the investment in the self-assessment and the SAR make sense.

7.4 The Self-Assessment Report (SAR)

After completing the self-assessment, the outcomes of the assessment will be written down in a Self Assessment Report (SAR). The SAR is an important document. On one hand it contains the basic information for the external expert team that will come and assess the quality of the program. On the other hand it is the basic document for the faculty/department for the formulation of an action plan or quality plan for the coming years. The content of the SAR follows the lines of the cells, discussed during the self-assessment process. For each cell, one should: • Describe clearly the state-of-the art. An outsider must understand the situation. • Analyse the situation. What is your opinion about it? Satisfied or not? If not, why not? Table 14 defines the content of the self-assessment report. Be sure to discuss the report within the faculty and ensure that everybody owns the process. Table23: Content of a self-assessment report at program level

Introduction • How was the self-assessment carried out? • Composition of the self-assessment team • Short description of the institution and the department responsible for the curriculum • Short description of the program (in such a way that an outsider has a good idea about the content of the program)

Chapter 1: The department 1.1. Mission and objectives 1.2 leadership and administration 1.3 Financial resources

Chapter 2: Requirements stakeholders and expected learning outcomes

Chapter 3: The process 3.1 Program specification 3.2 Program content 3.3. Program organisation 3.4 Didactic concept 3.5 Student assessment

Chapter 4: The input 4.1 quality of the staff 4.2 Quality of the support staff 4.3. The students 4.4. Student advice/support 4.5 Facilities and infrastructure

Chapter 5: Quality assurance 5.1 Student evaluation 5.2 Curriculum design& evaluation 5.3. Staff development activities 5.4. Benchmarking

Chapter 6: achievements and graduates 6.1 Achieved outcomes (graduates)/graduate profile 6.2. Pass rate and dropout rate 6.3. Average time to degree 6.4 Employability

Chapter 7: Stakeholder satisfaction 7.1. Opinion - Students 7.2. Opinion - Alumni (graduates) 7.3. Opinion - Labour market 7.4 Opinion - Society

8. Community service

Chapter 9: Strengths-weaknesses analysis 9.1 Summary of strengths 9.2 Summary of weaknesses 9.3 Quality plan for the coming years

102

7.5 The Follow Up After the Self-Assessment

The outcomes of the self-assessment must be translated into a quality plan that shows what activities the department will undertake in the near future. The self-assessment will show us where we are now and will give us the direction to where we would like to be, say in 5 years' time. Only with a clear follow up and a quality and action plan, the investment in the self-assessment and the SAR make sense. When there is no external assessment by NACTE, the QA-unit may, in consultation with the department, organise an external assessment in the form of an inter-collegial assessment. Such external assessment is important because it gives authority to the findings of the department. Inter-collegial assessment also delivers confidence to stakeholders; provides evidence of quality to the public and shows that the standards agreed upon by the competent authorities are being implemented. At the same time, it provides mechanisms for continuous quality improvement in the sustainability and development of the program, and buffers against pressures to lower quality standards. The QA-unit does have various possibilities for an inter-collegial assessment:

The assessment remains within the institution, involving colleagues from adjacent departments and/or disciplines.

Colleagues from other institutions, national or international, may be invited. An advantage of this approach is that there are more opportunities for benchmarking. A disadvantage is that this approach is more expensive, especially when international experts are involved.

Of course, it is also possible to combine those two approaches and to invite some external colleagues to participate in an internal assessment.

7.6 Preparation for Program Accreditation When the self-assessment of the institution will be followed by a Program Accreditation by NACTE, the institution has to prepare itself for the accreditation. The following is important:

Take care that the SAR is on time at NACTE

Take care that the SAR is informative, analytical and honest. The SAR is the basic information for the audit team

Take care that all staff and students knows about the SAR

Take care that during the site the most important documents, showing evidence that you are meeting the criteria are displayed for the audit team

Take care that everybody is available for interviews during the site visit.

7.7 The organisation of the Program Accreditation

7.7.1 The expert team

A program assessment can be part of an institutional assessment or can be done separately (e.g. a country wide assessment of one discipline). The way the program assessment will be organised decide the composition of the team. a) Composition of the expert team, when program assessment is included in the institutional

accreditation. An effective assessment team, commissioned to conduct the Institutional Accreditation, will have at least 6 members + secretary. Membership of an expert team should include:

A chairperson, totally independent and unconnected with the institution to be assessed. The chair should have experience with management structures in Technical Institutions and have experiences with the developments that have taken place over the last few years

103

3 experts in the specific field of the institution. One coming from the academic world, one from the labour market area or from a professional body

An expert in quality assurance The 3 experts in the specific discipline will participate in the judgment of the institution, but they will especially be in charge of the assessment of the programs, included in the institutional assessment. (see also section 6.6) b) An effective expert team, commissioned to conduct an external program assessment, not included in an institutional assessment) will have 3 till 5 members. Membership of an expert team should include:

A chairperson, totally independent and unconnected with the program to be assessed. The chair does not need to be an expert in the field, but should have the confidence of those who are in the team. If possible, the chair should have experience with management structures in higher education institutions and with the developments that have taken place over the last few years

Two experts on the module area/discipline in question

An expert from the labour market area taking up graduates and/or from the professional association

An expert on education/learning processes. There are some conditions that members of the expert team have to meet:

Members should act independently

There should be no conflicts of interest. Members should stand to gain nothing from their verdict

Members must be accepted by the department to be assessed. This means that NACTE has to check if the institution has any serious objection against one of the members. It is up to NACTE to decide if the objection is justified.

Retired staff can be invited to participate on the grounds that they are more independent (and have more time available). However, it is also important to have members still working in the field and with a good knowledge of recent developments in the field.

7.7.2 Preparatory work of the expert team In general, the task for the expert team can be described as follows:

To form an opinion about the standard of the curriculum and the quality of the educational process, including the organisation of education and the standard of the graduates on the basis of information supplied by the department (the SAR) and by means of discussions held on site. In assessing quality, the team must look at the requirements and expectations of the student, the faculty/discipline and society, and, in particular, prospective employers.

To make suggestions on quality improvement. An assessment team trying to fulfil its task will encounter a lot of problems, because the generally formulated task means that the team will tend to form opinions about everything. Therefore, for the benefit of both the team and the faculty, the terms of reference should be operationalized into a number of questions that can be formulated on the basis of the SAR. Preparatory work of the team members16 As soon as NACTE has decided about the site visit, the SAR of the department involved will be sent to the members of the team. The members will study the report carefully before the team comes together in a preliminary meeting. As a starting point for the discussions during the preliminary meeting, each member will be invited to answer the following questions with regard to the self-assessment report:

Is the report sufficiently critical and analytical?

16When the program assessment is combined with the institutional accreditation, “team member” s should be read as” the 3 team members, specialized in the discipline to be assessed”.

104

Have the problems that face the department been clearly formulated? Has the department indicated clearly how it will cope with the problems?

Are you able to form a picture of the content of the curriculum, given the description in the report?

Have expected learning outcomes been satisfactorily operationalized?

Do you think the objectives and goals have been satisfactorily translated into the program?

Is the curriculum well balanced?

Can the program, as described in the report, be done in the set time?

Do you think it is possible to produce good graduates with this curriculum?

Do you believe the curriculum assures the expected qualifications (the NTA)?

Preliminary team meeting If possible the expert team will meet some time before the planned site visit for the training session and to prepare itself for the assessment. If this is not possible, the meeting will be held on the day before the site visit begins. The topics in the meeting are: o Discussion on the Qualification standard

Is there a job profile formulated with clear purpose of the Qualification? What are the learning outcomes? What are the consequences for the curriculum?

o Discussion on the self-assessment report During the meeting, the team will discuss the Self-assessment report and formulate questions to be asked during the site visit.

o Discussion on the site visit program The chairperson will set a program for the site visit in consultation with the department. The program will be discussed to see if it fits the team's approach. Table 14 provides a format for a site visit program of a separate program assessment.17 The preparatory meeting is also important for making the group of experts into a cohesive team. Many a review team has complained that the team did not act as a team until after the end of the site visit. The intensive discussion on the job profile and the SAR will serve to transform the loose group into a team that can start the site visits as a team.

The assessment protocol The expert team assesses the quality of the program. The team will already have discussed several aspects during the preliminary meeting. The SAR will already have provided detailed information. During the site visit, the team will be looking for evidence with the following questions in mind:

Are the expected learning outcomes clearly formulated?

How are these translated into the curriculum?

Do the exams reflect the content of the program and courses?

Have graduates really acquired the expected knowledge, skills and attitudes? Of course, the SAR is the basic source of information and should provide the basic information to the team. But other sources should also be used:

The interviews

The list of the literature used

The final graduation project

Assessment and examination papers

Course descriptions and readers This is why it will be necessary to reserve time in the visit program for studying these materials. During the assessment, the expert team has to assess the following criteria • Departmental capacity

17For the program of a site visit of a combined institutional and program assessment see table11.

105

• Requirements of stakeholders • Expected learning outcomes • Program content • Program specification or description • Program organisation • Didactic concept/teaching/learning strategy • Student assessment • Quality of academic staff • Quality of the support staff • Student profile • Student advice/support • Facilities & infrastructure • Student evaluation • Curriculum design & evaluation • Staff development activities • Benchmarking • Achievements /graduates • Satisfaction stakeholders

The expert team will use the same analysis model for a program as has been used by the department to conduct the self-assessment (see figure 11)

7.7.3 The Site Visit Program The chairperson of the team will confirm the program for the site visit in consultation with the department according to a given format (see table 24). Before hand, appointments will have been made with whatever staff members and students the team would like to talk to. The interviews start with a discussion involving the writers of the self-assessment report. In this interview, the team can ask for clarification of any obscurities and explanation of any topics that are not totally clear. The interviews with the students are purposely planned to take place before the interviews with the staff members. The students are a very rich source of information, but the information needs to be checked and tested against the ideas of the staff members. Student interviews are important to get an insight into the study load, the didactic qualifications of the staff, the coherency of the program, to find out if they are acquainted with the learning outcomes/objectives, the organisation of the curricula and the facilities. These student interviews should be held in the absence of staff members, so that they can speak freely. The size of the student groups is ideally about ten each time. The composition of the student panels requires special attention. It is important that the group is as far as possible representative of the whole student population in that field, i.e. that it not only includes the good students, but also the less gifted ones. It is better not to leave the invitation of students to the faculty or the staff. The best way is to ask a student organisation (if there is any) to nominate the students. If there is no such organisation, the expert team will invite students at random. Interviews with staff members will be used for discussion on the content of the curriculum, the goals and objectives/expected learning outcomes. "Why and how did you choose this program?”. Other topics to be discussed include the examinations, the final paper (if any), field attachment etc. Other interviews will be held with members of a curriculum committee and with members of the committee responsible for examinations. During the interview with the curriculum committee, the question of how the curriculum is kept up to date will be discussed as will the question on how innovations are planned and realised, etc. The interview with the examination committee must clearly show how the quality of the examinations and degrees is assured. In the program time is allocated for looking at the facilities: Lecture halls, working group rooms, laboratories, practical rooms, libraries, etc.

106

Table 24: draft program for a site visit Time Activity

15:00 on the day before the official visit

Team members meet in the hotel for information about their task and discussion on:

The SAR

Specific questions

The program

18:00 Reception by the head of the institution and other officials; dinner

Day 1 9:00 - 17:00

Interviews with:

the writers of the self-assessment report

students

staff members

curriculum committee/examination committee

student advisers

19:00-20:00 20:00

Dinner for the expert team Short meeting for discussion on the findings of the day and for setting the program for the next day

Day 2 9:00 - 11:00

Interviews with the department board Additional interviews, if needed Visit to facilities

11:00 - 12:00 Meeting with the management of the institution

12:00 - 13:00 Lunch for the expert team

13:00 - 16:00 Formulation of the findings

16:00 - 16:30 Feedback to the faculty board

An important question is: "Should a team attend lectures?" The quality of education depends foremost on the interaction between staff and students. It is logical that the experts should attend lectures, tutorials and seminar or research groups. However, given the short time for the site visit it is quite impossible to do so. To get an impression of how things are going in the lecture halls, a team can agree to walk into a lecture hall "in action" to feel the atmosphere. However, it must be stressed that it is not a responsibility of the team to assess an individual staff member.

7.7.4 Formulating the Findings The afternoon of the second day is used for drawing up the findings. There are about 3.5 hours available for this difficult task. The best method is to start with completion of the checklist (see appendix 3) by each member individually. The team members are requested to give a mark between 1 and 7 for the various aspects. To have some idea of the value of the figures, bear the following ideas in mind:

Score 1-2 when you believe this aspect should be considered as critical. The institution has to take action. Something has to be done and cannot wait.

Score 3 when you believe this aspect is unsatisfactory. It must be improved, but does not directly threaten the quality.

Score 4 when you believe the situation is satisfactory. The TI may be satisfied, but there is no reason to be proud.

Score 5 when you believe this topic can be assessed as more than satisfactory, but not excellent.

Score 6 when you believe this topic can be assessed as more than satisfactory and c an be seen as an example of good practice.

Score 7 when you believe this topic can be assessed as excellent. The institutions can be proud of it and it is certainly a strong point.

The overall assessment of the different aspects is based on the scores given to each sub-aspect in the category. But, of course not all sub-aspects have the same weight. This means that you cannot calculate mathematically an average. You have to balance the various sub-aspects and to judge the weighting of each of them. Positive aspects may compensate for some negative ones. Summarizing the outcomes per category is not a mathematical enterprise, nor ticking boxes. One has to balance

107

and weight the importance of each criterion. The same counts for the overall judgment of the program. One cannot calculate the average of the categories, but weight the importance. Summarizing the outcomes for each of the 22 categories shows the strengths and weakness of the institution (See table 25) Table 25: summary quality aspects program analysis

SN 1 2 3 4 5 6 7

1 Mission & objectives

2 Leadership& administration

3 Financial resources

4 Requirements stakeholders

5 Expected learning outcomes

6 Programme specifications

7 Content of the programme

8 Organization of the program

9 Didactic concept

10 Student assessment

11 Quality teaching staff

12 Quality support Staff

13 Profile of students

14 Student advice & support

15 Facilities & Infrastructure

16 Student evaluation

17 Curriculum design

18 Staff development activities

19 Benchmarking

20 Achievements

21 Satisfaction stakeholders

22 Community service

Overall judgment

After completing the list, the resting time will be used to draw up an inventory of the topics to be treated in the Exiting Meeting.

7.7.5 The Exiting Meeting The oral presentation to the management of the institution at the end of the visit holds a special position in the process. Sometimes, findings and conclusions are not really suitable for the report, but the team would like to make a critical statement about t hem. In that case, the oral presentation can be used to formulate strongly worded recommendations. In order to do justice to this principle, the oral presentation is not public; the team reports to the management board. The chairperson should stress that this is an interim report; some conclusions may change during the final discussion on the report. It is advisable not only to mention weaknesses, but also strengths. The chairperson does not tell the advice about accreditation.

7.7.6 The Expert Team's Report After the site visit, the chairperson and secretary will write a first draft of the report, using the completed checklists and the minutes of the oral presentation. Table 26 gives an outline of the content of the assessment report. Table 26: Content of an Assessment Report of A Program

Introduction • Composition of The Team • Short Description of The University and the Department Responsible for the Curriculum • Short Description of the Program (in such a way that an outsider has a good idea about the content of the program)

Chapter 1: The department 1.1. Vision, Mission and Objectives 1.2 leadership and Administration

108

1.3 Financial Resources

Chapter 2: Requirements Stakeholders and Expected Learning Outcomes

Chapter 3: The process 2.1 Program Specification 2.2 Program Content 2.3. Program Organisation 2.4 Didactic Concept 2.5 Student Assessment

Chapter 4: The input 4.1 Quality of The Staff 4.2 Quality of the Support Staff 4.3. The Students 4.4. Student Advice/Support 4.5 Facilities and Infrastructure

Chapter 5: Quality assurance 5.1 Student Evaluation 5.2 Curriculum design & Evaluation 5.3. Staff Development Activities 5.4. Benchmarking

Chapter 6: achievements and graduates 6.1 Achieved Outcomes (graduates)/graduate profile 6.2. Pass rate and Dropout rate 6.3. Average Time to Degree 6.4 Employability

Chapter 7: Stakeholder satisfaction 7.1. Opinion - Students 7.2. Opinion - Alumni (graduates) 7.3. Opinion - Labour market 7.4 Opinion - Society

8. Community service

Chapter 9: Strengths-weaknesses analysis 9.1 Summary of Strengths 9.2 Summary of Weaknesses 9.3 Summary Recommendations

For each topic the following format will be followed:

What is said in the SAR about this topics

What are the findings of the expert team?

Conclusion

Recommendations Making recommendations, the expert team must keep in mind: To summarize the recommendations at the end Prioritize the recommendations as far as possible Ask itself if it is a realistic recommendation The expert team must make clear for whom the recommendations is made The team should not ride hobby horses If there are many recommendations, the team has to ask itself if they all are important The first draft of the report will be discussed with the team members. The second draft will be sent to the institution for comment. The comments should concern only factual errors and inaccuracies, not the differences in opinion. The audit team will decide what to do with the comments.

7.8 NACTE and the Program Accreditation The final report(s) of the expert team will be sent to the Sector Board concerned. The expert team gives its advice about accreditation or not accreditation in a side letter. The Sector Board will check if

109

the Program Accreditation has been done according to the rules and see if the report contains all information needed. If the assessment report is accepted it will be sent to the Council. The Council, normally, will follow the advice of the expert team. The decision might be:

Conditional Accreditation: The Council grants conditional accreditation to a program where the Council is of the opinion that there are certain requirements to be fulfilled by the department. The term, in which the condition has to be fulfilled, is given. NACTE will check if the conditions are fulfilled. Where the department fails to fulfill the requirements within the time, then the Council may:

o Cancel the conditional Accreditation granted to the program; or o After being satisfied that, there was reasonable cause for failure to comply, extend the

time for conditional Accreditation to such other time, as it may be determined.

Full Accreditation: The Council grants full accreditation to the program, which meet adequately the accreditation requirements. Accreditation is valid for 5 years. If the institution disagrees with the way the assessment is done, disagree with the findings of the external team or the decision of the Council, it may contact NACTE. NACTE will install an independent committee that will investigate the complaints. The verdict of the independent committee is binding both for NACTE as the institution.

Appendix 1: Checklist Internal Quality Assurance

1 2 3 4 5 6 7

QA organization

1 The institution has a “quality management plan” containing clear and specific policies and procedures that provide a framework for the quality assurance activities. The institution commits expressively and clearly to disseminating the culture of quality among all its members.

2 The institution has a Quality Assurance Committee to ensure quality policies and objectives are set, implemented and evaluated

3 The institution has a center/ unit/ department linked to the higher management of the institution with the task is to execute, coordinate and monitor the quality assurance. The size of the quality center/ unit/ department suits the size of the institution

4 Student progress is systematically recorded and monitored, feed back to students and corrective actions are made where necessary.

Overall Judgement QA Organisation

Monitoring

5 An institution has a structured monitoring system to collect information about the success rates and the drop out among the students

6 The institution has a structured method to obtain feedback from all stakeholders for the measurement of their satisfaction. The monitoring system includes at least: Structural feedback from the labour market Structural feedback from alumni

7 An institution has a structured monitoring system to collect information on the quality of its core activities. This includes monitoring the research out put (number of publications), the number of grants of the staff etc. (if applicable)

Overall Judgement Monitoring

Evaluation Instruments

8 The institution makes use of student evaluation on a regular base. The outcomes of the student evaluation are used for quality improvement. The institution provides the students with feedback what is done with the outcomes

9 The institution has specific mechanisms approved by NACTE to design,

110

approve, monitor and review the programs within the frame of quality assurance system in the institution.

10 The institution, with a task in research, has a system for regular review of research outcomes.

11 The institution has a system for regular review of the community outreach.

Overall Judgment Evaluation Instruments

Special QA processes

12

An institution has clear procedures to assure the assessment of students. Students are assessed on the basis of published criteria, regulations and procedures that are applied consistently. There are clear procedures to assure the quality of the examinations. There is an appeal procedure.

1 2 3 4 5 6 7

13

An institution has means to satisfy itself that its staff are qualified and competent to conduct the core activities of the institution: education, (research) and the community outreach: Adequate staff appointment procedures An adequate staff appraisal system Staff development activities

14

An institution has clear procedures to ensure that the quality of its facilities needed for student learning are adequate and appropriate for each program offered:

Adequate checks on the computer facilities

Adequate checks on the library

Adequate checks on the laboratories

15

An institution has clear procedures to assure the quality of the student support and student advice. In establishing a learning environment to support the achievement of quality student learning, teachers must do everything in their power to provide not only a physical and material environment that is supportive of learning and is appropriate to the activities involved, but also a social or psychological environment.

Overall Judgment Special QA processes

1 2 3 4 5 6 7

Specific QA-Instruments

16

An institution regularly (but at least once every 5 years), conducts a self-assessment of its core activities and of the institution as a whole to learn about its strengths and weaknesses. This self-assessment must lead to a quality plan.

17

A self-assessment might be part of an External Quality Assessment (EQA) or accreditation process where the self-assessment report acts as input for the external review team. If the self-assessment is not connected to the EQA, the institution will be expected to organise an audit itself based on the self-assessment report.

18

The Institution should have an information system that documents its performance on the key performance indicators approved in the quality improvement plan. Updated, fair and objective information on programs should be published taken from the outcomes of its information system or any other comparative studies.

19

An institution has a QA handbook that documents all regulations, processes and procedures concerning Quality Assurance. This handbook is public and known to all the people concerned.

Overall jJudgment Specific QA instruments

Follow up

20

There is clear evidence that the institution used the information from the monitoring system and the outcomes of evaluations to improve the quality.

Overall Judgment IQA system

111

Appendix 2: Checklist Quality Aspects of an Institution

1 2 3 4 5 6 7

1.

The institution has a clear idea about the relevant demands and needs of all stakeholders

2.1

The institution has a statement of vision and mission that defines the institution, its educational purposes, its student constituency, and its place in the technical education and training community.

2.2 The institution demonstrates honesty and truthfulness in presentations to its constituencies and the public; in pursuit of truth and the dissemination of knowledge; in its treatment of and respect for administration, academic staff, other staff, and students; in the management of its affairs and in relationships with NACTE and other external agencies.

Overall Judgment Criteria 2.1 – 2.2

3

The institution plans constantly to achieve its vision, mission and objectives and evaluate the extent, way and quality of achieving them. Results of the annual assessment are used in the processes of continuous comprehensive planning and assessment. It also works on self-analysis and criticism and reviews its objectives, policies and procedures according to that.

The institution ensures the participation of all councils, committees, administrations, faculty members and students in planning and assessment.

The institution periodically reviews the assessment processes and planning activities to ensure their effectiveness.

Overall Judgment

4.1

The institution has a Governing Board or Advisory Board or Council responsible for the quality and integrity of the institution.

4.2

The Institution has effective Administration and Governance.

Overall Judgment Criteria 4.1 – 4.2

5

The institution takes care of high-quality faculty staff and support staff by clearly defining their responsibility, and by evaluating their performance on a regular basis by means of an adequate staff appraisal system

The institution provides for: - a system of staff development to enhance the knowledge and skills of faculty and

supporting staff in conducting activities that have a direct influence on the quality of teaching-learning. This should include the formulation of a concrete personnel development plan;

- evaluation of the effectiveness of the provided training - compilation of records of education, experience, training, and other essential qualifications

required of lecturers and supporting staff.

The institution establishes an activity plan and evaluates activities to encourage students, faculty members and other personnel to be conscientious in their thoughts and speech.

The institution enhances the professional ethics of its students, faculty members and other personnel.

Overall Judgment

6

The provision of financial resources to meet academic requirements is effectively planned and reviewed.

7 The provision of physical resources to meet academic requirements is effectively planned and reviewed.

8.1

The programs at offer in the institution: o are meeting the expectations of the stakeholders o are meeting the current competence requirements o have clearly formulated expected learning outcomes o are coherent o are up-to-date

Stakeholders have appropriate opportunities to be involved in review of program outcomes.

Overall Judgment

112

1 2 3 4 5 6 7

8.2

The institution has well functioning student assessment systems through all programs at offer and clear rules to assure the quality of the assessments.

The institution I has a clear policy to promote that the examinations are objective, equivalent and trustworthy

The institution take care of the consistency of the examinations; consistency between the programs and consistency in time

The institution has a policy to promote a variety of assessments methods

The institution takes care that examination committee’s function adequately and performs the statutory task.

Students have an opportunity to appeal the results of assessment in a manner that is fair and equitable.

Overall Judgment

8.3

Field attachment/workplace training components are effective and integrated into curricula

8.4 The institution fosters good teaching-learning process

8.5

Appropriately qualified staff are employed to enable quality provision of programmes.

The institution actively encourages the development of all staff.

8.6

Prospective and continuing students have effective guidance to assist them in making informed decisions on their programme of study

Entry and selection criteria are appropriate for the level of each programme, are well publicised, and are applied consistently.

The institution has implemented effective credit transfer policies and procedures

The institution makes appropriate provision for the recognition of prior learning and current competency for its enrolled students.

Overall Judgment

8.7 The provision of physical resources to meet academic requirements is effectively planned and reviewed.

Overall Judgment Criteria 8.1-8.7

9

The institution has clear guidelines for consultancy and community outreach

10 The institution uses the instrument of benchmarking for analyzing the quality of its core activities and its management

11 The institution has an efficient internal quality assurance system

12

An institution has the means/opportunity to check whether the achievements are in line with the expected outcomes

13

An institution has a structured method for obtaining feedback from the stakeholders

113

Appendix 3: Checklist Quality Aspects of a Programme

1 2 3 4 5 6 7

1.

The vision and mission of the department is consistent with the vision and mission of the institution. It is applied based on specific objectives and requirements associated with the nature of specialization. It should be clearly stated and be influential in guiding, planning and implementing the program.

The mission of the department is the guide for the processes of planning and decision-making. It is carefully employed while preparing and reviewing the academic program.

The program and its objectives are periodically reviewed in light of performance assessment and the interaction of implementers of programs with their surroundings.

Overall Judgment

2.1

The department plans to achieve its mission and objectives and assess the extent, way and quality of achieving its mission and objectives. Results of the annual assessment are used in the processes of continuous comprehensive planning and assessment.

The department ensures the participation of all councils, committees, administrations, faculty members and students in planning and assessment.

The department periodically reviews the assessment processes and planning activities to ensure their effectiveness.

2.2

The head of the department decides on the program's priorities, prepares development plans, supports administrators and faculty members and the environment of teaching and learning as to achieve the mission and objectives of the program. There is also clear specification of the responsibility of the academic program

2.3 The departmental committee is responsible in the first place of the quality and credibility of implementing the program. It gives priority for the effective development of the program as to ensure the interests of the students and community it serves.

Overall Judgment Criterion 2

3.1

The process of financial planning is directed to achieve the objectives and priorities of the program

3.2

Financial affairs are effectively managed as to ensure balance between the required flexibility in units and the central accountability and responsibility.

The budget and accounts of the department are managed by a specialized financial manager.

Overall Judgment Criterion 3

4

The department, responsible for the program has a clear idea about the relevant demands and needs of all stakeholders, based on situation analysis

5

The program/curriculum has clearly formulated learning outcomes (knowledge, skills, attitude) reflecting the relevant demands and needs of all stakeholders, especially the labour market

The expected learning outcomes are competence based

The program has formulated learning outcomes on entrepreneurial skills

The program has a learning outcome, enabling the student to get a gender sensitive attitude

Overall Judgment

6.

The department publish, for each program they offer, a program specification/description which gives the intended learning outcomes of the program in terms of:

knowledge and understanding that the students will have acquired upon completion of the program

cognitive skills, such as an understanding of methodologies or ability in critical analysis

module specific skills, such as laboratory skills, clinical skills, etc.

Overall Judgment

7.

The program shows a balance between specialist contents and general knowledge and skills.

The program takes into account and reflects the vision, mission, aims and objectives of the institution.

The objectives of the program and expected learning outcomes are explicit and are known to staff and students.

The program shows the expected learning outcomes of the graduate. Each course is clearly designed to show the expected learning outcomes of the course. To obtain this, a curriculum map is constructed. A program map is available.

The program shows evidence of attention to develop entrepreneurial skills

The program shows evidence of attention to develop a gender sensitive attitude

Overall Judgment

114

1 2 3 4 5 6 7

8

The program is designed in such a way that the module matter is integrated and also strengthens other courses in the program

The program shows range, depth and coherence of the courses

The program is organised in a modular way

Overall Judgment

9

The faculty has a clear didactic concept

The didactic concept is student oriented. Hence, the conception of teaching is the facilitation of learning.

In promoting responsibility in learning, teachers should: o create a teaching-learning environment that enables individuals to participate

responsibly in the learning process o provide curricula that are flexible and enable learners to make meaningful choices in

terms of module content, program routes, approaches to assessment and modes and duration of study

Overall Judgment

10 The system of assessments and examination provides an effective indication whether the students have reached the expected learning outcomes of the program or its components.

The tests, evaluations and examinations are in line with the content and learning objectives of the various parts of the program.

The program provides individual students with adequate feedback concerning the extent to which the various learning objectives have been achieved.

The program ensures adequate consistency of the student assessments.

The assessment is adequately organized (as regards e.g. announcement of the results, opportunities to re-sit tests or examinations, compensation arrangements etc.).

The examination committee functions adequately and performs its statutory tasks.

Overall Judgment

11

The staff is competent and qualified

The size of the teaching staff is sufficient to deliver the curriculum and suitable in terms of the mix of qualifications, experience, aptitudes, age, etc.

Recruitment and promotion of staff are based on merit system, which includes teaching, (research) and services

Duties allocated are appropriate to qualifications, experience, and aptitude.

Time management and incentive system are directed to support quality of teaching and learning

There are provisions for review, consultation, and redeployment.

Termination, retirement and social benefits are planned and well implemented.

There is a well-planned staff appraisal system based on fair and objective measures in the spirit of enhancement which are carried out regularly

Overall Judgment

12

There is adequate support in term of staffing at the libraries, laboratories, and administration and student services.

13

There are clearly formulated admission criteria for the programs

If there is selection, the procedure and criteria are clear, adequate and transparent

The planned study load is in line with the real study load

Overall Judgment

14

Student progress is systematically recorded and monitored, feed back to students and corrective actions are made where necessary.

In establishing a learning environment to support the achievement of quality student learning, teachers do all in their power to provide not only a physical and material environment which is supportive of learning and which is appropriate for the activities involved, but also a social or psychological one.

Overall Judgment

15

The physical resources to deliver the program, including equipment, materials and information technology are sufficient

Equipment is up-to-date, readily available and effectively deployed

Information technology systems are set up or upgraded

The computer centre continuously provides a highly accessible computer and network

115

infrastructure that enables the campus community to fully exploit information technology for teaching, research and development, services and administration.

Overall Judgment

1 2 3 4 5 6 7

16

The department uses student evaluation on a regular base

The outcomes of the student evaluation are used for quality improvement

The department provides the students with feedback on what is done with the outcomes of the evaluation.

Overall Judgment

17

The curriculum design (or redesign) is done in a structured way, involving all stakeholders, especially employers

There is a well functioning program or curriculum committee

The curriculum is regularly evaluated

Revision of the curriculum takes place at reasonable time periods

Quality assurance of the curriculum is adequate

Overall Judgment

18

Staff development needs are systematically identified, in relation to individual aspirations, the curriculum and institutional requirements.

Teaching and supporting staff undertake appropriate staff development programs related to identified needs

Overall Judgment

19

The faculty/department uses the instrument of benchmarking for analysing the quality of its program and its performance

20.1

The final qualifications achieved by the graduates are in line with the formulated NTA level

The content and level of the graduation projects are in line with the NTA degree and the NQF.

Graduates are able to operate adequately in the field for which they have been trained

Overall Judgment

20.2 The department responsible for the program has set targets for the student success rate (i.e number of graduates per year) and the duration of studies comparable with those for other relevant programs.

The actual student success rate is in line with these targets.

Overall Judgment

20.3

The average time for graduation is in line with the planned time for finishing the program.

20.4

The employment/unemployment rate of the graduate are in line with the target set by the faculty.

Overall Judgment Criteria 20.1 – 20. 4

21

The department must have a structured method to obtain feedback from all stakeholders for the measurement of their satisfaction.

22

The department has clear guidelines for consultancy and community outreach

Overall Judgment Criteria 1-22

116

Appendix 4: Independence of Team Members

The expert teams are expected to assess the quality of the institution or program in an authoritative, critical and independent way. Therefore, the teams must conform to high standards of quality. Safeguards are necessary to make sure that these standards can be met and to demonstrate that they are actually fulfilled. The team and the team members have to act independently. The independence of the team and its members means that their judgement is not influenced by the institution or program under review or by any other interested parties. An important safeguard in this respect is the disclosure procedure, which means that any potential conflict of interest, bias or undue influence is reported and undesirable effects are minimised through clear agreements. This is not only aimed at finding and preventing actual undesirable influences, but also to detect what could give the impression of undue influence. A number of evidently undesirable situations (such as financial interests) are explicitly forbidden. The rules of conduct (section 3) describe how to deal with such situations. General Safeguards General safeguards regarding the independence of panels are:

team members who are (or were) committed to institution or programs under review, do not participate in the assessment thereof

the team as a whole is responsible for the definitive assessments

the definitive assessments are presented in draft to the participating institutes for factual correction and to check whether adequate use was made of all relevant information

there is a procedure for appeal against the assessments.

Specific Measures

The department under review must report any potential conflict of interest, bias or undue influence regarding candidates for panels.

The members of the expert team will sign a letter of independence. The members commit themselves to maintain an independent position during the assessment and not to allow undue influence to affect their judgement. Completing and signing the independence form is a requirement for installation as a team member

Potential conflicts or tensions that are reported in the independence form (or by other means) are discussed in the committee and an assessment is made to what extent these could unduly affect the judgement (or appear to do so). Measures are then taken to avoid undesirable effects. Such measures range from completely or partially excluding an expert from the assessment, to carefully counterbalancing or otherwise neutralising undesirable effects. The report states how potential tensions were detected and how these were dealt with in order to warrant the independence of the judgement.

the team members must reconfirm or update their declaration during the final committee meeting and state that they have actually fulfilled the requirements.

Rules of Conduct for the Expert Team

A team member must avoid any influence in the assessment from persons or parties committed to the programme or institute under review, or from other interested parties.

A team member must maintain sufficient distance from personal ideas, convictions or preferences about the area under review.

A team member uses the following information for the assessment: • the self study and annexed documentation provided by the faculty/department • any additional data provided at the request of the expert team • the interviews held in the course of the review • observations made during site visits.

117

The assessment made by a team member must conform to quality standards that prevail in the scientific world in general and in the relevant academic disciplines in particular. Relevant aspects in this respect are:

expertise and professionalism independence and objectivity carefulness and consistency transparency and absence of bias.

A team member does not use information gathered in the course of the review for personal purposes. Confidential information is treated appropriately.

A team member who is (or was) closely involved with the institute or program under review, does not participate in that particular assessment or in the interviews concerned.

A team member does not accept presents or remunerations from the program or institute under review.

A team member does not have financial or commercial stakes in the programme or institute under review, nor in any associated companies or organisations.

118

Independence and Disclosure Form for Members of the Expert Teams

1. Conflict of Interest Assessment Do you perceive any risk of conflict of interest or serious appearance of such conflict in your participation in the external assessment?

( ) No ( ) Yes If the answer is yes, please provide a brief description and analysis of the potential for conflict.

2. Declaration about Financial Interests “I declare that I have no financial links with any of the persons, programmes or institutes under review and that I have not accepted and will not accept any financial or other remunerations from outside sources for my participation in the external assessment. I declare that I will report any reward and other offers of such remuneration to the chairperson of the review committee.”

3. Declaration of Independence “I have read the principles and rules applying to this assessment and I declare that I will follow these to the best of my ability and that I ill judge without influence from the institute, program or other stakeholders, and without bias, personal preference or personal benefit.”

4. Declaration about Confidential ity “I declare that I will keep all information, gathered during the assessment and will treat with confidentially.”

Name: Date: ........ ...................... ................................. Signature: .......... .............................

Note: If your situation with respect to potential conflict of interest changes in the course of the review, you are obligated to submit an updated disclosure statement. Information provided in this disclosure form will be restricted to authorised persons.

119

Abbreviations CBE College of Business Education

CV Curriculum Vitae

ELO Expected learning outcome

ENQA European Association for Quality Assurance

EQA External Quality Assessment

ICT Information & Computer Technology

IQA Internal Quality Assurance

IUCEA Inter University Council East Africa

MSM Maastricht School of Management

NACTE National Council for Technical Education

NGQ National Qualification Framework

NICHE Netherlands Initiative for Capacity Development in Higher Education

NTA National Technical Awards

PI Performance indicators

QA Quality assurance

QA Quality Assurance

QC Quality control

SAR Self-Assessment Report

SWOT Strengths, Weaknesses, Opportunities and Threads

TI Technical Institution

TVET Vocational Education and Training Authority

ZIToD Zanzibar Institute for Tourism Development

Certified