Language Teaching Research 13,3 (2009); pp. 259–278

© The Author(s), 2009. Reprints and permissions: 10.1177/1362168809104698http://www.sagepub.co.uk/journalspermissions.nav

Address for correspondence: Matthew Peacock, Department of English, City University of Hong Kong, 83 Tat Chee Avenue, Kowloon, Hong Kong; email: enmatt@cityu.edu.hk

The evaluation of foreign-language-teacher education programmesMatthew Peacock City University of Hong Kong

This article presents a new procedure for the evaluation of EFL teacher-training programmes based on principles of programme evaluation and foreign-language-teacher (FLT) education. The procedure focuses on programme strengths and weaknesses and how far the programme meets the needs of students. I tested the procedure through an evaluation of a TEFL programme, collecting data from students, teachers, and alumni through interviews, questionnaires, essays and analysis of programme materials. The programme had many strengths, including the teaching of pedagogic skills and promoting reflection and self-evaluation, but also shortcomings. It should increase the amount of practice teaching and increase input in certain areas such as knowledge of teaching within the local sociocultural context and classroom management. I suggest that the procedure may be useful for other contexts.

Keywords: teacher education, programme evaluation

I Introduction

While the literature on programme evaluation and foreign-language-teacher (FLT) education is extensive, the literature seems to contain very few if any descriptions of a procedure for the overall evaluation of FLT education programmes. It was the aim of the research on which this paper reports to design and test such a procedure. The method was to review the literature on programme evaluation and the recommended content and procedures of FLT education programmes, then design a procedure that uses recognized meth-ods of programme evaluation to assess programmes in terms of how far they match those recommendations.

I tested the procedure by using it to evaluate an existing programme, and assessed its value for use in other contexts.

II Background to the research

The present research was based on two main premises. First, it is important for every teacher-training programme to have a system for regular internal

260 FLT education programmes

evaluation. There is a lot of support for this in the literature on language-teacher preparation (much of which dates from the 1990s): Rea-Dickins and Germaine (1998, p. 8) stress ‘the importance of having systematic evaluation at the heart of a programme’. Many others also emphasize this, e.g. Richards (1990), Wallace (1991), Reid (1996) and Lynch (2003). Second, the field of teacher education remains under-researched. In 1996 Freeman noted (p. 351) that ‘scant attention has been paid … still an unstudied problem’. He calls for the development of the area ‘as an important field of enquiry and study … subject to little critical scrutiny and less organized study’ (p. 374). Freeman and Johnson (1998, p. 397) say research in the area is ‘noticeably missing’.

A number of authors make recommendations regarding the content and procedures of FLT education programmes. Wallace (1991, p. 141) sug-gests they need a clearly stated philosophy, and that programme content should reflect it. He also asserts (p. 14–15) that programmes should balance received versus experiential knowledge. Wedell (1992, p. 344) proposes that programmes balance the required components of FLT training – linguistic, pedagogic and managerial competence. Linguistic competence means lan-guage proficiency; Cots and Arno (2005) note that we must distinguish this from knowledge about language. Crandall (1993, p. 507) says programmes should promote trainee flexibility in using different teaching approaches for different situations, and Wallace (1991, p. 145) adds the ability to use and adapt teaching materials.

Another suggestion (Wallace, 1991) is that programmes incorporate and encourage trainee reflection on the experiences and values they have when they enter; in particular, they should encourage trainee reflection on the effects of Lortie’s (1975, p. 26) ‘apprenticeship of observation’. This refers to preservice teachers having spent 10 000+ hours watching teach-ers in school. Reflection may be absent from many TEFL programmes: Grosse (1991, p. 43), after posting a questionnaire on 94 TESOL methods courses (not programmes), says too many lack a reflective component. Wu (1996, p. 20), describing the Chinese view of TEFL training, notes that ‘trainees are passive receivers … seeing what to do is more important than understanding why it is done’ and that for trainees, experienced teachers are ‘models to be observed and followed’ (p. 20). Liou (2001, p. 198) says this view still holds in Taiwan. Wallace (1991) asserts that programmes need to promote the skill of reflection and self-evaluation as a teacher. A connected suggestion (Stoynoff, 1999, p. 148; Lo, 2005) is that programmes need to promote the long-term and developmental character of learning to teach (that is, post-qualification teacher growth and development) and others add that they must promote future reflective practice (Crandall, 1993; Bartolome, 1994; Lynch, 2003).

Further proposals for programme content are that they should embody coherence among courses in linkage and avoid overlaps (Weir and Roberts, 1994, pp. 112–4); be up-to-date; and balance teacher- and student-centred

Matthew Peacock 261

learning (Bartels, 2005, p. 416; Wallace, 1991, pp. 14, 113–4). Bartolome (1994, p. 179) adds that it is important to evaluate programmes in terms of how well they prepare EFL teachers to function in the sociocultural context in which they will work. Freeman and Johnson (1998, pp. 400, 406, 408–9) develop this and suggest it is essential to expand views of the knowledge-base of foreign-language teachers to include knowledge of the social context of learning (i.e. classrooms), because learning cannot be fully understood without it. They add (pp. 401–2) that some programmes may place too much emphasis on theoretical and teaching skills. Johnson (2006, pp. 236–7) adds that this knowledge, and the fact that learning is a ‘dynamic social activity’, is an important part of teacher cognition and that its importance has not been adequately recognized in teacher education (pp. 237, 243). However, Yates and Muchisky (2003, pp. 136–40, 144) raise the objection that their argu-ments might marginalize the more central study of language and of second-language acquisition, and I feel they have a very strong point. Freeman and Johnson in a response article (2004, pp. 120, 122) say they did not do this; Muchisky and Yates (2004, pp. 135, 137, 139), in a further response, empha-size that while they do recognize a cultural component to language, they wish to stress the critical and central role in language-teacher education of the nature of language and how it is learned.

Further suggestions for evaluation are to ask students whether programmes meet and are relevant to their needs (Wallace, 1991, p. 147), and how well the programmes prepare them for classroom teaching (Reid, 1996, p. 3). Roberts (1998) also makes valuable contributions. He notes (p. 235) that evaluations are ‘never neutral’, as they expose stresses and competition in institutions, creating a need to identify clear questions and plan evaluations carefully (p. 236). He adds that they increase teacher feelings of ownership if teachers share the information gathered, and he also provides a detailed history and examination of Certificate in English Language Teaching to Adults programmes (pp. 198–203). His reflective description of one such programme in Tokyo (pp. 203–5) is particularly useful, providing a balanced view of programme strengths and weaknesses.

For this research, I adopted Robinson’s definition of programme evaluation (2003, p. 199): ‘the collection, analysis, and interpretation of information … for forming judgments about the value of a particular programme’. Among the aims of evaluation she describes are providing information on ‘perceptions of a programme’s value’, measuring how far they meet their objectives, and giving feedback to course providers on necessary improvements. She adds that we need to build evaluation plans in to programmes.

1 Rationale for creating a new evaluation procedure

There seem to be very few detailed descriptions of how to conduct overall evaluations of FLT education programmes. Grosse (1991) notes a lack

262 FLT education programmes

of research in the area. I agree that teacher-training programmes must embody systems for internal programme evaluation and wish to design such a system. Their use increases the accountability of the programme to stakeholders (White, 1998; Lynch, 2003). Another important reason for con-ducting programme evaluation is to contribute to programme improvement (Candlin, 1998; Rea-Dickins and Germaine, 1998).

I suggest there is a need for an evaluation procedure for FLT-training programmes, including a mechanism for obtaining and using feedback on whole programmes, not individual courses, from students, teachers and others. This would be a step towards the professionalization of the field of English-language teaching and make a useful contribution to theory. I also suggest that the new procedure will be of value in helping to answer Zhong’s (1985, p. 61) question ‘What constitutes adequate training of a foreign language teacher?’, and that the procedure will facilitate and encourage the evaluation of other teacher-education programmes elsewhere. The model could be useful for other programmes.

2 The evaluation procedure

I created the procedure by reviewing literature on what others recommend regarding the content and procedures of FLT education programmes. It aims to collect appropriate data in order to assess programmes in terms of how far they match those recommendations. The procedure is:

Review the literature and produce a set of questions.• Establish appropriate sources of data in your setting.• Choose and design data collection methods and instruments.• Collect and analyse each set of data against your questions.• Construct an account by relating each interpretation to the others.•

3 Testing the procedure

I tested the system by evaluating the effectiveness of one TEFL programme, and then evaluated the system. The programme stakeholders in the University are the students and teachers on the BA in TEFL, those responsible for education in Hong Kong, the Government, and the University. Bartolome’s ‘sociocultural context’ here means EFL classrooms in Hong Kong, where English is regarded as a foreign, not second language. The questions in this case were as follows:

Does the programme:

1) … have a clearly stated philosophy?2) … reflect programme philosophy?3) … promote trainee flexibility in using different teaching approaches for

different situations?

Matthew Peacock 263

4) … promote the ability to use, and to adapt, foreign-language-teaching materials?

5) … balance received versus experiential knowledge? 6) … incorporate and encourage trainee reflection on the experiences and

values they have when they enter the programme? In particular, does it encourage trainee reflection on their ‘apprenticeship of observation’?

7) … promote the skill of reflection and self-evaluation as a teacher? 8) … promote future reflective practice? 9) … promote the ‘long-term, developmental nature of learning to teach’ –

does it promote post-qualification teacher growth and development?10) … have good linkage among courses, avoiding overlaps?11) Is the programme up-to-date?12) … balance teacher- and student-centred learning?13) … prepare EFL teachers to function in the sociocultural context in

which they will work?14) Do students believe the programme meets their needs, is relevant to

their needs, and adequately prepares them for classroom teaching?15) … incorporate and balance linguistic, pedagogic, and managerial

competence to an appropriate degree? Linguistic competence here means L2 proficiency. Pedagogic competence refers to teaching skills plus knowledge of language and second language acquisition.

III Research method

This section, section 4 ‘Results’ and part of section 5 ‘Discussion and Conclusions’ describe the evaluation of an existing programme. The research design included qualitative and quantitative research methods. Aksamit et al. stress qualitative methods (1990, p. 217; also see Ross, 2003; Kogan and Shea, 2007), and they suggest that evaluators should be subjective partners with stakeholders in reflecting on the programme (p. 219). I used multiple methods and triangulation of data, integrating the views of differ-ent participants at different stages; Wang (2007, pp. 1014–16), Rea-Dickins and Germaine (1992; 1998, p. 8), and Roberts (1998, p. 235) all stress the importance of this for programme evaluation. I gathered the views of vari-ous participants, e.g. students and teachers, about the whole programme, as Wallace (1991) and Weir and Roberts (1994) recommend. Baartman et al. stress in particular the importance of teacher opinions (2007, pp. 858–60).

1 Subjects, school and programme

The subjects were 166 third-year students (all trainee teachers, 98% Hong Kong Chinese, 2% Westerners; average age 21; 20% male, 80% female), eight programme teachers, the coordinators of all 35 courses on the programme, and those programme alumni who responded to a posted questionnaire.

264 FLT education programmes

The students were all full-time undergraduates on the TEFL programme in the Department of English at the City University of Hong Kong, a three-year full-time preservice EFL teacher-training programme for local students who wish to teach English in secondary schools. The degree is an initial teaching qualification, though graduates need a postgraduate Diploma in Education for promotion, and enjoys a high degree of international recognition; it has a yearly intake of 38. Trainees complete a large number of courses e.g. ‘The practice of language teaching’ and ‘How languages are learned’. Each course is three hours a week for 13 weeks. The minimum starting level of English for programme students is TOEFL 525/IELTS 5.5. Students do not take either test but enter on the results of local EFL tests.

The programme has a procedure for regular external evaluation: an exter-nal examiner evaluates it biannually, though visits last only three days. The Programme Committee meets regularly but there is no system for evaluation of the whole programme, as opposed to individual courses. I suggest that the current mechanisms for internal evaluation of the whole programme are inadequate.

2 Aims of the project

The aims of this project arose out of Section II.1 ‘Rationale for creating a new evaluation procedure’:

1) Design a procedure for evaluating FLT training programmes based on existing principles of programme evaluation and models of teacher education, and answer Zhong’s question ‘What constitutes adequate training of a foreign-language teacher?’

2) Test the new procedure through a global internal evaluation of one programme. Programme effectiveness was assessed regarding the extent to which it meets the needs of the students, i.e. preparing them to teach in the local context.

3) Evaluate the new procedure and assess its value for use in other contexts.

3 Data-collection instruments and procedure

This research took several years and used six data-collection methods. I collected and analysed most data; exceptions were help from trained assistants with student interviews, another programme teacher with evalu-ating course materials, and the programme leader who collected alumni questionnaires.

a Student interviews (n = 101): The assistants conducted individual 10–15 minute interviews of 101 third-year students, asking them (1) ‘What are the

Matthew Peacock 265

strengths and weaknesses of the programme? (2) How can it be improved?’ Interviews lasted 10–15 minutes, and the interviewers took notes.

b Student questionnaires (n = 65): I collected questionnaires over several years from 65 third-year students. The questionnaire has 22 items (all may be seen in Table 1 below, section 4.2). It collects student opinions on a number of the above 15 questions. I constructed the questionnaire and piloted it before use. Questionnaires collected quantitative data and students answered on a five-point scale, strongly agree-agree-neither agree nor disagree-disagree-strongly disagree. Students did not write their names on the questionnaires.

c Teacher interviews (n = 8): I interviewed eight randomly selected pro-gramme teachers to elicit their views and comments, showing them the list of 15 questions and asking them to give comments, particularly on programme strengths and weaknesses and how to improve the programme. Interviews lasted 20 minutes.

d Student essays – programme philosophy (n = 35): I asked 35 third-year TEFL students to write a 100–150 word essay on the question ‘Does the pro-gramme reflect programme philosophy?’ The aim was to collect opinions on question (2) above; it also gathers some information on questions (5), (12), (13) and (15). Students were given a copy of the philosophy (Appendix), and one month to write.

e Evaluation of course materials: This was done in two stages, independ-ently by me and another programme teacher: (1) we contacted the coordinators of all 35 courses on the programme and gathered full written descriptions of the goals, objectives, assignments, and assessment criteria. (2) We evaluated courses in terms of the balance between (i) linguistic competence/L2 proficiency, (ii) pedagogic competence, and (iii) managerial competence. ‘Pedagogic competence’ refers to both teaching theory, and to teaching practices. Theory involves teaching skills plus the essential knowledge of language and language acquisition. Practices means teaching, planning for teaching, and reflecting on it afterwards. The aim was to gather data on question (15) above.

f Alumni questionnaires: The programme leader designed and posted short questionnaires to all alumni, asking for suggestions on how to improve the programme and on problems they have teaching.

4 Data analysis

I searched the qualitative data from student interviews, teacher interviews and alumni questionnaires for useful information and extracted com-ments, and quantified data from student essays. I calculated percentages of

266 FLT education programmes

agreement/disagreement for all student questionnaire items and collapsed the data into three categories for clearer presentation, i.e. combining strongly agree/agree and disagree/strongly disagree. I checked correlations among questionnaire items using Pearson’s r in the Statistical Package for the Social Sciences with statistical significance set at p < .01. As noted above, two pro-gramme teachers independently evaluated course materials and assessed the proportion for each area; after this we calculated percentages for each area and assessed inter-rater reliability by comparing the results from the both teachers.

IV Results

I completed the three aims of the project. In this section I present the results of the application of the new procedure.

1 Student interviews (n = 101)

The individual interviews of 101 final-year students were very useful. The data fell naturally into two categories, following the interview questions: strengths of the programme, and weaknesses of the programme with suggestions for improvement. I will use these categories to present the results.

a Strengths of the programme The most common comment, made by a third of students, was that the programme is very practical in preparing them to become teachers. Typical comments were ‘The programme is very professional in training English teachers’ and ‘I have learned a lot of teach-ing skills’. Two other top strengths, mentioned by almost a third of students, were that practice teaching is very useful and that teachers are very helpful and easy to contact. Students said ‘Practice Teaching is a very good oppor-tunity for us to apply the learned teaching theories in practice’ and ‘I feel that many lecturers and tutors are very helpful ... we can easily contact them’. Many students also praised the amount of theory on the programme e.g. ‘It is good that the programme includes many teaching theories’ and ‘I have learned many teaching theories and skills’, while others said there is a good balance between theory and practice. Other common comments were that the programme improved their English, that it is professional, that grammar and phonetics are very useful courses, that the small tutorial groups allow them to receive more attention, and that the programme has been carefully designed.

b Weaknesses of the programme, and suggestions for improvement Students highlighted a number of weaknesses and made helpful suggestions for improvement. No less than half the students said the programme should include a postgraduate diploma in education, and more practice teaching. Typical comments were ‘It would be better if the programme could be

Matthew Peacock 267

changed from three years to four years like … Hong Kong University which includes the educational diploma’ and ‘It would be better if the period of practice teaching could be extended [we need this to learn how to teach, to get a job]’. Another common suggestion, made by nearly half the students was to increase time for English proficiency development e.g. ‘I think it would be better if more English enhancement courses could be added into the programme to consolidate our English proficiency’. Many students found the Year 3 assignment load too heavy and pace too fast. A typical comment was ‘Since some courses in year 3 are very heavy, such as Practice Teaching and the Final Year Project, it allows very little time for us to focus on these courses and the assignments’. Other common suggestions were to add educational technology/I.T. courses, that input on classroom management was ‘insufficient’ and should be taught ‘before we go out to do practice teach-ing’, to provide a workroom/computer room for students, and to let them do the Final Project in pairs.

2 Student questionnaires (n = 65)

I computed the internal consistency reliability of the third-year student questionnaire at.8687 using Cronbach alpha based on the entire sample. Results are below in Table 1:

Items 8, 11, 7 and 9 attracted the highest levels of student agreement: 71% of students agreed with 8 ‘The TEFL programme encouraged me to be a reflective teacher (when I start teaching)’, 60% with 11 ‘taught me how to teach English’, 58% with 7 and 9, ‘encouraged me to reflect on my past experiences as a language learner’ and ‘promotes flexibility in using different teaching practices for different situations’, and 55% with 12 ‘taught me how to evaluate myself as a teacher’. Items 19, 3, 5, 13, 18, 21, 15, 14, and 2 attracted the lowest levels of student agreement: only 13% of students agreed with 19 ‘The TEFL programme has a good balance between the teaching of: English, teaching skills, and classroom management skills’; 29% with 3, 5, and 13 – ‘gave me adequate training in English’, ‘gave me adequate training for the needs of the local context (teaching in Hong Kong schools)’, and ‘taught me classroom management skills’; 32% with 18 and 21 ‘is relevant to my needs’ and ‘met my needs’; and 33%/36% respectively with 15 and 14, ‘taught me how to adapt, and use, foreign-language-teaching materials’. Finally, item 2 shows that only 35% agreed that the programme avoids overlapping informa-tion between different courses.

I found modest correlations between some questionnaire items: 0.55 between items 3 and 11, 0.55 (items 4 and 13), 0.48 (11 and 13), 0.50 (11 and 17), 0.41 (12 and 18), 0.41 (11 and 19), 0.50 (18 and 19), 0.49 (17 and 21), and 0.44 (19 and 21). ‘Modest’ refers to relationship strength. Below 0.2 is very low, 0.2–0.39 low, 0.4–0.69 modest, 0.7–0.89 high, 0.9–0.99 very high (Bryman and Cramer, 1990, pp. 168, 171). All correlations were statistically significant

268 FLT education programmes

Table 1 Student (n = 65) questionnaire results

Number The TEFL programme … Agree or stronglyAgree

Neither agree nor disagree

Disagree or strongly disagree

1 … has good linkage between different courses.

42 32 26

2 … avoids overlapping information between different courses.

35 29 36

3 … gave me adequate training in English.

29 26 45

4 … gave me adequate training in teaching skills.

52 22 26

5 … gave me adequate training for the needs of the local context (teaching in Hong Kong schools).

29 39 32

6 … is up-to-date. 48 39 13 7 … encouraged me to reflect on my

past experiences as a language learner.

58 32 10

8 … encouraged me to be a reflective teacher (when I start teaching).

71 23 6

9 … promotes flexibility in using different teaching practices for different situations.

58 19 23

10 … balances teacher-centred and student-centred learning on its courses.

39 42 19

11 … taught me how to teach English. 60 33 712 … taught me how to evaluate myself

as a teacher.55 32 13

13 … taught me classroom management skills.

29 26 45

14 … taught me how to use foreign language teaching materials.

36 45 19

15 … taught me how to adapt foreign language teaching materials.

33 47 20

16 … increased my powers of self-evaluation.

51 39 10

17 … taught me foreign language testing and evaluation skills.

49 32 19

18 … is relevant to my needs. 32 52 1619 … has a good balance between the

teaching of: English,teaching skills, and classroom management skills.

13 29 58

20 … prepared me to teach English in the classroom.

45 42 13

21 … met my needs. 32 39 2922 By the end of the TEFL Programme,

I will be ready to teach English.35 42 23

Matthew Peacock 269

(p < .01), indicating they did not happen by chance. I suggest that they reveal student opinions about FLT education and being a teacher, and interpret them as follows: items 4, 11, 18 and 21 are all concerned with student opinions on whether their needs were met concerning their training to be English teachers. These items correlated with items 3, 12, 13, and 17. This indicates that students believe that important parts of learning how to teach English are receiving adequate training in the English language, learning how to evaluate themselves as teachers, learning classroom management, and learning foreign-language testing skills. The correlations between items 11, 18 and 21 and item 19 are a student comment on the programme: they indicate that students believe that it should have a good balance between the teaching of English, teaching skills, and classroom management skills.

3 Teacher interviews (n = 8)

Teacher interviews were valuable and provided a different perspective on the programme. Regarding programme strengths, several teachers said balance is a strength. Comments included: ‘Nice balance between language proficiency, the nature of language, and methodology … we give our students exactly what they need’ and ‘Nice balance of teaching theories and teaching experi-ence’. Several also said: ‘We have a lot of emphasis [on] reflection … raising awareness in students of issues of learning and teaching’ and ‘Very strong on pedagogical knowledge’. Others said that the programme promotes ‘the spirit of being a teacher’, meaning that it promotes the fundamental essence or real meaning of being a teacher.

Three teachers responded to question 9 and said the programme does promote the long-term, developmental nature of learning to teach, promote post-qualification teacher growth and development, and that it is career-oriented. Two teachers responded to question 13 and said that the programme does well in preparing students to function in the local context in which they will work, and that it is sensitive to that context, noting our close links with local schools. Other comments were that the programme is well-organized, ‘fosters creativity’, and that ‘staff–student relationships are good’.

Regarding weaknesses, several teachers said there is too little practice teaching, e.g. ‘Practice teaching, gaining experience in schools, is limited. Five weeks is insufficient’. Others said that more knowledge of linguistics and language awareness was needed, e.g. ‘We focus too much on the teach-ing side not the language side’ – though as they said pedagogical knowledge is strong, ‘teaching side’ must refer to knowledge of language and second language acquisition. Teachers also said ‘We need more on language aware-ness, especially about the wider sociological setting’. Three teachers said student workload is unbalanced, with too much at the end of semesters. Two said staff ‘do not know enough’ about the Hong Kong education system and are sometimes ‘out of touch with developments’ there, and that there is not

270 FLT education programmes

enough focus on being a member of a school community. They added that the programme needs to ‘promote the culture of teaching and what it means to be a teacher’.

4 Student essays: programme philosophy (n = 35)

The programme has a philosophy (Appendix 1), which is given in the student handbook. Essay analysis shows that 77% of students think the balance is poor among the three ‘sources’ of subject matter competence, skills and awareness on the programme. 64% said it needs more on ‘awareness’, mean-ing understanding learning and teaching and the sociocultural context, because they need this once they start teaching. 32% said the programme needs more on ‘subject matter competence’ or knowledge of and proficiency in English. 38% said it needs more on ‘skills’ (teaching methods), and another 21% said courses in this area are too theoretical and/or impractical. 86% of students said the programme does engage students as active participants in learning. However, 66% of students said lecturers do not ‘use a variety of teaching modes to stimulate real learning’. Most said the main mode was lecture plus handouts, and 14% said there was little or no ‘student-directed learning’. Only 20% said the programme uses ‘a variety of teaching modes’.

5 Evaluation of course materials

The two raters had very similar results: we assessed inter-rater reliability at above 0.9, an acceptable figure. Evaluation revealed the following:

1) Linguistic competence makes up 18% of the programme, mostly in year 1. It makes up 42% of the year.

2) Pedagogic competence makes up 60%. Teaching theory makes up 36%, and teaching practices, 24% (actual teaching, 5%; planning for that teaching, 12%; and reflecting on it afterwards, 7%). Teaching practices are covered mostly in the last two years.

3) Managerial competence makes up 1% of the programme.4) I will address the question of what an appropriate balance is among these

three areas below, in ‘Discussion and conclusions’.

6 Alumni questionnaires

This part of the research was not successful. The questionnaire return rate was only 20%, despite further efforts to contact alumni. Few respondents wrote comments on the questionnaire; most only circled a number on the job satisfaction scale. The teachers rated their job satisfaction at a mean of 4.8 on a scale of 1 ‘very unsatisfactory’ to 7 ‘very satisfactory’. Their sug-gestions, ranked commonest first, for improving the programme were to

Matthew Peacock 271

add components on improving student motivation, classroom management, counselling students, information technology, teaching grammar and teaching English literature.

V Discussion and conclusions

Student interviews, teacher interviews, student questionnaires, student essays and the evaluation of course materials all provided valuable data. They allow us to evaluate the programme from multiple perspectives, and try to answer the 15 questions from Section II.3. I will first discuss the case study results and their implications, and then, more importantly, comment on and assess the new evaluation procedure.

1 The case study: assessing the programme

The programme does have a clearly stated philosophy (Wallace, 1991, questions 1 and 2). However, programme content does not reflect programme philosophy well. Student essays indicate an imbalance among the three areas of subject matter competence, skills and awareness. Most students said the programme needs more on ‘awareness’ (understanding learning and teaching and the sociocultural context). Teacher interview comments back up this find-ing, e.g. ‘We need more on language awareness, especially about the wider sociological setting’, as do student answers on questionnaire item 5. These comments are somewhat surprising, considering students’ ‘ apprenticeship of observation’, and I therefore conclude that the comments refer to teaching in the sociocultural context, not learning in it (i.e. being part of it as a learner). A majority of students also wrote that lecturers do not ‘use a variety of teaching modes to stimulate real learning’, as the philosophy claims, and answers on questionnaire item 10 support this.

The programme does appear to promote trainee flexibility adequately in using different teaching approaches for different situations (Crandall, 1993, question 3). Student and teacher interview responses, e.g. ‘very professional … very strong’, indicate satisfaction with the teaching skills components of the programme, as do the 58% and 60% student agreement with questionnaire items 9 and 11. However, the programme does poorly in promoting the ability to use and adapt FLT materials (Wallace, 1991, question 4): only about a third of students agreed with questionnaire items 14 and 15.

Teacher interview comments indicate that the programme successfully promotes teacher reflection and self-evaluation, and incorporates experiential knowledge to a satisfactory degree (Wallace, 1991; Grosse, 1991; Liou, 2001; Bartolome, 1994; Lynch, 2003 and others, questions 5, 6, 7 and 8), as do levels of student agreement with questionnaire items 7, 8 and 12. Teachers also said it promotes the ‘long-term, developmental nature of learning to teach’ and teacher development (Stoynoff, 1999; Lo, 2005, question 9). On the other

272 FLT education programmes

hand, the level of student agreement (42% and 35%) with questionnaire items 1 and 2 suggest the programme does not do well in linking courses and avoiding overlaps (Weir and Roberts, 1994, question 10). And less than half of students think the programme is up-to-date (Bartels, 2005; Wallace, 1991, question 11, questionnaire item 6).

Student essays suggest that the programme does not balance teacher-centred and student-centred learning (Bartels, 2005; Wallace, 1991, question 12), and this is backed up by only 39% of students agreeing with questionnaire item 10. In their essays most students implied it is weak in preparing students to function in the sociocultural context in which they will work (Bartolome, 1994, question 13). Teacher interview data, and only 29% of students agree-ing with questionnaire item 5, both corroborate this. ‘Function’ here means teach in EFL classrooms.

The question of whether the programme meets and is relevant to student needs, and adequately prepares them for teaching (Wallace, 1991; Reid, 1996, question 14), is fairly broad. Student interview data indicate it does not adequately meet their needs in three areas: gaining a postgraduate diploma in education; sufficient practice teaching; and consolidating English proficiency. Teachers agreed with the second point during interviews. Additionally, only 32% of students felt the programme is relevant to and met their needs (questionnaire items 18 and 21). However, 52% were not sure whether it is relevant, leading us to speculate that students may not be aware of what their needs will be when they start teaching. Regarding the question of whether the programme adequately prepares them for teaching, students seem satis-fied with certain teacher preparation aspects. During interviews students said its top strength is its practicality in preparing them to teach; teachers also named the teaching of pedagogic knowledge as a strength. This is backed up by student agreement with questionnaire items 4 ‘adequate training in teach-ing skills’ and 11 ‘taught me how to teach English’, which at 52% and 60% respectively was higher than for most items. On the other hand, only 35% of students agreed with student questionnaire item 22, indicating that they would not feel ready to teach English by the end of the programme; yet this conflicts to some extent with question 11. It is difficult to know why only 35% of students agreed with this item, but I speculate that this was for two reasons: students do not yet feel ready to teach in the relevant sociocultural context, and year 3 students lack confidence in their teaching abilities.

The modest correlations I found between certain questionnaire items are also relevant to the question of whether the programme meets and is relevant to student needs, and adequately prepares them for classroom teaching. They appear to reveal what students believe some of their needs to be, i.e. receiving adequate training in English, learning how to evaluate themselves as teachers and learning classroom management and foreign-language testing skills.

Regarding an appropriate balance among linguistic, pedagogic and managerial competence (Wedell, 1992, question 15), I conclude that the

Matthew Peacock 273

programme incorporates but does not balance these three areas. It is difficult to say what an appropriate balance is, and to some extent I have to rely on participants’ opinions. Students certainly perceive a misbalance; only 13% of students agreed that it balances these three (questionnaire item 19). I suggest that the stronger focus on language skills in year 1, and increase in teaching practices by year 3, are desirable features of the programme. However, only 24% of the programme engages students in actual teaching practices, and only 5% in actual teaching. Half the students we interviewed said this is not enough, and a weakness; interviewed teachers agreed. Regarding pedagogic competence, I conclude that the programme is stronger in promoting teaching skills than the vital knowledge of language and second language acquisition. Additionally, students said the programme needs more on English profi-ciency. Almost half the students we interviewed said they wanted more time here; and student essays, where 32% said they wanted more, and question-naire item 3, back this up to some extent. Only 29% of students feel it has enough. On the other hand, I note that teachers in interviews said that there was an appropriate balance among English proficiency, the nature of lan-guage, and teaching methods, and I speculate that many other experts would feel that 42% of year 1 is sufficient.

Regarding managerial competence, I suggest that few experts would consider that 1% is appropriate. Student interviewees and alumni mentioned lack of input on classroom management skills as a programme weakness; and on questionnaire item 13, only 29% of students agree the programme taught them this. Lack of management skills may be a problem. Richards (1990, p. 11) stresses that discipline problems are fewer, motivation higher and learning more efficient in a well-managed class and, as I said above, many experts emphasize its importance.

Other weaknesses emerged: not enough on testing or educational technol-ogy; the heavy and unbalanced workload (particularly in year 3); and teaching methods courses being too theoretical and/or impractical. Other strengths were teachers being very helpful and easy to contact, and the programme having ‘plenty’ of theory and a good theory–practice balance.

It is possible to use these results to make suggestions for programme improvement, and they also have implications for EFL teacher training. Robinson (2003, p. 199) called on evaluators to form judgments about a programme’s value and how far it meets its objectives. Regarding the pro-gramme’s value for preservice EFL teacher training, I note that the latest external examiner’s report called it a high-quality professional preparation programme of high academic standard that stands comparison with similar programmes elsewhere. While I do not disagree with the external examiner, this evaluation revealed certain areas where improvements are possible and desirable. The programme appears strong in teaching certain aspects of pedagogic skills (including flexibility in using different teaching approaches for different situations) and promoting teacher reflection and self-evaluation.

274 FLT education programmes

Staff–student relationships are good, and the programme also seems sound at promoting the ‘long-term, developmental nature of learning to teach’ and post-qualification teacher development, and in having sufficient theory and a good theory–practice balance. However, I suggest it needs strengthening in vari-ous areas: more input on classroom management and the ability to use/adapt teaching materials; more teaching practice; better linkage among courses (avoiding overlaps); and more instruction regarding teaching within the socio-cultural context (strengthening this may give students more confidence in their teaching ability) and in the important area of language and second language acquisition. It also needs to balance the workload more carefully. Regarding the philosophy, the programme should adhere to it; I also question whether it is adequate. It could be strengthened, as it talks more about programme content than philosophy (beliefs and values). Finally, the programme does not have a built-in procedure for internal evaluation of the whole programme, as Robinson (2003) recommends. We can add one. In addition to these areas, stu-dents say the programme is not up-to-date and does not balance teacher- and student-centred learning, and that it needs more input on English proficiency, foreign-language testing and educational technology. However, I cannot con-firm the need for more in these areas. That needs further evaluation.

Several writers (e.g. Robinson, 2003, p. 210) note that an important prob-lem with programme evaluation is getting the information across to decision-makers, who can then act on it. Detailed evaluation results must be sent in writing to programme committees. However, a serious and difficult problem for programme planners may well be that adding components involves adding student hours and de-emphasizing or removing other components. We must communicate these evaluation results to other stakeholders: students, teach-ers, the Government and departmental and university planners and adminis-trators. Programme committees must set up a clear mechanism for doing this; methods can include reports and presentations.

2 Assessing and improving the programme evaluation procedure

While this procedure evaluated a programme in one context, I suggest others can adapt it for other contexts. To assist this I will generalize out from the specifics of the case study and assess the validity, reliability, strengths and weaknesses of the procedure. I will discuss how well the data-collection instruments worked and what I recommend others to do.

I do not propose that this evaluation procedure is valid for all contexts. Clearly, informants and situations differ. However, as I said above, I suggest it will facilitate and encourage the evaluation of other teacher-education pro-grammes. I also suggest the procedure is useful in that it has a comprehensive list of 15 questions that arose from relevant literature, and collects a lot of valuable information from various programme stakeholders. Other key factors in the procedure are collecting data from participants in sufficient numbers,

Matthew Peacock 275

e.g. entire student cohorts, teachers and alumni, and collecting a variety of data, e.g. from student interviews, teacher interviews, alumni questionnaires, student essays and course materials.

I propose that the student and teacher interviews are very valuable. I concluded that one helpful feature was that participants were not constrained by overstructured questions, but asked to comment on programme strengths and weaknesses. This, particularly for students, allowed them to reflect and comment on areas of particular and practical importance to them, e.g. the lack of a PGD in education, practice teaching, and work on English proficiency. The additional step with teachers of showing the list of 15 questions also helped them provide a more comprehensive analysis, because it focused their thoughts on a wide range of relevant areas.

I found the student questionnaire to be of value and it provided a lot of useful data. The internal consistency reliability of the questionnaire was an acceptable 0.87. However, possible differences between student and teacher understanding of some concepts (e.g. ‘adequate’ in questions 3, 4, and 5, ‘good’ in question 19, and ‘reflect/reflective’ in questions 7 and 8) slightly restrict the value of the questionnaire. Also, item 5 could be clearer; items 11, 18, 20 and 21 are rather broad and generalized; and students may not be fully competent to answer on item 6, and possibly item 19.

The student essays were also valuable, but a programme teacher collecting them could possibly have constrained students. The evaluation of course mat-erials provided crucial information and inter-rater reliability was above 0.9, an acceptable figure. The alumni questionnaires, on the other hand, were not at all successful. The return rate was 20%, normal for postal questionnaires. This was probably because of changing addresses and lack of interest, but more efforts to contact alumni would help.

A weakness in this research was not contacting employers of programme graduates, which was beyond the scope of this study. However, this missed step is an important one for future improvement of the process. Other sug-gestions for improving further research (not carried out during this project, some because of lack of time and access to classrooms, others because they emerged only on later reflection) are:

to collect questionnaires from more students; • to give similar questionnaires to teachers and compare student and • teacher results; to interview a larger number of teachers; • to conduct lesson observations focusing on the content and process of • lessons; and to collect more data from alumni, e.g. by school visits, extensive inter-• views, essays and a better questionnaire.

Finally, checking course readings would help answer the question of whether the programme is up-to-date.

276 FLT education programmes

This article provides an account of the design and test of a procedure for evaluating FLT education programmes, assesses its strengths and weaknesses, and makes suggestions for improvement. I believe that all teacher-education programmes should incorporate a built-in procedure for overall internal evaluation, and use it regularly. I hope that the new procedure will facilitate and encourage this. Critical review of programmes is a beneficial and essential exercise.

VI References

Aksamit, D.L., Hall, S.P. and Ryan, L. (1990). Naturalistic inquiry applied to the evaluation of a teacher education program. Teaching and Teacher Education, 6(3), 215–26.

Baartman, L.K.J., Bastiaens, T.J., Kirschner, P.A. and Van der Vleuten, C.P.M. (2007). Teachers’ opinions on quality criteria for Competency Assessment Programs. Teaching and Teacher Education, 23(6), 857–67.

Bartels, N. (2005). Applied linguistics and language teacher education: What we know. In Bartels, N. (Ed.), Applied linguistics and language teacher education (pp. 405–24). New York: Springer.

Bartolome, L.I. (1994). Beyond the methods fetish: toward a humanizing pedagogy. Harvard Educational Review, 64(2), 173–94.

Bryman, A. and Cramer, D. (1990). Quantitative data analysis for social sciences. London: Routledge.

Candlin, C.N. (1998). General editor’s preface. In Rea-Dickins, P. and Germaine, K.P. (Eds.), Managing evaluation and innovation in language teaching: building bridges (pp. xiv–xxi). London: Longman.

Cots, J.M. and Arno, E. (2005). Integrating language teachers’ discipline knowledge in a language course. In Bartels, N. (Ed.), Applied linguistics and language teacher education (pp. 59–78). New York: Springer.

Crandall, J. (1993). Professionalism and professionalization of adult ESL literacy. TESOL Quarterly, 27(3), 497–515.

Freeman, D. (1996). The ‘unstudied problem’: research on teacher learning in language teaching. In Freeman, D. and Richards, J.C. (Eds.), Teacher learning in language teaching (pp. 351–78). Cambridge: Cambridge University Press.

Freeman, D. and Johnson, K. (1998). Reconceptualizing the knowledge-base of language teacher education. TESOL Quarterly, 38(1), 397–417.

Freeman, D. and Johnson, K.E. (2004). Readers react… Common misconceptions about the quiet revolution. TESOL Quarterly, 38(1), 119–27.

Grosse, C.U. (1991). The TESOL methods course. TESOL Quarterly, 25(1), 29–49.Johnson, K.E. (2006). The sociocultural turn and its challenges for second language

teacher education. TESOL Quarterly, 40(1), 235–57.Kogan, J.R. and Shea, J.A. (2007). Course evaluation in medical education. Teaching

and Teacher Education, 23(3), 251–64.Liou, H.-C. (2001). Reflective practice in a pre-service teacher education program for

high school English teachers in Taiwan, ROC. System, 29(2), 197–208.

Matthew Peacock 277

Lo, Y.-H.G. (2005). Relevance of knowledge of second language acquisition. In Bartels, N. (Ed.), Applied linguistics and language teacher education (pp. 135–57). New York: Springer.

Lortie, D. (1975). Schoolteacher: a sociological study. Chicago, IL: University of Chicago Press.

Lynch, B.K. (2003). Language assessment and programme evaluation. Edinburgh: Edinburgh University Press.

Muchisky, D. and Yates, R. (2004). The authors respond… Defending the discipline, field, and profession. TESOL Quarterly, 38(1), 134–40.

Rea-Dickins, P. and Germaine, K. (1992). Evaluation. Oxford: Oxford University Press.

Rea-Dickins, P. and Germaine, K.P. (1998). The price of everything and the value of nothing: trends in language programme evaluation. In Rea-Dickins, P. and Germaine, K.P. (Eds.), Managing evaluation and innovation in language teaching: building bridges (pp. 3–19). London: Longman.

Reid, J.M. (1996). Let’s put the ‘T’ back in TESL/TEFL programs. TESOL Matters, 5(6), 3.

Richards, J.C. (1990). The dilemma of teacher education in second language teaching. In Richards, J.C. and Nunan, D. (Eds.), Second language teacher education (pp. 3–15). Cambridge: Cambridge University Press.

Roberts, J. (1998). Language teacher education. London: Edward Arnold.Robinson, B. (2003). Evaluation, research and quality. In Robinson, B. and

Latchem, C. (Eds.), Teacher education through open and distance learning (pp. 193–211). London: Routledge/Farmer.

Ross, S.J. (2003). A diachronic coherence model for language program evaluation. Language Learning, 53(1), 1–33.

Stoynoff, S. (1999). The TESOL practicum: An integrated model in the US. TESOL Quarterly, 33(1), 145–51.

Wallace, M.J. (1991). Training foreign language teachers. Cambridge: Cambridge University Press.

Wang, W. (2007). Evaluation of 2 + 2 alternative teacher performance appraisal pro-gram in Shanxi, People’s Republic of China. Teaching and Teacher Education, 23(7), 1012–23.

Wedell, M. (1992). Pre/in service training of ELT teacher trainers: Planning the regional MATEFL project. In Flowerde, J., Brock, M. and Hsia, S. (Eds.), Perspectives on second language teacher education (pp. 337–50). Hong Kong: City University of Hong Kong.

Weir, C. and Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell.White, R. (1998). What is quality in English language teacher education? English

Language Teaching Journal, 52(2), 133–39.Wu, X. (1996). Train ourselves first? The Teacher Trainer, 10(3), 20–22.Yates, R. and Muchisky, D. (2003). On reconceptualizing teacher education. TESOL

Quarterly, 37(1), 135–47.Zhong, Y. (1985). Towards adequate E.F.L. teacher education in China: an analysis of

the problem. Australian Review of Applied Linguistics, 8(1), 61–69.

278 FLT education programmes

Appendix 1: Programme philosophy (Student handbook)

The teaching of English as a second language draws on three sources: sub-ject matter competence, skills and awareness [emphasis in the original]. Subject matter competence relates to knowledge and proficiency in the sub-ject of teaching, i.e. the English language. Skills relates to methodologies of teaching and course development as well as basic competence in the classroom. Awareness relates to an understanding of learning and teaching processes and of the sociocultural contexts in which learning and teaching take place.

The programme seeks to engage students in the learning process as active participants in learning. Lecturers use a variety of teaching modes to stimu-late real learning. Students often work in pairs or groups on class projects, and student-directed learning forms the core of many activities.