Geert Driessen & Peter Sleegers (2000) SESI Consistency of teaching approach and student achievement An empirical test

School Effectiveness and School Improvement 0924-3453/00/1101-0057$15.002000, Vol. 11, No. 1, pp. 57–79 © Swets & Zeitlinger

Consistency of Teaching Approach and StudentAchievement: An Empirical Test

Geert Driessen1 and Peter Sleegers2

1ITS, University of Nijmegen and 2Dept. of Educational Sciences,University of Nijmegen, the Netherlands

ABSTRACT

This article describes the results of a study into the relations between school, teacher,class, and student characteristics in Dutch elementary schools. Central to the study werethe socio-ethnic background of the students, socio-ethnic class composition, language andmath test results, teaching approach, and consistency of teaching approach within theschool. The major question was whether student achievement levels vary according to theconsistency of the teaching approach after controlling for socio-ethnic background at boththe individual and class levels. The sample consisted of 7,410 grade 8 students and in total1,714 teachers from 567 schools. The results of multilevel analyses showed consistencyof teaching approach to be of no relevance to achievement levels. The most importantfactor appeared to be the socio-ethnic background of the students. Ethnic minority stu-dents perform less well than native Dutch working-class students, who in turn performless well than the other students studied. In addition, students in classes with a relativelyhigh number of so-called disadvantaged students perform less well independent of theirindividual socio-ethnic background.

THEORETICAL NOTIONS

In the literature on school effectiveness, consistency of teaching approachis a regularly recurring notion. Research by Mortimore, Sammons, Stoll,Lewis, and Ecob (1988) has shown consistency among teachers to be oneof 12 key factors in the promotion of school effectiveness. In schoolswhere all teachers are found to follow guidelines in the same manner, theimpact on student progress is found to be positive (also see Chitty, 1997).

Creemers (1994a, 1994b) introduced the notion of consistency as atheoretical concept and suggested that isolated instructional components

Correspondence: G. Driessen, ITS, University of Nijmegen, P.O. Box 9048, 6500 KJNijmegen, the Netherlands. Tel.: +31 24 3653545, Fax: +31 24 3653599. E-mail:[email protected].

Manuscript submitted: November 5, 1998Accepted for publication: February 22, 2000

58 GEERT DRIESSEN AND PETER SLEEGERS

do not show a strong effect on student achievement. An integrated ap-proach at the class level is therefore needed along with the introduction ofthe consistency principle. The combined and coordinated elements of ef-fective teaching behavior, teaching materials, and group composition pro-duce a synergetic effect, according to Creemers. It is also important thatthe goals, structure, and evaluation of the various elements occur along thesame line: They need not be the same but should certainly support eachother. In Creemers (1991) and Creemers and Reezigt (1996), the consist-ency principle has been worked out into the following four distinct ele-ments:

• Consistency: at the school level, conditions for effective instructionrelated to curricular materials, grouping procedures and teaching be-havior should be in line with each other.

• Cohesion: all members of the school team show consistency of effec-tiveness characteristics. In this way effective instruction between class-es can be guaranteed.

• Constancy: effective instruction is provided during the total school ca-reer of students. Too often students are confronted with differences ininstruction between teachers and grade levels. Constancy means thatconsistency and cohesion are guaranteed over longer periods of time.

• Control: control refers not only to the evaluation of student achieve-ment and teacher behavior, but also to an orderly and quiet schoolclimate, which is necessary to achieve results. Control also refers toteachers holding each other responsible for effective instruction.(Creemers & Reezigt, 1996, pp. 215–216).

The elements of cohesion and consistency as distinguished by Creemersand Reezigt (1996) also stand out in the extended overview presented byScheerens and Bosker (1997, pp. 99-138). These authors come to a total of13 general factors (e.g., educational leadership, school climate, evaluativepotential) and distinguish a number of additional components within thesefactors. One of the factors is consensus and cohesion among the staffincluding the type and frequency of meetings and consultations, content ofthe cooperation, satisfaction about cooperation, importance attributed tothe cooperation, and indicators of successful cooperation. A detailed spec-ification of the relevant notions is virtually lacking with regard to consist-ency: The degree to which head teachers agree on the importance of align-ing the curricula for the different grade levels, the similarity in teachingapproaches across grades and classrooms, and a common policy on stu-dents with special learning or behavioral problems.

59CONSISTENCY OF TEACHING APPROACH

In a recent publication, the four elements of consistency, cohesion,constancy and control as distinguished by Creemers and Reezigt (1996)have been applied as formal criteria to guarantee teaching effectiveness(Creemers, 1997) and assumed to promote teaching effectiveness at anumber of different levels (i.e., the class, school, and context (policy)levels). According to Creemers, however, consistency is the only elementcontributing to all three levels of school effectiveness.

The strengthening of consistency in the manner of teaching and thedegree of cohesion and consensus among the teaching staff are thus viewedas important factors for expanding school effectiveness (also see Har-greaves, 1995). Closer examination of the notion of consistency neverthe-less shows the concept to not be used unequivocally and multiple interpre-tations to lurk behind the concept: Consistency sometimes pertains to theprinciple of effectiveness promotion, sometimes to one of the dimensionsor elements that have been found to be important for the promotion ofschool effectiveness, and sometimes to a formal criterion for school effec-tiveness that is assumed to apply at a number of different levels. Thenotion of consistency is also often mentioned in relation to cohesion andconsensus within the team without further specification of how the notionsshould analytically relate to each other. In such a manner, Creemers distin-guishes cohesion as one of the four elements of consistency while Scheer-ens and Bosker (1997) take consistency to be one of the dimensions ofcohesion and consensus among the staff. Consideration of the aforemen-tioned in fact reveals two – indeed connected but in principle independ-ent – elements, which can perhaps be best defined as occurring at differentlevels. The one level is the “ideal” level pertaining to all of the ideas,visions, norms, values, objectives, plans, opinions, and policies of theschool team and school administration. The other level is the “practical”level pertaining to the manner in which the school actually and concretelyoperates. Sammons, Hillman, and Mortimore (1995) speak in this connec-tion of the cultural and normative versus the more practical aspects ofconsistency and collegiality.

The division made by Reynolds, Sammons, Stoll, Barber, and Hillman(1996) is largely in keeping with the preceding. Although based on some-what outdated literature, these authors distinguish three elements underthe rubric of “shared visions and goals”:

• Unity of purpose: consensus on values;• Collaboration: collegiality and teacher involvement in decision making;• Consistency of practice: adopting a particular approach to school cur-

riculum guidelines and to discipline. (Reynolds et al., 1996, p. 141).


With regard to this division, one can once again speak of different levels.Consensus with regard to values constitutes the “ideal” level while con-sistency with regard to actual behavior constitutes the “practical” level.

In the present contribution, we will be primarily concerned with con-sistency in the teaching approach. Our definition of “consistency” corre-sponds to what Reynolds et al. (1996) call “consistency of practice.” Con-strued in such a manner, consistency largely pertains in the present articleto the practical level distinguished above. In an extension of this, we canspeak of “consensus” as the correspondence among the teachers within aschool at the ideal level or unity of purpose as distinguished by Reynoldset al.

EMPIRICAL STUDIES

Empirical studies with consistency of teaching approach as the topic ofresearch are virtually nonexistent. We only found two studies in whichthis topic was examined. The first study was the one conducted by Weide(1995). In her research, the effectiveness of the teaching of basic skills forethnic minority students was examined. The data pertain to the teachers ofgrades 6 and 7 of Dutch elementary schools1. On the basis of their ques-tionnaire responses, a number of effective school characteristics were iden-tified. Two of the characteristics are subsumed by Weide under the rubric“cohesion” or the degree to which all of the teachers in a particular schoolcorrespond with regard to the manner in which they effectively teach. Twoscales based on a series of items are involved, namely achievement-orient-ed attitude and degree of differentiation. Given that the study involvedteachers in different grades, Weide first averaged the scores for the twogroups. The degree of consistency was then determined by examining thestandard deviations for the two dimensions. The analyses showed schoolsto strongly differ in the degree of consistency for achievement-orientedattitude and differentiation. No correlation exists between the two dimen-sions: Consistency with regard to being achievement-oriented does notautomatically relate to consistency with regard to differentiation. With theaid of multilevel analyses, Weide then estimated the effects of the differ-ent approach characteristics that were distinguished on the achievement ofthe students in the areas of language and math. These analyses clearly

1. In the Dutch education system, elementary school is intended for children aged 4 to 12years. The first two grades are kindergarten classes in which play occupies a centralplace. In grade 3, formal instruction in reading, math, and writing commences. Grade8 is the highest grade, after which the students move on to secondary school.


show neither of the two consistency dimensions to lead to (extra) signifi-cant effects.

The second study was the one by Creemers (1997) who also examinedthe effects of consistency on the language and math achievement of stu-dents in secondary analyses of data from the Dutch National Evaluation ofthe Educational Priority Policy databases (cf. van der Werf, 1995). Theanalyses pertained to elementary grades 4, 6, and 8. Consistency wasdetermined by performing exploratory cluster analyses on 11 class varia-bles (e.g., learning time, opportunity to learn, degree of differentiation,homework) and 6 school variables (e.g., instructional rules, time rules,evaluation policy). The results of the cluster analyses revealed five lan-guage clusters and three math clusters at the level of the teacher/class. Forlanguage, two clusters on which the teachers scored high or low on morethan 50% of the features could be distinguished. Creemers labeled theseclusters at the level of the teacher: general effectiveness (36% of the teach-ers) and general ineffectiveness (14% of the teachers). For the other threeclusters associated with language, the teachers scored high on a smallernumber of characteristics. Depending on the particular characteristics in-volved, Creemers typified the clusters as: emphasis on homework andobjectives (28% of the teachers), emphasis on evaluation (4% of the teach-ers), and emphasis on differentiation (18% of the teachers). For math atthe teacher level, generally ineffective clusters were detected (39% of theteachers) but not, in contrast to language, generally effective clusters. Theremaining two clusters detected for math were typified by Creemers as:emphasis on subject matter (48% of the teachers) and emphasis on instruc-tion (13% of the teachers).

Cluster analyses on the six characteristics at the school level producedthree interpretable clusters, which were then typified by Creemers (1997)as follows: generally ineffective (43% of the schools), emphasis on rulesand order (36% of the schools), and emphasis on school policy (21% of theschools). No generally effective cluster was found at the school level,which is similar to the findings for the teacher level.

Creemers (1997) next examined the effects of the clusters on the lan-guage and math achievement of the students in multilevel analyses at boththe class and school levels. The expectation was that the effects of thegenerally effective (consistent) clusters in particular would be found tohave a positive effect. The results showed that the generally effectiveclusters at the level of the teacher did not have a positive effect on thelanguage and math achievement of the students; this finding is thus inkeeping with the findings for the generally ineffective clusters at the levelof the teacher. The same picture presents itself for the clusters at the


school level. More generally, the results provide no empirical support forthe assumption that consistency leads to better achievement on the part ofstudents.

In closing, it should be noted that the degree to which consistency in theteaching approach produces particularly effective teaching or not couldnot be determined on the basis of the literature we examined. In the firstplace, the notion of consistency was not employed in an unequivocal man-ner, which makes what is meant by consistency and at which level (i.e.,vision, objectives, concrete class practices) not completely clear. In thesecond place, the results of the two studies described in this section pro-vided no empirical support for the assumption that consistency leads tosignificant effects on school achievement.

RESEARCH OBJECTIVES

An extensive database recently became available and thereby placed us in aposition to examine the effects of consistency on the achievement of Dutchelementary school students. To do this, we characterized “consistency” asfollows: the correspondence between the teachers in a school with regard tothe manner in which they concretely give form to their teaching.

The following three questions constitute the basis for our research:

(1) To what extent can we speak of consistency in the teaching approach-es of teachers from different grades within a school?

(2) Which aspects of teaching approach appear to play a role in the expla-nation of the observed differences in achievement?

(3) What relation exists between the effects of teaching approach and theconsistency of the teaching approach within the team on studentachievement?

The last point pertains to the interactive effects of teaching approach andconsistency on student achievement. Our hypothesis is that the differencesin test results for the different approach levels will be greater to the extentthat the approach is applied more consistently within the school. Moreconcretely, we expect the combination of strong consistency with a weakapproach (consistently doing nothing) to produce the lowest achievementresults, the combination of strong consistency with a strong approach(consistently strongly guided) to produce the highest results, and the com-bination of non-strong consistency with a strong approach to produceintermediate achievement results.


The effects of consistency itself on student achievement will not beexamined. Given that consistency can mean consistently doing nothingwith regard to a particular approach or consistently doing a great deal withregard to a particular approach, the total effect of consistency directlydepends on the relation between the number of classes with the two com-binations of strong consistency and the level of approach. If consistentlydoing nothing negatively influences student achievement and consistentlydoing a great deal positively influences student achievement, then a nega-tive effect of consistency can be expected for an overrepresentation of thefirst combination and a positive effect for an overrepresentation of thesecond combination. In light of the fact that the total effect of consistencydirectly depends on the correlation between approach and consistency, thetotal effect of consistency is basically uninteresting. Consistency is onlyof importance to the extent that it further specifies the effects of teachingapproach (cf. Scheerens & Bosker, 1997).

From a methodological perspective, consistency is determined by stand-ardizing the scores for the different aspects of teaching approach withineach of the grades into z-scores and calculating the standard deviation foreach school thereafter.

It was decided to conduct the analyses on the data from the final ele-mentary school year, grade 8. The influence of the school on the studentsin this grade is presumably the longest, and the cumulative effects of theteaching approach applied in the preceding years will presumably be mostvisible (cf. Bosker, 1991).

Numerous studies have satisfactorily shown educational achievementto be strongly determined by the socio-economic environment in which astudent grows up and the ethnic origin of the student (cf. Driessen &Jungbluth, 1994; Rossi & Montgomery, 1994). The socio-ethnic back-ground of the student not only plays a role at the level of the individual,however, but also at the level of the class. Research has shown, namely,that the socio-ethnic composition of a student’s class has an additionalinfluence on educational opportunities (cf. Tesser, van Praag, van Dugter-en, Herweijer, & van der Wouden, 1995). We will therefore include bothindividual background and class composition as control variables in ouranalyses (cf. Chitty, 1997).

METHOD

SampleIn order to answer the research questions, we made use of the data col-lected within the framework of the national cohort study “Primair Onder-


wijs” (Primary Education; PRIMA). This study was started in the 1994/95 school year with an initial data-collection round among approximate-ly 60,000 students attending approximately 700 elementary schools (i.e.,10% of the total number of elementary schools in the Netherlands). ThePRIMA data were collected using various research instruments from:students and their parents, schools, school management teams, and class-room teachers. For a more detailed explanation of the design and datacollection methods, the reader is referred to Jungbluth, van Langen,Peetsma, and Vierke (1996).

In brief, PRIMA was designed as follows. At the end of 1994 and thebeginning of 1995, tests were administered to the students in grades 4, 6,and 8; at the same time, the teachers who had taught the students in theprevious year (grades 3, 5, and 7) completed a questionnaire with regard tothe teaching strategy they used. Such an approach was opted for becausethe teachers who had taught the students in the previous year were presum-ably more familiar with the students than the teachers currently teachingthem (i.e., teachers with a full year versus 6 months of experience with thestudents). It was also assumed that any effects of teaching strategy weremore likely to be a result of the teachers who had taught the students in theprevious year than of the present teachers.

The present grade 8 subject sample was selected from the PRIMA dataand consisted of 7,410 students from 492 grade 8 classes and 447 schools.Analyses to check for selective non-response showed the analysis sampleto be a good reflection of the original sample (for details, see Driessen &Doesborgh, 1998).

Variables

Socio-Ethnic BackgroundIn view of the fact that no baseline measurements were taken and it wastherefore not possible to check the entry capacities of the children, indi-vidual socio-ethnic background was considered. We applied the classifi-cation system that the Dutch Ministry of Education employs for the im-plementation of Educational Opportunities Policy (cf. Driessen &Dekkers, 1997; Driessen & Mulder, 1999). The three categories distin-guished within the present research were: (1) non-disadvantaged chil-dren; (2) native Dutch disadvantaged children (i.e., children of parentsborn in the Netherlands with a low educational and occupational level);and (3) ethnic minority disadvantaged children (i.e., children of immi-grant parents with a low educational or occupational level). The socio-ethnic variable combines information on socio-economic background


and country of origin.2 Although this variable is actually at an ordinalmeasurement level, it is interpreted at the interval level here because therelations to the test results appear to be linear. The distribution of thegrade 8 sample of 7,410 students according to socio-ethnic backgroundwas found to be: (1) 41.6% non-disadvantaged; (2) 38.6% Dutch disad-vantaged; and (3) 19.8% ethnic minority disadvantaged. Socio-ethnicbackground was included in the analyses as a control variable.

Socio-Ethnic Class CompositionSocio-ethnic background not only plays a role at the individual level butalso at the class level as indicated by recent research. A high concentrationof disadvantaged students within a class can produce additional disadvan-tages for individual achievement (Tesser et al., 1995). In order to gainsome insight into the possibly detrimental effects of disadvantaged chil-dren in the class, socio-ethnic class composition was included as a sepa-rate variable in our analyses. The mean individual socio-ethnic backgroundper class was calculated in order to do this.

Consistency of Teaching ApproachAs already mentioned, the grade 3, 5, and 7 teachers were actually theteachers that the students had the year prior to testing. The same question-naire was administered to the teachers in each grade (cf. Ledoux & Over-maat, 1996). The questions were based on an extensive examination of theliterature using the so-called dragnet procedure. In contrast to Creemers’(1994c) review on the topic of effective instruction, the present review hada more limited focus: Only Dutch empirical studies into those factors thatcan possibly explain differences in elementary school performance wereincluded (cf. Overmaat & Ledoux, 1995). The research questions were notformulated from a particular theoretical point of view but, rather, based onempirical relevance. Almost 40 questions were formulated to gain insightinto the particular teaching approach used. By applying explorative factoranalyses we hoped to obtain a smaller number of approach dimensions. Inorder to do this, z-scores were calculated around the same average of 0with a standard deviation of 1 for each of the three grades. After factoranalyses, still 15 approach ‘dimensions’ (in some instances a better label

2. At present, some 7% of the Dutch population of 15.5 million inhabitants belongs to anethnic minority group; this 7% is largely of Surinamese, Turkish, or Moroccan origin.The percentage minority students in elementary school is considerably higher: it was8% in 1985 and has now grown to more than 12%. In the four big cities (Amsterdam,Rotterdam, The Hague, Utrecht), minority students currently constitute almost 50% ofthe total school population.


would probably be ‘aspects’) remained.3 This result was somewhat disap-pointing: the various teaching aspects proved to be rather loosely correlat-ed. In Table 1, a brief overview of the resulting dimensions of teachingapproach is presented. Despite the more limited focus of our literaturereview, the dimensions of teaching approach we identified closely resem-ble the characteristics of effective instruction distinguished by Creemers(1994c). Some typical elements are: emphasis on the acquisition of basicskills; high expectations for the students; regular evaluation of learningprogress; opportunity to learn; and time on task (also see Creemers, 1994d;Scheerens, 1992).

In addition to the 15 content dimensions of teaching approach, a gener-al measure of approach intensity was constructed by counting the numberof times that a teacher showed a strong approach (i.e., z > .5; also see thenext section “Teaching Approach in Grade 7”) and transforming this into awithin-group z-score. The consistency of the teaching approach was nextcalculated as the standard deviation of the z-scores for all of the grade 3, 5,and 7 teachers within a school (cf. Weide, 1995, for a slightly differentapproach). This concerns a total of 1,714 teachers in 567 schools. Giventhat consistency was defined as the standard deviation of the z- scores, thevalue was almost always between 0 and 2. In order to make the content ofthe scores easier to interpret, the scores were re-coded to vary from 0 (= noconsistency) to 2 (= maximum consistency). In the first two columns ofTable 2, the averages and standard deviations for the different dimensionsof consistency of teaching approach are presented.

As can be seen, the scores for all of the different dimensions areabove the middle of the range 0–2. On average, thus, a reasonable de-gree of consistency was observed in teaching approach. The amount ofvariation around the averages was also found to be sufficient. Thestrongest degree of consistency was found for expectations with respectto the students and instructional orientation; the weakest degree of con-sistency was found for extra emphasis on basic skills. In order to dis-cern any relations between the various dimensions of consistency, inter-correlations were calculated. Almost all of the correlations were foundto be around 0, however. The relative consistency of a particular didac-tic approach thus says nothing about the relative consistency of the oth-er aspects of teaching approach.

3. A number of teacher questions were asked separately for language and math. Thereason why we did not analyze the subject-specific teaching approaches and consist-ency indices, is because we were specifically interested in the general direction, whichis: is there a strongly guided approach or not?


Table 1. Aspects of Teaching Approach.

label characteristics n items

reading time how many hours and minutes are devoted per week to(a) technical reading and (b) reading comprehension 1

language and math time how many hours and minutes are devoted per week to(a) language (writing, speaking, and listening skills;spelling, grammar, language study) and (b) math 2

homework are the students assigned homework in the areas of (a)language and (b) math 2

progress registration how often is student progress recorded for (a) technicalreading, (b) reading comprehension, (c) language(vocabulary, spelling, language progress), (d) math(via, for example, a student monitoring system) 6

instructional orientation in which form (class versus individual) is instructionprovided for (a) reading lessons, (b) language lessons,and (c) math lessons 3

attention reading strategy to what extent is attention paid to the acquisition ofreading strategies

test frequency how often are (a) external tests (method independent)and (b) diagnostic tests (to adapt teaching materialsand instruction) administered for (a)language and (b) math 4

checking: grade was the reason for checking the (a) language and (b)math work: to assign a grade 2

checking: seriousness was the reason for checking the (a) language and (b)math work: to determine whether the students seriouslydid their work 2

checking: understanding was the reason for checking the (a) language and (b)math work: to determine whether the material wasexplained well 2

checking: new assignments was the reason for checking the (a) language and (b)math work: to identify new assignments of tasks 2

checking: error analysis was the reason for checking the (a) language and (b)math work: to undertake an error analysis for a weakstudent 2

own capacities what is the estimate of the teacher’s own capacities(dealing with heterogeneous groups; improvingperformance of disadvantaged students; enabling slowstudents to achieve; teaching difficult students) 4

expectations for students what are the expectations of the teacher with regardto the students (students should do their very best;make hard demands on weak students; in a school likethis, you can’t expect much of the students; skepticalabout the learning capacity of some students) 4

emphasis basic skills does the teaching of this particular teacher distinguishitself from that in other schools through extra emphasison basic skills (language, reading, math) 1

approach intensity total number of “strong” aspects of teaching approach 38


Teaching Approach in Grade 7In order to determine the didactic approach of the grade 7 teachers, thesame features were used as for the determination of consistency. Thesample of grade 7 teachers included 565 teachers. The solution producedby factor analyses pretty much matched the results of the consistencyanalyses across the three grades.

In order to obtain a highly differentiated indication of teaching ap-proach, we derived variables expressing the degree of guidance in theteacher behavior. The z-scores for the original items were dichotomized.In doing this, the following criterion was applied: z > .5 was coded as 1and all other values were coded as 0. The dichotomized scores thus indi-cate strong versus no strong guidance. About 30% of the teachers werefound to have a z-score above .5. The average of the dichotomized scoreswas next calculated, which means that the number of items indicatingstrong guidance for a particular dimension was divided by the total numberof items relevant to that dimension. Given such similar ranges with 0indicating no strong guidance on any item whatsoever and 1 indicating

Table 2. Consistency of Teaching Approach (Means and Standard Deviations), Intensityof Teaching Approach (Means Multiplied by 100) and Correlations Consistencyand Approach.

consistency approach cons.* appr.M (SD) M r

reading time 1.39 (.40) 25 –.22language and math time 1.43 (.41) 20 –.17homework 1.28 (.35) 70 .08progress registration 1.47 (.32) 44 –.09instructional orientation 1.51 (.37) 16 –.41attention reading strategy 1.21 (.44) 18 –.29test frequency 1.48 (.32) 25 –.29checking: grade 1.26 (.42) 24 –.25checking: seriousness 1.23 (.46) 60 .25checking: understanding 1.26 (.51) 61 .24checking: new assignments 1.22 (.42) 32 –.02checking: error analysis 1.21 (.42) 39 .02own capacities 1.44 (.33) 24 –.07expectations for students 1.52 (.28) 42 –.05emphasis on basic skills 1.17 (.51) 51 .04

approach intensity 1.18 (.44) 41 –.04


strong guidance on all of the items, the different dimensions could bedirectly compared. Just as for consistency, a general approach variablewas also constructed by counting the number of times that the teachingapproach z-score was above .5, that is, the number of approach aspects onwhich the teacher demonstrated strong guidance.

The second column in Table 2 provides an overview of the intensity ofthe various aspects of the teaching approach used by the teachers. Themeans (multiplied by 100) indicate the percentage of teachers intensivelyusing the relevant approach. In order to examine the relation betweenconsistency of teaching approach within a school and the teaching ap-proach of the grade 7 teachers, the relevant correlation coefficients havebeen reported in the fourth column.

The averages show considerable variation in the intensity of the teach-ing approach. Assigning language and math homework is the most inten-sively employed approach, followed by examination of assignments forseriousness and understanding of the lesson. Instructional orientation andattention to the acquisition of reading strategies appear to have a very lowdegree of intensity, in contrast. On average, one can speak of an intensiveapproach or strongly guided learning on 41% of the aspects.

The correlations in the last column of Table 4 clearly show the degreeof consistency within a school to say nothing about the level of the teach-ing approach used in grade 7. For instructional orientation, frequency oftesting, language and math time, reading time, attention to reading strate-gies, and importance of checking student work to assign grades, the class-es with a high degree of consistency at school are overrepresented at a lowlevel of approach, which means consistently doing little or nothing. Con-versely, the classes with high consistency at school when it comes to theimportance of checking assignments for seriousness and understandingthe lesson are overrepresented at a relatively high level of approach.

Test PerformanceThe scores of the children on a language and a math test were taken ascognitive effect measures. Two multiple choice tests were administeredper grade by external test managers; the classroom teachers were alsogenerally present. An extensive justification for the choice of these instru-ments is provided in Driessen, van Langen, and Oudenhoven (1994). Thelanguage test included 64 problems and the math test included 40 prob-lems, with a reliability of .86 (KR20) and .87 (Cronbach’s α), respectively.The test scores represent the number of correct responses. For language,the average score was 78.7 with a standard deviation of 11.7; for math, theaverage score was 63.8 with a standard deviation of 18.3. When we break


the scores down according to socio-ethnic background, the following pic-ture emerges. On the language test, the mean scores for the non-disadvan-taged children, the Dutch disadvantaged children, and the ethnic-minoritydisadvantaged children were found to be 83.9, 78.2, and 68.6, respectively(Eta = .48); on the math test, the scores were 70.5, 61.9, and 53.5, respec-tively (Eta = .35).

RESULTS

In the following, we will consider the central question in our study, name-ly: How do the various student and teacher/class characteristics relate toeach other? The characteristics were measured at different levels. Thestudent background information and test results were collected for eachindividual child, with a minimum of two children per class. The informa-tion on teaching approach was measured at the class level, with only onescore per class. In addition, aspects of consistency were measured as con-textual variables at the class level and used as predictors.4 With studentsnested within classes, we applied multilevel analysis (cf. Bryk & Rauden-busch, 1992; Kreft & de Leeuw, 1998). Multilevel analysis allows us toavoid certain generalization problems, such as different class composi-tions influencing the variable to be explained differently. Multilevel anal-ysis also provides an estimate of the degree of association among thevariables while explicitly taking the fact that the variables have been meas-ured at different levels into account. For the analyses reported below, usewas made of the computer program MLn (Rasbash & Woodhouse, 1996).

The analyses can be explained in terms of model testing. In short thesemodels can be described as follows:

4. For the multilevel analyses, three levels could be distinguished in principle: studentlevel (background, test results), teacher/class level (class composition, didactic ap-proach), and school level (consistency, which is an aggregated variable). A three-levelmultilevel analysis nevertheless did not prove useful. A series of pilot analyses showedmost of the schools (80%) to have only one grade 8 class. Only 45 of the schools hadtwo grade 8 classes. This makes the calculation of a reliability estimate on the basis ofthe “class within school” variance impossible. For this reason, a two-level analysiswas decided on with student and teacher/class as the levels and consistency as a(contextual) predictor variable at the level of the class. Meijnen and Sontag (1997)encountered a similar problem in their research, which also included three levels inthe analyses initially. As the number of teachers per school was found to vary toolittle, no separate school level was included in the analyses. As in the present study,thus, Meijnen and Sontag conducted their analyses at the levels of the student and theteacher/class and not at the level of the school.


Model 0. In a multilevel analysis, the total variance in the test scores forthe students is split into two components, namely student level varianceand teacher level variance. In model 0, no predictors were included foreither of the two levels. This model shows just how much of the variancein the test scores is linked to each of the two levels. Student variance issubsequently predicted on the basis of student characteristics and teacher/class variance is predicted on the basis of teacher/class characteristics.

Model 1. In this model, the socio-ethnic background of the student isintroduced at the level of the student to predict the test results. The resultsshow the extent to which the test scores are corrected for student back-ground and how much of the variance at each level is predicted by studentbackground.

Model 2. In this model, the socio-ethnic composition of the class andthe interaction between the individual socio-ethnic background and theclass (socio-ethnic) composition are introduced at the teacher/class level.The independent effects of the different level socio-ethnic variables onstudent test scores can then be examined.

With the introduction of the socio-ethnic variables and their interaction,the student test scores are then fully corrected at both levels for any initialdisadvantage. Subsequently, this model is used as a basis for evaluatingthe extra effects of teaching approach and consistency.

Model 3. In this model, each aspect of the teaching approach in grade 7is introduced (total and separate). In such a manner, the independent effectof approach, which is the partial effect of approach while socio-ethnicbackground and class composition are held constant, can be estimated.5

Model 4. In this model, the product of the teaching approach in grade 7and consistency is added to get an estimate of the linear class-level inter-action effect. As interaction is always a partial effect (partial upon themain effects it consists of), both the main effects and the product term areincluded at the same time. Not doing so would overestimate the interactioneffect as the main effects would then be included in the interaction term.As already explained in the “Research Objectives” section, this interaction

5. Correlational analyses showed class composition to be virtually uncorrelated withteaching approach. The total effect of approach will thus be more or less equivalent toits independent effect. Only three weak correlations were found to occur: In groupswith many disadvantaged students, examination of their assignments for error analy-ses occurred less frequently (r = –.15), the teacher had somewhat lower expectationsfor the students (r = –.16), and somewhat more attention was paid to the acquisition ofbasic skills (r = .13). The remaining aspects of teaching approach correlate virtually 0with class composition. Apparently, little structure exists in the attunement of teach-ing approach to the degree of disadvantage characteristic of a class.


is of central interest to the present research and constitutes the basis foranswering our main research question.

In order to detect any differences in the two cognitive measures, the anal-yses were performed separately for language and math. The results forlanguage are presented in Table 3 and those for math are presented in

Table 3. Results Multilevel Analyses Language Test (Unstandardized Regression Coeffi-cients and (Standard Errors); Grand Mean: 78.5).

model

0 1 2 3 4

REGRESSION COEFFICIENTSStudent level:socio-ethnic background –6.3 (.2)** –5.2 (.2)**

Teacher level:socio-ethnic class composition –3.7 (.4)**

interaction –1.7 (.4)**

approach intensity .1 (1.3) 5.8 (2.9)*

reading time –.6 (.4) 1.3 (1.1)language and math time .0 (.6) –1.1 (1.4)homework .5 (.5) 1.6 (1.3)progress registration .3 (.6) .0 (1.7)instructional orientation –2.4 (.7)** –2.9 (1.6)attention reading strategy .3 (.5) 2.5 (1.2)*

test frequency .3 (.7) 1.0 (1.7)checking: grade .5 (.5) –.3 (1.2)checking: seriousness –.4 (.4) –.4 (.9)checking: understanding .1 (.4) .4 (.8)checking: new assignments –.5 (.4) –.9 (1.0)checking: error analysis .3 (.4) –.1 (1.0)own capacities –.8 (.7) .6 (2.0)expectations for students .4 (.6) 2.9 (2.1)emphasis on basic skills .5 (.4) .8 (.7)

VARIANCE COMPONENTSvariance explained student level 75% 8% 8%variance explained teacher level 25% 65% 75%

deviance 56298 55357 55237

* p < .05; ** p < .01


Table 4. The Grand Mean for each model was found to be 78.5 for lan-guage and 63.4 for math. The two tables show the unstandardized regres-sion coefficients and the standard errors in parentheses. Those effects witha p-value less than .01 are indicated with a **; those effects with a p-valueless than .05 (very weak effects) are indicated with a *.

Table 4. Results Multilevel Analyses Math Test (Unstandardized Regression Coefficientsand (Standard Errors); Grand Mean: 63.4).

model

0 1 2 3 4

REGRESSION COEFFICIENTSStudent level:socio-ethnic background –7.1 (.3)** –6.4 (.3)**

Teacher level:socio-ethnic class composition –4.6 (.8)**

interaction 1.2 (.7)

approach intensity 7.3 (2.6)** 5.0 (5.8)reading time –.7 (.9) .8 (2.2)language and math time 2.0 (1.2) –2.1 (2.9)homework 2.1 (1.0) .3 (2.6)progress registration 2.8 (1.2)* 4.7 (3.5)instructional intensity –5.7 (1.5)** –5.9 (3.2)attention reading strategy .4 (.9) 4.1 (2.5)test frequency 2.8 (1.4)* 1.6 (3.5)checking: grade 2.6 (.9)** 2.2 (2.4)checking: seriousness –1.1 (.8) 2.1 (1.8)checking: understanding 1.2 (.8) 1.1 (1.7)checking: new assignments –.3 (.9) –3.1 (2.0)checking: error analysis 1.2 (.8) –1.6 (2.0)own capacities –.7 (1.4) –3.7 (4.1)expactations for students 1.6 (1.3) 3.8 (4.3)emphasis on basic skills 2.1 (.7)** 2.5 (1.5)

VARIANCE COMPONENTSvariance explained student level 76% 5% 5%variance explained teacher level 24% 35% 40%

deviance 63002 62511 62480

* p < .05; ** p < .01


As can be seen from Table 3 (Model 0), 75% of the language testvariance is associated with the student level and 25% with the teacherlevel. The correlation between the test results and the socio-ethnic back-ground of the children (Model 1) is –6.3 with a standard error of .2, whichis therefore significant at the 1% level. In terms of content, this means thata difference of one category of socio-ethnic background is, on average,associated with a 6.3 point change in the language test scores. Ethnicminority children thus perform worse than Dutch working class childrenwho perform worse, in turn, than non-disadvantaged children. Socio-eth-nic background explains 8% of the student-level variance for language and65% of the teacher-level variance. The remaining part of the student-levelvariance stays the same in the subsequent models (2–4). The variance atteacher level is now further explained on the basis of teacher/class charac-teristics. The socio-ethnic composition of the class (Model 2) is found tohave a significant independent effect on the language test results (namely–3.7 with a standard error of .4; p < .01). This means that classes with arelatively large number of disadvantaged children produce particularlylow language test scores even when the individual socio-ethnic back-grounds of the students have been accounted for. One can also speak of asignificant cross-level interaction between individual student backgroundand class composition. This means that the individual backgrounds of thestudents in classes with many disadvantaged students play a larger role asa (negative) predictor of language achievement than in classes with fewerdisadvantaged students. Together with the interaction, class compositionexplains 75% of the teacher-level variance.

With regard to teaching approach (Model 3), the effects of both theparticular aspects of the teaching approach and the overall teaching ap-proach were examined. The only aspect to show a significant correlationwith the language test scores (at the 1% level) was instructional orienta-tion (class versus individual). This means that in classes where the teach-ers make intensive use of an individual didactic approach, the mean scoreon language is 2.4 points lower than in classes where this is not the case.Individual instruction appears to be paired with a on average lower levelof language achievement. A causal interpretation of this effect is not im-mediately apparent, however. The results under Model 4 show no interac-tion effects whatsoever between teaching approach and consistency on thelanguage test results. “Attention to reading strategies” and “intensity ofthe teaching approach” may marginally interact but only at a 5% level ofsignificance.

A comparison of the results for language (Table 3) with those for math(Table 4) shows the differences between the results for the two tests to be


relatively small. The language and math scores largely consist of system-atic variation at the level of the class, namely 25% and 24% respectively.Individual socio-ethnic background explained more of the systematic classvariation for language than for math, namely 65% versus 35%. Class com-position similarly explained more of the remaining class variance for lan-guage than for math, namely 10% (75%–65%) versus 5% (40%–35%).Considered as a whole, we can state that the variation in the languagescores is more strongly and systematically related to background factorsindicating some disadvantage than the variation in the math scores.

With regard to the effects of teaching approach on math achievement,the strongest effect is found similar to language for “instructional orienta-tion.” The amount of variance in the math scores explained by instruction-al orientation is 3.2%, which means a standardized effect of –.18. Theother effects found to have a p-value less than .01 were “importance ofchecking student work to assign a grade” and “special emphasis on theacquisition of basic skills,” which were also found to have a standardizedeffect of about .15. These findings do not meet the criterion of relevance,however, which also holds for the total number of approach aspects (i.e.,approach intensity). In closing, no significant interactions between teach-ing approach and consistency of teaching approach were found for math.

CONCLUSIONS

The first conclusion is that class composition (i.e., socio-ethnic composi-tion of a particular student’s class) is basically uncorrelated with the di-dactic approach of the teacher. Only three aspects of teaching approachbore a (weak) relation to class composition. In groups with a relativelyhigh number of (Dutch and minority) disadvantaged children: the studentwork is checked somewhat less for purposes of error analyses, the teacherexpects somewhat less of the students, and slightly more attention is paidto the acquisition of basic skills. These relations are largely in keepingwith what we expected. It is striking that the remaining, more specific,didactic aspects of teaching approach were found to be virtually uncorre-lated with class composition. There is apparently little structure in theattunement of a specific didactic approach to the degree of disadvantagethat characterizes a class.

For both language and math, significant negative effects of individualsocio-ethnic background and class composition were observed. This meansthat ethnic minority children achieve less than Dutch working class chil-dren who achieve, in turn, less than non-disadvantaged children. In addi-


tion to this negative effect of the individual socio-ethnic background ofthe child, however, the composition of the relevant student’s class wasalso found to have a clearly negative effect on test results. In classes withnumerous disadvantaged children, the level of achievement was found tobe particularly low. With regard to language, a significant cross-levelinteraction between the individual socio-ethnic background of the childand the class composition was also observed. This means that the individ-ual disadvantage in classes with a large number of disadvantaged childrenplays a larger role as a (negative) predictor of language achievement thanin classes with fewer disadvantaged children.

With regard to specific aspects of the teaching approach and their inter-action with the consistency of the teaching approach, the following can beconcluded. With regard to language achievement, one significant effect ofapproach was observed, namely “instructional orientation.” The provisionof more individualized instruction is, on average, associated with lowertest scores. For math, “instructional orientation” was again found to pro-duce the most significant effect. The other significant effects observed formath achievement were: “importance of checking student work to assign agrade” and “special emphasis on the acquisition of basic skills.” Thesetwo aspects of teaching approach are associated with on average highermath scores. In this context, it should be noted that these effects shouldprobably not as yet be interpreted as causal. With regard to the main topicof this study, teaching approach and consistency of teaching approachwere not found to significantly interact at a level which is of interest forour research. This means that our hypotheses concerning the effects ofconsistency are not confirmed: consistency of teaching approach in noway influences the language and math performance of students at the endof their elementary school careers.

How can we explain this result? One possible explanation concerns themethodology employed. In the present study, we analyzed data collectedfrom teachers via written questionnaires. Although numerous questionswere explicitly posed with regard to teaching method, all of the relevantinformation may not have been gained using this method. In other re-search, moreover, differences have been found between what teachers saythey do and what they actually do (cf. Charlesworth et al., 1993). Toincrease the validity of such information in the future, it is therefore rec-ommended that questionnaire data be supplemented by observations andin-depth interviews (cf. Schaffer, Nesselrodt, & Stringfield, 1994; Weide,1995). In future research, an accurate indication of the amount of time theteacher actually devotes to the various parts of the curriculum and toindividual students should also be obtained. In addition, the complex rela-


tions between the educational objectives of the school, the motivation ofthe teacher to realize these objectives, and the actual details of concreteeducational practice should be submitted to more critical consideration inthe future. In order to help achieving this, the use of multiple methodolo-gies (i.e., triangulation) is recommended to guarantee the reliability andvalidity of the data collected and analyzed.

Another potential explanation for the absence of an effect of consisten-cy pertains to the nature of actual school practice. A critical feature ofdaily school life is the fact that teaching generally occurs in an isolatedsetting. Teaching is and continues to be a lonely profession. Teachers havea large degree of autonomy and decision space, particularly when it comesto giving form to the teaching process within the classroom. Such profes-sional autonomy may be largely at odds with the idea of consistency andthus explain our failure to find an effect of consistency. This does notmean that consistency may not be a useful concept, but that it will be hardto demonstrate it in educational practice.

ACKNOWLEDGEMENTS

The authors would like to thank J. Doesborgh and J. Hutjes for their methodologicaladvice and help with the multilevel analyses. The Netherlands Organization for ScientificResearch (NWO) is gratefully acknowledged for funding the project on which this articleis based. This research was supported by grant #575-35-002 from the Foundation forBehavioral and Educational Sciences of this organization to Dr. G. Driessen of ITS (Insti-tute for Applied Social Sciences), University of Nijmegen, the Netherlands.

REFERENCES

Bosker, R. (1991). De consistentie van schooleffecten in het basisonderwijs [The consist-ency of school effects in elementary education]. Tijdschrift voorOnderwijsresearch, 16, 206–218.

Bryk, A., & Raudenbusch, S. (1992). Hierarchical linear models: Applications and dataanalysis methods. Newbury Park: Sage.

Charlesworth, R., Hart, C., Burts, D., Thomasson, R., Mosleu, J., & Fleege, P. (1993).Measuring the developmental appropriateness of kindergarten teachers’ belief andpractices. Early Childhood Research Quarterly, 8, 255–276.

Chitty, C. (1997). The school effectiveness movement: Origins, shortcomings and futurepossibilities. The Curriculum Journal, 8, 45–62.

Creemers, B. (1991). Effectieve instructie. Een empirische bijdrage aan de verbetering vanhet onderwijs in de klas [Effective instruction. An empirical contribution to theimprovement of the instruction in the class]. Den Haag: SVO.

Creemers, B. (1994a). Effective instruction: An empirical basis for a theory of educationaleffectiveness. In D. Reynolds, B. Creemers, P. Nesselrodt, E. Schaffer, S. String-


field, & C. Teddie (Eds.), Advances in school effectiveness research and practice(pp. 189–205). Oxford: Pergamon.

Creemers, B. (1994b). Effective instruction as a basis for effective education in schools.Tijdschrift voor Onderwijsresearch, 19, 3–16.

Creemers, B. (1994c). The effective classroom. London: Cassell.Creemers, B. (1994d). The history, value and purpose of school effectiveness studies. In D.

Reynolds, B. Creemers, P. Nesselrodt, E. Schaffer, S. Stringfield, & C. Teddie(Eds.), Advances in school effectiveness research and practice (pp. 9–23). Oxford:Pergamon.

Creemers, B. (1997). Onderwijseffectiviteit: Factoren, condities en misschien ook visies[Educational effectiveness: Factors, conditions and maybe visions too]. In Th.Bergen, F. Knoers, & P. Sleegers (Eds.), Perspectieven op de school in dynamischeontwikkeling [Perspectives on the school in dynamic development] (pp. 181–212).Alphen aan den Rijn : Samsom/H.D. Tjeenk Willink.

Creemers, B., & Reezigt, G. (1996). School level conditions affecting the effectiveness ofinstruction. School Effectiveness and School Improvement, 7, 197–228.

Driessen, G., & Dekkers, H. (1997). Educational opportunities in the Netherlands. Policy,students’ performance, issues. International Review of Education, 43, 299–315.

Driessen, G., & Doesborgh, J. (1998). Consistentie in didactische werkwijze binnen deschool en prestaties van leerlingen in het basisonderwijs [Consistency in teachingapproach within the school and achievement levels of elementary school stu-dents]. Nijmegen: ITS.

Driessen, G., & Jungbluth, P. (Eds.). (1994). Educational opportunities. Tackling ethnic,class and gender inequality through research. Münster/New York: Waxmann.

Driessen, G., van Langen, A., & Oudenhoven, D. (1994). De toetsen voor het cohortPrimair Onderwijs. Verantwoording [The tests for the Primary Education cohort].Nijmegen: ITS.

Driessen, G., & Mulder, L. (1999). The enhancement of educational opportunities of disad-vantaged children. In R. Bosker, B. Creemers, & S. Stringfield (Eds.), Enhancingeducational excellence, equity and efficiency. Evidence from evaluations of sys-tems and schools in change (pp. 37–64). Dordrecht/Boston/London: Kluwer Aca-demic Publishers.

Hargreaves, D. (1995). School culture, school effectiveness and school improvement.School Effectiveness and School Improvement, 6, 23–46.

Jungbluth, P., Langen, A. van, Peetsma, T., & Vierke (1996). Leerlinggegevens basis-onderwijs en speciaal onderwijs. Technische rapportage PRIMA-cohortonderzoek1994/95 [Pupil data primary and special education. Technical report PRIMA co-hort study 1994/95]. Amsterdam/Nijmegen: SCO-Kohnstamm Instituut/ITS.

Kreft, I., & De Leeuw, J. (1998). Introducing multilevel modeling. London: Sage.Ledoux, G., & Overmaat, M. (1996). School-en klaskenmerken basisonderwijs. Technische

rapportage PRIMA-cohortonderzoek 1994/95 [School and class characteristicselementary education. Technical report PRIMA cohort study 1994/95]. Amster-dam: SCO-Kohnstamm Instituut.

Meijnen, G., & Sontag, L. (1997). Effective education for young children in primaryschools. Educational Studies, 23, 87–104.

Mortimore, P., Sammons, P., Stoll, L., Lewis, D., & Ecob, R. (1988). School matters. Thejunior years. Wells: Open Books.

Overmaat, M., & Ledoux, G. (1995). De constructie van de instrumenten school- enklaskenmerken voor het cohort-onderzoek primair onderwijs. Verslag van tweevoorstudies [The construction of the school and class characteristics instrumentsfor the cohort study elementary education]. Amsterdam: SCO.


Rasbash, J., & Woodhouse, G. (1996). MLn command reference. Version 1.0a. London:University of London.

Reynolds, D., Sammons, P., Stoll, L., Barber, M., & Hillman, J. (1996). School effective-ness and school improvement in the United Kingdom. School Effectiveness andSchool Improvement, 7, 133–158.

Rossi, R., & Montgomery, A. (1994). Educational reforms and students at risk. A review ofthe current state of the art. Washington DC: US Department of Education.

Sammons, P., Hillman, J., & Mortimore, P. (1995). Key characteristics of effective schools.A review of school effectiveness research. London: OFSTED.

Schaffer, C., Nesselrodt, P., & Stringfield, S. (1994). The contribution of classroom obser-vation to school effectiveness research. In D. Reynolds, B. Creemers, P.Nesselrodt, E. Schaffer, S. Stringfield, & C. Teddie (Eds.), Advances in schooleffectiveness research and practice (pp. 133–150). Oxford: Pergamon.

Scheerens, J. (1992). Effective schooling. Research, theory and practice. London/NewYork: Cassell.

Scheerens, J., & Bosker, R. (1997). The foundations of educational effectiveness. Oxford:Pergamon.

Tesser, P., Praag, C. van, Dugteren, F. van, Herweijer, L., & Wouden, H. van der (1995).Rapportage minderheden 1995. Concentratie en Segregatie [Report on ethnicminorities 1995. Concentration and segregation]. Rijswijk: SCP.

Weide, M. (1995). Effectief onderwijs voor allochtone kinderen [Effective instruction forethnic minority children]. Groningen: RION.

Werf, M. van der (1995). The Educational Priority Policy in the Netherlands. Content,implementation and outcomes. Den Haag: SVO.