AGREEMENT AMONG CLASSROOM OBSERVERS OF CHILDREN’S STYLISTIC LEARNING BEHAVIORS

helen hamlet buchanan and paul a. mcdermott

University of Pennsylvania

barbara a. schaefer

Pennsylvania State University

Standardized and reliable rating scales have an important role in educational assessment and be-havioral classroom intervention. The Learning Behaviors Scale (LBS) is a standardized behaviorrating scale designed to report how individual students respond to classroom learning situations.This study investigated the interobserver agreement of the LBS with the use of linear and intraclasscorrelation methods. The methods jointly assessed the three salient aspects of observer judg-ments—severity level, rank order, and directionality. Participants were 72 students enrolled inspecial education programs as observed by 16 educators in eight self-contained classrooms. Bothlinear and intraclass coefficients were substantial (averages 5 .83 and .84, respectively). No sig-nificant observer effect was found. Moreover, the LBS produced comparable levels of differentiallearning styles for assessments of individual children. © 1998 John Wiley & Sons, Inc.

Standardized rating scales frequently provide relatively unobtrusive assessments of classroombehavior (Merrell, 1994). The prime element in such assessments is the competence of the observeror rater (Edelbrock, 1983; Groth–Marnet, 1990; Mulliken & Buckley, 1983). The observer must bea natural and constant participant in the classroom environs and must have the opportunity to dis-tinguish children’s behavioral constancy and variability. In the classroom, adult educators play thecentral roles in delivery of instruction, group management, and performance evaluation. They are,because of their experience with many different children over time and in many diverse social con-texts (learning, discipline, play, etc.), the natural informants for classroom phenomena (Baumgart-ner & Jackson, 1991; McDermott, 1986; Watkins & Canivez, 1997). And to the extent that ratingscales are summative and may be completed without the direct cognizance of the children observed,the resultant assessments are optimally unobtrusive.

Among relevant classroom phenomena, basic learning-related behaviors and dispositionsemerge as most fundamental (Power & Ikeda, 1996; Reid & Maag, 1994). The relationship betweenlearning behaviors (sometimes called learning or cognitive styles) and a child’s academic success iswidely recognized (Alexander, Entwisle, & Dauber, 1993; Hinshaw, 1992; Sattler, 1992; Stott,1985). Indeed, the identification and shaping of those behaviors is often regarded as the key elementin bolstering successful school attainments and averting faulty learning habits and poor attitudes.Thus, information about a student’s typical competence motivation, learning strategies, flexibility,persistence, and response to error, correction, and novelty, afford important clues to the formation ofgood learning styles and consequent achievement. Moreover, there is ample evidence to support thebelief that basic learning behaviors (unlike the ubiquitous and relatively unchangeable elements thatcomprise general intellectual ability; see Brody, 1985, and Snow, 1986) are themselves relativelyalterable or teachable, and thus provide clear direction for fruitful remediation and enrichment pro-grams.

One of the newest, cost beneficial, and potentially effective means for assessing basic learningbehaviors is the Learning Behaviors Scale (LBS; McDermott, Green, Francis, & Stott, 1997). TheLBS is a 29-item summative rating device that is completed by a knowledgeable teacher or class-room aide. It is standardized nationally on a large sample that is representative of students 5–17 years

Psychology in the Schools, Vol. 35(4), 1998© 1998 John Wiley & Sons, Inc. CCC 0033-3085/98/040355-07

355

Requests for reprints should be sent to Paul A. McDermott, Graduate School of Education, University of Pennsylvania,3700 Walnut Street, Philadelphia, PA 19104.

of age, including their geographic regions, community sizes, family structures and education levels,and disabling conditions. Each teacher assessment renders specific information on a student’s leveland pattern of competence motivation, attitudes toward learning, attention and persistence, problem-solving strategies, and flexibility.

A substantial body of empirical evidence supports the construct validity and generalizability ofLBS information across time and across pertinent population subgroups (including ages, gender, andethnicity). It is further clear that the information provided for prediction of academic attainment is aunique and significant complement to predictions offered by intelligence measures and that the pre-dictions from LBS information and its interactions with IQ often exceed those availed by IQ infor-mation alone (Schaefer, 1996; Weiss, 1997).

Notwithstanding broad potential utility, the practical usefulness of LBS-type measures is de-pendent upon the accuracy of assessments at the level of the individual child. That is, it does not re-ally suffice that LBS assessments remain internally consistent for groups or that they remain rela-tively stable over reasonable periods of time (as the evidence demonstrates). Rather, for child-levelassessments to be useful for planning and intervention, it must be shown that independent observersof the same phenomena in the same context substantially concur on the assignment of distinguish-ing learning styles. This is the matter of interobserver agreement (Bartko, 1991; Fleenor, Fleenor, &Grossnickle, 1996)—a quality that is often quite difficult to establish, either because competent ob-servers do not ordinarily co-occupy the same classroom or because such agreement does not exist.Needless to say, assessments that are idiosyncratic to the observer, and not to the student, are fairlylimited or even useless or misleading for intervention purposes. One must ensure that the pertinenttarget phenomena (learning behavior), and not observer idiosyncrasies or presentiments, are whatground child assessments.

Within this framework, the present study was undertaken to examine the interobserver agree-ment for LBS information. Both interclass (linear) and intraclass correlation strategies were used toquantify the concordance of independent LBS evaluations. Special attention was given to the rela-tive weight of observer versus individual child influences on assessments (what is called the rater ef-fect or rater source error; see Deitz, 1989, and James, Demaree, & Wolf, 1984).

Method

Participants

The sample consisted of 72 children drawn from students attending special education schoolsin the suburban area of Philadelphia, PA. Ages ranged from 7 to 16 years with a mean age of 11.3years. The children were enrolled in grades one through seven and were diagnosed with learning dis-abilities, conduct disorders, attention deficit disorders, or physical disabilities, in accordance withlocal implementation of federal guidelines (Individuals with Disabilities Education Act, 1990) viainterdisciplinary panels. Males comprised 58% and females 42% of the sample, with 59% beingWhite and 41% African-American.

Instrumentation

The 29-item LBS (McDermott et al., 1997) was normed nationally. The sample (N 5 1,500)was designed to represent the population of all noninstitutionalized 5- through 17-year-old youth at-tending school in the United States at the opening of the 1990 decade. Sampling specifications con-formed to the 1988–90 U.S. Census (U.S. Department of Commerce, 1992) with matrix blocking forsex (750 males and 750 females) and approximately equal distributions of subjects and gender ateach one-year age and grade interval. Stratified random sampling was used to configure the sampleto Census proportions, both marginally and within cells, across the four regions of the country and

356 Buchanan, McDermott, and Schaefer

for major and minor metropolitan and rural residences, race/ethnicity (White, African-American,Hispanic origin, and other), family structure (mother and father, mother only, father only), and yearsof formal schooling completed by mother and father (or guardian).

Stratification provided approximate matching to national distributions of special education sub-jects (as per the U.S. Department of Education, 1989) classified by multidisciplinary child studyteams, including 6.9% handicapped (with proper subcategorical distributions of the learning dis-abled, emotionally disturbed, speech impaired, etc.), and 4.0% gifted/talented. Intellectual ability andacademic achievement were assessed for 1,350 of the 1,500 through the Differential Ability Scales(Elliot, 1990), a battery individually administered in all cases by psychologists or supervised psy-chology graduate students. In each performance area, including General Conceptual, Verbal, andNonverbal Ability, and reading and mathematics achievement, the derived quotient Ms and SDs forthe normative sample were 100 and 15, respectively.

Each of the 29 items presents a specific learning-related behavior (e.g., “Is willing to be helpedwhen a task proves too difficult,” “Has enterprising ideas which often don’t work out,” “Follows pe-culiar and inflexible procedures in tackling tasks,” and “Carries out tasks according to own ideasrather than in the accepted way”) and the observer is required to indicate whether the behavior Mostoften applies, Sometimes applies, or Doesn’t apply to describe the child’s typical classroom behav-ior over the past 2 months. The valence (positive or negative) of item wording is varied as a mea-sure to reduce response sets. Assessments take 3–5 minutes per student and are not permitted untilthe respondent has had at least 50 days to observe the student.

Based on multiple exploratory and confirmatory factor analyses, observer ratings for the 29items are summed by unit weighting (Wainer, 1976) to form four learning style dimensions—Com-petence Motivation, Attitude Toward Learning, Attention/Persistence, and Strategy/Flexibility. AnLBS total score (as based on second-order factor analyses) further represents the sum of the uniquebehaviors constituting the four learning style dimensions. Dimension and total scores are trans-formed by area conversion (Thorndike, 1982) to normalized T scores (M 5 50, SD 5 10) that areassociated with comparable percentile distributions across dimensions.

Development history and validity evidence for the LBS are extensive. Essentially, numerouspublished research editions provided the groundwork for the current version (Glutting & McDer-mott, 1988; McDermott, 1984; McDermott & Beitman, 1984; McDermott & Watkins, 1987; Stott,1985; Stott, McDermott, Green, & Francis, 1988), demonstrating construct continuity, temporal sta-bility, and incremental predictive validity. The dimensions for the present version also have beenfound invariant and comparably reliable across age, gender, and ethnic subpopulations (McDermottet al., 1997). As noted previously, LBS forecasting ability significantly increments and/or outpacesconventional cognitive ability measures (Schaefer, 1996; Weiss, 1997). Moreover, the predictionsare not biased against gender or particular ethnic groups (Schaefer, 1996).

Procedure

Classrooms where two adults simultaneously worked with students were identified and eightclassrooms were selected for participation as based on teacher volunteerism and a desire to diversi-fy the observer pool as much as possible. Classroom behaviors were simultaneously but indepen-dently observed by one professional and one paraprofessional teacher. Each observer pair (16separate observers) consisted of a special education teacher and a special education aide in a self-contained classroom. Financial compensation was offered to participant observers for the classroomfund.

All observers were White and female. All teachers possessed a Bachelors degree and nearly allhad completed 30 graduate credit hours, with all having taught for more than 10 years. All of theteacher aides had graduated high school and completed at least 4 years of classroom teaching.

Agreement 357

Completion of the LBS occurred after a minimum of a 2-month opportunity to observe eachchild in one of the classrooms. Each observer completed ratings on an average of 7.5 children, withno fewer than 7 nor more than 10 children assigned to any observer.

Data Analyses

Interobserver agreement was determined through application of interclass and intraclass strate-gies. Interclass evaluations proceeded through the two-step test of related rankings and directional-ity (i.e., positive and statistically significant Pearson rs for linear relationship) and failure to rejectthe null hypothesis of equality of means across observers (McDermott, 1988; Watkins & Canivez,1997). For the intraclass approach, the two-way random effects model of the intraclass correlation(ICC) is indicated (Shrout & Fleiss, 1979). This ICC is appropriate when the focus is the absoluteagreement among measurements and it informs the integrity of the individual student’s performanceas it relates to an independent criterion level and the effectiveness of the scale regardless of the ob-server (Cho, 1981; McGraw & Wong, 1996). The Winer (1962) ICC formula was used as confirma-tion of results (as recommended by Hamer, 1993).

Results

Table 1 displays means and standard deviations for LBS assessments by the respective ob-servers. None of the discrepancies across observers is significant statistically, indicating that, as agroup, the children are viewed with comparable levels of performance on each learning dimension.Interestingly, it will be noted that the reported mean T scores fall nearly one SD below the popula-tion average (50), suggesting that the selected special education children were generally not deemedto possess effective learning styles. This discovery comports with the expectation that disabled stu-dents may manifest faulty learning behaviors and additionally speaks to the relevance of the select-ed sample, inasmuch as marked variability was found for faulty learning behaviors.

Table 2 presents inter- and intraclass correlations. All values are positive, substantial,1 and sta-tistically significant at p , .001. Values are almost identical across correlation methods, demon-strating that LBS observations are essentially comparable over independent observers in terms oflevel (severity), rank ordering, and pattern. As evaluated via the Shrout–Fleiss (1979) ICC method,no significant effect was found for observer. Thus, child rather than observer qualities underpinnedthe assessments.

358 Buchanan, McDermott, and Schaefer

1Widaman (1993) reported that reliabilities in excess of .60 are commonly regarded as adequate for most scaling appli-cations. This level is considered substantial because it ensures that most variance is not error variance.

Table 1T-Score Means and Standard Deviations of Learning Behaviors ScaleDimensions and Total Score for Independent Observers

Teacher Teacher Aide

LBS Dimension M SD M SD

Competence motivation 41.9 6.7 42.3 7.3Attitude toward learning 41.5 11.8 41.1 9.7Attention/persistence 40.0 9.3 41.3 8.6Strategy/flexibility 41.6 8.0 41.0 9.9LBS Total 40.2 7.8 40.2 8.5

Note. N 5 72. LBS 5 Learning Behaviors Scale.

Discussion and Conclusion

The interobserver agreement for independent assessments of stylistic classroom learning be-havior was established. Whereas LBS evaluations were congruent over observers for each child, idio-syncratic observer influences were not detected. Consequently, the use of the LBS in a special edu-cation environment, with a teaching professional or paraprofessional as observer, should providereliable information regarding the behavioral learning style of a given student. The informationshould be reproducible regardless of the observer, as long as the context remains relatively constant.

Two decades ago, it was formally recognized by the U.S. Office of Child Development (An-derson & Messick, 1974) that certain foundation behaviors undergirded educational success. Thesebehaviors were viewed as developmentally acquired, learned, and relatively responsive to modifi-cation programs—unlike the phenomena associated with cognitive ability. The experts identified be-havioral skill areas referred to as competence motivation, control of attention, problem solving strat-egy, persistence, curiosity, and flexibility. These comport closely with LBS dimensions andcomponent behaviors. More recently, Barnett and associates (Barnett, Bauer, Ehrhardt, Lentz, &Stollar, 1996) have argued that such behaviors be regarded as keystone instructional targets becausethey form a foundation for future successes and their modification promotes change across manysimilar learning behaviors and transfer to developmental domains outside of schooling.

A variety of successful research programs and available instructional materials have shownmuch promise for the application of enrichment and remediation procedures based on informationfrom LBS assessments (McDermott & Watkins, 1987; Stott, 1978, 1981; Stott et al., 1988). De-pending on the procedure, LBS profiles provide the footing for either large-scale screening initia-tives or individual assessment, followed by systematic production of individualized instructional se-quences (based on hierarchies of learning-related behaviors presented with incremental complexity).Nevertheless, the value of these initiatives is fully dependent upon the verity of the initial classroomassessments. The current research would appear to confirm that such assessments can be accurateand reproducible from the perspectives of independent observers.

References

Alexander, K. L., Entwisle, D. R., & Dauber, S. L. (1993). First-grade classroom behavior: Its short- and long-termconsequences for school performance. Child Development, 64, 801–814.

Anderson, S., & Messick, S. (1974). Social competency in young children. Developmental Psychology, 10, 282–293.

Agreement 359

Table 2Interobserver Agreement Coefficients for Learning Behaviors Scale Dimensionsand Total Score

Agreement Coefficienta

Learning Behaviors Scale Dimension Interclass Intraclass

Competence motivation .88 .88Attitude toward learning .87 .85Attention/persistence .68 .68Strategy/flexibility .88 .85LBS Total .91 .91

Note. N 5 72.aInterclass coefficients are Pearson product–moment rs and intraclass coefficients are

Shrout–Fleiss (1979) indices. All values are statistically significant at p , .001.

Barnett, D. W., Bauer, A. M., Ehrhardt, K. E., Lentz, F. E., & Stollar, S. A. (1996). Keystone targets for changes:Planning for widespread positive consequences. School Psychology Quarterly, 11, 95–117.

Baumgartner, T. A., & Jackson, A. S. (1991). Measurement for evaluation. Dubuque, IA: Brown.Bartko, J. J. (1991). Measurement and reliability: Statistical thinking considerations. Schizophrenia Bulletin, 3, 483–489.Brody, N. (1985). The validity of tests of intelligence. In B. Wolman (Ed.), Handbook of intelligence (pp. 353–389). New

York: Wiley.Cho, D. W. (1981). Inter-rater reliability: Intraclass correlation coefficients. Educational and Psychological Measurement,

41, 223–226.Deitz, J. C. (1989). Developing norm-referenced standardized tests. Seattle, WA: Haworth Press.Edelbrock, C. (1983). Problems and issues in using rating scales to assess child personality and psychopathology. School

Psychology Review, 12, 293–299.Elliot, C. D. (1990). Differential Ability Scales: Introductory and technical handbook. San Antonio: Psychological Cor-

poration.Fleenor, J. W., Fleenor, J. B., & Grossnickle, W. F. (1996). Interrater reliability and agreement of performance ratings:

A methodological comparison. Journal of Business and Psychology, 10, 367–380.Glutting, J. J., & McDermott, P. A. (1988). Generality of test-session observations to kindergartners’ classroom behav-

ior. Journal of Abnormal Child Psychology, 16, 527–537.Groth–Marnet, G. (1990). Handbook of psychological assessment. New York: Wiley.Hamer, R. M. (1993). Intraclass correlations [Computer code]. Cary, NC: SAS Institute.Hinshaw, S. P. (1992). Externalizing behavior problems and academic underachievement in childhood and adolescence:

Causal relationships and underlying mechanisms. Psychological Bulletin, 111, 127–155.Individuals with Disabilities Education Act, 20 U.S.C. 1400 et seq. (1990).James, L., Demaree, R., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias.

Journal of Applied Psychology, 69, 322–327.McDermott, P. A. (1984). Comparative functions of preschool learning style and IQ in predicting future academic per-

formance. Contemporary Educational Psychology, 9, 38–47.McDermott, P. A. (1986). The observation and classification of exceptional child behavior. In R. T. Brown & C. R.

Reynolds (Eds.), Psychological perspectives on childhood exceptionality: A handbook (pp. 136–180). New York:Wiley–Interscience.

McDermott, P. A. (1988). Agreement among diagnosticians or observers: Its importance and determination. ProfessionalSchool Psychology, 3, 225–240.

McDermott, P. A., & Beitman, B. S. (1984). Standardization of a scale for the Study of Children’s Learning Styles: Struc-ture, stability, and criterion validity. Psychology in the Schools, 21, 5–14.

McDermott, P. A., Green, L. F., Francis, J. M., & Stott, D. H. (1997). Learning Behaviors Scale. Philadelphia: Edu-metric and Clinical Science.

McDermott, P. A., & Watkins, M. W. (1987). Microcomputer systems manual for McDermott Multidimensional Assess-ment of Children (IBM version). San Antonio: Psychological Corporation.

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. PsychologicalMethods, 1, 30–46.

Merrell, K. W. (1994). Assessment of behavioral, social & emotional problems. New York: Longman.Mulliken, R. K., & Buckley, J. J. (1983). Assessment of multihandicapped and developmentally disabled children.

Rockville, MD: Aspen Publication.Power, T. J., & Ikeda, M. J. (1996). The clinical utility of behavior rating scales: Comments on the diagnostic assessment

of ADHD. Journal of School Psychology, 34, 379–385.Reid, R., & Maag, J. W. (1994). How many fidgets in a pretty much: A critique of behavior rating scales for identifying

students with ADHD. Journal of School Psychology, 32, 339–354.Sattler, J. M. (1992). Assessment of Children (3rd ed.). San Diego: Author.Schaefer, B. A. (1996). Relationship between learning behaviors and academic achievement among school-aged Ameri-

can children and adolescents (Doctoral dissertation, University of Pennsylvania, 1996). Dissertation Abstracts Interna-tional, 57(04), 1490.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86,420–428.

Snow, R. E. (1986). Individual differences and the design of educational programs. American Psychologist, 41, 1029–1039.Stott, D. H. (1978). The hard-to-teach child: A diagnostic-remedial approach. Baltimore: University Park Press.Stott, D. H. (1981). The Flying Start Learning-to-Learn Kits. Chicago: Science Research Associates.Stott, D. H. (1985). Learning style or “intelligence”? School Psychology International, 6, 167–174.

360 Buchanan, McDermott, and Schaefer

Stott, D. H., McDermott, P. A., Green, L. F., & Francis, J. M. (1988). Learning Behaviors Scale and Study of Chil-dren’s Learning Behaviors research edition manual. San Antonio: Psychological Corporation.

Thorndike, R. L. (1982). Applied psychometrics. Boston: Houghton Mifflin.U.S. Department of Commerce. (1992). Current population survey, March 1992 [Machine-readable computer file].

Washington, DC: Bureau of the Census.U.S. Department of Education. (1989). Eleventh annual report to Congress on the implementation of the Education of

the Handicapped Act. Washington, DC: Author.Wainer, H. (1976). Estimating coefficients in linear models: It don’t make no nevermind. Psychological Bulletin, 83,

213–217.Watkins, M. W., & Canivez, G. L. (1997). Interrater agreement of the Adjustment Scales for Children and Adolescents

among exceptional students. Diagnostique, 22, 205–213.Weiss, R. V. (1997). Cognitive ability, classroom learning behavior, and achievement responsibility as predictors of con-

current academic performance (Doctoral dissertation, University of Pennsylvania, 1997). Dissertation Abstracts Inter-national, 58, in press.

Widaman, K. F. (1993). Common factor analysis versus principal component analysis: Differential bias in representingmodel parameters? Multivariate Behavioral Research, 28, 263–311.

Winer, R. J. (1962). Statistical principles in experimental design. New York: McGraw–Hill.

Agreement 361