The Judge Factor: The Effect of an Adjudicator’s Teaching

The Judge Factor: The Effect of an Adjudicator’s Teaching Level

on Assigned UIL Ratings

By

Cody Newman, B.M. A Thesis

In Music Education

Submitted to the Graduate Faculty of Texas Tech University in

Partial Fulfillment of the Requirements for

the Degree of

Master of Music Education

Approved

Dr. Janice Killian Chair of Advisory Committee

Dr. Keith Dye

Dr. Jacqueline C. Henninger

Dr. Mark A. Sheridan Vice Provost for Graduate and Postdoctoral Affairs

Dean of the Graduate School

December, 2014

© 2014, Cody Newman

Texas Tech University, Cody Newman, December, 2014

ii

ACKNOWLEDGMENTS

I would like to say thank you to the many people who have guided me on my

journey through my years at Texas Tech and into the world of music education. The

constant support and inspiration I received from the Texas Tech faculty, my beautiful

wife Candice, and my peers in music education have lifted me to this point. I would like

to specifically thank Dr. Killian for the years of training and teaching she invested in me

as well as her boundless patience through this journey.


iii

TABLE OF CONTENTS

ACKNOWLEDGMENTS ii

ABSTRACT v

LIST OF TABLES vi

LIST OF FIGURES vii

CHAPTER

I. INTRODUCTION 1

Vignette 1

Justification 2

II. REVIEW OF RELATED LITERATURE 4

Director Controlled Influences 4

Non-Director Controlled Influences 5

Judge Reliability 6

Purpose 7

III. METHODOLOGY 8

Terminology 8

Participants 11

Procedures and Materials 14

IV. RESULTS 15

Comprehensive Results by Director 15

Comprehensive Results by Grade Level Experience 22

Comprehensive Results Final Comparison 24


iv

VI. DISCUSSION 26

Implications for Further Research 27

Implication for Educators 29

REFERENCES 30

APPENDIX 33


v

ABSTRACT

Previous research established the reliability of adjudicators and their rankings

and ratings when judging in their field of expertise. This study examined the extent to

which the grade level of judges’ prior teaching experience affects the ratings they assign

during band adjudication (N = 11, 7 high school judges, 4 middle school judges).

Adjudication data were collected from a 14 year period, 2001 – 2014 and included varsity

band ratings from the UIL Concert and Sightreading Contest in the state of Texas.

At each contest studied, judging panels consisted of three Texas Music

Adjudicator Association approved judges. Each of the three judges individually assigned

a rating, and the three ratings were combined for a composite rating. Results were

determined by evaluating the statistical significance between the rating assigned by each

of the judges in this study as compared to the composite rating of the panel for that

contest. These results were then compared based upon the adjudicator’s previous grade

level of teaching experience in relation to the grade level of ensembles adjudicated. The

emphasis of this study was on the relationship between the ratings assigned to ensembles

in which the adjudicator had the most experience teaching in that grade level compared to

the ensembles the adjudicator had little or no experience teaching in that grade level.

Results indicated that there was no statistical significance found in any of the judge’s

ratings when compared to the adjudicators’ previous grade level of teaching and the

ensembles in which they adjudicate.


vi

LIST OF TABLES

1. UIL Rating Table p. 10

2. Participant Data p. 13

3. Unpaired T Test Results p. 25


vii

LIST OF FIGURES

1. High School Judge A Analyzed Results p. 16

2. High School Judge B Analyzed Results p. 16

3. High School Judge C Analyzed Results p. 17

4. High School Judge D Analyzed Results p. 17

5. High School Judge E Analyzed Results p. 18

6. High School Judge F Analyzed Results p. 18

7. High School Judge G Analyzed Results p. 19

8. Middle School Judge AA Analyzed Results p. 19

9. Middle School Judge BB Analyzed Results p. 20

10. Middle School Judge CC Analyzed Results p. 20

11. Middle School Judge DD Analyzed Results p. 21

12. All High School Analyzed Results p. 22

13. All Middle School Analyzed Results p. 23


1

CHAPTER I

INTRODUCTION

Vignette

To sit in the audience and listen to a high school or middle school band is

something many people have done, whether as a parent or friend of a performer or just

a supportive community member. It is a common occurrence, but every year in the

state of Texas, thousands of bands go to the University Interscholastic League,

henceforth referred to as UIL, Concert and Sightreading Contest to test their abilities

against a standard, each seeking the coveted sweepstakes award signifying the group

as a “superior” performing ensemble. Having taken my own band to this contest

numerous times, I know the incredible importance and expectation placed on this one

day of the year and the pressure associated with it. Then I had the opportunity to

experience the other side of the auditorium, the job of the judge. The weight of my

decision along with the rest of the judging panel, could positively shape another

director’s career or, frankly, dissuade them from the profession. It could make for a

celebration by the students celebrating their hard work and the journey they took

together, or, unfortunately, something else altogether. As brutal as it seems, this is the

reality, but there are always deeper layers to every scenario and situation. Each

adjudicator, like the ensembles he or she judges, brings a vast array of experiences in

his or her field. Some have a treasure trove of experience in one level, and next to

nothing in another, yet they often judge across grade levels. Is this just a technicality

that really does not matter because “band is band” regardless of the grade level, or

does the previous grade level of teaching have an impact in the ratings assigned?


2

Justification

In Texas, performance ratings are an everyday part of a music educator’s life,

and much work and focus is based on the performance at the spring contest. The

University Interscholastic League (UIL) is the governing body of all public school

academic, athletic and music contests in the state of Texas (University Interscholastic

League, 2014 a). The Concert and Sightreading Contest occurs each spring and is

attended by middle school and high school bands, choirs, and orchestras (2014 a). This

contest is not a competition between groups or schools but an assessment on an

established standard. That standard is specifically described by judging rubrics created

by the Texas Music Adjudicators Association (2014). This association not only

provides the rubric by which the performing group will be compared, but also

identifies high-quality judges in each field. The TMAA member judges have applied

for and been accepted to the Texas Music Adjudicator’s Association (TMAA),

attended specific training seminars, and displayed a proven track record of excellence

at the school in which they teach (Texas Music Adjudicators Association, 2014).

These judges are then charged with comparing a group’s performance to the rubric and

assigning a rating. A rating of Division 1, or “Superior”, is the goal of each group and

the highest rating possible. A rating of a Division 2 is considered “Excellent”, a

Division 3 is “Average”, a Division 4 is “Below Average” and the lowest rating of

Division 5 is considered “Poor” (University Interscholastic League, 2014 b). Many

factors influence an adjudicator’s assigned rating (Geringer & Johnson, 2007), and

with this being an important piece of a student’s educational experience each year, it

could be beneficial for music educators to understand the impact a judge’s teaching


3

career may have had on his or her final ratings at UIL Concert and Sightreading

Contest.

Beyond the rubric that is used to judge these bands, however, is a human being

who has been charged with using his or her own knowledge and experiences to

accurately rate the group’s performance. All judges are selected on the basis of their

successful teaching experiences (Texas Music Adjudicators Association, 2014), but

each judge has different backgrounds and teaching experience, which raises the

question: How does a judge’s previous experience, particularly the grade level on

which he or she taught, affect the rating assigned to not only the group they have the

most experience teaching, but also the grade levels he or she has less, little, or no

experience teaching? Therefore, the purpose of this study was to determine the extent

to which the grade level of a judge’s prior teaching experience affects the assigned

rating of the bands he or she adjudicates.


4

CHAPTER II

REVIEW OF RELATED RESEARCH

There have been several in depth studies related to the adjudication of music

contests (Bergee, 2003, 2007; Brakel, 2006; Cavitt, 1997; Conrad, 2003; Fiske, 1975,

1978; King & Burnsed, 2009; Smith, 1999), as well as studies and publications

regarding the numerous factors that music educators must address while preparing

their groups for competition that might ultimately affect their contest rating (Baker,

2004; Geringer & Johnson, 2007; Hewitt, 2007; Killian, 1998, 1999, 2000). However,

a study identifying an adjudicator’s previous grade level of teaching in relation to the

ratings he or she gave has not been discovered. Although there are numerous studies

related to music adjudication, the spectrum of information relevant to the proposed

study included studies closely related to the following concepts:

1. Director controlled influences over the rating received

2. Non-director controlled influences over the rating received

3. Previous research related to judge reliability

Director Controlled Influences

Each year, the director of any organization makes many choices that can

positively or negatively affect the year-end outcome and overall success of that

organization. The same is true for music organizations in the state of Texas (May,

1989).

Among the many influences a music director has over his or her ensemble is

the selection of repertoire. Much emphasis has been placed upon the director to make


5

wise repertoire choices for his or her ensemble’s contest performances as described by

Killian (1998, 1999, 2000) in regards to choirs in Texas. Moving from repertoire

choices made outside of the classroom, Smith (1999) examined the actual rehearsal

procedures and practices that music educators, specifically marching band directors,

utilize to influence the eventual contest ratings of their groups. These two factors,

repertoire selection and rehearsal behaviors, are both decisions controlled primarily by

the music director. Further, Price (2006) and Morrison, Price, Geiger and Cornacchio

(2009) explored the director’s physical conducting itself and the many impacts the

level of preparation and conducting style have on the overall outcome of the ensemble

being conducted.

Non-Director Controlled Influences

Although the director of a music program has the ability to positively or

negatively affect the outcome of the group’s performance through decisions made

previous to and during the year, there are factors that can affect the rating and a

judge’s perception during a contest. Previous research would indicate that in solo and

ensemble competitions, judges’ ratings were influenced positively by an afternoon

performance time, belonging to a large, high expenditure school, and being a solo

vocalist (Bergee 2006; Bergee & McWhirter 2005; Bergee & Piatt, 2003). This

example of non-director controlled influence on a judge’s rating was contrary to the

findings of Brakel (2007) who probed ratings of solo musical performances across the

variables of occasion, sequence, and rater, finding lack of bias and general consistency

within the performances evaluated.


6

In 2008, Rickels examined multiple variables that influenced the results of

marching band festivals in the state of Arizona in 2004. Rickels found that elements

such as band budget, number of directors and assistants, marching band enrollment,

the number of festivals attended, school enrollment, and the concurrence of a concert

band program to have the greatest relationship to final scores, but that the director’s

tenure and total years of teaching experience had no significant statistical relationship

to the outcome of the contest.

Judge Reliability

The question of judge reliability is one that has been studied quite extensively

by Brakel (2006), King and Burnsed (2009) and Bergee (2007) who each examined

large contests at the state level for judge reliability.

In 2006, Brakel examined the reliability of judge’s ratings in the state of

Indiana; the results indicated a high level of reliability between judges. The groups

that performed at a higher performance quality yielded the closest agreement between

judges, but as the performance quality of the group declined, so did the agreement

between judges’ ratings, but only on a miniscule level. Brakel gave much credit to the

Indiana State School Music Association Instrumental Festival organization for their

comprehensive examination of judging results and the adjudication process as well as

the implementation of a training session for judges.

Similarly, King and Burnsed (2009) examined the 2005 Virginia Band and

Orchestra Directors Association State Marching Band Festivals and the judge’s ratings

assigned to the 124 competing bands. Just as in Brakel’s study, the reliability of the


7

judge’s ratings was very high. Additionally, King and Burnsed explored further,

studying the judge’s overall ratings in comparison to detailed captions such as: Quality

of Sound, Technique, and Musicianship. The results indicated the inter-correlations of

the caption ratings were very high as well. On the contrary, Bergee (2007) examined

the relationship between the adjudicator, performer, occasion and sequence of the

performance. The findings suggested virtually no measurable error when considering

occasion and sequence, but did find a substantive measurement error amongst the

adjudicators themselves.

Purpose

Given the data indicating the influence of a variety of factors on ratings, but the

lack of data regarding grade level of adjudicators’ prior teaching experience, the

purpose of this study was to determine the extent to which the grade level of a judges’

prior teaching experiences affects the assigned rating of the bands he or she

adjudicates.


8

CHAPTER III

METHODOLOGY

This study was designed to examine the extent to which the grade level of a

judge’s prior teaching experience affects the assigned rating of the bands he or she

adjudicates.

The data collected for this study consisted of ratings from UIL Concert and

Sightreading Contests from across the state of Texas over a period of fourteen years,

2001-2014. The ratings were gathered and sorted into two groups, an “assigned rating”

group (n = 3,493) and a “composite rating” group (n = 3,493), both of which are

defined below.

Terminology

University Interscholastic League Concert and Sightreading Contest

The University Interscholastic League (UIL) was created by the University of

Texas at Austin in 1910 to govern Texas public school academics, athletics and music

contests. It is the largest inter-school organization of its kind in the world

(University Interscholastic League, 2014 a). Within the UIL is the division of music,

which governs the various music specific events that occur annually throughout the

state (University Interscholastic League, 2014 b). Through the supervision of the UIL

Music Division and under the Constitution and Contest Rules (2014 b, 2014 c), the

Concert and Sightreading Contest is conducted each spring. These contests consist of

musical performances of a concert component and a sightreading component. The

concert component includes three prepared pieces by the ensemble adjudicated by a


9

three member-judging panel. The sightreading component consists of a reading of

literature composed specifically for the sightreading contest. As is on the concert

component, a three-judge panel adjudicates the sightreading performance.

Varsity Ensemble

In the state of Texas, the highest quality group at a given school is designated

the varsity group and performs a more difficult level of literature and sightreading than

the non-varsity groups during the UIL Concert and Sightreading contest (University

Interscholastic League, 2014 b). Second ensembles are designated as non-varsity and

lower ensembles are designated as sub non-varsity. Only varsity ratings in both

concert and sightreading were considered in this study to eliminate any discrepancies

caused by each school’s own guidelines for determining the makeup of a non-varsity

group.

Assigned Ratings

An assigned rating is the rating an individual adjudicator gave to a varsity

ensemble at a UIL Concert and Sightreading Contest within the time period of this

study. An assigned rating is specific to each judge and can vary from judge to judge.

The term “assigned rating” was applied because the adjudicator, through the use of a

judging rubric provided by the Texas Music Adjudicators Association (Texas Music

Adjudicators Association, 2014), assigned a rating of one through five to that

ensemble’s performance on that day.


10

Composite Ratings

A composite rating is the combined rating the ensemble received from the

judging panel that day. There were three judges on each panel (University

Interscholastic League, 2014 c) and each judge gave an assigned rating. This

composite rating takes into account each judge’s rating, but is not precisely an average

among the three-judge panel. Using the three assigned ratings applied to the University

Interscholastic League Rating Table (2014 c), Table 1, the composite rating is

determined for the group. The composite rating is in bold, and the assigned ratings of

the three judges and all various combinations are below.

Table 1 University Interscholastic League Rating Table

Rating I Rating II Rating III Rating IV Rating V

1-1-1 1-2-2 1-3-3 1-4-4 1-5-5

1-1-2 1-2-3 1-3-4 1-4-5 2-5-5

1-1-3 1-2-4 1-3-5 2-4-4 3-5-5

1-1-4 1-2-5 2-3-3 2-4-5 4-5-5

1-1-5 2-2-2 2-3-4 3-4-4 5-5-5

2-2-3 2-3-5 3-4-5

2-2-4 3-3-3 4-4-4

2-2-5 3-3-4 4-4-5

3-3-5

(University Interscholastic League, 2014 c)


11

Participants

Each participant was selected by identifying individual judges who had both a

high overall frequency of judging UIL contests, and less than nine years experience as

a head director at one grade level with the remainder of said career at the opposite

grade level. The Texas Music Adjudicators Association has, as an organization, been

challenged with creating a set of standards, not only for the ensembles they adjudicate,

but also for the adjudicator. The requirements to be a University Interscholastic

League judge necessitate a person to:

1. Display successful teaching

2. Maintain employment in music education

3. Attend applicable TMAA approved workshops once every four years

4. Have a record of superior performance

5. Be active as an adjudicator

(Texas Music Adjudicators Association, 2014)

Each judging panel according to the UIL Constitution and Contest Rules (2014

b) must be comprised of at least two TMAA qualified adjudicators.

First, to gather a pool of possible directors, I examined scores publicly

available on UILforms.com website (UILforms, 2014), Brynn Park Productions

(2014), and on the Texas Music Adjudicators Association (2014) website. This process

eliminated many of the possible candidates, as they did not meet the constraint of nine

years or less at one grade level, middle school or high school, with the remainder of

the career spent at the opposing grade level. Next, further research was done into the

frequency with which the participant actually was contracted by the UIL to adjudicate


12

Concert and Sightreading Contests over the time period 2001-2014. These data are

publically available at Brynn Park Productions (2014). There was not necessarily a

minimum number of times hired by UIL constraint because each UIL Concert and

Sightreading Contest can be very different. One contest may have only varsity

ensembles, another may have a mix of varsity and non-varsity, and yet another contest

may contain only non-varsity ensembles. The important data for this study were the

number of varsity ratings assigned by the adjudicator over the time period. In addition

to being hired to judge varsity ensembles at a UIL contest, each participating judge

had to have no fewer than 20 ratings assigned in each grade level, middle school and

high school, and no fewer than 80 ratings assigned overall. After these constraints

were met, eleven (N = 11) participants emerged. The average number of assigned

ratings for the high school judge group was 148 and the average for the middle school

judge group was 464. Details regarding the number of judging experiences per each

selected adjudicator appear in Table 3.


13

Table 3

Participant Data _____________________________________________________________________

Starting in the year 2001 and ending in the year 2014, all ratings assigned to

varsity bands in the state of Texas by these eleven adjudicators (N = 11) were

compiled amassing 3,493 (n = 3,493) assigned ratings and the same number of

corresponding composite ratings (n = 3,493).

Level of Teaching

Experience

Relative to Study

Total High School

Ensemble Ratings

Assigned

Total Middle

School Ensemble

Ratings Assigned

Total Varsity

Ratings

Assigned

Judge A 26 HS / 0 MS 53 35 88

Judge B 15 HS / 8 MS 135 183 318

Judge C 40 HS / 0 MS 108 183 196

Judge D 25 HS / 0 MS 75 22 97

Judge E 30 HS / 3 MS 295 68 363

Judge F 29 HS / 6 MS 163 28 191

Judge G 26 HS / 8 MS 157 226 383

Judge AA 2 HS / 25 MS 65 423 488

Judge BB 9 HS / 21 MS 145 616 761

Judge CC 3 HS / 27 MS 102 212 314

Judge DD 0 HS / 32 MS 26 268 294


14

Procedure and Materials

For each judge, data were compiled and analyzed into four mean numbers:

1. The judge’s assigned rating mean for high school ensembles

2. The corresponding composite rating mean for high school ensembles

3. The judge’s assigned rating mean for middle school ensembles

4. The corresponding composite rating mean for middle school ensembles

After these calculations were completed for each judge, the judges and the

calculated means above were grouped into four different scenarios for comparison.

Each scenario would then yield a mean assigned rating and mean composite rating

based upon the judge’s level of experience, middle school or high school, and the type

of ensemble adjudicated, middle school or high school.

1. High school experienced judges adjudicating high school ensembles

2. High school experienced judges adjudicating middle school ensembles

3. Middle School experienced judges adjudicating high school ensembles

4. Middle school experienced judges adjudicating middle school ensembles

These final mean numbers were then compared using t-tests for all combinations,

testing for statistical significance.


15

CHAPTER IV

RESULTS

The data consisted of information collected regarding the assigned ratings of

specific, experienced, judges (N = 11) and was compared then to the composite rating

for each varsity ensemble judged. The data gathered were used to determine to what

extent the judge’s previous grade level of teaching experience affected the assigned

rating in comparison to the composite rating of the judging panel.

Comprehensive Results by Director

These ratings were then compiled for each adjudicator and calculated into a

judge’s mean rating assigned and a corresponding composite rating assigned. This

allows for a comparison over a 14-year period of each adjudicator’s mean assigned

rating to the overall composite rating the ensemble received. See Figures 1 through 7

for each high school experienced judge’s analyzed results. See Figures 8 though 11 for

each middle school experienced judge’s analyzed results. The judges have been

assigned letter names, single letters for high school experienced adjudicators and

double letters for middle school experienced adjudicators.

When reading each figure, please note that the number atop each column

represents the mean of all ratings applicable to that column and can be compared

directly to the analyzed means of the other columns. Also note that a lower number

indicates a more favorable rating because ratings could range from 1 (high) to 5 (low).


16

Figure 1

High School Judge A Analyzed Results

_____________________________________________________________________

Figure 2

High School Judge B Analyzed Results


17

Figure 3

High School Judge C Analyzed Results

_____________________________________________________________________

Figure 4

High School Judge D Analyzed Results


18

Figure 5

High School Judge E Analyzed Results

_____________________________________________________________________

Figure 6

High School Judge F Analyzed Results


19

Figure 7

High School Judge G Analyzed Results

_____________________________________________________________________

Figure 8

Middle School Judge AA Analyzed Results


20

Figure 9

Middle School Judge BB Analyzed Results

_____________________________________________________________________

Figure 10

Middle School Judge CC Analyzed Results


21

Figure 11

Middle School Judge DD Analyzed Results

_____________________________________________________________________


22

Comprehensive Results by Grade Level Experience

After reviewing the results for each individual judge and the their data over the

14 year period, all high school data were then compiled for a broader look at the

judging outcomes. See Figure 13 for the calculated mean of all high school directors,

and the comparison between the assigned and composite ratings.

_____________________________________________________________________

Figure 13

All High School Judges Combined Analyzed Results

_____________________________________________________________________


23

Similarly, all data collected for each of the middle school experienced directors

were also compiled and analyzed as can be seen in Figure 14.

_____________________________________________________________________

Figure 14

All Middle School Judges Combined Analyzed Results

_____________________________________________________________________


24

Comprehensive Results Final Comparison

After these calculations were completed for each judge, the judges were grouped

into four different scenarios for comparison:

1. High school experienced judges adjudicating high school ensembles

2. High school experienced judges adjudicating middle school ensembles

3. Middle School experienced judges adjudicating high school ensembles

4. Middle school experienced judges adjudicating middle school ensembles

Results were analyzed using an unpaired t-test in each of the four groups. Through

each test, the results returned to reveal a lack of statistical significance in any

comparison.


25

Table 3

Unpaired T Test Results

Mean Assigned Rating

Mean Composite Rating

Two-Tailed P Value

High School Judge

Adjudicating

High School Ensembles

1.40304343 1.38604 0.7211

High School Judge

Adjudicating

Middle School Ensembles

1.426025 1.35671283 0.5285

Middle School Judge

Adjudicating

High School Ensembles

1.354239 1.321053275 0.8628

Middle School Judge

Adjudicating

Middle School Ensembles

1.48094175 1.4723975 0.8192


26

CHAPTER V

DISCUSSION

This study arose from personal experiences with UIL Concert and Sightreading

Contest in my early years of teaching as a high school band director. The results of

that contest did not go the way I wanted, but through that situation, this study arose.

The purpose of this study was to determine to the extent to which the grade level of a

judge’s prior teaching experience affects the assigned rating of the bands he or she

adjudicates.

Looking at the final results of the study, the consistency of the eleven (N = 11)

adjudicators was notable. The lack of statistical significance in the judges’ ratings in

relation to their previous grade level of teaching experience closely aligns with the

studies of King and Burnsed (2009) and Brakel (2006) who both found a high level of

reliability among judges’ scores. When looking at each judge’s results (See Figures 1 -

11), note each individual judge’s analysis with emphasis on how close all the judges

were in their assigned rating mean and composite rating mean. Considering the total

number of assigned ratings (n = 3,493), once again the consistency of the adjudicators

to accurately rate an ensemble’s performance is evident. Results should be generalized

with caution due to the fact that these data came from a single state and a single

adjudication training procedure.


27

Implications for Further Research

A few interesting scenarios developed while collecting data that could be cause

for further study, one of which was apparent pockets of ratings, sometimes lasting just

a few ensembles, and, more rarely, lasting several days of adjudication where a judge

displayed a higher level of “missed” ratings, meaning their assigned rating was not the

same as the composite for an extended period of ratings. Most of these strings of

misses were on the high side, meaning the adjudicator during this string of ratings was

judging more strictly, or more strictly than the other judges on the panel. Keep in

mind, in this contest, the ensemble’s goal is to attain a lower number rating, i.e. “1”,

which indicates a higher level of performance. An interesting study would be to isolate

these strings and conduct interviews with actual adjudicators to try and capture their

mindset during this period of judging. Could it have been the style of teaching in a

geographical region that influenced the judge, musical choices made by the directors

and students, or something that has affected the adjudicator on a more personal level?

Based on these findings, additional research could be conducted involving the

non-varsity ensemble. The non-varsity ensemble creates many different and interesting

scenarios for the adjudicator who knows that these are not necessarily the top players

at the school and has to apply the standards from the Texas Music Adjudicators

Association equally to them as he or she does to any other ensemble. Although this

study was limited to the examination of varsity ensembles, the researcher had the

opportunity to peruse the non-varsity ratings while collecting the varsity ensemble

data. After reviewing these non-varsity ratings assigned by the adjudicators that were

included in this study, there seems to be much more discrepancy in ratings. To


28

speculate, these discrepancies in ratings could be due to the very nature of the non-

varsity ensemble, which is generally comprised of less musically developed and less

technically advanced students. Those factors carry different weight with each

adjudicator, which may lead to instability in the accuracy and agreement between

judges. To study this might reveal how an adjudicator who has a stellar record with

varsity groups transfers that level of adjudication to the non-varsity ensemble.

Finally, I believe the most interesting result from this study would center

around how most of the adjudicators included in the study had higher mean assigned

ratings than composite rating assigned by corresponding judging panel. Keep in mind

Table 1, which illustrates how the composite rating system works. The composite

rating is not the mean rating of the three judges; in fact, it is not necessarily a

representation of agreement by the three judges on a rating. The only adjudicator to

have an assigned mean lower in both grade levels, middle school and high school, was

Middle School Judge CC, and only two adjudicators had one mean assigned higher

than the composite in one grade level, High School Judge B and Middle School Judge

BB. All three of these situations were ones in which mean average although higher on

the composite side, was still no more than 0.02 rating points in total difference. All

other judges in this study, in both grade levels, displayed a higher mean assigned

rating than composite rating, meaning that these changes on average resulted in a

higher rating to an ensemble than the judges on the panel sitting next to them. I would

speculate that this is because of the volume of ratings each of these judges has given

over their careers along with the confidence to assign a rating based upon that

experience.


29

Implication for Educators

After many months of collecting and analyzing data, the final results are what

we all as educators and adjudicators would hope for. No statistical significance in this

situation is an outstanding result that should be applauded by all parties involved.

Through the results of this study, it is clear that the intentions of the University

Interscholastic League and Texas Music Adjudicators Association are successfully

producing individuals who are not being swayed, positively or negatively, from a

ratings standpoint, by their previous grade level of teaching.


30

REFERENCES

Baker, V. (2004). The effect of repertoire selection on university interscholastic league Choral concert ratings. Texas Music Education Research. Retrieved from http://www.tmea.org/assets/pdf/research/Bak2004.pdf Bergee, M. J. (2003). Faculty interjudge reliability of music performance evaluation. Journal of Research in Music Education, 51, 137-150. doi:10.2307/3345847 Bergee, M. (2006). Validation of a model of extramusical influences on solo and

small-ensemble festival ratings. Journal of Research in Music Education, 54, 244-256.

Bergee, M. (2007). Performer, rater, occasion, and sequence as sources of variability

in music performance assessment. Journal of Research in Music Education, 55, 344-358.

Bergee, M. & McWhirter, J. (2005) Selected influences on solo and small-ensemble

festival ratings: replication and extension. Journal of Research in Music Education, 53, 177-190.

Bergee, M. J., & Piatt, M. C. (2003). Influence of selected variables on solo and small-

Ensemble festival ratings. Journal of Research in Music Education, 51, 342- 353. doi:10.2307/3345660

Bergee, M. J., & Westfall, С. R. (2005). Stability of a model explaining selected

extramusical influences on solo and small-ensemble festival ratings. Journal of Research in Music Education, 53, 358-374. doi:10.1177/002242940505300407

Brakel, T. (2006). Inter-judge reliability of the Indiana State School Music

Association High School Instrumental Festival. Journal of Band Research, 42, 59-69.

Brynn Park Productions (2014). Complete UIL Results. Retrieved from

http://www.brynnpark.com/#!archives/c1syx Burnsed, V., Hinkle, D., & King, S. (1985). Performance evaluation reliability at selected concert festivals. Journal of Band Research, 2/(1), 22-29. Cavitt, M. E. (1997). Effects of expectations on evaluators' judgments of music performance. Texas Music Education Research. Retrieved from http://www.tmea.org/assets/pdf/research/Cavl997.pdf


31

Conrad, D. (2003). Judging the judges: Improving rater reliability at music contests. NFHS Music Association Journal, 20(2), 27-31 Fiske, H. E. (1975). Judge-group differences in the rating of secondary school trumpet

performances. Journal of Research in Music Education, 23, 186-189. doi:10.2307/3344643

Fiske, H. E. (1978). The effect of a training procedure in music performance

evaluation on judge reliability. Ontario Educational Research Council Report, Canada.

Geringer, J. & Johnson, C. (2007). Effects of excerpt duration, tempo, and

performance level on musicians' ratings of wind band performances. Journal of Research in Music Education, 55, 289-301.

Hewitt, M. (2007) Influence of primary performance instrument and education level

on music performance evaluation. Journal of Research in Music Education, 55, 18-30.

Killian J. N. (1998). Characteristics of successful choirs in a contest setting. Texas

Music Education Research, 39-43. Retrieved from http://www.tmea.org/assets/pdf/research/Kill998.pdf

Killian J. (1999). Music selection of successful choirs at UIL and non-UIL contests.

Texas Music Education Research, 51-56. Retrieved from http://www.tmea.org/assets/pdfresearch/Kill999.pdf

Killian, J. (2000). Effect of music selection on contest ratings: Year three of a

continuing study. Texas Music Education Research. Retrieved from http://www.tmea.org/assets/pdf/research/Kil2000.pdf

King, S. & Burnsed, V. (2009). A study of the reliability of adjudicators ratings at the

2005 Virginia Band and Orchestra Directors Association State Marching Band Festivals. Journal of Band Research, 45, 27-32.

May, W. (1989, October). On Competition. Southwestern Musician, 6, 8. Morrison, S., Price, H., Geiger, C. & Cornacchio, R. (2009). The effect of conductor

expressivity on ensemble performance evaluation. Journal of Research in Music Education, 57, 37-49.

Price, H. (2006). Relationships among conducting quality, ensemble performance

quality, and state festival ratings. Journal of Research in Music Education, 54, 203-214.


32

Rickels, D. (2008). A comparison of variables in Arizona marching band festival results. Journal of Research in Music Education. 44, 25-39.

Smith, J.W. (1999). Correlation of discreet and continuous contest ratings and

marching band director rehearsal behaviors. (Doctoral dissertation, University of Kansas, 1999). Dissertation Abstracts International, 60 (09A), 3303.

Texas Music Adjudicators Association. (2014). Membership Requirements. Retrieved

from http://www.txmaa.org/tmaameminfo.php UIL Forms (2014). Results. Retrieved from http://www.uilforms.com/results.asp University Interscholastic League. (2014 a). About the UIL. Retrieved from

http://www.uiltexas.org/about University Interscholastic League. (2014 b). Constitution and contest rules: Section

1109-1110. Retrieved from http://www.uiltexas.org/files/constitution/uil-ccr-section-1109-1110.pdf

University Interscholastic League. (2014 c). Constitution and contest rules: Section

1111-1115. Retrieved from http://www.uiltexas.org/files/constitution/uil-ccr-section-1111-1115.pdf


33

APPENDIX


High School Judge MEAN

TOTAL SUM OF RATINGS

TOTALNUMBER OF RATINGS

AHS 53 HSassigned 1.377358491 73 assigned 1 1 1 1 2 1 1 1 1 1 2 2 2 1 2 1 2 2 1 1 1 1 2 3 1composite 1.320754717 70 composite 1 1 1 1 2 1 1 1 1 1 2 1 2 1 2 1 1 2 1 1 1 1 2 3 1MS 35 MSassigned 1.314285714 46 assigned 1 1 2 2 1 1 1 1 1 2 2 2 1 1 1 1 1 1 1 2 1 4 1 1 1composite 1.228571429 43 composite 1 1 2 2 1 1 1 1 1 2 2 2 1 1 1 1 1 1 1 3 1 2 1 1 1

HSassigned 2 1 2 2 1 1 1 1 1 1 1 1 2 1 1 2 2 1 1 3 1 1 1 1 2composite 2 1 2 2 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 3 1 1 1 1 2MSassigned 1 1 1 1 1 1 1 2 2 1composite 1 1 1 1 1 1 1 1 1 1

HSassigned 1 2composite 1 2MSassignedcomposite

34





BHS 135 HSassigned 1.348148148 182 assigned 1 1 2 1 2 1 1 1 1 1 1 1 1 1 2 1 2 3 1 3 3 1 1 1 3composite 1.355555556 183 composite 1 1 2 1 2 1 1 1 1 1 1 1 1 1 2 1 1 3 1 3 3 1 1 1 3MS 183 MSassigned 1.513661202 277 assigned 2 1 1 3 3 1 4 1 1 2 1 3 2 3 1 2 1 2 1 3 4 1 2 1 1composite 1.49726776 274 composite 2 1 1 3 3 1 4 1 1 2 1 2 2 3 1 2 1 2 1 3 3 1 2 1 1

HSassigned 3 1 1 1 1 2 1 4 1 1 1 2 2 1 1 1 3 2 2 1 3 1 1 1 1composite 3 1 1 1 1 2 1 4 1 1 1 2 3 1 1 1 3 2 2 1 3 1 1 1 1MSassigned 1 1 1 1 3 1 1 1 1 1 4 1 2 2 1 2 1 3 2 1 2 1 2 1 2composite 1 1 2 1 2 1 1 1 1 1 4 1 2 1 1 2 1 3 2 1 2 1 2 1 1


35


B (2 of 2)

HSassigned 2 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4 1 1 2 2 1 1 1 1 2 1 1 1 1 1 1 3 2 1 3 1 1 1 1 1composite 2 1 1 1 1 1 1 1 2 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 1 2 2 1 1 1 1 2 1 1 1 1 1 1 3 2 1 3 1 1 1 1 1MSassigned 3 2 1 1 1 1 2 1 1 2 1 3 2 1 1 1 1 1 2 1 1 1 1 2 1 2 3 2 1 1 3 3 1 1 1 2 2 1 2 1 3 1 1 2 1 1 1 2 1 1 1composite 3 2 1 1 1 2 2 1 1 2 1 3 2 1 1 1 1 1 2 1 1 2 1 2 1 2 3 2 1 1 2 2 1 1 1 2 2 1 2 1 3 1 1 2 1 1 1 3 1 1 1

HSassigned 1 2 1 1 1 1 1 1 1composite 1 1 1 1 1 1 1 1 1MSassigned 2 2 3 1 1 1 1 3 1 1 1 1 1 1 1 2 1 2 1 1 1 3 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 1 2 1 1 1 1 1 1 1 1 1 2 1 1composite 2 1 3 1 1 1 1 2 2 2 1 1 1 1 1 2 1 2 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 2 1 3 1 2 1 1 1 1 1 1 1 1 1 1 1 1

HSassignedcompositeMSassigned 1 2 1 1 3 1composite 1 2 1 2 2 1

36




TOTAL NUMBER OF RATINGS

CHS 108 HSassigned 1.305555556 141 assigned 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 2 2 1 1 3 1 1 1 1 1composite 1.305555556 141 composite 2 1 1 1 1 1 2 1 1 1 1 1 1 1 1 2 2 1 1 3 1 1 1 1 1MS 88 MSassigned 1.613636364 142 assigned 2 1 3 1 3 2 2 1 1 1 2 1 2 1 2 1 1 3 1 1 1 1 1 1 1composite 1.568181818 138 composite 2 1 3 1 3 2 2 1 1 1 1 1 2 1 2 1 1 3 2 1 1 1 1 1 1



37


C (2 of 2)

HSassigned 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1composite 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1MSassigned 1 2 1 2 2 3 4 3 3 1 1 1 1composite 1 1 1 2 2 2 3 2 3 1 1 1 2

HSassignedcompositeMSassignedcomposite


38





DHS 75 HSassigned 1.546666667 116 assigned 3 4 2 1 1 1 1 1 2 1 3 1 1 5 3 1 1 1 3 2 1 1 2 1 1composite 1.533333333 115 composite 4 4 2 1 1 1 1 1 1 1 3 1 1 5 3 1 1 1 3 2 1 1 2 1 1MS 22 MSassigned 1.318181818 29 assigned 2 1 1 1 1 2 2 1 2 1 1 1 3 1 1 1 1 2 1 1 1 1composite 1.272727273 28 composite 2 1 1 1 1 2 1 1 1 1 2 1 3 1 1 1 1 2 1 1 1 1

HSassigned 1 1 1 3 3 1 3 3 3 1 1 3 1 1 1 1 1 1 1 2 1 1 1 1 3composite 1 1 1 3 3 1 2 3 3 1 1 3 1 1 1 1 1 1 1 2 1 1 1 1 3MSassignedcomposite


39





EHS 295 HSassigned 1.376271186 406 assigned 1 2 1 1 1 1 2 1 1 1 1 1 1 1 1 2 1 1 4 1 3 2 3 1 1composite 1.359322034 401 composite 1 2 1 1 1 1 2 1 1 1 1 2 1 1 2 2 1 1 4 1 3 2 3 1 1MS 68 MSassigned 1.102941176 75 assigned 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1composite 1.073529412 73 composite 2 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1


HSassigned 1 1 2 1 4 4 2 1 1 2 1 2 1 1 1 1 2 1 1 1 1 1 1 2 2composite 1 1 2 1 5 3 2 1 1 2 1 1 1 1 1 1 2 1 1 1 1 1 1 2 2MSassigned 1 1 3 3 1 1 2 1composite 1 1 2 3 1 1 2 1

40


E (2 of 3)

HSassigned 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 2 1 1 2 1 1 1 1 1 1 2 1 1 1 1 1 3 1 2 1 2 1 3 1 1 1 1 2 1composite 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 2 1 1 1 1 1 1 2 1 1 1 1 1 2 1 2 1 1 1 3 1 1 1 1 1 1MSassignedcomposite



41


E (3 of 3)


HSassigned 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1composite 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1MSassignedcomposite

42





FHS 163 HSassigned 1.472392638 240 assigned 1 2 3 2 1 4 1 1 2 1 1 3 1 1 2 2 1 3 1 1 1 1 3 1 2composite 1.441717791 235 composite 1 2 3 2 1 4 1 1 2 1 1 2 1 1 2 1 1 3 1 1 1 1 3 1 2MS 28 MSassigned 1.5 42 assigned 2 2 1 1 1 1 2 1 1 1 1 2 2 1 2 2 4 2 3 1 2 1 1 1 1composite 1.5 42 composite 3 2 2 1 1 1 2 1 1 1 1 2 1 1 2 2 3 2 3 1 2 1 1 1 1

HSassigned 1 1 2 1 1 2 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 1 1 1 1composite 1 1 2 1 1 2 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 1 2 1 1MSassigned 1 1 1composite 1 1 1


43


F (2 of 2)


HSassigned 2 1 1 1 1 1 1 1 2 1 1 3 1 3 1 1 2 2 1 2 1 1 1 2 1 1 1 1 2 1 1 1 1 1 1 1 2composite 2 1 1 1 1 1 1 1 2 1 2 2 1 3 2 1 2 1 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 2MSassignedcomposite

44




TOTALNUMBER OF RATINGS G (1 of 2)

GHS 157 HSassigned 1.394904459 219 assigned 1 1 1 2 1 1 3 1 1 1 1 2 1 1 1 1 1 1 1 2 2 1 1 1 1composite 1.388535032 218 composite 1 1 1 2 1 1 3 1 1 1 1 2 1 1 2 1 1 1 1 2 2 1 1 1 1MS 226 MSassigned 1.619469027 366 assigned 4 3 2 4 1 2 5 1 2 2 1 1 1 1 2 1 1 1 1 1 1 1 2 1 1composite 1.553097345 351 composite 3 3 2 4 1 2 5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1



45


G (2 of 2)


HSassigned 1 1 1 2 3 1 1 1 2 1 1 2 1 1 1 1 1 2 1 3 1 1 1 2 1 2 4 3 2 1 2composite 1 1 1 2 3 1 1 1 1 1 2 2 1 1 1 1 1 2 1 2 1 1 1 2 1 2 4 3 2 1 2MSassigned 2 1 1 1 1 3 1 2 1 2 3 2 1 1 1 1 1 1 2 2 1 2 2 2 2 1 1 2 1 1 1 1 2 1 2 2 2 3 2 1 2 2 1 2 1 1 1 3 1 1 1composite 2 1 2 1 1 3 1 2 1 1 2 2 1 1 1 1 1 1 1 1 1 2 2 2 2 1 1 2 1 1 1 1 2 1 2 2 2 3 1 1 2 2 1 1 1 1 1 3 1 1 1

HSassignedcompositeMSassigned 1 2 2 2 1 3 2 2 1 3 3 1 2 2 3 3 1 1 1 1 1 1 1 1 2 1 2 1 1 2 3 3 1 1 1 3 2 1 3 2 1 2 1 2 2 3 3 2 2composite 1 2 2 2 1 3 2 2 1 3 3 1 1 1 3 3 1 1 1 1 1 1 1 1 2 1 2 1 2 2 3 3 1 1 1 3 2 1 3 2 1 2 1 2 1 3 3 2 2

46


Middle School Judge MEAN



AAHS 65 HSassigned 1.569230769 102 assigned 1 2 2 1 1 2 1 1 1 1 1 1 1 1 2 1 1 1 1 2 1 3 1 2 1composite 1.584615385 103 composite 1 2 2 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 3 1 2 1MS 423 MSassigned 1.517730496 642 assigned 3 3 2 3 1 1 4 1 1 1 1 1 1 1 1 1 2 1 1 1 2 2 3 3 2composite 1.48463357 628 composite 3 3 2 3 1 2 5 1 1 1 1 1 1 1 1 1 2 1 1 2 3 2 3 3 1


HSassigned 1 2 4 1 3 2 1 1 1 2 2 1 1 1 1composite 1 2 4 2 3 2 1 1 1 2 2 1 1 1 1MSassigned 1 1 1 1 1 3 1 1 2 2 1 2 1 1 2 1 1 1 4 1 2 1 2 3 1composite 1 1 1 1 1 3 1 1 2 2 1 2 1 1 2 1 1 1 4 1 2 1 2 2 1

47


AA (2 of 4)

HSassignedcompositeMSassigned 2 1 1 1 1 1 1 1 2 1 3 3 2 1 3 2 1 3 1 1 1 1 2 2 1 1 2 1 1 1 2 3 1 2 2 1 2 2 2 1 1 1 1 2 1 1 1 1 3 1 4composite 1 1 1 1 1 1 1 1 2 1 2 3 2 1 3 2 2 4 1 1 1 1 2 2 1 1 2 1 1 1 2 3 2 3 3 1 2 2 2 1 1 1 1 2 1 1 1 1 3 1 3



48


AA (3 of 4)




49


AA (4 of 4)

HSassignedcompositeMSassigned 1 2 2 1 1 1 1 2 2 2 1 1 1 1 2 2 1 3 1 1 1 2 1 1 2 2 1 3 2 2 1 1 1 1 1 2 1 1 1 1 1 1composite 1 2 2 1 1 1 1 1 2 1 1 1 1 1 2 2 1 2 1 1 1 1 1 1 2 2 1 3 2 2 1 1 1 1 1 2 1 1 1 1 1 1



50





BBHS 145 HSassigned 1.289655172 187 assigned 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 3 2 1composite 1.317241379 191 composite 1 1 1 1 1 1 1 1 1 2 1 1 2 1 1 1 1 1 1 1 1 2 3 2 1MS 616 MSassigned 1.50974026 930 assigned 3 3 3 2 1 2 5 1 1 1 1 1 1 1 1 2 1 2 3 1 1 2 1 1 2composite 1.5 924 composite 3 3 2 3 1 2 5 1 1 1 1 1 1 1 1 2 1 2 3 1 2 2 1 2 2



51


BB (2 of 6)


HSassigned 1 1 1 1 1 1 1 2 1 2 1 2 2 1 3 3 1 2 2composite 1 1 1 2 1 1 1 2 1 2 1 2 2 1 3 3 1 2 2MSassigned 2 3 1 1 1 1 1 2 2 1 1 1 1 1 2 1 1 2 1 1 1 1 1 1 1 2 1 4 3 1 1 1 1 3 2 2 1 2 1 2 1 5 4 1 2 1 3 2 1 1 1composite 2 3 1 1 1 1 1 2 2 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 3 3 1 1 1 1 3 2 1 1 2 1 2 1 5 4 1 2 1 3 2 1 1 1


52


BB (3 of 6)




53


BB (4 of 6)




54


BB (5 of 6)




55


BB (6 of 6)

HSassignedcompositeMSassigned 1 1 1 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 1 1 1 2 2 1 1 1 1 1 3 1 2 1 1composite 1 1 1 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 2 2 2 2 2 2 2 1 1 1 2 2 1 1 2 2 1 3 1 2 1 1

56





CCHS 102 HSassigned 1.519607843 155 assigned 3 3 1 2 2 1 1 3 2 1 1 1 1 1 1 1 1 1 1 3 2 2 1 1 1composite 1.529411765 156 composite 4 4 1 1 2 1 1 2 2 1 1 1 1 1 1 1 1 1 1 3 2 3 1 1 1MS 212 MSassigned 1.485849057 315 assigned 5 1 1 1 1 1 3 2 1 2 1 1 3 1 3 1 2 3 3 4 4 1 2 2 2composite 1.509433962 320 composite 5 1 1 2 1 1 3 2 1 2 1 1 2 2 3 1 2 4 3 4 3 2 2 3 2



57


CC (2 of 2)

HSassigned 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 2 2 1 3 3 1 2 2composite 1 3 1 1 2 1 1 1 1 1 1 1 1 1 2 2 1 2 1 2 2 1 3 3 1 2 2MSassigned 3 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 2 1 1 1 1 1 3 1 1 3 2 1 1 1 2 1 1 3 2 1 1 1 2 2 1 2 4 3 1 2 4 1 1 1composite 3 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 2 1 1 1 1 1 2 1 1 3 2 1 1 1 2 1 1 3 2 1 1 1 2 2 1 3 4 3 1 2 4 1 1 1


HSassignedcompositeMSassigned 1 2 1 1 1 1 2 2 1 1 1 2 1 1 2 1 1 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 2composite 1 2 1 1 1 1 2 3 1 1 1 2 3 1 2 1 1 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1

58





DDHS 26 HSassigned 1.038461538 27 assigned 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1composite 1.038461538 27 composite 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1MS 268 MSassigned 1.410447761 378 assigned 2 1 1 2 1 2 1 1 1 2 2 1 2 1 2 1 1 2 2 1 1 1 2 2 1composite 1.395522388 374 composite 1 1 1 1 1 2 1 1 1 1 2 2 2 1 2 1 1 2 2 1 1 1 1 2 1

HSassigned 1composite 1MSassigned 1 1 1 2 3 1 1 1 1 2 2 1 1 1 1 1 2 1 1 1 1 2 1 1 1composite 1 1 1 2 3 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1

HSassignedcompositeMSassigned 1 1 1 1 1 1 1 1 2 3 1 1 1 1 2 1 1 1 2 1 1 2 1 2 1composite 1 1 1 1 1 2 1 1 2 3 1 1 1 1 1 1 1 1 2 1 1 2 1 2 1

59


DD (2 of 3)




60


DD (3 of 3)

HSassignedcompositeMSassigned 1 1 3 1 2 1 2 3 2 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 2 1 1 1 2 1 2 1 2 1 3 2 1 1 1 2composite 2 1 3 1 2 1 2 3 3 1 1 1 2 1 1 1 1 1 1 2 1 1 1 1 2 1 1 1 2 1 2 1 2 1 3 2 1 1 1 2



61

Documents

The Judge Factor: The Effect of an Adjudicator’s Teaching