Developing a Program and Music Teacher Evaluation Tool for Music Education - Creating Your Own Accountability Report

Developing a Program and Music Teacher

Evaluation Tool for Music Education - Creating

Your Own Accountability Report

Douglas C. OrzolekUniversity of St. Thomas St. Paul,

Minnesota

Developing Evaluation Tools for Music Education

Session Outline External Factors Internal Factors Administrators/Supervisors Literature On Developing an Evaluation Tool The Teacher Performance Assessment An Accountability Report What Next?


External Factors Attitude (Peck) * Need for Composite Rating / “% Problem” (Burling) * Inconsistency Policy from State to State, District to District VAMs (Darling-Hammond, 2012) “Transparency or Accuracy?” (Prince, Friedman) Evaluator training, understanding of music, criteria don’t

apply $ to develop a Proper Assessment Reliability and Validity Legal Issues and Policy Validity (Prince)

Developing Evaluation Tools for Music EducationEvaluation Components for

Teachers in Non-tested Subjects or Grades – D.C. IMPACT Program (Steele,

2010)

75% Administrator or Master Educator Observation

10% Student Growth on a Teacher-Chosen Measure

10% Commitment to the School Community

5% School-wide Value-Added


Internal Factors How Our Work is Percieved (Orzolek) * Subjective/Affective (Maranzano, Barrett) * Student Achievement? Connected to Learning? (Bowman) One Size Does Not Fit All - Multifaceted Jobs – Create, Respond,

Perform; Classroom, Ensemble, Individual (Grant, Taebel) Criteria for Excellent Music Teaching? (Brophy, Taebel);

“Successful” Teaching versus “Good” Teaching (Barrett) More Research – Connect to Other Non-tested Areas? The “My Program” Problem – Evaluate me on Program Success “If we say that music kids are smart, then why are we afraid of

VAMs? Wouldn’t that prove our point?” (Sean Barker, UST ‘12)


MSBA Survey: Accountability in Music (Orzolek, 2004)No Complaints

“Good” ConcertsSuccessful Trips

No Letters to the EditorTrophies

Athletic BandsKids Seem Happy

C&I Person is HappyA Local “Expert”

Mostly at Budget Time“What else is there?



Reminders from Music Leaders Literature Clear Goals and Objectives – Cowden and

Klotman (1991), 97 “Evaluation ought to involve looking critically at

the process of gathering and weighing evidence that will reveal changes in terms of a desirbable musical product.” – Weyland (1960), 295

Very Subjective – Snyder (1959), 114 Multifaceted – Dykema and Gehrkens (1941), 368 “Seek competent evaluation of your work.” –

Prescott (1938), 259


On Developing Evaluation Tools – ExternalBurling (2012) (aka Pearson Corporation)

Define a construct – What is an effective educator? Deploy multiple indicators – What evidence characterizes good

teaching and school leaderhip? Develop a clear composite rating – What weights should each

indicator have and who should be involved in the decision? Clarify differentiated performance levels – What distinguishes

varying levels of educator effectiveness? Build strong data analysis and reporting tools – What does the

information reveal about student, educator, and school performance?

Improve Instructional Leadership and Practice – How can the information target professional development to boost educator practice, student learning outcomes, and school efficacy?


On Developing Evaluation Tools – ExternalOthers

Promote consistency in any student measures used to assess teachers (Steele)

Implement tools after experimentation (Steele) Should supply teachers with feedback – “close the loop”

(Prince) Teachers should determine the extent to which

math/reading scores are included (Prince) Portfolio (Prince) Always use tools that are valid and reliable – Charlotte

Danielson. (Peck)


On Developing Evaluation Tools - InternalBrophy (1993)

Personal characteristics, as evidenced through student-teacher, teacher-faculty, and teacher administration interactions;

Music competence and performance skill, as evidenced when teaching a lesson;

Effective use of nonverbal strategies, such as modeling/demonstrating;

Effective use of verbal strategies, such as eliciting performance from students;


On Developing Evaluation Tools - InternalBrophy (1993)

Classroom management, including discipline, group participation, and the creation of a positive learning environment;

Effective planning for concept learning and aesthetic appreciation across a wide age span;

An objective assessment of teaching style based on empirically supported criteria;

Appropriate professional development activities should be recommended.


Three Categories of Teacher EvaluationNAfME (2011)

Teacher Evaluation linked exclusively to student outcomes.

Teacher evaluations linked exclusively to teacher practice.

Teacher evaluations that combine measures of student outcomes and teacher practice.


Question

How do we help prepare teachers for these evaluations?


TPA Background Teacher Performance Assessment – Stanford,

AACTE, CCSSO, Pearson, MACTE. 21 states actively involved.

Assesses “Effectiveness” of Pre-service Educators

The entire assessment is outlined in a “TPA Handbook” – “The Performing Arts”.

Completed during Student Teaching Scored by Trained Scorers via E-folio


How does the assessment work? “An Authentic Assessment of Teaching

Competence” Students complete 4 Tasks:

Describe their Plans for Instruction & Assessment; Engage Students in Learning; Assess Student Learning; Analyze Teaching.

Parameters “Central Focus of Learning Segment” 3-5 Lessons Evidence provided on Video Use of “Academic Language” Detailed Rubrics Provided


Starting Point: A Voluntary

AccountabilityReport

2002 - 2003 Accountability ReportInstrumental Music Education Department - George Washington High School

Contact: Douglas C. Orzolek ([email protected])

Mission Statement : High Quality, Comprehensive Music Education

Goals/Results for 2003-2004 Increase number of national standards completed /

Implemented composition unit this year Improve student rankings in state portfolio assessments / See

below Enhance music theory understanding by adapting new

software / New software purchased and implemented Provide outstanding performance opportunities for students /

Performances were well received - CD attached!

National StandardsThis year we added a composition (Standard 4)unit to our instruction. Several guests spoke oftheir experiences as composers, we took a fieldtrip to the American Composer’s Forum andwrote our own pieces that were performedduring our recital at the end of the year.

State/District Standards11th grade portfolios were sent to ourneighboring schools for evaluation andassessment by music educators in thosedistricts. The following are the results basedupon established rubrics: 4 (Exemplary) - 15% (10% in 2002) 3 (Proficient) - 45% (40% in 2002) 2 (Developing) - 35% (45% in 2002) 1 (Poor/Incomplete) - 5% (5% in 2002)The results suggest that our students areperforming well in these assessments of their work. This year’s scores show animprovement from the previous year.

Comments from Professional Reviews“Strongest performing group I have heard in years.” - LeonardBernstein

Comments from Student/Peer Evaluations “I feel like I have a better understanding of my instrument

after I’ve studied a composer and her intentions.” “”Billy’s perceptions of the performance were much different

than mine. But, I appreciated his points - they reminded me that music can be something different for everyone.”

“Opera - I still don’t like it.”

Testing Results for 2003-2004 9th Graders completed the NAEP music skills written portion

and scored in the 72 percentile nationally. Music reading skills measured by the Watkins-Farnum

performance scale continue to improve. The mean etude completed by students in grades 9-12 was #6 in 2002 and rose to #8 in 2003.

Average SAT score of a student in band was 1425 (of 1600.)

Other Highlights 97% of our students participated in the MN solo/ensemble

festival this year. 35% received Superior and 40% received Excellent ratings.

Our Concert Band earned the top ranking of Superior at the large group festival.

Our Marching Band was selected to perform in the Rose Parade.

We commissioned a new work from composer Libby Larsen. 6 band members were selected to perform with the All-State

band. Our students gave 60 public performances during this past

year. 80% of senior band members are planning to attend a 4

year college.


What Next? Top of NAfME Agenda – Policy Analysis, Leadership Stay Informed – Websites, Readings Share Stories – Forum, Meetings Coordinate and Advance Research Efforts - Grants The Reality - Multifaceted and Messy –

Experiment. This is about People not Composite Ratings. Until we agree on what constitutes effectiveness,

it will be difficult to measure and reward it.


Darling-Hammond & Bransford (2005)Effective Teachers: Understand subject matter deeply and flexibly; Connect what is to be learned to students’ prior knowledge and

experience; Create effective scaffolds and supports for learning’ Use instructional strategies that help students draw connections, apply

what they’re learning, practice new skills, and monitor their own learning;

Assess student learning continuously and adapt teaching to student needs;

Provide clear standards, constant feedback, and opportunities for revising;

Develop and effectively manage a collaborative classroom in which all students have membership.


“When a call for accountability elicits discussions about

education, the results produced are usually positive.”

R. Colwell, (2003) “The Status of Arts Assessment: Examples from Music”, Arts Education Policy Review,

105 (2), 16.

Documents

Developing a Program and Music Teacher Evaluation Tool for Music Education - Creating Your Own Accountability Report