Performance-Based Assessments

Jennifer Weber

Arizona Department of Agriculture

Presentation Overview

Overview of standardized multiple-choice exam and performance based exam formats

Activity involving each type of exam Compare and contrast formats based on your test

results Describe characteristics of pesticide handlers Draw conclusions about effectiveness of these two

types of exams for the agricultural community

Assessing Knowledge. . .

through a hands-on activity

through multiple-choice questions on a written test

Assessing Knowledge. . .

Hmm, now what is wrong with this picture?

Hmm, now which of these is the correct answer?

Assessing Knowledge. . .

Working together in small groups to demonstrate how to respond to a

pesticide-related situation

Individually answering questions about how to respond to pesticide-

related situations

O.K. class, let’s begin.

Correct Answers

Your Results?

Multiple-Choice Exam Activity

You will receive a 5 question, written, multiple-choice exam

Please read each question thoroughly and circle the correct answer

You will have 3 minutes to complete the exam Do not discuss questions or answer options with

others in the room I will provide you with the correct “letter” but will

not explain the answer itself. You may begin your test

Performance-Based Exam Activity

Do we have a volunteer?

Test Format Preference

Discussion

How many people in this session preferred the multiple-choice exam format? Why?

How many people in this session preferred the performance-based exam format? Why?

Test Format Preference Quote:

“Personally I like multiple choice as it is not open to interpretation and judgment of the person administering the exam and embarrassment, should I not have a clue about the answer. I have a 25% chance of getting the correct answer with multiple choice.”

(Jack Peterson, personal communication, April 11, 2007)

Advantages of Multiple-Choice Exams Over Performance-Based Exams

Easier to administer Less time consuming for department staff Can be scored quickly and easily if a Scantron form and

scoring machine are used More economical Fewer logistics Can test more individuals in a short amount of time

Multiple-Choice Exams for Pesticide Applicators

May be the ideal format for a fluent English-reader who grew up on a farm, has completed high school in the United States and is accustomed to taking multiple-choice exams

Anyone without the aforementioned characteristics could struggle with the multiple-choice exam format.

(Brenda Ball and Jack Peterson, personal communication, April 11, 2007)

Diversity of Agricultural Community

Language skills Literacy levels Test taking experience Learning styles

Educational background

Work experience Age

What is an Effective Way to Measure Knowledge and Skills?

The story of the three brothers

The story continues. . .

Language and Literacy

The following table is from the 2005 NAWS report and displays the participants’ English speaking and reading abilities by place of birth and ethnicity data.

English Speaking and Reading Ability, by Place of Birth and Ethnicity

Language and Literacy

Transferring knowledge and terminology from one’s native language to English can be difficult, especially when the same knowledge was acquired in a different context. In this type of situation, standardized exams can become tests of English reading and comprehension abilities and not of content knowledge and skills.

(Carlon Ami, Relevance of Assessment and Culture in Evaluation (RACE) Conference. Tempe, AZ 2004)

Language and Literacy

“Thinking processes aside, non-English speakers often must learn additional words. For example, while all of them know that they must protect their eyes and skin during pesticide application, ‘ocular’ and ‘dermal’ are two more words that have to be learned to pass many exams.”

(Gerald Kinro, personal communication, May 6, 2007)

Multiple-Choice Exams - Spanish

Private applicator licensing exams are offered in Spanish in Oregon, Washington, and California.

Results:

Oregon: 55% pass rate among pesticide applicators who took the Private Applicator’s exam in English and a 3-5% pass rate among those who take the same exam in Spanish

(Myron Shenk, personal communication, 2001)

Monterey County, California: 80% of the English test takers and 15% of the Spanish test takers passing the Private Applicator exam on their first attempt

(Maria Vidauri, personal communication, May 6, 2004)

Learning Styles and Test Taking Experience

Native American students are holistic, visual learners, a contrast to the more logical and linear thinking skills that are often needed to excel on multiple-choice exams.

They possess unique intelligences and a vast amount of knowledge and skills that cannot be captured through standardized exams.

(Carlon Ami, RACE Conference. Tempe, AZ 2004)

Work Experience and Knowledge

Becky Hines provides a test preparation course at WSU for landscapers

Most have been working in the industry for several years and have vast amount of knowledge about pesticides and pest control

Excel in group work, individual tasks, and practice quizzes during the three day test preparation course

Years of Experience Handling Pesticides

Very few have passed the pesticide applicator’s licensing exam that is administered immediately following the course

“It’s so frustrating! They’re not stupid. They know the information but they just can’t seem to pass these tests.”

(Becky Hines, personal communication, March 30, 2007).

Equity Concerns

The problem arises when these single scores are low despite one’s experience and expertise in the area of study.

Pesticide handlers who have limited English language and/or test taking abilities may struggle with the traditional multiple-choice format and be unable to display their true knowledge on the subject.

Educational Background and Test Taking Experience

“The multiple-choice tests are hard for the Spanish-speaking growers from Mexico. Where I grew up in Mexico, we didn’t have multiple-choice exams or the Scantron form. It’s hard to adjust to these types of tests.”

(Maria Vidauri, personal communication, 2004)

Possible Solutions

One might think that a simple solution would be to present a course on multiple-choice test taking skills, to teach people how to use process of elimination tactics, pace themselves on timed tests, and fill in Scantron “bubbles” clearly and correctly

While such a session might be helpful for some, efforts should really be made on finding the right tool or tools to assess what pesticide handlers already know or need to learn about pesticide safety

Alternative Form of Assessment

The provision of an alternative form of assessment, such as performance-based exam, would allow test takers who might have difficulties passing a multiple-choice exam with an opportunity to demonstrate their knowledge and skills through authentic, hands-on activities, case studies, and question and answer sessions designed to reflect actual workplace situations.

Examples of Performance-Based Exam Programs

Crane operator’s exam Business school admission

requirement Driver’s license driving

demonstration Computer technician

“laboratory” Clinical skills for nurses

and veterinarians Air traffic controller’s

communication skills test

Additional Advantages of Performance-Based Exams

A performance-based exam provides an opportunity to exchange safety information as well as assess individual knowledge and skill.

In the long run, this assessment process has the potential to prevent hazardous situations at the work site.

Conclusion and Suggestions

Develop and offer two testing formats: a multiple-choice exam, which may be the desired

format for people who have attended school in the U.S. and are accustomed to this testing format

a performance-based exam that would enable test takers from various educational backgrounds with diverse language and test taking skills equal opportunity to demonstrate their knowledge on the safe and effective use of pesticides