Pioneering the Next Generation of Measurement: Developing West Virginia's Performance Assessments

Pioneering the Next Generation of

Measurement:Developing West

Virginia's Performance Assessments

WV CTE Global 21 Performance Assessments

Sandra Starr Vision and Its Background

Development Process

Nathan Taylor Writing Process

Vicki Jenkins Administration Experience

Ryan Haught Student Experience

Sandra Starr Problems and Next Steps

2

West Virginia Department of Education

The Vision and Its Background

Sandra Starr

“I realized we (CTE) were missing a lot of what we do by issuing the 50 question multiple-choice end-of-course exams.”

Dr. Stan HopkinsAssistant Superintendent

Division of Technical and Adult Education Services

4


CTE terms to know

CTE is Career and Technical Education.

Clusters organize both academic and occupational knowledge and skills into a coherent course sequence.

5


CTE terms to know continued

Concentrations are groups of at least 3 courses that when combined make up a program. The courses may be from a single CTE

content area or a combination of content areas which may be predefined or linear in nature.

Some CTE students build meaningful cohesive concentrations based on individual interests.

Completers are students who finish a minimum of three courses, including the core required courses.

6


CTE present

West Virginia has

1,198 CTE programs in 55 counties/districts

100,000+ students enrolled in CTE

43,000 students enrolled in CTE concentrations

7


CTE Global 21 Performance Assessments

West Virginia currently has

86 CTE concentrations72 CTE concentrations that assessed

completers

≈5,600 students who completed the performance assessments in 2010

8


CTE future

“21st century career and technical education is purposeful, personalized and performance-based, focusing on the numeracy, literacy, employability, and technical skills required for success in postsecondary education and careers.”

Dr. Stan HopkinsAssistant Superintendent


9


Carl Perkins Vocational and Technical Education Act of 1998

Incorporates secondary/postsecondary education elements

Includes coherent and rigorous content aligned with standards

Includes ways of earning postsecondary credits

Leads to credentials/certificates/degrees

Requires accountability for technical skills

10


Perkins grant continued

Levels of performance "shall, at a minimum--

(I) be expressed in a percentage or numerical form, so as to be objective, quantifiable, and measurable;

(II) and require the State to continually make progress toward improving the performance of vocational and technical education students." [sec. 113(b)(3)(A)(i)]

"Indicators of performance... shall be established solely by each eligible agency with input from eligible recipients." [sec. 113 (b)(2)(D)]

11


The Development Process

Sandra Starr

Blueprint and design

Researched available models and fields of thought.

Compartmentalized test into three areas: live performance (typically 3

interrelated tasks based on a scenario)

written componentoral component

13


Blueprint and design continued

Rubrics were standardized

Consulted with Dr. Gary Phillips, Vice President, American Institute for Research

14


Writing/Construction

Teachers wrote assessments with guidance from CTE Coordinators

Copy and paste with editing from teacher drafts AFTER alignment was ensured.Alignment based on Webb method.Consulted with Dr. Norm Webb,

University of Wisconsin, prior to process.

15


Writing/Construction continued

Time limit of 3 hours was determined for live performanceEach task carried a suggested

timing set by writers and/or state coordinatorsAllowed flexibility to move time

from task to task, as needed, but not to exceed three hour limit

16


Sample script excerpt17


Sample scenario

ET1980 Welding Technology

The local welding shop is accepting applications for entry level welders. First you must submit a resume and complete the interview process. Then, you will complete a hands-on welding test.

18


Sample problem set

Problem 1: Using the blueprints, layout the equipment. Problem 2: Develop a list of tools and materials needed to complete the assessment. This list of materials will be used to complete the rest of the examination, so accuracy is a MUST. Problem 3: Construct the items on a set of blueprints.

19


Writing/Construction continued

Written and oral components to be given prior or post live performance http://careertech.k12.wv.us/documents/WrittenComponentRubric.pdf

http://careertech.k12.wv.us/documents/OralComponentRubric.pdf

Trainings for each level of administration were created and housed on Internet to ensure defensibility of trainings

Test Administration Manual created with ancillary materials http://careertech.k12.wv.us/documentsTestAdministrationManual_R.pdf

20


Welding

21


Scoring/Rubrics

Standardized rubrics used to create a “Master Rubric” from which each coordinator could pull to create rubrics for each concentration

Due to the specificity of some concentrations, some criteria had to be customized

Customized criteria were used across all concentrations when appropriate

22


Master rubric

Content Area Knowledge

Critical Thinking & Problem Solving

Social/Personal Accountability

Oral Presentation Skills

Visual & Media Literacy

Work Readiness Applications

Written Project Support

23


Sample criterion24


Future revisions and changes

Use the field test from 2010 and look at the suggested revisions/corrections from the teachers and use that input to correct and improve each assessment

Use improved form as “OPERATIONAL” in 2011

Development of second form - must undergo “social moderation” to ensure comparability

25


The Writing ProcessNathan Taylor

Teacher selection

CTE Coordinators invited teachers that were specialists in the content areas being assessed.

Teachers who had completed the most recent Content Standards and Objectives (CSOs) revisions were invited first.

27


Planning

Dates for workshops were scheduled around convenient times for each CTE area based on CTSO activities.

28


Writing workshops

Training on depth of knowledgeTeachers were introduced to the depth

of knowledge method of alignment and participated in evaluation of materials to identify levels of cognitive complexity.

Teachers evaluated CSOs to identify appropriate DOK levels to be assessed.

Teachers created an overarching scenario and three problems within the scenario that aligned with the CSOs.

29


Writing workshops continued

Teachers produced a rough draft of each assessment, peer reviewed the drafts for content and bias, and made appropriate changes.

Coordinators reviewed the drafts (content and bias) and identified areas for corrections as needed.

Final drafts were submitted at end of each writing workshop.

30


Workshops by cluster

Agriculture, Science and Natural Resources

Coordinators—Jason Hughes, Nathan Taylor

Workshop held one time—December 2-4

30 Participants

31


Workshops by cluster continued

Engineering and Technical Education

Coordinators—Rick Gillman, Kathy Gillman, Kobie Coburn

2 workshops held

70 participants total

32



Business and Marketing Education

Coordinators—Abby Reynolds and Betty Sias

2 workshops


33



Health Science Education

Coordinators — Cynthia Sundstrom and Becky Davis

1 workshop

30 participants

34



Human Services

Coordinators — Kathy Mace, Susan Rice, Donna Wilkes

3 workshops


35


Follow ups

Follow ups were conducted by telephone and in person when possible when questions or conflicts arose during the test building process.

36


Editing and revision process

CTE Coordinators reviewed performance assessments for content accuracy.

Industry professionals reviewed randomly selected performance assessments for content validity, accuracy, and applicability to field.

37


Editing and revision process continued

CTE Coordinators and the Assessment Coordinator revised and edited to ensure conformity to an acceptable CTE format and assessment parameters.

Assessment Coordinator made final edits for bias, mechanics/spelling, and formatting.

38


The Test Administration

ExperienceVicki Jenkins

Completers

A student who completes the required four courses within their unique concentration

Teacher generated list/WVEIS record

Generate roster of completers for each concentration (16 at JRTI)

Double check roster with administrator

40


Preparing

Rational for test-Industry standard

Have teachers who conducted pilot provide testimony/provide handout for organizing testing

Encourage teachers to “talk it up” with students

41


Preparing continued

Set testing dates and contact & confirm dates with home schools (check for calendar conflicts)

Publish testing schedule-distribute to all parties

Announce in newsletters, put on Edline for parents, letter to parents

42


Pre-assessment responsibilities

Secure materials upon arrival (in locked location)

Copy assessment booklets, gather necessary number of pens/pencils/highlighters

Make student test packets

Prepare “do not disturb” signs

43


Pre-assessment responsibilities continued

Prepare class packets with all required information - manual, manipulatives, signs, pencils, etc.

Order thank you gifts and materials for judges

Order supplies/materials for each test

44


Trainings

Copy the Test Administration Manual for each faculty member participating in testing

Complete training for teachersReview IEP and 504 requirements

Sign agreements/maintain on file

45


Trainings continued

Emphasize security agreements, i.e. procedures, directions, rubric, reporting procedure breach of security

Inform examiners about checking for pertinent information

Review return procedures for all materials

46


Trainings continued

Review responsibilities for evaluators’ training, complete evaluator’s training and/or confirm with examiner that examiner training has been completed

Provide folder with assessment materials to each evaluator

Prepare cards containing the test irregularity site-distribute to examiners/evaluators

47


Post-assessment responsibilities

Secure materials after testing

Enter scores by required dateData Management Profile System

(operated by state)

Maintain hard copies for 3 yearsRubrics, answer sheets, etc.

(documents with recorded scores)

48


Post-assessment responsibilities continued

Ask examiners/evaluators to share concerns/comments for improvement

Destroy secure materials by shreddingCD’s created for assessmentPerformance assessmentsScratch paperProjects created for assessments

49


The Student Experience

Ryan Haught

Preparation of students

Focus on teaching the CSOs throughout each course during the school year and over a two-year period for two-year programs

51


Assessment scheduleAssessment Date Concentration # of Completers

May 3rd Computer Repair and Networking Technology

9

May 3rd Emergency Medical Preparedness

17

May 12th Automotive Technology 19

May 12th Building Construction 7

May 12th E-Business Publishing 8

May 12th Health Science Occupations

13

May 12th Welding Technology 11

May 13th Facility Maintenance 4

52


Students’ perspective

There were a few who reacted negatively, but the majority of students reacted positively to the process

In comparing rigor, most students believed the performance assessments were far more applicable in comparison to their experiences with paper-and-pencil exams

53


Students’ perspective continued

The experiences of the three problems were preferred over the experiences with the written and oral components of the performance assessment

In comparing the assessment of higher level thinking, the majority of students felt the “hands-on” portion of the exam was much better in comparison to paper-and-pencil assessments in “making them think.”

54


Students’ perspective continued

Some students expressed concern and frustration over having difficulty recalling skills they had learned during the first year of their program

The majority of students were eager to find out how they scored

55


And the students said . . .

56


Problems and Next StepsSandra Starr

Problems

Poor communication caused administrative and security breaches.

Testing window conflicted with end-of-year senior activities which affected participation.

Test writers assumed no changes would be made to their work; some returned the work to its original state before administering.

Writing process needs tweaking; editing process needs to revised.

58


Problems continued

Examiners who had taught CTE for several years tended to administer the test without the benefit of the provided administration directions.

Not all examiners had taught a concept in the manner in the manner in which it was tested.

Many participants were territorial in their approach to the “new test” which hurt the performance assessments.

59


Contact information

For more information about the WV CTE Global 21 Performance Assessments, please contact:

Dr. Kathy D’Antoni

Assistant State Superintendent of Schools


304-558-2346

[email protected]

60


Presenters’ contact information

Ryan Haught

[email protected]

Vicki Jenkins

[email protected]

Nathan Taylor

[email protected]

Sandra Starr

[email protected]

61


Documents

Pioneering the Next Generation of Measurement: Developing West Virginia's Performance Assessments