Validity, reliabiltiy and alignment to determine the effectiveness of assessment

VALIDITY, RELIABILTIY AND ALIGNMENT TO DETERMINE THE

EFFECTIVENESS OF ASSESSMENT

ASSESSMENT

VALIDITY RELIABILITY

STRATEGIES STANDARDS

ALIGNMENT

ASSESSMENT Assessment is the process of gathering and discussing information from multiple and diverse sources in order to develop a deep understanding of what students know, understand, and can do with their knowledge as a result of their educational experiences. (Huba and Freed,2000)

VALIDITY- refers to what the assessment is actually testing.

RELIABILITY- refers to the consistency of the

assessment.

ALIGNMENT- refers to the connection between learning objectives, learning activities and assessment

A. VALIDITYDefinitions:Validity - refers to what the assessment is actually testing - refers to how accurately a conclusion, measurement or concept corresponds to what is being tested - defined as the extent to which an assessment accurately measures what it is intended to measure.

Factors Affecting Validity:1. Student’s reading ability - Educators should

ensure that an assessment is at the correct reading level of the students.

2. Student self-efficacy - If students have low self-efficacy or beliefs about their abilities in the particular area they are being tested in, they will typically perform lower.

3. Student test anxiety level - Students with high test anxiety will underperform due to emotional and physiological factors

Types of Validity:1. Face Validity - ascertains that the measure appears to be

assessing the intended construct under study. 2. Construct Validity - is used to ensure that the measure is

actually measure what it is intended to measure and not other variables

3. Criterion-Related Validity - is used to predict future or current performance - it correlates test results with another criterion of interest.

4. Formative Validity - when applied to outcomes assessment it is used to assess how well a measure is able to provide information to help improve the program under study.

5. Sampling Validity - ensures that the measure covers the broad range of areas within the concept under study

Kinds of Validity Claim:

1. Predictive evidence - form of construct validation that examines whether performance on an assessment is strongly related to real-world success in the domain that the assessment is meant to reflect.

2. Consequential evidence - refers to the kinds of consequences an assessment and its uses have for learners and for instruction.

Why is validity necessary?While reliability is necessary, it alone is not sufficient. For a test to be reliable, it also needs to be valid

Ways on how to improve validity:1. Make sure your goals and objectives are clearly defined and operationalized. 2.Match your assessment measure to your goals and objectives. 3.Get students involved; have the students look over the assessment for troublesome wording, or other difficulties. 4. If possible, compare your measure with other measures, or data that may be available.

B. RELIABILITYDefinitions:

Reliability - refers to the consistency of the assessment

- the degree to which an assessment tool produces stable and consistent results

Factors Affecting Reliability:

1. The length of the assessment 2. The suitability of the questions or tasks for the students being assessed.3. The phrasing and terminology of the questions.4. The consistency in test administration 5. The design of the marking schedule and moderation of marking procedures.6. The readiness of students for the assessment

Types of Reliability:1. Test-retest reliability - is a measure of reliability

obtained by administering the same test twice over a period of time to a group of individuals.

2. Parallel forms reliability - is a measure of reliability obtained by administering different versions of an assessment tool to the same group of individuals

3. Inter-rater reliability - is a measure of reliability used to assess the degree to which different judges or raters agree in their assessment decisions

4. Internal consistency reliability - is a measure of reliability used to evaluate the degree to which different test items that probe the same construct produce similar results.

a.) Average inter-item correlation - It is obtained by taking all of the items on a test that probe the same construct determining the correlation coefficient for each pair of items, and finally taking the average of all of these correlation coefficients

b.) Split-half reliability – The process of obtaining this begun by “splitting in half” all items of a test that are intended to probe the same area of knowledge in order to form two “sets” of items.

C. ALIGNMENTDefinition: It is the connection between

learning objectives, learning activities and assessment

OBJECTIVES ACTIVITIES ASSESSMENT

3 Components of Alignment:1. Objectives - What do I want students to know

how to do when they leave this course?2. Activities - What kinds of activities in and out

of class will reinforce my learning objectives and prepare students for assessments?

3. Assessment - What kinds of tasks will reveal whether students have achieved the learning objectives I have identified?

Why is alignment important? Proper alignment keeps you going in the

right direction. If assessments are misaligned with objectives or strategies, it can undermine both student motivation and learning.

HOW CAN WE DEVELOP AN EFECTIVE ASSESSMENT?

StrategiesStandardsAlignment

STRATEGIES FOR DEVELOPING AN EFFECTIVE ASSESSMENT

1. Remember the effect of assessment on student learning behaviors and outcomes.2. Align the course assessment with the learning outcomes and curriculum.3. Prepare students for assessment by providing formative tasks and explaining the structure of the assessment for their course.

STRATEGIES…4. Design quality assessment tasks and items.5. Review Assessment Data.6. Understand how to set standards and grade cut-offs.7. Give feedback to students.

STANDARDS1. Assessment of Higher-Order

Cognitive Skills2. High-Fidelity Assessment of Critical

Abilities3. Assessments that Are Internationally

Benchmarked4. Use of Items that Are Instructionally

Sensitive and Educationally Valuable

1. Assessment of Higher-Order Cognitive Skills

One widely used approach in conceptualizing knowledge and skills represented in curriculum, teaching, and assessment Webb’s Depth of Knowledge (DOK) taxonomy.

Using the DOK framework as a guide, if assessments are to reflect and encourage transferable abilities, a substantial majority of the items and tasks (at least two-thirds) should tap conceptual knowledge and abilities.

Webb’s Depth of Knowledge (DOK) taxonomy

2. High-Fidelity Assessment of Critical Abilities

This standard identifies a number of areas of knowledge and skills that are clearly so critical for college and career readiness that they should be targeted for inclusion in new assessment systems. As described in the standard, these includes: Research Analysis and Synthesis of Information Experimentation and Evaluation Communication in Oral, Written, Graphic, and Multi-Media

Forms Collaboration and Interpersonal Interaction Modeling, Design, and Complex Problem Solving

3. Assessments that Are Internationally Benchmarked

The assessments should be as rigorous as those of the leading education countries, in terms of the kind of content and tasks they present, as well as the level of performance they expect.

Such assessments sought were, in order of importance: 1) teamwork2) problem solving3) interpersonal skills4) oral communications5) listening skills

4. Use of Items that Are Instructionally Sensitive and

Educationally Valuable

Assessment tasks should also be instructionally sensitive and educationally useful. That is, they should 1) represent the curriculum content in ways that respond to instruction, and 2) have value for guiding and informing teaching.