Embed Size (px)
Citation preview
Linda McKee, Chief Operations OfficerTX Teachers
October 24, 2019
1
Agenda Welcome, Introductions, and Agenda AAQEP Updates and Overview Check-In, Objectives, and the Action Plan AAQEP Standards and Data Expectations Data Quality Concepts Validity, reliability, fairness & trustworthiness
Enacting your Plan Objectives and Check-out
3
AAQEP’s Start: Renewing a Conversation
43 individuals: 34 faculty, deans, coordinators 2 PK-12 educators 4 reps from 3 SEAs (HI, MI, UT) 3 AAQEP initial staff 39 institutions in 14 states
Working in three groups: Expectations, Process, Consistency
2017
4
AAQEP Today: 90+ members, 9 site visitsMembers in 16 states,Guam, & Canada
Diverse membership:• Initial• Advanced• Traditional• Alternative • Online• Large, medium, &
small• Public• Independent• Proprietary• Community
colleges5
Principles Supporting the Conversation
6
Provider collaboration• AAQEP values and provides collaboration
for educator preparation to move forward in multiple ways through the accreditation process.
Improvement-focused with innovation friendly protocols
• AAQEP recognizes that innovation is integral to the continuous improvement cycle.
Philosophy of partnership• AAQEP encourages partnerships involving
providers, state authorities, and the accreditor.
Principles Supporting the Conversation
7
Comprehensive–all providers, all programs
• AAQEP created a comprehensive system that is open to all providers and all types of programs with the same quality expectations.
Consistency • AAQEP promotes consistency and calibration of all reviews and decisions
Efficiency and frugality in operations
• AAQEP maximizes efficiency and strives for frugality while maintaining quality.
Context and mission • AAQEP recognizes the importance of context and respects institutional mission.
Association Vision and
Mission
VISIONExcellent, effective, innovative educator preparation that is committed to evidence-based improvement, engages with the P-20 system, and holds high public confidence.
MISSIONTo promote and recognize quality educator preparation that strengthensthe education system’s ability to serve all students, schools, and communities.
What questions do you have about AAQEP?
Parking Lot
10
Check-In: Our Baseline(1) We know we need to work on this and could use some support
(2) We have made a good start but want to learn more
(3) This is well underway but we need to engage and support colleagues
(4) I could teach this (part of the) workshop; take me on the road!
Plans for establishing validityPlans for establishing reliabilityPlans for establishing fairness and trustworthinessProcesses for managing data qualityAction plans for establishing data quality in my program
Design a plan for ensuring that:
the instruments used allow you to make valid inferences
all who participate in the assessment of performance in clinical practice have a shared understanding of the key concepts and practices, and
all who participate in the assessment of performance in clinical practice recognize and evaluate them consistently
opportunities for innovation and improvement are identified
Today’s Objectives
12
Action Plan
13
Action Plan
14
1. Clearly identify priorities—it’s better to have one place to start. The big picture needs to be painted, but knowing where to start and where to continue moves things forward.
2. Clearly identify partners—knowing who will lead and who will be on the team is essential. And share the plan!
3. Engage local stakeholders in data quality work and in using data to find opportunities for innovation and improvement.
Action Plan Considerations
15
Don’t do things that don’t make sense.
Standards and Expectations Dimensions
Essential Questions for Standards 1 & 2
1. Program completers perform as professional educators with the capacity to support success for all learners.
At the end of the program, are completers ready to fill their target professional role effectively?
2. Program completers adapt to working in a variety of contexts and grow as professionals.
Were completers prepared to work in diverse contexts, have they done so successfully, and are they growing as professionals?
Confidence that completers perform as effective educators and continue to grow
20
Standard 1: Candidate/Completer Performance
Program completers perform as professional educators with the capacity to support success for all learners.
21
Standard 1 Aspects
Content/pedagogical/professional knowledge (relevant to license)
Learners / learning theory, including SEL Cultural competence Assessment and data literacy Positive learning/work environment Professional dispositions/behaviors
22
Candidate/Completer Performance
Standard 1 Evidence Evidence for this standard will include: multiple measures multiple perspectives (including program faculty,
P-12 partners, program completers, graduates’ employers)
direct measures evidence of performance in a field/clinical
setting appropriate to the program
Does the evidence address the essential question:At the end of the program, are completers ready to fill their target professional role effectively?
23
Multiple Settings/Times
Multiple Measures
Multiple Perspectives
Standard 2: Completer Professional Competence and Growth
Program completers adapt to working in a variety of contexts
and grow as professionals.
24
Standard 2 Aspects
Engage local school and cultural community, caregivers, and families
Culturally responsive practice with diverse learners Can develop productive learning environments in diverse
contexts Support increasing global perspectives Grow professionally Collaborate for professional learning
25
Completer Professional Competence and Growth
Standard 2 Evidence Evidence for this standard will show that program completers:
have engaged successfully in relevant professional practice
are equipped with strategies and reflective habits that enable them to serve effectively in a variety of educational settings appropriate to the credential or degree sought
have, in fact, done so—using whatever evidence can be gathered
Does the evidence address the essential question:Were completers prepared to work in diverse contexts, have they done so successfully, and are they growing as professionals?
26
Data from Preparation
Data from Practice
Confidence that quality will be sustained, enhanced, improved
27
Essential Questions for Standards 3 & 4
3. The program has the capacity to ensure that its completers meet Standards 1 and 2.
At the end of the program, are completers ready to fill their target professional role effectively?
4. Program practices strengthen the P-20 education system in light of local needs and in keeping with the program’s mission.
Were completers prepared to work in diverse contexts, have they done so successfully, and are they growing as professionals?
Standard 3: Quality Program Practices
The program has the capacity to ensure that its completers meet
Standards 1 and 2.
29
Standard 3 Aspects
Coherent curriculum (part of Appendix C) Quality clinical experiences Stakeholder engagement (part of Appendix D) Admission/monitoring process linked to success (Appendix A) Internal quality assurance/continuous improvement
(Appendix D) Capacity for quality (Appendix C)
30
Quality Program Practices
Standard 3 EvidenceEvidence related to this standard will include documentation of programpractices and resources as well as the program’s rationale for its structure and operation.
Does the evidence address the essential question:Does the program have the capacity (internal & with partners) to ensure that completers are prepared and succeed professionally?
31
Program practices
Resources
Rationale for structure and
operation
Standard 4: Program Engagement in System Improvement
Program practices strengthen the P-20 education system in light of local needs and in keeping with
the program’s mission.
32
Standard 4 Aspects
Engages stakeholders to support schools and reduce disparities Supports diverse educator workforce and addresses state and
local needs Supports completer career entry and growth (Appendix B) Uses available evidence on completers for program improvement Meets relevant regulatory requirements (general, and Appendix E) Investigates effectiveness vis-à-vis institutional mission (Appendix F)
33
Program Engagement in System Improvement
Standard 4 EvidenceEvidence for this standard addresses the identified issues in light of local and institutional context.
Does the evidence address the essential question:Do program practices strengthen the P-20 education system in light of local needs and in keeping with the program’s mission?
34
Engages with stakeholders
Supports diverse workforce
Is informed by completer data
Evidence Requirements and Priorities:
Evidence is brought forward in
the proposal
and Quality
Assurance Report
Multiple measures with reasonable continuity
Quality of evidence must be investigated and shared
Priority is given to direct performance measures
Down-stream data must be considered for improvement purposes
Differentiate evidence by license, level, location, mode of delivery
Assessments appropriate to program (not uniform across programs)
Contents Overview of the provider and its context Identification of assessments linked to aspects of Standards 1 & 2 Explanation of how validity, reliability, fairness, and trustworthiness has/will
be established Description of innovations/ contextual challenges
How is the proposal reviewed and what role does it play? Peer reviewers provide 1 or 2 rounds of feedback AAQEP reviews final proposal for completeness Proposal and summary of feedback becomes part of case record
Data Quality in the Proposal
36
Evidence used to make the case for accreditation must be: Valid—How have/might (content) validity be established? Reliable and Reliably Collected—How are raters calibrated? Fair & Address Bias—Do measures work well for all candidates? Trustworthy— For qualitative data, how is analysis done and how
is consistency assured?
Evidence regarding assessments should be appropriate to type
*Quality Assurance Report (the self-study document)
Data Quality in the QAR* - Appendix G
What are your questions?
Parking Lot Revisited
validity, reliability, fairness, trustworthiness…and the role of the action plan
Validity (have your action plan handy!)
Does an instrument actually measure what it says it measures?
Is the evidence produced by the measure useful?
Who needs to be engaged in establishing validity, and what process will you use?
Validity: What’s the Point?
41
Face validity: whether a measure “looks valid” to those who use it or complete it.
Content validity: refers to the extent to which questions/tasks represent important aspects of the intended content
Criterion validity: refers to the extent to which a measure’s results correlate with other variables that measure intended outcomes (concurrent or predictive)
Convergent/Divergent: is seen when logically related measures correlate positively, (or where measures of opposite constructs are negatively correlated.)
Types of Validity: Ways of looking at measures
Approaches to investigating validity Locally established methods developed by the provider with
a clear rationale and clear procedure. Research-based methods, such as the Lawshe, can provide
a useful framework for evaluating items on a measure. Similarly, the Angoff method has been used to establish
passing criteria or “cut scores” on some types of measures.
Content Validity: Art or Science (or magic)?
Method for gauging agreement among raters or judges regarding how essential items are.
Lawshe (1975): subject matter expert raters (SMEs) on the judging panel respond to the following question for each item: “Is the skill or knowledge measured by this item 'essential,' 'useful, but not essential,' or 'not necessary‘ to the performance of the construct?”
According to Lawshe, if more than half the panelists indicate* that an item is essential, that item has at least some content validity. Greater levels of content validity exist as larger numbers of panelists agree that a particular item is essential. Using these assumptions….
*Note the role of professional judgment in this formal process
Content Validity as Science
Lawshe developed a formula termed the content validity ratio: CVR = (ne-N/2)/(N/2)
where CVR = content validity ratio, ne = number of SME panelists indicating “essential” for an item, N = total number of SME panelists.
The formula yields values which range from +1 to -1; positive values indicate that at least half the SMEs rated the item as essential. The mean CVR across items may be used as an indicator of overall test content validity.
Content Validity as Science
Watch Video Elementary Classroom and complete observation rubric
Score each item based on the 3-point scale (essential, etc.)
Together, use our formula to determine the CVR for 5 items Which items need to be addressed based on the CVR scale? At your table have a conversation about these items
Whole group debrief
Hands On! Validity in Action
1. What might be valuable about this process?
2. In what way(s) might this process be limiting or problematic?
3. In what ways might you adapt or learn from this process?
4. Who would be key experts in content to bring to your table?
5. What program- versus provider– level work might need to be done?
To Lawshe or Not to Lawshe…that is the question
The Angoff score is the lowest pass/fail cutoff score that a minimally qualified candidate is likely to achieve on a test.
The score is calculated by using a panel of experts to determine the difficulty of each test question included in an assessment.
Angoff Score Method
How do you think about establishing validity of….
Locally developed measures (observation ratings, course assignment rubrics, surveys, interview protocols)?
State-mandated, I-didn’t-pick-this-or-ask-for-it…measures?
Measures developed by third parties that we are adopting?
How might different types of measures inform each other?
Different Treatments for Different Measures
Action Plan: Validity
Situational Validity (OK, I made that up)
51
Reliability
Reliability: Are Results Consistent…
53
•Test-retest reliability•when might that be useful?
Across time for individuals
•stability of results•how might looking at this be helpful?
Across time for successive cohorts
• internal consistency• split-half reliability; when might other statistical treatments
help?Across items
• interrater reliability• this might be where the biggest payoff comes!Across different scorers
•classroom, course level/program/degree level• this could help you know where a measure
works/doesn’t workAcross levels
Note: Consistency itself isn’t the aim; variation might tell you something.
Reliability: Perspective Matters
54
Perfect coincidence of views may not be the goal, but rather, productive comparative conversation
How do you think about establishing reliability of….
Locally developed measures (observation ratings, course assignment rubrics, surveys, interview protocols)?
State-mandated, I-didn’t-pick-this-or-ask-for-it…measures? Measures developed by third parties that we are adopting
(e.g. the CPAST student teaching evaluation, from Ohio)
How might different types of measures inform each other?
Different Treatments for Different Measures?
Using only standard 6 items on the observation form…
1. Watch the Secondary Classroom 2. Individually rate the observed teacher using only 3. Identify a partner at the table and compare your ratings4. On which criteria were your ratings the same? Different?
Hands-on: Reliability in Action
56
As a table group discuss videos and your results…
How have you investigated inter-rater reliability in your program(s)? How might viewing common videos help?
What might you do to increase inter-rater reliability in your program? Who would need to be involved?
Inter-Rater Reliability
Action Plan: Reliability
58
How about a break? See you in 10!
59
Fairness & Trustworthiness
60
A fair assessment (system) ensures all stakeholders are equally represented and protected.
An assessment (system) that’s fair will embody the values of equity, diversity, and integrity.
A fair assessment assesses what has been taught and reflects all performance fairly.
Fairness
61
Bias in assessment refers to the presence of systematic differences in the meaning of test scores associated with group membership.
It’s difficult to write an assessment without introducing your own bias into the questions in some way.
What groups of students are most impacted by fairness?
Fairness
4 Key Components:
Credibility Dependability Transferability Confirmability
See Establishing Trustworthiness in Qualitative Research
Trustworthiness
When considering fairness…
What words or phrases or might bea cause for concern?
…Keep in mind: context plays a role!
Popcorn Brainstorming!
At your table, exchange your own survey with your partner and review it for fairness and trustworthiness. Review in pairs/small groups and then discuss
How is the survey accessible to the user - both in terms of taking the assessment and also receiving understandable results?
How does the survey meet the needs of a diverse population? How will your team check the language in the survey for cultural bias?
Could you ask multiple stakeholders the same questions? What changes would be needed for whom, and why?
Hands-On: Fairness and Trustworthiness
Action Plan:
What will you attempt on your own campus?At your table discuss:
What’s the strongest aspect of your data quality work? What’s the biggest challenge you see ahead? What key allies are on board, need to be on board? What additional resources/information would help you? What are your priorities in investigating data quality?
Enacting Your Plan: What happens next?
68
Action Plan: Next Steps
A Review of Today’s Objectives
71
Design a plan for ensuring that:
the instruments used allow you to make valid inferences
all who participate in the assessment of performance in clinical practice have a shared understanding of the key concepts and practices, and
all who participate in the assessment of performance in clinical practice recognize and evaluate them consistently
opportunities for innovation and improvement are identified
What are your remaining questions?
Parking Lot
Check-Out: Our Progress(1) We know we need to work on this and could use some support
(2) We have made a good start but want to learn more
(3) This is well underway but we need to engage and support colleagues
(4) I could teach this (part of the) workshop; take me on the road!
Plans for establishing validityPlans for establishing reliabilityPlans for establishing fairness and trustworthinessProcesses for managing data qualityAction plans for establishing data quality in my program
A strong quality assurance system can assure accountability, support
innovation and improvement, and foster professional collaboration.
aaqep.org
Our Belief
74
Follow Us on Social Media
75
twitter.com/aaqep1
linkedin.com/company/aaqep
