Embed Size (px)
Citation preview
#pearsoncite
Let’s Launch a CBE Program: An A-Z Workshop
#pearsoncite
Paul Bowers
Strategic Consultant
Pearson
@pebowers1
Joan Bouillon, PhD
Director Regulatory Compliance
Pearson
Barbara Rowan, PhD
Director of Academic Research, Efficacy, and Evaluation
Pearson
Kate Kazin, J.D., PhD
Chief Academic Officer
Southern New Hampshire University
#pearsoncite
Mary Adkison
Academic Strategist, Institutional Efficacy
Pearson
Elizabeth Adkins
Director of Instructional Design
Pearson
Russ Lewinter
Manager, Product Development
Pearson
#pearsoncite
Workshop Agenda
1. Getting Started- WHY CBE and Key Realities
2 Designing CBE Programs
3. Institutional Supports for CBE
4. Leading Change and Engaging Your Campus
#pearsoncite
Workshop Goals
• Engage Planning for CBE
• Identify Key Drivers and Purposes
• Focus on Essential Elements and Key Decisions
• Simplify Many Tentacles of CBE
• Develop Strategies to Engage Others
#pearsoncite
Useful Frameworks
• CBE Playbook
• C-BEN Design Elements for CBE
• C-RAC Accreditation Criteria
#pearsoncite
CBE PlaybookMaintaining coordination across 7 key workstreams
Summary
Our framework for the CBE Playbook – it provides the 7 key work streams necessary for CBE strategy.
Note on use
The CBE work streams are not sequential. Rather, they run in parallel with specific activities occurring within in each stream throughout the program development phase.
1. CBE Strategy & Integration 5.
Technology & Infrastructure
4. Student Success
2. Organization
3. Program
Development
6. Program
Management
7. Marketing &
Recruiting
#pearsoncite
Key functions across workstreams
Strategy & integration
Strategy development
Project management
Metrics & Accountability
Executive leadership
Financial modeling
External relations
Accreditation & approvals
Change managementO
rg.
Faculty Development
Curriculum D&D
Delivery Model D&D
Assessment D&DD
ev.
Orientation & Readiness
Retention strategy
Service Processes
PLA & Degree pathways
Succ
ess
Learning infrastructure
Reg & Assessment mgmt
Data & learning analytics
Retention & support solutions
Tech
Resource planning
StaffingVendor
relationsPartnerships
Tracking & reportingM
gmt
Marketing strategy
Lead genFunnel
managementRegistration &
advisingEM
#pearsoncite
C-BEN Program Design
Elements
How strongly do CBE programs agree the
following design elements are needed for a robust and healthy program?
#pearsoncite
C-RAC Accreditation Criteria
#pearsoncite
Why CBE?What Is CBE?
#pearsoncite
Thinking about Learning
Pick a time you learned something new.
Complete the handout.
#pearsoncite
In other words, CBE:
• Focuses on what students can do with what they know
• Assesses learning directly rather than relying on proxies (e.g., credit hours and seat time)*
• Allows multiple attempts to acquire mastery
• Does not define how the learning should be achieved
*But may require “translation” into credit hours for administrative or regulatory purposes
#pearsoncite
Why CBE?
• It’s the latest fad in higher education
• Your president or provost said you needed aCBE program
• Your IT team has too much time and too many resources
✗✗
#pearsoncite
The conventional transcript is a black box
=
#pearsoncite
Advantages for programs and institutions
• Clear evidence of what students know and can do
• Insights into student progress
• Easier alignment with industry – employers “speak” competency
#pearsoncite
Advantages for students:
• Flexibility (schedule and pace)• Transparency (progress and outcomes)• Lower cost (potentially)• Greater individual support for learning (usually)• Removal of irrelevant barriers (like time)• Greater relevance, especially for adult learners
#pearsoncite
Discussion Exercise:Tell us about YOUR Interests in CBE
• What is driving YOUR Institution/Thinking?
• What goals does a CBE program help you achieve?
• What are you most unsure about regarding CBE?
#pearsoncite
Key Realities:Accreditation and Financial Aid Eligibility
#pearsoncite
Compliance Agencies
• State agency approval
• Programmatic accreditor
• Licensing board
• Institutional accreditor
• Department of Education
#pearsoncite
Accreditation Requirements
• Competencies are externally referenced• Outcomes emphasize performance, not just
knowledge• Regular and substantive interaction with faculty• Competencies reflect achievement associated
with the degree level
#pearsoncite
• Good practices in assessment and measurement
• Validate quality through feedback
• Credit hour equivalency method
• Admission requirements and screening
• New student orientation
• Learning activities including externships
• Advanced standing policy
Accreditation Requirements
#pearsoncite
• Faculty roles and qualifications
• Grading policy – pass/fail, points, other
• Satisfactory academic progress
• Cancellation and refund policies
• Transcripts and student information system (SIS)
• Transfer and transition to traditional delivery plan
Accreditation Requirements
#pearsoncite
DOE Requirements
• “an institution must establish a methodology to reasonably equate the direct assessment program…to credit or clock hours”
• “a full-time student is expected to complete the equivalent of at least 24 semester or trimester credit hours…900 clock hours” in a minimum of 30 weeks
#pearsoncite
DOE Requirements
• “…process to document a student’s Enrollment Status”, i.e. full time, etc.
• In “any seven-day period…at least one day of educational activity occurs… educational activity in a direct assessment program includes regularly scheduled learning sessions”
#pearsoncite
DOE Requirements• Satisfactory Academic Progress (SAP) policy “must include
procedures to determine whether a student has completed sufficient competencies to complete the program within the maximum time frame…” and be “…based on the number of competencies associated with a specified number of credit hours, clock hours, or equivalencies…”
#pearsoncite
Discussion Exercise
• The follow slide has THREE accreditation issues you will need to address at some point
• We will divide into three small groups. • Each groups takes one question• Take 5-7 minutes to discuss how you would likely
respond• Report out to the whole what you concluded
#pearsoncite
Accreditation Application Items
1. What are the procedures to transition a CBE student to the institution’s traditional delivery program.
2. Describe the refund policies for the CBE program.
3. Provide policies on advanced standing, PLA, experiential learning, and/or transfer of credit for CBE programs.
#pearsoncite
Designing CBE Programs: Key Decisions
#pearsoncite
Exercise: Decisions, decisions
• Who are your stakeholders?– Internal – External (e.g., employers)
• What is your goal/purpose in offering CBE?• What’s the scope of your CBE initiative?
– Single course? (not recommended)– Program?– Institution or system?
• Credit-based or direct assessment?
@cathrael_kazin
#pearsoncite
Two options
• Start from scratch– Build new CBE from the
bottom up
• Retrofit – Try to “competency-ize”
existing programs”
@cathrael_kazin
#pearsoncite
Once that’s settled…
• You need a set of claimsabout what your graduates know and are able to do
• These are your competency statements
#pearsoncite
What makes a good competency statement?@cathrael_kazin
Measurable and
observable
Clear
Significant
Can stand alone
Reflects doing, not
just knowing
Has external meaning
#pearsoncite
Exercise: Evaluating Competency statements
Graduates of CITE University will be able to think critically
Understands the concept of supply and demand
Can analyze a problem and recommend solutions in writing
Is able to solve quadratic equations
Can name the three types of receptors in the human body
Conveys information by creating charts and graphs
#pearsoncite
Assessing Competencies: the Heart of CBE Programs
#pearsoncite
Assessment
• Driving force behind a strong CBE program
• Two-step Process– Assessment development and score validation
– Determining cut scores
#pearsoncite
Assessment Development
StandardsPsychometric
EvaluationValidity
#pearsoncite
Assessment Best Practices
• Each competency must be measured more than one time
• Each competency must be measured more than one way
• Assessments should focus on clusters of competencies as they might appear in real world
#pearsoncite
Assessment Types
Performance-based Assessment (real-world
focus)
Objective Measures of
Competencies
Diagnostic Tests of Knowledge
@BarbaraRowan
#pearsoncite
Assessments
• Assessment maps to competencies
• Assessment measures a student’s knowledge and skills AND how well the student can integrate, synthesize, and use the knowledge and skills necessary to become a community of practice
#pearsoncite
Assessments
The more “real world” the assessment is, the more value the results hold for determining a student’s mastery of the competencies.
#pearsoncite
Psychometric Evaluation
• Reliability – indicates consistency of test scores
• Item Analysis – indicates items that are performing poorly
• Rubric Evaluation – to ensure that the rubric is properly evaluating subjective measure
@BarbaraRowan
#pearsoncite
Score Validation
Test Content
Relations to Other
Variables
Response Processes
Test Consequences
Internal Test Structure
@BarbaraRowan
#pearsoncite
Framework for Determining Mastery
• Must determine what separates masters from non-masters
• Validity evidence required to support decisions
@BarbaraRowan
#pearsoncite
Cut Scores – Questions to Ask
What do Students Need to Know and Be Able to Do at Each Level?
What Performance Levels will be
Reported?
Are Cut Scores Necessary?
@BarbaraRowan
#pearsoncite
Cut Score Options
@BarbaraRowan
#pearsoncite
Five Basic Steps
Select Judges
Define borderline knowledge and skills
Train the judges
Collect judgments
Choose a cut score
by combining judgments
@BarbaraRowan
#pearsoncite
Choosing a Method
• No one best method
• Choice should depend on what kind of judgments can be gotten and believed
• Based on types of questions in the test (multiple choice; constructed-response; rubric-based
@BarbaraRowan
#pearsoncite
NAEP Suggested Levels
Basic Level
• Partial Mastery of knowledge and skills
Proficient Level
• Solid Academic Performance and Demonstrated Competence over Challenging Subject Matter
Advanced Level
• Signifies Superior Performance on Challenging Subject Matter
@BarbaraRowan
#pearsoncite
Cut Score Process
@BarbaraRowan
Set Provisional Cut Scores
Establish Operational Cut Scores
Document the ProcessEvaluate the Results of
Using the Cut Scores
#pearsoncite
Provisional Cut Scores
• Determined by people who are aware of what students should know and be able to do
• Determine how many score points students have to earn to demonstrate they have reached the level of knowledge and skill for each level
@BarbaraRowan
#pearsoncite
Provisional Cut Scores
• Choose Cut Score Method
Step 1
• Select Judges, that is, 5-15 subject matter experts
Step 2• Train Judges in
the Cut Score Method Chosen
Step 3
@BarbaraRowan
#pearsoncite
Operational Cut Scores
@BarbaraRowan
Is it better to pass a student who should fail?
Or fail a student who should pass?
#pearsoncite
Operational Cut Scores
Test Score Fluctuations
Pass
Fail
@BarbaraRowan
#pearsoncite
Document the Process
@BarbaraRowanRationale
Qualifications
Training
Steps
Data shared with Judges
Judges’ Ratings
Evaluation Forms from
Judges
Rationale for Adjustments
#pearsoncite
Evaluating the Results of Using Cut Scores
• Occurs after at least one round of testing and reporting of results has occurred
• Requires the validation of cut scores
• Ongoing process
@BarbaraRowan
#pearsoncite
Validity Study
• Are the results reasonable?
• Are failure rates far higher or lower than expected?
• Are outstanding students failing to reach the highest level?
• Are poorly prepared students reaching the highest level?
@BarbaraRowan
#pearsoncite
Discussion
• What types of assessments do you envision for YOUR CBE program?
• How would you envision addressing the development of assessments – who should be involved?
#pearsoncite
Developing The Institutional Support Model
#pearsoncite
Three Program Models
#pearsoncite
ProgramManagement
#pearsoncite
Program Model
#pearsoncite
Student Support Model
College for America
Academic planSelf directed student progression through competencies
Source: Adapted from EDUCAUSE Next Gen Grant Recipient Profile
CoachSet and monitor
goals
Online networksPeer support
Learning PartnerSupport and encourage
MentorEmployer /
community member
#pearsoncite
Kentucky Virtual Technical College Model
Source: Adapted from EDUCAUSE Next Gen Grant Recipient Profile
Pre-assessment Activity Post-assessment
Learn on demand course design: pre-assessment allows path to test out
FacultyLead each
course, provide
feedback
Student support
staffIntensive support
#pearsoncite
Texas Affordable Baccalaureate Program
Source: Adapted from EDUCAUSE Next Gen Grant Recipient Profile
Content ExpertsDevelop content
materials
CoachesAssigned
enrollment to graduation
TutorsAssigned to
content domains
FacultyFace-to-face &
online
Lower-division coursework Upper-division coursework
Oversight teamFaculty & Business leaders develop competencies
#pearsoncite
Exercise: Evaluating Competency statements
Graduates of CITE University will be able to think critically
Understands the concept of supply and demand
Can analyze a problem and recommend solutions in writing
Is able to solve quadratic equations
Can name the three types of receptors in the human body
Conveys information by creating charts and graphs
#pearsoncite
Key Questions
• What is the role of faculty?
• What is the role of staff?
• How are activities and information shared between the two?
#pearsoncite
Discussion Exercise
• In Small Groups:– Identify KEY Constraints for Support Model and Roles
– What would be the Key Characteristics of YOUR Support Model?
#pearsoncite
Data Informed Learning Design
#pearsoncite
Data Informed
Learning Design
DIGITAL LEARNING ENVIRONMENT
INFORMED DECISIONS
DATA
Pearson Digital Footprint
INFORMED DECISIONS
PREDICTIVE ANALYTICS
Enterprise-level: Programs and courses across the entire institution
Program-level:Departments and/or Programs (e.g. CBE)
Course-level:Course(s) taught by
individual instructors
Student-Level: Student
behavior & performance
Data Informed Learning Design
INFORMED DECISIONS
Analysis
Strategy + Action
Measurement
Data Modeling
& Predictive
Alerts
Learning Design
Professional Development
Launch
Monitor
Measure
Iterate
Actionable Data Dashboard
INFORMED DECISIONS
PREDICTIVE ANALYTICS
#pearsoncite
Discussion Exercise
• What are the MOST important things you would like to know about your learners?
#pearsoncite
Conversation Roundtables
#pearsoncite
Closing Thoughts:Change Leadership Strategies
#pearsoncite
Change Leadership
• Getting stakeholders to buy into
change
• Ensuring the process stays under
control
• Minimizing risk
• Keeping the initiative on budget
• Articulates a vision for the
future
• Mobilizes and coordinates
resources necessary for the
change
• Creates engines for accelerating
innovation
Change LeadershipChange Management
#pearsoncite
Change Leadership Strategies •Involve key stakeholders as early as possible
•Create faculty innovation models
•Clearly articulate goals and outcomes -WHY CBE?
•Empower key stakeholders through distributed problem-solving and
iterative design
•Engage employers and other external stakeholders in meaningful ways
•Understand key constraints and limits–Which can you overcome and which do you need to live with?
#pearsoncite
Change Leadership Examples
• U Texas Center for Transformational Learning
• Purdue IMPACT Program
• Edmonds Community College
#pearsoncite
Final Question
• What are your greatest concerns about change?
• What will YOU do to lead change on your campus?
