Embed Size (px)
Citation preview
USING Tk20 SOFTWARE TO OPTIMIZE OUTCOMES ASSESSMENT:
A CASE STUDY
John W. Rogers Ph. D.VP Institutional Effectiveness
American International College
• Founded in 1885• Small Private, Non Profit College• 3500 students: half UG/half graduate• 40 UG programs and 15 graduate programs:
include Nursing, Graduate Business, and Masters programs in a variety of areas
• Masters and Doctorates in Education are offered in US and international satellite locations
NEASC 10 Year Review
January, 2010: Monitor institution’s progress:• Improvement in financial health;• “establishing a campus-wide assessment
program focused on enhancing institutional effectiveness, especially with regard to assessment of student learning outcomes;”
• Faculty development and academic program improvement;
• Student information literacy
The Challenge
• Centralized data collection vs. decentralized data management– Each program needs to take ownership of its outcomes
assessment plan– Campus wide responsibility for aggregating data
• Continuous improvement and evidence based resource allocation
• Accrediting bodies need to see a coherent story of institutional quality and performance management
Criteria for Software Selection• Hosting in-house
– Cash flow advantage– Flexibility in application – Portfolio durability – Alumni relations
• Usability: common user interface• Survey function is fully integrated with system and is
confidential • In-house hosting process: we build server, we design
the system, integrate with SIS, vendor sets it up and provides maintenance
Implementation Planning
• Train the trainer model• Select early adapters• Communicate to the entire community:
Assessment Day • Involve non-academic departments:
– Administration – Finance, Business Office, Financial Aid, Admissions
– Student Activities – Residence Life, Athletics, Security, Career Services
The Assessment Process
• The Mission of the College• Learning Objectives and Broad based Goals• Define program responsibilities and a hierarchy
of authority• Program Learning Objectives related to artifacts
for measurable results• Map artifacts to the program curriculum• Gather data and aggregate it for continuous
improvement
Tk20 Applications
• Manage assessment plans: learning objectives, artifacts, curriculum map, and streams of evidence
• Electronic course evaluations• Practica and internships • Portfolios for assessment and career planning• Faculty artifacts and development• Student advising• Surveys of faculty, students, and staff• Assess “non-academic” programming • Strategic Plan Dashboards
Criteria/Ratings Unacceptable Acceptable Good ExcellentOrganization No overall principle of
organization; paragraphs random in order, with no clear topics.
Some organization evident. Most paragraphs are properly formed. No effort to argue a particular view is made.
Most parts are well-organized, but some deficiencies noted. Effort to present a strong argument is present but not necessarily successful.
Well-organized at every level, with evident effort to make a strong argument.
Mechanics Typos and misspellings common, poor grammar, and inconsistent punctuation.
Few typos or misspellings, grammar correct for the most part.
Typos or misspellings rare, correct grammar throughout, with careful proofreading evident.
Correct in every way.
Sophistication Vocabulary, sentence construction, and overall language use are simple, with no effort at college-level vocabulary or usage. Failure to identify areas that need explication.
Vocabulary typical of early college student, sentence construction goes beyond basics, and some attempt at persuasion or explication of complex material is evident.
Vocabulary and sentence construction is advanced and varied, with competent explanation of complicated topics and successful persuasion.
Vocabulary and sentences are advanced and nuanced, demonstrate deep understanding of complex ideas and topics, and persuasive language is compelling.
Integration No effort made to connect ideas and concepts from lectures, texts, or labs with the topic of the paper. No effort to seek out alternative views.
Ideas from lectures and labs are sometimes used to shed light on the topic, but no effort made to explore alternative views.
Ideas from lectures and labs are explored in appropriate contexts, alternative views are presented, and multiple source types are explored.
Ideas from labs and lectures are used well and expanded upon completely. Alternative views are explored fully. Multiple sources are compared and judged.
References No in-text references, web sites are used as a source of information, few if any sources primary.
In-text references are appropriate, web sites are not referenced, but bibliography is sparse and references include secondary sources.
In-text citations appropriate and unambiguous, no web sites or secondary references used, but not much evident effort to find a wide variety of primary sources.
Perfect use of in-text citations, no web sites referenced, no secondary references used, references are all primary and a wide variety of sources is used.
Level of Mastery
Categories
Institutional Effectiveness
CourseEvaluations
Results
Course and Program
Improvement
Electronic Course Evaluations• Replace paper and pencil/Scantron system• Give real time feedback on instructor performance• Timely aggregation and reporting to deans and program directors
Survey Design
Aggregate Results by Instructor
Longitudinal Information
Longitudinal Information
E-Portfolio
• Critical aspects of assessment software• Already required in physical or limited
electronic form by program specific accreditors
• Electronic Portfolio system becomes easy and comprehensive method of assessment; assessment is “baked in”
Analogy of the Map
The curriculum map allows an institution, college, or program to demonstrate:
• where you are going (goals/outcomes to be met)
• how you are getting there (skills taught and assessments used).
Analogy of the Map
Where Are You Going?
Huba and Freed. Learner-Centered Assessment on College Campuses: Shifting the Focus from Teaching to Learning. Needham Heights: Allyn & Bacon, 2000.
What are transition points?
Key points in a program when a unit assesses
candidate knowledge, skills, and professional
dispositions to determine if candidates are
ready to proceed to the next stage in a
program.
Defining Transition Points
• Should be defined per program
• Minimum of 4 Stages
Program
Stage 1 Stage 2 Stage 3 Stage 3
Secondary EducationAdmission to
Teacher EducationAdmission to Student
TeachingProgram
CompletionFollow-up
Complete application Cumulative GPA of 2.5 or higher
Student Teaching Portfolio Employer Evaluation
FOED 1110 and FOED 2110 with grade of C or better
Completion of all Professional Education courses with a grade of C- or higher
Assessment of Student Teacher Preparation Form
Candidate evaluation of performance and preparation
Cumulative GPA of 2.5 or higher
Negative tuberculin test Acceptable ratings on dispositions
Acceptable scores on Praxis I or approved substitute test (ACT, SAT, GRE)
Acceptable ratings on dispositions
Summative evaluation forms from both placements
Complete 45 hours of coursework
Acceptable ratings on Critical Performances
Self Assessment of Student Teacher Performance Form
Acceptable ratings on dispositions
Senior, graduate, or post-bac classification
Completed Background Check
Reports
• The following types of reports are available based on data collection has begun:– Success Rate at Each Transition Point– Distribution of Students Meeting each Transition
Point Requirement– Percentage of Students Meeting each Transition
Point Requirement
Success Rate at Each Transition Point
Admission to Program
Entry to Clinical Practice
Exit From Clinical Practice Exit From Program
90% 78% 70% 66%
Distribution of Students in Early Childhood Education Program by Transition Points
Stage # Students
Admission to Program 144
Entrance to Student Teaching 125
Exit from Student Teaching and Recommendation for Licensure 98
Exit From Program 85
Aggregate Report of Students Passing Each Transition Point
Requirement
Entrance into the Mathematics Education
ProgramAdmission to Student
Teaching
Exit from Student Teaching and
Recommendation for Licensure Follow-up
Submission of Program Admission Form 100% x x x
Department Interview 100% x x x
Dispositions-based letter of recommendation from university faculty member 99% x x x
Overall GPA 2.75 or higher, 3.0 or higher in all EDU courses 92% x x x
Maintained overall GPA of 2.75 or higher x 85% x x
Disposition based letter of recommendation from Education faculty members x 84% x xDevelopmental portfolio including student-selected artifacts which show evidence of progress towards attainment of standards
x 82% x x
Passing Praxis I score x 80% x x
Standard Mapping
• Align course work to standards for:– State Boards of Education– Professional accrediting bodies: Business, Nursing,
PT, OT, CJ, Education– Regional accreditors
• Relate competencies developed in curriculum to specific standards for accreditation
ReportsNAEYC Initial Licensure Standards in Early Childhood Education
Field Experience
• Collect demographic data on sites where students are placed
• Students use electronic templates to submit field experience work
• Mentors assess student work in relation to agreed criteria; and comment on student preparedness
• Aggregate data on:– % of students who complete practica– Effectiveness of sites in achieving program objectives
Faculty Artifacts
• Demonstrate that faculty have credentials and levels of engagement required to fulfill program objectives
• Reports provide evidence of faculty qualification, research, teaching effectiveness, and institutional contributions
• Treat faculty as the essential human capital of the institution
Strategic Plan Dashboards
• Align Broad based Goals and Strategic Initiatives with:–Action Plans–Budgets–Assessment Measures
• Dashboard for tracking and monitoring • Include “non academic” departments
Benefits to the Institution• Track student progress
• E-Portfolio aid students and advisors in developing resume,
graduate school applications
• Identify trends in programs
• Identify areas of improvement
• Generate reports to fulfill accreditation needs
• Communicate with senior management and Board on
Performance Management – “close the loop” with Strategic
Plan
