View
215
Download
1
Category
Preview:
Citation preview
Conducting Assessment of
Academic Advising
2013 NACADA Summer InstituteJacksonville, Florida
Dr. Karen BostonDr. Rich Robbins
NACADA Executive OfficeKansas State University
2323 Anderson Ave, Suite 225Manhattan, KS 66502-2912
Phone: (785) 532-5717 Fax: (785) 532-7732
e-mail: nacada@ksu.edu
© 2013 National Academic Advising Association
The contents of all material in this presentation are copyrighted by the National Academic Advising Association, unless otherwise indicated. Copyright is not claimed as to any part of an original work prepared by a U.S. or state government officer or employee as part of that person's official duties. All rights are reserved by NACADA, and content may not be reproduced, downloaded, disseminated, published, or transferred in any form or by any means, except with the prior written permission of NACADA, or as indicated below. Members of NACADA may download pages or other content for their own use, consistent with the mission and purpose of NACADA. However, no part of such content may be otherwise or subsequently be reproduced, downloaded, disseminated, published, or transferred, in any form or by any means, except with the prior written permission of, and with express attribution to NACADA. Copyright infringement is a violation of federal law and is subject to criminal and civil penalties. NACADA and National Academic Advising Association are service marks of the National Academic Advising Association.
The Global Community for Academic Advising
• Review of reasons for performing assessment• Review of differences between evaluation and
assessment• The assessment cycle• Identifying stakeholders• Advising is teaching• Identifying student learning outcomes from various
sources• Outcome measurement• Acting upon the results• Assessment as research
Overview
The Global Community for Academic Advising
• Program effectiveness
• Program improvement
• Program accountability• Enhancing student
• success• persistence• retention
• Activities aimed at student success, program improvement and accountability are all important
Assessment Has Multiple Purposes
The Global Community for Academic Advising
Assessment is
• An on-going cycle of activity
• A gathering of a variety of information and data
• A team effort with faculty, staff, students, and administrators all actively engaged
• A complex process of comparison
The Global Community for Academic Advising
Remember…
“…a lack of assessment data can sometimes lead to policies and practices based on intuition, prejudice, preconceived notions, or personal proclivities – none of them desirable bases for making decisions”
Upcraft and Schuh (2002. p. 20)
The Global Community for Academic Advising
A Distinction with a Difference
Assessment Evaluation
Focus Programmatic Individual
Process Continuous & Embedded
Episodic
Measures Outcomes Effectiveness
Part of assessment, part of measuring outcomes
The Global Community for Academic Advising
The Assessment Cycle (Maki, 2002, 2004)
Gather
Evidence
Interpret
Evidence
Identify
Outcomes
Implement Change
Mission/PurposesEducationalObjectives
The Global Community for Academic Advising
The Assessment Flowchart
Student Learning OutcomesCognitive, Psychomotor, AffectiveProcess/Delivery Outcomes
Mapping the ExperienceWhat experiences?When or by when?
Gathering EvidenceWhen gathered?Where gathered?
How often gathered?From whom gathered?
How gathered?Minimum performance criteria for success?
ValuesVision
MissionGoals
Programmatic Outcomes
Sharing and Acting Upon the ResultsInterpret how results inform practice
How and with whom to share interpretationFollow up on implemented changes
Start the process all over again!
(adapted from Darling, 2005, 2010)
The Global Community for Academic Advising
The Assessment Matrix/TableInstitutional Mission Statement
Local Mission Statement
Specific Goal or Objective
Specific Process/Delivery Outcome or Student Learning Outcome
Where Outcome Occurs
By When You Want Outcome To Occur
Outcome Measure
Data Instruments
Minimum Performance Criteria for Success (Threshold)
Action(s) Based on Outcome Data
Mapping of Outcomes
Adapted from Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12). San Francisco: Jossey-Bass.
Robbins, R. (2011). Assessment and accountability of academic advising. In J. Joslin & N. Markee (eds.), NACADA Monograph #22: Academic Advising Administration: Essential Knowledge and Skills for the 21st Century (chapter 4). Manhattan, Kansas: NACADA.
Robbins, R. & Zarges, K.M. (2011). Assessment of Academic Advising: A Summary of the Process. NACADA Clearinghouse of Academic Advising Resources: http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/assessment-Robbins-Zarges.htm
The Global Community for Academic Advising
Exercise 1:Identify Your Stakeholders
The Global Community for Academic Advising
Key Terms• Student Learning Outcomes – Articulate what students are
expected to know, do, and appreciate as a result of involvement in the academic advising experience
• Process/Delivery Outcomes – Articulate the expectations for how academic advising is delivered and what information should be delivered through the academic advising experience
• Mapping – The process of determining when, where and how the outcomes for advising will be accomplished over the students’ academic career and beyond
The Global Community for Academic Advising
Process/Delivery OutcomesProcess/Delivery Outcomes
• Do not assess student learning, but evaluate what processes occurred during the advising interaction
• Typically what is evaluated via student satisfaction surveys• e.g., “the advisor provided accurate information”• e.g., “the setting for the advising session was
appropriate”• etc.
The Global Community for Academic Advising
Focus on Student Learning:Advising is Teaching
The Global Community for Academic Advising
Advisor as TeacherAdvisor as Teacher
• facilitator of communication
• coordinator of learning experiences
• referral agent who connects students with all of the institution’s resources and co-curricular opportunities that can help them be successful
The Global Community for Academic Advising
Exercise 2:What skills used in teaching are also used in advising?
The Global Community for Academic Advising
Teaching
Knowledge of subject matter
Preparation - planning and organization of course material
Advising
Knowledge of institutional policies, procedures, programs, resources
Preparing for advising meetings
The Global Community for Academic Advising
Teaching
Engaging students in participation of their learning
Giving students feedback on their progress
Helping students learn to analyze and problem solve
Advising
Guiding students to be self-directed and autonomous
Working with student to regularly evaluate the student’s goals and progress
Assisting students in decision-making skills
The Global Community for Academic Advising
Teaching
Clarity of presentation of subject matter
Establishment of dialogue with students in the classroom
Demonstration of excellent listening skills
Advising
Sharing information in a clear manner
Leading students to question and interact with the advisor
Listening to what advisees are saying verbally and non-verbally
The Global Community for Academic Advising
Teaching
Assigning out-of class work
Working from a developmental perspective
Providing a learner-centered environment
Advising
Assigning out-of-session tasks
Working from a developmental perspective
Providing a learner-centered environment
The Global Community for Academic Advising
Teaching
Identification and communication of student learning outcomes for the curriculum or course
Use of a course syllabus
Advising
Identification and communication of student learning outcomes for the advising process/interaction
Use of an advising syllabus
The Global Community for Academic Advising
Focus on the Learner/AdviseeFocus on the Learner/Advisee
As a result of academic advising, what do we want students to demonstrate that they…
• Know (cognitive student learning outcome)
• Are able to do (behavioral student learning outcome)
• Value and appreciate (affective student learning outcome)
The Global Community for Academic Advising
It’s All AboutStudent Learning
The Global Community for Academic Advising
Student Learning Outcomes
What do you expect students to be able to
• Know (cognitive learning)• Do (behavioral learning)• Value and Appreciate (affective learning)
as a result of participating in academic advising?
The Global Community for Academic Advising
Cognitive Elements
What do we want students to KNOW as a result of participating in academic advising?
Know general education requirements
Know about academic support services
Know how to use the student information system to register
Know how to use the catalog
Etc…….
The Global Community for Academic Advising
Exercise 3:Identify Three Things You Want Students to Know as a Result of
Academic Advising
The Global Community for Academic Advising
Tips on Developing SLOsTips on Developing SLOs
• avoid compound SLOs, e.g., “students understand and value the purpose of a liberal arts education,” “students understand and utilize their degree progress reports”
• as you are developing each SLO, think about where and when the opportunities for student learning may occur for that desired SLO
• as you are developing each SLO, think about how you might measure whether or not the SLO has been achieved
The Global Community for Academic Advising
Skills/Behavioral/Psychomotor Elements
What do we want students toDo as a result of participating in academic advising?
Generate their degree audit
Make advising appointments
Keep advising appointments
Ask for help
Access course descriptions and degree requirements using the online catalog
Etc….
The Global Community for Academic Advising
Exercise 4:Identify Three Things You Want Students to Do as a
Result of Academic Advising
The Global Community for Academic Advising
Affective Elements
What do we want students to Value or Appreciate
as a result of participating in academic advising?
Value/Appreciate general education
Value/Appreciate the advising relationship
Value/Appreciate the process of learning
Etc….
The Global Community for Academic Advising
Exercise 5:Identify Three Things You
Want Students to Value or Appreciate as a Result of
Academic Advising
The Global Community for Academic Advising
Developing SLOs from Programmatic Goals
The Global Community for Academic Advising
What do students need to demonstrate they know, can do, or appreciate in order to say that your programmatic goals have been achieved?
The Global Community for Academic Advising
Exercise 6:
• Identify the services your advising program offers
• Identify 1 student learning outcome for each service
• Prioritize• If you could assess only two to start, which
ones would they be?
The Global Community for Academic Advising
Deriving SLOs from other sources
The Global Community for Academic Advising
Clear and measurable student learning outcomes for academic advising may also be derived from
• the NACADA Concept of Academic Advising
• the NACADA Core Values
• the CAS Standards for Advising
The Global Community for Academic Advising
NACADA Concept of AdvisingNACADA Concept of Advising
http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/Concept-advising-introduction.htm
• multi-dimensional and intentional
• grounded in teaching and learning
• includes specific purpose and content
• includes specific student learning outcomes for academic advising
The Global Community for Academic Advising
NACADA Statement of Core Values
• Six values that NACADA adopted in 2004 as being important regarding what academic advisors are expected to do as part of the advising interaction
• May be easily translated into Student Learning Outcomes…
http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/Core-Values.htm
The Global Community for Academic Advising
NACADA Core Values: Advisors NACADA Core Values: Advisors Teach Students…Teach Students…
• to value the learning process
• to apply decision-making strategies
• to put the college experience into perspective
• to set priorities and evaluate events
• to develop thinking and learning skills
• to make choices
The Global Community for Academic Advising
NACADA Core Values as Student NACADA Core Values as Student Learning OutcomesLearning Outcomes
As a result of academic advising, students will
• value the learning process• apply decision-making strategies• put the college experience into perspective• set priorities and evaluate events• develop thinking and learning skills• make choices
The Global Community for Academic Advising
Using the CAS Standards in Program Assessment
Council for the Advancement of Standards in Higher Education (CAS) Standards and Guidelines for Academic Advising
http:// http://www.nacada.ksu.edu/Resources/Clearinghouse/View-Articles/CAS-Advising-Standards.aspx
The Global Community for Academic Advising
CAS Standards for Academic AdvisingCAS Standards for Academic Advising
• 16 “relevant and desirable” student learning outcomes for academic advising (revised in 2006)
• Developed based on every type of higher education academic setting
• Each outcome will be achieved based on exposure to student and student maturity
• May require a systematic approach re: introducing these concepts and expectations in order for students to achieve these student learning outcomes
The Global Community for Academic Advising
Advising Programs…Advising Programs…
…promote learning and development in students by encouraging experiences which lead to:
• intellectual growth• the ability to communicate effectively• appropriate career choices• leadership development• the ability to work independently and collaboratively• and so on for all 16 student learning outcomes…
The Global Community for Academic Advising
It is not necessary to attempt to use all of the CAS Standards as your outcome statements…
…for those you may choose to use, be sure to develop related student learning outcomes as well as process/delivery outcomes
The Global Community for Academic Advising
Developmental SLOs
• The offering of opportunities for students to learn the desired SLOs will vary by SLO
• The time by which your students need to demonstrate achievement of a specific SLO will vary by SLO
• Not all SLOs are achieved at same time
The Global Community for Academic Advising
Thus, you may also take a developmental approach, and utilize different SLOs as desired outcomes for students at different times during their college careers
The Global Community for Academic Advising
Mapping of Outcomes
The process of determining when, where and how the SLOs for academic advising will be accomplished over the students’ academic career and beyond
The Global Community for Academic Advising
Mapping the Learning Experience
• What should be learned: e.g., student knows the components of the institution’s General Education requirements
• Where it should be learned: e.g., orientation workshops, advising sessions, personal reading of catalog or curriculum guide
• When it should be learned: e.g., prior to first year (orientation); by end of first year (via advising sessions); by end of first year (via personal reading)
The Global Community for Academic Advising
Mapping of Outcomes
The Assessment Matrix/Table
Institutional Mission Statement
Local Mission Statement
Specific Goal or Objective
Specific Process/Delivery Outcome or Student Learning Outcome
Where Outcome Occurs
When or By When Outcome Occurs
Outcome Measure
Data Instruments
Minimum Performance Criteria for Success (Threshold)
Action(s) Based on Outcome Data
The Global Community for Academic Advising
Exercise 7:Mapping Your Desired
Outcomes:Identifying Where/When
Outcomes Occur
The Global Community for Academic Advising
Measuring Outcomes
Once the desired Process/Delivery outcomes and Student Learning outcomes have been identified, as well as when and where they will occur, the next step is to determine who or what will be measured and how the data will be gathered…
…using multiple measures of varying types
The Global Community for Academic Advising
Initial ConsiderationsTrue assessment of academic advising should involve multiple measures
– beyond a simple satisfaction survey or advisor evaluation
– individual measures evaluate (rather than assess) a single dimension of a single phenomenon
The Global Community for Academic Advising
Types of Measures• Qualitative
– exploratory– small samples– open-ended– emerging information– subjective, inductive interpretation of data
• examples– focus groups
– case studies
– naturalistic observation
Information/data in form of rich, in-depth responses (words)
The Global Community for Academic Advising
Types of Measures continued…
• Quantitative– descriptive– large samples– structured– objective, deductive interpretation of data
• examples– questionnaires
– surveys
– experiments
Information/data in form of numbers, measures (statistics)
The Global Community for Academic Advising
Types of Measures continued…
• Direct– may be qualitative or quantitative– examples
• direct observation of advising interaction• pre-test/post-test of variable leading to desired outcome• standardized test or inventory measuring student learning• tracking of student data (enrollment rates, retention rates, GPAs,
transcript analysis, etc.)• counts of use of services• advisor:student ratios
The Global Community for Academic Advising
Types of Measures continued…
• Indirect– may be qualitative or quantitative– examples
• focus groups• surveys, questionnaires• interviews• reports• tracking of student perceptions (satisfaction, ratings of advisors,
ratings of service, etc.)• tracking of advisor perceptions (student preparedness, estimation
of student learning, etc.)
The Global Community for Academic Advising
Examples of Existing Instruments
• To be used as just one measure among multiple measures– ACT Survey of Academic Advising– Noel-Levitz Student Satisfaction Inventory (SSI)– Winston and Sandor’s Academic Advising
Inventory (AAI)– NACADA Assessment of Advising Commission
www.nacada.ksu.edu/Commissions/C32/index.htm
The Global Community for Academic Advising
Other Sources
• To be used as just one measure among multiple measures– Data from National Survey of Student
Engagement (NSSE)– Data from other nationally normed, standardized
instruments- Existing institutional data
The Global Community for Academic Advising
Institutional Data• Why reinvent the wheel, so to speak, if there are existing
institutional data that relate to a question you would like answer regarding your advising program?
• Become knowledgeable regarding what type of data your institution is already collecting
• Get to know your institutional data people, and include them as a stakeholder in the assessment of your advising program
• Remember that any single type of data should be utilized as just one measure among multiple measures for each phenomena you are evaluating within your overall assessment effort
The Global Community for Academic Advising
For both process/delivery and student learning outcomes, you need to identify the minimum criteria for success of the outcome measure, e. g.,•number of students exhibiting a specific learning performance•percentage of students exhibiting a specific learning performance• advisor rating of student performance• student rating of specific aspect of advising process• advisor rating of specific aspect of advising process• etc.
The Global Community for Academic Advising
In the absence of any existing relevant measures of the identified student learning outcomes for academic advising, the initial cycle of assessment for any given desired student learning outcome should be considered a baseline data gathering effort
Suggestion
The Global Community for Academic Advising
Mapping of Outcomes
Adapted from Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12). San Francisco: Jossey-Bass
The Assessment Matrix/Table
Institutional Mission Statement
Local Mission Statement
Specific Goal or Objective
Specific
PDO or SLO
Where Outcome Occurs
When or By When Outcome Occurs
Outcome Measures
Data Instruments
Minimum Performance Criteria for Success (Threshold)
Action(s) Based on Outcome Data
The Global Community for Academic Advising
Exercise 8:Identifying Multiple Measures
Select one student learning outcome, and identify three ways to measure whether or not that specific outcome has been met
The Global Community for Academic Advising
An Additional And Important Consideration
- even if your outcome data supports the achievement of the desired SLO – how do you know that the student learning was a result of academic advising???
- you literally have to ask students “where did you learn the information,” “how did you know to do the behavior,” and/or “how did you develop the appreciation” in regard to the SLO
The Global Community for Academic Advising
Advising Syllabus as a Tool for Communicating Key Elements
• If academic advising is teaching…• Advisors are teachers• Teachers have a discipline• The advisor’s discipline is academic advising• Academic advising is a discipline• Individual academic advisors offer the “course” academic
advising• Individuals in disciplines author unique courses• Each course has a syllabus
The Global Community for Academic Advising
What Does an Advising Syllabus Include?
• Purpose of academic advising• Scope of academic advising• Tools, texts, resources• SLOs for academic advising• Advisor responsibilities• Advisee responsibilities• Criteria for successful academic advising experiences
(outcome measurements)• Other elements as individually appropriate
The Global Community for Academic Advising
Acting Uponthe Outcome Data
The Global Community for Academic Advising
Determine How and With Whom Results Are Shared
• Administration: President, Provost, various committees
– via annual report, strategic plan, white paper, Web sites, etc.
• Faculty: all faculty, curricular committees, faculty advisors
– via performance reviews, annual reports, strategic plans, Web sites, etc.
• Students: all students, student advisees, student senate, student groups
– via newsletters, annual reports, Web sites, etc.
• Budgeting entities
– via annual reports, budget requests, Web sites, etc.
• Accreditors
– via self-studies, accreditation reports, Web sites, etc.
The Global Community for Academic Advising
• Revise pedagogy or curriculum
• Develop/revise advisor training programs
• Design more effective programming– advising, orientation, mentoring, etc.
• Increase out-of-class learning opportunities
• Shape institutional decision making– planning, resource allocation
• Other…
Interpret How Results Will Inform Decision Making
The Global Community for Academic Advising
Decide How You WillFollow-up on Implemented
Changes• Timetable to implement changes
– implement all or specific components on a schedule
• Assessment of implemented changes– repeat assessment cycle again
• Continuous assessment– assessment is on-going
The Global Community for Academic Advising
Professional Development• Using assessment to inform and support
professional development
• Revise advisor training and development programming accordingly
• Demonstrate the need for additional training and development
• Demonstrate the need for additional resources to meet goals
The Global Community for Academic Advising
Building a Culture and Capacity for Assessment
• The Culture• Commitment• Communication• Collaboration
• The Capacity• Support
The Global Community for Academic Advising
Exercise 9:Acting Upon the Outcome Data
Identify at least one use for the outcome data of your assessment of advising
The Global Community for Academic Advising
At the end of the day, assessment of academic advising
is all about…• developing consensus around collective expectations about student learning that should occur in advising
• gathering evidence in order to understand student learning resulting from academic advising
• using this evidence to support improvements in academic advising that will contribute to improvements in student learning
The Global Community for Academic Advising
assessment is much morethan just a single evaluation
Remember:
The Global Community for Academic Advising
Assessment as Research
The Global Community for Academic Advising
Why Research Academic Advising?Why Research Academic Advising?
• Academic advising is a collective experience among most college students
• Academic advising is an integral part of student development
• Academic advising is teaching, with a discipline and pedagogy
• Professional disciplines utilize inquiry and resulting data to inform decision making
• There is much anecdotal information re: the power of effective advising, but little empirical research
The Global Community for Academic Advising
Wide Range of Audiences
• Peers in academic advising
• Advising administrators
• Higher Education
• Field of Advising
• Deans, Provosts, and Presidents
• Individual advisors
• Students
• Parents
The Global Community for Academic Advising
Scholarly Inquiry May Include…Scholarly Inquiry May Include…
• Hypothesis testing• Replication of existing knowledge in new
setting• Discovery of a novel phenomenon• Development of a new theory• Creation of new knowledge• Evaluation of effectiveness of new
implementation or approach
The Global Community for Academic Advising
Methods of InquiryMethods of Inquiry
• Experimental• Quasi-experimental• Ex-post facto (after-the-fact)• Correlational• Historical• Ethnographic (cultural interpretation)• Phenomenological (description of experience)• Case study• Longitudinal• Program Assessment
The Global Community for Academic Advising
Continuing Assessment as Continuing Assessment as ResearchResearch
Once you have obtained assessment data and act upon it via implementing a new strategy or model, the subsequent assessment of the new implementation serves as research:
• experimental assessment• current versus historical data• ethnographic inquiry• qualitative inquiry• quantitative inquiry• other research/inquiry
The Global Community for Academic Advising
Cognitive Learning Outcomes:• Understand the differences between assessment and
evaluation• Understand the differences between Process/Delivery
Outcomes (PDO) and Student Learning Outcomes (SLO)
• Understand the concept of “mapping” of desired outcomes
• Understand the importance of using multiple measures for any single desired outcome
Learning Outcomes for Presentation
The Global Community for Academic Advising
Behavioral Learning Outcomes:• Identify your stakeholders in the assessment of advising• Identify at least one SLO for your assessment of
advising process• Map at least one SLO• Identify multiple measures for at least one SLO• Identify at least one use for the outcome data of your
assessment of advising
Learning Outcomes for Presentation
The Global Community for Academic Advising
Affective Learning Outcomes:• Value the cyclical process of assessment• Appreciate the importance of assessment (besides
accountability)• Appreciate the value of student learning in advising• Appreciate the importance of assessing student learning
in advising• Value the process of identifying opportunities for
learning (mapping) in the advising process• Appreciate the need for multiple measures for any single
desired PDO or SLO• Value the role of outcome data in informing decisions
Learning Outcomes for Presentation
The Global Community for Academic Advising
Questions/Discussion
The Global Community for Academic Advising
THANK YOU!
The Global Community for Academic Advising
Recommended