HOW DO YOU KNOW IT WORKS?

Measuring the Effectiveness of Career Counselling

Interventions

Joan McCurdy Myers

Felicity Morgan

HOW DO YOU KNOW IT WORKS?• UTMississauga Career Centre • Career Counselling Interventions and

Evaluation• Ace Your Career Case: content and

evaluation• Results• Challenges of integrating practice and

research• Next steps

UTMississauga CAREER CENTRE

• 10,500 students, mainly undergraduate• Broad range of academic programs• Large proportion of commuter students, living at home, GTA• Services include career counselling, employment advising,

workshops, panels, employer events, job postings• Staffing: 15 non-student staff

– 7(5.4 fte) staff in Career Development Unit

– 4 Employer Relations & Marketing

– 3 admin– 18 student staff

• Partnerships with academic programmes & student groups: key to tailoring for student needs

CAREER COUNSELLING INTERVENTIONS

• Individual, workshops & events• Some in class, some not• Workshops & events:

– Evaluate using post session evaluations, learning objectives, follow up surveys

– Students report high levels of satisfaction with sessions and meeting learning objectives

• Standard: 75% rate 4/5 on 5 point scale

EVALUATION OF INTERVENTIONS

• In 2006, decided to go beyond post-session satisfaction/feedback & surveys & attempt to measure what the students were truly learning– Our own questions– Learning outcomes focus at UT– Student Research Assistant position– Funding: through the Enhancing the Student

Experience Fund for two projects; Ace Your CareerCase and Intro Psych student research credit for Career Development workshop

ACE YOUR CAREER CASE

• 2nd and 3rd year Management and Commerce students

• Goal: career goal clarity: through self assessment, gathering of information and interaction with industry professionals

• Held annually in January for three years; facilitated by career counsellor and job coach consultant

ACE YOUR CAREER CASE

• 2 half days; Day One - self assessment through career

counselling exercises; career research methods and sources

Day Two – business introduction development and practice; lunch and interaction with employers; reflection on what was learned and next steps

• Co-sponsored with Management Department, funded by Provost’s Office

LEARNING OUTCOMES• Identify clear career goal, based on self

knowledge and industry knowledge• Locate and use career research

resources• Develop, practice and deliver business

introduction• Understand benefits and opportunities

available in employer events• Participate more effectively in recruitment

METHODS OF EVALUATION

• Post workshop evaluation• Follow up survey• Video-taped business introduction (years 2

& 3)• Students completed Career Decision

Profile pre Day One and post Day Two

CAREER DECISION PROFILE

• Developed by Prof. L. Jones (emeritus), North Carolina State, President Career Key Inc.

• Measures: – Decidedness– Comfort– Self Clarity– Knowledge about Occupations & Training– Decisiveness– Career Choice Importance

WHY THE CDP?

• Previously used it with at risk students; gave them clarity and purpose

• Scales break down the components of career planning, separates them out

• Questions are process oriented, not final answer oriented; fits with our approach

• Can be used as a counselling tool• Had support from the author

CDP SCALES

• Decidedness: how decided are you about your choice of career?

• Comfort: how comfortable are you with where you are in the process of making this choice?

• Self Clarity: how clear is your understanding of your abilities, interests and personality and how they fit with careers

SELF CLARITY SAMPLE QUESTIONS

• I wish I knew which occupations best fit my personality

• I need to have a clearer idea of what my interests are

• I need to have a clearer idea of my abilities, my major strengths and weaknesses.

• 8 point scale; strongly disagree to strongly agree

CDP SCALES

• Knowledge: how well informed you believe you are about occupations and educational programs that fit your interests

• Decisiveness: how capable you believe you are in making decisions without unnecessary delay

• Career Choice Importance: importance of choosing a career at this time

HOW EFFECTIVE WAS ACE?

• Results from post workshop evals• Results from follow up survey• Results from video-taped business intro• Results from CDP

RESULTS

• Post-workshop Evaluation– 100% recommended to a friend every year– Met learning outcomes criteria re: clarity of career

goal, strengthened career plan, more effective participation in recruitment,

– Did not meet learning outcomes criteria for: • The strength of their self marketing strategies • Satisfaction (wanted more time with industry professionals)

– Sometimes met: resources and how to use opportunities

RESULTS

• Follow Up Survey (one month post)– Overall, students felt all measures increased

significantly from pre-ACE to post-ACE• they understood themselves and the work world

better• they were more prepared for the recruitment

season• their career plan was more thorough and clear• their ability to self market had improved

RESULTS

• Video-taped Business Intro– Yr 2- 45% scored in 7-10 range, average 6.35

– Yr 3-66% scored in 7-10 range, average 7.4– Identifying 2 key strengths was a challenge

RESULTS• CDP

– Overall, scores moved in positive directions• Decidedness yielded mixed results

– Students significantly (<.05) increased their scores in knowledge 2 of 3 years (year 1 <.10)

– Year 3 saw the most significant changes: comfort, self clarity, knowledge

• Decidedness (significant but decreased)• Decisiveness (approaching significant <.10)

– Importance of Career Decision

RESULTS-for our student programming

• Great opportunity to experiment with new teaching/learning approaches– Business introduction (improved our teaching; used same

assessment in subsequent sessions)– Positive response of industry reps and students to reps

encouraged us to incorporate format in more of our activities

– Sharpened our focus on how to help students learn to reflect-incorporated direct reflection opportunities in a range of programming

– Style of greater student engagement on the spot also now used in Fall Management Conference with very positive results

CONCLUSIONS: How do you know it works?

• Take multiple perspectives to see what works– Students’ perceptions, satisfaction, learning

outcomes, feedback– “objective” measures e.g. CDP, behavioural

measurement e.g. business intros, 3rd party evaluations, rubrics

• Student participation in services, recruitment activities• Student “success” e.g. # interviews at recruitment

– Develop database for comparison e.g. UTM CDP

• Use results to improve programming

INTEGRATING PRACTICE & RESEARCH

• Proper evaluation without too much evaluation

• Need more time & expertise to plan, analyze, understand & use results

• Difficulty of balancing service provision with need for more structured assessment

• Speaking the language of the academic e.g. statistical results

INTEGRATING PRACTICE & RESEARCH

• Managing partnerships with academics is challenging

• Gratifying to be able to document & report changes

• Increased our skills and knowledge as professionals; identified new learning goals

• Benefits other students through improved programming

NEXT STEPS• Using CDP in other interventions; assists us to

assess where our students are at the beginning and improve our interventions to increase positive outcomes

• Follow up with participants• Encouraged us to go beyond satisfaction

measures; now “see” more opportunities for broader range of assessment/evaluation

• Increasing our expertise as a Centre & division• Looking for more partnerships

QUESTIONS?

Joan McCurdy-Myersjoan.mccurdy.myers@utoronto.ca

Felicity Morganfelicity.morgan@utoronto.ca