Graduate Learning Outcomes: The challenge of assessment

Serge DesmaraisUniversity of Guelph

Learning Outcomes Conference

October 17, 2014

Fostering a learning outcomes culture

U of G established 10 graduate learning objectives in 1987: Inspirational and lofty “desired characteristics of graduates” Used to guide new grad and undergrad course and new

program proposals

Since 2005, we re-engaged the community in an effort to develop a learning outcomes culture 2005: Provost’s White Paper 2006-2007: 21st Century Curriculum Committee 2008: UUDLEs & GDLES 2010: IQAPs at U of G & across the province

Fostering the Culture First strategic steps

Introduced “Outcomes” as a priority for governance: First addressed at Board of Undergraduate Studies and

Program Committees Following by introduction of same strategy at Board of

Graduate Studies

Created a new Senate Committee: Quality Assurance – new Senate committee; assist

departments in meeting requirements under IQAP

Engaged Associate Deans with program-level responsibility in the discussion

Increased staff support: more teaching & learning staff

University Learning Outcomes Two year process of broad consultation with

faculty, staff and students

New learning outcomes combine:

U of G’s learning

objectives (1987)

LEAP+Bases of

Competence (Evers, Rush &

Bedrow)

+

5 University-wide Outcomes approved by Senate for both undergraduate (2012) and graduate (2013) programs:

Critical and Creative Thinking Literacy Global Understanding Communicating Professional and Ethical Behaviour

University Learning Outcomes

Critical and Creative Thinking

Introduce Reinforce Mastery

Inquiry and Analysis

Problem Solving

Creativity

Depth and Breadth of Understanding

. . . one applies logical principles . . . to solve problems with a high degree of innovation, divergent thinking and risk taking.

. . . show evidence of integrating knowledge across disciplinary boundaries. Depth and breadth of understanding of disciplines essential to this outcome.

Critical and Creative Thinking

Introduce Reinforce Mastery

Inquiry and Analysis

Problem Solving

Creativity

Depth and Breadth of Understanding

x

Depth and Breadth of Understanding

Level 3: Compares the merits of alternate hypotheses in many different disciplines. Demonstrates mastery of a body of knowledge and critically evaluates the limits of their own knowledge and how these limits influence analyses.

Assessing L.O.s in Undergrad programs

Collaboration with Desire2Learn to develop the assessment tools within our current LMS

Our eventual goal: an assessment method that will be used by all departments and programs

The initial plan engaged two programs: Bachelor of Arts and Science Bachelor of Engineering

Now working with the full B. Eng program and beginning process with DVM (Veterinary Medicine)

Lessons learned… so far

Software development is not the most complex aspect of the project

Decisions re: which outcomes to assess at what point in the curriculum progression is complex

Long term, challenge is to gain full departmental and faculty engagement: Much resistance, paranoia, and hope for continued status

quo

Identify your champions and the most engaged programs, and leverage their enthusiasm and persuasiveness

How to proceed?

A “one size fits all” model is not the right approach Must respect disciplinary differences Efforts should be programmatic, not only

course by course Graduate programs must have a clear

sense of when and where specific learning outcomes will be addressed and assessed

Curriculum Mapping Assessment must be documented and

evidentiary based

So… how do we assess in grad programs?

Moving to a culture of evidence-based continuous improvement will be hard work Requires a re-envisioning of the entire

graduate program process Requires a commitment and broad

agreement on departmental and program strategy

Let’s work on a few examples….

Communicating Rubric

  Introduce1

Reinforce2

Master3

Oral CommunicationIncludes interpersonal skills and oral speaking

Demonstrates the ability to present information in a comprehensive manner, clearly, and effectively.

Demonstrates oral communication skills that are organized, and presented in a creative and interesting manner. The student speaks in a clear, loud voice.

Demonstrates the ability to present information in that the receiving party can easily understand the information. The speaker is clear and shows confidence in public speaker. Has mastered oral communication skills.

Written CommunicationThe ability to express one’s ideas through writing

The ability to write clearly, and demonstrate a general knowledge about describing an idea.

Has the ability to write a clear message with good vocabulary and little grammatical, spelling or functional errors.

The student has shown a breadth of vocabulary and can write in a sophisticated manner that clearly conveys the message of the speaker. Grammar, spelling, and functional errors are almost non-existent.

Reading ComprehensionThe understanding of writing

The student develops the key components in reading at their age and grade level

The ability to read a wide vocabulary, and understand more sophisticated writing is evident.

The ability to read and extract information from the text has developed. The student can answer questions, and generate ideas from writing.

ListeningInvolves being attentive when others are speaking, and responding effectively to others’ comments during a conversation

The ability to pay attention when a speaker is talking.

The receiver shows evidence they are intently listening, occasionally asking questions.

The receiver not only demonstrates bodily gestures indicating listening, but also engages with the speaker, including asking questions, and reiterating points the speaker states.

Information Literacy Rubric

  Introduce1

Reinforce2

Master3

Collaborative LiteracyThe ability to know where there is a need for information

Is able to locate and use materials from a variety of resources.

Can not only locate materials, but also is able to understand when more information is needed and how to evaluate its effectiveness.

This skill is developed where one can easily identify, locate, evaluate many resources and effectively and responsibly use the information discover new information.

Quantitative LiteracyIncludes numeracy, and a comfort in working with numerical data

Understands the need for evidence and use of numerical data, and is able to generally interpret this data.

Is able to use quantitative data to one’s advantage by using it as evidence for a claim. Is comfortable with everyday situations with numeracy.

Possesses the ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations, and from this, can develop sophisticated arguments supported by quantitative evidence. Is also able to create and communicate numerical data.

ComputingThe ability to use technology and programs

Uses technology to research and support communication of knowledge.

Is able to use computers and technology to compliment knowledge and understanding, and has a large skill set in helpful programs.

Has a breadth of knowledge in computer and technology skills which one uses to enhance presentations and knowledge. Uses technology in a sense that does not overshadow knowledge, but rather compliments it.

Depth and Breadth of UnderstandingAn ability to break disciplinary boundaries and bring together information

Realizes the differences of information from a variety of disciplines.

Is able to extract information, and understands the advantages of using a variety of disciplines.

Is able to understand a variety of expressions from different disciplines, and can use information to ones advantage, i.e. many supporting perspectives on a claim enhances the reliability and objectivity.

Thank You

Serge Desmarais

Acting Provost and Vice-President (Academic) s.desmarais@exec.uoguelph.ca

www.uoguelph.ca/vpacademic/avpa/outcomes