Assessing Generic Skills-2-GordonV2

Al Hussein Fund for Excellence - Teaching, Learning & Assessment in HE

Training workshop, 28-29 November 2010 1

Assessing Generic Skills

Prof. Gordon Stanley

“Teaching, Learning and Assessment in Higher Education”

Workshop organized by

Al Hussein Fund for Excellence

Amman – Jordan, 28-29/11/2010

Reporting Standards

� Grading based on required learning

outcomes

� Assessment tasks are designed to indicate

progress towards learning outcomes

� Assessment grade is a measure of the extent

to which learning outcomes have been

achieved.

28-29/11/2010 2Teaching, Learning & Assessment in HE



Graduate Knowledge

What is a student who has just graduated

expected to be able to know and do?

The next slide describes what standards a

student graduate in biosciences who

specialised in organisms would have

achieved.


GenericThreshold standard

On graduating with an honours degree

in biosciences, students should be able

to access bioscience information

from a variety of sources and to

communicate the principles in a

manner appropriate to the

programme of study

Typical standard

On graduating with an honours degree in

biosciences, students should be able

to access and evaluate bioscience

information from a variety of sources

and to communicate the principles

both orally and in writing (e.g.

essays, laboratory reports) in a way

that is well organised, topical and

recognises the limits of current

hypotheses

Subject-specific

(organisms)Threshold standard


biosciences in which the study of

organisms forms a significant

proportion, students should be able to

show knowledge of the basic genetic

principles relating to the organisms

studied

Typical standard


biosciences in which the study of

organisms forms a significant

proportion, students should be able to

describe and analyse patterns of

inheritance and complex genetic

interactions relating to the lives and

development of the organisms

studied




Assessing generic skills

With the assessment of generic skills the emphasis shifts

from mastery of disciplinary specific knowledge and skills

to the use of knowledge and skills in a range of contexts.

Through mastery of specific knowledge generic reasoning

and learning skills should also have developed.

In many systems of higher education there is growing interest

in those generic skills that are relevant to employment.

There are a number of approaches to defining and measuring

what are sometimes termed ‘competences’ or

employability skills.


European Tuning Process

� Tuning makes the distinction between learning outcomes and competences to distinguish the different roles of the most relevant players: academic staff and students/learners.

� Desired learning outcomes of a process of learning are formulated by the academic staff, preferably involving student representatives in the process, on the basis of input of internal and external stakeholders.

� Competences are obtained or developed during the process of learning by the student/learner.




Tuning Competences

Tuning distinguishes three types of generic competences:

• Instrumental competences: cognitive abilities, methodological abilities, technological abilities and linguistic abilities;

• Interpersonal competences: individual abilities like social skills (social interaction and cooperation);

• Systemic competences: abilities and skills concerning whole systems (combination of understanding, sensibility and knowledge; prior acquisition of instrumental and interpersonal competences required).


Employability Skillssss� communication skills that contribute to productive and

harmonious relations between employees and customers;

� team work skills that contribute to productive working relationships and outcomes;

� problem-solving skills that contribute to productive outcomes;

� initiative and enterprise skills that contribute to innovative outcomes;

� planning and organising skills that contribute to long-term and short-term strategic planning;

� self-management skills that contribute to employee satisfaction and growth;

� learning skills that contribute to ongoing improvement and expansion in employee and company operations and outcomes;

� technology skills that contribute to effective execution of tasks.




Measuring generic skills

While there has been a lot of interest in generic skills the first

approaches at system level have occurred as part of

accountability requirements of government funding

agencies.

In response to the Spelling Report in the USA and various

State funding requirements emphasising outcome

indicators a voluntary use of tests of generic skills has

begun in the USA


Voluntary System of Accountability

(VSA).

The American Association of State Colleges and

Universities and the National Association of State

Universities and Land-Grant Colleges developed a

programme called the Voluntary System of

Accountability (VSA).

VSA evaluates core educational outcomes in public

universities and colleges by assessing skills that are

common, multidisciplinary and university-wide.

These generic ‘outcome skills’ are written

communication, critical thinking and analytical

reasoning.




An institution chooses one of three standardized tests:

� The Measure of Academic Proficiency and Progress (MAPP) developed by ETS,

� the Collegiate Assessment of Academic Proficiency (CAAP) developed by ACT and

� the Collegiate Learning Assessment (CLA) offered by the Council for Aid to Education.

The VSA involves looking at the value added in these critical skills as assessed by one of the three tests between fresher and senior years.

The emphasis is on getting evidence about institutional performance rather than individual performance.


OECD AHELO Project

The CLA, has been adopted to test generic skills by an

initiative funded by the OECD and the EU called the

Assessing Higher Education Learning Outcomes

(AHELO) project.

AHELO will potentially be the largest, most comprehensive

assessment of universities yet devised.

The feasibility study is composed of four “strands” of work:

three assessments to measure learning outcomes in

terms of generic skills and discipline-related skills (in

engineering and economics) and a fourth – research

based– value-added strand.




Generic skills, such as critical thinking, analytic reasoning

and problem-solving, can be tested reliably across

institutional, occupational and cultural contexts:

“The kind of competencies to be covered and the methods to

be employed could be similar to those used for the

Collegiate Learning Assessment Project (CLA). A key

advantage is that such competencies are largely

invariant across occupational and cultural contexts

and could be applied across higher education institutions,

departments and faculties. Since such assessments

capture, at least to some extent, the cumulative learning

outcomes of prior schooling, they should be combined with

data on prior learning, for which the CLA provides

methods as well” (OECD, 2007).


The Collegiate learning Assessment

(CLA)

Tasks are derived from a domain of real world jobs

suggested by activities found in education, work, policy,

and everyday practice.

To perform the tasks, test-takers need to think critically

and analytically about the information they are given

and communicate their decisions, judgments, or

recommendations clearly and with appropriate

justification.

The next slide illustrates a performance task.




28-29/11/2010 Teaching, Learning & Assessment in HE 15

The CLA program uses two types of

essay questions:

1. Make-An-Argument

The 45-minute make-an-argument type prompts present students with a point-of-view about a topic of general interest and asks them to respond to it from any perspective(s) they wish. An example of one of these prompts is:

“In our time, specialists of all kinds are highly overrated. We need more generalists -- people who can provide broad perspectives.”

Students are instructed to provide relevant reasons and examples to explain and justify their views.




2. Break-An-Argument

� The 30-minute break-an-argument type prompts

present an argument and asks students to critique it,

including their analysis of the validity of the author’s

arguments, rather than simply agreeing or disagreeing

with the author’s position


Example of a Example of a Example of a Example of a 30303030----Minute PromptMinute PromptMinute PromptMinute Prompt

The University of XX is generally considered one of the best

universities in the world because of its instructors'

reputation, which is based primarily on the extensive

research and publishing record of certain faculty members.

In addition, several faculty members are internationally

renowned as leaders in their fields.

For example, many of the English Department’s faculty

members are to teach at universities in other countries.

Furthermore, two recent graduates of the physics

department have gone on to become candidates for the

Nobel Prize in Physics. And 75 percent of the students are

able to find employment after graduating. Therefore,

because of the reputation of its faculty, the University of XX

should be the obvious choice for anyone seeking a quality

education.28-29/11/2010 18Teaching, Learning & Assessment in HE



The capacity to provide these rich tasks without

overburdening students is in part afforded by recent

developments in information technology.

All the assessments are delivered on an interactive internet

platform that produces a paperless, electronic

administration, and online scoring and reporting of

results.

Answers to the essay type questions are scored using

natural language processing software.


What value generic skills testing?

Generic skills tests are difficult to construct as they usually

provide all the information necessary for problems to be

solved or else they assume common knowledge.

Most are similar to general ‘ability’ tests often used to select

students for university entrance.

If students are high scorers on entry to university they may

not show any significant change as a result of university

study (‘ceiling effect’).




Future Developments

Rather than attempting to assess generic skills in a non-

disciplinary specific manner it makes more sense to

embed these skills in regular course activities and

assessments.

This means designing an assessment programme that

involves assessing outcomes in a range of ways to ensure

that employability skills are developed.


Reflection Exercise

What are the advantages of tests such as the CLA for a

national system that is unsure about consistency of

outcomes?

How can aspects of the generic skills approach be

incorporated in discipline focussed assessment

programmes?

Using the list of employability skills what changes would

you suggest for your university assessment programme?


Documents

Assessing Generic Skills-2-GordonV2