Vurdering og læring – får vi det til?Assessment and Learning – Fields Apart?

Summary and Comments

Jan-Eric GustafssonProfessor II

Purposes of assessment

• of – for – as learning at individual – institutional – system levels

• Assessments typically serve multiple purposes and purposes often change over time

• Different stakeholders ascribe different purposes to the same assessment

• The validity of an assessment must be evaluated against its purpose(s).

Validity

• Amazing development of neo- and post-Messick models and of argumentation-based models

• The chain is only as strong as its weakest link• The generalisation link: Strong in standardized, itemized,

assessments; weak in complex, authentic assessments • The extrapolation link: Strong in complex, authentic

assessment; weak in standardized, itemized, assessments• If generalisation and extrapolation are both important,

validity is difficult to achieve

Methodological approaches• Much methodological debate starts from a dichotomy

between quantitative and qualitative methods. However, Ercikan and Roth (2006) argued that this dichotomy is fallacious:–Quantitative research is typically based on qualitative distinctions in

data generation and in conceptualisation–Much qualitative research aims at, and does achieve, generalizations

• They proposed instead a continuous scale that goes from the lived experience of people on one end (low-level inference) to idealized and generalized patterns of human experience on the other (high-level inference). –Low-level inference research is characterized by contingency,

particularity, being affected by the context, and concretization, –High-level inference research is characterized by standardization,

universality, distance, and abstraction.

A metaphor for low- and high-level inference research: weather and climate• Weather affects our daily lives, how we dress, what we do and talk

about. We may adapt to weather but there is not much we can do about it. In the short run we can predict weather, but beyond a week or so weather is unpredicatble.

• Climate is generalized weather over a longer period of time. We experience weather, and through aggregating these experiences, we get a sense of climate. In a more precise manner scientists define climate as aggregate aspects of weather, using indicators such as mean temperature and mean rainfall. Thus, climate is an abstraction.

• While weather is unpredictable and chaotic, climate and climate changes are stable phenomena, which we can be understand theoretically and for which empirically based models may be constructed, that predict long-term development.

• In terms of this metaphor, high-level inference research is concerned with climate, while low-level inference research is concerned with weather.

The foundation of high-level inference research: aggregation• Climate is a social construction, and research on climate is

based on a highly developed technology of devices for generating data, on agreed-upon definitions, and analytical models. But the fundamental idea is to aggregate multiple observations of different aspects of weather.

• In the same manner quantitative research in education is based on aggregation of observations of different aspects of phenomena of teaching and learning: Two types of aggregation:– aggregation over observational units, such as students, classes, schools,

municipalities and school-systems. – aggregation over different observations for the same unit, such as when

students are followed and observed over longer periods of time.

High-level inference procedures

• Aggregation of observations is used to create measures of abstract constructs, such as reading literacy, end-of-term grades, or a licensing assessment.

• With aggregation over observations the generalization link is strengthened, according to the general principle that combination of fallible observations from different contexts and contents yields a more generalizable score

• However, agggregation over contexts and contents weakens the extrapolation link because the more abstract measure is not easily connected to such real-life situations that we want to make extrapolations to.

When can we take advantage of high-level inference procedures?

• In construction of standardized tests of and for learning

• In class-room based assessment of learning over longer periods of time

• In aggregation of student responses to institutional and system levels

When can we take advantage of low-level inference procedures?

• In class-room based assessment for learning, focussing on specific content and tasks (teacher assessment, supported by qualitative information and high pedagogical content knowledge)

Some conclusions

• When generalization is of little importance, systematic low-level inference procedures support assessment for learning and short-term assessment of learning.

• When generalisation is important high-level inference procedures are essential

How does assessment support learning?

• Through assessment for learning at individual, institutional and system levels

• Through assessment as learning, developing student metacognition and knowledge how to monitor their own learning

• Through intended and unintended impact on teaching and learning

• However, we need to be aware of the problem of multiple purposes, and the risk of corruption caused by high stakes

Thank you very much for your attention!