Assessment and Teaching of 21st Century Skills

Claire Scoular

Assessment Research Centre

University of Melbourne

2nd April 2014

San Jose, Costa Rica

2

Outline

1. Background of ATC21S

2. ATC21S: The Latin American Chapter

3. Costa Rican results and outcomes

4. The future of the project in Costa Rica

SECTION 1 : Background of ATC21S

4

CHANGE BEGINS WITH GLOBAL PARTNERSHIP

ATC21S is a joint effort to mobilize international, educational, governmental and business communities to bring 21st-century skills into the classroom.

5

ATC21S GOAL

To develop new assessment approaches matched to new 21C skills and to advise systems, schools, and teachers on the uses of assessment data to help students develop higher order performances

6

ATC21S PROCESS

Phase 1

Conceptualize C21 skills and

education output needs

Phase 2

Skill identification

and hypotheses

Phase 3

Task development

Phase 4

Pilot studies and trials

Phase 5

Dissemination scale and policy

formation

SECTION 1 : Phase 1Conceptualize C21 Skills

8

PREPARING STUDENTS TO SUCCEED IN THE 21ST CENTURY

• Changes in employment trends

• Increase in workforce by 2050

• For Latin America expected increase of 26% (Wang)

9

PREPARING STUDENTS TO SUCCEED IN THE 21ST CENTURY

e are surrounded by technology, and kids are digital natives. They go into the classroom, and they don’t see the same use of technology in their education as they might in their outside lives. If education doesn’t start to address that issue, there will be a disconnect in what kids see in their lives growing up and when they get out into the workplace, which is one of the motivations for us to join this project.”

• — Michael Timms, National Project Manager ATC21S for USA

“We are surrounded by technology, and kids are digital natives. They go into the classroom, and they don’t see the same use of technology in their education as they might in their outside lives. If education doesn’t start to address that issue, there will be a disconnect in what kids see in their lives growing up and when they get out into the workplace, which is one of the motivations for us to join this project.”

— Michael Timms, National Project Manager ATC21S for USA

10

TRANSFORM EDUCATION FOR THE 21ST

CENTURY

11

PHASE 1: CONCEPTUALIZING THE SKILLS

Assemble experts

Define methods

Explore practical and technical needs in the

classroom

Create a new framework

12

SECTION 1 : Phase 2Theoretical and Conceptual Framework

14

SELECTED SKILLS

WAYS OF WORKING

AND THINKING

Collaborative problem solving

TOOLS FOR WORKING AND

LIVING IN THE WORLD

ICT digital literacy

Personal and social responsibility through social networking

15

PROCESS OF ASSESSMENT DEVELOPMENT

Formulation of tasks

Teacher view Cognitive laboratory Pilot Trial

Formulation of developmental continua

Identify the dimension/s Characterise the evidence Draft three profile levels

Definition of constructs

Collaborative Problem-solving ICT Literacy

16

COLLABORATIVE PROBLEM SOLVING: Definition

16

Collaborative problem solving as a 21st Century Skill may be seen as a combination of its component parts. These are collaboration, problem solving and the new technologies currently believed to characterise the 21st century

• Collaboration – working together with a common goal

– Communication– Cooperation

• Problem solving– Knowledge – Strategies

17

Purpose, functions and outcomes

• Why?– When different expertise, knowledge or skills are

both needed and available

• What?– Formulate and express ideas; argue and defend

– Value, review, evaluate

• Outcomes– Problem solution

– Enhanced skills and knowledge

18

COLLABORATIVE PROBLEM SOLVING FRAMEWORK

18

Collaborative problem solving

Social

ParticipationPerspective

takingSocial

regulation

Cognitive

Task regulation

Knowledge buiilding

19

Social skills strands

Participation

Action

Interaction

Task completion

Perspective taking

Adaptive responsiveness

Audience awareness

Social regulation

Negotiation

Self evaluation

Transactive memory

Responsibility initiative

20

Cognitive skills strands

Task regulation

Resource management

Information collection

Systematicity

Ambiguity tolerance

Organisation

Goal setting

Knowledge building

Knowledge acquisition

Represents relationships

Identifies consequences

Hypothesises

21

ICT Literacy

Intellectual Capital

Social Capital

Producer

Consumer

SECTION 1 : Phase 3Task Development

23

Process

Theoretical Framework

Task Design Score

Interpret

&

Report

Intervention

24

25

Laughing Clowns

25

26

Indicative Behaviours

Skill Behaviour in task

Interaction- Chatting with partner

Presence of chat before allowing partner to make a move

Audience awareness- Adapts contributions to increase

understanding for partner

Number of ball moves attempted before stopping and waiting for partner to respond

Reponsibility initiative- Takes responsibility for progress for the

group task

Communicating with partner before first half of balls have been used

Resource management Realises that balls are meant to be shared and only uses allocated half

Relationships - Identifies patterns between information

Both students coming to agreement on how their machine works

27

Balance Beam

28

Indicative Behaviours

Skill Behaviour in task

Action- Active in scaffolded tasks

Student A passes B a mass

Task completion – undertaking part of task individually

Follows instructions, moves 100g to position 4

Responsiveness – responding to contributions of others

Realising that some masses cannot balance. If Student A resends 50 or 500, B returns it immediately

Systematicity- Implements possible solutions to a problem

Trying out all combinations of masses on different beam positions

Solution - Correct answer

Number of successful balances achieved (3 optimum)

29

Skill progressions

SOCIAL

LEVELS PARTICIPATION

Level 4 The student initiates and promotes interaction with their partner; sharing resources,

information and ideas.

The student perseveres in solving the task as shown by repeated attempts and/or

multiples strategies

Level 3 The student is actively participating in the task in familiar and unfamiliar contexts.

The student interacts with their partner more (i.e. responds to communication cues

and requests) and shares information and/or resources with partner.

Level 2 The student is actively participating in the task when it is familiar and/or scaffolded.

There is limited interaction with partner and this is mainly prompted by task

instructions.

Level 1 The student participates in the task but is working quite independently

Task activity is usually directed by system instructions

30

Reporting to Students: Learning Readiness

Real time reports based on empirically developed learning progressions. They will identify a point of readiness to learn for each student.

This type of report can be linked to teaching interventions associated with readiness to learn as indicated by the black bar in the spine of the chart.

SECTION 1 : Phase 4Pilot studies and trials

SECTION 1 : Phase 5Dissemination & Policy Formation

33

34

Dissemination

Resources will now be placed in the public domain

Atc21s.org

35

Trialing Tools

Concept Checking

Cognitive Laboratories

Pilots

Trials

Teacher pedagogy workshops

SECTION 2 : THE LATIN AMERICAN CHAPTER

37

Objectives and Benefits to Costa Rica

• Validate the assessment through pilot and trials

• Results could continue the agenda and lead Latin America in 21st Century skills

• Contributing to large international project

• Gain ideas about future implementation of 21C skills

• Evaluate curriculum

• Regional dialogues with peers as well as feedback from stakeholders in the global initiative

38

ASSESSMENT OF COMPETENCIES OF THE 21ST

CENTURY

Three exemplary contributions:

– Language adaptions

– Customization of ICT Literacy Tasks

– Localized website

39

LANGUAGE ADAPTIONS

40

LANGUAGE ADAPTIONS

41

Componente cognitivo

Descripciones de nivelNivel 5 Resolución refinada de problemas

En este nivel, los estudiantes pueden reconocer errores previos y transferir y usar ese conocimiento en las secciones más complejas de las pruebas. Reconocen qué recursos han causado errores previos y los evitan. Están desarrollando confianza en el manejo de las pruebas y esto les permite corregir o superar las respuestas/acciones de sus parejas. Los estudiantes comparten con sus parejas recursos apropiados y usan estrategias consistentes que han identificado como efectivas en pasos anteriores de la solución del problema. Todo el planeamiento y la mayoría de las estrategias implementadas se realizan dentro del contexto del equipo.

Nivel 4 Trabajo eficiente

En este nivel, los estudiantes comienzan a resolver problemas de manera colaborativa y a usar estrategias de cooperación más efectivas. En este momento pueden simplificar el problema y revisar varias veces la ejecución de secciones previas de las pruebas para determinar si la regla es transferible a las páginas o secciones siguientes. En este nivel, el intercambio de información entre estudiantes incrementa y se hace más eficiente. Su secuencia y el juicio y la error acciones requieren pocos intentos y por lo tanto se concluyó en una cantidad óptima de tiempo / intento. Las completan en un tiempo óptimo. Las acciones parecen estar bien pensadas y planeadas, y cada acción parece tener un propósito. Pueden identificar causas y efectos y usar estrategias adecuadas para lograr una vía de solución correcta. El estudiante dirige y guía a su pareja hacia acciones y recursos apropiados.

Nivel 3 Planeamiento y ejecución estratégicos

En este nivel, los estudiantes pueden completar tareas de menor dificultad de manera correcta e independiente. Son capaces de completar el último paso o sección y ayudar a su pareja en este proceso. Incrementan su trabajo juntos al planear estrategias de trabajo, establecer objetivos y compartir sus recursos. El estudiante tiende a enfocarse en los recursos relevantes y desecha otros que no fueron beneficiosos en intentos anteriores. Son capaces de llegar a respuestas de común acuerdo cuando esto es necesario. En pruebas con varias secciones el estudiante tiende a identificar estrategias y recursos exitosos y aplicar las mismas reglas repetidamente. El estudiante continúa intensificando la investigación secuencial y el comportamiento sistemático en las siguientes secciones de mayor dificultad. Esto lleva a intentos por probar hipótesis junto con

LANGUAGE ADAPTIONS

42

ICT TASK CUSTOMIZATION

Poetry

43

Antarctic Trek

ICT TASK CUSTOMIZATION

44

LOCALIZED WEBSITE

45

Trialing the assessment tools

Trials 2011 (data collection CPS)

16 elementary and high schools

90 11-year-old students, 222 13-year-olds, and 188 15-year-olds

Trials 2013 (ICT and system testing)

9 high schools

776 students

SECTION 3 : SPECIFIC COSTA RICAN OUTCOMES

47

ALL COUNTRIES COMBINED

Valid CPS construct

Item Separation reliability 0.999

Person separation reliability 0.814

Regression variable -0.182

48

ALL COUNTRIES COMBINED

Two separate dimensions (social & cognitive)Social Cognitive

Separation reliability 0.998

Person separation reliability

0.754 0.770

Mean Latent Ability 0.560 -0.824

Correlation between dimensions

0.788

49

Summary of data in Costa Rica

• 575 students were scored

• Sample only not representative• Focus is to validate the test not to compare

countries

50

COUNTRY COMPARISON

CostaRica AllotherCountries

MeanofLatentAbility(StDError) -0.121(0.021) 0.145(0.01)

EAP-PVreliability(SimilartoCronbach’salpha) 0.711 0.783

ReliabilityofItemSeparation 0.993 0.999

The psychometric properties of the tasks in Costa Rica were similar as in other countries, indicating that these tasks can be completed in Costa Rica with little to no language, cultural, or country specific bias.

51

Differences in the difficulty of the items were minor or non-existentOut of 150 items only two were more difficult for CR than other countries

Difficulty = Frequency

Could be explained by time constraints

Hexagons - Answering the final answer correctly

Small Pyramids – Responding to questions from partner

52

Sunflower - presence of chat when growth appears

Survey (self assessment) - Replying to chats

Two items were easier for CR than other countries

However, there are many items that assess each skill and these differences are based on single items so they are not representative of overall ability in a specific skill

There is no particular task that is more difficult for Costa Rica than the other countries.

SECTION 4:IMPLICATIONS AND FUTURE DIRECTIONS

54

Now we have…

Share results for teaching

Assessments to test skills

Two skill constructs

55

Now we need…

Developmental assessments

PD for teaching

Resources for teaching

Strategies for teaching

56

Expected to teach subject content and develop 21st

century skills

• assess and intervene in both key learning area and general capabilities

Required to think beyond discipline expertise

• current teacher training mainly discipline-based pedagogy

• most effective methods of teaching skills in one discipline may not be same in another

Domain specific or general curriculum

57

Literacy Numeracy

ICT CapabilityCritical and Creative

Thinking

Personal and Social Capability

Ethical Understanding

Intercultural Understanding

Science

History

Math

English

LEARNING

AREASGENERAL CAPABILITIES

Australian Curriculum and Reporting Authority, ACARA (2013).

EMBEDDED INTO CURRICULUM : Australian Curriculum

58http://www.moe.gov.sg/media/press/2010/03/moe-to-enhance-learning-of-21s.php

EMBEDDED INTO CURRICULUM : Singapore Curriculum

59CONVIVENCIA

VIDA COTIDIANA Y ARTES INDUSTRIALES

SEXUALIDAD

ÉTICA, ESTÉTICA Y CIUDADANÍA PRIMARIA(MÚSICA, ARTES PLÁSTICAS, EDUCACIÓN FÍSICA,

CÍVICA Y ESTUDIOS SOCIALES)

MATEMÁTICAS

ESPAÑOL

ÉTICA, ESTÉTICA Y CIUDADANÍA SECUNDARIA

(MÚSICA, ARTES PLÁSTICAS, CÍVICA, EDUCACIÓN FÍSICA)

EMBEDDED INTO CURRICULUM : Costa Rica

60

SUSTAINABILITY

61

62

63

64

Poetry

65

Antarctic Trek

66

Why engage in the assessments?

Establish a baseline for academic standards

and

Influence curricula at local, regional, national and global levels

Influence curricula at local, regional, national and global levels

Generate new approaches to evaluation and teaching

© Copyright The University of Melbourne 2014

Claire Scoularc.scoular@unimelb.edu.au

Esther Caree.care@unimelb.edu.au

www.unimelb.edu.au/arcwww.atc21s.orgwww.actrc.org