ALOA conceptual model: Aligning Learning Outcomes and Assessment

Click to edit Master title style

• Click to edit Master text styles– Second level• Third level

– Fourth level» Fifth level

03-05-23

Evaluation of the Application of E-learning Methodologies to the

Education of EngineeringRita Falcão de Berredo

Programa Doutoral em Media Digitais

Especialidade Criação de Audiovisual e de Conteúdos InterativosUniversidade do Porto :: FEUP

Porto, 21st of January of 2013




03-05-2303-05-23

Main concepts

ASSESSMENTLEARNINGOUTCOMES

What do we hope students will learn?

How do we know that they have learned?

What should be assessed?

ALIGNMENT




03-05-2303-05-23

Before…

ASSESSMENT

LEARNINGOBJECTIVES

What will we teach our students?

ALIGNMENT

CONTENT




03-05-2303-05-23

• Demand for new skills and competences• Mobility and recognition• Quality approach and accreditation

Why change?...

THE FOCUS IS ON THE STUDENTS!




03-05-2303-05-23

• Students may have prior knowledge and skills• Students must learn what they need for the

profession and future• Provide evidence of acquired competences

And now what?

How can we help teachers select an adequate assessment strategy for the learning outcomes?

ASSESSMENTLEARNINGOUTCOMES

ALIGNMENT




03-05-2303-05-23

• Students• Faculty• Course and curriculum developers• Internal quality and accreditation• Mobility and recognition• Employers• General public

To whom is this relevant?




03-05-2303-05-23

Why this theme?

European project VIRQUAL




03-05-2303-05-23

Learning outcomes are statements of what a learner is expected to know, understand and/or be able to demonstrate after completion of learning. (AHELO - Assessment of Higher Education Learning Outcomes by OECD)

Learning Outcomes

A common language, building blocks, genetic code




03-05-2303-05-23

Assessment: Any procedure used to estimate student learning for whatever purpose. (Brown et al)

Assessment

e-Assessment is the use of ICT and the Internet in particular for the assessment of learning, including design, delivery and/or recording of responses. (JISC)




03-05-2303-05-23

Alignment

Alignment: The level of correspondence between objectives, instruction and assessment.

(Anderson et al)




03-05-2303-05-23

Initial Research

LEARNINGOUTCOMES

Where can I find LOs in Engineering Education?

Courses Programs Qualification Frameworks

ASSESSMENTALIGNMENT




03-05-2303-05-23

• Four qualification frameworks– ABET: Accreditation Board for engineering and

Technology – EUR-ACE: European quality label for engineering

degree programmes– CDIO: Conceiving, Designing, Implementing,

Operating– EQF: European Qualification Framework

Learning Outcomes in Engineering




03-05-2303-05-23

Initial Research

ASSESSMENT Which types of assessment should be considered?

peer

formative

group

essay

project

email

chat

MCQ

exam




03-05-2303-05-23

• Multiple choice questions • Short Answer Questions • Problems• Essays• Practical work• Reflective practice

Assessment




03-05-2303-05-23

Alignment

Specific

Assessment tasks

Course and Programme

LOs

General

General assessment

methods

LOs QFs

ALIGNMENT




03-05-2303-05-23

Alignment scenariosAt course level…




03-05-2303-05-23

Simple problem…?




03-05-2303-05-23

The ALOA conceptual modelAligning Learning Outcomes and Assessment

ALIGNMENT




03-05-2303-05-23

The rBloom Matrix

Adapted from Anderson et al




03-05-2303-05-23

The ALOA conceptual modelAligning Learning Outcomes and assessment

ALIGNMENT

rBLOOMMATRIX




03-05-2303-05-23

• Write LOs statements

• Define content

• Define learning activities

• Define assessment

• Ensure alignment

Teacher with NEW course

ALOAHELP!!




03-05-2303-05-23

Analysis of LOsThe student should be able to describe the main

components of a personal computer.

NOUN / KNOWLEDGE

VERB / COGNITIVE PROCESS

rBloomMatrix




03-05-2303-05-23

Compare matrixes

ALOADB

ASSESSEMENT 1

ASSESSEMENT 2

ASSESSEMENT 3

ALIGNEDASSESSMENT STRATEGY

MATRIXES OF LOs of course

ALOA SYSTEM

LOsMATRIXESLOs

MATRIXESLOsMATRIXES




03-05-2303-05-23

• LOs statements written

• Content defined

• Existing learning activities

• Assessment strategy defined

• Verify alignment

Teacher with EXISTING course

ALOA

HELP!!




03-05-2303-05-23

ALOADB

ASSESSEMENT 1

ASSESSEMENT 2

ASSESSEMENT 3

ASSESSMENT METHODS(EXISTING)

MATRIXES OF LOs of course

ALOA SYSTEM

ALIGNMENT REPORT

e.g.: Accreditation purposes

Compare matrixes

LOsMATRIXESLOs

MATRIXESLOsMATRIXES




03-05-2303-05-23

Work overview• Four case studies• LOs and assessment methods• Application of the ALOA• Improvement of ALOA




03-05-2303-05-23

• The ALOA conceptual model• A structured approach to assessment• Linking Bloom and assessment• Bloom as a translation for LOs• Clarification of the alignment concept

Major contributions of this work to the field of study

Reflection Quality Mobility




03-05-2303-05-23

Future work




03-05-23

Thank you.




03-05-2303-05-23

• Describing– Analyzing and describing the LOs – Analyzing and describing assessment methods

• Aligning– Aligning assessment and LOs

• Mapping– LOs of different levels (programme/course/QF) – LOs of different sources (recognition)– Assessment

Roles of the rBloom Matrix




03-05-2303-05-23

• In terms of knowledge it is considered that the design process includes mainly procedural and conceptual.

• The cognitive tasks will include planning and producing an application according to criteria. Some interpretation of criteria will be necessary as well as iterative evaluation of the application against the given requisites.

Step 1: Analysis of LOsStudents should be able to design and implement a simple distributed application.

KNOWLEDGE

Procedural Conceptual

COGNITIVE PROCESS

Understand: interpretingApply: implementingEvaluate: checking, critiquingCreate: planning, producing




03-05-2303-05-23

STEP 2: Build rBloom matrixKNOWLEDGE

Procedural Conceptual

COGNITIVE PROCESS

Understand: interpretingApply: implementingEvaluate: checking, critiquingCreate: planning, producing




03-05-2303-05-23

Scenarios of applicationApplication and stakeholders

Proposal of internal alignment

Single courseTeachers, curriculum developers

Verification of internal alignment

Single courseTeachers, curriculum developers, accreditation

Vertical alignment Course/programme/QFcurriculum developers, accreditation

Horizontal alignment Several courses, prior learningTeachers, recognition and mobility




03-05-2303-05-23

• RQ1) Which type of Learning Outcomes in the field of Engineering are relevant and should be considered? – Defining and describing LOs in engineering

• RQ2) Which are the e-assessment methods that should be considered? – Defining and describing assessment methods and

e-assessment tasks

Research questions




03-05-2303-05-23

• RQ3) What type of intended Learning Outcomes can be measured by assessment methods? – Defining a model of achieving alignment between

learning outcomes and assessment• RQ4) Is it possible to propose specific

assessment strategies for each type of LO in EE?– Applying the model to link specific learning

outcomes with specific assessment tasks

Research questions




03-05-2303-05-23

Alignment criteriaCriteriaMatch LO statement = TLA = assessment

Emphasis LOs have different levels of importance

Coverage The content of the course should be covered

Precision A complex LO may include different competences with different values.




03-05-2303-05-23

Backwash effect (Biggs)Perspective of the teacher

Perspective of the student




03-05-2303-05-23

• Difficulties of applying the model• Testing the templates • Validity of selected list of assessment • Validity of classification and description of the

assessment methods • Validity of selected QFs• Validity of tool

Case studied




03-05-2303-05-23

Assessment cycle




03-05-2303-05-23




03-05-2303-05-23

Validity and reliability

Precision Accuracy




03-05-2303-05-23

• Remember• Understand• Apply• Analyse• Evaluate• Create

MCQ




03-05-2303-05-23

• speculative• quote to discuss• assertion,• write on• describe/explain• discuss• compare• evaluate• problem

Essays




03-05-2303-05-23

• Simple closed ended• Complex closed ended• Open ended

• Routines• Diagnosis• Strategy• Interpretation• Generation

Problems




03-05-2303-05-23

• Select crucial evidence• Explain methods, procedures and relationships• Present arguments• Describe limitations of data• Formulate valid conclusions• Identify assumptions• Formulate hypothesis• Formulate action plans

SAQs




03-05-2303-05-23

• Concrete experience • Reflective observation• Abstract Conceptualisation • Active Experimentation•

Reflective practice




03-05-2303-05-23

MCQ Computer based test / online testing, Optical reading, CAT: Computerized Adaptive Testing

Essays File Upload, Essay Question in online exam, Discussion ForumPublished media, Wiki, Concept maps, Videoconferencing, Chat

Problem solving Computer based test / online testing, File upload, Chat, Concept maps and Diagrams, Simulation, Scenario-based activity

Practicals File upload, Computer based tests/Online testing, Video file, animations and sequence of images, Videoconferencing, Diagrams Publish media or wiki, Chat and discussion forum, Virtual Labs and Remote Labs, Simulation, Scenario-based activity, Game based learning

SAQs Computer based test / online testing, Chat or Forum, Concept maps and Diagrams

Reflective Practice

Portfolios