• Diana Laurillard and Dejan Ljubojevic • London Knowledge Lab

Institute of Education, London

Evaluating learning designs through the formal representation of

learning patterns

A Learning Design Support Environment (LDSE)

IOE/LKLBrock Craft (Technical)Sarah Gelcich (Admin)Diana Laurillard (PI)DejanLjubojevic (RF)Javier Calzada-Prado (Intern)

OxfordLiz Masterman (CoPI)Marion Manton (CoPI)

Birkbeck/LKLGeorge Magooulas (CoPI)Patricia Charlton

LondonMetTom Boyle (CoPI)

LSESteve Ryan (CoPI)Ed WhitleyRoserPujadas(Studentship)

RVCKim Whittlestone (CoPI)Stephen MayCarrie Driver (Studentship)

OUTLINETeachers are interested in learning design and the use of technology but lack the tools they need LDSE

Planning, advice, exemplars – could it also evaluate a design?

•Design principles in the literature

•Learning design patterns – to capture pedagogic principles?

•Representing learning theories as an evaluative framework

•Evaluating the pedagogy in a learning design

•Testing the evaluative capability of the framework

Evaluating learning designs

Design principles in the literatureGood feedback practice:1. helps clarify what good performance is - goals, criteria, expected

standards2. facilitates the development of self-assessment (reflection) in

learning3. delivers high quality information to students about their learning4. encourages teacher and peer dialogue around learning5. encourages positive motivational beliefs and self-esteem6. provides opportunities to close the gap between current and

desired performance7. provides information to teachers that can be used to help shape

the teaching.

(Nicol& MacFarlane-Dick, 2006)

Design principles in the literature1. Encourage contacts between students and faculty 2. Develop reciprocity and cooperation among students 3. Use active learning techniques 4. Give prompt feedback 5. Emphasize time on task 6. Communicate high expectations 7. Respect diverse talents and ways of learning.(Chickering and Gamson, 1991)

Where is the pedagogy that aids learning design?

What active learning techniques?What kind of feedback?

Can pedagogical patterns assist with designing and evaluating the quality of a learning design?

Learning design patterns

• The LDSE( learning design support environment) aims to evaluate the pedagogy in a learning design in terms of theory

• We look for pattern templates that will help to evaluate pedagogy in a design

• A learning design pattern as a learning activity sequence designed to lead to a specific learning outcome

Comparing pedagogy pattern templatesiCOPER Planet TELL

Author & Copyright Credits

Summary/ Thumbnail Context

Rationale Rationale Rationale

Subject/ Discipline Context

Learning outcomes Problem Problem

ForcesGroup size

Duration (part)

Learner Characteristics Audience

Type of Setting

Graphical Representation Diagram

Sequence of Activities

Solution SolutionRoles

Type of Assessment

Where is the effective pedagogy captured?

We need categories within the ‘solution’ or ‘sequence of activities’ to help teacher:• Develop reciprocity and cooperation among students• Use active learning techniques• Give prompt feedbackSome accounts of learning activity sequences are quite elaborate

Categories do not sufficiently discriminate the form of pedagogy that will make the critical difference to learners

Comparing pedagogy pattern templatesiCOPER Planet TELL

Rationale Rationale Rationale

Learning outcomes Problem Problem

Sequence of Activities

Solution SolutionRoles

Type of Assessment

Needs further detailed analysis to support learning design

Pattern for understanding authentic practice

Pattern for helping whole class benefit from individual experiences

Representing learning patternsStructured text-based accounts , e.g. iCoper, Learning Designs

Text cannot be easily interpreted by a program.Computationally defined activities can be (e.g. LAMS).Therefore can be evaluated against theory.

Box-and-arrow diagrams for a sequence of activities, e.g. LAMS, iCoper, ISiS-ECTEL

Representing learning theories

The Conversational Framework: developed to provide a representation of what it takes to learn in education

based on the main design principles for learning and pedagogy drawn from the literature

can test a learning design in terms of which aspects of the framework it covers.

The learner learning

LCTC

LPTP

Thoughts

Action plans

Guidance

OC

OP

Articulating ideas

Others’ ideas

Preparing Outputs

Others’ Outputs

Asking Questions

Investigating

Listening/ Reading

Reflection

Producing

Revising

Working to a goal

Feedback

LC

LP

Adaptation

being supported, actingthinking,

Learning theories represented in the Conversational Framework

LCTC

LPTP

Thoughts

Action plans

Guidance

OC

OP

Articulating ideas

Others’ ideas

Preparing Outputs

Others’ Outputs

Asking Questions

Investigating

Listening/ Reading

Reflection

Producing

Revising

Working to a goal

Feedback

LC

LP

Adaptation

1. Encourage contacts between

students and faculty

4. Give prompt feedback

2. Develop reciprocity and cooperation among

students

3. Use active learning techniques

Learning theories represented in the Conversational Framework

LCTC

LPTP

Thoughts

Action plans

Guidance

OC

OP

Articulating ideas

Others’ ideas

Preparing Outputs

Others’ Outputs

Asking Questions

Acting

Listening/ Reading

Reflection

Producing

Revising

Working to a goal

Information

LC

LP

AdaptationReflectionAdaptation

Learning theories represented in the Conversational Framework

LCTC

LPTP

Thoughts

Action plans

Guidance

OC

OP

Articulating ideas

Others’ ideas

Preparing Outputs

Others’ Outputs

Asking Questions

Acting

Listening/ Reading

Reflection

Producing

Revising

Working to a goal

Information

LC

LP

AdaptationReflectionAdaptation

Learning theories represented in the Conversational Framework

LCTC

LPTP

Thoughts

Action plans

Guidance

OC

OP

Articulating ideas

Others’ ideas

Preparing Outputs

Others’ Outputs

Asking Questions

Acting

Listening/ Reading

Reflection

Producing

Revising

Working to a goal

Feedback

LC

LP

AdaptationReflectionAdaptation

Representing technologies in support of learning

LCTC

LPTP

Thoughts

Action plans

OC

OP

Reflection

LC

LP

Adaptation

Evaluating a learning design

The Conversational Framework: can test a learning design in terms of which aspects of the framework it covers

- IF we can link each LAMS activity (or similar computationally defined activity) in the sequence to part of the framework

LAMS – representing learning design as a temporal sequence of computationally defined activities

Chat

Chat and Scribe

ForumForum and Scribe

MCQNotebookNoticeboardQ&AResources&ForumShare resources

SubmitVoting

To understand the processes within a system

Chat and Scribe

Mapping LAMS activities to theory via CF

LCTC

LPTP

Guidance

OC

OP

Articulating ideas

Others’ ideas

Preparing Outputs

Others’ Outputs

Asking Questions

Investigating

Listening/ Reading

Producing

Revising

Working to a goal

Feedback

LC

LP

Forum and ScribeMCQ

Notebook

Q&A

Share resourcesSubmitVoting

Chat and Scribe

Notebook

Noticeboard

Noticeboard

Q&A

Share resources

Submit

Forum and Scribe

MCQ

Voting

LAMS – representing learning design as a temporal sequence of computationally defined activities

To understand the processes within a system

Interpreting the quality of the design

LCTC

LPTP

Guidance

OC

OP

Articulating ideas

Others’ ideas

Preparing Outputs

Others’ Outputs

Asking Questions

Investigating

Listening/ Reading

Producing

Revising

Working to a goal

Feedback

LC

LP

Notebook

Chat and Scribe

Noticeboard

Q&A

Share resources

Submit

Forum and Scribe

MCQ

Voting

System can interpret design as scoring:4 out of 10 activities;No iterations;Time for each task is good;Sequence is good for ‘awareness’; less good for ‘understanding’

Evaluating the quality of the design

Improving the quality of the design

LCTC

LPTP

Guidance

OC

OP

Articulating ideas

Others’ ideas

Preparing Outputs

Others’ Outputs

Asking Questions

Investigating

Listening/ Reading

Producing

Revising

Working to a goal

Feedback

LC

LP

Chat and Scribe

Noticeboard

Q&A

Voting

System can propose:Adding in designs or patterns that complete the other cycles;Checking that the sequence follows the motivating cycles of iteration

Testing the framework

The Conversational Framework: can differentiate between sequences in terms of coverage of the framework, amount of iteration, and effective ordering of activities

But to what extent can it recognise the relative effectiveness of learning designs identified in the literature?

Pattern 1

Contrasting cases text

Analyse data

S analyses data

S produces graphs

S listens to lecture

S selects and predicts

Very accurate MCQs

High transfer to prediction

Pattern 2

Theory and examples text

Read text

S reads text

S produces summary

S listens to lecture

S selects and predicts

Very accurate MCQs

Much lower transfer to prediction

Comparing learning design patternsIntended learning outcome: To predict accurately the outcomes for a hypothetical experiment on memory [Schwartz and Bransford,1998]

Learning activities

Reading

T sets a task goal

Investigating

Producing output

Listening

Working to achieve goal

Actual outcomes

Pattern 1

Contrasting cases text

Analyse data

S analyses data

S produces graphs

S listens to lecture

S selects and predicts

Very accurate MCQs

High transfer to prediction

Pattern 2

Theory and examples text

Read text

S reads text

S producessummary

S listens to lecture

S selects and predicts

Very accurate MCQs

Much lower transfer to prediction

Critical differences between patterns are not just the nature of the activity, but the internal relations between contents of activities in the sequence

Interpreting the quality of the designs

LCTC

LPTextData

Guidance

3. Summarising3. Investigating

1. Reading

4. Producing

Revising

2. Working to a goalFeedback

LC

LP

System must be able to differentiate between producing graphs and summaries;And between providing Text and Data in the practice environment

5. Listening

4. ProducingReflectingAdapting

Asking Questions

Representing the range of good pedagogy

LCTC

LPTP

Guidance

OC

OP

Articulating ideas

Others’ ideas

Preparing Outputs

Others’ Outputs

Asking Questions

Investigating

Listening/ Reading

Producing

Revising

Working to a goal

Feedback

LC

LP

Notebook

Chat and Scribe

Noticeboard

Q&A

Share resources

Submit

Forum and Scribe

MCQ

Voting

A System like LAMS can represent good pedagogy in terms of type and range of learning activity

Cannot represent good pedagogy within the practice environment – simulation, model, digital library, data repositories, type of task and type of feedback

SummaryDesign principles in the literature

Learning design patterns

Representing learning theories as a framework for design

Evaluating the pedagogy in a learning design

Testing the evaluative capability of the framework

Too general to be helpful for teacher-designers

May miss categorising the critical pedagogy

Makes it possible to relate theory to computational activities

Needs mapping from framework to LAMS or similar system

May not capture all the key differences in effectiveness

Pedagogy operates both across and within activities• We need user-editable patterns for activity sequences

– One role for learning technologists is to create the means for teachers to adopt and adapt pedagogic patterns for activity sequences – e.g. LAMS

– One role for teachers is to these to adopt, adapt and share pedagogic patterns for activity sequences

• We need customisable programmed patterns for activities– Another role for the learning technologist is to discover and

create the means for teachers to adopt and adapt pedagogic patterns for practice environments – e.g. GLO tool, NetLogo

Evaluating learning designs through the formal representation of learning patterns

Made publicly available by ALT under a Creative Commons Attribution-Non-Commercial 2.0 UK: England & Wales license http://creativecommons.org/licenses/bync/2.0uk/