UTS CRICOS PROVIDER CODE: 00099F

ANALYTICS FOR HOLISTIC EDUCATIONAL IMPROVEMENT:

cic.uts.edu.au

THE STORY IN A SLIDE

•  Organisations are complex and ‘messy’

•  There are different ‘ways of knowing’ through which humans seek knowledge and represent that knowledge through data.

•  Using data to derive actionable insights for leadership decisioning must take account of such complexity and be aligned to purpose & timely.

•  Case study – hierarchical process modelling is a way of structuring data for leadership decisioning

•  The perimeta dashboard – what it looks and feels like

•  Actionable insghts for leadership decisioning

•  What next? Key issues?

cic.uts.edu.au

http://www.youtube.com/watch?v=jBl6ALNh18Q

Jurgen Habermas b 1929

Knowledge seeking human interests Empirical analytical Hermeneutical Emancipatory

DIFFERENT FORMS OF RATIONALITY

cic.uts.edu.au

Learning Analytics UNESCO Policy Briefing — Simon Buckingham Shum

Empancipatory Rationality individual learner progress

Hermeneutical Rationality institution-wide contextual

Strategic Rationality region/state/national/international

4

DATA FOR LEARNING ANALYTICS

Learners

Leaders/Teachers

Policy Makers Researchers

Measurement models for learning journeys: using different types of data and different quality criteria

Narrative; self-report; interpretation. Self report questionnaires, observation, self-assessment. Tests of knowledge, depths of understanding, MCQs, visualisations. Community of Peers, public presentations, evaluations Hierarchical Process modelling

Identity purpose

motivation

Personal qualities

Knowledge structuring

Performing

Authenticity Trustworthiness Credibility Coherence Reliability Validity Replicability Usefulness

BACKGROUND

The Education Charter

The Echo Project

Three Oasis Academies

WHAT TO DO?

"Here is Edward Bear, coming downstairs, bump, bump, bump, on the back of his head, behind Christopher Robin. It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it"

Alan Alexander Milne The opening paragraph of “Winnie-the-Pooh”, 1926

Systems thinking – managing complexity Developing resilience – becoming learning organisations Responding to emergence – unpredictable outcomes Taking responsibility for our own development Engaging all stakeholders in a networked improvement community Using rich data for actionable insights

KEY QUESTIONS

How can we measure the full range of what we value in schools? How can we collect and manage evidence to understand the big picture as well as the detail? How can we lead and manage in conditions of complexity? How can we used data for real-time leadership decisioning as well as for evaluating performance

Macro Layers and feedback loops……

CommunityLearning

Leaders Learning

Teachers Learning

Students Learning

Learning is a core process at multi-levels in learning organisations

Blockley, D. (2010) The Importance of Being Process, Civil Engineering and Environmental Systems, 27(3)189-199

cic.uts.edu.au

THE COMPLEX PROBLEM OF CREATING LEARNING COMMUNITIES

IMAGINE ……

13

Define purpose

Analyze system

Design Model

Design measurement parameters for

evaluating success

Collect data according to

measurement model over time

Enter into HPM model and analyse

performance

Feedback to system leaders

Explore solutions as prototypes

Implement prototypes and

evaluate

Hierarchical Process Modeling

for a Holistic Approach to

System Improvement

P

A SD

What sort of design should we use to model our learning communities?

HIERARCHICAL PROCESS MODELLING

cic.uts.edu.au

Purpose = How (who what where when)

Outcome   1.1 Establishing and sustaining a group of high achieving learning communities that enables everyone to realise their full potential and refuses to put limits on achievement  

2. Outputs   2.1 Developing the learning of students so that they realise their full potential  

2.2 Developing the learning of teachers so that they realise their full potential  

2.3 Developing the learning of leaders so that they lead the learning of teachers and students effectively  

2.4 Engaging parents/carers effectively in the learning activities of the Academy and in supporting the learning of their children  

3. Inputs  

3.1

Feel

ing

valu

ed a

nd in

clud

ed in

our

com

mun

ity  

3.2

Feel

ing

part

of a

com

mun

ity th

at fo

cuse

s its

ac

tiviti

es o

n le

arni

ng a

nd a

chie

vem

ent  

3.3

Kno

win

g th

at I

am b

ecom

ing

an e

ffect

ive

lear

ner  

3.4

Kno

win

g th

at w

e ar

e be

ing

help

ed to

reac

h ou

r ful

l pot

entia

l  3.

5 M

aint

aini

ng a

goo

d ra

te o

f pro

gres

s at

key

tra

nsiti

ons,

for e

xam

ple,

prim

ary

to s

econ

dary

sc

hool  

3.6

Ach

ievi

ng re

sults

that

mee

t my

aspi

ratio

ns

and

expe

ctat

ions

 

3.7

Feel

ing

that

my

succ

esse

s ar

e re

cogn

ised

 3.

8 C

ontri

butin

g ac

tivel

y to

a c

omm

unity

whi

ch

focu

ses

on s

ervi

ce to

oth

ers  

3.9

Con

tribu

ting

activ

ely

to a

com

mun

ity w

here

sh

ared

lead

ersh

ip is

pro

mot

ed  

3.10

Pro

vidi

ng e

vide

nce

that

trai

ning

and

de

velo

pmen

t opp

ortu

nitie

s ha

ve h

elpe

d m

e to

im

prov

e m

y cl

assr

oom

pra

ctic

e an

d/or

my

effe

ctiv

enes

s as

a le

ader

and

man

ager  

3.11

Con

tribu

ting

activ

ely

to a

com

mun

ity th

at

lear

ns to

geth

er, s

hare

s w

hat w

orks

bes

t and

kn

ows

wha

t it n

eeds

to d

o to

impr

ove  

3.12

Fee

ling

incl

uded

and

invo

lved

with

my

child

ren’

s ed

ucat

ion

at th

e A

cade

my

and

know

ing

that

ther

e is

an

open

doo

r for

con

tact

whe

n I

need

it  

3.13

Bei

ng e

ffect

ive

role

mod

els

as le

ader

s of

le

arni

ng  

Sources of

evidence  Students   Teachers   Leaders   Parents/ carers  

PURPOSE H

ow

data

A Hierarchical Model: Defined by Purpose

What type of data do we need to collect to measure the core processes which fulfil our purpose?

20

Define purpose

Analyze system

Design Model

Design measurement parameters for

evaluating success

Collect data according to

measurement model over time

Enter into HPM model and analyse

performance

Feedback to system leaders

Explore solutions as prototypes

Implement prototypes and

evaluate

Hierarchical Process Modeling

for a Holistic Approach to

System Improvement

P

A SD

CAPTURING DATA FOR CONTINUAL IMPROVEMENT

cic.uts.edu.au

Surveys for Open Learning Analytics (SOLA ) Platform hosting apps to support self-directed change

An integrated platform for systemic improvement

DATA WAREHOUSING FOR SYSTEM WIDE RESEARCH

LEGACY SYSTEMS

BLACKBOARD

HIERARCHICAL PROCESS MODELLING

KNOWLEDGE STRUCTURING TOOLS

SOCIAL LEARNING ENVIRONMENTS

Stakeholders Type of data Focus 1 Y7 students Numerical scores per dimension Learning power 2 Y8 & 10

Students Numerical values for each question students as learners and

values & citizenship 3 Y9 students Stories rated by researchers Personal transformation 4 Y11 leavers Numerical values for each question Aspirations, trajectories

5 Parents/carers Numerical values for each question Participation

6 Teachers Numerical values for each question Impact of professional learning on student outcomes

7 Leaders Numerical values for each question Leaders as learners

8 KPI s Numerical values of performance Behavioural indicators

9 Y11 Numerical values of performance Attainment

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Taking responsibility for my own learning over time through defining my purposes, understanding and managing my feelings, knowing how I go about learning & planning my learning journey carefully.

Making connections between what I already know & new information & experience. Making meaning by linking my story, my new learning & my purpose.

Wanting to get beneath the surface & find out more. Always wondering why and how.

1

Dependent Closed Openness to learning

An emotional orientation of being open & ready to invest in learning, having flexible self-belief, willing to persist & manage any self-doubt. A necessary pre-requisite for developing resilience in learning

Using my intuition & imagination to generate new ideas & knowledge. Taking risks & playing with ideas and artefacts to arrive at new solutions.

Being part of a learning community at work, at home, in education & in my social networks. Knowing I have social resources to draw on when I need them

Having the optimism & hope that I can learn & achieve over time. Having a growth mindset; believing I can generate my own new knowledge for what I need to achieve

Being able to work with others, to collaborate and co-generate new ideas and artefacts. Being able to listen and contribute productively to a team.

THE NATURE OF UNCERTAINTY

Fuzziness Incompleteness Randomness

.. COMBINED WITH PERFORMANCE MEASURES THAT COPE EXPLICITLY WITH UNCERTAIN EVIDENCE

1 0

Evidence that A is successful Lack of Evidence

Evidence that A is not successful

We don’t really know what is going on!

Performing well with little uncertainty

Performing poorly with some uncertainty

From Likert type scales to measures of uncertainty

Hall et al’s (2004) mapping from linguistic descriptions of ‘performance’ and ‘confidence in judgement of performance’ to interval values of performance

So what does our Perimeta model tell us about Parents/carers ..…an example

29

Define purpose

Analyze system

Design Model

Design measurement parameters for

evaluating success

Collect data according to

measurement model over time

Enter into HPM model and analyse

performance

Feedback to system leaders

Explore solutions as prototypes

Implement prototypes and

evaluate

Hierarchical Process Modeling

for a Holistic Approach to

System Improvement

P

A SD

cic.uts.edu.au

PERIMETA DASHBOARD FEATURES

Input data – one node for each Respondent and each Question

Results compiled for each Question

Results compiled for each Gender

Results overall for specified area of performance

Results compiled for each Academy

cic.uts.edu.au

PERIMETA MODELS FROM DIFFERENT EVIDENCE STRANDS

S7: Learning of leaders (SLT)

S4: Learning of students in Year 11

S6: Learning of teachers

S5: Quality of relationship w. parents/ carers

cic.uts.edu.au

PERIMETA MODEL RESULTS

cic.uts.edu.au

DRILLING DOWN INTO THE EVIDENCE TO UNDERSTAND KEY ISSUES

Statement POS I was helped and supported when I struggled 0.98 My teachers taught me well 0.90 The Academy made good use of technologies 0.90 I was valued and included 0.89 I was often challenged to do better 0.89 There were great facilities 0.89 I felt part of a learning community 0.89 I developed responsibility towards others 0.89 I had a wide range of opportunities for learning 0.89 I felt safe 0.89

Outcome Establishing and sustaining learning communities Outputs Developing the learning of students so that they realise their full potential Inputs Feeling valued and

included in our community

Feeling part of a community focused on learning & achievement

Knowing that I am becoming an effective learner

Knowing that we are being helped to reach our full potential

Statement UNC I supported activities that would improve things for other people

0.38

I led activities 0.34 I am looking forward to the move to Sixth Form, FE, training or work

0.33

My parents/carers felt involved 0.15 Everyone worked very hard to make it a great place to learn and to do well

0.15

I enjoyed my time 0.15 I could not have gone to a better school 0.14 I was offered opportunities for leadership 0.14 I never felt that anyone was putting limits on what I could achieve

0.13

My teachers accurately predicted my GCSEs 0.13

Statement NEG I am looking forward to the move to Sixth Form, Further Education, training or work

0.18

I led activities during my time at the Academy 0.17 No others above 10%

THE PERIMETA TOOLKIT SUPPORTS COLLABORATIVE ACTION TO IMPROVE LEARNING

Focusing on PERFORMANCE vs broad outcomes

Understanding COMPLEX PROCESSES

Weighing up uncertainty of EVIDENCE

Informing robust CHOICES

Enabling robust & collaborative ACTION

KEY FEATURE: “Italian Flag” evidence model & rich uncertainty calculus

KEY FEATURE: Excel interface for linked data and predictive models

KEY FEATURE: Hierarchy of interactions focused on outcomes

So how do we respond to what Perimeta is telling us as a learning community?

37

Define purpose

Analyze system

Design Model

Design measurement parameters for

evaluating success

Collect data according to

measurement model over time

Enter into HPM model and analyse

performance

Feedback to system leaders

Explore solutions as prototypes

Implement prototypes and

evaluate

Hierarchical Process Modeling

for a Holistic Approach to

System Improvement

P

A SD

Professional Enquiry through Rapid Prototyping to address key challenges highlighted by Perimeta….

Plan

Act Study

Do

…..within a disciplined evaluation framework

Experiential knowing, hunches, ideas

P D

SA

P D

SA

P D

SA P D

SA

P D

S A

P D

S A

P D

S A P D

S A

P D

S A

P D

S A

P D

S A P D

S A

Measurable System Improvement: New Knowledge and Know How

Bryk, A. Gomez L. Gunrow A. (2010) Getting Ideas into Action: Building Networked Improvement Communities in Education, Carnegie Foundation for the Advancement of Teacher Education, Carnegie, Stanford.

How can we use technology to enhance our ability to develop organisational learning through Perimeta

cic.uts.edu.au

AN ACADEMY WITH SYSTEMIC ANALYTICS

Strategic Progress

Learner Dispositions

Teacher Dispositions

Stories of Change & Evidence

Reflective Enquiry Blogs

Academy X

cic.uts.edu.au

AN ACADEMY NETWORK WITH SYSTEMIC ANALYTICS

42

Analytics Integration

Evidence Hub

Data   Data   Data   Data  

cic.uts.edu.au

Share  and  debate  evidence…  

About…  

SYSTEMS LEARNING & LEADERSHIP EVIDENCE HUB HTTP://SYSLL.EVIDENCE-HUB.NET

cic.uts.edu.au

SYSTEMS LEARNING & LEADERSHIP EVIDENCE HUB HTTP://SYSLL.EVIDENCE-HUB.NET

44

Replay Oasis/Univ Bristol/Open Uni event: http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning

cic.uts.edu.au

Data Sets + Data Streams

LEARNING ANALYTICS HUB (3.0)

Learning Analytics System Simulations

Recommendation Engines Social Learning Environments

Online Tools designed to measure core processes and to

stimulate self-directed change

Buckingham Shum, S. & Deakin Crick, R. (2012) Learning Dispositions and Transferable Competencies: Pedagogy, Modelling, and Learning Analytics, Proceedings of LAK12: 2nd International Conference on Learning Analytics & Knowledge, April 29 - May 2, 2012, Vancouver, BC. ACM: New York. ISBN 978-1-4503-1111-3.