LAK17 Reflective Writing Analytics

UTS CRICOS PROVIDER CODE: 00099F utscic.edu.au

REFLECTIVE WRITING ANALYTICS FOR ACTIONABLE FEEDBACKAndrew Gibson, Adam Aitken, Ágnes Sándor (XRCE), Simon Buckingham Shum, Cherie Tsingos-Lucas, and Simon Knight

[email protected] • @andrewresearch

REFLECTIVE WRITING

What is reflective writing and why is it important?

Mental processing (Moon,1999)Self referential bending back upon oneself (Ryan,2013; Archer,2010)A way of navigating, proceeding in the face of uncertainty (Reidsema et. al., 2010)Also… Mezirow, Schön

Kat – Ponder (flikr)

EXAMPLEIt is hard to believe that I started placement almost three months ago. It has definitely been full of experiences which have shaped my understanding of the role of a pharmacist. During my time at Chester Square Pharmacy I was able to observe different sides of pharmacy including pharmacist-patient interactions, retail, administration and the clinical aspects. To be honest my first thoughts going into placement were negative. I dreaded the idea of having to interact with patients as well as engaging with employees of the pharmacy. I felt that my lack of experience would cause an inconvenience and I would leave a negative impression in front of the pharmacist and other employees. However, I came to realise that my preceptor is an exceptional teacher and as the weeks progressed I begun to look up to him as a mentor. Despite my lack of experience my preceptor ensured I observed different counselling situations and even encouraged me to engage directly with patients from the early days of placement. By allowing me to engage with patients I have been able to build on important communication skills. My preceptor would also observe my weaknesses and bring them to my attention so that each week we would work on overcoming my weaknesses and turning them into strengths.

EXAMPLE

ACTIONABLE FEEDBACK• Reflective writing requires a departure from objective analytical academic

writing for an external audience• It is personal, subjective, and speaks primarily to self• For students » adjusting to the style requires timely feedback • For educators » feedback for large numbers of students is difficult

• How can students be given timely actionable feedback on their reflective writing in a way that is scalable?

HUMAN != NLP FEATURES

• Computers can help with timely feedback and address issues of scalability, however…

• We need to avoid the assumption that human understanding of reflective writing can be directly represented by computational features.

• We need a way of linking our understandings of reflective writing within learning to computational techniques for the analysis of natural language.

Jim Cortez – Pondering (flikr)

REFLECTIVE WRITING ANALYTICS (RWA)• Our approach to bridge learning features to computational text features:

1. Conceptual framework based on reflection and reflective writing literature

2. Genre approach linking conceptual framework to high level rhetorical (reflective) moves.

3. Construction of high level features from a combination of NLP techniques (and low level tools).

4. Communication of high level constructs as feedback to writer

• More than a computational model… a complex of different ‘models’

CONCEPTUAL FRAMEWORK

Developed…• From the reflection and reflective writing literature• Through Psychological and Sociological lenses• A framework from common dimensions of prevailing theory

Intention

What am I going to doabout it? In which

direction do I head next?How do I want to

change? INTENTIONand PREDICTION:

change likely to lead tocertain future benefits.

MODALS : will can, hopeto

Integration

What impact will thishave on me and mygoals/aspirations?

CAUSALITY: Impact onself in context

What other ideas could Iuse to improve myself?

CITATION andEVALUATION

(appraisal) of ideas

What do others suggestis a way forward? How

do others address thesechallenges? CITATIONS

and hedging (could,might)

How can I learn from thepast? How can I learn

from other perspectives?What perspectives are

best for me?REFLECTING on

beginning of learningexperniece

Internalisation

What do these feelingsay about me?CAUSALITY:

explanations forimpressions and feelings

How is this a problem?DESCRIPTIVE

NARRATIVE. How doesthis challenge me?

CAUSALITY: effect ofexperience on self

What should I improve?Why do I need to

improve? Causality:explain changed belief or

approach, newunderstanding,competence,

empowerment

How can I improve?CITATIONS used to

present a theory for selfimprovement. PAST/

NOW contrast to showchange, learning,

Interpretation

What does it mean forme? Why is itsignificant?

COMPARATIVESCAUSALITY,REFLECTIVE

QUESTIONING,DEFINITION (of issues)

Why do I feel this way?FEELINGS as a

measure of expectations,surprise, uncertainty,

lack of confidence etc.

Impression

What do I notice aboutmy situation? What ishappening to me and

around me? TEMPORALLINKS,

Thoughts Feelings Challenge Self critique Potential solution Learning opportunity

Martin et al

Luk 2008

Depth

Orientation Complication (or Problematisation AA) Resolution (or ACTION)

REFLECTIVE NARRATIVE (may be embedded in Subject Rubric Scaffold e.g Intro/conc; "Concrete Experience"/ "Abstract conc

Identify critical issues Analysis of Issue Recommendation and Solution of Issue

SIMPLIFIED FOR ACTIONABLE FEEDBACK

• The complexity of depth dimension• The complexity of structural dimension• Possible 30 indicators, or a least 12 possible feedback options• Reduced to 3 + 1 primary indicators• 3 Additional characteristics

REFLECTIVE MOVES

Conceptual framework

AWA categories

Tutor highlights* AWA highlights

*We thank Peter Jones from UTS to have provided us with his annotations of some student essays.

Rhetorical moves are conveyed by pre-defined patterns of constituent concepts.

The constituent concepts are instantiated by words or expressions in sentences which are linked by syntactic dependencies.

Concept-matching framework

Ontology of concepts of reflection

CONTEXT

That early role-play felt distant and impersonal, as I made

a conscious effort not let my emotions interfere with the

job I had been given.





CHALLENGE

I do not feel as though I have sufficiently developed my

practical knowledge, or phronesis (Schwandt 2007), to a

point where I can speak up and challenge my manager.





CHANGE

In the future it would be beneficial for me to sit with them

and take notes on how they completed the task as this

would mean that I am still taking responsibility for my

role by learning how to face the challenge next time it

occurred.




Implementation in AWA: Xerox Incremental ParserTransition to open source code (Stanford Parser)

RWA PLATFORM

AWA – UI CONSIDERATIONS

• Annotation levels, Icons, Textual feedback

• Making the feedback actionable!

STUDENT USE – PHARMACY

• 59 Pharmacy students, 120 posts.• Of the 52% of students that provided feedback on the software, 85%

considered it helpful• 60% of students posted more than once• 27% showed evidence of drafting• Most draft modifications appeared to improve the quality of reflection• General indications of actionability of the feedback

LIMITATIONS AND POTENTIALS

• Pedogogical framing was a significant factor in reception of the tool• Difficulties with generalisation in a participatory design process• Lack of specifics resulted in smaller than anticipated buy-in• Ongoing work with specific subjects is addressing these issues• Paragraph level feedback was problematic• A redesign is likely to replace paragraph feedback with key discipline level

indicators• A TAP redesign is making sectional division of text more flexible

THANK YOU AND QUESTIONS?

A big thank you to authors not present: Adam Aitken and Cherie Lucas

And also to other project partners: Natalia Nikolova, Walter Jarvis, Alan Parr, Andy Leigh, Peter Jones, Jo McKenzie, Rosalie Goldsmith, Susan Hoadley, Isabelle Benne , SarabMansoor, Keenan Wilson, and Je Browi, Xiaolong (Shawn) Wang, and Xerox Research Centre Europe (XRCE).

For more info on our work, see:http://utscic.edu.au

STUDY WITH US @ CIC

utscic.edu.au/research/phd

Education

LAK17 Reflective Writing Analytics