WLE Centre for ExcellenceInstitute of EducationUniversity of London20 Bedford WayLondon WC1H 0AL

Tel +44 (0)20 7911 5531Fax +44 (0)7092 288 882Email n.pachler@ioe.ac.ukWeb www.wlecentre.ac.uk

Embedding evaluation in distance courses – using narrative methods

9 February 2000

Caroline Daly, Norbert Pachler, Jon Pickering, Jill Russell and Jon Wardle

Rationale for the research projects• We need course development to be informed by e-learners’ experiences• The ‘primacy of improving the student learning experience’ (HEA Strategic Plan 2005 – 2015) for overseas and home students• The inadequacies of evaluation based on ‘satisfaction’ and ‘exit’ models• We perceive coherence in our learning designs where the students may not• We need a sustainable way of understanding learners’ experiences in distance e-learning contexts• We need to avoid the ‘sisyphus effect’

E-learners’ experiences – four types of ‘newness’ which affect participation for all (CDE 1)

The experience of managing work, life and learning with technology

The experience of writing with fellow students in an online discussion in order to learn

The experience of collaborative learning in an online tutor group

The experience of social relationships with other e-learners

Narrative evaluation methods are based on…• a ‘way of knowing’ or ‘mode of thought’ (Bruner, 1985)

• a way of filtering experience in order to make sense of it

• giving practitioners access to informal as well as formal experiences of students’ learning

• respondents in engaging with knowledge they have which is often tacit and unarticulated

• a qualitative way of understanding something for both the learner and the practitioner - an interpretation of reality the learner organises their experiences by ‘narrating’ the practitioner constructs the meanings to be coherent within a

view of what has happened

Why narrative methods?

A way of capturing the learner voice as a prime research or evaluation tool in qualitative approaches

A way of developing situated accounts of respondents’ experiences

To provide ongoing information about our practice as providers while it is happening

To impact on learners’ abilities to adapt to new learning contexts (Levy, 2006)

A way of better understanding the realities of ‘being a learner’ in complex contexts

To inform effective education practitioner-development

• Loosely structured• “without overspecifying

the substance or the perspective of the talk” (McCracken,1988)

• Naïve interviewer stance

• Invitational, not interventional

• Long• ‘Confessional’

Narrative interviews

What is going on in a narrative interview?

• Emplotment • Chronology• Opportunities for individual sense-making (e.g. metaphor-

building in this example)• Opportunities for deep levels of reflection by the

respondent can elicit rich data• The agenda becomes the respondent’s

Contemporary contexts for narrative evaluation

`Patterns of student participation are changing. A range of new contexts raise challenges for distance education and training, in terms of how we can evaluate the learner experience:

E-learning Distance education Mass education Work-based learning Professional accreditation Retention, isolation, dilution, serial change, multiple pressures…

CDE 2 project 2006-7Context Design

2 online masters degreesUCL & Bournemouth University

Longitudinal – 9 months

Participants (n=18) new students - 11 in primary healthcare (UCL) - 7 in creative media (BU)

Quantitative data – pre-course questionnaire

E-learning at a distance - online discussions - electronic text-based resources - wikis (BU) - blogs (BU) - podcasts (BU)

 

Qualitative data – learner narratives - online discussion – course-based - narrative interviews by telephone - online commentaries - individual email correspondence - focus groups with tutors International

- UCL cohort includes a high proportion of South African students - CEMP cohort is mostly UK based

Analytical approach based on systematic reading and open-coding of range of narratives to establish the effectiveness of methods

Main aims: to develop methods to collect evaluation information from distance learnersto design an embedded approach to evaluation in fully online contexts

 

Narrative evaluation methods for distance learners• Group narratives aimed at collective sense-making of experiences. The UCL

‘intensive introduction to e-learning’ as an embedded evaluation activity;

• Telephone interviews. Adapting individual, loosely-structured narrative interviews using ‘postmodern’ techniques (Gubrium, 2003) from face to face contexts;

• Virtual focus groups using online forums to elicit reflective group discussion on key issues;

• Individual email correspondence (post-interviews and post-online forums) to invite reflective review of earlier narratives.

Narrative interviewing at a distance• The power relations between tutor/evaluator and learner changed• The learner knew more than the interviewer about the

context of the interview • The learner controlled the context of the interview• The learner questioned the interviewer • The interviewer exercised measured tentativeness• The interview became a dialogue where prompts must be

given, but control over the agenda is handed over to the learner as far as possible

Tutors’ responses• The content of narrative interviews did not ‘surprise’ the tutors much

• The narratives provided a coherent picture – told the ‘whole story’ of the learners’ experiences in a way not normally available

• The data did expose experiences not anticipated by tutors (e.g. the ‘feeling sick’ about the experience of posting to the forum; scale of the struggle to balance work/life/study; the fact that students participate in the middle of the working day e.g. ‘between patients’)

• Highlighted the need for more close engagement with students’ experiences

• Highlighted students’ desire to ‘own’ evaluation processes, e.g. to decide when and how it is convenient to discuss their experiences

Dialogue – the future for meaningful evaluation? The value of narrative evaluation is much more than

viewing it as an effective means of eliciting rich evaluation data.

“An alternative view of evaluation redefines it as a social practice in itself and so the emphasis becomes as much on the value of the substantive processes of evaluation as on the product (data) elicited by the evaluation”

Russell (2008)

e-VALU-ation “…involves not only collecting descriptive information …but

also using something called “values” to (a) determine what information should be collected and (b) draw explicitly evaluative inferences from the data…

…research can tell us “what’s so” but only evaluation can tell us “so what”.

Davidson (2005)

Project publications

• Daly, C. (2008) ‘Evaluation for new learning contexts – how can it be ‘fit for purpose’?’ Reflecting Education 4 (1) 127-138. http://www.reflectingeducation.net

• Daly, C., Pachler, N., Pickering, J. and Bezemer, J. (2007) Teachers as e-learners: exploring the experiences of teachers in an online professional master’s programme Journal of In-service Education 33 (4) 443-462.

• Daly, C., Pachler, N., Pickering, J. and Bezemer, J. (2006) ‘A study of e-learners’ experiences in the mixed-mode professional master’s programme, the Master of Teaching’ WLE Centre, Institute of Education, www.wlecenre.ac.uk