Use of dialogue for Fellowship - Higher Education … · a dialogue, professional conversation or to underpin an oral. This can be seen from summary of the different dialogue formats

Use of dialogue for Fellowship

Dr Ruth Pilkington - July 2016

Contributors:

Mandy Asghar – York St John University

Vicky Davies – Ulster University

Dr Sarah Floyd – Ulster University

Dr Fiona Smart- Edinburgh Napier University

2

Wider contributions from

Dr Charlotte Chalmers – Edinburgh Napier University

Kelly Ho – York St John University

Dr Joan McLatchie – Edinburgh Napier University

Robert Mason – Edinburgh Napier University

Dr Jane Rand – York St John University

Dr C. Paul Sellors – Edinburgh Napier University

Philip Vincent – York St John University

3

Contents

Section Page

Contents 3

1. Introduction 5

1.1 Context of this report 5

1.2 Why dialogue? – a discussion of the wider use of dialogue for

Fellowships 5

1.2.1 Combining Elements 9

1.2.2 Benefits of Non-Written Review Options 10

1.2.3 Challenges of Non-Written Review Options 11

1.2.4 Missing data 11

2. How should dialogue work? 13

2.1 A Model for Dialogue 13

2.2 Detailing issues for the Assessed Conversation 18

2.3 Defining dialogue (Purpose) 19

3. Planning the assessed dialogue 22

3.1 Structuring an assessed dialogue: 22

3.2 Format of an assessed dialogue 23

3.3 Case Study evidence on structure for the dialogue process 25

3.3.1 York St John University (YSJU) 25

3.3.2 Ulster University 25

3.3.3 Edinburgh Napier University 26

3.3.4 Summary of the structure of the dialogue process for each case

study 27

4. Supporting assessors/reviewers 29

4.1 Techniques and approaches adopted by assessors 30

4.2 Challenges of assessing dialogue 32

4

4.3 Case Studies to illustrate approaches to assessor/reviewer

development 35

4.3.1 Edinburgh Napier University 35

4.3.2 York St John University 35

4.3.3 Ulster University 35

5. Other oral mechanisms of assessment 37

6. Key recommendations for dialogue 38

Appendix 1 40

Appendix 2 modes of review identified in 2014-15 annual CPD review 46

Appendix 3 – Institutional Case Studies 47

Appendix 3.1 Edinburgh Napier University: ENroute Scheme 47

Appendix 3.2 Ulster University: ENHANCE PD&R Scheme 58

Appendix 3.3 York St John University - Recognising Academic Practice

CPD Framework 66

Appendix 4 – Samples of case study guidance for reviewers 71

Appendix 4.1 Edinburgh Napier University 71

Appendix 4.2 Ulster University 72

Appendix 4.3 York St John University 74

References & Further Reading 75

5

1. Introduction

1.1 Context of this report

The following report is the culmination of a short collaborative project, which

aimed to draw together case studies of good practice and resources to support

those seeking to incorporate a dialogue route for awarding HEA Fellowship. The

report itself outlines lessons, issues and recommendations for those using a

dialogic route and accompanies a set of resources developed by practitioners

from across the Higher Education (HE) sector. A group of academics from

Edinburgh Napier University, Ulster University, and York St John University were

invited to produce and share good practice case studies and resources from

their work as Scheme Leads using dialogue successfully. These exemplars

emerge from cycles of implementation, review and enhancement by these

academics and provide invaluable insights to guide practitioners. Drawing also

on original research by Pilkington (Escalate), Davies (unpublished thesis), and

Asghar and Pilkington (in progress), the report provides a resource for those

considering, developing and/or implementing dialogue within accredited CPD

schemes, an area that is generating considerable attention within institutions.

1.2 Why dialogue? – a discussion of the wider use of dialogue for

Fellowships

Dialogue is much talked about amongst those involved with supporting the

professional learning of HE staff, especially with respect to the learning and

development of HE academics (Barnett, 2008; Knight, ; Kahn et al, 2006; Roxa

and Martensson, 2009; Pilkington, 2014, 2013). Within institutions adopting the

UK Professional Standards Framework (UKPSF) and a strategy of embedding a

HEA-accredited scheme for awarding Fellowship, dialogue is widely recognised

as adding an extra dimension to the value of a scheme. Within the report

summarising findings from the first annual HEA-accredited scheme review

(2014-15)1 undertaken for the HEA by Pilkington (2016), the following findings

were noted:

1 The 2015-16 annual accredited CPD scheme review considered activity self-reported by scheme leads for the period 1

May 2014-30 April 2015. The full summary report (Pilkington, 2016) is available on the HEA website.

https://www.heacademy.ac.uk/system/files/downloads/annual_cpd_review_report_2014-15.pdf

6

‘A point of interest is the frequency with which dialogue emerges as having

value and impact, both as an assessment mechanism and as a consequence

of scheme embedding’ (p11).

‘Dialogue around learning and teaching is mentioned frequently, as are

mentors. Mentors and the use of faculty or departmental support are

particularly crucial to the success of schemes and are widely cited. Around

these roles and activities dialogue, communities and wider exchange

emerge, supporting the applicants and also drawing staff in to seeing

learning and teaching as a valid area for development. This is supported and

enhanced by the development of dissemination mechanisms to share

outcomes from assessment processes…’

Similarly, workshops are widely used and are strong across many schemes.

They are often structured around the process of application: introduction

and overview, and preparing submissions. They bring staff together around

fellowship and encourage exchange, learning and dialogue’ (p14).

In fact, from an examination of all 2014-15 annual CPD scheme reviews, it was

apparent that the embedding of dialogue - through the wider conversations

around teaching and learning, and the raised profile and credibility generated

for learning and teaching as a result of such schemes - was recognised by nearly

every institution as a significant benefit and outcome. Of particular interest in

the review of institution scheme reports was the use of, and interest in, dialogue

for the actual process of awarding Fellowship within institutional schemes.

‘(43%) 38 of 88 use varieties of oral forms of assessment. These include

presentations, vivas, dialogue, although where there is a choice of route,

written forms are often reported as being preferred –seemingly there may

be a lack of confidence/conviction in this approach which requires further

work’ (Pilkington, 2016; 6)

With this growth in interest in dialogue, the HEA has sought to provide resources

for institutions seeking to incorporate dialogue within the assessed processes of

the schemes.

The analysis of the 2014-15 annual accredited CPD scheme reviews received for

the academic year undertaken for the Higher Education Academy (HEA) by

Pilkington (2016) provided an overview of the use of dialogue2 as an assessment

option across the accredited institutions. The following section extends

Pilkington’s (2016) analysis, raising questions to inform further discussion about

2 The term dialogue is used at this point in the paper; further attention is given to nomenclature in Section 2.

7

how dialogic approaches are, and might be, used; and the extent to which what

is described and enacted across the reports is indicative of a robust process. It is

appreciated that accredited UKPSF providers are often concerned by a number

of factors associated with the use of dialogue (or non-written options for

assessing Fellowship) including robustness, efficiency and how a non-written

option can be effectively incorporated within an accredited scheme.

The starting point for this discussion of dialogue use is Pilkington’s (2016) paper

and the data it provided. Institutional schemes3 in 2014-15 were identifying

dialogue as a component within the Fellowship assessment (43%). One

institution was using dialogue to support D4 in particular and a further nine

institutions reported using oral presentation in the accredited scheme

assessments. This meant around 43% were using an oral component for the

purposes of assessing Fellowship. Of all those institutions (88) reporting at the

review, the majority noted wider dialogue about teaching and learning emerging

as a wider benefit of an accredited CPD scheme. Furthermore, the majority of

institutions (>90%) mentioned dialogue as a wider benefit of scheme

introduction. Because of the wide range involved when institutions identified a

component of dialogue, it was evident that ‘dialogue’ as a term was unclear and

required some clarification.

In an attempt to assist in facilitating further discussion, the data from 2014-15

annual CPD reviews first worked with by Pilkington (2016) was re-explored.

Using a simple process of reviewing the language in the annual reviews a word

cloud was produced to draw out the language associated with non-written

Fellowship assessment. Whilst ‘dialogue’ (n.15) featured most commonly in the

type of scheme described (sometimes prefixed by ‘professional’), ‘oral’ (n.14)

featured too, not uncommonly attached to ‘presentation’ (n.12). Less common in

the range of terms used were ‘verbal’ (n.2), ‘conversation’ (n.2) and ‘viva’ (n.1) as

shown below.

3 The HEA annual accredited CPD review requires reporting only on institutions’ accredited CPD schemes; taught/credit

bearing programmes such as PG Certificates are not included in the annual review.

8

Figure 1 – Frequency of Language Used in Connection with Non-Written Fellowship

Assessment in 2014-15 Annual CPD Reviews

Figure 1 clearly shows a lack of consistency across the sector for how a non-

written approach might be conceptualised. The obvious question is whether we

are talking about the same or a different entity. For example, what makes a

dialogue, different to an oral or a viva? Arguably, what is important is that the

accredited institution knows for itself what its non-written review option(s) is,

and can articulate it to participants in the UKPSF scheme. An example of this is

the use by Edinburgh Napier (one of the Case Studies included here) of the term

‘review’ rather than ‘assess’. They justify this as follows:

‘As a team, we believed that language, and wider discourse, constructs

meaning and from this meaning we symbolically represent our values and

ethics. Consequently, for example, we decided that we would not speak of

judgment, but rather review. We would not use the terms assessment or

assessor, but rather review and reviewers. And individuals coming into the

scheme seeking Fellowship would be participants, rather than applicants’.

Across all the case studies informing this report, there are a variety of labels

used to describe elements within their dialogue options:

Ulster: Assessed Professional Conversation (APC)

York St John: Final Summative Dialogue

Edinburgh Napier: Professional Dialogue.

This said, as the ultimate arbiter of quality when accrediting schemes, the HEA

requires clarity in relation to the nature and purpose, and description, of non-

9

written components of accredited schemes4. As we can see from the case

studies, the actual process of managing and holding a dialogue varies

significantly within an overarching conceptualisation of a dialogue: this posits a

‘dialogue’ as an environment; a mechanism creating space for reflection,

exchange, exploration and recognition of practice through formalised

conversation, as well as providing a mechanism to assess and award.

In examining the 2014-15 annual CPD review data for those institutions using

non-written mechanisms for assessing Fellowship, further detail on process and

the format emerges, in particular around the component elements of non-

written forms of assessment.

1.2.1 Combining Elements

It is apparent from this that there is considerable variation in the ways in which

institutions combine assessment options. With one exception, non-written

options are presented as a choice, with at least one alternative available to

participants. Whilst the written option still dominates numerically as a means of

assessing Fellowship, portfolios/e-portfolios also commonly feature as a means

to present information, and this may or not may be used, for example, to initiate

a dialogue, professional conversation or to underpin an oral. This can be seen

from summary of the different dialogue formats used at Ulster, York St John and

Edinburgh Napier.

Ulster: Assessed Professional Conversation plus an e-portfolio, which is a

complementary component for the dialogue and is part of the assessment.

York St John: Final Summative Dialogue plus e-portfolio, which is formative;

mentored dialogue elements are crucial components; the portfolio acts as an

aide memoire and stimulus for the dialogue.

Edinburgh Napier: Professional Dialogue and the portfolio they use is

formative and acts as a stimulus to the dialogue, and is not assessed.

Overall, what stands out is choice, something to be welcomed in schemes

seeking to recognise individuals and their achievements in teaching and

supporting learning. Interestingly, although the written option features large,

4 Refer to the HEA Guide for addressing the criteria for accreditation

https://www.heacademy.ac.uk/system/files/downloads/guide_to_addressing_the_criteria_for_accreditation_2016-17_final.pdf

10

comments about its benefits for participants, assessors or the wider institution

are very quiet in the 2014-15 annual review data. Where it is raised, the benefits

are often framed around the dialogue that happens during formative stages of

preparing a written submission. This view contrasts with non-written review

options, where report writers offered frequent commentary highlighting

benefits and value of this option.

1.2.2 Benefits of Non-Written Review Options

Actual and potential beneficiaries of including non-written options in the choices

available fall into three categories from 2014-15 annual CPD review comments

(see Figure 2 below); they are the applicants, assessors and the wider institution.

1. For applicants, benefits were seen to include: enabling interaction with

the panel; being affirming of their practice as a process; a means to

develop thinking; a way to share practice and an opportunity to develop

confidence. One respondent’s observation was that “participants regard

the dialogue process as a celebration of their expertise and a chance to

articulate and scrutinise their philosophy of teaching”.

Figure 2: The Benefits of Non-Written Review Options

2. Benefits for assessors, also reported for mentors in some schemes, were

seen to include: encouraging communities of practice; learning from

others, especially from outside of their own disciplinary context; offering a

way “to give back”; and a means to maintain good standing.

1. Participant

2. Reviewer

Benefits

3. Wider Institution

11

3. The data identify clear gains for the wider institution too, with comments

offered about the wider dissemination of innovative practice; being better

enabled by hearing about it; the process prompting conversations about

learning and teaching; sharing and dialogue being a catalyst; and thoughts

about the facilitation of a cultural shift.

1.2.3 Challenges of Non-Written Review Options

Not surprisingly the data also raise challenges with non-written review options,

again from the perspective of applicants, reviewers and the wider institution.

There were, for example, comments about a reluctance to participate, with a few

schemes acknowledging that non-written options had still to be used. Whether

this was solely due to applicants not wanting to engage or other factors was not

always clear. One respondent felt that the professional dialogue was a risk both

for the participant in speaking ‘live’ about their practice, and for the assessors5 in

evaluating ‘standards met’, whilst engaging in the conversation. This challenge is

addressed within the Edinburgh Napier dialogue by using turn taking, where two

reviewer (their term) roles switch between note-taking, and leading the

conversation. Some scheme leads reporting in 2014-15 raise time and

administrative challenges associated with non-written options, something also

raised by Ulster and addressed within their preparatory planning cards. There is

a sense in which the positives of non-written review options outweigh the

disadvantages/ risks, however. This would be expected because all of the review

writers could be categorised as early adopters in their introduction of dialogue

as part of a variety of options into their accredited schemes.

1.2.4 Missing data

Exploration of the data suggests some missing information too. It was

anticipated that there might be more detail in respect of internal and external

quality assurance and enhancement, for example in respect of training of

assessors, the role of the externals (examiners, moderators or critical friends),

recording dialogues/orals, reflections on what can go wrong (and how situations

are managed). Of course, this does not mean to say that institutions are not

engaged in discussions and decision making in these areas, but in terms of

sharing lessons learned into the wider community of accredited schemes, it is

5 HEA requirements for the number and Fellowship status of reviewers required for review at each category of

Fellowship are set out in the HEA accreditation policy

https://www.heacademy.ac.uk/system/files/downloads/hea_accreditation_policy_2016-17_final_version.pdf

12

possible to conclude that there is more to be known. This is where the case

studies, and the planning tools provided by Floyd and Davies (Ulster) building on

their experience of introducing non-written forms of assessing for Fellowship,

are of value.

Recommendations from an analysis of the 2014-15 annual CPD review data on

non-written assessment for fellowship indicate institutions should, when

introducing a ‘non-written (dialogic)’ form of assessment:

Ensure they are clear in definitions and descriptions of the assessed process

about what is entailed in and expected from a ‘dialogic approach’. This means

providing clarity in language and descriptions of the processes involved,

providing definitions, or even using glossaries of terms.

Offer clarity on why combinations of review options have been identified.

This relates to being transparent to all the participants about any overarching

strategy of offering choice, and also within descriptions of actual

mechanisms.

Ensure that processes for QA are clearly outlined. This includes internal

processes of e.g. assessment, moderation, referral and complaints; being

clear on the role/ function of the external; and finally addressing internal staff

development.6

The review of data from the 2014-15 annual CPD review suggests as a final point

that there are wider sector needs in respect of sharing lessons gained from

implementation through annual reporting, and a need for a wider evidence base

as it concerns orality. This is being addressed within this publication in the

supplementary bibliography.

6 Institutions must also provide a mechanism for referees to verify the ‘claim’ made by the applicant

13

2. How should dialogue work?

2.1 A Model for Dialogue

The most widely used form of dialogic assessment around the UKPSF has been

developed using methods that combine the provision of evidence of practice

(usually written, sometimes oral), a process of peer critique and support, and a

final summative dialogue. This draws on research and work undertaken by

Pilkington and others as part of the HEA-funded Project on the use of dialogue

for assessing professional learning (ESCALATE). Building on this, a model for

dialogue in this context has been widely promulgated and adapted. It involves a

two-stage process of peer-mentored dialogue, which has evolved from the

model from Brockbank and McGill (2007) and has been widely circulated by

Pilkington (2011, 2013, 2014). The research was undertaken in 2009-10 as part of

a HEA-funded Escalate project exploring dialogue in assessment

(http://escalate.ac.uk/6333). The professional dialogue developed for the project

and piloted in five institutions adopted a two-stage process; a preparatory stage

and an assessed stage as follows:

Stage One – preparation phase (comprising multiple conversations)

This stage is about preparation for the assessed component. It is also about

facilitating a process whereby the applicant can begin to relate their practice

not only to UK PSF in a meaningful way, but also relate it to scholarship and

become familiar in the language of HE teaching and learning. It involves an

introductory dialogue held to engage and familiarise with UK PSF, process,

evidence required and assessment. This can also take place as part of an

introductory workshop.

Subsequent, follow up dialogues with a mentor (minimum 2 follow ups in this

model) facilitate an exploration of the applicants’ pedagogic practice, how

they are engaging with the UKPSF Dimensions and applying them to their

work. Supporting evidence for dialogues can be provided or constructed in

this stage through e.g. a portfolio or sample documents. The model suggests

these should be kept as succinct as possible – good practice examples use

e.g. Moodle, Blackboard, Mahara pages for sharing these elements.

Alternatively, a poster or short reflective commentary (audio or video) may be

created.

For FHEA, preparatory (mentor-facilitated) dialogues might adopt question

prompts structured around UK PSF Areas of Activity with use of examples

http://escalate.ac.uk/6333

14

and evidence from practice to explore application of Core Knowledge and

Professional Values. Dialogues to meet other Descriptors may focus on

crucial Descriptor components such as providing integrated and holistic

coverage of all Dimensions plus specific focus on leadership for SFHEA, and

strategic activity and impact for PFHEA. These enable applicants to familiarise

themselves with the reflective, evidence-led and scholarly approach expected

in Stage Two.

Within this model, each Stage One dialogue feeds into an iterative process of

further reflections by participants, further dialogue and, when ready (self and

peer-judged), the candidate goes on to Stage Two.

Stage Two – assessed dialogue

This stage is about the final summative or recognition assessment. It involves

assessors and potentially external assessor(s), and is structured to meet HEA

requirements.

The minimum assessed dialogue length within the model proposed here is

normally 30-45 minutes for Descriptor 2, although timings can vary in

practice. A recording is made, the judgment determined, and written

feedback may be drafted and agreed by assessors. Physical supporting

evidence can form part of this stage providing evidence and/or structure for

the assessed dialogue and a focus for probing and exploring practice or gaps.

Judgements will be formulated around this Stage Two process and decisions

often confirmed at a subsequent panel (board) as with student assessment.

An external moderator reviews samples of assessed dialogue recordings and

evidence and quality of the process and judgment assured.

Institutions involved in the project included UCLan, University of Liverpool,

University of Chester, York St John and University of Worcester. Building on this

work, dialogue is being successfully used by a number of institutions in formats

similar to that outlined above and as we have seen in Section 1, they can also

vary widely. In the case studies used in this report (full details in Appendix 3),

three approaches are provided, which all reflect the above model to a greater or

lesser degree:

York St John University uses formal mentor-led dialogues (2/3 or more for the

preparatory Stage One prior to a final summative dialogue)

15

Ulster University uses groups and workshops to support the preparatory

process (Stage One) for Associate Fellowship and Fellowship with dialogue as

an option: for Senior and Principal Fellowship, the dialogue is compulsory

with mentors used for the preparatory stage one. An Assessed Professional

Conversation forms the focus for Stage Two.

Edinburgh Napier University uses group based, peer-led mentoring circles at

the preparatory Stage One, and has a review dialogue for Stage Two.

All the case studies in this report support their approaches to dialogue with a

mechanism whereby the applicant presents a form of written evidence. This

gives focus for the final assessed dialogue and is an approach widely adopted

across the sector.

Evidence of practice is a desirable element within the process and a written

element provides a framework for this. The written evidence supports decision-

making and rigour of decisions; it forms an essential component that underpins

the final assessed dialogue7. Such evidence should encourage creativity however

alongside reflective engagement with professional practice, and be presented as

supportive rather than creating an additional and significant burden. The value

of dialogue constructed around this is that it can probe quickly and effectively

into practice and contexts and disclose contradictions. The types of evidence

most often used across the sector take the form of e.g. written pro formas, short

reflective documents, portfolios using VLEs or hard copy. The list below expands

this drawing on illustrations from examples mentioned in 2014-15 annual CPD

reviews.

1. Documentary summaries of mentored preparation dialogues, indicating

focus of discussion, evidence discussed, peer review, etc.;

2. Use of portfolios (simplified) either presented online or hard copy;

3. Summary CVs, records of practice mapped to the relevant UKPSF

Descriptor (which incorporates the UKPSF Dimensions), short reflective

pieces to provide a focus and prompt for questioning;

4. Presentations or video recordings, blogs, presentation of artefacts in

creative disciplines.

7 The written element may also provide opportunity for referees to verify the evidence provided is a fair and accurate

reflection of the applicant’s practice.

16

The aim of such evidence is to provide a basis for preparing the focus for the

assessed dialogue: discussion, exploration and probing around Descriptors,

Dimensions or gaps. Within HEA-accredited Fellowship processes, advocate and

referee input are also required for dialogue as with written submissions. These

can serve as additional sources of evidence and authentication, and the use and

preparation of such statements should align with HEA guidance.

The benefits of this approach is that these background resources, whether

assessed or not, provide a stepping off point for the dialogue, and for wider

exploration and sharing of practice. It is around this facility for creativity,

affirmation, exploration and deeper sharing of practice that the strength of the

non-written, dialogue form of assessing Fellowship resides.

The preparation dialogues espoused in the above model (using mentors or

critical friends, workshops or groups) take time, for example up to an hour in

each case. However, it is argued that if such formative dialogues are structured

as part of a wider strategy promoting teaching and learning, they can be

managed within schools and departments as a beneficial and constructive

approach enabling broader and mutual learning and sharing of practice

(Pilkington, 2013).

Ulster University on Senior Fellowship and Principal Fellowship mentor roles:

‘As they prepare their submission, applicants are allocated a trained mentor

who holds the category of fellowship they are aiming for. One key element of

the mentor’s role is to help them to develop their skills in discussing their

L&T practices and in particular their leadership of L&T’.

These activities can be used for continuing professional development for the

mentor as well as the mentee. In fact, all three case studies cite mentoring as

being a developmental opportunity for both parties, and this resonates with

findings from the 2014-15 annual CPD review. Roxa and Martensson (2009) in

their research into how academics learn, describe the use of dialogue within

significant networks as essential for practice-based learning. The preparatory

(mentored) dialogues are often positive, constructive opportunities for genuine

and powerful exchanges about practice from which considerable wider gains

can be drawn. In the model discussed above, these mentored (Stage One)

dialogues were undertaken over a minimum of 3-5 months, and this meant that

the preparatory Stage One period might be managed effectively within

workloads of mentors without excessive burden because of the value, discursive

and collegial aspects of the process.

17

Some institutions reporting in the 2014-15 annual CPD reviews have creatively

adopted the use of Skype, mentoring circles, action learning approaches, a

dialogic approach using workshops, or even team-led dialogues within subjects

for this purpose. A critical success factor for achieving Fellowship reported

across the reviews involves engagement and active participation in the Stage

One (preparation) process by Fellowship applicants. To conclude therefore, the

findings from dialogue experience supported by research recommend an

approach to dialogue that involves:

Drawing on this initial discussion, we can model the engagement process for

dialogue as suggested below. It can be seen that, in each of the overlapping

interactions, a space for learning and exploration of practice emerges, which

benefits all participants. These provide,

1. A space for exploration and feedback as part of Stage One between

applicant and, e.g. mentor, in which meaning is constructed explored,

shared;

2. A space in which during the ‘assessment’ conversation knowledge is

surfaced, recognised and made visible for review;

3. A space created through e.g. training and assessment, and in discussions

at panels and between team members (mentors, assessors, externals), in

which knowledge is shared more widely and potentially generates actions.

This is shown in the following diagram (Figure 3).

A two-stage process, structured over time as a supported preparation

stage (Stage One), followed by the applicant’s submission of ‘evidence’,

which complements or frames an assessed dialogue between peers

(Stage Two).

The Stage Two dialogue acts as a space for exploring, affirming, and

recognising practice within the framework of the relevant UK PSF

Descriptor.

18

Figure 3: Learning Benefits for Participants in Dialogue

2.2 Detailing issues for the Assessed Conversation

There are – it is agreed – a number of challenges associated with using dialogue.

These are extremely well summarised in the figure produced by Davies and

Floyd (Ulster University case study). This draws together their learning from

evaluations and their experience of developing and enhancing their dialogue-

based scheme. They identify six components for consideration when planning

and designing a dialogue-based approach to assessing Fellowship.

1. Purpose of Dialogue: Descriptor, formative, summative;

2. Format of Dialogue: component elements and relationships;

3. Guidance around Dialogue: for all participants (see 6 below);

4. The Focus and Process of Assessment: who, what, when, how:

5. Logistics for undertaking Dialogue: rooming, timing, location, ethics and

conflict of interest, moderation, award;

19

6. Training for participants in Dialogue: applicants, mentors, team,

assessors, externals.

The issues and planning questions associated with using dialogue for assessing

Fellowship in this figure are deconstructed within the case study for Ulster.

Figure 4: A Framework for HEA Professional Recognition through Dialogue (Davies,

2016)

2.3 Defining dialogue (Purpose)

It is absolutely essential first and foremost that everyone is clear about what the

dialogue involves, and how it is defined. Whilst definition is challenging, the

following highlight some crucial factors:

Talk: Speak in order to give information or express ideas or feelings;

converse or communicate by spoken words.

Conversation: A talk, especially an informal one, between two or more

people, in which news and ideas are exchanged.

Definitions of dialogue itself tend to shift into more formal territory:

Dialogue: A discussion between two or more people or groups, especially

one directed towards exploration of a particular subject or resolution of a

problem

Meaningful dialogue: A free and open debate between two parties (as states,

political factions, etc.), especially with opposed interests or with a history of

conflict; any open and unbiased exchange of views.

www.oxforddictionaries.com/definition/english/

http://www.oxforddictionaries.com/definition/english/

20

In Ellinor and Gerard (1998) they highlight the Greek roots of dialogue (dia –

through; logos – meaning). They discuss the value of divergent dialogue that

opens up and explores and on p16 they highlight six key qualities for dialogue:

1. Suspension of judgment;

2. Release of the need for specific outcomes;

3. An inquiry into and an examination of underlying assumptions;

4. Authenticity;

5. A slower pace with silence between speakers;

6. Listening deeply to self, others and for collective meaning.

Simply reviewing these characteristics, it is clear that the first and second are

precluded because of the award/recognition purpose of a ‘dialogue for

Fellowship’, but points 3-6 are positive characteristics that can be attributed to

and exploited within the dialogue approach.

From comments from colleagues, key terms associated with ‘dialogue’ in the

context of this report are

There are also physical components of face-to-face communication, which

provide additional layers of meaning within a conversation. In other words,

whilst dialogue is often the term used, what is actually being applied is far more

powerful. It lies between a conversation and dialogue. And the best dialogues

allow the elements of exchange, affirmation and meaning making to take place

for all parties to the process. There is of course a difference between what might

occur within ‘Stage One’ dialogue and ‘Stage Two’. ‘Stage Two’ dialogues are the

focus of an assessment / judgment linked to a Fellowship claim, so there are

important additional elements of inequality and relationship, of power involved,

and these have to be acknowledged and accounted for.

An assessed dialogue should, in sum, have clear purpose as a discrete activity

for surfacing practice and beliefs about practice, and - potentially - gaps in

Communication, exchange, sharing, exploration

Dynamic, flexible, surfacing, being in-the-moment

Narrative, story-telling, enabling, layered, richness, complexity

Affirming, person-centred, celebratory

Purposeful

21

evidence. It is a constructed space for flexibly and dynamically exploring,

exposing, and recognising professional practice. This is where its strength and

value lies. It is also where the challenge lies, meaning that training and

communication of expectations are crucial to its success.

22

3. Planning the assessed dialogue

3.1 Structuring an assessed dialogue:

What should be assessed within the assessed dialogue clearly has to be driven

by the Descriptor and the UKPSF Dimensions, and of course, by the applicants’

practice itself. This latter subject emerges for the majority through the

submission by the applicant of some form of evidence as either a framework,

prompt or complementary part of the assessment.

The Descriptors should act as criteria and a framework, exactly as with written

submissions. The Descriptors provide an invaluable structure for the dialogue

itself, especially where assessors can review supporting evidence from

applicants and target a conversation around potential gaps, strengths and key

Descriptors.

Dialogue can be aligned to any of the Descriptors and has strengths that fit with

each in terms of the potential for planned and active, in-the-moment, probing of

key Descriptor components. It should not result in a repeated (dual format)

submission, for example, where a supporting written submission of

evidence/reflection is rehashed again in the assessed dialogue space. For

example,

A dialogue for Associate Fellowship (Descriptor 1) might be structured to

explore two areas of activity with accompanying questions exploring

engagement with values and the knowledge base in the decision-making and

evaluations of the applicant, as well as recognising and sharing good practice.

Dialogues for Fellowship (Descriptor 2) can be framed by the Areas of Activity,

through which the Values and the Knowledge base can be explored and

surfaced. It is however important to manage time, so dialogue for this

Descriptor should be framed to strengthen and explore gaps in evidence and

to identify and celebrate good practice. This might require appropriate use of

evidence in any dialogue preparation so as to avoid repetition of effort and

content. It may also call for strong time management by assessors to avoid

being swept up by single Areas of Activity and ensuring balance.

Furthermore, simply rehashing an exploration of what has been amply

evidenced in supplementary documentation by an applicant would reduce

the point and value of the dialogue process.

23

Within the context of Senior Fellowship (Descriptor 3) a dialogue should focus

on the key Descriptor elements in particular D3.VII. Hence case studies and

evidence presented by applicants might provide a focus for dialogue to

surface how Dimensions underpin practice and further how ‘successful co-

ordination, support, supervision, management and/or mentoring of others in

relation to teaching and learning’ is enacted, evaluated and informed. The

sharing and highlighting of strengths for the applicant can be an important

outcome of dialogue for this Descriptor, potentially celebrating and

enhancing the individual’s sense of esteem, worth and impact.

An illustration of the particular issue of Senior Fellowship is evident in how an

institution defines what is meant by D3.VII. For example Edinburgh Napier

has deliberately defined this as, ‘influence and impact through others on

students’. Others have used this Descriptor as an opportunity to begin to

explore the challenging concept of ‘academic leadership’ enacted through

influencing the practice of peers in curriculum design, innovation, mentoring,

as well as management and co-ordination.

As with Senior Fellowship above, there are key Descriptors that

focus/structure dialogue for Principal Fellowship (Descriptor 4). These

naturally centre on D4.II and D4.III. In a PFHEA dialogue, the value of the

exchange can be the clarification of strategic impact for both assessors and

applicant to the benefit of their practice. The danger for a dialogue at PFHEA

is that sharing and celebrating the good practice of an esteemed applicant

can distract attention from the core differentiators in relation to the

Descriptor.

3.2 Format of an assessed dialogue

The assessed dialogues themselves may vary in duration. On average,

experience suggests durations may vary between 30 minutes (e.g. AFHEA, FHEA),

45 minutes (e.g. F/SFHEA), and 60 minutes (e.g. S/PFHEA). This will of course

reflect the structure and purpose of the dialogue being adopted as well as the

use of supporting evidence, and the Descriptor being targeted.

The HEA sets clear expectations for numbers of assessors involved in assessing

written and non-written applications within accredited schemes. The structure of

a ‘Stage Two’ Dialogue needs to allow sufficient time for exploration of practice

by the HEA-regulated number of assessors, and timings may also allow time for

review of any supporting documentary evidence, and space to agree a judgment

24

or to construct feedback on decisions. A concern raised about dialogue often

relates to the time involved and its organisation (logistics – Davies & Floyd). The

assessed dialogue is reputedly resource intensive involving several assessors

and the applicant, but it can be argued that this is also the situation for any

written assessment. Indeed time was identified across the review of data from

the 2014-15 annual CPD reviews by Pilkington, whether written or non-written.

Time was seen as a challenge for applicants in prioritising and meeting

deadlines, participating in formative and training events, and time was a

challenge for assessors/reviewers in terms of the actual process of assessment,

scheduling meetings, panels, and participating in e.g. non-written events. In

short, time was a universal issue.

Davies and Floyd highlight planning for and organising dialogues under their

issue of logistics. This relates to the challenges of bringing various participants

together at a specific time and place, and the planning of the location of the

dialogue to ensure resources (recorder, computer if necessary) and a quiet,

positive and reassuring environment. The solution it appears lies in support of

effective and good administration and clear systems; use of scheduling and the

establishment of structure to aid timely completion of the application and

assessment processes. All three case studies for this report reflect on how they

have evolved systems to manage this. Ulster manages particular issues with a

dispersed campus and uses Skype if necessary.

‘It can be easy to underestimate the administrative time that is needed to

organise and schedule APCs once you are dealing with growing numbers of

applications. I have needed to establish very clear processes for timetabling,

booking slots, allocating assessors, avoiding conflict of interest etc. I would

advise that at an early stage it is useful to set out all the stages that are

needed throughout the applicant journey’.

Edinburgh Napier has developed a very clear review period and process.

‘All participants, irrespective of category of Fellowship sought or review

option decided upon, follow the same process. Having expressed their

interest in Fellowship via an online system managed by the University’s

Human Resources Department, they gain access to resources located in a

virtual learning environment (VLE) and are invited to book on three

mentoring circles. They are advised that there are three review periods per

year:

a. Starting in May with applications submitted in September,

b. Starting in September with applications submitted in December and

c. Starting in January with applications submitted in May’.

25

York St John also highlights the need for managing:

‘Trying to organise assessments and ensuring that everyone was where they

should be at the same time was one of our biggest challenges. When the

CPD framework was first in place there was no administrative support and it

is a critical success factor. … Keep members of staff, who are engaging in the

dialogue process, together in groups. We have three cohorts a year that

mostly start and finish in a three-month period. If you do not do this it

becomes very chaotic’.

3.3 Case Study evidence on structure for the dialogue process

3.3.1 York St John University (YSJU)

The Dialogue process in the YSJU CPD Framework initially requires applicants to

engage in two, one hour dialogues with peers to explore how their practice

meets the criteria of the UK Professional Standards Framework (UKPSF). These

are formative dialogues that provide valuable opportunities for applicants to

reflect on their teaching and consider what evidence they might use to showcase

in the final summative dialogue. Applicants are required to use an e–portfolio

(Mahara) as a space to bring together their thoughts as to how they meet the

criteria of the UKPSF at the appropriate Descriptor level and to showcase their

evidence. The e-portfolio is deemed to be a formative aspect of the process and

contributes to the developmental nature of the recognition process. It provides

a means for staff to reflect on their pedagogic practice using a range of media.

Applicants are supported in the process by engaging in a mandatory initial

development workshop that provides training in the e-portfolio system and

encourages conversations about how participants might evidence their

engagement with the UKPSF. An audio-recorded final summative dialogue of

one hour occurs between the applicant and assessors. Prior to this event the

assessors are given access to the applicant’s e-portfolio. The e-portfolio is also

used as an aide memoire in the final summative dialogue and to showcase good

practice.

3.3.2 Ulster University

Applicants for Associate Fellowship (Descriptor 1) and Fellowship (Descriptor 2)

are able to choose between two options:

Dialogic Route consisting of an e-portfolio providing an outline record of

activities and evidence aligned to the UKPSF requirements for Descriptor

26

1/Descriptor 2, advocate statements, plus an Assessed Professional

Conversation (20-40 minutes).

Full E-portfolio Route providing detailed case-studies plus supporting

evidence and advocate statements.

A small number only have opted to use the full e-portfolio route.

All applicants can avail of group mentoring and/ or writing retreats as they

prepare their submissions. For the dialogic route, once evidence is submitted,

assessment of applications takes place through scrutiny of submitted evidence,

followed by a recorded Assessed Professional Conversation (APC), carried out by

two trained ENHANCE PD&R Scheme assessors. Assessors all hold SFHEA or

PFHEA. The e-portfolio and supporting statements are used as reference points

during the conversation. Evidence is triangulated against the Dimensions of the

UKPSF commensurate with the category of Fellowship being applied for.

Decisions are communicated within one week of the Panel meeting (held 6 times

per annum at the end of a series of APC assessments held over a 1-2 week

period).

All applicants for Senior Fellowship (Descriptor 3) and Principal Fellowship

(Descriptor 4) prepare an e-portfolio containing a detailed narrative plus

supporting evidence, advocate statements and participate in an Assessed

Professional Conversation. This is based on robust evaluation of the scheme,

where it is evident that there are clear benefits for both applicants and

assessors in assuring the veracity of the decision-making and the developmental

process for the applicant. As they prepare their submission, applicants are

allocated a trained mentor who holds the category of fellowship they are aiming

for. One key element of the mentor’s role is to help them to develop their skills

in discussing their learning and teaching practices and in particular their

leadership of learning and teaching.

3.3.3 Edinburgh Napier University

All participants follow the same process. Having expressed their interest in

Fellowship via an online system managed by HR, they gain access to resources

located in a virtual learning environment (VLE) and are invited to book on three

mentoring circles. There are three review periods per year, each of around 5

months. Any deferrals or withdrawals must be notified to the ENroute

administrator.

27

The mentoring circles are supported by individuals holding Fellowship – AFHEA,

FHEA and SFHEA; mentors8 are drawn from across the University, a feature

which is understood to strengthen ENroute, especially its mission to be inclusive

and respectful of the ‘discipline’. Guests are invited to the circles to speak about

their experience of the process of being recognised as Fellows. Each mentoring

circle has a different theme. The first focuses on categories of Fellowship and

review options, the second developing the claim and the third readiness to

submit.

Participants submit draft written submissions for formative feedback and can

ask for comment on their ‘presentation page’ in the e-Portfolio (the page which

is the focus for the professional dialogue).

Prior to the professional dialogue (about 3 weeks in advance of the review

period commencing), a session is held for participants to talk about what to

expect on the day and to go through the questions, which form the outline for

the professional dialogue.

The review period is planned in advance and takes into consideration a number

of factors, including reviewer availability and conflict of interest (prejudice

towards/bias against identified by both participants and reviewers). The review

period spans 3-4 weeks and is used by reviewers to make their decisions in

respect of documentary and dialogue applications. The review period concludes

with the notification of the decision to the participant, an updating of HR records

and the communication of information to the HEA.

3.3.4 Summary of the structure of the dialogue process for each case study

Stage One dialogue – focus and support; meaning-making

York St John University – workshops and mentoring; training in use of

Mahara: VLE support;

Ulster University – workshops; training in the e-portfolio; mentoring at

Descriptors 3-4;

Edinburgh Napier University – VLE and optional participation in structured

mentoring circles; optional formative feedback on e-portfolio.

8 Mentors can be reviewers, but they cannot be mentors and reviewers in the same review period.

28

Stage Two process – review, award and recognition; assessment

York St John University – Final summative dialogue of varied timings using e-

portfolio as an aide memoire

Ulster University – Final Assessed Professional Conversation with e-portfolio

and supporting statements used as reference points for conversation

Edinburgh Napier University – Review dialogue using e-portfolio as reference.

All assessed dialogues within the case studies are audio recorded, and

applicants are able to review these although the mechanisms differ; one

encourages all applicants to have a copy for review and informing future

reflections; for others this is available on request; and another allows supervised

access once on ethics grounds

Within the ‘Stage Two’ assessed dialogue the approaches differ in terms of how

dialogues are managed. Enroute has taken a very structured approach and

provides a crib sheet.

29

4. Supporting assessors/reviewers

Within the research by Pilkington carried out during the Escalate project, an

analysis of transcripts from the assessed dialogues produced two clear

resources to illustrate how dialogue might work in the context of assessment for

Fellowship. The first item is a model for the variety of dialogue or conversational

cycles that might occur within the framework of the assessed dialogue. This

supports assessors in their thinking about the nature and process of the

dialogue they might manage and how it might evolve around topics.

Figure 5: Model showing how the process of Dialogue might evolve around a variety of

topics related to prompts from Descriptors or submitted evidence

The model highlights a flow and the variety to how exchanges may emerge, and

also a broader overall structure for the conversation with an introductory and

end stage. Assessors can prepare questions and techniques to manage this, e.g

setting up a formal but affirming space and process at the outset, and giving an

opportunity for the applicant to reflect on, review their comments, and add to

the dialogue in the closing stages. This diagram of the process within an

30

assessed (Stage Two) dialogue contrasts significantly with the less exploratory

structure of a dialogue-as-interview shown in Figure 6.

Figure 6: Showing the contrast of formal probing involved in interview

Out of the analysis of transcripts from the Escalate project, it was furthermore

possible to identify stages and potential interventions by the assessor, which

might help a dialogue progress, and enable a positive exchange to occur.

4.1 Techniques and approaches adopted by assessors

The list below in Table 1 identifies some of the types of language and

conversational ‘techniques’ that might be adopted.

These techniques have been grouped around five main areas, of rapport

building, managing the discussion, formulation of questions, providing feedback

and linking to criteria, and managing face (emotions).

31

Table 1: Conversational techniques used by assessors/reviewers

Technique Different approaches

Building

Rapport

Explaining process of dialogue

Encouraging dialogue - Confirmatory statements - Reassurance -

Compliments

Checking applicant happy with dialogue

Jokes, laughter, smiles – body language

Personal engagement with discussion - Developing points further -

Presenting alternative perspective -Acknowledging shared experiences

Expressing personal beliefs - Reflecting on point made by applicant -

Expressing collegiality / solidarity with applicant - Expressing empathy

Answering applicant’s questions

Managing

Discussion

Focusing and re-focusing - Filtering - Reformulation of, e.g questions

and participant answers - Clarifying and making sense of responses

Making connections within the dialogue and to professional standards

framework

Redirecting and making openings - Giving applicant opportunity to

develop points further - Giving applicant opportunity to set agenda

Interruptions – Managing/allowing diversion arising from dialogue -

Making links and connections within and beyond dialogue

Formulating

questions

Reference to written evidence provided by applicant; to professional

standards, themes in Descriptors; to planned outline of questions

Drawing on knowledge of applicant outside dialogue

Open questions - Closed questions

Invitation to discuss (e.g. ‘tell me a little bit about’)

Requests for clarification, further information, evidence / example,

reflection, opinion, consideration of theoretical underpinnings

Using prompts from Professional Standards

Prompting questions to link scholarship or theory

Responding to point made in dialogue and taking it further

Evidence of own beliefs / perspectives in question

Feedback

and linking to

criteria within

Dialogue

Summarising themes and evidence for professional standards – making

link

Assessor developing points further

Giving feedback through affirmation, body language, noises, or direct

evaluative comment

Suggesting new approaches to issues raised in discussion

Suggesting areas for further consideration in the dialogue, linking to

future development

Asking applicant to relate a comment to professional

standards/Descriptor

32

Managing

Face

Challenging statements

Repairing where a conversation starts to go wrong

Creating a pause for reflection, recovery

In contrast to dialogue, written documents lack the dynamism and ‘divergent’

elements of a dialogue (Ellinor & Gerard, 1998). They often struggle to evidence

reflective, values related aspects of practice, and are static artefacts. The

benefits of dialogue for assessing Fellowship are that it is dynamic, flexible and

allows effective and detailed probing of knowledge and understanding.

Dialogue as an assessment process is widely recommended for enabling

meaning-making, and for surfacing the experiential and reflective aspects of

learning within and around practice, that is potentially a defining characteristic

of CPD scheme approaches to practice. These benefits make it a useful tool for

assessing professional espoused and enacted practice and knowledge.

Across the case studies, the benefits leading to the use of dialogue reflect this

interest in the dynamic, flexible, surfacing potential of dialogue strongly. Further

details about the approach taken by each of the three institutions are included

in Appendices 3 and 4.

4.2 Challenges of assessing dialogue

There are challenges. Disadvantages are that the skills and judgment of

assessors are foregrounded within the active, in-the-moment nature of the

dialogue process; these mean that training and support for assessors in the

criteria and process require time and effort.

Training and support is regarded as a crucial aspect of dialogue when using it as

assessment. Mentoring is already being used across CPD schemes in the UK as a

means of supporting the professional reflection and preparation process

involved in presenting any claim for Fellowship. In the 2014-15 annual CPD

review of such schemes, the majority of institutions identified participation in

mentoring as a ‘crucial success factor’ for Fellowship. Reports highlight that for

most schemes early investment in mentor training and growing capacity at

Senior Fellowship was a significant focus in implementation. Each of the case

studies used here draw on a form of mentoring or peer support within the

preparatory Stage One of their dialogue approach.

33

Mentor training generally highlights active listening, questioning techniques,

strong familiarity with Fellowship criteria, and a focus on supporting the mentee

to construct his/her professional claim drawing on evidence and an empirically

informed, pedagogic understanding.

Assessor training is potentially more challenging. Assessors require an ability to

build rapport with applicants, manage time and the power relationship of the

assessment; maintain a focus on Descriptor criteria; exercise an ability to

actively listen and internally judge ‘sufficiency’; and they need the confidence to

steer dialogue in a supportive and rigorous process. The response by Edinburgh

Napier to identify a lead ‘reviewer’ and another reviewer to focus on note-taking

and assuring criteria are met is a creative way of dealing with this challenge. It is

intended to enable one person at a time to focus entirely on the conversation as

it evolves. When roles are swapped the second (and third if Descriptors 3-4)

reviewer can then pick up on any gaps to ensure that the Descriptor is met.

Participants are informed about this arrangement in advance and reminded

before the dialogue starts, so they know what to expect.

‘What happens in the professional dialogue depends on the individuals who

together create it. However there are different responsibilities depending on

roles. For example, it is for the reviewers to establish the sense of a safe

space in which participants feel enabled to talk; it is for participants to be

prepared, ready for the reviewers to direct the dialogue to ensure that the

information gathered is on focus for the category of Fellowship sought.

However it is not always as straightforward as that, therefore there is a need

to be prepared for what to do in certain instances e.g. the participant going

off piste, the participant finding it hard to articulate their practice in respect

of the category of Fellowship sought, the reviewer losing focus, the reviewer

not probing when needed….’

Within the 2014-15 annual CPD reviews and also within case studies given here,

the ‘best practice’ process of training is rigorous, because all recognise the

dangers inherent in having untrained assessors and mentors: their role is crucial

to the success of any scheme approach for awarding Fellowship – and is

particularly important for the active, dynamic, in-the-moment environment of an

assessed dialogue. Such training is essential to the credibility of any scheme and

its wider embedding.

The model adopted for training assessors and mentors that appears to have the

most success involves:

Attendance at training

Undertaking/participating in a dialogue as an applicant

34

Shadowing a panel

Shadowing a dialogue

Seconding in dialogue

Engaging in reflective and debriefing events

Fronting (leading) a dialogue

Roles of mentor and assessor are often framed as professional development

towards Senior Fellowship, or conditional upon being a Senior Fellow, and

provide ongoing CPD for those who are Senior Fellows.

Another disadvantage highlighted is the oft-cited issue of evidence of rigour

accompanied by concerns over moderation and external input. This is arguably

a non-issue for dialogue in assessment: linguists have been using audio and / or

video recording with accompanying feedback sheets for decades. In examining

dialogue as an assessment tool, the same principles of professional judgement

apply here as elsewhere, namely the alignment to criteria and evidence of

achievement of learning outcomes (Biggs, 1999).

In training External roles therefore the above requirement for thoroughness,

experience and understanding of the process applies as with internal assessors.

Externals therefore should be familiar with dialogue and UK PSF, hold Senior or

Principal Fellowship and should be suitably experienced at making Fellowship

judgements. They should ideally have experience of, and an understanding of

dialogue, and its philosophy and purpose within the institution. They should

have access to all elements of the process including the portfolio, feedback,

mechanisms for determining judgment, etc.

Recordings can be exchanged using dropbox, links to a VLE, or via a USB or

CDRom. Storage of this data must be archived for three years and stored

ethically.

35

4.3 Case Studies to illustrate approaches to assessor/reviewer

development

4.3.1 Edinburgh Napier University

[In July 2016] the reviewer part of the ENroute team has grown from 7 to 23

since May 2015. It is appreciated that the increase in numbers exacerbates the

risk of inconsistent decision making. Added to this is the potential for mentors at

the mentoring circles to offer guidance that is not aligned with review decision

making. To address this, a joint programme of CPD for mentors and reviewers

was put in place in May 2015. There are now 9 ‘touch points’ per year, 3 per

review period which bring the team together. These incorporate:

1. Lunch with the External Reviewer to discuss the process of supporting

participants and making review decisions;

2. One week after the Review Board to explore ‘lessons learned’ and action

plan for refinements in respect of mentoring or reviewing;

3. A formal CPD session usually focused on a topic, which has emerged in

discussion with the External Reviewer.

4.3.2 York St John University

Getting mentors, assessors and applicants to understand what dialogue is and

isn’t was an initial problem and we overcame this by inviting Ann Brockbank to

deliver workshops about dialogue. These involved role play as well as

stimulating thinking about what it really was we were trying to achieve. I think

there was also a sense of allowing flexibility and fluidity across the three

dialogues and that often no two encounters would be the same, although

ultimately they were achieving the same end. We identified early on the

importance of training and development for those supporting the CPD

framework in order that the ethos of what we were trying to achieve was

maintained.

4.3.3 Ulster University

To build your capacity for operating a dialogic process, you will need more

trained assessors due to the nature and logistics of the process. Include some

documentary evidence as part of the submission, this will allow assessors to

prepare, and for the ensuing discussion to be richer and more relevant. It is

important also to ensure that the same assessors do not always work together

36

as there are real opportunities for assessors to learn from each other and to

appreciate the range of disciplinary and pedagogic perspectives that may

surface in an APC.

37

5. Other oral mechanisms of assessment

Examples of wider use of oral mechanism for assessment include video

recordings of reflective narratives (an option not included in figures above); oral

presentations with and without accompanying questions; and vivas for Principal

Fellowship or in borderline cases. Oral presentations raise challenges because,

as with written applications, they are to some extent static: a one-way

monologue. They rely on the ability of the presenter to make a case strongly to

meet Descriptors without being distracted by personal interests. As with written

submissions, such presentations also have to be judged on their merits without

further exploration and so lack the versatility of the interactive dialogue form.

Questions address this to some extent although there is often limited time or

scope for detailed exploration. It is recommended good practice to target

specifically the Descriptor criteria in structuring such mechanisms.

38

6. Key recommendations for dialogue

Ensure both assessors and mentors are conversant and experienced in the

Descriptors and Dimensions of PSF as these underpin judgment

Provide and require appropriate training of mentors and assessors

Encourage shadowing for mentors and assessors, as well as provide

experience of dialogue (either through having completed their own,

shadowed events or reviewing them)

Train assessors in active listening, summary and closure techniques

Active listening should be informed by ongoing internal reference to

Descriptors by assessors

Ensure that any assessed dialogue has a core structure aligned to the

relevant Descriptor built around the supporting evidence, whilst allowing and

encouraging exploration and reflection by participants

Support dialogue planning and focus by review of evidence and encourage

assessors to make time to schedule briefings for discussion pre- and post-

assessments

Ensure evidence requirements allow creativity and minimise excessive

burden to applicants

Train mentors as above and in reflecting back, summary, supporting meaning

making etc.

Schedule in time for pre-briefing and post dialogue discussion

Use a mix of pre-designed and targeted questions around Descriptors in

assessment

Use Dimensions to direct questions and frame dialogue for AF/F

Use Descriptors to direct questions and frame dialogue for SF / PF, focusing

on the crucial components: D3.VII, D4.II and D4.III

Do not be afraid to probe around Descriptors in the assessed dialogue, e.g

leadership, strategic leadership

Keep to time and use Descriptors and pre-planning to ensure time is well

directed

39

Ensure applicants are prepared e.g. know what to expect in terms of time,

that it is up to them to take ownership of the process, not to necessarily

expect it to be like an interview or viva etc.

Record dialogues for moderation purposes (audio is recommended to

minimise file size and access for moderation). Draw on local institution

norms for sample sizes accounting for HEA requirements, and the need to

invest more time in this process during early stages of using dialogue.

40

Appendix 1

The Table below sets out the responses relating to dialogue or other forms of

oral assessment as reported to the HEA in the annual review of accredited CPD

schemes (2014-15). These data provide the basis for the summary report

authored by Pilkington (2016).

Type of Scheme Comments on Dialogue

Certificate, narrative &

dialogue; narrative currently

preferred route

Personal mentor support through process. All mentors D3 and

D4. Anticipate challenges for dialogue in the timing, recording and

rooming. Workshop at writing retreat for dialogue.

Accredited taught and

assessed route for D1&D2.

CPD route written with

dialogic interview for D4

Faculty-based mentors, drop-in sessions and writing workshops.

Written Portfolio application

(applicant not present at

panel), oral presentation

(applicant present at panel),

or digital presentation

(applicant compiles digital

portfolio plus video

presentation)

Pairings of applicants across the institution helped generate

community of practice. Most conversations one-to-one. Peer

mentoring between SFHEA. Peer Groups used at SF and PFHEA.

Challenges articulating to candidate and panel what looking for in

PF.

3 Modes: written, eportfolio

and presentation, written

most popular

Initially, participants considered presentation route an

easier option- led to particularly diff icult recognition panel.

Now stress presentation route needs to be very well

prepared so demonstrates all UKPSF succinctly and

thoroughly. All routes provide evidence to support reflective

account. Concerns that dialogic opportunities may

advantage applicants; however, not evident from

experience. Where panels require supplementary

information for a decision , applicants offered written or

dialogic route for providing this

Written or presentation for scheme route, none used presentation yet

Written and portfolio plus

dialogue, e.g. RPA and PD

Labour intensive for mentor/assessor, our ‘professional dialogue’

route particularly popular with certain colleagues. Dialogue route

requires half the written information plus dialogue of approx. one

hour. Two colleagues discuss practice with the applicant in more

depth. Completers evaluate the experience highly. Some initial

difference about what constitutes evidence of

leadership/mentorship at D3

41

Scheme using dialogue Presentations and professional dialogues for assessment.

Extracts of the recordings used for repository of learning and

teaching innovation. Videos of applicants’ presentations - to be

used for academic development activities. Professional dialogue

based on observation for D1 - feedback from participants

positive. For D2 and D3, panels felt 40 minute presentation /

professional dialogue unhelpful to applicants and panels, so

changed to 20 mins prest plus dialogue. Challenge has been the

distinction between D2 and D3

Case study and viva The case studies and viva (optional D1 and D2, mandatory D3 and

D4) are especially valuable parts of the process

Dialogue review option

(eportfolio submission and

professional conversation)

plus document option

Applicants wanted professional dialogue. Now developed and

application process established

‘Mixed media’ portfolios,

written individual application

or oral submission possible

The UK PSF is becoming a standard or language in teaching and

learning conversations adding legitimacy for pedagogic research.

Oral narrative or written or

portfolio

The oral critical reflective narrative –enables interaction with

panel, questions about evidence and reflections. Panel members

provide feedback on strengths in person. Range of reviewers from

across university community; encourages dissemination of

effective practice; supports community of practice in teaching and

learning. Evidence from critical reflective narratives (oral and

written) and evidence suggests measureable impact on students.

Increased confidence in practice from positive outcome. Support

by one-to-one coaching.

Portfolio plus oral Cross-faculty leads and critical friends (mentors) ensure

interdisciplinary learning. The portfolio-based dialogic assessment

provides examples of practice for internal dissemination and is

reported by participants as affirming process.

42

CPD scheme only has

dialogue route- D1-3

Staff articulate teaching experience and philosophy in supportive

environment and explore rationale for choices, situating practice

within UKPSF. Staff utilise dialogue and feedback process to

develop their thinking. Personalised nature of dialogue enables

strategic approach to staff development. Staff in programme

teams ‘buddy up’ to work collaboratively on applications using

application process for discourse about teaching practice.

Significant time commitment for team members, e.g. length of

dialogues. Future FHEA dialogues to be 45 minutes; SFHEA 1

hour. Future developments: use SFHEA via dialogue route to

build mentors and assessors; enhance probing and questions for

SFHEA dialogues around e.g.‘sustained effectiveness’, pedagogic

research, and scholarship.

(1) portfolio of evidence with

reflective commentary

(written or oral) or (2) an

account of Professional

Practice and Development

using 2 case studies (written

or oral), plus two references

For (2) challenge for colleagues to conclude presentations with

summative statement on match to Descriptor requirements. Also

UKPSF mapping document of their professional and development

activities to illustrate breadth of experience alongside depth

conveyed by case studies; limit of 2pp max.

Oral presentation Oral presentation creates space for peers from different

disciplines to share accounts of teaching practice. Several joint

ventures initiated between panel members and claimants over

year.

Variety of options: a wholly

written form, a presentation

and interview and video

submission

The flexibility and support of scheme welcomed by applicants but

taxing for mentors. Mentor system success is having close

colleague to work with on application. Key learning for panel is

difficulty accessing and assessing a variety of materials.

3 routes: written, oral

dialogue and taught

programmes

Both pathways share criteria for submission (references, a peer

observation, summary of evidence document); dialogue pathway

involves 25 minute recorded dialogue with two independent

assessors. Both pathways popular with participants. Refining

guidance to participants on the dialogue route to clarify

expectations for assessment, and training day for dialogue route

for assessors/mentors so can better support participants on this

route. Challenge so far for dialogue route is number of

independent assessors. Scheme acting as catalyst for wider

discussions.

43

Dialogue option possible D1-

3

Writing workshops, mentor and 1:1 advice sessions support

applicants. Other support includes mentor and assessor training,

and advice sessions for referees. 1:1 advice sessions by mentors

and scheme assessors welcomed.

Professional Dialogue,

portfolio plus recorded

dialogue

Recorded professional dialogue is 45 minute conversation

between reviewer and candidate, involves conversation around

applicant’s practice and the evidence they provide. Primary

judgement made by reviewer, the recording is then assessed by

independent external reviewer and pathway co-ordinator. For

most, dialogue is extremely rewarding on both sides. Some

candidates and reviewers avoid this form of assessment, because

of risks attached to a live conversation about one’s practice and

demands placed on dialogue partner to evaluate standardsmet

whilst engaging in conversation with candidate

Written, application form;

can do podcast 2mins per

500 words

Planned community of critical friends to support colleagues from

departments.

Artefact submission, seen by 2 assessors [D1/2], plus moderation. Also video submissions

Largely written; verbal

option not taken; requests

for multi media

High quality conversations with staff, one-to-one guidance offers

opportunities to observe range of practice across the institution.

The quality of discussion at panels high.

Scheme with written

narrative or professional

dialogue route

Positive comments on opportunity to reflect on teaching practice

with colleagues from outside discipline area; is growing dialogue.

Consistency of judgement can vary, and some concerns about

penalising staff by requiring high standards.

Assessed dialogue D1-3 ‘Assessed dialogue’ route aimed at experienced staff encourages

applicants’ reflection on teaching and learning journey via

dialogue process. Initial reluctance for staff to engage; once they

start on the ‘assessed dialogue’ route, they realise the value to be

gained from critical reflection

Portfolio and dialogue, also

written; D1-4

Acting as assessor/ mentor allows staff to ‘give something back’

and maintain good standing. Dialogic approach to development

and assessment. Participants can access their assessed

professional conversation recording as part of reflection on

learning. Dialogic approach develops staff confidence in practice,

establishes communities of practice and that learning and

teaching is valued. Administration load is significant and impacts

on workload.

Written; oral dialogue only for clarification not assessment

44

Written and dialogue – all

have done dialogue

Recorded dialogic assessments are working well and staff

engages positively in this process. Mentor capacity built by

engaging those who have been through process.

CPD route, written or

presentation

Staff undertaking on CPD route report favourably on process and

reflective practice. Presentations have greater resource

investment attached in terms of panel time. However, with

support of panel members and appropriate moderation plus

training for applicant, presentation format is real strength.

All Descriptors, written, peer

dialogue around case studies

The peer dialogue praised by D3 and D4 applicants as useful

opportunity to reflect on case studies related to practice.

D1-4 written, oral Many of the case study examples from fellows are captured in

video and article form on the University website, to illustrate the

key education strategy themes and ambitions

All Descriptors

Verbal and written routes

Integrated mentoring scheme encourages dialogue and sharing of

practices between staff at different levels. Panel membership

drawn from across faculties

Descriptors 1-3, oral

presentation or written

submission

D3 panels consider written submission and oral discussion,

enable in-depth understanding of individual practice and support

for and guidance of others. Feedback from panel members very

positive. Orally assessment has set questions. Has been useful,

also constrains and limits learning exchange.

Scheme- written and presentation route; written form preferred

All Descriptors-

Written form, evidence

matrix and professional

dialogue

In PF, applicants can choose advocate to speak to panel prior to

professional dialogue. The professional dialogue is supportive in

tone, and provides additional opportunity for applicants to make

a claim. Staff welcomes opportunity to talk about learning and

teaching. Also shares good practice. Panels comprise staff from

across faculties; they value chance to hear examples of good

practice from outside own context. Process encourages staff to

reflect on practice and prompts conversations about learning and

teaching in institution.

One route: case studies in written or video format

3 routes: a written case study route (most popular); an oral presentation and an online reflection

(least). Also taught course route

Oral presentation possible or eportfolio – most used

2 routes: written or oral. PF

is both written and oral

Oral route can be a presentation, a discursive encounter or other

interactive session. Oral route is resource intensive; the panel

needs to be present not reviewing in their own time, but is more

valuable in broadening understanding of institutional learning

and teaching practice for participants and panellists.

45

Portfolio and professional

conversation

Portfolio designed to lead participants to provide appropriate

evidence. Mentoring process, when used, usually leads to success.

Review process has sound basis for making decisions, with high

reliability. Mentors and Panels hear about novel or innovative

practice for wider dissemination.

Taught and experiential;

written and oral – oral not

yet used

Allows emphasis on creative and practice-based outputs as part

of a submission– to be developed

Oral possible but not yet active, written

Teaching dialogue based on

reflective account of

practice

The Teaching dialogue well received as method to assess

applications where applicant has opportunity to elaborate on

experience and approaches

A written application or an

oral presentation

(accompanied by a mapping

exercise against the UKPSF, a

500 words summary of

personal educational

philosophy and a summary

of the oral presentation).

Most applicants opt for oral presentation followed by Q&A.

Experienced members of staff on Scheme present their case

orally to a panel of two external members and two internal

members. In revised scheme, the presentation is accompanied by

substantial teaching and learning portfolio mapped against

Descriptor.

D1-4, dialogic and written

uses eportfolio to underpin

oral.

Respondents valued mentoring relationship, opportunity for self-

reflection, recognition of good practice. Participants regard

dialogue process as celebration of expertise and chance to

articulate, and scrutinise philosophy of teaching. Positive shift in

institutional culture towards supporting and promoting

conversations about teaching, use of SoTL.

46

Appendix 2 modes of review identified in 2014-15 annual CPD review Institution Narrative

(written) Portfolio/ ePortfolio Written Case

Study(ies) Dialogue Dialogic

interview Oral Oral

Presentation Verbal Viva Video Digital

Presentation

1

2 (D4)

3

4

5

6 with dialogue

7

8

9 with prof conversation

10 (PgC)

11

12 with dialogue

13 PgC

14 with written or oral commentary

written or oral

15

16

17

18

19 + recorded dialogue

20 podcast

21

22

23

24

25

26 for clarification only

27

28

29 (peer)

30

31

32

33

34

35 written or video

36 online reflection

37

38

39 prof conv

40

41

42 (teaching dialogue)

43

44 to underpin oral

TOTALS 1 15 13 1 8 12 2 1 2 3

47

Appendix 3 – Institutional Case Studies

Appendix 3.1 Edinburgh Napier University: ENroute Scheme

Dr Fiona Smart, SFHEA

1. What led you to consider dialogue for your CPD/Fellowship Scheme?

Edinburgh Napier University’s decision, taken in 2013/14 to set an expectation

that all staff members on academic contracts would be recognised as Fellows

(AFHEA, FHEA, SFHEA or PFHEA) by 2020, led to a period of reflection within the

ENroute9 team10 during 2014/15. The team’s deliberations were informed by

discussions within the University and visits to a number of other institutions

running Higher Education Academy (HEA) UKPSF-accredited schemes. The end

point of the team’s review of our own Framework was a continuing commitment

to offer Fellowship opportunities across the landscape of the University (not just

to staff on academic contracts) and to be as enabling as possible to participants

seeking Fellowship. For the team, being enabling meant thinking about the

discourse of our scheme and our intention to enact collegiality in the way we

behaved and in the language we used. As a team, we believed that language,

and wider discourse, constructs meaning and from this meaning we symbolically

represent our values and ethics. Consequently, for example, we decided that we

would not speak of judgment, but rather review. We would not use the terms

assessment or assessor, but rather review and reviewers. And individuals

coming into the scheme seeking Fellowship would be participants, rather than

applicants. Thus, through this conscious collegial discourse, we set the

foundations for our UKPSF scheme, signifying an approach that would guide us

and create a framework of expectation for mentors, reviewers and, most

importantly, participants. Survey responses indicate we have gone some way to

achieving our aim:

9 ENroute is the University’s UKPSF CPD scheme and includes its PgC Learning, Teaching and Assessment Practice in HE,

its PgC Blended and Online Education and its experiential route to Fellowship.

10 The ENroute team includes membership from across the University; ENroute is managed from within the Department

of Learning and Teaching Enhancement.

48

Thank you. You have demonstrated the behaviors, values and principles that sometimes are not

so clear to see in other aspects of university work. It has been a very

rewarding experience. (Anonymous, Dialogue route applicant).

The ENroute team also committed to being responsive. One of the

conversations, which had taken place since the HEA Accreditation in 2013,

concerned the privileging of the written word in the making of a Fellowship

claim. The potential of a dialogue option within the experiential route to

Fellowship had been raised on a number of occasions. This was perhaps not

surprising. The PgC in Teaching and Learning in Higher Education had included

an integrative oral (IO) as part of the final assessment of its students for a

substantial period of time. The IO was designed to facilitate a focused discussion

with the student completing their studies in respect of two things: their learning

enabled via each of the three modules, and their professional development

through the lens of SEDA’s11 values and the UKPSF (D2). Feedback from External

Examiners had confirmed the robustness of the IO. Members of the ENroute

team had either experienced it as a member of the PgC team or as a student. It

was recognised as being valuing in its purpose, but also having the capacity to

facilitate discussion about key topics so as to enable the student to articulate

their knowledge, practice and value base. The question the ENroute team

contemplated was whether a similar opportunity might be created for

participants seeking Fellowship within the experiential route. The consensus was

that we should establish such an option. We debated in some depth what we

should call the option. And we used the literature to guide our thinking,

exploring, for example, Haigh's (2005) thinking on the professional conversation,

but decided in the end to adopt the term ‘professional dialogue’, confident that it

would facilitate the process described by Pilkington (2012, p. 254):

The structured dialogue enabled exposure, exploration and clarification of

the individual’s constructed professional pedagogical subject knowledge and

values to meet UKPSF descriptors to gain Fellowship of the HE Academy.

Having sought and gained authorisation from the HEA for the change, from May

2015 individuals seeking Fellowship via the scheme‘s experiential route can elect

one of two review options: a documentary submission or a professional

11 The programme was/ is approved by the Staff Education and Development Association (SEDA).

49

dialogue12. Both review options require participants to submit a record of

professional activity (RPA) and two referee statements.

Note: During 2014/15, both PgCs within the overall ENroute scheme were

revalidated. The PgC Blended and Online Education retained the option for

individuals who are employees of the University to seek to be recognised as

Fellows (FHEA). The mechanism for this was as before: students would reflect on

their learning through the PgC and give an online presentation, with a

discussion, evidencing D2 requirements. In contrast, the decision to take the

integrative oral out of the new PgC Learning, Teaching and Assessment Practice

was taken – the consequence of a completely new curriculum design and

assessment strategy. Fellowship at D2 is decided at the point of the final

summative assessment of the student’s patchwork text. Learning opportunities

throughout the single 60-credit provision facilitate focus on the UKPSF, so, too,

the five points of formative assessment and feedback.

2. Roughly and globally outline how your dialogue process works13

All participants, irrespective of category of Fellowship or review option decided

upon, follow the same process. Having expressed their interest in Fellowship via

an online system managed by the University’s Human Resources Department,

they gain access to resources located in a virtual learning environment (VLE) and

are invited to book on three mentoring circles. They are advised that there are

three review periods per year:

Starting in May with applications submitted in September,

Starting in September with applications submitted in December and

Starting in January with applications submitted in May.

Once individuals have decided that they want to make a Fellowship claim and

are therefore formally ‘active’ within ENroute, they can defer or indeed

withdraw, but they need to notify the ENroute administrator.

12 Participants seeking Fellowship via the professional dialogue review option submit an ePortfolio, which isn’t of itself

reviewed; it is explained as the “conversation starter” for the dialogue.

13 Focus from here in the case study is on the experiential, and not the accredited (credit-bearing PgC) route to

Fellowship.

50

At present the mentoring circles, referred to above, are campus-based; plans for

them to be virtual have been discussed but not progressed at this point. The

mentoring circles are facilitated by individuals holding fellowship – AFHEA, FHEA

and SFHEA, offering a breadth of support and guidance to the process and they

are drawn from across the University, a feature which is understood to

strengthen ENroute, especially its mission to be inclusive and respectful of the

‘discipline’. Guests who have gained Fellowship via the experiential route are

invited to the circles to speak about their experience. Again, this approach for

mentoring is valued by participants because it offers the opportunity to explore

ENroute from a lived perspective.

Attendance at each of the three mentoring circles is not mandatory at this point,

but this has been discussed with our External Reviewer. Each mentoring circle

has a different theme. The first focuses on categories of Fellowship and review

options, the second on developing the claim and the third on readiness to

submit.

The primary purpose of the mentoring circles is to create a space within which

participants come together to learn and to share. They draw from Habermas’

model of “communicative action” summarised by Bolton (2014, p. 8) as:

social action in which the actors seek to reach common understanding and

to coordinate group actions by reasoned argument, consensus, and

cooperation rather than action strictly in pursuit of their own goals.

Bolton (2014) notes that, for Habermas, communicative action sits in opposition

to “instrumental action”, which is essentially concerned with self-interest and

“strategic action”, wherein individuals seek to change the decisions of others in

order to meet a strategic goal, regardless of any personal adverse impact. The

mentoring circles seek to create an environment for “ideal speech

opportunities”, in which the information given is factually correct and relevant;

and where the intention of communication is both sincere and ethical

(Habermas, 1987). While we seek the ideal, the mentoring circles are not without

their challenges and will be returned to later in the case study.

In addition to the mentoring circles, all participants are offered the opportunity

of gaining formative feedback on their record of professional activity and also on

their draft written submissions (documentary option) and ‘presentation page’ in

the ePortfolio (professional dialogue option). Following discussion with the

External Reviewer, and an identified link between participants who do not seek

formative feedback and an increased likelihood of their not being successful in

their Fellowship claim, we now ask all participants to take advantage of this

51

opportunity. We do not, however, require it, recognising that individuals need to

make their own choices.

To further support participants taking the professional dialogue review option,

approximately three weeks in advance of the review period commencing, an

information session is held. It facilitates discussion in a group setting about the

professional dialogue process, reviewers’ expectations and participant

responsibilities. Time is taken to talk through the questions, which form the

outline for the professional dialogue14. Although all of the information explored

at this session is available in the VLE, feedback from attendees highlights the

value of the coming together to talk. Importantly, the significance of the session

extends beyond providing information, responding to questions and clarifying

expectations. It offers the space to enact ENroute’s commitment to collegiality,

communicated by the facilitator being warm in approach, receptive and valuing.

The review period spans 3–4 weeks and is used by reviewers15 to make their

decisions in respect of documentary and dialogue applications. Participants are

not informed of review decisions until after the Review Board sits. An academic

member of staff who is a Principal Fellow convenes the Board. This individual is

neither a mentor nor a reviewer. The primary purpose of the Review Board is to

ratify the review decisions and to receive formal feedback from the External

Reviewer, who is in attendance together with the ENroute lead, the organiser of

the mentoring circles, the administrator and the ‘empty seat’ observer16.

A pre-Board chaired by the Review Board convenor, and attended by the

External Reviewer, the ENroute lead, the organiser of the mentoring circles and

the administrator is used to discuss particular applications, identified either by

reviewers or the External Reviewer. When an application for Fellowship has been

selected for discussion, at least one of the reviewers involved in the decision-

making is asked to attend the pre-Board at a specific time. This feature of the

14 Question 1 – which category of Fellowship and why; Question 2 – participant invited to select something from their

presentation page “of which they are proud” and to talk about it; Final Question – “anything you’d have liked to have talk

about but haven’t had the chance to”. Topic areas for other questions – approaches to learning, teaching and

assessment, managing change, CPD and scholarship. These topics are tailored by reviewers according to the category of

Fellowship sought.

15 All reviewers are Senior Fellows of the HEA.

16 The ‘empty seat’ observer was introduced in May 2015 as part of our commitment to openness and transparency. In

advance of the Review Board, an invitation is sent to the whole of the University inviting anyone who has a role in

teaching/supporting learning to sit at the Board to observe due process. An individual is randomly selected from the

expressions of interest received to take the empty seat.

52

provision was introduced to prevent unnecessary discussion of colleagues

coming forward for Fellowship across the body of the reviewers and is seen as

further confirmation of our commitment to be respectful by upholding

confidentiality wherever possible.

The review period concludes with the notification of the decision to the

participant, an updating of HR records and, with the participant’s permission, the

communication of the successful Fellowship outcome to the HEA.

3. What were the challenges associated with introducing/implementing a

dialogue approach for your scheme, and how did you surmount these?

The challenges and their solutions can be grouped under four headings:

technical, organisational, consistency and feedback.

Technical – The VLE is used to receive documentary submissions, including

the RPA and references. From here, the administrator transfers the individual

folders for each participant to secure eStorage. We had thought that we

would be able to capture Mahara pages (the ePortfolio we use for the

dialogue) similarly, but cannot do so in the way we planned. However, the

presentation page is the conversation starter and is not itself reviewed.

Therefore, for participants taking the professional dialogue option, the audio

recording of their dialogue is stored securely by the administrator together

with their RPA and references submitted by participants into the VLE.

Another technical challenge was enabling the External Reviewer to access

audio-recordings remotely and securely. With support from Information

Services the issue was resolved.

Organisational – The review period is planned in advance and, as noted,

spans a three–four week period. The introduction of the professional

dialogue meant that we had to revisit what had previously been an

anonymous reviewer process. Now all participants irrespective of their

review option – documentary or professional dialogue – know who their

reviewers are in advance. Again, this supports a commitment to openness

and transparency. In advance of deciding on the reviewers, both participants

and reviewers are asked to identify if there is / might be a conflict of interest,

explicitly defined as ‘the risk of prejudice against / bias towards’. It has been

reassuring that both participants and reviewers have felt able to say where

there is, or could be, a conflict. This information, together with participant

and reviewer availability, and room allocation at each of the three campuses,

is used to create the review partnerships. Other information taken into

53

account at the point of planning is reviewer experience17. Overall, it is a

complex process, not commonly, but often enough, disrupted by a reviewer

becoming unable to attend a professional dialogue in the designated time

slot. To date, no dialogues have had to be cancelled.

From the perspective of room allocation, we have to ensure that the room is

big enough so each person feels comfortable in the space, has internet

access so we can access the ePortfolio, is sufficiently sound proof to protect

confidentiality and where disruptions are unlikely. We also learned, after the

first series of professional dialogues, that we needed to book rooms for two-

hour slots. This enables:

The reviewers to meet together for 30 minutes before the participant

arrives and to access the participant’s ePortfolio presentation page so

that it is on view for everyone;

The dialogue itself to have 60 minutes set aside (45 minutes of that for

the actual dialogue, with introductions, the audio consent process and

the overview of the process using the first 15 minutes, if needed);

30 minutes for the reviewers to make the provisional decision.

Consistency – The reviewer part of the ENroute team has grown from seven

to twenty-three since May 2015 [written July 2016]. The increase in numbers

is understood to increase the risk of inconsistent decision-making. Added to

this is the potential for mentors at the mentoring circles to offer guidance

that is not aligned with review decision making. To address this, a joint

programme of CPD for new and established mentors and reviewers was put

in place in May 2015. There are now nine ‘touch points’ per year, three per

review period, which bring the team together. They are:

1. Lunch with the External Reviewer to discuss the process of supporting

participants and making review decisions;

2. One week after the Review Board to explore ‘lessons learned’ and

action plan for refinements in respect of mentoring and/or reviewing;

17 All new reviewers shadow documentary and professional dialogue Fellowship claims as part of their induction into

role. Reviewers who have completed the ‘shadowing’ phase but are less experienced are partnered with more

experienced individuals, creating the opportunity for further learning and development.

54

3. A formal CPD session usually focused on a topic, which has emerged in

discussion with the External Reviewer.

The impact of the CPD programme can be evidenced in feedback from the

External Reviewer, who has noted the robustness of the scheme; equally

important is the development of the ENroute team into a cohesive whole,

operating so as to be consistent in the information provided, guidance given

and decisions made.

Feedback – Giving feedback to all participants, successful or not, has been

hotly debated in the ENroute team and is sometimes raised by participants

themselves. The challenge we talked through was giving meaningful feedback

to participants at the conclusion of the Fellowship claim. We agreed that

participants who were ‘not yet successful’ would receive written feedback, the

opportunity to meet with one of the reviewers and also guidance to work

through the tailored support plan agreed at the Review Board. However, it

was decided that participants whose claims were successful would not

receive what would be, in effect, summative feedback. Instead the available

resource would be committed to providing formative feedback at a point

when the participants would be able to make best use of it – see Section 2

above. This approach is consistent with a University-wide change process

centred on prioritising high-quality, timely formative feedback, rather than

using scarce resource where its benefit is less likely – that is, at the point of

the summative decision.

4. What would be the key considerations you would highlight for others to

consider when introducing dialogue, and why?

The mentoring circle format was born of necessity. The resource-intensive

1:1 model, which had been in place, was recognised as problematic given the

100% Fellowship goal. But there was more to the decision to introduce

mentoring circles than this. Mentoring circles feature in the literature as

valuable additions to an organisation’s mentoring repertoire. Driscoll, Parkes,

Tilley-Lubbs, Brill, & Pitts Bannister (2009), for example, challenge the efficacy

of the traditional mentoring dyad, instead acclaiming the potential of group

peer mentoring process, in a way which is not dissimilar to Habermas’ model

of communicative action (Bolton, 2014). Yet, just as Habermas (1987)

understood that the model of communicative action was an ideal type,

ENroute’s mentoring circles are not perfect; they need to accommodate a

number of variables, which include the facilitator and their skill in working

55

with the participants to create the learning/sharing space and the fact that

each mentoring circle, despite the fact that it is part of a series, is differently

constituted each time as the membership inevitably changes. Adding to the

complexity is that not all the participants are seeking the same category of

Fellowship; neither have they all selected documentary or professional

dialogue as their review option. From a communicative action perspective,

the space might appear ‘‘distorted” (Habermas, 1987). However, by

acknowledging the complexities, understanding pluralism and enacting our

values, facilitators can and do ameliorate the recognised difficulties, seeking

to remain true to the intention to begin the process of professional dialogue

in the mentoring circles in a way which is consistent with the scheme’s

discourse – captured for us in the term ‘collegiality’.

To be clear about the nature and purpose of the dialogue, including securing

agreement about the word used to describe this review option. As indicated

earlier, we explored a number of alternatives including ‘professional

conversation’ and ‘viva’. For us, the use of the term ‘professional dialogue’ is

consistent with Pilkington's (2012, p. 254) thinking and is used in ENroute to

represent an elongated process which starts in the mentoring circles and

concludes in the audio-recorded session, in which the participants speak

about their practice within a framework of expectation – that is they know

the first, second and final questions and also know the topics which will

explored, nuanced to reflect the category of Fellowship sought, during the

discussion.

Just as is the case in the mentoring circles, what happens in the professional

dialogue depends on the individuals who together create it. However, there

are different responsibilities depending on roles. For example, it is for the

reviewers to establish the sense of a safe space in which participants feel

enabled to talk; it is for participants to be prepared, ready for the reviewers

to direct the dialogue to ensure that the information gathered is on focus for

the category of Fellowship sought. However, it is not always as

straightforward as that, and whilst it is the case that ENroute reviewers are

seeking to adopt what Kvale (2008) presents as an approach which suggests

‘travelling with’, rather than ‘mining out’, it can sometimes be that a

professional dialogue feels like the reviewers are needing to ‘dig’ the

information out of the participant. There are other considerations too,

including what to do if the participant goes off-piste or when the participant

finds it hard to articulate their practice in respect of the category of

Fellowship sought, the reviewer losing focus or the reviewer not probing

56

when needed. Our experience is that the CPD process we have with its touch

points during the academic year builds skills and confidence across the

ENroute team. We are an example of a learning community wherein our

knowledge and skills are never complete.

We have found that one reviewer leading the dialogue for the first 30

minutes while another makes the record against category requirements and

then swapping over defined roles works well. Participants are informed

about this in advance and reminded before the dialogue starts, so they know

what to expect.

We work continually to ensure to the best of our ability an equivalence of

review opportunity between the dialogue and documentary review options. It

is likely that no matter how experienced we get, we will need to maintain

focus on this.

Finally, we took time to secure support for and to get guidance about the

audio-recording of the dialogues. Immediately prior to the dialogue starting,

participants are reminded about why the recording is made, who can access

it and its purpose – information that has all been made available previously.

All parties – participants, reviewers and, where relevant, shadow reviewers18

– sign the consent form. The form is securely stored in the participant’s

individual eFolder by the ENroute administrator.

5. What recommendations would you make to others to help them introduce

effective dialogue processes?

Accredited schemes need to facilitate engagement with the UKPSF and require

robust decision making. They also need to ‘fit’ with the institution – its priorities

and cultural milieu. Whilst every institution is different, there are some

recommendations which are thought to be of value elsewhere. They include:

Attention to the ethical underpinning of the scheme and its discourse;

18 As noted previously, shadow reviewers are new to role. They are required to develop by shadowing the dialogue,

observing the process and decision making and by shadowing the documentary review process. In neither case is their

view part of the decision taken. Participants are informed about the role of the shadow and know who they are too, just

as they know the names of their reviewers.

57

Ensuring clarity in respect of words and terms used and being consistent in

their use;

Ensuring transparency about review options and making clear the

commitment to an equivalence of review opportunity;

Ensuring clarity in respect of quality assurance and enhancement processes –

internal and external – including internal staff development and the role/

function of the external reviewer.

6. What have been the wider benefits/impacts of dialogue for your scheme /

your participants / your institution?

Edinburgh Napier has a proud history of valuing learning and teaching; see, for

example, its Teaching Fellows community, which pre-dates the UKPSF by years.

But even in this context, where pedagogy is a word that is used and understood,

having a dialogue option in ENroute is creating new conversations, with

participants, between participants and reviewer(s), between reviewers, with

mentors and reviewers. New ideas develop, practices are shared and questions

posed. In the Scottish context, where enhancement is the primary quality focus,

ENroute is privileging discussion about academic practice, in a way, which

documentary submissions perhaps can never replicate. There is something

about ‘being there’, hearing, listening, asking, responding, which makes practice

live.

7. What has been the impact of dialogue for you and have you any final

reflections and lessons learned to share?

Personally, my networks across the University have grown, I have gained

enhanced insight into disciplinary fields other than my own, I have experienced

pride in the quality of work lived in everyday practice and developed still further

my ability to think in the moment, and steer a dialogue without it becoming an

interrogation, or, in Kvale (2008) terms, an exercise in mining.

58

Appendix 3.2 Ulster University: ENHANCE PD&R Scheme

Dr Sarah Floyd PFHEA [email protected]

http://www.ulster.ac.uk/centrehep/pds/


The Ulster ENHANCE Professional Development &Recognition Scheme (PD&R

Scheme) utilises dialogue as a key component of its assessment approach for

the majority of applicants and it has proven to be a highly successful approach.

Prior to us introducing it, I would have only used other forms of oral assessment

such as presentations and vivas so it was a leap of faith to include it in the

scheme. However, this decision was evidence informed based on publications

and discussions with Ruth Pilkington, and as we also engaged her as external

assessor there was an element of a safety net as she would be able to support

our implementation of the approach.

Key drivers in opting to use dialogue were based around two key considerations:

As a HEA accreditor I was extremely aware that for some staff writing

reflectively can be a challenge but that they would be able to articulate their

experiences verbally with much greater ease. Additionally, there are times

when you read an application where you feel if you were just able to ask for a

little more information or evidence it would be so much easier.

At Ulster we have been striving to engage staff in meaningful dialogue

around L&T for a number of years and it was evident to us that there would

be many opportunities within a dialogic process to build staff confidence and

experience of doing this with a broader range of staff. We hope that this

would help to build a culture of L&T discourse.

Three years into my leadership of the scheme and having led on countless

professional dialogues I can now say it was the best decision we made, not one

without challenges, but more importantly it led to unexpected opportunities.

These have included:

our learning from the myriad rich examples of effective and innovative

practice that are discussed with applicants;

opportunities to build our own relationships with colleagues across the

university and to broker new inter-disciplinary relationships between staff

that lead to further enhancement of the student learning experience;

mailto:[email protected]

http://www.ulster.ac.uk/centrehep/pds/

59

Other transformative impacts on individuals and the institution are highlighted

below.

My reflections, as Scheme Lead, have been informed by extensive on-going

review and evaluation that has included inputs from a range of stakeholders

including applicants, internal assessors, our external assessor, senior staff, the

HEA and the core PD&R Scheme team.

2. Roughly and globally outline how your dialogue process works?

Applicants for Associate Fellowship and Fellowship (Descriptors 1 and 2) are able

to choose between the following evidence options:

Dialogic Route consisting of an E-portfolio providing an outline record of

activities and evidence aligned to the UKPSF requirements for D1 or D2,

advocate statements, plus an Assessed Professional Conversation (20-40

minutes).

Full E-portfolio Route providing detailed case-studies plus supporting

evidence and advocate statements.

We decided at our last accreditation to offer choice for D1 and D2 applicants and

a small number have opted to use the full e-portfolio route. However, in this

case study I will only refer to the dialogic route.

Applicants can avail of group mentoring and/ or writing retreats as they prepare

their submissions.

Once submitted, assessment of Associate Fellow and Fellow applications takes

place through scrutiny of submitted evidence, followed by a recorded Assessed

Professional Conversation (APC), lasting between 20-40 minutes, carried out by

two trained ENHANCE PD&R Scheme assessors. Assessors all hold SFHEA or

PFHEA. The e- portfolio and supporting statements are used as reference points

during the conversation. Evidence is triangulated against the dimensions of the

UK PSF commensurate with the category of fellowship being applied for.

Decisions are communicated within one week of the Panel meeting (held 6 times

per annum at the end of a series of APC assessments held over a 1-2 week

period).

All applicants for Senior and Principal Fellowship are required to prepare an e-

portfolio containing a detailed narrative plus supporting evidence, advocate

statements and participate in an Assessed Professional Conversation. This is

based on robust evaluation of the scheme, where it is evident that there are

60

clear benefits for both applicants and assessors in assuring the veracity of the

decision making and the developmental process for the applicant. As they

prepare their submission, applicants are allocated a trained mentor who holds

the category of fellowship they are aiming for. One key element of the mentor’s

role is to help them to develop their skills in discussing their L&T practices and in

particular their leadership of L&T.

Assessment of Senior Fellow applicants takes place through scrutiny of

submitted evidence, followed by a recorded Assessed Professional Conversation

carried out by two trained ENHANCE PD&R Scheme assessors. All applications

are also assessed by a third assessor with a sample moderated by the external

assessor. Assessors hold SFHEA or PFHEA. The focus of the Assessed

Professional Conversation is to explore the applicant’s achievement and

demonstration of “clear evidence of a record of sustained effectiveness in

relation to teaching and learning that demonstrates that you are developing

your strategic leadership role, incorporating for example, the organisation,

leadership and/or management of specific aspects of teaching and learning

provision, mentoring and leadership of others commensurate with Descriptor

3.” The APC lasts around 45 minutes.

The e- portfolio and supporting statements are used as reference points during

the conversation. Evidence is triangulated against the dimensions of the UK PSF

commensurate with the category of fellowship being applied for. Decisions will

be communicated within one week of the Panel meeting.

Assessment of Principal Fellow applicants takes place through a recorded

Assessed Professional Conversation carried out by two ENHANCE PD&R Scheme

Panel Members and an external assessor. At least one of the assessors holds

PFHEA with the others holding SFHEA or PFHEA. The e-portfolio and supporting

statements are used as reference points during the conversation. Evidence is

triangulated against the dimensions of the UK PSF commensurate with the

category of fellowship being applied for. The focus of the Assessed Professional

Conversation is to explore the applicant’s achievement and demonstration of

“clear evidence of a sustained and effective record of impact at a strategic level

in relation to teaching and learning, as part of a wider commitment to academic

practice commensurate with Descriptor 4. This may be within Ulster or wider

(inter)national settings.” The APC lasts around 60 minutes.

It is worth noting that the specific focus of the D4 dialogue has helped applicants

to focus their preparation on better positioning and articulation of their

approach to, and vision of, their own strategic leadership. They are very much

61

aware of the need to demonstrate how they can evidence their impact to

support their claim. This has led to stronger submissions and through both the

APC experience and the ensuing feedback applicants are provided with further

developmental advice. For aspirant PFHEA applicants this focus has also caused

them to seek opportunities to build a stronger profile of strategic leadership that

shows the reach and depth of impact needed for D4, and this has positive

benefits for the institution.



The Ulster ENHANCE PD&R Scheme utilises dialogue as a key component of its

assessment approach for the majority of applicants and it has proven to be a

highly successful approach. Three years into the scheme, we can now

confidently say it was the best design decision we made. My colleague Vicky

Davies undertook a rigorous evaluation of our early experiences and based on

these has established a model that explores the key practical and pragmatic

considerations we have had to deal with. These are summarised in the diagram

below.

A Framework for HEA Professional Recognition through Dialogue V Davies (2016)

Each year we have reflected on how the Assessed Professional Conversations

have operated, and as a result have refined our guidance, in particular the foci

and key questions we use.

Dialogue

Purpose

Format

Assessment

Training

Guidance

Logisitics

62

Key questions to be asked in all D1 & D2 APCs (wording may vary):

Reflecting on your experience as an educator (practitioner) in HE – could you tell

us about your overall approach to supporting and/ or enhancing the student

learning experience and how this relates to/aligns with the Professional Values?

Give an example of professional development you have engaged in and explain

how it positively influenced your learning and teaching practices.

Key questions to be asked in all D3 APCs (wording may vary):

Reflecting on your experience as an educator (practitioner) in HE – could you tell

us about your overall approach to supporting and/ or enhancing the student

learning experience and how this relates to/aligns with the Professional Values?

Give an example of professional development you have engaged in and explain

how it positively influenced your learning and teaching practices.

Using a representative example, tell us how you have demonstrated sustained

effectiveness in learning and teaching linked to one or more of the areas of

activity.

Illustrate, using an example, how you have demonstrated successful leadership

in learning and teaching that has impacted on and/or influenced the practice of

others (not students).


consider when introducing dialogue, and why? What recommendations

would you make to others to help them introduce effective dialogue

processes?

A number of particular aspects I wish to highlight are:

It can be easy to underestimate the administrative time that is needed to

organise and schedule APCs once you are dealing with growing numbers of

applications. I have needed to establish very clear processes for timetabling,

booking slots, allocating assessors, avoiding conflict of interest etc. I would

advise that at an early stage it is useful to set out all the stages that are

needed throughout the applicant journey;

63

It is evident that dialogue as an assessment approach can be much more

inclusive for staff from a wide range of disciplines and roles. Building on

documentary evidence, many individuals are able to demonstrate their ability

to reflect on their effective practice in a way that they may have struggled

with in the unfamiliar genre of reflective writing. However, staff can be very

nervous about the dialogue and we have worked hard to ensure that

assessors are trained in establishing an environment and atmosphere that is

supportive and conducive to a ‘conversation’ as oppose to an ‘examination’;

Build your capacity for operating a dialogic process, you will need more

trained assessors due to the nature and logistics of the process;

Include some documentary evidence as part of the submission, this will allow

assessors to prepare, and for the ensuing discussion to be richer and more

relevant;

It is important also to ensure that the same assessors do not always work

together as there are real opportunities for assessors to learn from each

other and to appreciate the range of disciplinary and pedagogic perspectives

that may surface in an APC.

5. What have been the wider benefits/impacts of dialogue for your scheme/


The sustained impact of engaging in discourse around L&T has been evident

with participants’ feedback on how valuable it has been, and how it has

encouraged them to talk more confidently and freely with others about their

L&T. This has started to permeate through the institution, developing a more

evident culture of "it is OK to talk about teaching".

“Engaging with colleagues on teaching practice is really rewarding as it makes you stop and

consider the discussions that you might have 'in your head' as you

are preparing teaching.”

“It validated my teaching practice and offered me an opportunity to reflect on the same and

think of ways of further engaging in continuous professional

education”

“Getting my Senior Fellowship has enhanced my confidence in my own abilities and helped me

realise the importance of my contribution to the educational

experience in my institution, something I hadn't fully realised before

I began this process.”

64

A value added for all who participate in the Assessed Professional Conversations

has been how much you learn about the range of effective, and at times

innovative, practice going on around the university. It is not unusual for

assessors and applicants to arrange to meet up later to discuss a particular

practice further, or to plan to work collaboratively. We have also been able to

connect applicants with others in the University doing similar work. For the

ENHANCE PD&R Scheme Leads this exposure to practice has been extremely

useful, and has provided a wealth of examples to use in other fora, such as

within Postgraduate Certificate in Higher Education Practice sessions.

Our approach has been supported and endorsed by both our external

assessors:

“What is very strongly apparent is the important contribution made by mentoring to the success

of the Scheme. This is essential in particular at SFHEA and PF. My

impression of the work of mentors and Faculty leads is that they

have contributed significantly to engendering a culture change

across faculties and have successfully embedded dialogue within

the institution. This has been enhanced by the strong emphasis on

scholarship and evidence across both the PG Cert and APC routes.

Dialogue emerges overall as an enjoyable positive, celebratory and affirming experience for the

majority of staff undertaking the APC. The final dialogue element for

the PFHEA applicants (which I attend) is particularly powerful and

rigorous indicating the strength of the dialogue within the process.

It significantly exposes both the strengths of the applicant and any

weaknesses which might indicate a candidate is not yet ready for

Principal Fellow. It is however also recognised that the PF process is

challenging and often creates tension because it is necessarily a

matter of working with those in sensitive leadership positions. This

might be overcome by having greater numbers of PFHEAs to act as

mentors and assessors from the university management team.”

External Assessor Report R Pilkington, July 2015

and another External who was involved in a recent PFHEA dialogue:

"As I mentioned on the day - it was one of the most positive recognition experiences I have had,

and whilst, if I'm honest, being slightly sceptical about 'dialogue'

previously, the experience has completely convinced me of its

potential." Dr Colleen Connor, April 2016

Seeing and learning about the value of dialogue has led to us attempting to

embed it more fully in other processes and, moving forward, we hope to be able

to transfer some of our learning in to our institutional approach to appraisal. We

are also continuing to provide formal opportunities for dialogue through use of

65

SFHEAs/ PFHEAs as mentors, and also as a community to consult with on new

L&T developments e.g. through Centre for Higher Education, Research and

Practice initiatives and projects.


reflections and lessons learned to share?

Evaluation of the ENHANCE PD&R Scheme has highlighted for us a range of

wider benefits for the institution:

the dialogic approach has acted as a trigger for ongoing engagement with

learning and teaching, and the development of meaningful communities of

practice where effective practice and innovations can be disseminated;

the value of dialogue in brokering new professional relationships and

expanding the usual “significant networks”;

recognising and gauging the role of professional recognition in fostering

cultural change in SoTL;

the surfacing and further development of more hidden, and previously

unrecognized, academic leaders.

The learning about dialogue has not only had impact at Ulster but has also been

of interest externally and I have, together with Vicky Davies, had opportunities to

share our experiences formally and informally.

66

Appendix 3.3 York St John University - Recognising Academic Practice

CPD Framework

Mandy Asghar PFHEA


The role of dialogue in professional learning and developing pedagogic practice

has always been very important to me. When I arrived in 2009 at York St John

University (YSJU), and became responsible for academic development, it struck

me that there were limited opportunities for staff to get together to “talk” about

their teaching practice. As a consequence I set about creating both informal and

formal opportunities to do this and moved away from traditional didactic

formats. This involved re-invigorating the annual learning and teaching

conference to something more akin to an open space event, creating lunchtime

forums using Pecha kutcha formats, and evolving a network of University

Teaching Fellows who regularly met to talk about their teaching. In shifting the

culture of traditional pedagogic development towards a more dialogic approach

it was then a natural extension to conceptualize it as a key recognition route in

our HEA accredited CPD framework in 2011.

2. Roughly and globally outline how your dialogue process works?

The process requires applicants to engage in two, one hour dialogues with

peers, to explore how their pedagogic practice meets the criteria of the UK

Professional Standards Framework (UKPSF) at the appropriate descriptor. These

are formative dialogues, an opportunity for applicants to reflect on their

teaching, and consider what evidence they might use in their third, and final,

summative dialogue.

Applicants are required to use an e portfolio (Mahara) as a space to bring

together and organize, the evidence they are going to talk about. We have found

this helps applicants shape their thinking about the UKPSF as many have difficult

choosing what evidence to use, particularly when they have been teaching for

many years.

Applicants attend a mandatory initial development workshop. This provides

training in the e-portfolio system, encourages conversations amongst applicants

as to how they can evidence their engagement with the UKPSF; and explains

how the dialogue will work formatively and summatively. The final aspect of the

process is an audio-recorded summative dialogue of one hour between the

67

applicant and assessors. Prior to this event, the assessors are given access to the

applicant’s e-portfolio. The e-portfolio is also used as an aide memoire in the

final dialogue and to showcase good practice. For further information about the

integration of the e portfolio into our dialogue process please see Asghar (2014).



Getting mentors, assessors and applicants to understand what dialogue is and

isn’t, was an initial problem and we overcame this by inviting Ann Brockbank to

deliver workshops about dialogue. This stimulated thinking about what it was we

were trying to achieve, and helped us develop a shared understanding. We

practiced dialogue using role play and talked about the challenges of the

process. It also became important to allow flexibility and fluidity across the three

dialogues. Often we found that no two encounters would be the same, although

ultimately they were achieving the same end. We identified, early on, the

importance of ongoing training and development for those supporting the CPD

framework so that the ethos of what we were trying to achieve was maintained.

The e-portfolio was the most often complained about aspect of the process,

despite the support we put in place. We have overcome this by providing

additional support sessions often on a one to one basis. That said, we have

seen some amazing portfolios that tell incredible stories of members of staff

who have created innovative learning opportunities for their students.

Applicants sometimes struggle with trying to decide what a portfolio page

should contain. Use of examples in the workshop has helped in showing what is

possible. We have avoided created templates as we want to keep the portfolio

flexible, allowing a variety of approaches and use of different media.

Trying to organise assessments and ensuring that everyone was where they

should be at the same time was one of our biggest challenges. When the CPD

framework was first in place there was no administrative support and it is a

critical success factor.

68


consider when introducing dialogue, and why?

Start small, take a pilot group through to assessment and learn as you develop

your processes, so that you iron out problems and gain confidence in your

decisions.

As the leader of a CPD Framework although it might feel awkward to do so, go

through your own scheme when seeking personal recognition. Use your external

to ensure quality and robustness of the outcome. This way you role model the

process to others, you know what it’s really like, and are more convincing in

encouraging others to engage in it.

Keep members of staff, who are engaging in the dialogue process, together in

groups. We have three cohorts a year who mostly start and finish in a three

month period. If you do not do this it becomes very chaotic.

Bring a couple of senior academics on board and support them through the

process. Early on I mentored a Dean to become a Principal Fellow through

dialogue. This had an enormous impact on her Faculty and she became a real

advocate for the dialogue process and engagement with Fellowship recognition.

Be careful not to overuse your early adopters who often become your

champions and participate as mentors and assessors. We found that we were

over reliant on a few who soon became fed up of the extra workload.

Think carefully about how you will engage with your externals in a dialogue

scheme. You may have externals involved in your assessments or you may just

have one overall external who will review a sample of your dialogues19. Consider

how you will share the files with them and remember it can be time consuming

to listen to lots of dialogues, longer than it would to review a sample of

documentary applications.

Learn from others who have been there before you in developing dialogue

routes. Increasing numbers of schemes in the UK are using a dialogic process,

you might find it useful to visit, talk to scheme leaders and watch dialogues in

action.

19 Please refer to the HEA accreditation policy for minimum requirements relating to externality (Section 4.6)

69

5. What recommendations would you make to others to help them introduce

effective dialogue processes?

Make sure your framework is effectively resourced and that you have your

champions who can talk positively about using dialogue in this way.

Keep encouraging those who have gone through the dialogue route themselves

to become involved as mentors and assessors so as to make your framework

sustainable.

Introduce new assessors to your framework gradually; give them an opportunity

to observe one or two assessments prior to doing any independently.

Be flexible and if you do not think someone is ready for the final assessed

dialogue make space for further mentor meetings. This ensures that applicants

are ready and helps to avoid the challenges of some might react to not

succeeding.

When running an assessment particularly with SFHEA and PFHEA where there is

a requirement for more than two assessors, consider how one may take the lead

in terms of the dialogue. This helps to avoid the final dialogic assessment from

becoming more like a viva.

Make sure your applicants are apprised of what to expect in the final summative

dialogue. It’s not an interview and you should avoid treating as one.

Think about how you might use edited recordings of dialogues for training

purposes.

Have discussions with mentors and assessors to share ideas about how best to

integrate scholarship and leadership into the dialogues.

Keep up with mentor/assessor support and training to ensure the ethos you set

out to create in your scheme does not become derailed. I believe it is important

to remain curious about dialogue and what we are trying to achieve. I often

return to Ellinor and Gerard (1998) who suggest that the best thing about

dialogue is that it “slows the pace of our conversation down so we can see what

is behind all the rushing around”. They explore the differences between dialogue

and discussion which they regard as on a conversation continuum. Dialogue is

seen as “learning through inquiry” and “creating shared meaning” whereas

discussion is much more about defending a position, persuasion and finding the

right answer.

70

6. What have been the wider benefits/impacts of dialogue for your scheme/


Many of those who have had success have suggested that it has had a big

impact on their self-confidence and self-esteem. Many applicants talk about

being proud to be e.g. a Senior Fellow. They also talk about enjoying the process

and the opportunity to talk about what they do in the classroom.

We have identified potential National Teaching Fellows from the dialogues we

have had and who we would otherwise not have known about.

We have created increased opportunities for colleagues to share their practice

with others.

Many more staff now use e-portfolios with their students as a result of their

engagement in the dialogue process.


reflections and lessons?

The impact of the dialogue process has been invaluable within the institution. It

has brought many staff together who would otherwise not have met each other.

I am confident that the dialogue route is more developmental than the

alternative documentary process, and that can only be good for the institution.

Remember to celebrate success, not just those who have gained recognition but

your mentors and assessors without whom the dialogue process would be

unsustainable.

71

Appendix 4 – Samples of case study guidance for

reviewers

Appendix 4 sets out some examples of the types of questions and approaches

that the three case study institutions take to questioning in dialogue. The three

sections below include some samples of guidance used for reviewers.

Appendix 4.1 Edinburgh Napier University

The questioning is intended to clearly frame an open dialogue following the

model of drawing on evidence and Descriptors but with open-ended (scene

setting and mop-up) questions to encourage ownership and input by the

applicant.

Initial question – focus on what category of Fellowship and why? Don’t look for a

long answer, the purpose of the question is to help the participant settle.

Follow with – ask the participant to select something from their presentation

page “of which they are proud”

Other question areas, which you may need to ask, depending on the dialogue.

Remember you need to think about the questions (and responses) through the

lens of the relevant Descriptor (e.g. Associate Fellowship, Fellowship, Senior

Fellowship or Principal Fellowship) :

Approaches to learning, teaching and assessment practice

Managing change

Recent CPD and its influence on practice

Scholarship and its influence on practice

Final Question – is there something you wanted to tell us about that you

haven’t had the chance to?

72

Appendix 4.2 Ulster University

Some institutions structure dialogue around the e-portfolio and Descriptors,

focusing on the process of dialogue management. For example, Ulster provides

guidance to assessors on how to manage the dialogue as set out in the ‘boxes’ in

Appendix 3.2 above and in the extracts of the guidance below:

Extracts from the guidance for assessors about managing the Assessed

Professional Conversation (APC)

Prior to the APC

Ensure that you are familiar with the requirements for the category of

Fellowship that you are assessing, particularly the dimensions of the UKPSF

against which the application is to be assessed.

Review the e-Portfolio to identify areas where the evidenced provided

matches the appropriate/relevant dimensions of the UKPSF commensurate

with the category of Fellowship being applied for;

Areas where there appears to be insufficient or partial evidence

supplied against one or more of the appropriate/relevant

dimensions of the UKPSF commensurate with the category of

Fellowship being applied for.

Check that the advocate statements broadly support the

application.

Based on the above you should begin to formulate questions that will allow

applicants to explore in more detail areas that you think are examples of good

and/or insufficient evidence.

During the APC

Establish the purpose of the APC: an opportunity for further exploration of

the evidence provided to ensure coverage of the appropriate/relevant

dimensions of the UKPSF commensurate with the category of Fellowship

being applied for.

Key questions will be asked of all applicants together with a range of more

tailored questions adapted for their practice (see indicative question hand-

out for the relevant category of fellowship).

Demonstrate respect and interest for the applicant’s perspective. Establish an

atmosphere of trust where the applicant is comfortable to take risks and

explore issues honestly.

73

Maintain the focus on the overarching issues (UKPSF) and avoid tangential

issues.

Use positive presuppositions. E.g. “What patterns do you see in student

coursework?” rather than “What are the patterns in student coursework?”

Although these two questions may seem to be asking the same thing, the

former suggest that the applicant is already alerted to such patterns and has

already noted them.

Avoid closed questions which evoke a yes/no response. Where the applicant

answers yes/no, invite further comment and thinking: e.g. “tell me more

about...”

Use plural rather than singular forms. E.g. “What possible explanations are

there for...?” This indicates that there is not a single correct answer but

multiple possibilities that are all worthy of consideration.

Promote analytical thinking by using “would ....if...” constructions to promote

hypothetical thinking and reflection.

Encourage metacognition – extract general principles from the

evidence/experience provided and see how the applicant’s thinking has come

about and/or evolved.

Examine assumptions and implication – make observations and probe

further

Paraphrase what the applicant says – this shows that you have been listening

and have understood what’s been said

Ensure that the questions/conversation allow for the areas where evidence

was insufficient or unclear to be explored in more depth, but you should also

encourage further reflection on areas where you felt good evidence had been

provided in the portfolio. Applicants should have the opportunity to reflect

on what has influenced their evolving practice and/or beliefs.

(Please also refer to the key questions contained in the boxes in Appendix 3.2, Question 3 above).

74

Appendix 4.3 York St John University

Some suggested questions for dialogue to frame and promote an open

dialogue;

I think of a fellowship dialogue as a moment in time to stand still, to reflect

on where you have come from and think about where you are going. Can you

tell us what the future holds for you in terms +of continuing to develop your

pedagogic practice?

Can you tell me about a teaching practice that was particularly successful/

unsuccessful? What did you learn from it?

Can you tell me about a time when you influenced someone else’s teaching

practice? What did you do and what impact did it have?

Tell us about where your ideas come from when you are designing learning

opportunities for students?

Is there anything you have learnt through the dialogue process that will

influence your future practice as a teacher?

75

References & Further Reading

Appleby, Y. and Pilkington, R. (2014). Developing Critical Professional Practice in

Education. Leicester: NIACE

Asghar, M. (2014) Using e-portfolios to evidence the UKPSF. SEDA Educational

Developments Issue 15.4 p22-26.

Barnett, R. (2008). Critical Professionalism in an Age of Supercomplexity. In: B.

Cunningham (ed.), Exploring Professionalism, pp.190-207. London: Bedford Way

Papers

Bolton, R. (2014). A Comparison of a Habermas-Inspired Approach and

Economists´Approaches to social capital. North American Regional Science

Conference Version, 1–51

Claude, C. (2011) Protocols for Professional Learning Conversations: Cultivating

the Art and the Discipline. Bloomington: Solution Tree Press

Danielson, C. (2009) Talk About Teaching! Leading Professional Conversations.

Thousand Oaks: Corwin Press

Davies, V. (2016) A Framework for HEA Professional Recognition through

Dialogue Unpublished

Driscoll, L., Parkes, K., Tilley-Lubbs, G., Brill, J., & Pitts Bannister, V. (2009).

Navigating the lonely sea: Peer mentoring and collaboration among aspiring

women scholars. Mentoring & Tutoring: Partnership in Learning, 17 (911724993),

5–21. http://doi.org/10.1080/13611260802699532

Ellinor, E & Gerard, G (1998) Dialogue. Rediscover the Transforming Power of

Conversation. New York. Wiley & Sons Ltd.

Eraut, M. (1994). Developing Professional Knowledge and Competence. London:

Routledge

Evans, L. (2008). Professionalism, Professionality and the Development of

Education Professionals. British Journal of Educational Studies, 56(1), 28-38

Floyd, S. and Davies, V. (2016) Dialogue as an institutional change agent.

Unpublished

Johnson, K. and Golombek, P. (2002) Inquiry into experience: Teachers' personal

and professional growth. In: Johnson, K. and Golombek, P. eds. Teachers'

Narrative Inquiry as Professional Development. Cambridge: Cambridge

University Press, 1-15

76

Habermas, J. (1987). The Theory of communicative action, Volume 2: Lifeworld

and system: A critique of functionalist reason. (T. McCarthy, Trans.). Boston,

Massachusetts: Beacon Press

Haigh, N. (2005). Everyday conversation as a context for professional learning

and development. International Journal for Academic Development, 10(1), 3–16.

http://doi.org/10.1080/13601440500099969

Kahn P, R Young, S Grace, R Pilkington, L Rush, B Tomkinson, I Willis, (2006) The

role and effectiveness of reflective practices in programmes for new academic

staff: a grounded practitioner review of the research literature University of

Manchester, The HE Academy

Kvale, S. (2008). Doing Interviews. London: Sage Publications Ltd.

http://doi.org/10.4135/9781849208963

Pilkington, R (2012) Professional dialogues: exploring an alternative means of

assessing the professional learning of experienced HE academics’ IJAD, 18:3.

August 2012

Pilkington, R. (2014) ‘Constructing practice space to support professional

learning’ Educational Developments 15.4 SEDA Dec 2014, pp5-9

Pilkington, R. (2013). Professional dialogues: Exploring an alternative means of

assessing the professional learning of experienced HE academics. International

Journal for Academic Development, 18(3), 1–13.

http://doi.org/10.1080/1360144X.2012.717225

Pilkington, R. (2016) Annual Review of HEA accredited CPD Schemes, HEA

https://www.heacademy.ac.uk/system/files/downloads/annual_cpd_review_repo

rt_2014-15.pdf

Roxå, T and Mårtensson, K (2009). Significant conversations and significant

networks - exploring the backstage of the teaching arena. Studies in Higher

Education, 34(5), 547-559

Sachs, J. (2001). Teacher professional identity: competing discourses, competing

outcomes, Journal of Educational Policy, 16(2), 149-161

Wenger, E. (1998). Communities of Practice: Learning, Meaning and Identity.

Cambridge, UK: Cambridge University Press



77

78

Contact us+44 (0)1904 717500 [email protected]

Innovation Way, York Science Park, Heslington, York, YO10 5BR

Twitter: @HEAcademy www.heacademy.ac.uk

© The Higher Education Academy, 2017

The Higher Education Academy (HEA) is the national body

for learning and teaching in higher education. We work

with universities and other higher education providers to

bring about change in learning and teaching. We do this to

improve the experience that students have while they are

studying, and to support and develop those who teach

them. Our activities focus on rewarding and recognising

excellence in teaching, bringing together people and

resources to research and share best practice, and by

helping to influence, shape and implement policy - locally,

nationally, and internationally.

The views expressed in this publication are those of the

author and not necessarily those of the Higher Education

Academy. This publication maybe transmitted in its

current form (electronically or mechanically), downloaded,

photocopied and printed for personal non-commercial

educational purposes. All other rights are reserved. Any

storage of this publication in repositories, reproduction of

extracts, republication or any other use requires the

written permission of the Higher Education Academy. For

permission requests, please e-mail

[email protected].

To request copies of this report in large print or in a

different format, please contact the communications

office at the Higher Education Academy: 01904 717500 or

[email protected]

The Higher Education Academy is a company limited by

guarantee registered in England and Wales no. 04931031.

Registered as a charity in England and Wales no. 1101607.

Registered as a charity in Scotland no. SC043946.

The words “Higher Education Academy”, “HEA” and the

Higher Education Academy logo are registered

trademarks. The Higher Education Academy logo should

not be used without our permission.

Documents

Use of dialogue for Fellowship - Higher Education … · a dialogue, professional conversation or to underpin an oral. This can be seen from summary of the different dialogue formats