Http:// ePortfolios Using ePortfolios to support annual appraisal in undergraduate medicine Simon Cotterill, Philip Bradley, Rosie

http://www.eportfolios.ac.uk

ePortfolios

Using ePortfolios to support annual Using ePortfolios to support annual appraisal in undergraduate medicine appraisal in undergraduate medicine

Simon Cotterill, Philip Bradley, Rosie StacySchool of Medical Education Development


ePortfolios

Reflective approach to evidencing achievements:

"Clinical logbooks and personal portfolios, which allow students to identify Clinical logbooks and personal portfolios, which allow students to identify strengths and weaknesses and to focus their learning appropriately, can strengths and weaknesses and to focus their learning appropriately, can provide such information. Using these will emphasise the importance of provide such information. Using these will emphasise the importance of maintaining a portfolio of evidence of achievement, which will be necessary maintaining a portfolio of evidence of achievement, which will be necessary once they have become doctors and their licence to practise is regularly once they have become doctors and their licence to practise is regularly revalidated." revalidated." Tomorrow's Doctors, GMC, 2002

Appraisal"Students must receive regular, structured and constructive appraisal from Students must receive regular, structured and constructive appraisal from their teachers during the mainly clinical years of the curriculum. This their teachers during the mainly clinical years of the curriculum. This allows the medical school to judge their clinical knowledge and allows the medical school to judge their clinical knowledge and competence against the principles set out in Good Medical Practice.competence against the principles set out in Good Medical Practice." Tomorrow's Doctors, GMC, 2002

Background #1


ePortfolios

Built on 2 years experience of using ePortfolios in MBBS from Sept 2003

Background #2

FDTL-4 Collaborative project: http://www.eportfolios.ac.uk/FDTL4

Cotterill SJ., McDonald AM., Drummond P., Hammond GR. Design, implementation and evaluation of a 'generic' ePortfolio: the Newcastle experience Proc. ePortfolio 2004 (ISBN 2-9524576-0-3)


ePortfolios

Aims and Objectives of the Appraisal Project (2005/6)

• Support and further develop the use of portfolios and related assessment processes in the MBBS curriculum.

• Provide students with the facility to evidence their achievement of specific programme outcomes in their ePortfolios

• These will be seen by and discussed in an annual appraisal.

• Support the one-to-one discussion/coaching and the evaluation and refinement of the process with the intention of eventually rolling it out across the full 5 years of the programme.

• Support the evaluation of this


ePortfolios


ePortfolios


ePortfolios


ePortfolios

Student Perceptions of the ePortfolio and Appraisal Process

Online Questionnaire (LSE)14 fixed-response questions (6 point Likert scale) 9 open-ended questions

213 Respondents (90% response rate)

Evaluation 2005/6


ePortfolios

PortfolioThe ‘My Evidence’ section of the portfolio within LSE was used to support the appraisal process.

questionquestion % agree% agree(scores 4-6)(scores 4-6)

Overall the ePortfolio was easy to use 74%

The ‘My Evidence’ section of the portfolio was easy to navigate 81%

Attaching evidence (files, links etc). to the portfolio was straight forward

91%

It was easy to print off the ‘My Evidence’ section 87%

Having a word limit on the ‘reflective accounts’ helped me improve my summative skills

64%

I felt that there was sufficient guidance and instructions about using the portfolio

48%

Overall I found the process of evidencing my development useful 47%


ePortfolios

The Appraisal SessionThis relates to your face-to-face appraisal session

questionquestion % agree% agree(scores 4-6)(scores 4-6)

I had a clear understanding of what I needed to do for the appraisal

60%

I felt apprehensive about having an appraisal 63%

My appraiser made me feel at ease 91%

The session was well facilitated 86%

The appraisal session referred to my evidence from the portfolio 90%

Overall the appraisal session was a positive experience for me 70%

Having had the appraisal will influence the way I approach my learning next year

58%


ePortfolios

Analysis of the qualitative data

What did you most helpful about the appraisal session ?

102 students responded

Reflection 43-Holistic / Overview 12Personal / Interaction 35Reassurance 8Process (portfolio / appraisal) 17Learning Evidencing Skills 17Planning / goal setting 14Specific Advice 8

Categorisation of anonymised data by SC


ePortfolios

“The fact that you actually spoke about your progress made you feel that it wasn't just about exams”

Personal / Interaction (n=35)

“It was really good to talk about the course and how I was doing and what I could do better. I really found it beneficial to have someone to bounce things off.”

“It was nice to feel the university is kind of 'monitoring' us. That we aren’t just candidate numbers.”


Reflection (n=43)

“It made me think about what I have achieved this year.”

“Allowed me to see my own achievements.”

“I was able to clarify to myself via the facilitator both the things that have gone well regarding my learning this year, and the things that I needed to improve.”


ePortfolios

Holistic / Overview (n=12)

Included recognition of ‘life-wide learning’:

“The appraiser helped me realise that there were other activities that I did which would also constitute evidence, something that I hadn't known previously…”

“Brought things together. Made me sum up my overall progress of the year, which was a really nice thing to do. Positive reflection.”

“Help to give an overall view of this academic year.”

“It brought together different aspects of my learning to date.”

“It linked in the extra curricular activities to the medical course.”



ePortfolios

Process (portfolio / appraisal) n=17


“Having a timetabled session forced me to compile my portfolio better as it gave me a deadline to work towards.”

“It allowed me to find out if I was completing the portfolio and using it in the correct way. It gave me some feedback on my general progress this year.”

Learning Evidencing Skills n=17

“before the appraisal session I didn’t know what sort of evidence could be included in some of the sections, for example, 'managing my own learning' seemed vague to me and I didn’t know what to include as evidence…”

“helpful to be able to reflect upon my development especially learning what counted as evidence.”


ePortfolios

Reassurance n=8

“Chance to talk through how I felt the year went and what I was worried about.”

“It reassured me that what I had been doing was right.”


Planning / goal setting n=14

“…thinking about how I will continue next year was useful as it is good to make plans now rather than just let yourself slip back into bad habits.”

“It highlights what I need to improve on and how it could be done”


ePortfolios

What did you least helpful about the appraisal session ?

75 students responded

Personal / Interaction 20Timing (near exams / lectures) 19Rational / understanding the process 9Limited time of session 7No benefit (reason unspecified) 7


ePortfolios

Timing (near exams / lectures) n=19

“It came at a time when I wanted to be revising for exams. It would be more useful after the exams.”

“It was at a time when we all had better things to be doing, such as revising for exams. It would have been better to have it before Easter.”


“… the session itself seemed to be more for the benefit of the appraiser than for me, as it mainly comprised of me simply reading out sections from my portfolio, which felt a little pointless.”

“I suppose there was a lot of ticking boxes and it wasn't very personal.”

Personal / Interaction (n=20) n.b. this is a very ‘mixed bag’


ePortfolios


Rational / understanding the process (n=9)

“I didn't really know before the session how much evidence or how many reflections were required to satisfactory evidence.”

“I didn't really know what I had to do. It made preparation difficult.”

Limited time of session (n=8)

“It felt a little rushed”

“The time - too short which meant we could not discuss fully some of the issues I raised.”


ePortfolios

Following the session what aspect of your development do you feel that you most need to improve on ?

(open ended question, answers categorised by SC)

“Learning some Geordie to be able to communicate more effectively with patients during hospital visits!”

Organisation & Time management 34Study Skills 21Creativity / Resourcefulness 11Motivation 9Communication Skills 6Reflection 6Extra curricula activities 6Clinical skills 2Misc. 9


ePortfolios

Appraiser Feedback (n=7)

““The other thing that would definitely make the appraisal easier The other thing that would definitely make the appraisal easier would be if the students were asked to let us have their form would be if the students were asked to let us have their form in advance. Even a couple of days before would be okay by in advance. Even a couple of days before would be okay by me and it would mean you entered the appraisal with a me and it would mean you entered the appraisal with a clearer picture of how each student felt they were doing.”clearer picture of how each student felt they were doing.”

““A common problem was the students filled in the summary of A common problem was the students filled in the summary of evidence (in a hurry the night before) and had no evidence evidence (in a hurry the night before) and had no evidence attached to support it. I therefore spent a fair amount of time attached to support it. I therefore spent a fair amount of time explaining how to get and attach evidence and how the explaining how to get and attach evidence and how the should only really refer to this evidence in the summary of should only really refer to this evidence in the summary of evidence box.”evidence box.”


ePortfolios

Appraiser Feedback (n=7)

““Some did not grasp what the appraisals were about and clearly Some did not grasp what the appraisals were about and clearly placed a low value on them. Many said they would rather be placed a low value on them. Many said they would rather be revising. I think some form of tangible reward should be revising. I think some form of tangible reward should be offered - though not sure what - they seem to value grades, offered - though not sure what - they seem to value grades, and you don't want this to be an assessment process - bit of a and you don't want this to be an assessment process - bit of a conundrum there.”conundrum there.”

““Most sessions went over the 15 minutes. I'd say 20 minutes Most sessions went over the 15 minutes. I'd say 20 minutes would be adequate.”would be adequate.”

““I would urge you to consider doing this in second year as well I would urge you to consider doing this in second year as well as I believe the appraisal gives the student an opportunity to as I believe the appraisal gives the student an opportunity to reflect on the year; something that only a small proportion of reflect on the year; something that only a small proportion of the year do as a matter of course!”the year do as a matter of course!”


ePortfolios

Summary #1• Overall, 70% of Stage 1 students found it a positive

experience

• Students found the process useful for reflection, interaction/discussion & planning.

• Some students found this process instructive on how to evidence their development for future appraisals/assessment

• 58% felt it would influence their learning in Stage 2

• Many students were not clear on the purpose/rationale and specific requirements (a common issue in portfolio learning, increased training planned).

• Timing – near exams – was an issue


ePortfolios

Summary #2

• Overall, appraisal supported by ePortfolio has been successful for MBBS stage 1

• Following successful evaluation it is being rolled across the programme

• Refinements to the process were implemented for 2006/7• 4 extra skills to evidence for the later years• Timing of appraisals brought forward – away from exams

• Features of ‘My Evidence’ were incorporated back into the ‘generic’ ePET portfolio

• Later expanded to link Blogs to structured skill/outcomes (EPICS-2)


ePortfolios

Further information:Further information:

[email protected]


Documents

Http:// ePortfolios Using ePortfolios to support annual appraisal in undergraduate medicine Simon Cotterill, Philip Bradley, Rosie