Upload
dangcong
View
216
Download
1
Embed Size (px)
Citation preview
GPST 1 and 11 Modular Programme 2007-8
EVALUATION
Remote and Rural Healthcare Educational Alliance
Remote and Rural Healthcare Educational Alliance
RRHEAL 2010 2
Acknowledgements
The authors wish to thank Alison Crosbie, Laura Stewart and Roslyn MacDonald for their
assistance with this first RRHEAL team effort
RRHEAL 2010 3
Contents
INTRODUCTION AND BACKGROUND ....................................................................4
METHOD................................................................................................................6
Consent, approval and confidentiality ............................................................................ 6
1. Investigator familiarisation conducted through exploratory interviews with faculty
members ......................................................................................................................... 6
2. Observation of a video-conferenced educational intervention................................... 8
3. Semi-structured telephone interviews with a purposive sample of participant
learners ........................................................................................................................... 8
4. A questionnaire administered to the full cohort conducted through telephone and
email communications .................................................................................................... 9
RESULTS.............................................................................................................10
1. Interviews with the GPST Faculty (n.4) .................................................................... 10
2. Observation of a single video-conferenced educational session............................. 11
3. Interviews with a purposive sample from GP training sites ..................................... 14
DISCUSSION .......................................................................................................28
RECOMMENDATIONS..........................................................................................29
CONCLUSIONS....................................................................................................29
APPENDIX 1 ................................................................................................................. 30
APPENDIX 2 ................................................................................................................. 35
APPENDIX 3 ................................................................................................................. 37
APPENDIX 4 ................................................................................................................. 41
APPENDIX 5 ................................................................................................................. 42
APPENDIX 6 ................................................................................................................. 48
RRHEAL 2010 4
INTRODUCTION AND BACKGROUND
In 2002 the 'Unfinished Business: Proposals for Reform of the Senior House Officer' 1document proposed a revision of the way in which postgraduate medical education is
organised. Doctors who have chosen a career in General Practice enter three years of
specialised training: General Practice Specialist Training (GPST).
In responding to the framework that has arisen from Modernising Medical Careers
(MMC) the North West Deanery has designed a modular education programme for 2007-
8 using a proportion of the maximum of 30 days that they could claim - 20 days in total
- for an educational programme that relies heavily on video-conferencing; trainees could
feasibly be based in any one of ten hospitals in the four North of Scotland Health Boards
and re-design was to alleviate some of the geographical access barriers faced by GPST
learners who access training from remote locations.
RRHEAL undertook this work in partnership with the North Deanery GP Faculty at the
request of Dr Ronald MacVicar of the North of Scotland Deanery GPST Programme
Board. This work leads to an understanding of the advantages and disadvantages of this
type of education delivery for the purposes of identifying best practice and informing
development of future remote and rural programmes.
By developing a baseline description of the 2007-8 programme the completed work will
provide the GPST Faculty with a platform for future audit and evaluation and provide
RRHEAL with intelligence around the rural proofing of educational interventions.
Assessment of the level of skills and knowledge acquisition using this method is
conducted elsewhere and is not the subject of this study.
1Unfinished Business - proposals for reform of the senior house officer grade, September 2002, Sir Liam Donaldson, Chief Medical Officer for England, RCGP summary paper 2002/13 http://www.rcgp.org.uk/pdf/ISS_SUMM02_13.pdf
RRHEAL 2010 5
Kirkpatrick’s model of evaluation2 was selected as a framework; this is based on four
data outputs; Reaction, Learning, Behaviour, Results. The model effectively measures:
1. Reaction of student - what they thought and felt about the training
2. Learning - the resulting increase in knowledge or capability
3. Behaviour - extent of behaviour and capability improvement and
implementation / application
4. Results - the effects on the business or environment resulting from
a trainee's performance
It was agreed that a questionnaire would be administered by telephone. This was based
on a number of factors:
• Time constraints - the necessity for this evaluation to be completed in time for the
design of year two of the training, due to start in August 2008.
• Logistics - the geographical spread of the respondents meant that travel to each
site would be difficult
• Value for Money - the evaluation was cost-neutral at the point of application
• Format of Questionnaire - the questions were not complex, requiring only yes/no
responses, with the exception of four open questions
The work conducted by the RRHEAL core team with GPST Faculty and 1st and 2nd year
trainees provided the following investigation outputs:
• Description of the characteristics of this new intervention.
• Baseline measures of satisfaction, learning, self-reported behaviour change and
impact on local health services.
RRHEAL staff resource was made available and a schedule of staged reports was
provided to the GPST faculty.
2 Evaluating Training Programmes – The Four Levels, Donald Kirkpatrick (2006) Berrett-Koehler
RRHEAL 2010 6
METHOD
The study was conducted in four parts and with the agreement of the GPST faculty
(appendix 1)
1. Investigator familiarisation conducted through exploratory interviews with faculty
members
2. Investigator observation of a video-conferenced educational intervention
3. Semi-structured telephone interviews with a purposive sample of participant
learners
4. Questionnaire administered to the full cohort conducted through telephone and
email communications
Consent, approval and confidentiality Individual consent to participation was gained at each interface with a learner.
Management approval for the exercise was achieved by the GPST Faculty.
Confidentiality is maintained throughout; access to the RRHEAL drive of the NES server
is restricted to members of the RRHEAL core team and is password protected.
1. Investigator familiarisation conducted through e xploratory interviews with faculty members
Four members of NES staff were involved in this part of the investigation. Three
interviews were reported before development of the semi structured telephone interview
of the purposive sample and one during the pilot phase of the survey tool development
for the full cohort.
RRHEAL 2010 8
2. Observation of a video-conferenced educational i ntervention
The educational sessions are conducted via video conferencing (VC) links with up to 8
sites. The North of Scotland NHS VC bridge is booked by Deanery administrative staff
and the local equipment is managed by administrators, IT technician, educators and
learners. The level of technical support is variable across the sites and educator / learner
expertise has improved with practise.
All Tandberg VC units are compatible and were purchased with Scottish Telemedicine
Action Forum funds, and for the most part, administered through the former North of
Scotland Tele-Education Project.
A member of the investigation team was a silent observer at a locally delivered
emergency scenario workshop that was also delivered via video conference to remote
learners. Photographs were taken with consent. A report was shared immediately post
event with the GPST Faculty for validation and to support educator learning.
3. Semi-structured telephone interviews with a purp osive sample of participant learners
A purposive sample of eight current or recent learners was selected from the full cohort.
This sample was selected based on gender, previous experience of learning in a remote
setting, clinical location, employing Health Board, experience of group-based learning,
undergraduate education, experience of traditional undergraduate education and
ethnicity. The questionnaire consisted of seven open questions (Appendix 2).
Information from the ‘observed education via video conference’ and faculty interviews
was used to construct open questions for the semi-structured telephone interviews.
A baseline qualitative telephone interview was conducted with 6 respondents.
Each member of the learner cohort was given a unique ID to identify double reporting
when a learner participated in both initial interview and subsequent telephone
administered survey.
RRHEAL 2010 9
4. A questionnaire administered to the full cohort conducted through telephone and email communications
The responses to the baseline questionnaire conducted with the purposive sample
informed the design of a topic-specific structured questionnaire based on Kirkpatrick’s
model of evaluation. This questionnaire contained twenty nine questions including four
open questions. (Appendix 3).
The GPST Training Programme Board provided the contact details of a cohort of 39
trainees, which included the six who had completed the baseline telephone interview.
The telephone administered questionnaire was viewed by a member of the GPST
Programme Board prior to commencement of the telephone questionnaires, and
deemed to be fit-for-purpose. The telephone technique and estimate of time was piloted
in-house and the questionnaire was found to be straightforward and the timing of the
interview reasonable.
A script was provided for telephone interviewers (Appendix 4) Respondents were
contacted by telephone at their place of work and offered the choice of completing the
questionnaire immediately or later through a confirmed email address.
Interviewers recorded telephone responses onto a paper version of the questionnaire
before entering the data onto an Excel spreadsheet to allow data verification. Emailed
responses were printed off and entered onto the Excel spreadsheet by a member of the
RRHEAL team. Respondents were offered feedback opportunity on completion of the
evaluation and collation of the results.
The RRHEAL office issued one reminder by email to each non respondent (Appendix 6)
Qualitative data
Comment from telephone conversations was transcribed and recorded with data
extracted from the response to the open question in the final survey tool.
Comment was grouped into themes for reporting purposes.
RRHEAL 2010 10
RESULTS
The overall response rate was 23 from a possible 37 (The full cohort had been 39
however 2 learners resigned their posts before evaluation was completed); 6 took part in
initial interview, and survey respondents numbered 20. Three took part in initial interview
but did not complete the final survey.
1. Interviews with the GPST Faculty (n.4)
GPST Faculty know that learners hold information about the acceptability of the new
delivery mode that may not otherwise be captured. A conventional survey tool i.e. a
simple questionnaire could not capture the nuances of complex educational delivery.
They were aware of the number of surveys that will be conducted with this year group
across Scotland by professional bodies and are not able to wait for formal reporting as
they have to begin preparation for the next season of learners starting on August 4th
2008.
GPST faculty video conferencing competences were acquired on an ad hoc basis;
through observation and through experimentation. Training faculty list complications and
frustrations as:
Bridge failings: lost connection with a remote site, loss of connection from the
host site, loss of voice package, combinations of audio and visual reliability/loss
Human error: little experience of VC technique, i.e. visiting clinical experts who
were unfamiliar with controls
There is a wealth of educational expertise available within this team. They are conscious
of the need to prepare trainee GPs for independent practice where they will often be
isolated from peer support; decision making and problem solving skills are therefore of
paramount importance.
RRHEAL 2010 11
The aim of the educators is implicit. When the aim is made explicit to the learners there
is some reporting of ‘too many introductions’ and a potential gap in a learner’s
understanding of the reasons for using group work pedagogy.
2. Observation of a single video-conferenced educat ional session
Preparation : Preparation started well in advance of the date with dissemination of
meeting dates, alternative dates, programme and handouts
Technical: The host site did not have a technician, all sites were using Tandberg
equipment, the VC bridge manager was not directly involved and GP trainers managed
all technical aspects of the intervention
Equipment used: Screen, Microphone, VC, digital camera, BASICS resuscitation bag,
ALS manikin, Automated External Defibrillator, patient trolley, mobile phones.
Participants:
Remote sites: Fort William - 1 Lochgilphead - 1 Oban - 2
Host site: GP trainers – 2 Visiting consultant – 1 Learners – 3 Observer – 1 Actors – 2
Visual: Camera angles were considered in the placement of teaching aids and
positioning of actor ‘casualties’ during the preparation period. Camera angles at remote
sites were corrected without prompting. A dynamic scenario was used so the view from
the remote sites was limited at times. There was good collaboration between learners
without prompting, for example: holding equipment up to camera level for the remote
colleagues.
Audio: Good sound levels were maintained throughout.
RRHEAL 2010 12
Dialogue: There was good introduction from the host with a description of contingency
for example ‘If we struggle to maintain links, stay on line, if you struggle then re-dial
etc…’
The scenario was explained as exploring reality in a GP rural setting with request for
audience participation ‘Don’t sit back to watch a show! Volunteer here!’
The role of the remote learners was described ‘You will play the role of mentor or
observing angel!
The host reminded everyone about remote observers and actively managed the group
discourse
Advice was given to maintain VC links during a brief refreshment break. Opportunity was
available locally for informal ‘coffee queue learning’ that is denied to remote sites
Presentations: Not used
Confidentiality: Maintained
Good practice was observed
• Presenters were well prepared and self taught;
• Host site speakers were conscious of and responsive to camera angles and
sound levels;
• Telephone numbers for mobile phones were available for remote/host site
contact;
• Handouts were distributed to remote sites 2 days in advance;
• Introductions were made with description of contingency plans and timing of
breaks;
• Familiarity was demonstrated with equipment – both faculty and learners;
• Administrative support was available to alert participants via mobile phone to ‘sit
tight’ when the host site bridge ‘failed’ and whilst waiting for re-connection;
conducted locally by re-dialling.
RRHEAL 2010 13
• Active group management kept the learning interactive;
• Group - both learners and faculty were ‘spotting’ for those wishing to speak;
• A brief re-capitulation was given when the bridge was re-established after a 5
minute failure;
• Camera angles were checked throughout;
• Sound clarity was checked throughout;
• Organisers included actors and expert in the feedback and closure session;
• Timing was accurate.
The skills used are:
• 1:1 dial to remote centre;
• Arranging teaching meetings, and co-ordinating diaries and facilities for use of a
VC bridge;
• Use of the technology to give a powerpoint presentation from a linked laptop PC;
• Using digital photography to present Xrays and clinical photographs of injury for
education purposes;
• Remote management of dispersed groups and PBL sets (Problem Based
Learning);
• Demonstration of practical skills in a clinical workshop scenario
RRHEAL 2010 14
3. Interviews with a purposive sample from GP train ing sites (n.6) and open
comment from researcher administered telephone surv ey (n.20) reported early as
qualitative data (Appendix 5)
• Initial reaction
There are access benefits for the remote cohort however there is potential for this to be
interpreted as disadvantage by the Inverness cohort whose more personal face to face
session is fragmented when audio visual aids fail:
‘Modular programme ok but VC detracts’ (C6)
‘More structured than I am familiar with’ (R2)
Learners report being ‘sceptical’ and ‘nervous at first’ before improving their
understanding, knowledge and skills of the techniques being used.
Learners report a double set of learning for remote sites:
1) learning to use video conferencing
2) learning curriculum content
• Preparation for learners
There is awareness that participation is crucial and this includes preparation of the
environment for example remembering to have a mobile phone to link back to the Host
administrator. Interactive sessions are thought to be helpful in highlighting local needs
and thereby setting out the context of future practice
• Preparation of tutors
There was a mixed response in this theme, content is seen to be relevant to most and
follows the curriculum, some suggest that only a handful are relevant.
‘Suboptimal [delivery].’( R3)
‘Lecturers condescending / patronising.’(C2)
‘Training is Inverness orientated’(R13)
‘Trainers should involve peripheral sites or otherwise it is like watching someone else
getting a lecture’ (R13)
RRHEAL 2010 15
• Good practice
Learners appreciated the facts that there were no cancellations, that sessions were
repeated and ran to time. They noted the good organisation and valued receiving papers
in advance.
• Rotation
Some of the cohort experienced repetition of learning opportunity on the rotation due to
geographic setting and there were perceptions of lack of recognition of previous
experience. Remote learners could not access ENT, Opthalmology, Obstetrics and
Gynaecology etc so they perceived disadvantage and lack of choice for those taking a
remote placement.
‘One year in rural is too long’(R2)
• Content
There is a request for more typical day-to-day patient scenarios
‘rather than the theoretical ones you may never come across.’ (R1)
Although designed by GPs for GPs learners suggest that this is not GP focussed; that
they have a sense of being assessed rather than leaning new skills and that the
curriculum is daunting. Re-design is suggested to include
‘asking a Consultant in the speciality to identify what a GP should know’(C2)
It was suggested that links could be made from one topic to the next and that guidance
could be given on information and helpful hints on what to expect as a GP registrar.
…and appreciation
‘We have completed more than half of the headings in the e portfolio’(R11)
• Content choice
Some would appreciate opportunity, expressed by others as flexibility that is built on
learner reflection in order to influence content of the programme
RRHEAL 2010 16
‘My hands are pretty tied; I take it or leave it’ (R14)
• Method
There were many comments about repeated introductions with suggestions for an
introductory lead lecture (locally delivered) followed by discussion across sites via video
link and introduction reminders that could come out with advance papers.
Styles were discussed with some wanting
‘More direct lectures and less hypothesis’ (C1)
On line modules and short attachments to
‘increase range of hands- on’ (R1)
‘Video linking has its place but it would be helpful to have short attachments to other
hospitals to undertake training’ (R1)
A full 5 day training course was proposed by one learner as a preferred option with use
of pre and post course evaluation with scoring to support future design
There appeared to be lack of understanding of the use of group discussion for learning
with some requesting more lectures, visiting specialists speakers, less group discussion,
shorter sessions and noting that
‘Scenarios make peripheral site people feel more isolated’ (C8)
‘Workshops don’t work by VC ’ (R4)
There is a note that some content feels irrelevant if one cannot access the clinical setting
for example Orthopaedics, Paediatrics, ENT, Obstetric and Gynaecology etc
…and finally ‘this…
‘Has ensured that all GPS trainees have had continuity in teaching & training availability’
(R3)
RRHEAL 2010 17
As a result the teaching is more standardised and more organised and this has meant
that it is of a higher quality. (R13)
• Technology
There is recognition that poor connectivity and technical problems let the educational
team down and that sound and audio can be improved.
‘Using video conference education is a great idea especially for trainees in remote
places. However, if we are taught how to use it and make sure it is working properly
each site would help video conference better’(R11)
• Comment on new learning
‘Preparation is key to the sessions. I normally attend with a colleague so we can prepare
together and go into the sessions totally ready for them’(R15)
Comparing this model to another Deanery’s model ‘ My sessions address the curriculum,
his [friend’s] Deanery doesn’t offer this option’ (R14)
‘Well - current teaching program we hardly learn anything new and I strongly believe that
3 hours is better spent in the wards seeing patients’ (R1)
• Changes in the learner and raised awareness
‘Better use of formalised reflective learning -a better appreciation of my learning style
after 25yrs of education.’ (C5)
Comments on changes in the learner include raised awareness of their knowledge and
learning needs, a new attitude towards self-directed learning and better knowledge of
where to source information. On use of the experiential log
‘It is a good learning tool and I use it very regularly’ (R3)
Some nervousness is noted
‘I still feel very nervous about becoming a GP registrar because I don't really know what
to expect’ (R10)
RRHEAL 2010 18
the need to contact and involve Primary Care colleagues for their input
‘I now know that I have the support of colleagues outwith my area’ (R12)
‘I am more likely to make use of my colleagues in other locations now’ (R4)
and other comments include those noting improved community orientation, holistic
approach and better problem solving skills
‘I think the best changes I have adapted is treating patients holistically which has
positively reflected on my decision-making process’(R9)
• Impact on local health service delivery
‘It is difficult to get away from the workplace for 3 hours, it is hard for the hospital to
cover service commitments and training too. Foundation year doctors have a compulsory
day off and have to leave Western Isles for their sessions.’ (R11)
‘When I was in Lochgilphead I had to go to Argyll College. It is hard to miss a whole
morning, in a small hospital there are so few doctors on the rota, so you can’t like say to
someone will you hold by bleep.’ (R14)
‘Not too bad. They are well provided for in terms of staff cover to allow them to
undertake the training.’ (R1)
‘With the on-call situation, it has been difficult to find time to attend the training courses’.
(R2)
‘There has consistently been someone missing from the rota, so I have had to miss
some sessions due to lack of cover.( R15)
‘It has been difficult to attend due to rota problems’. (C8)
‘fantastic way of involving trainees in remote sites with out cost of travel in terms of time,
money and environmental impact.’ (C1)
RRHEAL 2010 19
‘It is more of a hassle, because the rest of the team thinks we are skiving away from
work in the name of teaching and we get only two hours of whole day especially after
being busy on call or other work. We don't really get half day off for teaching. No
compensatory time off when we attend teaching when we are on nights or late shifts.’
( R2)
‘NHS has saved money, good cover has always been available’ (C2)
‘To be honest, given that I was in the central group, VC education per se (i.e. divorced
from the content of presentations/group work/etc) has probably made no difference to
local health care delivery, except for an excellent session structured and given by Dr.
Douglas at Fort William’ (C7)
‘ There is a strain on the service due to time away for training. Unfair on colleagues’
(R8)
‘On the whole, I think for me it has been a successful programme with variable effects
(mostly negative) on the health service system’ (R9)
‘It's cheaper than flying us over to Inverness for face-to-face teaching sessions, and
therefore saving them cash.’ (R10)
Researcher administered telephone survey for the fu ll cohort of 37 learners
The quantitative results are based on responses from 20 respondents. A selection of
the yes / no responses have been presented in graphical form for ease of
understanding. The responses as a whole have been compared with the responses from
remote sites where it was felt that the information was relevant.
RRHEAL 2010 20
Figure 1
0
2
4
6
8
10
12
14
16
Yes No
Have you previously received training in this forma t?
75% of respondents have never received training in this format.
Figure 2
0
2
4
6
8
10
12
14
Yes No
Prior to recieving this training were you taught ho w to use the equipment sufficiently to allow you to use it?
Total
Remote Sites
RRHEAL 2010 21
Figure 3
0
2
4
6
8
10
12
14
Yes No
Have you enjoyed learning in this format?
Total
Remote Sites
65% of respondents said they enjoyed training in this format. Of that number, 70% were
respondents from remote sites.
Figure 4
0
2
4
6
8
10
12
14
16
Yes No Sound Connection
Have you experienced technological difficulties wit h this form of training?
Total
Remote Sites
A large proportion of trainees reported technological problems associated with receiving
the training in this format, both in sound and connection.
RRHEAL 2010 22
Figure 5
0
2
4
6
8
10
12
14
Yes No
Have you felt isolated by this form of training?
Total
Remote Sites
35% of trainees felt some level of isolation from receiving training in this format, while
65% did not. Of note is the fact that, of the 35% of trainees who did not feel isolation,
61% of these trainees were from remote sites.
Figure 6
0
2
4
6
8
10
12
14
Yes No
If this format of training was not available to you , would you have to travel for more than one hour to
access tutor-led education?
RRHEAL 2010 23
Figure 7
0
2
4
6
8
10
12
14
16
Yes No
Do you feel that this form of training allows you t o learn more effectively than other forms of training you have received?
Total
Remote Sites
25% of trainees felt that this form of training allowed them to learn more effectively than
other forms of training they had received. All were from remote sites.
Figure 8
0
2
4
6
8
10
12
14
Workshops Direct Lectures
Workshops and Discussions Versus Direct Lectures
65% of respondents reported that workshops and discussions enabled them to learn
more effectively than direct lectures. However, those who did express a preference for
direct lectures did so very strongly and, at times, named individual speakers or specialty
lectures that they had found particularly inspiring or useful as learning tools.
RRHEAL 2010 24
Figure 9
0
2
4
6
8
10
12
Yes No
Do you feel that there are aspects of the training that are repititious?
55% of trainees felt that some aspects of the training were repetitious. In some
responses to these questions, respondents mentioned by name specialty training that
they felt they were either not receiving or were receiving too much.
Figure 10
0
2
4
6
8
10
12
14
16
Yes No
Do you feel that there is a natural flow of learnin g from one session to the next?
75% of respondents felt that there was no natural flow of learning from one session to
the next. However, of the 25% who did feel there was a natural flow, 80% were from
remote sites.
RRHEAL 2010 25
Figure 11
0
2
4
6
8
10
12
After each session At completion of training
At what stage did you feel that you were able to apply the skills you learned?
65% of respondents felt able to put their skills into practice after each session, while 35%
only felt able to do this after completion of the training module. One respondent did not
feel able to answer this question.
Figure 12
0
2
4
6
8
10
12
Yes No
Do you agree with this statement? 'Learning in this setting prepares me for the reality of being a GP'
Total
Remote Sites
40% of respondents agreed that learning in this setting prepared them for the reality of
being a GP. Of this number, 87.5% were from remote areas.
RRHEAL 2010 26
Figure 13
0
2
4
6
8
10
12
14
16
Yes No
Do you feel disadvantaged by this form of training?
Total
Remote Sites
75% of trainees do not consider themselves to have been disadvantaged by this format
of training, and over half of these are from remote sites.
Figure 14
0
2
4
6
8
10
12
Yes No
Do you feel that your experience as a whole support s the continuation of this form of training?
Total
Remote Sites
60% of trainees felt that, as a whole, their experience supported the continuation of
training in this format.
RRHEAL 2010 28
DISCUSSION
It is encouraging to note that two thirds of trainees feel that they are learning
successfully from workshops and discussions, however, one third of students appear to
be missing the importance of the group work as an integral part of the course design and
General Practice training itself.
Some of the trainees used the interviews and survey opportunity to express their
concerns about specialities in which they were not receiving any training, and suggested
attachments in ENT, Orthopaedics, Ophthalmology, Obstetrics and Gynaecology.
There is a ‘lead in’ time before learners grasp the skill to use the video technology and
understand the role of ‘group work’ education.
There is some evidence that use of powerpoint presentations that are visible only to the
local or host site enhances feelings of powerlessness and isolation from the main body
of the learner cohort.
The faculty interviews did not include guest speakers and tutors and this may have
provided a more rounded view of stakeholder opinion on utility of this modified
intervention
RRHEAL 2010 29
RECOMMENDATIONS
Videoconferencing skills are learnt in an ad hoc way that in future could be more
formalised for both learner and faculty
Some remedies can be suggested for immediate application for example:
• Hand signals or cue cards when wishing to speak
• Develop a set of cue card to indicate yes, no, a question or loss of sound
• Presentations by power point to all or none
Further research is required into working out the correlation of rate of attendance at
educational sessions and satisfaction with the delivery format.
CONCLUSIONS
This study has enhanced our understanding of the delivery method
It provides baseline data for future audit
It will form part of a more academic discussion around adaptation of GP education to
meet the needs of learner and service requirement in the remote setting.
RRHEAL 2010 30
APPENDIX 1
First plan as agreed with R MacVicar
GPST Evaluation
Contact:
Dr Ronald MacVicar
Assistant Director of Postgraduate General Practice Education
NHS Education for Scotland
Centre for Health Science
Old Perth Road
Inverness
IV2 3JH
EMail: [email protected]
Tel: 01463 255710
Investigators:
Gillian Swan, Michaela Rodger, RRHEAL
Introduction
Description of postgraduate GPST vocational training delivery
Extract from Associate Dean’s report April 2008
General Practice Specialty Training
A group from the local Adviser team (the “GPST Group”) oversees issues relating to
GP Specialty Training (GPST) in the Highlands & Western Isles and copies of
minutes of meetings are freely available from [email protected]
RRHEAL 2010 31
2.2.1. Background
We are now nine months in to the new world of GP Specialty Training and as
such we have left behind terminology such as “Vocational Training”, “Green
Card Schemes” and “Self Construct Schemes”. While it may be tempting to
think of this as simply a change of nomenclature, there are substantial changes
in this new world, including bespoke three year GPST Programmes , a GP
Curriculum (available at www.rcgp.org.uk/default.aspx?page=2561), Educational
Supervision for the whole programme coming from General Practice and an
appropriate competency based assessment process.
Current configuration of our GPST Programmes in the Highlands and Western
Isles is less that ideal. As a result we have been unable to offer our trainees
either the breadth of experience or the geographical stability we would wish
them to have. Most of our trainees have to spend at least six months
geographically distant from their training base. More problematically, our current
programmes are made up of 24 months in hospital and 12 months in a General
Practice setting, whereas the curriculum is designed to be delivered over a
training programme that includes 18 months in each setting. Changes to the
programmes from August 2008 will address this problem
2.2.2. Current Delivery
Notwithstanding the fact that the programmes are less that ideal, it is the task of
the educational system (Programme Directors, Educational Supervisors and
Clinical Supervisors) to ensure that the new curriculum is delivered across the
three years of the training programme.
As with much of the jargon in the post MMC world, the terminology is new but
the requirement to support trainees through their training is familiar. One
significant change however is that Educational Supervision for the whole three
year programme comes from General Practice and, despite a lack of adequate
funding support, our GP Trainers have enthusiastically taken up the challenge of
RRHEAL 2010 32
this important role. The result is an improved focus on General Practice
throughout a GP Specialty Training experience and this is enhanced by practice
attachments in the Educational Supervisor’s practice (for one week during each
six month hospital attachment).
The Programme Director role is a new one for our team and these functions are
shared by Doctors Jerry O’Rourke, Sue Tracey, Jim Douglas (Fort William
focused programmes) and Bob Dickie (Western Isles focused programmes). A
major focus for the Programme Directors in this first year of GPST has been in
the design and delivery of a modular educational programme for trainees on
their first year (ST1) and second year (ST2). This is no mean task given that our
ST1 and ST2 trainees can be in any one of ten hospitals around the North of
Scotland. For this group we have implemented an innovative modular
educational programme which relies heavily on video-conferencing and which
has been evaluated very positively so far. We will undertake a formal evaluation
of this innovative approach to GP Specialty Training before the end of July.
Project Aims
• To provide a description that characterises the 2007-2008 GPST delivery mode
in NW Deanery, NES, Scotland
• To provide baseline measures for future audit
• To identify best practice
• To inform iterative development of future programmes
Outputs
An understanding of the benefits of this type of educational delivery
RRHEAL 2010 33
Method
Investigator familiarisation:
Meetings with Ronald MacVicar, Sue Tracey, Jerry O’Rourke
Completed: Gillian Swan
Observe an educational intervention
Action: Gillian Swan 9 th June
Develop structure for telephone interviews
Completed: Gillian Swan
Management approval
Action: Ron MacVicar to arrange for 3 Health Boards
Complete: NHS Highland
Identify a purposive sample of 8 learners to represent gender, previous experience of
learning in a remote setting, each clinical location, each Health Board, experience of
group based undergraduate education, experience of traditional undergraduate
education and if necessary age range and ethnicity. (appendix1)
Completed: Sue Tracey/Laura Stewart
Provide telephone contact details of sample to Investigator
Complete: Laura Stewart
Gain consent and conduct 8 semi structured telephone interviews
Action: Gillian Swan
2 completed
Analyse interviews and develop a questionnaire for telephone administration
Action: Michaela Rodger/Gillian Swan
RRHEAL 2010 34
Provide telephone contact details of all learners to investigation team
Action: Laura Stewart
Complete telephone questionnaire with entire cohort
Action: Michaela Rodger and RRHEAL Core team
Record and analyse data
Action: Michaela Rodger
Complete reports to NW Deanery team
Action: Gillian Swan and Michaela Rodger
Analyse and identify benefit, barriers and contribution to remote and rural practice. Use
results to contribute to RRHEAL Impact Assessment project.
Action: Gillian Swan and Michaela Rodger
Limitations
Explorations of group work dynamics are out with the scope of this project. The benefits
of group work learning and co-operative learning are well documented elsewhere
(references needed)
Compliance with curriculum requirements will be assessed on an individual basis and by
the success rate of learners completing this GPST year.
Confidentiality
All data will be stored on password protected computers in the RRHEAL Administrative
office.
RRHEAL 2010 35
APPENDIX 2 Interviews with a purposive sample from the 8 GP training sites:
1. Raigmore Hospital
2. Belford Hospital (Fort William)
3. Lorn & Isles Hospital (Oban)
4. Western Isles Hospital (Stornoway)
5. Argyll & Bute Hospital (Lochgilphead)
6. Caithness Hospital (Surgery)
7. Balfour Hospital (Orkney)
8. Gilbert Bain Hospital (Shetland)
Introduction Hello, My name is …... I have been co-opted onto the North Deanery GP ST programme to lead the baseline evaluation. I will be designing a formal questionnaire for all participants and wondered if you have 5 minutes to talk to me now? Confirm confidentiality and researcher contact details Questions Q1 What were your first impressions of the way the education and training is being delivered? Q2 How suitable is it for your workplace setting? Q3 What has worked well? Q4 What should the training team do differently? Q5 What if anything would you do differently? Q6 (added after interview 1) Have you used PBL (Problem based learning) before?
RRHEAL 2010 36
Q7 (added after interview 2) Have you got previous experience of learning from a remote setting? Closure
• Researcher contact details confirmed • Advised re future phone call to all • Thank you
RRHEAL 2010 37
APPENDIX 3
GP Specialist Training Telephone Questionnaire
Dear Doctor
As part of the Remote and Rural Healthcare Educational Alliance we have been asked
by the North of Scotland Deanery GPST program to evaluate the new format of GP
Specialist Training.
Our aim is to capture the experiences of trainees such as yourself to assist with this
evaluation and to inform the development of future training programs. All of your
answers will be recorded and analysed anonymously.
We require a 100% response by Tuesday 8th July 2008 at 4pm and are grateful for your
co-operation.
Name ………………………………………………………… GPST 1 or 2
Specialty
………………………………………………………………………………………………
Place ………………………………………………………………………………………………
QUESTIONNAIRE
QUESTIONS YES NO
1. Have you previously received training in this format?
2. Prior to the beginning of this training, were you taught how to
use the video-conferencing equipment to a standard that
allowed you to use it confidently?
3. Have you enjoyed learning in this format?
4. Have you experienced technological difficulties with this
RRHEAL 2010 38
format of training?
• Connection? (please indicate)
• Sound quality?
5. Have you felt isolated by this format of training?
a. If yes, is this because you are receiving the training as the
only learner at your site?
6. Do you think you would feel less isolated if you had
colleagues in the room with you?
7. If this format of training was not available to you, would you
have to travel for more than one hour to access tutor-led
education?
Please explain your reaction to education via this type of media in one sentence
8. Do you feel that this format of learning physically allows you
to learn successfully?
9. Do you feel that you are actually learning new skills?
10. Do you feel that you are being assessed on what you already
know rather than learning new skills?
11. Do you feel that this format allows you to learn more
effectively than other formats of training you have
experienced?
12. Do you feel that direct lectures enable you to learn more
effectively than workshops and discussions?
13. Do you feel that workshops and discussions enable you to
learn more effectively than direct lectures?
14. Do you feel that there are any aspects of the training that are
repetitious?
15. Do you feel that there is a natural flow of learning from one
session to the next?
Please describe in one sentence what you would change in the program if you had the choice
RRHEAL 2010 39
16. Do you feel confident enough to be able to immediately put
the skills you have learned at each session into practice
when you are back in your workplace?
17. Did you feel that you were only able to put the skills you
learned into practice once you had received the whole
training module?
18. Do you agree with this statement? ‘Learning in this setting
prepares me for the reality of being a GP’
19. Did you feel that there was a noticeable improvement in your
performance after each session?
20. Did you feel that there was only a noticeable improvement in
your performance after the whole training program was
completed?
Please describe in one sentence the best changes in performance or decision-making that you
have made since taking part in this program
21. Do you feel that the quality of your input into the workplace
has improved from having received your training in this
format?
22. Do you feel that you are more able to retain what you have
learned from having received training in this format?
23. Do you feel that you are less able to retain what you have
learned from having received training in this format?
24. Do you feel that, on the whole, you have been disadvantaged
from having received training in this format?
25. Do you feel that your learning experience as a whole
supports the continuation of GPST training in this format?
Please describe in one sentence the impact that this use of video conference education has had
on the health service where you are working
RRHEAL 2010 40
Please return this form to:-
Michaela Rodger
RRHEAL Research Assistant
Email: [email protected]
RRHEAL 2010 41
APPENDIX 4
Questionnaire
GP SPECIALIST TRAINING TELEPHONE QUESTIONNAIRE
My name is Michaela Rodger. I am part of the Remote and Rural Healthcare Educational
Alliance. We have been asked by the North of Scotland Deanery GPST program to
evaluate the new format of GP Specialist Training.
Our aim is to capture the experiences of trainees such as yourself to assist with this
evaluation and to inform the development of future training programs. All of your
answers will be recorded and analysed anonymously.
Do you have 10 minutes now to answer some questions or can I phone you back at a
time that suits you?
Alternatively I could email the questions to you but I do need the responses by Tuesday
8th July 2008 at 4pm.
RRHEAL 2010 42
APPENDIX 5
In the interest of timing and planning for the next round of GPST training (08-08)
RRHEAL is sharing this list of comment from the Survey tool; many other aspects of the
information collected have yet to be analysed and these comments should be read in the
light of that knowledge.
Response is anonymised.
CAUTION:
• This is a list of data collected from open question s only
(Full questionnaire attached as appendix)
• No attempt at analysis has been made yet
• This represents comment from 15 respondents (n. 39)
• Some participants are on holiday
• Some have committed to returning a paper response
• Final reminder will be sent on Tuesday 15 th July
Please explain your reaction to education via this type of media in one sentence
• As I was in the central group, it made little difference apart from allowing
interaction with colleagues in remote sites
• Did not complete
• It is quite difficult to learn in this setting because of isolation
RRHEAL 2010 43
• I think learning through video-link education has been extremely valuable and
very innovative
• Adequate when it works properly. Can never be as good as face-to-face
teaching.
• Very efficiently organised and delivered
• I was sceptical about the VC tutorials at first but as I became used to them I
enjoyed them more and realised that you needed to join-in the discussions to get
the most out of them - they would be pointless if you sat back and didn't
contribute
• Simply improve the technology of the VC connectivity - it's the technology that
lets and educational team down
• Ineffective
• Potentially a very good format of training however some barriers to flow of
presentation, can be v slow; presenters very patronising, sometimes very
condescending
• sub-optimal but necessary
• nothing entered
• Not ideal because of the technology problems. I don't feel quite part of it, but
isolation is too strong a word
Please describe in one sentence what you would chan ge in the program if you had
the choice
RRHEAL 2010 44
• More lecture-type sessions would be advantageous, with less group discussion
• I would invite more specialist like ENT, Ophthalmologists etc and I would make
each session an hour shorter
• More information on what to expect as a GP registrar and how best to guide our
own learning.
• Some helpful hints from GPs and GP registrars would be useful.
• To offer better sound quality and not perform scenarios like CPR through
videolinking as perhiopheral sites feel isolated. Also we were not able to see the
PowerPoint presentations delivered from Inverness hence that session was more
or less pointless without knowing what's happening
• Technological improvements, including audio and visual
• Make sure all the trainers involve the peripheral sites - most are very good but
occasionally the trainer/lecturer would only address the Inverness trainees and
you could feel like you are watching them get a lecture rather than being
involved, which was a waste of time.
• more input from specialists for direct teaching rather than group hypothesizing
• Training should be provided locally as far as possible using the local resources,
more like short attachments in different specialties, more hands-on, rest can be
done something like online modules
• I would do full five days training course with all trainees in one centre and involve
pre- and post-course evaluation with scoring
• Roll 4 intros into 1 intro - could email out intro rather than present intro at
session; whole presentation should be emailed out either before or after session.
RRHEAL 2010 45
• The whole curriculum is too daunting should ask a consultant in speciality what
he would want the GP to know and then build traiing around that.
• more scope to input our own learning needs - eg at perhaps 3 times / course
check what learning has already taken place and therefore what the needs are
• no answer
• I would rather be trained face-to-face. I would rather have more hard facts than
workshops. Workshops don't work by video link. A recent dermatology lecture
was very good.
Please describe in one sentence the best changes in performance or decision-
making that you have made since taking part in this program
• Generating an attitude towards self-directed learning
• Did not complete
• I am more aware of the need to contact and involve my Primary Care colleagues
for their input
• I think the best changes I have adapted is treating patients holistically which has
positively reflected on my decision-making process
• I still feel very nervous about becoming a GP registrar because I don’t really
know what to expect
• I certainly have improved my problem solving skills and holistic approach
• I now know that I have the support of colleagues outwith my area
• each teaching session left me feeling more aware of all the things I still needed to
learn about the topic.
RRHEAL 2010 46
• Well - current teaching program we hardly learn anything new and I strongly
believe that 3 hours is better spent in the wards seeing patients.
• No
• more on-going use of experiential log. It is a good learning tool and I use it very
regularly
• no answer
• I am more likely to make use of my colleagues in other locations now
Please describe in one sentence the impact that thi s use of video conference
education has had on the health service where you a re working
• To be honest, given that I was in the central group, VC education per se (i.e.
divorced from the content of presentations/group work/etc) has probably made no
difference to local health care delivery, except for an excellent session structured
and given by Dr. Douglas at Fort William
• It is alright. This questionnaire is difficult as there is not one clear yes or no
answer
• There is a strain on the service due to time away for training. Unfair on
colleagues
• On the whole, I think for me it has been a successful programme with variable
effects (mostly negative) on the health service system
• It's cheaper than flying us over to Inverness for face-to-face teaching sessions,
and therefore saving them cash.
RRHEAL 2010 47
• Using video conference education is a great idea especially for trainees in remote
places. However, if we are taught how to use it and make sure it is working
properly each site would help video conference better
• There is good cover in Fort William for attending the sessions, and if for any
reason you miss any, they are repeated. Overall it is a very efficient way to learn,
but it is no substitute for face-to-face learning, but good nonetheless
• VC has allowed us to get better quality teaching and has reduced the need for
either lots of travelling or more regional tutorials. As a result the teaching is more
standardised and more organised and this has meant that it is of a higher quality.
• fantastic way of involving trainees in remote sites with out cost of travel in terms
of time, money and environmental impact.
• No response
• It is more of hassle, because the rest of the team thinks we are skiving away from
work in the name of teaching and we get only two hours of whole day especially
after being busy on call or other work. We don't really get half day off for
teaching. No compensatory time off when we attend teaching when we are on
nights or late shifts.
• NHS has saved money, good cover has always been available
• Has ensured that all GPS trainees have had continuity in teaching & training
availability.
• No response
• Good in theory because it helps bring places and people together, but there is no
comparison with face-to-face programs. It is well organised and run
impressively, but does not work in practice
RRHEAL 2010 48
APPENDIX 6
Final Email reminder
Dear Doctor,
Re: GPST Modular Course
The North Deanery has been working with you through a modular approach to teaching.
They asked me as a manager with RRHEAL and as Hon Research Fellow with the
Centre for Rural Health to supervise the evaluation work and to ensure that research
governance and confidentiality protocols were in place.
Some are on holiday and we note that you have not yet replied to our survey.
Please do so now by clicking 'Reply' and modifying the text below, appending Y or N to
each statement.
1. Have you previously received training in this format?
2. Prior to the beginning of this training, were you taught how to use the video-
conferencing equipment to a standard that allowed you to use it confidently?
3. Have you enjoyed learning in this format?
4. Have you experienced technological difficulties with this format of training?
•Connection?
•Sound quality?
5. Have you felt isolated by this format of training?
5a. If yes, is this because you are receiving the training as the only learner at your site?
6. Do you think you would feel less isolated if you had colleagues in the room with you?
7. If this format of training was not available to you, would you have to travel for more
than one hour to access tutor-led education?
8. Do you feel that this format of learning physically allows you to learn successfully?
9. Do you feel that you are actually learning new skills?
10. Do you feel that you are being assessed on what you already know rather than
learning new skills?
RRHEAL 2010 49
11. Do you feel that this format allows you to learn more effectively than other formats of
training you have experienced?
12. Do you feel that direct lectures enable you to learn more effectively than workshops
and discussions?
13. Do you feel that workshops and discussions enable you to learn more effectively
than direct lectures?
14. Do you feel that there are any aspects of the training that are repetitious?
15. Do you feel that there is a natural flow of learning from one session to the next?
16. Do you feel confident enough to be able to immediately put the skills you have
learned at each session into practice when you are back in your workplace?
17. Did you feel that you were only able to put the skills you learned into practice once
you had received the whole training module?
18. Do you agree with this statement? ‘Learning in this setting prepares me for the reality
of being a GP’
19. Did you feel that there was a noticeable improvement in your performance after each
session?
20. Did you feel that there was only a noticeable improvement in your performance after
the whole training programme was completed?
21. Do you feel that the quality of your input into the workplace has improved from
having received your training in this format?
22. Do you feel that you are more able to retain what you have learned from having
received training in this format?
23. Do you feel that you are less able to retain what you have learned from having
received training in this format?
24. Do you feel that, on the whole, you have been disadvantaged from having received
training in this format?
25. Do you feel that your learning experience as a whole supports the continuation of
GPST training in this format?
If you have time now please add comment to the following four open questions. If you
are rushed then Thank You and please 'Send' now.
1.Please explain your reaction to education via this type of media in one sentence
2.Please describe in one sentence what you would change in the program if you had the
RRHEAL 2010 50
choice
3.Please describe in one sentence the best changes in performance or decision-making
that you have made since taking part in this program
4.Please describe in one sentence the impact that this use of video conference
education has had on the health service where you are working
Thank You, please 'Send' now. You reply will help us to refine the course for the next set
of learners.
with best wishes
Gillian Swan
Educational Projects Manager
RRHEAL and Strategic Engagement (N)
NHS Education for Scotland
Centre for Health Science
Old Perth Rd
Inverness, IV2 3JH
Direct dial 01463 255702
Mobile 07795 077741
Inverness department telephone 01463 255000
Fax 01463 255736