Technology as an Instrument to Improve Quality, Accountability, and Reflection in Academic Medicine

Leadership, Faculty Development, and Infrastructure

456 http://ap.psychiatryonline.org Academic Psychiatry, 30:6, November-December 2006

Technology as an Instrument to Improve Quality,Accountability, and Reflection in Academic Medicine

Michael S. Wilkes, M.D., Ph.D., Lydia Howell, M.D.

Received March 12, 2006; revised August 9, 2006; accepted August9, 2006. Drs. Wilkes and Howell are affiliated with the Universityof California, Davis, Sacramento, California. Address correspon-dence to Dr. Wilkes, One Shields Avenue, Davis, California 95616;[email protected] (e-mail).

Copyright � 2006 Academic Psychiatry

Objective: This article describes two complementary technologysystems used in academic medicine to 1) improve the quality oflearning and teaching, and 2) describe the barriers and obstaclesencountered in implementing these systems.

Method: The literature was integrated with in-depth, case-basedexperience with technology related to student progression, facultypromotion and school administration.

Results: Academic medicine concerns itself with data and out-comes. Psychiatrists need to attend to their learning and teachingpaths as much as to developing the knowledge and skills to man-age their patients.

Conclusions: Technology enables us to track, manage, and re-port these data with increasing ease, making transparency andaccuracy more achievable.

Academic Psychiatry 2006; 30:456–464

To a great extent academic medicine concerns itselfwith data and outcomes. Some of these data are re-

lated to the development of new knowledge, some to themanagement of our patients, some to the mission of edu-cation, some to the management of academic careers, andsome to quality assurance. Technology has enabled us totrack, manage, and report these data with increasing ease,making transparency and accuracy more achievable. Thepurposes of this article are to describe 1) two complemen-tary technology systems used in academic medicine to im-prove quality; 2) the barriers and obstacles encountered inimplementing these systems; and 3) how technology fa-cilitated positive change for the organization.

Medical Student Portfolios: My Oleport (MyOnline Electronic Portfolio)

No one would question an architect’s or artist’s need fora comprehensive portfolio to demonstrate the depth andbreadth of his or her ability. Traditionally, evaluation ofthe medical students has relied on written exam scores,objective structured clinical exams (OSCEs), oral exams,360-degree evaluations from patients, nurses, and peers,and narrative reports (usually very brief) from faculty.These student evaluations were typically completed at theend of a course, with a copy kept in folders in a lockedcabinet in the registrar’s office. Faculty advisers were re-quired to request copies or go to the office to examine astudent’s folder. Those few students in academic difficultyfound that a school official was indeed following their pro-gress as they were called into the Dean’s office. However,for those students who performed at or above average,final grades provided the only solace that they were movingforward.

Self-AssessmentMissing from the student’s folder is any attempt on the

part of the student at self-assessment or peer-assessment.This means that faculty are limited in their ability to de-termine whether a student can engage in lifelong adult

WILKES AND HOWELL

Academic Psychiatry, 30:6, November-December 2006 http://ap.psychiatryonline.org 457

learning, including the ability to identify weaknesses anddevelop a focused learning plan to address those weak-nesses—two skills that are crucial to the practicing physi-cian. To address these deficiencies, some medical educa-tors have come to rely upon Web-based electronicportfolios (ePortfolios) as a data management system, amethod to stimulate reflective learning, and an assessmenttool. These tools also facilitate communication andprovide more effective mentoring, particularly whengeographic distances make communication difficult.ePortfolios are intended to encourage systematic reflec-tion by mandating that students maintain narrative logsthat foster a critical examination of their performance, andattempt to manage their own professional developmentwith feedback from mentors.

The ePortfolio drives the interaction between a facultymentor and advisee at least three times a year. Two weeksprior to a required meeting with their mentor, students aresent an e-mail asking them to log in to the Oleport andaddress newly posted self-reflective questions. Some ques-tions relate to career development (Figures 1 and 2) andothers ask students to engage in reflective learning, as isthe case following an OSCE. Each structured entry re-quires students to address three domains with regard toparticular assigned graduation competencies or objectives:1) what are the weaknesses/strengths in achieving specifiedcompetencies? 2) what is the learning plan to achieve com-petency? and 3) how will the student identify outcome(s)measures to demonstrate competency? As a simple ex-ample, students might watch a video recording of theirexam of a 55-year-old man with back pain. After watchingthe encounter with a skill assessment guide, the studentsare asked to 1) identify the strengths and weaknesses oftheir exam; 2) develop a learning plan to improve theirexam of a patient with back pain (e.g., “I will find a neu-rologist to demonstrate an evidence-based approach to theback pain or review an online tutorial of the back exam”);and 3) identify outcome measures to prove competency(e.g., “I will have a neurologist observe my exam of theback and sign off that I am competent”). At the conclusionof each ePortfolio session, the entry is e-mailed to the fac-ulty mentor who is encouraged to respond to the studentin writing. In any case, mentors are required to meet withtheir students to review their reflection and approve learn-ing plans.

The ePortfolio as a Learning ToolReflection should be a crucial part of adult learning. It

involves an active recollection and assessment of past

events intended to encourage adults to learn from expe-rience. It is no surprise that adults benefit from continu-ously looking back on their actions/behavior and analyzingthem in the context of how things might have been im-proved. Ideally, reflection is the beginning of a cycle andis followed by practicing new approaches and then againreflecting back with the goal of continuous quality im-provement.

The ePortfolio is an online tool intended to track andmanage this reflection. If used appropriately, this reflectivetool helps students develop more in-depth understandingof how and why they learn, and it asks them to consideralternative approaches to learning that might be tried inthe future. It is important for the students to receive reg-ular feedback on the specifics of their reflective learning(calibration) that fosters a critical attitude toward theirperformance and helps them plan and manage their owndevelopment. Remedial plans are needed to address theneeds of those students who lack the critical ability to ap-praise their strengths/weaknesses. Measurable outcomeswill be necessary to document impact.

Limitations of the ePortfolioePortfolios are effective learning tools for most, but not

all, learners. To achieve maximum effectiveness, certainconditions must be met, including 1) a good explanationof the ePortfolio, including its goals/objectives and what isexpected of the student; 2) clear user-friendly technology;3) appropriate follow-through with a mentor; 4) appropri-ate faculty development to ensure strong coaching skills;and 5) a summative assessment (1). Even when each ofthese goals is met, faculty go to considerable lengths toconvince students of the benefits of reflective learning.While providing structure is crucial to helping a studentlearn self-assessment skills, too much structure can be sti-fling and reduce reflective learning. We have also foundthat reflection for the sake of reflection without directionis not fruitful. Requests for students to “log on” to theePortfolio system need to follow clearly identified clinicalexperiences where reflection would be of benefit, such ashome visits, hospice visits, OSCEs, the end of clerkships,or career planning exercises. Following written reflection,the students receive timely discussion and feedback on thequality of their ePortfolio entries and on their identifiedlearning plan.

Though students object to being graded on their port-folios, faculty feel that completion of the portfolio is animportant part of the students’ assessment in the domainof professionalism. Completion of the portfolio requires

USING TECHNOLOGY TO IMPROVE QUALITY IN ACADEMIC MEDICINE


FIGURE 1. Examples of Questions Posed to Students

Students receive questions specific to a course, their development as a doctor or on their learning process. Advisers may also be invited to addthoughts, which are used for discussion.

time from both the student and faculty. In recognition ofthis, we provide students with a pass-fail summative as-sessment on reflective learning, and we encourage facultyto provide narrative comments to encourage students (andfaculty) to take the exercise seriously. Merely developingan ePortfolio is no guarantee that reflective learning willoccur. All stakeholders (faculty, students, and administra-tion) must clearly understand its purpose and objectives.

Faculty Portfolios: MyInfoVaultFor many years, American medical schools have had a

growing interest in improving methods to document andmanage information related to faculty accomplishmentsfor promotion and tenure evaluations, including creationof faculty portfolios to document teaching contributions.Many of the publications that describe faculty portfoliosare limited to documentation of accomplishments in theeducational mission (2–7).

The University of California (UC) has well-establishedrequirements regarding preparation of faculty portfolios(often referred to as packets or dossiers) for faculty meritand promotion reviews. These requirements address ac-complishments in all missions (education, research, clini-cal service and university/public service) and include spec-ifications on what information can and must be includedand the format in which it is presented (8,9). UC facultymembers are required to undergo a merit review every 2to 3 years and review for promotion after a maximum of7 years as an assistant professor and 6 years as an associateprofessor.

Each major step requires preparation and submission ofpackets which must contain:

• A departmental supporting letter• A list of publications• A document describing contributions to jointly au-

thored work

WILKES AND HOWELL


FIGURE 2. An Ongoing, Reflective Process Facilitated by Asking the Student to Reexamine Past Answers

• A list of research grants and contracts• A list of teaching activities that includes courses

taught• A list of student advisees, including a description of

service on thesis and doctoral committees• A list of public service activities, such as committee

service and/or offices held for the department, university,professional societies, government, and other organiza-tions

• Teaching evaluations for three courses, and copiesof publications

Both faculty and staff complain that such frequent sub-mission of so much detailed information and creation ofthe required packets are time-consuming and often redun-dant, since the faculty are requested to submit the sameinformation for different purposes multiple times eachyear.

To address these issues, in 2002, the University of Cali-fornia, Davis, School of Medicine created a mission-based

reporting (MBR) program. Our MBR program has beenreported previously and was a Web-based program inwhich faculty members report their activities on an annualbasis (10–14). We believed faculty members would be morelikely to participate in MBR if the information collectedcould be saved in a data repository used for other purposesas well, such as preparation of their merit or promotionpackets or a grant submission, thus minimizing the re-quests from a department for information and the effortrequired to provide this. Though MBR was ultimately dis-continued after a pilot program in our school, we did de-velop an online faculty data repository, MyInfoVault(MIV). Implementation of MIV is currently in progress.It is capable of producing three documents: a packet in therequired format for faculty merit or promotion reviews, acurriculum vitae for personal use, and NIH BiosketchForm for grant submissions. All MIV pages are con-structed dynamically based on the user’s profile. The sys-tem is written in Java, an industry de facto, object-oriented



Internet programming language. The major design con-cept of the MIV is to use the central data repository witha variety of integrated applications that generate a seriesof professional reports and documents. Complete infor-mation on MIV and its development is available on theMIV homepage (14).

Data EntryTo enter data, the faculty member or a designated as-

sistant logs in to gain access to the MIV system. Figure 3illustrates the general data entry screen, and Figure 4 il-lustrates the screen for publication entry. After the dataare entered and saved, they can potentially be used in anyof the three MIV applications: CvOnline, PacketOnline,or NIH Form. Since not all data elements entered are per-tinent to each application, the user can choose which ele-ments to include, and these choices are saved. Formattingoptions also exist and the packets for merit/promotion canbe sent directly to the different levels of review throughMIV. Each level of review can add its components intoMIV.

ImplementationAs of January 2006, over 2,000 individual faculty ac-

counts were created in MIV from 75 different departmentsacross the UCD campus. These include all 25 departmentsin the School of Medicine, in addition to departments inthe School of Veterinary Medicine, College of Agriculturaland Environmental Sciences, College of Biological Sci-ences, College of Letters and Sciences, School of Educa-tion, and College of Engineering.

As is typical of implementation of most new systems,frustrations were noted regarding the learning phase,largely due to unanticipated “bugs.” Additionally, the ini-tial data entry to create an MIV record was considered tobe time-consuming and tedious. Uniformly across theschool, department academic assistants performed thedata entry. Very few faculty members have entered data ormaintain their own records. In general, academic assistantshave reported data entry and packet preparation to be nomore difficult than most other computer tasks they regu-larly perform, and this opinion was shared by both first-time users and experienced users. They found that final-izing and electronically sending packets for review to bevery easy. Departments found it particularly advantageousto have packets saved in a central repository that only re-quired updating at review time. Prior to MIV, frequentstaff changes caused repetitive loss of packets from pre-vious years, necessitating re-creation of entire packets eachtime a faculty member was undergoing a review.

Department analysts responsible for preparing packetsand the School Personnel Committee members were sur-veyed regarding their experience using MIV packets foracademic reviews. Questions related both to experiencepreparing packets and to experiences reviewing completedpackets. Respondents answered using a scale of 1 to 7 with1 defined as “easy” or “better than conventional methods,”4 as “the same as conventional methods,” and 7 as “worse.”The ability to generate the packet in the standard format,appearance, and readability was rated by all users to be atleast as good as, and often much better than, the packetprepared by conventional methods, and found that MIVpackets significantly conserve space and paper. Most mem-bers of the School Personnel Committee, which reviewspackets for academic advancement and the AssociateDean for Academic Affairs, find electronic review of pack-ets to be more convenient and easier to find and abstractpertinent information than the conventional paper packet.School Personnel Committee members also felt that hav-ing packets available electronically to all members facili-tated their group discussion since paper packets had pre-viously only been printed for the primary and secondaryreviewers. All users share the opinion that MIV is a veryfavorable concept with much future promise, though allfeel that additional functional enhancements are neces-sary, particularly regarding increasing speed of the system.Few faculty members have as yet taken advantage of thecurriculum vitae or NIH biosketch features in this initialstage of implementation.

Barriers to Change

Both the ePortfolio and the MyInfoVault have requiredchange and buy-in from stakeholders. Development ofboth tools required considerable time and effort over sev-eral years by a multidisciplinary team, including computerprogramming and systems experts, and personnel fromboth the Dean’s office and departments that oversee andare intimately familiar with the requirements. Users of alllevels must also be included in the development process toachieve the necessary perspective and buy-in. All users an-ticipate significant long-term benefits.

To understand change, in general, is to understand therelationships between organizational issues, political is-sues, and technical issues.

In our society, political issues often drive change, in in-ternal or external, and micro or macro sectors. The com-plex inner workings of academic medical schools and cen-ters are at the micro political level, with a classic

WILKES AND HOWELL


FIGURE 3. General Data Entry Screen From MyInfoVault

Note the large number of general categories. Data from any of these can be chosen to create the merit/promotion packet, CV or NIH data sketch.Additional categories can be created.

bureaucracy (formal jurisdictional areas, chain of account-ability hierarchy, rules, records, dedicated training), on thebasis of departments, seniority, research/publications, pro-

ductivity and other factors. At the macro political level,government (State Department of Health or state or fed-eral government) is required to be responsive to external



FIGURE 4. Data Entry Screen for Journal Publications

A similar format exists for other data categories such as research grants, and teaching activities.

pressures (the public, the press). In short, it is politics (mi-cro and macro) that determines in what direction the shipwill move. However, a vision for direction with no methodof execution would be no vision at all.

Change requires an appreciation of how an organization(or a profession) functions. Most academic medicalschools exist in divided territory, divided by specialty, se-niority, primary role (researcher versus clinician), location

(academic hospital versus ambulatory center), ability/will-ingness to teach, and place of training (15). These orga-nizational aspects dictate interrelationships between indi-viduals, groups, and external forces. Institutional history,local personalities, stakeholders/opinion leaders, local cul-ture, and internal and external pressures also have an im-pact on the organization. Since change may involve gains/losses to individuals, groups and the institution, the dy-

WILKES AND HOWELL


namics of ownership, leadership, and power arise. Socialsystems preserve members’ interests and resist change.Fault lines in the quasi-stable environment are opportu-nities for change (e.g., a lack of concordance in members’ideology, competition between subsystems).

It is important to understand an organization’s responseto pressure and how it affects change. At the two extremesof change are public conformity and private acceptance.As Kelman describes, there are three ways the memberscan be encouraged to change: compliance, identification,and internalization (16). Compliance is the application ofpositive or negative incentives intended to direct behavior.Identification occurs when one member seeks to establishor maintain some relationship to another and lasts as longas the reason that initiated the change (15). Internalizationis the acceptance of influence because the change is con-gruent with the member’s existing value system, usually toreduce cognitive dissonance or because it is required byinternal values (17).

Technical aspects of change can be altered throughtraining/education, software, rebudgeting, and other inter-ventions. Each organization must decide for itself whatpaddles to use and how to coordinate the rowing. Histori-cally, choices in medical education were left to the rowers(departments, courses, or individual instructors) with re-sultant Brownian motion. Critical thinkers felt that, sittingat the level of the cresting waves, rowers lacked the per-spective to chart a meaningful forward direction. Severalparties have key roles, with political entities pointing thedirection, while priorities and strategy were left to the or-ganization to develop and plan, and to individuals to im-plement the technical aspects.

Change Initiated by TechnologyIn our two examples (ePortfolio and MyInfoVault),

change occurred for several reasons. External pressure(Accreditation Council for Graduate Medical Educationcore competencies, Liaison Committee on Medical Edu-cation requirements for competency-based education, andUniversity-imposed faculty standards) forced changes inthe organization. For an institution to succeed withePortfolio, all participants must see the relevance and re-ceive training and prompts to use it. It must be integratedin other advising activities, for example, so adviser-adviseevisits coincide with diary work. For those with vision, thisis an opportunity to create a new dialogue, whereas forfaculty it is a change, and for coordinators it is a significantlogistical challenge. In the case of MyInfoVault it was notonly the faculty member who needed to believe that a com-

pilation of outcomes measures related to their perfor-mance was beneficial to their career, but the staff memberwho needed to see it would ultimately reduce their work-load.

Conclusions

Academic medicine is under siege and needs new pro-totypes for managing educational, clinical and research vi-sions. In research and clinical care, changes have beendriven by reimbursement patterns, a drive to reduce error,and increased productivity—all of which depend uponclear documentation. To a large extent, technology hasbeen an enabling force for much of this change, includingthe two complementary technology systems (ePortfolioand MyInfoVault) used in academic medicine to facilitateand measure outcomes, improve performance, and allowtransparency. The implementation of both systems in-volved overcoming significant barriers and obstacles, butin the end, technology has become a part of the solutionextending our armamentarium and facilitating informationtransfer.

References

1. Dreiessen EW, VanTartwijk J, Overeem K, et al: Conditionsfor successful reflective use of portfolios in undergraduatemedical education. Med Educ 2005; 39:1230–1235

2. Lindemann JC, Beecher AC, Morzinski JA, et al: Translatingfamily medicine’s educational expertise into academic success.Fam Med 1995; 27:306–209

3. Carroll RG: Professional development: a guide to the edu-cator’s portfolio. Am J Physiol 1996; 271:S10–13

4. Hafler JP, Lovejoy FH: Scholarly activities recorded in theportfolios of teacher-clinician faculty. Acad Med 2000;75:649–652

5. Collins J, Smith WL: Promotion based on teaching effortsrequires on-going documentation of scholarship teaching ac-tivities. Acad Radiol 2001; 8:771–776

6. Pinsky LE, Fryer-Edwards K: Diving for PERLS: working andperformance portfolios for evaluation and reflection on learn-ing. J Gen Intern Med 2004; 19:582–587

7. Simpson D, Hafler J, Brown D, et al: Documentation systemsfor educators seeking academic promotion in U.S. medicalschools. Acad Med 2004; 79:783–790

8. Checklist for Merit Increase. Available at http://manu-als.ucdavis.edu/apm/220_Check2.htm

9. Checklist for Promotion. Available at http://manu-als.ucdavis.edu/apm/220_Check4.htm

10. Howell LP, Hogarth M, Anders TF: Creating a mission-basedreporting system for an academic health center. Acad Med2002; 77:130–138

11. Howell LP, Hogarth M, Anders TF: Implementing a mission-based reporting system at an academic health center: a



method for mission enhancement. Acad Med 2003; 78:645–651

12. Howell LP, Green R, Anders TF: A mission-based reportingsystem applied to an academic pathology department. HumPathol 2003; 34:437–443

13. Anders TF, Hales R, Shahrokh N, et al: Mission-based re-porting in academic psychiatry. Acad Psychiatry 2004; 28:129–135

14. mediaworks.ucdavis.edu:8080/confluence/pp/viewpage.action?pageId�3082. Accessed January 25, 2006

15. Wilkes MS, Raven B: Understanding social influence in medi-cal education. Acad Med 2002; 77:481–488

16. Kelman HC: Processes of opinion change. Public OpinionQuarterly, Spring 1961

17. Wilkes MS, Srinivasan M, Flamholtz E: Effective organiza-tional control: implications for academic medicine. Acad Med2004; 80:1054–1063

Documents

Technology as an Instrument to Improve Quality, Accountability, and Reflection in Academic Medicine