EMPIRICAL REPORT

Psychiatry Residents’ Experiences with Systems-Based Practice:A Qualitative Survey

Joanna L. Fried & Melissa R. Arbuckle & Michael Weinberg & Anthony Carino &

Hunter L. McQuistion & Sosunmolu O. Shoyinka & Anna Skiandos & David A. Stern &

Jules M. Ranz

Received: 30 August 2013 /Accepted: 9 January 2014# Academic Psychiatry 2014

AbstractObjective The aim of this study is to analyze qualitative datacollected during field-testing of an instrument to assess psy-chiatric residents’ experiences with systems-based practice(SBP).Methods A total of 237 psychiatry residents from 6 levels oftraining in 12 different psychiatry residency training programsresponded to a 60-item instrument measuring their experi-ences with SBP during residency. Qualitative techniquesadapted from content analysis were used to review narrativeresponses to open-ended questions on the instrument.Results Certain themes emerged in the residents’ answersreflecting their opinions about the opportunities for (and bar-riers to) performing SBP in their work.Conclusions Psychiatric residents express an eagerness foropportunities to learn about and perform SBP but often feelconstrained by the lack of resources, teaching, and

supervision. Moreover, many residents desire a better under-standing of healthcare economics and how to factor costconsideration into clinical care.

Keywords Systems-based practice . Psychiatry residents

The Accreditation Council for Graduate Medical Education(ACGME) broadly defines systems-based practice (SBP) asthe demonstration of “an awareness of and responsiveness tothe larger context and system of health care and the ability toeffectively call on system resources to provide care that is ofoptimal value” [1]. Many studies have found that residentswant to be better informed in systemic, policy, and socioeco-nomic factors affecting their patients and healthcare delivery[2]. Despite the incorporation of SBP as one of six corecompetencies set by the ACGME, this desire for more infor-mation has not changed over the past three decades [2–4]. In a2009 study, residents found systems-based issues important totheir overall education but felt that these issues were notsufficiently or effectively addressed in their residency educa-tion [3]. Another 2010 survey of internal medicine residentsfound that residents perceived topics within SBP as highlyrelevant to their education and practice but that they did notfeel they were being adequately trained in these topics, par-ticularly in the areas of practice management and health careeconomics [4].

In 2008, the Mental Health Services (MHS) committee ofthe Group for Advancement of Psychiatry (GAP) began aproject to define, measure, and quantify the concept of SBPwith the ultimate goal of helping residency programs betterprepare their trainees to meet this ACGME core competencyrequirement. As such, the MHS committee field tested aresident survey with the goal of refining SBP into specificobservable behaviors [5]. Here, we discuss residents’ re-sponses to the items at the end of the instrument, whichprovided the opportunity to respond in free text to questions

J. L. FriedNew York University School of Medicine, New York, NY, USA

M. R. Arbuckle :M. Weinberg :A. Carino : J. M. RanzNew York State Psychiatric Institute/Columbia University,New York, NY, USA

H. L. McQuistionGouverneur Health System, New York UniversitySchool of Medicine, New York, NY, USA

S. O. ShoyinkaUniversity of Missouri, Columbia, MO, USA

A. SkiandosMt. Sinai Health System, New York, NY, USA

D. A. SternNew York Medical College, Valhalla, NY, USA

J. L. Fried (*)Janian Medical Care, New York, NY, USAe-mail: joannafriedMD@gmail.com

Acad PsychiatryDOI 10.1007/s40596-014-0038-6

about personal experiences with SBP. It is a unique collectionof first-person responses on this subject from residents at 12different programs across 6 levels of training (PGY1-6). Wehope that this study will help clarify further directions foreducation and research.

Methods

The MHS committee reviewed 35 specific measurable behav-iors developed from the common ACGME requirements forSBP by multidisciplinary expert consensus [6]. By methodsdescribed in detail elsewhere [5], these were modified to thecontext and specific requirements for training in psychiatry[7]. The resulting 60 items each represented a specific observ-able behavior which, when performed by psychiatric resi-dents, would indicate proficiency in SBP. The resulting SBPinstrument asked residents to rate the extent to which theywere encouraged to perform each of these 60 behaviors duringtheir most recent clinical rotation in the following contexts:general clinical settings, developing clinical treatment plans,team meetings, written communication with supervisors, andoral communication with supervisors. Sample items in theinstrument included the following: “In general clinical set-tings, I am encouraged to seek input about patient care from allmembers of the treatment team” and “In clinical treatmentplans, I am encouraged to consider the psychiatric rehabilita-tion goals of patients in relationship to available communityresources.” All 60 items are available upon request. The finalsection of the instrument posed open-ended questions, en-abling residents to respond in free text. The questions includedthe following 1: In what ways would you like to see SBPmorefully incorporated into your residency curriculum?2: With regard to your most recent rotation, please describelimitations in you opportunity to perform SBP behaviorsitemized in this instrument, 3: In what way, if any, did yourunderstanding of SBP change as a result of completing thisinstrument?

The IRB approved survey was distributed to a total of 457residents across 12 sites from April 2009–November 2010and included both general psychiatry residents and fellows inchild and adolescent psychiatry. Qualitative techniquesadapted from content analysis were used to review narrativeresponses to open-ended questions for recurring themes andideas. Two investigators independently coded all comments.Theme-specific groupings and detailed summaries were thenestablished by consensus.

Results

A total of 237 residents and fellows (52 %) completed the fullinstrument. Sixty-three percent were women and 37 % were

men. Respondents fell into the following age groups: 25–29(34 %), 30–34 (48 %), 35–39 (15 %), and over 40 (3 %).Sixty-four percent were Caucasian, 13 % Asian, 7 % South-Asian, 5 % each African-American and Hispanic, and 3 %each reported multiracial and other. The breakdown of respon-dents according to year of training was PGY1 (12 %), PGY2(31 %), PGY3 (21 %), PGY4 (21 %), PGY5 (12 %), andPGY6 (4 %). The survey respondents’ most recent rotationwas reported as follows: inpatient (40 %), outpatient–adult(30 %), outpatient–child (10 %), emergency room (8 %), andother (12 %). Residents’ responses to each item are describedbelow.

1. In what ways would you like to see SBP more fully incor-porated into your residency curriculum? A total of 195 resi-dents responded to the question. Twenty-six residents(13.3 %) indicated that they thought SBP was well taught intheir program and that no further training was necessary. Twoindividuals indicated that they had “no idea what SBP is.”However, the remaining 85.6 % provided specific suggestionsfor improvement in SPB training. Many felt that SBP shouldbe more formally incorporated into the curriculum throughdidactics, lectures, or journal clubs (26.2 %). In addition,residents also indicated that more should be done to integrateSBP concepts directly into their clinical training (23.6 %)including supervision and team meetings/rounds.

Residents identified several specific topics that theythought should be expanded in training, such as the cost orfinances of care (27.2 %), as well as available resources(13.3 %). One wrote, “Although we work with a low-incomepopulation, we don’t always pay attention to the practicalrealities of financing treatment because we assume it’s ‘paidfor’ by the hospital, and other long-term care options areorganized by social workers. More information about financ-ing options would be important to discuss.”

Other suggestions included an introduction to SBP earlierin training, with specific expectations for SBP delineated(7.2 %), more protected time with a priority on SBP set atthe administrative level (3.1 %), and a focus on addressingadverse outcomes (2.0 %). Several residents noted that SBPcould be addressed more in the outpatient setting (2.6 %).

Residents also indicated that availability and coordinationwith other team members such as social workers and casemanagers would help them learn more about SBP (3.1 %).Some residents noted that their supervisors also needed moreeducation regarding SBP (2.5 %). One resident gave thefollowing example: “When questioning whether an MRI andEEG were truly helpful for a patient’s diagnosis, I brought intothe conversation the patient’s family’s financial concerns. I wastold by my supervisor ‘if you do that on the board[s], it’s anautomatic failure. I can’t believe you would even bring thatup.’ Some education needs to be done on the faculty side aswell.”

Acad Psychiatry

2. With regard to your most recent rotation, please describelimitations in your opportunity to perform SBP behaviorsitemized in this instrument. A total of 191 residents respondedto this question. Approximately 13 % did not feel there wereany barriers to their ability to perform SBP in their most recentrotations. The remaining 87% of residents identified a numberof barriers to the learning and practice of SBP in their workwith patients.

Lack of supervision and lack of knowledge about SBPwere limiting factors for 22.3 %. One respondent commented,“I often feel out of the loop as an MD with regards to some ofthese structural concerns…attendings seem minimally interestedin these details,” and another referenced “massive gaps in knowl-edge among psychiatrists.” One respondent wrote “I’m not al-ways aware of the systems out there and therefore don’t know toask about them or consider them in the discharge process,” or asanother put it,”You cannot know what you don’t know.”

Time limitations were a prominent limitation for many(21.7 %), with responses citing overwhelming caseloads,quick evaluations and encounters, busy inpatient services,and high patient turnover. For example, one resident wrote,“An enormous amount of time and energy would be spent onpaperwork for social services for patient[s], referrals, lettersfor evaluations, disability, etc.Much of this work entails a fulltime social worker or case manager.”

A number of respondents (21.2 %) cited lack of a treatmentteam, poor communication among members of the team, or apoorly defined role for the psychiatrist on the team. Severalcomments indicated that social workers are the ones expectedto perform many of the SBP behaviors. One wrote, “Staffmembers equipped to provide additional services for patientsare not on-site in this clinic. These staff members are in otheroffices in other buildings and I am unsure how to coordinatesome services for patients.”

Limited attention to and discussion about issues of cost ofcare and healthcare economics was a barrier for 13.6 %. Aresident wrote, “Consideration of costs generally don’t appearto be taken into account as long as insurance is coveringcosts,” and another commented, “There is little opportunity tolearn or discuss patient finances and cost effectiveness oftreatment plan.” Some respondents (8.7 %) cited lack ofresources in the hospital and/or community, as well as lackof administrative support for SBP. Another 4.4 % cited patientfactors, including nonadherence to treatment, high “no-show”rates, inability to get collateral, and limited patient and familyresources.

Several participants (2.7 %) responded that their opportuni-ties to perform SBP were limited by the fact that their patientswere privately insured; “haven’t had Medicaid/Medicare pa-tients on current service,” “all patients had insurance,” “Pa-tients generally had private resources and did not need toutilize public systems,” suggesting that many residents maysee SBP as primarily existing in the realm of public psychiatry.

3. In what way, if any, did your understanding of SBP changeas a result of completing this instrument? Fewer than half ofthose surveyed (n=96) responded to the final question,and 26.0 % of them did not feel that completing theinstrument changed their understanding. However, manyrespondents indicated that the instrument did impact theirthinking about SBP in a number of ways. Many (26 %)noted an increase in their understanding of the concept ofSBP. One resident wrote, “This survey provided a betterperspective…[on] the various components of SBP,” andanother wrote, “This survey attempts to spell out manyof the notions that are considered implicit in psychiatricresidency.”

Others (20.8 %) wrote that completing the survey madethem more aware of deficits, both in their fundamentalunderstanding and practice of SBP behaviors (“It’s evenmore complex than I thought,” “I’m more aware of howlittle I know!” and “It made me reflect on areas I could bemore proactive at mastering”), as well as in training andresources for SBP, commenting that completing the instru-ment “made me realize how much we lack,” “made merealize the deficits in our training.” One commented, “Ican appreciate that we let this go a lot in our clinic.”Another theme that arose was an increased awareness ofthe importance of economic considerations.

Several responses suggested that completing the surveyheightened the respondent’s interest in more mindfully incor-porating SBP into clinical practice as follows: “Sounds likeSBP should be part of the general assessment that we make ofa patient”; the following comment suggests that completingthe survey not only clarified the concept of SBP but alsoframed it as an educational/training goal: “The instrumentarticulated very clearly elements of SBP which I previouslyunderstood (piece-meal) to be important but not in this logicalway for physician education.”

A number of responses indicated that residents either werenot entirely familiar with the concept of SBP or were notaware that many of their tasks of routine patient care actuallyconstitute SBP (“It’s more integrated into my daily experiencethan I would have thought,” “It allowed me to see how muchSBP [is] part of our everyday decision making and practice,”“I wasn’t sure what SBP meant, but now I understand that Igenerally have been incorporating the principles anyway,”“Realization that many things I ordinarily do with patientsare part of SBP”).

Discussion

The responses to the open-ended questions about SBP re-vealed to us that residents are eager for opportunities to learnabout and perform SBP but often feel constrained by lack ofresources, teaching, and supervision. Moreover, many

Acad Psychiatry

residents desire a better understanding of healthcare econom-ics and how to factor cost consideration into clinical care.Residents often cite the dynamic between social workers andpsychiatrists as an area where effective teaching of SBP couldcome into play. The roles and responsibilities of these twodisciplines often overlap, and the distinctions are often unclear,resulting in role confusion.

Our previous study found that some of the challenges inteaching SBP may stem from its complexity. In evaluating theresults from our 60-item survey, we found that the practice ofSBP can be divided into 4 separate but related functions asfollows: team member, patient care advocate, informationintegrator, and resource manager [5]. Focusing on the role ofthe psychiatrist within these four functions could help mitigatethese difficulties. It appears that merely completing the surveyhas the potential to increase interest, awareness, and under-standing of the concept of SBP.

Legitimizing SBP as a something that residents are alreadydoing has the potential to profoundly affect their approach totheir clinical work. If communicating with a patient’s familymember or collaborating with a social worker on an applica-tion for an ACT team is conceptualized as a valuable clinicalskill (as opposed to a barrier to doing “real” psychiatric work),residents may feel more valued or effective as both cliniciansand treatment team members. By identifying, valuing, andformalizing these skills, training programs have the opportu-nity to reframe tasks that some residents may see as auxiliary(or even burdensome) as skills that are central to medicalpractice and essential for good clinical care. The survey re-sponses highlighted the need for formal didactics, mentorship,and modeling.

Directions in SBP Training Residency training programs andresearchers across all medical specialties have explored waysto teach residents SBP, increase their performance of SBPbehaviors, and evaluate their competency in SBP. CallingSBP a “difficult to teach competency,” a 2010 workgroup ofemergency medicine residency directors determined that ex-pert modeling, informal small group discussion, and formalsmall group activities were considered to be the optimalmethods to teach SBP [8]. A Canadian survey of psychiatryresidents found the majority of those surveyed preferred learn-ing through workshops, small groups, and mentorship [9].Teaching approaches in other programs include using meta-phor to explore the concept of SBP. Basing its approach on thesaying “It takes a village to raise a child,” one internal med-icine residency program used the metaphor of a “village” toexplain SBP to its residents, emphasizing the importance ofthe medical community and the larger context and system ofhealth care in providing the best care to patients [10]. Web-based curricula, resident-led conferences, and incorporatingan SBP curriculum into resident continuity clinics are otherpossible avenues of education in this area [11–14].

A further avenue of inquiry remains tailoring an SBP cur-riculum specifically to the needs of psychiatry residents. ThePublic Psychiatry Education Program at Columbia UniversityMedical Center Department of Psychiatry developed a modelcurriculum incorporating SBP, community psychiatry, and re-covery into residency training and is currently collaboratingwith programs around the country to augment existing SBPtraining [15]. In addition, the American Academy of Child andAdolescent Psychiatry Work Group on Community-Based-Systems developed a training toolkit for systems-based prac-tice, specifically geared towards meeting the ACGME require-ments for child and adolescent training in SBP [16].

As indicated by some of the responses in this study,psychiatry faculty may not possess the knowledge, under-standing, or skills necessary to teach SBP. Continuingmedical education for faculty may be necessary becausemany clinical teaching faculty do not have formal trainingin SBP [17]. In addition, many faculty are not aware ofavailable resources and teaching materials for SBP [18].Future directions might include additional surveys in orderto assess faculty understanding of SBP. Given the growingimportance of systems-based practice in the context ofhealth care reform, and the fact that many of these conceptscut across disciplines, there may be value in interdisciplin-ary approaches to teaching SBP at the institutional level(i.e., hospital CEO, medical school dean’s office, publichealth dean’s office). Hospital administrators may have moreexperience with activities like assessing risk of re-hospitalizationduring the process of discharge planning or considering theoverall cost of “high-utilizer” patient populations, which aretwo of the specific instrument items relevant to the resourcemanager role of SBP.

Challenging Assumptions While carrying out this qualitativeanalysis, we found that we had to reframe some of our ownassumptions about SBP. Most of this study’s authors areaffiliated with institutions and practice psychiatry in the publicor academic field. Thus, our design of the instrument wasapproached from a public psychiatry perspective with theunderstanding that resident training occurs almost exclusivelyin hospital or clinic-based settings and the exposure to privatepractice is limited. Although some outpatient rotations mayapproximate a private practice model, they largely occur ininstitutional settings. However, as one respondent observed, “Irealize (SBP) is both in private practice AND communitybased.” This highlighted the fact that skills in SBP are essen-tial regardless of the practice setting. The vague nature of theACGME definition of SBP (pertaining to “the larger contextand system of health care”) leaves the parameters of whatconstitutes a system undefined. In the broadest sense, “thelarger context” of health care is nearly limitless and “thesystem” can be constructed as being any one of a number offormal and informal elements. All clinical care, whether it

Acad Psychiatry

consists of caring for uninsured patients in a communityclinic, evaluating agitated patients in the ER, making homevisits with an ACT team, or doing psychoanalysis in a privateoffice, requires the clinician to traverse and coordinate thesesystems in an infinite variety of ways. Figuring out where torefer a patient for ongoing treatment in the community, tryingto get prescriptions filled for an uninsured patient, talking tofamily members about a patient’s prognosis, fielding callsfrom a patient’s parole officer, submitting reimbursementforms to a patient’s private health insurance company—allof these require some combination of the skills evaluated bythis instrument.

Limitations The limitations of this study include the lowresponse rates for several of the open-ended questions andthe small sample size (only 12 residency programs participat-ed in the study; there are 182 ACGME-accredited psychiatricresidencies in the US). Moreover, the residency programsincluded in the study may not be representative of psychiatryresidencies in the US because potential sites were identifiedthrough contacts available to the authors. In addition, 8 of the12 programs are located in NewYork City (with the remaining4 sites also located in relatively large urban settings: Philadel-phia, Dallas, Pittsburgh, and Los Angeles).

One might expect that senior residents and child and ado-lescent psychiatry fellows would have more training insystems-based practice. On the basis of the qualitative natureof the data collected and the small sample size for each groupof residents (PGY1-6), we are unable to determine if there aresubstantial differences in responses to these open-ended ques-tions from level of training. This would be an important areafor future study, particularly in the context of the new mile-stones for psychiatry residents [19].

Conclusions

The rapidly changing face of healthcare makes the teaching ofSBP an especially relevant endeavor. There are broad cultural,economic, and political forces working to radically reshapehealthcare in this country. Healthcare systems are beingredesigned on national and local levels, and ideologies areshifting as patient-based practice, health homes, integratedcare, and other new methodologies of care emerge. Under-standing and participating in these seismic shifts requires allthe skills imparted by well-taught SBP.

As evidenced by our results, psychiatry residents seem tobe, on the whole, eager for opportunities to learn about andperform SBP but often feel constrained by lack of resources,teaching, and supervision. Moreover, many residents desire abetter understanding of healthcare economics and how tofactor cost consideration into clinical care. Simply completing

the survey can potentially increase interest, awareness, andunderstanding of the concept of SBP.

Implications for Educators

& Psychiatry residents seem to be, on the whole, eager for opportunitiesto learn about and perform systems-based practice (SBP)

& Psychiatry residents often feel constrained in their efforts to learn SBPby lack of resources, teaching, and supervision.

& Many psychiatry residents desire a better understanding of healthcareeconomics and how to factor cost consideration into clinical care.

Implications for Academic Leaders

& By identifying, valuing, and formalizing SBP skills, training programshave the opportunity to reframe tasks that some residents may see asauxiliary (or even burdensome) as skills that are central to medicalpractice and essential for good clinical care.

& Residents report a need for formal didactics, mentorship, andmodeling of systems-based practice.

& Continuing medical education for supervisors may be necessary asprograms begin to build additional training in SBP.

& Programs should be aware of various teaching resources andapproaches available for incorporating SBP into residency training.

& Involvement of hospital administrators in teaching various aspects ofSBP may help address the perceived need for training pertaining tohealthcare financing.

Disclosures On behalf of all authors, the corresponding author statesthat there is no conflict of interest.

References

1. Accreditation Council for Graduate Medical Education. CommonProgram Requirements. 2013. http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/ CPRs2013.pdf AccessedJuly 16, 2013.

2. King BF, Kindig RC, NoahWH, Tuteur PG: Internal Medicine chiefresidents suggest need to improve health care delivery and publicpolicy education. Internal Medicine Center to Advance Research andEducation 1990; 31:suppl 3–15.

3. Elwood D, Kirschner JS, Moroz A, Berliner J. Exploring systems-based practice in a sample of physical medicine and rehabilitationresidency programs. Physical Medicine & Rehabilitation. 2009;1:223–8.

4. Didwania A, McGaghie WC, Cohen E, Wayne DB. Internal medi-cine residency graduates’ perceptions of the systems-based practiceand practice-based learning and improvement competencies.Teaching and Learning in Medicine. 2010;22:33–6.

5. Ranz JM,Weinberg M, Arbuckle MR, Fried J, Carino A, McQuistonHL, et al. A four-factor model of systems-based practices in psychi-atry. Acad Psychiatry. 2012;36:473–8.

6. Graham MJ, Naqvi Z, Encandela J, Harding KJ, Chatterji M.Systems-based practice defined: taxonomy development and roleidentification for competency assessment of residents. J Grad MedEduc. 2009;1(1):49–60.

7. Accreditation Council for Graduate Medical Education. ACGMEProgram Requirements for Graduate Medical Education in Psychiatry.

Acad Psychiatry

Effective: July 1, 2007. http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/400_psychiatry_07012007_u04122008.pdf accessed July 16, 2013.

8. Wang EE, Dyne PL, Du H. Systems-based practice: summary ofthe 2010 council of emergency medicine residency directorsacademic assembly consensus workgroup-teaching and evaluatingthe difficult-to-teach competencies. Acad Emerg Med. 2011;18Suppl 2:S110–20.

9. Sockalingam S, Stergiopoulos V, Maggi J. Physicians as man-agers: psychiatry residents’ perceived gaps in knowledge andskills in administrative psychiatry. Academic Psychiatry.2007;31:304–7.

10. Ziegelstein RC, Fiebach NH. “The mirror” and “the village”: a newmethod for teaching practice-based learning and improvement andsystems-based practice. Acad Med. 2004;79:83–8.

11. Hauge LS, Frischknecht AC, Gauger PG, Hirshfield LE, HarkinsD, Butz DA, et al. Web-based curriculum improves residents’knowledge of health care business. J Am Coll Surg. 2010;211:777–83.

12. Sultana CJ, Baxter JK. A resident conference for systems-based prac-tice and practice-based learning. Obstet Gynecol. 2011;117:377–82.

13. Eskildsen MA. Review of web-based module to train and assesscompetency in systems-based practice. J Am Geriatr Soc. 2010;58:2412–3.

14. Perez Jr JA, Faust C, Kenyon A. The virtual practice: using theresidents’ continuity clinic to teach practice management and

systems-based practice. Journal of Graduate Medical Education.2009;1:104–8.

15. LeMelle S, Arbuckle MR, Ranz JM. Integrating systems-based prac-tice, community psychiatry, and recovery into residency training.Academic Psychiatry. 2013;37:35–7.

16. McGinty K; Chenven M; Winters N et al: Training toolkit forsystems-based practice in child and adolescent psychiatry.Washington DC. American Academy of child and AdolescentPsychiatry. Work Group on community-based systems of care. Oct.2009. Available at http://www.aacap.org/AACAP/Resources_for_Primary_Care/Training_Toolkit_for_Systems_Based_Practice.aspxAccessed January 2, 2014.

17. Philibert I. Accreditation Council for Graduate medical Educationand Institute for Healthcare Improvement 90-Day Project. Involvingresidents in quality improvement: contrasting “top-down” and “bot-tom-up” approaches. August 2008. https://dconnect.acgme.org/acWebsite/ci/90DayProjectReportDFA_PA_09_15_08.pdf.Accessed January 2, 2014.

18. Lee TG, Cox JR, Walker SC. Child welfare training in child psychi-atry residency: a program director survey. Acad Psychiatry.2013;37(5):308–12.

19. The Psychiatry Milestone Project. A joint initiative of TheAccreditation Council for Graduate Medical Education and TheAmerican Board of Psychiatry and Neurology. November 2013www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/PsychiatryMilestones.pdf Accessed January 2, 2014.

Acad Psychiatry