THEnet’s Evaluation Framework for Socially Accountable Health … · 2012. 2. 12. · and Zorayda Leopando Editing of the Monograph: Björg Pálsdóttir, Simone Ross, Robyn Preston,

THEnet’s Evaluation Framework for SociallyAccountable Health Professional Education

Version 1.0

Monograph I

Suggested Citation: The Training for Health Equity Network. THEnet’s Social AccountabilityEvaluation Framework Version 1. Monograph I (1 ed.). The Training for Health Equity Network, 2011.

© The Training for Health Equity Network 2011

All rights reserved. Publication can be obtained at www.thenetcommunity.org All reasonable precautions have been taken by THEnet to verify the information in this publica-tion However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies withthe reader. In no event shall THEnet be liable for damages arising from its use.

The Training for Health Equity Network54, Rue FostyB-1470 Belgium

Design and Layout: Monika Tillman

Table of Content

Contributors .....................................................................................................................................................4

Acknowledgments..........................................................................................................................................4

Preamble............................................................................................................................................................5

Redefining the Role of Health Professions Education ...............................................................................5

Finding New Ways to Measure Success of Academia ...............................................................................6

Introduction to Training for Health Equity Network: .....................................................................8

THEnet’s Aims and Guiding Principles ..................................................................................................9

THEnet’s Values............................................................................................................................................10

THEnet’s Social Accountability Operational Model........................................................................11

Introduction to the Framework .............................................................................................................12

The Framework............................................................................................................................................13

THEnet’s Evaluation Framework for Socially Accountable Health Professional Education................................................................................................................14

How does our school work? .......................................................................................................................14

What do we do?...........................................................................................................................................16

What difference do we make? ...................................................................................................................21

Guide to Evaluation Framework Terminology .................................................................................25

References ......................................................................................................................................................27

Contributors

Many individuals were involved in the development and review of the framework. They include:

Evaluation Framework Group that oversaw the project: Sarah Larkins, Iris Lindemann, Marie Matte, Jose Alvin P Mojica, André-Jacques Neusy,Björg Pálsdóttir, Robyn Preston, Rex Samson, Simone Ross, David Buso, Filideto Tand-inco, and Afdal Kunting

Other contributors to the development of the Framework: Charles Boelen, Kate Brennan, Juan Carrizo, Pasqualito Conception, Fortunato L. Cristobal, Aaron Goldstein, Jenenne Greenhill, Dan Hunt, Jehu Iputo, Joel Lanphear,David Marsh, Khaya Mfenyana, Ileana del Rosario Morales Suárez, Richard Murray,David Prideaux, Jusie Lydia Siega-Sur, Roger Strasser, Paul Worley, Sarah Strasser, and Zorayda Leopando

Editing of the Monograph: Björg Pálsdóttir, Simone Ross, Robyn Preston, Sarah Larkins, and Andre-Jacques Neusy

Acknowledgments

The Evaluation Framework and this publication was funded by the generous support ofthe Atlantic Charitable Trust. THEnet would also like to thank the Build Project and Arcadia Foundation for their generous support.

We are indebted to Charles Boelen and Robert Woollard who developed the Conceptual-isation–Production–Usability Model1 that THEnet used as a basis for the development ofthe Framework.

We are very grateful to Louise O’Meara from the Interaction Institute for Social Change,who skillfully facilitated many challenging meetings and discussions on the Framework.

We are also appreciate the work of Georgina Gomez Tabio who translated pre-meetingonline discussions and face-to-face dialogue during our meeting in Cuba.

Finally, this work could not have been done without the participation of faculty, stu-dents, staff and others who participated in workshops, pilot study focus groups and in-terviews at each school.

page 4

Redefining the Role of Health Professions Education

More than a billion people globally never see ahealth worker in their lives2. Such health in-equities are not only unacceptable, but costly, limiting individual opportunity and slowing societalgrowth. Great disparities in health outcomes existeven in high income countries illustrating the im-portance of the social determinants of health, i.e.the environment in which people develop, live andwork and the systems that serve them3.

Inequitable and inefficient health systems4 areweakened by a shortage and mal-distribution ofthe health workforce, a major impediment for longterm improvements in health outcomes in manyregions5. In many poor countries health profes-sional schools end up training students for theglobal market, rather than to meet their own

needs, at considerable cost to their own oftenweak health systems6. While reducing inequities is complex and requires the involvement of a mul-titude of stakeholders, health professional schoolscan play a central role in improving health equity.They produce key components of the health sys-tem: the health workforce and knowledge. Yet,they are often not active participants in develop-ing more equitable health systems. Addressingthese challenges call for a radical shift in our understanding of the role of health professionseducation institutions.

Today’s health professional schools must form ef-fective partnerships with the health sector, policymakers and communities to identify and helpsolve priority health needs. They can and shouldbe vital contributors to health system develop-ment and agents of innovation and reform.Schools help develop the values, norms, behav-iours and worldviews, held by key groups in thehealth system. They can shape and influence theirgraduates with potentially wide-ranging effectsthroughout the health system.

Unfortunately, current models of health workforceeducation are not producing the people, researchand services needed to attain basic universalhealth coverage5 and the calls for reform aregrowing louder5 7 8 9 10 . The Independent GlobalCommission on Education of Health Professionalsfor the 21st Century (2010) calls for transforminginstitutional and educational approaches to bettermeet changing health system needs5. Further-more, in late 2010, The Global Consensus for Social Accountability of Medical Schools (GCSA)urged schools to improve their response to cur-rent and future health-related needs and chal-lenges in society and reorient their activitiesaccordingly10.

However, since 2008 several health profession education institutions in underserved and rural regions of Africa, Asia, Europe the Americas andAustralia have been pioneering innovative ap-proaches to tackle these challenges. Supported

by The Atlantic Charitable Trust, eight suchschools, striving towards greater social accounta-bility, founded the Training for Health Equity Net-work (THEnet) in December 2008. THEnet schools define social accountability as an institutional responsibility to orient teaching, research andservice activities to addressing priority healthneeds with a particular focus on the medically underserved.

THEnet founding schools are: Ateneo de Zam-boanga University School of Medicine in the Philippines; Comprehensive Community PhysicianTraining Program in Venezuela; Faculty of HealthSciences, Walter Sisulu University in South Africa;Flinders University School of Medicine in Australia; James Cook University Faculty of Medi-cine, Health and Molecular Science in Australia;Latin American Medical School in Cuba; NorthernOntario School of Medicine in Canada and Univer-sity of the Philippines Manila - School of HealthSciences in Leyte.

THEnet’s Evaluation Framework for Socially Accountable Health Professional Education page 5

Preamble

Frameworks on social accountability have existedsince the mid-1990s. However, THEnet schoolsdetermined that there were no robust, practicaltools for assessing the progress of their institu-tions towards social accountability that effec-tively evaluated and compared strategies,processes and programs within and across contexts.

The focus of accreditation bodies for health professional schools (particularly for medicalschools) is firmly on the process of education,with little attention paid to governance, commu-nity partnerships, distribution of resources orbroader outcomes. Thus, the initial priority ofTHEnet was to develop a comprehensive Evalua-tion Framework to identify key factors that affecta school’s ability to positively influence healthoutcomes and health systems performance aswell as to develop ways to measure them acrossinstitutions and contexts.

Finding New Ways to Measure Success of Academia

What should health professional institutions beheld accountable for? THEnet schools define theirsuccess, not by how many graduates they pro-duce or how many articles they publish, butwhether their graduates have the right compe-tencies to meet the needs of their reference pop-ulations and whether a high portion of them stayand work in regions where they are mostneeded. They also assess whether their researchand services positively affect health policies andpractice and thereby improve health in disadvan-taged communities.

Using WHO’s social accountability framework11

and Boelen and Woollard’s Conceptualization,Production and Utilization Model1 as a founda-tion, THEnet schools worked together and withtheir stakeholders, to develop and test an Evalu-ation Framework for Socially Accountable HealthProfessional Education.

Measuring the impact of educational and institu-tional strategies on the health system and itsbeneficiaries is challenging, not least because“impacts” are usually the result of a myriad offactors, relationships, and events that in turntrigger ripple effects in the health system and its sub-systems. Consequently, it is difficult toattribute a specific impact to one intervention,program, or institution. However, additional em-pirical research is needed to identify the key fac-tors that institutions can affect to improve health

www.thenetcommunity.org page 6

Preamble; Redefining the Role of Health ProfessionsEducation continued


and health system impact. In addition, our mem-bers, while insisting on rigor in the quest for evi-dence and measurement, kept in mind thatinnovation comes from thinking out of the box. As a result they agreed that the framework had tobe flexible enough to provide adequate space forchallenging orthodoxies, allowing for contextualdifferences and experimenting with new ideas.

This document provides the first version ofTHEnet’s Evaluation Framework for Socially Ac-countable Health Professional Education. It allowsschools to get a sense of where they are on theroad towards greater social accountability and intheir ability to increase impact on health andhealth services. This first version of the Frame-work centers on medical education. However, theFramework focuses on core common elementsand was tested across health disciplines at two ofTHEnet schools, and we believe it can serve as afoundation to evaluate other health professionaleducation as well. THEnet’s Evaluation Frameworkalso serves as a starting point for its collaborative

research activities. Hence, it also helps identifyresearch and data gaps to strengthen the evi-dence base.

THEnet is excited to share its Framework andwould like to partner with those schools interestedin using it. Institutions willing to partner withTHEnet will have the opportunity to contribute torefining this organic tool, that will continuouslyevolve in response to feedback and as root causesand causal links are examined in greater detail.Partners will in turn receive access to THEnet’sFramework Implementation Manual. They canalso receive mentoring, guidance, and supportdocuments such as focus group and survey in-struments. More importantly collaborating withTHEnet provides all of us with a more solid evi-dence base by harmonizing data collection andanalysis. Together we can create better tools tosupport schools seeking to increase their socialaccountability and thereby make health equity a

re

Preamble; Finding New Ways to Measure Success of Academia continued

realistic and actionable goal.


Several health professional schools, working inmarginalized urban, rural, and remote regions inhigh and low income countries, agreed to joinforces and constitute THEnet in late 2008. Thegoal was to build evidence to support effectiveand credible change towards greater impact andaccountability of academic institutions. It is acommunity of practice of schools of medicine andhealth sciences with a core commitment toachieving equity in health services and improvinghealth outcomes.

THEnet institutions operate in highly diverse con-texts. Training settings vary from poor communi-ties in the United States, remote indigenouscommunities in Canada and Australia and rural regions of Africa to urban slums and marginalizedcommunities in the Philippines including conflict-ridden Mindanao. Embedded in underserved com-munities and committed to principles of equity,they see communities as vital actors in the healthsystem. The schools thus work with health systemstakeholders and community members to developinnovative ways to address the spectrum of issuesaffecting health. Not only are students and facultyoften providing health services where there werenone, they are mobilizing communities to take re-sponsibility for their own health. This ranges fromreducing cardiovascular risk factors in neglectedpopulations, to bringing municipal authorities andcivil society groups together with students andfaculty in projects as diverse as increasing useof latrines, and growing vegetable gardens.

Founding member schools are:

1. Ateneo de Zamboanga University School ofMedicine, Philippines (ADZU)

2. Comprehensive Community Physician Training Program, Venezuela (CCPTP)

3. Flinders University School of Medicine, Australia (FLINDERS)

4. James Cook University Faculty of Medicine, Health and Molecular Sciences, Australia (JCU)

5. Latin American Medical School, Cuba6. Northern Ontario School of Medicine,

Canada (NOSM)7. School of Health Sciences Leyte, University

of the Philippines Manila, Philippines (SHS) 8. Walter Sisulu University Faculty of Health

Sciences, South Africa (WSU)

Four new schools that joined THEnet in late 2011will be testing the Framework:

1. Gezira University Faculty of Medicine, Sudan

2. Ghent University Faculty of Medicine and Health Sciences, Belgium

3. Patan Academy of Health Sciences in Nepa4. The University of New Mexico Health

Sciences Center, United States

Introduction to the Training for Health Equity Network


THEnet’s Aims and Guiding Principles

THEnet’s Aims:

1. To generate new evidence and advocate for effective institutional strategies that help health professions schools meet needs of underserved communities

2. To increase the number of health profes-sions schools using social accountability principles to meet needs of underserved populations

3. To support health professional schools engaged in reform through evidence-based policy guidance, practical tools and capacity development

4. To increase the focus on health equity and social accountability in health profes-sions education and health system reform

THEnet’s Guiding Principles:

Despite highly different operational contexts,THEnet schools have embraced several commonprinciples/strategies, some of which are the focusof its collaborative research.

1. Health and social needs of targeted com-munities guide education, research and service programs

2. Social accountability is demonstrated in action through a "whole school” approach

3. Students recruited from the communities with the greatest health care needs

4. Programs are located within or in close proximity to the communities they serve

5. Health professions education is embedded in the health system and takes place in the community and clinics instead of predomi-nantly in university and hospital settings

6. Curriculum integrates basic and clinical sciences with population health and social sciences; and early clinical contact in-creases the relevance and value of theo-retical learning

7. Pedagogic methods are student-centered, problem and service-based and supportedby information technology

8. Community-based practitioners are re-cruited and trained as teachers and mentors

9. Health system actors are partners to produce locally relevant competencies

10. Faculty and programs emphasize and model commitment to public service

The following values underpin our work, and havebeen defined collectively with THEnet partnerschools.

Equity: The state in which opportunities for health gainsare available to everyone. Health is a social prod-uct and a human right, and health equity (that is,the absence of systemic inequality across popula-tion groups) and social determinants of healthshould be considered in all aspects of education,research and service activities. This incorporatesthe principles of social justice, or redressing in-equitable distribution of resources, and access to education.

Quality: The degree to which health services for individualsand populations increase the likelihood of desiredhealth outcomes and are consistent with currentprofessional knowledge. These health servicesmust be delivered in a way which optimally satis-fies both professional standards and communityexpectations.

Relevance: The degree to which the most important and lo-cally relevant problems are tackled first. This in-corporates the values of responsiveness tocommunity needs. In addition, it incorporates theprinciple of cultural sensitivity and competency.Cultural competency is not seen as a specific knowledge, attitudes and practices acquired,

but a process of removing barriers to effective andopen communication in the service of the patient.

Efficiency: This involves producing the greatest impact onhealth with available resources targeted to ad-dress priority health needs and incorporates theprinciple of cost-effectiveness.

Partnerships: Partnership with all key stakeholders in develop-ing, implementing and evaluating efforts is at thecore of THEnet schools’ activities. The partnershipvalue includes partnerships between and amongall stakeholders including faculty and students;communities being served; all health and educa-tion system actors; the school and the larger academic and social accountability community. It incorporates the values of mutual transforma-tion, equipping students and faculty to be agentsof change and open to be changed through theirpartnerships. It also incorporates the value ofinter-professionalism, or a belief that all health professionals must respect each other’sknowledge and culture and understand the rolethat each team member plays on the health careteam. Inter-professionalism includes the key fea-tures of partnership, participation, collaboration,co-ordination and shared decision making. Whereinter-professionalism is practiced, health profes-sionals from all disciplines work together as teamswith and in service of the patient and the wholecommunity.


THEnet’s Values


The operational model guiding the development of THEnet’s schools programs and its Framework, assumes that to meet the needs of the populations it serves, a health professional school must be designed based on a thorough needs assessment and understanding of the environment it operates in. This includes the social systems it seeks to impact and how various systemic and other factors may influence its operations and outcomes. The assessment is conducted in collaboration with key stake-holders including health system actors and underserved communities.

Guided by the values it espouses the school then needs to set outcome objectives and select strategiesbased on the information and evidence that is available. Desired competencies of health professionalsand research priorities are defined based on the need assessment. It then designs and delivers pro-grams to meet their defined outcomes. The school then evaluates its processes, strategies, outcomesand the impact the school is having on the systems, communities and individuals it serves to ensure itsactivities are meeting needs. This is an ongoing process and the school must continue to examine theirunderlying assumptions, be proactive and responsive to changing needs and demands.

Measured through the lens of:

Quality•

Equity•

Relevance•

Efficiency•

Partnership•

THEnet’s Social Accountability Operational Model

IDENTIFYresearch,

competencies and attitudes to meet needs

DELIVEReducation,

research and services

EVALUATEneeds,

process,outcomes and

impact

ADJUSTgovernance,education,

research andservices

ASSESShealth system

communityand student

needs

To strengthen the evidence base on socially ac-countable health workforce education, as our firstpiece of work, we chose to develop a comprehen-sive Evaluation Framework (the Framework). TheFramework seeks to identify key factors that af-fect a school’s ability to positively influence healthoutcomes and health systems performance and todevelop ways to measure them across institutionsand contexts. This first generation of the Frame-work was created for medical schools. However,given the basic nature of most of the questionsand after having tested it across health disciplinesat two schools, we believe it can serve as a foun-dation that is adaptive to evaluate other healthprofessional education.

The Framework was created as an aspirationaltool. It is not designed as a summative pass/failexercise, but rather a formative exercise to help

schools take a critical look at their performanceand progress towards greater social accountability,and assist schools in establishing priority areas forresearch and improvement.

There is increased discussion globally on develop-ing and incorporating social accountability stan-dards in medical school accreditation and qualityimprovement assessments (Global Consensus forSocial Accountability of Medical Schools, 201110).While THEnet is highly involved in this effort andsome standards might eventually be developedfrom this framework, it is important to note thatthe Framework is not designed to be used as a‘tick box’ for social accountability in accreditation.It is an aspirational evaluation tool for critical andhonest institutional quality improvement.


Introduction to the Framework

To date, five of the founding THEnet schools havepilot tested the Framework (ADZU, FLINDERS,JCU, NOSM, and SHS). Each of these schoolsfound the Framework effective, not only in evalu-ating the school's progress towards greater socialaccountability, but also in raising awareness of theissue among students, staff and stakeholders. It allowed each school to review their schools’ social accountability mission, vision and goals and identify strengths, weaknesses and gaps.

The Framework is also a comprehensive tool toidentify key factors that affect a school’s ability topositively influence health outcomes and healthsystems performance and to develop ways tomeasure them across institutions and contexts. Itis an organic tool that will continue to evolve asroot causes and causal links are established ingreater detail.

The Framework is designed to be used in its en-tirety, as schools may vary widely in their ap-proach across different contexts. In addition, it isintended to be used on an institution-wide basisnot at department or program levels. Moreover,for the Framework to be effective and so that itsfindings be acted on, institutional buy-in at high-est levels is essential.

The Framework has been designed to be useful atthree levels (individual schools, network andglobal level):

Individual School Level: At individual school level the goal is to learn fromothers, validate and continually assess and im-prove schools’ performance in socially accountablemedical education, research and service accordingto agreed standards.

Network/Partnership Level: At a network/partnership level the goal is to en-gage in self-reflection and renewal, share collec-tive experience and use the resulting critical massof data for continuous improvement.

Wider Level: At a wider level the goal is to facilitate, and advo-cate for, sustainable improvement in health serv-ices and outcomes through demonstrating thevalue/impact of socially accountable health pro-fessional schools. It is also to challenge otherbodies and orthodoxies at local, national and international levels.

The Framework

Using Boelen and Woollard’s Social AccountabilityConceptualization–Production–Usability model(CPU model)1 as the foundation, the Framework seeks to assist health professional schools in assessing their social accountability by directingschools to a series of question’s under the threesections as follows:

Section 1: How does our school work?

This section addresses important aspects of theorganization and planning of the school, fre-quently neglected in existing evaluation and ac-creditation frameworks, including an assessmentof values, governance and decision-makingprocesses and partnerships with the health sector,community groups and policy makers. It also in-cludes documentation and understanding of thereference populations that the school serves, withparticular note of underserved groups within this.

Section 2: What do we do?

This section focuses on the schools’ programs in-cluding composition of students and teachers,curriculum, learning methodologies, research,service and resource allocation. This correspondsmore closely to usual accreditation processes.

Section 3: What difference do we make?

This section includes an assessment of graduateoutcomes (location, discipline, and practice), en-gagement and effect on health services andhealth system outcomes and influence with otherschools. Again, this section is largely overlookedin most accreditation measures, and is frequently considered outside the scope of health profes-sional education institutions.


Introduction to the Framework continued

THEnet’s Evaluation Framework for Socially Accountable Health Professional Education

1.1. What do we believe in? (Values*)

Social accountability values areexplicit and known and under-stood by students and staff.

Staff/faculty and students andkey stakeholders/communitypartners are able to give exam-ples of how values are opera-tionalised.

Measurement Tools:THEnet Interviews/surveys/focus groups with students and staff/faculty, health care providers and community.

Data Sources:School/University Mission or Vision statement and strategic plan. Annual Report, course materials, student manuals.

Section 1: How does our school work?

Key Component Aspirations Indicators Suggested Sources of Evidence

1.2. Who do we serve,what are theirneeds and whatare the needs ofour health system?(Reference popula-tion and healthsystem)

Clear rationale for identificationof populations

Underserved populations arewell defined and emphasised

Priority health and workforceneeds identified and regularlyreviewed

Awareness and understanding of reference population by students, staff/faculty and key stakeholders

School, faculty/staff and stu-dents are involved in influencingand developing key policies andpractices to improve healthservices and policies, with a par-ticular emphasis on PHC

Measurement Tools:Needs assessment that identifies key population, underserved populations, priority health and workforce needs

Interviews/surveys/focus groups with students and staff/faculty, health care providers and community

Data Sources:Documents and/or data describing reference populations as well ashealth services, health system, health workforce priority needs andplans to address them

National/regional health workforce plan

Evidence of faculty/staff and students involvement in advocacyAccreditation documents

Documents demonstrating community partnerships

We recognize and define thepopulations we serve with par-ticular reference to underservedpopulations

We define the reference popula-tions’ needs in collaborationwith these communities andother key health system stake-holders and hold ourselves ac-countable for addressing theseneeds

We collaboratively plan to ad-dress the priority health andworkforce needs of our refer-ence populations and healthsystems

We are active contributors tothe health system of which weare a part and play a role in ad-vocacy and reform. Our particu-lar emphasis is on increasingthe provision of and access tocomprehensive Primary HealthCare (PHC), and addressing thesocial determinants of health

We uphold and demonstrateshared values of social account-ability as defined by THEnet.

page 14*The terms in parentheses are from the CPU Model1

Section 1: How does our school work? continued


1.3. How do we workwith others? (Partnerships)

Measurement Tools:Interviews/surveys/focus groups with students and staff/faculty, health care providers and community

Data Sources:Documents describing partnerships; suggested examples could include:

- Documented collaborative programs of teaching, service and research

- Audit of community participation and partnership outcomes

- Meeting minutes and content

- Memoranda of Understandings with health departments and communities

- Financial/infrastructure audit of community co-contribution

page 15

We operate in partnership withall relevant stakeholders, with aprimary focus on the priorityhealth needs and social needsof our target populations. Ourpartnerships reflect our genuinecommitment to meaningful col-laboration with communities,health services and health careproviders.

Level of involvement of keystakeholders in planning, devel-oping and supporting teaching,service and research programsand student and faculty/staff recruitment

Community is engaged andmakes and receives meaningfulin-kind and financial contribu-tions, reciprocally betweenschools and communities

1.4. How do we makedecisions? (Governance)

Governance structure andprocess ensures meaningfulparticipation of key stakehold-ers in corporate, fiscal and academic governance.

Constitution or mission state-ment reflective of stakeholderinvolvement in decision making

Examples of changed policies/processes in response to stake-holder feedback

Our strategic decision-makinginvolves meaningful participa-tion from all stakeholders.

Measurement Tools:Interviews/surveys/focus groups with students and staff/faculty, health care providers and community

Measurement tools to assess meaningful participation12

Data Sources:Examples of changed policies/programs/services in response to community feedback

School organisational chart

Memoranda of Understanding

School mission statement/charter

Minutes/meeting notes from key committees

Evidence of how decisions are made and who is making them

Stakeholder feedback and consultation

Reference groups/community committees (including student groups)

www.thenetcommunity.org

2.1. How do we man-age our resources? (Field operations)

Resources for community en-gagement and program opera-tionalization are distributedaccording to priority needs (SeeKey Component 1.3)

Sufficient resources are avail-able to operationalise theSchool’s strategic plan

Community makes and receivesmeaningful in-kind and financialcontributions

Stakeholder satisfaction withresource allocation

Documentation of partnershipagreements for engagementwith local communities

Equivalence in student assess-ment results across sites

“Champions” are identified andsupported in community andstakeholder groups

Funding to support engagementand services in priority areas issought from a range of sources

We allocate resources to opera-tionalise our plans for commu-nity engagement and deliveryof the program in communitieswhere there is the greatestneed for the provision of highquality health services

We encourage reciprocal contri-butions among the school,community and other stake-holders

Measurement Tools:Credible, relevant needs assessment

Community and student interviews assessing satisfaction

Case studies of communities/teaching sites

Interviews/surveys/focus group discussions with ‘champions’ that aresupportive of the school as well as those not familiar or supportive ofthe school

Mapping of external partnerships including external Memoranda of Understanding

Assessment results across sites (matched with resource allocation)

Data Sources:Workforce plans and budgets

Evidence of grant funding to support underserved populations

Strategic plan and other planning documents

Student handbooks/manuals and policies/cultural manuals outliningroles and responsibilities

Documented community engagement policies

Section 2: What do we do?



2.2. What, where andhow do we teach? (Education program)

Curriculum contentCurriculum reflects identified priority health and socialneeds of the community

Stakeholders involved in curriculum design and placements

Curriculum design, delivery, assessment and evaluationreflect the desired graduate attributes, the principles ofgeneralism and integration of basic and clinical scienceswith population health and social sciences

Inter-professional education opportunities available to allstudents

Assessment is designed to assess the acquisition of theknowledge, skills and behaviours required by socially ac-countable practitioners in responding to health needs ofunderserved populations

Teaching methodologyRationale for teaching methodology clearly outlined interms of social accountability

Teaching methodology is relevant and appropriate tolearner’s needs and context

Community placementsField placements developed to provide adequate exposureto priority health needs whilst learning in context

Continuity of community and clinical experience through-out the curriculum

Clear rationale for teaching site selection

Length of time student spends in supported, educationallysound placements congruent with learning needs

We have an education programthat reflects the priority healthsystem, workforce and needs ofthe communities we serve, asdefined by community partner-ships. This is evident withincurriculum, pedagogy, assess-ment and teaching sites.

Learning objectives are devel-oped based on professionalroles and responsibilities thatare reflective of current andprojected health and workforceneeds.

Measurement Tools:Interviews with curriculum committee members

Community profiles showing demographicsand health indicators reflecting site selection

Qualitative data supporting innovative, qualityinter-professional experiences (e.g. focusgroups with students and faculty/staff)

InInterviews/surveys/focus groups with stu-dents and staff/faculty, health care providersand community

Data Sources:Curriculum documents – planning documents,curriculum maps (including community place-ment plan), workshops, publicationsStudent placement policies and database

Student handbooks

Student community placement reports

Curriculum database in which learning objec-tives align with priority health needs

Section 2: What do we do? continued



2.3. Who do we teach? (Learners)

Percentage of student intakefrom reference population

Proportional intake of under-served groups

Explicit and targeted supportand pathways for underservedpopulations

Student progress and comple-tion rates are equivalent acrossstudent groups

Advocacy to support medical ed-ucation for underserved groups

We enrol and support studentswho reflect the socio-demo-graphic charactistics of our ref-erence population, especiallyunderserved populations

Measurement Tools:Interviews/surveys/focus groups discussions with students from underserved groups

Data Sources:Student database

Selection policy and processes

Rural classification

Student support policies: including secondary school recruitment policies

Wider university polices on student support and selection

Student attrition, progress and completion statistics



2.4. Who does theteaching? (Educators)

We recruit and support educa-tors and other staff who:

Reflect the demographics of ourreference population

Reflect the balance of clinical,biomedical and social sciences

Demonstrate commitment toSA principles

We engage and support com-munity and community healthservice providers as educatorsin a manner which strengthenslocal health services

Faculty/staff and promotions re-flect a diverse mix of professional,cultural and community back-grounds

Proportional representation andretention of underserved groupson faculty and non-academic staff

Health service providers and community can engage in the design and implementation of the program

Staff and faculty undertake crosscultural training

Presence of community precep-tors in the underserved commu-nity

Role of community is formalizedthrough adjunct appointments

Measurement Tools:Log of community preceptors, including geographical distribution

Validated tools assessing social accountability values of staff

Audit of percentage of school staff/faculty who have undertaken cultural awareness training

Interviews/surveys/focus groups with students and staff/faculty, health care providers and community

Data Sources:Log of community preceptors, including geographical distribution

Validated tools assessing social accountability values of staff

Audit of percentage of school staff/faculty who have undertaken cultural awareness training

Interviews/surveys or focus groups with students and staff/faculty,health care providers and community

page 18THEnet’s Evaluation Framework for Socially Accountable Health Professional Education

2.5. How does our re-search programrelate to our mis-sion and values?(Research)

Projects are community-orientedinvolving community/populationmembers and other key stake-holders at all stages

Projects focus on culturally ap-propriate, affordable and innova-tive solutions to priority healthand health service problems andhealth promotion

Proportion of research projectsemploying participatory methodologies

Research priority agenda reflectssocial accountability and alignswith national and regional priorities

Translational impact of researchon policy and practice

We have a research agendathat reflects priority health andhealth system needs of our ref-erence population developedand undertaken in partnershipwith key stakeholders with afocus on participatory method-ologies

Measurement Tools:Audit of student engagement in community action projects

Audit of publications, presentations and joint authorship withcommunity partners

Data Sources:Memoranda of Understanding

Community action research/participatory action research projects

Number of relevant grant applications

Number of internal school and faculty grants supporting social accountability projects.

Number of higher research degree and Honours students under-taking projects addressing priority health needs

Records of ethics applications

Record of community requests for partnerships and projects andactual projects and partnerships

Research priority agenda


Aspirations Indicators Suggested Sources of EvidenceKey Component


2.6. What contributiondo we make to delivery of healthcare? (Service)

Service provider participation ineducation/training activities inthe community

Students provide services as partof their training that benefits localcommunities in terms of access,utilization and quality of care

Percentage of student and stafftime directly involved in servicedelivery

Number/proportion of co-appoint-ments

Examples of school responsive-ness to community feedback onservice delivery

Community able to describechanges as a consequence ofschool involvement

Service learning is valued withinthe program and by stakeholders

Student projects partnered withthe community

Educators and students are in-volved in service delivery re-lated to changing priority healthneeds of reference populationsand reflective of future workingenvironments

Educators and students are in-volved in community develop-ment

Measurement Tools:Community/reference population interviews/focus group discussionson faculty/staff contribution to communities

Audit of student placement diaries

Data Sources:Staff human resource policy for health professional work

List of adjunct staff

Curriculum framework

Record of student service activity

Training and placement records for faculty and students

Faculty placement database

Mission statement

Records of student projects

Record of community representation

Examples of changed policies in response to community feedback,and examples of new programs/services/initiatives/projects




3.1. Where are ourgraduates andwhat are theydoing? (Human resources)

Graduate knowledge, attitudes and skills are appropri-ate to their practice and settings

Number/proportion of graduates participating in continuing professional development appropriate topractice and setting

Number/proportion of graduates working and remain-ing in communities of need

Performance of graduates in certification exams

Distribution of graduate specialization proportional toneed

Balance of graduates working in public versus privatesystem, urban versus rural areas, primary versus sec-ondary versus tertiary care settings

Evidence that gaps in health services are being addressed by local graduates

Graduate support programs

Graduate linkage to school after graduation

Processes in place facilitating quality care (e.g. rurallocum support)

Number/proportion of alumni available to mentor students and new graduates

Graduates and students recognized by community and government as key advocates

We produce graduates techni-cally, socially and culturallysuited to address the healthand social needs of referencepopulations and health system.This will be reflected in geo-graphical location, careerchoice, and professional behaviour

Our graduates particularly en-gage in comprehensive primaryhealth care, (addressing the so-cial determinants of health) andbroader advocacy and reformWe are involved in the contin-uum of medical education andsupport alumni beyond gradua-tion to promote access, qualityand efficiency of health care

Measurement Tools:Validated tools for measuring social accountabil-ity/social responsibility13

Survey of heads of departments on junior doctorperformance

Interviews with community

Interviews/survey with key staff and graduates

Longitudinal graduate outcome survey and otherknowledge, attitude and behaviour surveys

Data Sources:Internal graduate tracking processes

Measurement of integration of graduates includ-ing community integration

Department of Health score cards or evaluations

Alumni tracking processes

School publications

Continuing Professional Development policiesand processes and MoUs with postgraduatetraining providers

List of advocacy activities and meetings tomeasure reciprocity

Section 3: What difference do we make?


page 21www.thenetcommunity.org

3.2. What differencehave we made toour reference pop-ulation and refer-ence healthsystems? (Partnerships andeffects on healthoutcomes)

Perceived strength of partner-ships with socially accountableentities

Models of partnerships reflectsocially accountable values

Numbers of community meet-ings held and joint activities andlevel of participation

Perceived impact of medicalschool by community and healthservice

Volunteer work of graduatesCommunity satisfaction withcare

Measures of access to care

Health, economic and socialoutcome indicators

Access to health education op-portunities and facilities

Retention of health profession-als in the community

Outcomes and impact of stu-dents and faculty projects

We have a positive impact onpriority health and social needsof our reference population. In partnership with stakehold-ers we contribute to the trans-formation of health systems tobe more relevant to the healthneeds of our reference popula-tions

We are recognized agents ofpositive change by our refer-ence population/partners andstakeholders

Measurement Tools:Qualitative tools such as Most Significant Change, a participatory mon-itoring and evaluation of community engagement14

Longitudinal study of population health outcomes

Longitudinal studies assessing the efficiency (incorporating cost-effectiveness) of socially accountable health professional education

Economic and social impact study (case studies). For example see Exploring the Socio-Economic Impact of the Northern Ontario School of Medicine. Available at: http://www.nosm.ca/reports/

Data Sources:Memoranda of Understanding or other arrangements with key stakeholders

School annual report

Interviews or focus groups with communities, health service providers

Social, economic and infrastructure data for populations

Section 3: What difference do we make? continued



3.3. How have weshared our ideasand influencedothers? Dissemination/Promotion andsustainability/transformationalchange)

Number of relevant publications

Number of relevant conferencepresentations

Examples of changes to healthservice delivery/policy as a resultof school’s activity

Level of community support ofthe school

Quality improvement frameworks(e.g. accreditation)

Contribution to professionalgroups

Partnerships with relevant stake-holders including other universi-ties

Faculty/staff and student exchanges

Joint conferences and profes-sional organizations

We are engaged in a continuousprocess of critical reflection andanalysis with others and dis-seminate these learnings inmany ways

We influence policy makers, ed-ucation providers and otherstakeholders to transform thehealth system

Measurement Tools:Social network analysis

Data Sources:Relevant school publication/conference presentations and meetingrecords

Community meetings or newsletters

Workshops

Accreditation visits/ external examiners reports

Exchanges students/staff/academic or community visitors

Policy records and quality improvement frameworks

Proof of research utilization

Annual and financial reports

Qualitative methodologies such as Most Significant Change, a partici-patory monitoring and evaluation of community engagement14

interviews with communities and service providers

Number of benefactors supporting the school

Web search of other medical schools annual and financial reports

Office holders involvement in non-profit groups and professionalgroups

Enquiries from other health professionals about social accountability




3.4. What impact havewe made withother schools?(Peer support andreplication)

Number of schools partneringwith THEnet referred/facilitatedby the school

Number of schools joined in so-cially accountable projects orrecognizing social accountabilityas a core value

Schools assisted to adopt socially accountable health professional education

Relevant joint research projects– number and topic

Number of publications andconference presentations

Number and site of peer educa-tion/mentoring visits

Mentoring ties across schools

Site visits to other schools

We actively engage with andsupport other institutionsacross national boundaries to achieve common social accountability goals

Measurement Tools:Tracking system of collaborative efforts

Data Sources:THEnet website

Website activities/hits

Number of requests for collaboration

Meeting notes

Joint publications/conference papers

Documentation of mentoring relationships

Records of joint projects and project outcomes



page 24THEnet’s Evaluation Framework for Socially Accountable Health Professional Education

Aspirations: A vision statement; the end objec-tive for schools aspiring to social accountability.

Champions: These are defined as communitypersonnel (often but not always health profession-als) who will act as key linkage personnel in sup-porting a partnership or community engagementprocesses. They may be referred to by differentnames in different contexts, including communityliaison personnel, linkage officers and so on.

Community Development: This is defined asformal and informal activities that students andstaff undertaken in partnership with communitiesthat enhance community health and social wellbeing.

Community Engagement: THEnet schools be-lieve that ‘real’ community engagement involves agreater depth of involvement, or engagement, indecision-making processes and all stages of plan-ning, implementation and evaluation15 16.So, theterms ‘engagement’ and ‘participation’ may covera multitude of ways in which the community and/or stakeholder groups are actually involved inschools. It is useful to be aware that there are different levels of community or stakeholder en-gagement. It is important to note that these levelsare not mutually exclusive, but rather represent a continuum in the ways that stakeholders andcommunity members may be engaged in program planning and evaluation.

Community Partnerships: We define partner-ships as conscious high quality engagement withcommunities and/or individuals where the intent is shared decision making.

Comprehensive Primary Health Care: This isdefined according to the WHO as “... socially ap-propriate, universally accessible, scientificallysound, first level care provided by health servicesand systems with a suitably trained workforcecomprised of multidisciplinary teams support byintegrated referral systems in a way that givespriority to those most in need and addresseshealth inequalities; maximizes community and in-dividual self-reliance, participation and controland; involves collaboration and partnership withother sectors to promote public health.”17.

Existing Data Sources: This includes data anddocuments already being collected at schools. Itmay include existing surveys, evaluations or re-search projects. For example a graduate survey or a qualitative research project with communityleaders.

Faculty and Non-academic Staff Definitions:We recognize that different terms are used torefer to academic and non-academic universitystaff in different parts of the world. For the purposes of this framework we will refer to academic staff as faculty, and other staff as non-academic staff.


Guide to Evaluation Framework Terminology

Focus Groups/Surveys/Interviews: THEnetschools found that some information was not doc-umented and was best captured through inter-views or focus group discussions. THEnetImplementation Guide outlines suggested focusgroup/interview questions.

Health and Social Needs: We use the WorldHealth’s Organization’s definition of health: "...astate of complete physical, mental, and socialwell-being and not merely the absence of diseaseor infirmity."18.

Health System: We use the World Health’s Orga-nization’s definition of a health system: “A healthsystem consists of all organisations, people andactions whose primary intent is to promote, re-store and maintain health….Its goals are improv-ing health and health equity in ways that areresponsive, financially fair and make the best, or most efficient, use of available resources.”19.

Indicators: These are measures, both quantita-tive and qualitative, of the progress of a school to-wards social accountability.

Key Component contains the key elements of the framework written plain English.

Measurement Tools: These include tools thathave been developed by THEnet for use byschools, and existing validated tools that are appropriate to the aspiration.

Participatory Action Research has several defi-nitions including “Participatory research is definedas systematic inquiry, with the collaboration ofthose affected by the issue being studied, for pur-poses of education and taking action or effectingchange.”20. It referce to research that is based onshared ownership of the research project, a com-munity-based analysis of social problems, and anorientation towards community action. It often includes a commitment to bring together broadsocial analysis, and action to improve things.

Partner Schools/Individuals has been used in-terchangeably to describe tertiary educationschools and individuals members who are partnersof the Training for Health Equity Network.

Professional Behaviour: In addition to tradition-ally accepted definitions of clinical professional-ism, for THEnet schools desirable professionalbehaviour is embodied in competent and confidenttransformational scholars dedicated and equippedto be agents of change in health systems and so-cieties. We therefore expect our graduates to havea thorough understanding of the biological, psy-chological and social determinants of health, anda strong commitment to providing high qualityhealth services and addressing inequities in health care delivery.

Resources: We refer to human, financial or in-kind resources.

Reference Communities/Populations are de-fined as the population that the school serves andthe communities within this. This needs to be de-fined for each participating school with particularreference to the underserved populations in termsof health services.

Service-Learning "...is a structured learning ex-perience that combines community service withpreparation and reflection. Students engaged inservice-learning provide community service in re-sponse to community-identified concerns andlearn about the context in which service is pro-vided, the connection between their service andtheir academic coursework, and their roles as citi-zens”.21.

Social Accountability: THEnet schools collec-tively agreed to adapt the World Health Organiza-tion’s21 definition of social accountability, amendedto bring a focus on the underserved in line withtheir core mission (in brackets): “Social accounta-bility is the obligation to orient education, re-search, and service activities towards priorityhealth concerns of the local communities, the re-gion and/or national (schools) one has a mandateto serve. These priorities are jointly defined bygovernment, health service organizations, and thepublic, [and especially, the underserved”].


Guide to Evaluation Framework Terminology continued

Boelen C, Woollard RF. Social accountability and accreditation: a new frontier for educational 1institutions. Medical Education. 2009 Sep;43(9):887-94.

The Save the Children Fund. No Child Out of Reach - Time to End the Health Worker Crisis.2Save The Children UK; 2011.

Commission on the Social Determinants of Health. Closing the gap in a generation: health3equity through action on the social determinants of health. Final Report of the Commissionon Social Determinants of Health. Geneva: World Health Organization; 2008.

Frenk J. The Global Health System: Strengthening National Health Systems as the Next Step4for Global Progress. PLoS Med [serial on the Internet]. 2010; 7(1).

Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al. Health professionals for a new5century: transforming education to strengthen health systems in an interdependent world.The Lancet. 2010;376(9756):1923-58.

Mills EJ, Kanters S, Hagopian A, et al. The financial cost of doctors emigrating from sub-6Saharan Africa: human capital analysis. BMJ 2011.Nov 23;343:d7031.

Committee on the Robert Wood Johnson Foundation Initiative on the Future of Nursing at 7the Institute of Medicine. The Future of Nursing Leading Change, Advancing Health. Washington DC: The National Academies Press; 2010. Available from:http://www.nap.edu/openbook.php?record_id=12956&page=R1.

Institute of Medicine. Health Professions Education: A Bridge to Quality. Washington DC: 8National Academy Press; 2003.

Cooke M, Irby D, O’Brian B. A summary of educating physicians: a call for reform of medical9school and residency. Stanford (CA): The Carnegie Foundation for the Advancement ofTeaching; 2011. Available from: http://www.carnegiefoundation.org.

Global Consensus for Social Accountability of Medical Schools. Global Consensus for Social10Accountability of Medical Schools: Position statement Global Consensus for Social Accounta-bility of Medical Schools; 2010. Available from: http://healthsocialaccountability.org/.

Boelen C, Heck J. Defining and measuring the social accountability of medical schools.11Geneva: World Health Organisation 1995. Report No.: WHO/HRH/95.7.

Taylor J, Wilkinson D, Cheers B. Working with communities in health and human services. 12Melbourne: Oxford University Press; 2008.

Crandall SJ, Volk RJ, Loemker V. Medical students' attitudes toward providing care for the13underserved. Are we training socially responsible physicians? JAMA. 1993 May19;269(19):2519-23.

Dart JJ & Davies, RJ (2003), A dialogical story-based evaluation tool: the most significant14change technique, American Journal of Evaluation 24, 137–155.

Oakley P. Community involvement in health development: an examination of the critical 15issues. Geneva: WHO 1989.

Aslin HJ, Brown VA. Towards whole of community engagement: a practical toolkit. Canberra:16Murray-Darling Basin Commission 2004.

WHO, UNICEF. Declaration of Alma-Ata. Geneva World Health Organization.17(http://www.who.int/hpr/NPH/docs/declaration_almaata.pdf

page 27

References

Grad FP. The Preamble of the Constitution of the World Health Organization. Bulletin of the18World Health Organization [serial on the Internet]. 2008; 80(12): Available from:http://www.who.int/bulletin/archives/80(12)981.pdf

World Health Organization. Everybody’s Business: Strengthening Health Systems to 19improve Health Outcomes: WHO’s Framework for Action. Geneva: WHO 2007.

Green LW, George MA, Daniel M, Frankish CJ, Herbert CP, Bowie WR, et al. Appendix C:20Guidelines for Participatory Research in Health Promotion. In: Minkler M, Wallerstein N, editors. Community-Based Participatory Research for Health. San Francisco,CA: Jossey-Bass Inc.; 2003.

Cashman SB, Seifer SD. "Service-Learning: An Integral Part of Undergraduate Public21Health." American Journal of Preventive Medicine; 35(3):273-278. 2008.

World Health Organization. World Health Assembly Resolution WHA48.8: Reorientating 22medical education and medical practice for health for all. Geneva: World Health Organization 1995.

page 28

References continued