Upload
stephen-dennis
View
215
Download
2
Tags:
Embed Size (px)
Citation preview
Program Evaluation
• A systematic effort to describe the status of a program
• Extent to which program objectives achieved
Uses of Health Program Evaluation
Insight - Needs
- Barriers
- Activities
Improvement- Social mobilization
- Inter- sectoral Coordination
- Implementation
- Client Conveniences
Affordability
Accessibility
Availability
- Cost - Benefit
Outcomes / Impact- Skills development
- Behavioral change
- Level of success in achieving objectives
- Accountability
Types of Evaluation
Needs Assessment
To identify
• Goals
• Products
• Problems
• Conditions
Types of Evaluation Contd…
Formative (Process) Evaluation
To improve developing or ongoing program
Role as helper/advisor/planner
Progress in achievements
Potential problems/needs for improvements in
• Program Management
• Inter-sectoral coordination
• Social mobilization
Implementation
Outcomes
Types of Evaluation Contd…
Summative (Coverage) Evaluation
(To help decide ultimate fate)
Summary statement about
Program’s achievements
Unanticipated Out comes
Comparison with other programs
Sample Size
Factors
- Purpose of study
- Population size
- Level of precision (sampling error)
- The confidence / Risk level
- Degree of variability
- Appropriate for the analysis
- Appropriate for comparative analysis of sub groups
- Add for non respondents
Sample Size Contd…
Strategies
- Using a census - small population
- Using sample size of a similar study
- Using published tables / software
- Using formulas
Health Program Evaluation - Quantitative Research Methods
Approach
-Measures the reaction of a great many
people to a limited set of questions
-Comparison and statistical aggregation of
the data
-Broad, generalizable set of findings
presented succinctly and parsimoniously.
Health Program Evaluation Qualitative Research Methods
• Participant Observation• Key Informant Interviews• Open ended Interview• Focus Group Discussions • Pile sort
Health Program Evaluation - Qualitative Research Methods Contd…
Findings- Presented alone / in combination with quantitative data
- Validity and reliability depends on methodological skills, sensitivity, integrity of the researchers
- Skillful interviewing - more than just asking questions.
- Content analysis - more than just reading to see what’s there.
- Generate useful and credible findings through observation, interviewing and content analysis
How?- Discipline, knowledge, training, practice, creativity, hardwork
Data Processing
• Raw field notes should be corrected edited and typed
• Tape recordings need to be transcribed and corrected
• Texts by field worker should not be changed to make it ‘writable’ or
‘readable’
Data Reduction
• Process of selecting, focussing, simplifying, abstracting and transforming data from field notes and transcripts
• Researcher retains some data chunks, pulls out some and gets an idea of story to tell
Analysis Steps
• Free listing
• Domain Evolution
• Coding
• Tabulation
• Summarizing
Quotable Quotes
• Give a vivid, meaningful flavor which is far more convincing than pages of summarized numbers
- These should not be distracters
- Should not take the reader away from the real issues in hand
“This is an unprecedented event where all people irrespective of caste, creed and religion take part in PPI program on the same day (NID) throughout the country”
•Health worker (150): Burdwan
“He (my husband) told me that everybody is going for polio drops. Then why should we be left out ? After all, everybody is not a fool”
•Utilizer (1422): Delhi
“We have not at all immunized our son. My husband was very stubborn. He said ‘those who are immunized are also getting this disease (polio) and whatever happens let it happen’. He has not allowed me to get the child immunized”
•Non utilizer (630): Hyderabad
Data Display
• This is an organized, compressed assembly of information that
permits conclusion drawing and action
• Matrices, graphs, charts and networks are used
Qualitative Vs Quantitative Research - the dichotomy
Qualitative Quantitative
Social theory Action Structure
Methods Observation, interview Experiment, survey
Question What is X? (classification)
How many Xs? (enumeration)
Reasoning Inductive Deductive
Sampling method Theoretical Statistical
Strength Validity Reliability
Qualitative Vs. Quantitative
• It is not qualitative Vs. quantitative but qualitative and quantitative
• Qualitative methods are rapid, exploratory and hypothesis generating
• Can be used as Impact evaluation research
• Allow the researcher to palpate the unique cross-cultural features
Multi-Centric Evaluation Studies
Issues• Common understanding of the program
• Common understanding of aims and objectives of evaluation exercise
• Standardization of research instruments
• Standardization of protocol implementation at various sites
• Regional variation in program implementation
Multi-Centric Evaluation Studies Contd…
Steps • Cohesive network of partners
• Multi-disciplinary team of investigators
• Piloting of instruments at different sites
• Development of common understanding
• Training of research teams
• Multiple layers of quality assurance measures
Sustaining the Network
During active period:
• Recognized individual efforts to excel - Co-opted as extended CCT members
• Recognized PMC efforts to excel
- Made regional centers
• Communication Channels (phone, fax, e-mail)
• Pilot of instruments (ownership)
During interface:
• IndiaCLEN / INCLEN meetings, workshops
• Feedback on completed reports
Network Dynamics
Network Dynamics Contd…
Quality Assurance Measures
• National orientation workshop (PI s)
• Regional orientation workshops (PI s, RA s)
• PMCs, Regional Centers:
Quality checks of interviews, schedules, tapes
• Central Coordinating Office:Random checks of recordings, interviewing procedure, transcripts, translations
• Regional Coordinators, CCT members:
Site visits / Supervision of FGDs
• Method triangulation using
In-depth interviews, FGDs
Quality Assurance Measures
• Development of Interview Schedules
• Consistency Checks / VALIDITYMethod Triangulation
Data Triangulation Data Interpretation
• Partner Medical Colleges
• Regional CEUs
• Central Coordinating Office
Capacity Building
National Level• Leadership transfer to coordinate MI project
- Thiruvananthapuram
State Level• PMCs upgraded to Regional Centers• Upgrading of physical facilities• Ten investigators INDEPENDENTLY took up
evaluation of national programs at state/district level
Network Partners• Research - individual/collaborative• Resource persons - local/regional/national• Extended CCT members
Interaction with Program Managers / Policy Makers
Program Evaluation: A partnership exercise
• Developing objectives, instruments
• Dissemination of Findings
• Support other program related activities
Dynamics of Establishing Partnerships with Policy Makers
Dynamics of Establishing Partnerships with Policy Makers
Evaluators express their opinions explicitly
- based on evidence gathered
- consistent quality assurance measures
- limitations of study accepted up front
- politics of evaluation
- remains a scientific endeavor
Evaluators express their opinions explicitly
- based on evidence gathered
- consistent quality assurance measures
- limitations of study accepted up front
- politics of evaluation
- remains a scientific endeavor
Dynamics of Establishing Partnerships with Policy Makers
Dynamics of Establishing Partnerships with Policy Makers
• Results to be presented in a manner which are perceived as VALID, RELIABLE & FEASIBLE TO IMPLEMENT
• Working in a strict TIME SCHEDULE for timely fine-tuning of strategies
• Program Evaluators have to establish CREDIBILITY with Program Managers
• Results to be presented in a manner which are perceived as VALID, RELIABLE & FEASIBLE TO IMPLEMENT
• Working in a strict TIME SCHEDULE for timely fine-tuning of strategies
• Program Evaluators have to establish CREDIBILITY with Program Managers
How can Evaluation Data be Used?
Program managers
– Redefining aims objectives – Modifying or fine tuning strategies (process)– Sustainability (including fatigue factor)– Judge the worth (impact)– Expense / cost– Interaction with other activities
Education
– Generalisability – Unique features (success/failure)
– Determinants of provider and client behavior
How can Evaluation Data be Used?
IndiaCLEN Program Evaluation Network
Activities
Studies Completed (1997-2001)
Title Year Funding Source
Pulse Polio Immunization Program (PPI-1)
1997- 98 USAID/INCLEN
Pulse Polio Immunization Program (PPI-2)
1997- 98 USAID/INCLEN
Intensive Pulse Polio Immunization Program (IPPI-3)
2000 USAID/INCLEN
Family Health Awareness Campaign (FHAC-1)
1999 USAID/INCLEN
Barriers in Polio Eradication (Moradabad, UP)
2000 WHO/SEARO
Family Health Awareness Campaign (FHAC-2)
2000 (Coverage & Process)
USAID/INCLEN
Forthcoming Studies
Title Year Funding Source
Integrated Disease Surveillance Program: Barriers in Surveillance Activities in three States (pilot-FGDs)
2001 World Bank
Vitamin-A and Iron Folate Supplementation Program
2001- 02 MI/IDRC
Safe Injection Practices (Coverage & Process)
2001- 02 World Bank
Evaluation of AFP Surveillance 2002 USAID/INCLEN
Family Health Awareness Campaign (Coverage)
2002 USAID/INCLEN
IndiaCLEN Members -1996
Delhi
Lucknow
ChennaiVellore
Nagpur
Thiruvananthapuram
IndiaCLEN Program Evaluation Network - 1997
•
•
•
•
•••
••
Srinagar
Delhi
AgraLucknow
Jodhpur Dibrugarh
Berhampur
Bhopal
Bilaspur
Tirupati
ChennaiVellore
Nagpur
MumbaiGulbarga
Bangalore
Calicut Madurai
Thiruvananthapuram
Ahmedabad•
•Panaji
Burdwan•
Patna•
Ballabhgarh•
IndiaCLEN Program Evaluation Network - 2000
•
•
•
•
••
••
Srinagar
Delhi
AgraLucknow
Jodhpur Dibrugarh
Berhampur
Bhopal
Bilaspur
Tirupati
ChennaiVellore
Nagpur
Mumbai Gulbarga
Bangalore
Calicut Madurai
Thiruvananthapuram
Ahmedabad•
•Panaji
Burdwan•
Patna•
Ballabhgarh•
Kangra•Rohtak•
Jaipur•Kota• Gwalior• •Kanpur Darbhanga•
Kolkata•• •
•••Guwahati Kohima
Imphal
Aizwal
Agartala
Sambhalpur Cuttack• •
Vijayawada
Hyderabad••
Kannur •••
Jamnagar•
Bikaner•
Agenda Item No.14- Conduction of Family Health Awareness Campaign
A brief (15 minutes) presentation on evaluation of FHAC round 2000 was made by Dr. N.K. Arora, IndiaCLEN, AIIMS.
“Addl. Secretary & Project Director (NACO) said that the short comings observed in evaluation of the campaign should be taken into consideration while preparing action plans for the next round of FHAC in the year 2001. After discussion (one hour 15 minutes) with the State Project Directors, it was decided that…”
Letter No.T.11014/2/2001-NACO dated 05.07.2001
IndiaCLEN Program Evaluation Network
VISION
Facilitate development
and implementation of
people friendly, effective
IndiaCLEN Program Evaluation Network
Investigators
CCT Members
N. K. Arora
M. Lakshman
Kiran Goswami
Sneh Rewal
R.M. Pandey
K. Anand
K.K. Ganguly
Naveet Wig
Leena Sinha
S. Vivek Adhish
N. Chaudhuri
H.K. Kumbnani
Thomas Mathew
Sandip K. Ray
S.L. Chadha
Rema Devi
K.C. Malhotra
R. Sankar
Sunita Shanbhag
Ballabhgarh
S.K. Kapoor
Praveen Kumar
M.K. Taneja
R.C. Agarwal
Rohtak
A.D. Tiwari
B.K. Nagla
Mohinder Singh
Kangra
T.D. Sharma
Swaran Lata
K.L. Ghai
G.L. Jaryal
Lucknow
R.C. Ahuja
Vikas Chandra
J.V. Singh
A.K. Srivastava
Kanpur
V.N. Tripathi
Joginder Singh
R.P. Singh
Agra
Deoki Nandan
S.K. Mishra
S.P. Agnihotri
Dibrugarh
Faruque Ahmed
Swapna D. Kakoty
Mayashree Konwar
Bijit Bhattacharya
Mir Shahadat Ali
Pranab Jyoti Bhuyan
Guwahati
Chiranjeeb Kakoty
Sajjad Ahmed
Alaka Bhattacharyya
Imphal
E. Yaima Singh
T.H. Achouba Singh
R.K. Narendra
Tiasunup Pongener
Umatula
Agartala
Partha Bhattacharjea
Aizwal
L. Lalhrekima
Jodhpur
S.L. Solanki
Suman Bhansali
Afzal Hakim
Y.R. Joshi
Kota
Raghuveer Singh
Gopal Bunkar
Hans Raj
Jaipur
Anurag Sarna
Rajesh Jain
Hemant Jain
Bhopal
Sheela S. Bhambal
A.K. Upadhyaya
R.K.S. Kushwaha
U.K. Dubey
Bilaspur
Vijay Tiwari
D.N. Mishra
Ajay Gurudiwan
Ashok Tiwari
Gwalior
A.G. Shingeweker
A. Shingeweker
Berhampur
B.C. Das
D.M. Satapathy
G.S. Patnaik
T. Sahu
Cuttack
S.C. Jena
S.K. Sahu
K. Misra
Sambhalpur
O.P. Panigarhi
H.P. Acharya
S.C. Panda
Nagpur
A.K. Niswade
Sanjay Zodpey
Sanjay Deshpande
Suresh Ughade
Prashant Langewar
Mumbai
Alka Jadhav
Nagaonkar
Nitin Deshpande
Shubhangi Upadhye
Chitra Nayak
Vijayawada
S. Narasimha Reddy
T.S.S. Manidhar
A. Rama Prasad
C. Usha Rani
Hyderabad
B.V.N. B. Rao
C. Bala Krishna
J. Ravi Kumar
Tirupati
K. Raghava Prasad
N.A. Chetty
G. Raviprabhu
Gulbarga
B. Mallikarjun
R.R. Rampure
B.N. Patil
Shreeshail Ghooli
Bangalore
Shivananda
Gopal
Premalata
Bangalore
A.S. Mohammad
Lalita Bhatti
R.M. Christopher
Chennai
R. Sathianathan
A. Vengatesan
R.K. Padmanaban
S. Karthikeyan
Vellore
Kurien Thomas
O.C. Abraham
Mary Kurien
Madurai
C. Kamaraj
M. Eswaran
T. Rajagopal
Thiruvananthapuram
M. Narendranathan
P.S. Indu
J. Padmamohan
S.M. Nair
Kannur
Jeesha C. Haran
T.P. Mubarack Sani
M. Jayakumary
Calicut
M. Ramla Beegum
C.R. Saju
N.M. Sebastien
Program Evaluations - Relevance to PolicyProgram Evaluations - Relevance to Policy
• ACADEMIA can play an important role in influencing the National Policy
- multi disciplinary teams
• Evaluations are not done in VACUUM, should be Policy Relevant
- central, state, district level
• RECOGNIZE Policy Makers & Other Stakeholders as partners
• ACADEMIA can play an important role in influencing the National Policy
- multi disciplinary teams
• Evaluations are not done in VACUUM, should be Policy Relevant
- central, state, district level
• RECOGNIZE Policy Makers & Other Stakeholders as partners
Models of Program Evaluation
• Goal oriented evaluation
Aimed to assess the progress and the effectiveness of innovations/ interventions.
• Decision oriented evaluation
Aimed to facilitate intelligent judgements by decision makers.
• Responsive evaluation
Aimed to depict program process and the value perspectives of key people.
• Evaluation Research
Focused on explaining effects, identifying causes of effects, and generating generalizations about program effectiveness.
• Goal free evaluation
To assess program effects based on criteria apart from the program’s own conceptual framework, especially on the extent to which real client needs are met.
• Advocacy - adversary evaluation
Evaluation should derive from the argumentation of contrasting points of view.
• Utilization - oriented evaluation
Structured to maximize the utilization of its findings by specific stakeholders and users.
Design Effect
Ratio of variance with cluster sampling to variance with simple random sampling
Var simple random sampling = p(1 - p)
n
Var cluster sampling = (pi - p)2
K(k-1)
Design effect = (pi - p)2 n
k(k-1) p(1-p)
Health Program Evaluation - Quantitative Research Methods
Approach
- Measures the reaction of a great many people to a limited set of questions
- Comparison and statistical aggregation of the data
- Broad, generalizable set of findings presented succinctly and
parsimoniously.
Summary
Qualitative methods aim to make sense of, or interpret, phenomena in terms of the meanings people bring to themQualitative research may define preliminary questions which can then be addressed in quantitative studiesA good qualitative study will address a clinical problem through a clearly formulated question and using more than one research method (triangulation)Analysis of qualitative data can and should be done using explicit, systematic, and reproducible methods
• Lessons of success & failure
• Wider application of program strategies
• Determinants of client behavior
• Impact on other health systems [national & international interest in later part]
Development of Program Objectives (Program Evaluators)