1
Day 3
Technical Sessions - 2Social Accountability Tools
Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Tool# 3
Mystery Visitor
2Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
The Concept
• Simple & popular tool used since 1940s
• Quantitative & Qualitative indicators
• Relies on observation techniques
• Best used for observing infrastructure, processes,
staff behaviour etc.
• Uses trained anonymous personnel
Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
The Methodology
• Identifying service/institution – one location Vs several
• Developing indicators – what to observe?
• Choose timings – when to observe?• Choosing method of enquiry – personal visit /
telephone calls• Developing recording protocols – how to collect
and organize information• Uses ‘scores’ or ‘ratings’ to provide objective data.
Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
An Example• Objective: Audit the service delivery processes at a
business licence issuance centre.• Indicators: - Access (time of operations); - Availability of information regarding fees, application
procedures, Processing time, complaints; - Infrastructure – cleanliness, seating availability,
availability of drinking water, toilet facilities; - Interaction – staff behavior, following queues, presence
of agents/middlemen- Developing scores
Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Indicator/Sub-indicator Scoring Format Actual Score
Maximum Score Possible
1. Access:1.1 Opening time
Opened on time = 10Opened with a slight delay= 5Opened after a long time = 0
5 10
1.2. Lunch Interval Followed timings = 10Did not follow timings = 0
0 10
2. Availability of information2.1 Fees
Information provided = 10No information given = 0
10 10
2.2 Procedure Information provided = 10No information given = 0
0 10
2.3 Time Information provided = 10No information given = 0
0 10
2.4 Complaints Information provided = 10No information given = 0
0 10
3. Infrastructure3.1 Cleanliness
Very clean = 10Somewhat clean = 5Dirty = 0
5 10
Total Score 20 70
Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
DiscussionCan Mystery Visitor Audits be implemented in
our context? If yes, which service is most enabling?
If no, why not? What are the key challenges? Can an adapted form be implemented?
Quickly list out some indicators you would like to observe and record for the service you have identified
7Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Tool# 4
COMMUNITY SCORECARDS
8 DGT - AA Myanmar Workshop, Mar 2013
What is CSC?
• A community based approach that combines key elements of:
- PRA techniques (Participation)- Public Hearings (Disclosure)- Citizen Report Card (Measurements)- Budget, Input Tracking & Infrastructure Audit
(Entitlements)
9Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
CommunityScoreCard
Participation
Disclosure
Measurements
Claims
Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Defining Features of a CSC
tracking of inputs or expenditures (e.g. availability of medicines)
monitoring of the quality of services/projects, generation of benchmark performance criteria
that can be used in resource allocation and budget decisions,
generating a direct feedback mechanism between service providers and community,
building local capacity and strengthening community voice and
empowerment. 11Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Preconditions for a CSC
• understanding of the socio-political context of governance at the local level,
• strong facilitation skills, • a strong awareness campaign to ensure
maximum participation from the community and other local stakeholders, and
• steps aimed at institutionalizing the practice for repeated actions.
12Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
CSC – The Methodology
13
I• Preparatory Phase
II• Input / Expenditure Tracking
III• Community Generated Score Card
IV• Self Evaluation by Provider
V• Interface Meeting
VI• Institutionalization
Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
1. Preparatory Phase Step 1: Identifying & Training of Facilitators: 3 day
workshop (with a manual) with a field practical Step 2: Scoping visit to meet community
leaders/representatives Step 3: Orientation meeting with service providers. Step 4: Get basic data on the community- Population
data, Service provision data, Social & Poverty Profiles (Social Map, Well Being Ranking)
Step 5: Logistics – No. of facilitators, No. of focus groups, Timing of Interface Meeting
Step 6: Awareness Building & Mobilization15Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
2. Input Tracking“Data on budget expenditure, physical assets, service inputs etc.”
Step 1: Decide and Obtain Information on Inputs to be Tracked- E.g., number of doctors allocated, number of para-medical staff allocated, number of beds, list of medicines at the hospital, budget – expenditure statements etc.
16Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
2. Input Tracking (Contd.)
Step 2: Share information on Entitlements (Plenary with community & provider)
Step 3: Divide gathering into focus groups
Step 4: Fill in the Input Details
Step 5: Record data in the form of an Input Matrix
Step 6: (Optional) Physical Inspection of infrastructure
17Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
A Sample Input Tracking Matrix
Dr. Gopakumar Thampi, Sanchi, April 12 2012
18
SrNo. Input
Indicator Entitlement Actual Remarks/evidence
1 Number of Doctors
2 doctors (1 full time and 1 Part time)
1 doctor (part time) Need for two doctors
2 Working Hours of Doctors
9-12 noon (morning)2-4 PM (evening)
11-1 PM (morning)
Need to followscheduled timings
3 No. of Beds 7 beds 2 beds available Absence of 5 beds
4 Availability of medicines
Prescribed list of medicines to be available at the PHC
Basic medicines are not available at the PHC
Poor Management
5 Waiting Hall A waiting hall for the patients and the person accompanying the Patient
Absence of any space for waiting
Poor Management
Performance Scorecard by Community contd..
Classify participants in focus groups • In a systematic manner, basis for classification must be
clear. Develop performance criteria • Brainstorm within the focus group to develop performance
criteria based on which the facility and services considered will be evaluated
• Based on community discussions, facilitator lists all issues mentioned and assists the group in organizing them into measurable performance indicators
• Ensure everyone’s participation in developing indicators• Finalize and prioritize 5-8 indicators
19Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Performance Scorecard by Community contd..
Scoring the indicators• Facilitate focus groups to give relative scores for each of the
indicator. Scoring can be done either through consensus method or through individual voting . A scale of 1-5 or 1-100 is usually used – higher scores are preferred!!
Explanations for the scores• Understand the reasons behind low and high scores; helps in
explaining outliers and generates valuable information about service delivery
• Community suggestions for service improvement
• Data recording in community generated performance card
20Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Ex. of Performance Score card by Community, Health
SNo
Performance Criteria /Indicators
Score(0-100) Reasons
1 Attitude of staff 38
1.1 Punctuality 50 Start late, but some work after hours
1.2 Polite behavior 40 Many shout at patients, rude towards children
1.3 Respect for patient 25 No respect shown
2 Management of centre 55
2.1 Cleanliness 70 Place is cleaned daily
2.2 Observing working hours 40 Open on time, staff come late, long lunch hours
3 Quality of services 16
3.1 Adequate supply of drugs 25 Drugs mostly not available
3.2 Adequate infrastructure 20 No admission room, no drinking water
3.3 Ambulance service 2One ambulance for several centres.
21
Ex. of Performance Score card by Community, School Education
SNo
Performance Criteria
Score(0-100) Reasons
Recommendations
1 Availability of Teachers 50
There are four teacher posts sanctioned, three of them are available. There is absenteeism as they come from far off place
Teachers must come regularly to school
2 Infrastructure 25No sufficient classrooms, no benches, No bathrooms (problem for girls). No water
Toilets to be constructed urgently
3 Cleanliness 30School premises are not kept clean; there is no one to sweep
Contributions can be given by the community to hire
4 Compound wall 25There is no compound wall in the school; animals are grazing
Compound wall must be made at least with dry grass and thorns
5 Communication with parents 35
Poor communication of school with parents; PTAs not held regularly
Regular PTA meetings to be held
22
4. Self Evaluation by Providers
• This evaluation is carried out by the service providers on their own performance. It brings out their own perspective of their performance.
• The assessment is based on indicators / criteria developed by the provider. The reasons for these scores are documented and become the basis for obtaining suggestions for change through greater trust and partnership, which is the objective of the process.
• It is important to understand their perspective since they sometimes work under numerous constraints.
23Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Self-Evaluation Scorecard contd..
Orient Service Providers• The first step in developing the self evaluation assessment
card/report for providers is to orient them properly about the purpose and use of the Community Score Card.
Ensure adequate Participation• It is important to set out a time in advance to ensure adequate
participation of service providers.
Deciding on Performance Criteria*• The service providers come up with their own set of performance
criteria. Ideally, these should then be classified in a manner that is easily comparable with the indicators chosen by the community.
24Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Provide scores for indicators and Reasons The members then need to fill in their relative scores for each of the indicators they came up with and provide reasons for the scores.
Recording DataThe data from the self-evaluation is also recorded in the form of a score card, which looks exactly like the community assessment card/report.
Suggestions for Change/ImprovementThe members are then asked about what changes or suggestions they have for improving the quality and efficiency of the services they provide.
Self Evaluation by Providers (Contd.)
25Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Ex. Self Evaluation Score Card from Health Pilot, Vizag
SNo
Performance Criteria
Score(1-100) Reasons/Comments
Recommendations
1Doctor‘s Timings / Availability
85 Doctors are available at the PHC
-
2Staff Behaviour / Working Style
75 The PHC staff behave well with the patients
There is great work pressure therefore sometimes they are Rude
3 Medicine Dispensing 85
Medicines are available at the PHC and are given to the patients as prescribed
There is sometimes shortage of medicines.
4 Cleanliness 75 Hospital premises are kept clean
There is only one person and she is old; needs a replacement
26
The Interface Meeting and Action Planning Process
It is a public forum or meeting where the service providers and users gather in order to present their respective scorecards and discuss ways to improve service delivery.
Enables the service users to present their evaluation of the service performance, along with their concerns and priorities regarding the service.
Steps involved:
• Prepare both parties for the meeting• Ensure adequate participation from both sides• Presenting findings of both groups • Brainstorming to come up with concrete changes/improvements • Develop an Action Planning Matrix• Divide roles and responsibilities for follow-up and monitoring
27Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Steps involved in the Interface Meeting
• Prepare both parties for the meeting• Ensure adequate participation from both sides• Community cluster meeting for services provided in more than
one village area (aggregation)• Presenting findings of both groups & input tracking matrix• Summarizing Scorecard findings*• Analysis of results in plenary discussion – Start with highlighting
common criteria and scores• Brainstorming to come up with concrete changes/improvements • Develop an Action Planning Matrix• Divide roles and responsibilities for follow-up and monitoring
28Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
Ex: of an Action Plan Matrix, Health
S. No.
Indicator Action to be taken By whom and when
1 Doctor’s Timings / Availability
Change of timings :Present: Morning: 9:00 AM to 12:00Noon, Afternoon: 3:00 PM to 5:00 PMNew: Morning: 10:00 AM to 1:00 PM, Afternoon: 4:00 PM to 6:00 PM
Doctor , immediately
2 Reporting complaints
Complaint box to be introduced: Every Saturday between 11:00 AM and 12:00 PM the complaints registered to be redressed
Doctor, Immediately
3 Staff Behaviour
Trainings to be conducted for the staff Health Department
4 Availability of medicines
List of medicines available at the PHC to be prepared and put up on the display board
Pharmacist and doctor
29
30
Satkhira, Bangladesh
Repeat Score Cards
• The scorecard process is repeated after a mutually agreed upon period of time. The repeat process is usually easier and faster since everyone has prior experience.
• The main purpose of the repeat scorecard is to review progress and provide inputs for a revised action plan by:
– Scoring the indicators again to reflect any changes in performance
– Reviewing progress related to implementing the action plan
– Discussing any changes experienced in service delivery
31Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
32
No. Indicators Scores (out of 100)
Comments and reasons
August 2002
March 2003
1.1 Punctuality of staff 50 60 They start late, but work during the lunch time and after hours
There is some improvement in time keeping
1.2 Polite behaviour 40 50 There is one health worker who shouts at us when children urinate or we sit at the wrong place
There is some improvement in staff attitude
1.3 Listening to patients’ problems
50 85 They give us a chance to explain our problems
People are now free to express themselves to the health workers
1.4 Respect for patients 25 95 There is more respect these days than was the case previously
Everybody is given due respect
Repeat Scorecard at the Chileka Health Centre, Malawi
Challenges and limitations
• Limitation of community’s role in monitoring quality:– Community can oversee attendance, requirement of
infrastructural facilities and can have general perception of quality in case of education
– Community can oversee regularity of doctors, availability of medicines and things in general in case of health
• Supply side information very crucial for the process • Scoring is not always applicable • Interface meeting can sometimes create conflict within
groups• Facilitator plays a key role- moderating skills and knowledge
about the subject matter important • Takes more time than planned
33Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
CSC – A TimelineStage/Phase Time Frame
Preparatory Groundwork 1-2 weeks
Input Tracking matrixCommunity ScorecardSelf Evaluation Score card
1-2 Half Day Meetings either on the same day or spread over 2-3 days
Preparation for Interface Meeting 1-2 weeks
Interface Meeting & Action Planning 1 Half Day Meeting
Follow up & Advocacy 1-3 months
34Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013
DiscussionCan SDS/CRC be implemented in our context?
If yes, which sector, program or service is most enabling? Give reasons
What aspects of the sector, program or service would you be interested?
How many Focus Groups will you form? Who would participate in these FGDs?
What are the key challenges or risks that you anticipate? How will you address them?
What additional skills and resources do we need to implement SDS/CRC?
35Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013