1 Day 3 Technical Sessions - 2 Social Accountability Tools Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013

1

Day 3

Technical Sessions - 2Social Accountability Tools

Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013

Tool# 3

Mystery Visitor

2Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013

The Concept

• Simple & popular tool used since 1940s

• Quantitative & Qualitative indicators

• Relies on observation techniques

• Best used for observing infrastructure, processes,

staff behaviour etc.

• Uses trained anonymous personnel


The Methodology

• Identifying service/institution – one location Vs several

• Developing indicators – what to observe?

• Choose timings – when to observe?• Choosing method of enquiry – personal visit /

telephone calls• Developing recording protocols – how to collect

and organize information• Uses ‘scores’ or ‘ratings’ to provide objective data.


An Example• Objective: Audit the service delivery processes at a

business licence issuance centre.• Indicators: - Access (time of operations); - Availability of information regarding fees, application

procedures, Processing time, complaints; - Infrastructure – cleanliness, seating availability,

availability of drinking water, toilet facilities; - Interaction – staff behavior, following queues, presence

of agents/middlemen- Developing scores


Indicator/Sub-indicator Scoring Format Actual Score

Maximum Score Possible

1. Access:1.1 Opening time

Opened on time = 10Opened with a slight delay= 5Opened after a long time = 0

5 10

1.2. Lunch Interval Followed timings = 10Did not follow timings = 0

0 10

2. Availability of information2.1 Fees

Information provided = 10No information given = 0

10 10

2.2 Procedure Information provided = 10No information given = 0

0 10

2.3 Time Information provided = 10No information given = 0

0 10

2.4 Complaints Information provided = 10No information given = 0

0 10

3. Infrastructure3.1 Cleanliness

Very clean = 10Somewhat clean = 5Dirty = 0

5 10

Total Score 20 70


DiscussionCan Mystery Visitor Audits be implemented in

our context? If yes, which service is most enabling?

If no, why not? What are the key challenges? Can an adapted form be implemented?

Quickly list out some indicators you would like to observe and record for the service you have identified


Tool# 4

COMMUNITY SCORECARDS

8 DGT - AA Myanmar Workshop, Mar 2013

What is CSC?

• A community based approach that combines key elements of:

- PRA techniques (Participation)- Public Hearings (Disclosure)- Citizen Report Card (Measurements)- Budget, Input Tracking & Infrastructure Audit

(Entitlements)


CommunityScoreCard

Participation

Disclosure

Measurements

Claims


Defining Features of a CSC

tracking of inputs or expenditures (e.g. availability of medicines)

monitoring of the quality of services/projects, generation of benchmark performance criteria

that can be used in resource allocation and budget decisions,

generating a direct feedback mechanism between service providers and community,

building local capacity and strengthening community voice and

empowerment. 11Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013

Preconditions for a CSC

• understanding of the socio-political context of governance at the local level,

• strong facilitation skills, • a strong awareness campaign to ensure

maximum participation from the community and other local stakeholders, and

• steps aimed at institutionalizing the practice for repeated actions.


CSC – The Methodology

13

I• Preparatory Phase

II• Input / Expenditure Tracking

III• Community Generated Score Card

IV• Self Evaluation by Provider

V• Interface Meeting

VI• Institutionalization


1. Preparatory Phase Step 1: Identifying & Training of Facilitators: 3 day

workshop (with a manual) with a field practical Step 2: Scoping visit to meet community

leaders/representatives Step 3: Orientation meeting with service providers. Step 4: Get basic data on the community- Population

data, Service provision data, Social & Poverty Profiles (Social Map, Well Being Ranking)

Step 5: Logistics – No. of facilitators, No. of focus groups, Timing of Interface Meeting

Step 6: Awareness Building & Mobilization15Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013

2. Input Tracking“Data on budget expenditure, physical assets, service inputs etc.”

Step 1: Decide and Obtain Information on Inputs to be Tracked- E.g., number of doctors allocated, number of para-medical staff allocated, number of beds, list of medicines at the hospital, budget – expenditure statements etc.


2. Input Tracking (Contd.)

Step 2: Share information on Entitlements (Plenary with community & provider)

Step 3: Divide gathering into focus groups

Step 4: Fill in the Input Details

Step 5: Record data in the form of an Input Matrix

Step 6: (Optional) Physical Inspection of infrastructure


A Sample Input Tracking Matrix

Dr. Gopakumar Thampi, Sanchi, April 12 2012

18

SrNo. Input

Indicator Entitlement Actual Remarks/evidence

1 Number of Doctors

2 doctors (1 full time and 1 Part time)

1 doctor (part time) Need for two doctors

2 Working Hours of Doctors

9-12 noon (morning)2-4 PM (evening)

11-1 PM (morning)

Need to followscheduled timings

3 No. of Beds 7 beds 2 beds available Absence of 5 beds

4 Availability of medicines

Prescribed list of medicines to be available at the PHC

Basic medicines are not available at the PHC

Poor Management

5 Waiting Hall A waiting hall for the patients and the person accompanying the Patient

Absence of any space for waiting

Poor Management

Performance Scorecard by Community contd..

Classify participants in focus groups • In a systematic manner, basis for classification must be

clear. Develop performance criteria • Brainstorm within the focus group to develop performance

criteria based on which the facility and services considered will be evaluated

• Based on community discussions, facilitator lists all issues mentioned and assists the group in organizing them into measurable performance indicators

• Ensure everyone’s participation in developing indicators• Finalize and prioritize 5-8 indicators


Performance Scorecard by Community contd..

Scoring the indicators• Facilitate focus groups to give relative scores for each of the

indicator. Scoring can be done either through consensus method or through individual voting . A scale of 1-5 or 1-100 is usually used – higher scores are preferred!!

Explanations for the scores• Understand the reasons behind low and high scores; helps in

explaining outliers and generates valuable information about service delivery

• Community suggestions for service improvement

• Data recording in community generated performance card


Ex. of Performance Score card by Community, Health

SNo

Performance Criteria /Indicators

Score(0-100) Reasons

1 Attitude of staff 38

1.1 Punctuality 50 Start late, but some work after hours

1.2 Polite behavior 40 Many shout at patients, rude towards children

1.3 Respect for patient 25 No respect shown

2 Management of centre 55

2.1 Cleanliness 70 Place is cleaned daily

2.2 Observing working hours 40 Open on time, staff come late, long lunch hours

3 Quality of services 16

3.1 Adequate supply of drugs 25 Drugs mostly not available

3.2 Adequate infrastructure 20 No admission room, no drinking water

3.3 Ambulance service 2One ambulance for several centres.

21

Ex. of Performance Score card by Community, School Education

SNo

Performance Criteria

Score(0-100) Reasons

Recommendations

1 Availability of Teachers 50

There are four teacher posts sanctioned, three of them are available. There is absenteeism as they come from far off place

Teachers must come regularly to school

2 Infrastructure 25No sufficient classrooms, no benches, No bathrooms (problem for girls). No water

Toilets to be constructed urgently

3 Cleanliness 30School premises are not kept clean; there is no one to sweep

Contributions can be given by the community to hire

4 Compound wall 25There is no compound wall in the school; animals are grazing

Compound wall must be made at least with dry grass and thorns

5 Communication with parents 35

Poor communication of school with parents; PTAs not held regularly

Regular PTA meetings to be held

22

4. Self Evaluation by Providers

• This evaluation is carried out by the service providers on their own performance. It brings out their own perspective of their performance.

• The assessment is based on indicators / criteria developed by the provider. The reasons for these scores are documented and become the basis for obtaining suggestions for change through greater trust and partnership, which is the objective of the process.

• It is important to understand their perspective since they sometimes work under numerous constraints.


Self-Evaluation Scorecard contd..

Orient Service Providers• The first step in developing the self evaluation assessment

card/report for providers is to orient them properly about the purpose and use of the Community Score Card.

Ensure adequate Participation• It is important to set out a time in advance to ensure adequate

participation of service providers.

Deciding on Performance Criteria*• The service providers come up with their own set of performance

criteria. Ideally, these should then be classified in a manner that is easily comparable with the indicators chosen by the community.


Provide scores for indicators and Reasons The members then need to fill in their relative scores for each of the indicators they came up with and provide reasons for the scores.

Recording DataThe data from the self-evaluation is also recorded in the form of a score card, which looks exactly like the community assessment card/report.

Suggestions for Change/ImprovementThe members are then asked about what changes or suggestions they have for improving the quality and efficiency of the services they provide.

Self Evaluation by Providers (Contd.)


Ex. Self Evaluation Score Card from Health Pilot, Vizag

SNo

Performance Criteria

Score(1-100) Reasons/Comments

Recommendations

1Doctor‘s Timings / Availability

85 Doctors are available at the PHC

-

2Staff Behaviour / Working Style

75 The PHC staff behave well with the patients

There is great work pressure therefore sometimes they are Rude

3 Medicine Dispensing 85

Medicines are available at the PHC and are given to the patients as prescribed

There is sometimes shortage of medicines.

4 Cleanliness 75 Hospital premises are kept clean

There is only one person and she is old; needs a replacement

26

The Interface Meeting and Action Planning Process

It is a public forum or meeting where the service providers and users gather in order to present their respective scorecards and discuss ways to improve service delivery.

Enables the service users to present their evaluation of the service performance, along with their concerns and priorities regarding the service.

Steps involved:

• Prepare both parties for the meeting• Ensure adequate participation from both sides• Presenting findings of both groups • Brainstorming to come up with concrete changes/improvements • Develop an Action Planning Matrix• Divide roles and responsibilities for follow-up and monitoring


Steps involved in the Interface Meeting

• Prepare both parties for the meeting• Ensure adequate participation from both sides• Community cluster meeting for services provided in more than

one village area (aggregation)• Presenting findings of both groups & input tracking matrix• Summarizing Scorecard findings*• Analysis of results in plenary discussion – Start with highlighting

common criteria and scores• Brainstorming to come up with concrete changes/improvements • Develop an Action Planning Matrix• Divide roles and responsibilities for follow-up and monitoring


Ex: of an Action Plan Matrix, Health

S. No.

Indicator Action to be taken By whom and when

1 Doctor’s Timings / Availability

Change of timings :Present: Morning: 9:00 AM to 12:00Noon, Afternoon: 3:00 PM to 5:00 PMNew: Morning: 10:00 AM to 1:00 PM, Afternoon: 4:00 PM to 6:00 PM

Doctor , immediately

2 Reporting complaints

Complaint box to be introduced: Every Saturday between 11:00 AM and 12:00 PM the complaints registered to be redressed

Doctor, Immediately

3 Staff Behaviour

Trainings to be conducted for the staff Health Department

4 Availability of medicines

List of medicines available at the PHC to be prepared and put up on the display board

Pharmacist and doctor

29

30

Satkhira, Bangladesh

Repeat Score Cards

• The scorecard process is repeated after a mutually agreed upon period of time. The repeat process is usually easier and faster since everyone has prior experience.

• The main purpose of the repeat scorecard is to review progress and provide inputs for a revised action plan by:

– Scoring the indicators again to reflect any changes in performance

– Reviewing progress related to implementing the action plan

– Discussing any changes experienced in service delivery


32

No. Indicators Scores (out of 100)

Comments and reasons

August 2002

March 2003

1.1 Punctuality of staff 50 60 They start late, but work during the lunch time and after hours

There is some improvement in time keeping

1.2 Polite behaviour 40 50 There is one health worker who shouts at us when children urinate or we sit at the wrong place

There is some improvement in staff attitude

1.3 Listening to patients’ problems

50 85 They give us a chance to explain our problems

People are now free to express themselves to the health workers

1.4 Respect for patients 25 95 There is more respect these days than was the case previously

Everybody is given due respect

Repeat Scorecard at the Chileka Health Centre, Malawi

Challenges and limitations

• Limitation of community’s role in monitoring quality:– Community can oversee attendance, requirement of

infrastructural facilities and can have general perception of quality in case of education

– Community can oversee regularity of doctors, availability of medicines and things in general in case of health

• Supply side information very crucial for the process • Scoring is not always applicable • Interface meeting can sometimes create conflict within

groups• Facilitator plays a key role- moderating skills and knowledge

about the subject matter important • Takes more time than planned


CSC – A TimelineStage/Phase Time Frame

Preparatory Groundwork 1-2 weeks

Input Tracking matrixCommunity ScorecardSelf Evaluation Score card

1-2 Half Day Meetings either on the same day or spread over 2-3 days

Preparation for Interface Meeting 1-2 weeks

Interface Meeting & Action Planning 1 Half Day Meeting

Follow up & Advocacy 1-3 months


DiscussionCan SDS/CRC be implemented in our context?

If yes, which sector, program or service is most enabling? Give reasons

What aspects of the sector, program or service would you be interested?

How many Focus Groups will you form? Who would participate in these FGDs?

What are the key challenges or risks that you anticipate? How will you address them?

What additional skills and resources do we need to implement SDS/CRC?


Documents

1 Day 3 Technical Sessions - 2 Social Accountability Tools Dr. Gopakumar Thampi. SDC/ACC Workshop. Thimphu, August 20-23, 2013