Pi STRATEGY CONSULTING EGPP Card-based Pilot Jan – Mar 2014 Session Evaluation Report April 2014

pi STRATEGY CONSULTING

EGPP Card-based PilotJan – Mar 2014 Session

Evaluation Report

April 2014

2pi STRATEGY CONSULTING

Overview


Introduction A card-based pilot was carried out from January to March

session of EGPP in two sub-districts: Bhurungamari and Dashmina

This presentation highlights the findings from an evaluation of the pilot program conducted by pi Strategy Consulting.

This evaluation is primarily based on information collected through face-to-face interviews, first hand focused observations and data provided by various stakeholders. They are:– EGPP central team – World Bank local office– Beneficiaries in two sub-districts– SARDARs in two sub-districts– PIOs of two sub-districts– Field Supervisors of two sub-districts– BCCP field trainers– Post office officials in two sub-districts


Introduction (contd.) This evaluation covers the entire geography of

the pilot program. However, due to time and budget constraints, statistically significant sampling of the beneficiaries could not be possible. Instead, purposive sampling method was applied to identify beneficiaries at the initial stages and then, respondents were chosen randomly from the list.


Methodology The preliminary questionnaire was developed by pi Strategy

Consulting. This was then validated through feedback from local World Bank team.

The updated questionnaire was subsequently tested in the field (Q-Test), and another round of edits were incorporated to arrive at the Final Questionnaire (see Appendix) for the evaluation.

The field evaluation primarily comprises three tools:1. One-to-one interview 2. FGD (Focus Group Discussion) and3. Focused Observation

As part of focused observation, pi Strategy team observed attendance and payment process in real time and identified major challenges.

BhurungamariOne-to-one Interview FGD

Beneficiary 5 1 (10 participants)SARDAR 1PIO 1

BCCP Field Trainer

1

Post office official

1

DashminaOne-to-one Interview FGD

Beneficiary 4 1 (6 participants each)SARDAR 1PIO 1BCCP Field Trainer

1

Post office official

1

Collective Sample Size: 33


Findings & Analysis


Beneficiaries were provided with 2.5h to 4h training. 30-40 beneficiaries received training at each session.

Training was perceived useful but not sufficient. Only 16% of the beneficiaries could read and recall their PIN.

No beneficiary was found who could complete the entire payment withdrawal on her own effort

Beneficiary training was not sufficient


A Personal Identification Number, or a PIN, is intended to be a confidential number. PIN is an integral part of the 2-factor authentication process that is deemed very important for security purposes, and a critical differentiator from cash transactions

Beneficiaries keep the Postal Cash Card and the PIN together. Most of the beneficiaries have the PIN written down on a piece of paper that is often pasted over the PCC. If the card is lost or stolen, the money could easily be withdrawn by anyone.

PIN was not a secret code at all

PIN was written over the card


Benefits from New Payment Mechanism

Beneficiaries in Bhurnagamari save around 28 minutes on a payment day with the new system. This was not observed in Dashmina, as with the old system Beneficiaries there received the payments from the UP members. Though the old system might be convenient for the beneficiaries but some hidden costs may be associated with it.

Beneficiaries in Bhurungamari save around BDT 7* on average during a payment day with the new system. Beneficiaries in Dashmina did not incur any cost with the previous system as they had to acquire payment from their nearby Union Member.

* excluding any unofficial fee.

Dashmina

Bhurungamari

Combined

3.5

0

3.0

3

3.2

2

0.9

5

3.5

0

2.4

8

Payment Time (in hours)

Old System New System

Dashmina

Bhurungamari

Combined

- 5 10 15 20 25 30 35

31

13

20

-

20

12

Travelling cost for collect-ing payment (IN BDT)


n=15

n=10

n=25

n=15

n=10

n=25


Beneficiaries lost PCC and/or PIN Some beneficiaries lost their

PCC and/or PIN and did not receive their payment

In Bhurungamari, both PCCs and PINs were reissued to the beneficiaries but no such step was taken in Dashmina

5 beneficiaries from Ramshagardi, Dashmina lost their PCCs and PINs, while they were crossing a river. But no PCCs and/or PINs were reissued to them

Example from Field: 1

2 beneficiaries from Joymonirhut, Bhurungamari lost their PCCs and later those were reissued to them



Problems in attendance recording In some cases, for improper

handling Sardars blocked the POS machines and no attendance was taken that day

Network failure interrupted the attendance process sometimes

A Sardar from Dashmina Union blocked the POS machine. Later he called the customer care center of the machine provider and restarted the machine


On February 26, 2014 no attendance was given at different worksites of Dashmina Union due to network failure.



PCCs were taken away from beneficiaries

In Alipur Union, Dashmina, PCCs were taken away by UP Members from the beneficiaries after one day of delivery.

PCC was taken away from Barek, a beneficiary of Alipur Union, Dashmina by the UP member. Since then no attendance or payment withdrawal was made by him


UP Members possessed all the beneficiary cards


Beneficiaries did not receive full payment

Beneficiaries in many cases did not receive full payment.

PCCs and PINs were taken away from the beneficiaries and they were also not informed about their payment amount.

Kalimullah of Alipur Union, Dashmina did not receive 10 working days payment . His PCC and PIN was also taken away from him



Inconsistency on payment day On one payment day, no

payment processing was observed in Alipur and Betagi-Shankipur Union of Dashmina.

Post office officials were present at the payment point, but because beneficiaries did not have their cards on them, no payments could be made.

After further inquiry with the local people, it was learnt that in Alipur, the UP Members sometimes obtained payments for all beneficiaries from the Post Office Officials and later disbursed the payments to the beneficiaries themselves from their own homes.

PO officials were waiting for the beneficiaries at one union office in Dashmina. It was 2.30PM and no payments were made up to that point.


Challenges and High-level Recommendations

Safekeeping the PIN and the Postal Cash Card

Recommendations

1. Small bag to keep the card could be provided that can be easily carried during work.

2. Effective training is required

Challenges

Beneficiaries have no clear knowledge about: • the amount of money

they are going to receive.

• the person to complain to if the amount received is less than he/she expected.

1. A list could be provided to Sardar populated with the current balance of the beneficiaries’ account.

2. A special-purpose POS device that only allowed balance checking could be used by beneficiaries (while waiting in queue or after receiving payment)

3. A grievance mitigation system could be incorporated.

Network Failure for an entire day causes interruption in attendance receiving process

1. There should be a dedicated channel of information for the Sardars and the FS. Through the channel they would have the access to the proper authority to address such issues of unsuccessful attendance taking process due to some authentic technical problem.

Beneficiaries have to wait for long periods on the payment day. It takes nearly 3-5 hours to receive the payment.

1. Number of POS devices per beneficiaries needs to be increased.

2. Number of payment day could be increased/staggered.

Safe

keepin

g

Lack

of

Info

rmati

on

Netw

ork

Failu

re

Wait

ing T

ime


APPENDICES1. Questionnaire (for beneficiaries and Sardars)2. Lines of Inquiry (for PIO, FS, trainers, agents)3. Responses to Capstone questions4. Responses to all other questions not already shared within

the Findings & Analysis section5. Findings from PIO/FS6. Findings from Trainers7. Findings from PO Officials8. Findings from Sardars


QUESTIONNAIRE FOR BENEFICIARIESSECTION-I: CAPSTONE DATA

1. Name

2. Beneficiary ID (from Job Card) 3. Note if she does not have Job Card (Y/N)

4. Age

5. Gender

6. Profession

7.Income (BDT) per month (outside of EGPP)

Personal: HH:

8. Expense (BDT) per month Personal: HH:

9. Savings (BDT) per month Personal: HH:

10. Medium of Savings (Informal, MFI, Bank etc.)

Personal: HH:

11. Land Ownership (in decimals)

12. Bank Account (Y/N)

1. if ‘Y’, Name of the Bank(s)

12. Do you have National ID card? (Y/N)

13. Mobile Phone Ownership (Y/N) If ‘Y’, then

a. a. Handset Modelb. Price of the Handsetc. Years of Usaged. Can you tell us your phone number from

memory? (Y/N)e. Phone Numberf. MNO providerg. Can you dial a number from your

handset? (Y/N)h. Can you read an SMS? (Y/N)i. Can you send an SMS from your

handset? (Y/N)j. Number of Handsets in HHk. Frequency of Airtime Purchase per month

(Personal and HH)l. Amount (BDT) of Airtime Purchase per

month (Personal and HH)


QUESTIONNAIRE FOR BENEFICIARIES (Contd.)SECTION-II: GENERAL SECTION

14. PCC Number:

21. How did you safe keep the PCC and/or PIN?

22. Do you keep/carry the PCC and PIN together? (Y/N)

23. Can you read your PIN? (Y/N)

24. Can you recall your PIN? (Y/N)

25. Do you know or have you heard of anyone who lost her PCC and/or PIN? (Y/N)

a) If yes, then how many of them? b) How did they lose the PCC?c) What did they do as next steps to recover the

PCC/PIN?

15. From whom did you collect the Postal Cash Card (PCC)?16. How long did it take (waiting in queue) to collect the PCC?17. Did you need to pay anyone to get the PCC? (Y/N). If Yes,

a. How much?b. To Whom?

18. What documents (KYC form, Photocopy of National ID, Picture etc.) did you need to submit to get the PCC?

19. How much did it cost you in travelling (both way) and preparing the necessary documents to get the PCC?

20. Did you participate in any training session? (Y/N). If Yes,

a) What was the length of the training session?

b) How many participants were there in one session?

c) How many instructors were there in one session?

d) What did you learn in the training session?

e) Was the training session helpful for you?


QUESTIONNAIRE FOR BENEFICIARIES (Contd.)SECTION-III: ATTENDANCE RELATED

26.How do you give the attendance in the new system? (step by step activities)

27.How long does it take (including waiting time) to confirm attendance in the new system?

28.How long did it take (including waiting time) to confirm attendance in the old system?

29.Which one do you prefer? (Old/ New)

30.Why do you prefer this?

31.How many times have you sent replacement people in the current session? (Note the total number of working days in the current session)

32.How many times did you send replacement people in the previous session?

33.Where do you keep your PCC while working in the field?

34.Do you find it cumbersome to carry the PCC with you during the work sessions? (Y/N)

SECTION-IV: PAYMENT RELATED

35.Describe a typical Payment day of EGPP pilot.

36.How do you know that your payment has arrived?

37.Do you know the current balance of your PCC account? (Y/N)

38.How do you verify (step by step activities) with this PIN while withdrawing cash from a PO branch?

39.How many times have you withdrawn wages during this session?

40.What amounts (Multiple) did you withdraw each time?

41.Was there any problem with liquidity at PO branch? (Y/N). If Yes,

a) Number of such occurrencesb) What did you do in such occasions?

42.Did you need to pay anybody to withdraw cash? (Y/N)

a) If yes, how much did you pay per transaction?

b) To whom?


SECTION-V: REMARKS

49. What challenges (if any) have you faced so far in the card-based attendance and payment system?50. What problems (if any) have you noticed with others around you in the program?51. What will be your suggestion (if any) for the improvement of the attendance verification and payment processes?

43.How long does it take to wait in queue for collecting cash in the new system?

44.How long did it take to wait in queue for collecting cash in the old system?

45.How much does it cost you in travelling (both way) to collect cash in the new system?

46.How much did it cost you in travelling (both way) to collect cash in the old system?

47.Which one do you prefer? (Old/New)

48.Why do you prefer this?

QUESTIONNAIRE FOR BENEFICIARIES (Contd.)


Line of Inquiries For FS/ PIO:

– Name – Contact Number – Designation – Role in the pilot – Challenges faced so far during EGPP Pilot – Problems with others around you in the

program – Suggestion for the improvement of the

program


Line of Inquiries (contd.) For Trainer

– Name – Location – Contact Number – Designation – A typical Training Day Description – Number of Participants in each session – Duration of each session – Challenges faced so far – Problems others faced during the program – Suggestions


Line of Inquiries (contd.) For Post Office Official

– Name – Contact number – Designation – Frequency of Transaction in PCC per week – Average amount transaction in PCC – Describe an EGPP payment day – Maximum number of EGPP beneficiaries served in one day – How much time it takes Average amount of cash out by EGPP

beneficiaries – Total amount of cash needed upfront for EGPP payment – Was having so much cash in advance a challenge – Run out of Cash Actions after running out of cash – Beneficiaries who can tell or enter PIN – Charge per transaction to EGPP beneficiary – How do you handle beneficiaries who lose PIN – Challenges faced during EGPP wage disbursement – Problems others around you have faced – Suggestion for the improvement of payment process


Q-3 Average Age varies among region Beneficiaries in Dashmina are slightly

older than beneficiaries in Bhurungamari

Dashmina

Vurungamari

0 5 10 15 20 25 30 35 40 45 50

43

33.5

Average Age

n=10

n=15


Q4- Uniform Gender Split There is an equal split between Male

and Female beneficiaries

Dashmina

Bhurungamari

Total

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

60%

53%

56%

40%

47%

44%

Gender Split

Male Female

n=15

n=10

n=25


Q 5 Professional Split Majority of beneficiaries in Dashmina are

day laborers. In Bhurungamari it’s equally split among Farmers, Housemaids and Unemployed

Dashmina Bhurungamri Total0%

10%

20%

30%

40%

50%

60%

70%

80%

Profession

Farmer Housemaid Day Laborer Unemployed

n=15n=10 n=25


Q6 Income Distribution Beneficiaries in

Bhurungamari have a 48% higher personal income on average than the beneficiaries in Dashmina

Dashmina Bhurungamari Combined 3

,350

4,9

67

4,3

20

4,6

00

7,6

00

6,4

00

Income Distribution

Average Personal IncomeAverage Household Income

Furthermore Bhurungamari Beneficiaries have about 65% higher Household Income on average compared to their Dashmina Counterparts

n=10 n=15 n=25


Q 8,9 Savings

20%

80%

Dashmina

Have Savings Don't have Savings

Beneficiaries in Bhurungamari are more likely to save than beneficiaries in Dashmina

But those who save in Dashmina save about 23% more on average than beneficiaries in Bhurungamari

67%

33%

Bhurungamri

Have Savings Don't have Savings

Those who save in Dashmina save about BDT 350 on average

Those who save in Bhurungamari save about BDT 285 on average

Only 1 respondent saves his money in a formal bank; rest save informally

n=10 n=15


Q 10 Land Ownership None of the Beneficiaries in

Dashmina have any kind of land ownership.

But 53% of Bhurungamari beneficiaries own land.

Those who own land, they own about 6.125 decimals of land on average 53%

47%

Bhurungamari Land ownership

Own Land Don't own land

n=15


Q 11,12 & 13 Bank account NID and Mobile Handset possession

8%

92%

Bank Account Posession

Have Bank AccountsDon't have Bank accounts

All of beneficiaries interviewed have a National Identification Card

Dashmina Bhurungamari

30%40%

70%60%

Handset Pos-session

Have a Handset Don't have a Handset

n=10 n=15

n=25


Q 13 Mobile Phone Ownership

8%

92%

Remember Phone Number from Memory

Yes No36%

64%

Handset Possess ion

Have Handset Don't Have Handset

Handset Model

5

2

2

Handset Model

Nokia SymphonyDon't Know

Average Phone Price BDT 1,425

n=25

n=9

n=9

n=9

Dial Number

Read SMS

Type SMS

36%

25%

0%

64%

75%

100%

MOBILE PHONE USE

Can perform Can't perform


Q15-24 PCC, Training & PIN related On average the beneficiaries have to wait

for 1 Hour and 26 minutes to collect the Postal Cash Card from Union Parishad office.

None of the beneficiaries had to pay money to acquire the Postal Cash Card

Read PIN

Recall PIN

0% 5% 10% 15% 20% 25%

Read and Recall PIN

Combined BhurungamariDashmina

All of the Beneficiaries carry the PCC and PIN with them during work

Content: how to use the Postal Cash Card, how to register the attendance, how to acquire the payment from

Post Office and the whole payment mechanism.

Training Session

Participants: 30-40Time: 2.5-4 hourTrainer: 2 BCCP Field trainer

Participants: 30-40Time: 2.5-4 hourTrainer: 2 BCCP Field trainer

Morning Evening

n=25

n=25


Q26-29 Attendance During attendance the beneficiary brings the card and the PIN and

gives them to the Sardar. The Sardar swipes the card, reads the PIN and enters the PIN himself. None of the beneficiaries enters the PIN by themselves.

Swipes Card Enters PIN Enters Ok Operation Successful

Dashmina Bhurungamari Combined

1

1.2

4

1.4

1.2

5 1.5

1.4

Attendance Time (In Minutes)

New System Old SystemAll the beneficiaries unanimously agree that they prefer the new system

n=25n=15n=10


Keep PCC and PIN

Q 30-34 Replacement and PCC

None of the beneficiaries sent replacement workers during the current session.

12% of the respondents say that they sent replacements during the previous session

Dashmina

In a polybag With Sardar Pocket

Bhurungamari

In a polybag With Sardar

Pocket Dashmina Bhurungamari

20%

67%

80%

33%

Find Carrying PCC Cumbersome

Yes No

n=10

n=10 n=15

n=15


Q 34-41 Payment process A typical payment day

The Union Parishad members informs them about the payment day

None of the beneficiaries know or can check their PCC balance

All of the beneficiaries give the card and PIN to the PO official PO official completes the transaction and delivers the money

The beneficiaries gather in front of the PO/Union Parishad office at around 9 A.M.

They need to wait 3 -5 hours on average to receive the money

When their turns comes they hand over the PCC card and the PIN to the PO officials.

PO officials swipe the card and enter the PIN in the POS machine

A thumb print is taken in a sheet against their name


Q 42- 46 Payment Time and Cost None of beneficiaries

had to pay extra to anybody to receive payments

In old system beneficiaries from Dashmina incurred zero travelling cost as UP members provided them the payments. Members collected the payments and gave it to the beneficiaries.

In old system beneficiaries used to receive payments from the union members directly in Dashmina. None of the beneficiaries in Dashmina went to a bank to receive payments. But in Bhurungamari it took them longer receive payments as they had wait in longer queues in Banks

n=15

n=15

n=10 n=25

n=10

n=25

Dashmina Bhurungamari Combined

3.5

0

3.0

3

3.2

2

0.9

5

3.5

0

2.4

8

Payment Time (in hours)

New System Old System

Dashmina

Bhurungamari

Combined

- 10.0 20.0 30.0 40.0

Travelling cost for collecting payment



Q 47-51 Preference, Challenges & ProblemsChallenges One of the major

challenge that the beneficiaries perceive is to safe-keep the PCC and the PIN

Don't have clear idea about what to do when someone misses an attendance

They receive the payments themselves.

It’s more accurateThe system is mechanized, very low probability

of manipulation

Reasons for preferring the new system

Problems Network failure caused them to sacrifice one day’s work/payment.

No attendance was given at that particular day. Waiting time during the payment day is burdensome to them.

They have to stay 3/4 hours in-front of the Post Office.

Beneficiaries unanimously agree that they prefer the new system


Findings from PIO DashminaName of the Project Implementation Officer

Sohrab Hossain

What was your role in the Pilot? 1. Overall labor selection2. Monitoring3. FS recruitment 4. Payment Monitoring 5. Sardar Selection

What challenges did you face so far implementing the Pilot?

1. Illiterate people2. New process3. Payment is given at a particular day from Union Post office

Problems with others around you in the program

Payment day problem, with long queues

What are your suggestions for improving the pilot?

1. Incentive for Upazilla & Union members 2. Solution for lost Card


Findings from FS BhurungamariName of the Project Implementation Officer

Towhidul Islam

What was your role in the Pilot? 1. Beneficiary list preparation2. Monitoring of beneficiary training3. Monitoring of Sardar training4. PCC and PIN disbursement5. Project initiation6. Overall management

What challenges did you face so far implementing the Pilot?

1. Illiterate people2. New process3. Payment is given at a particular day from Union Post office

Problems with others around you in the program

1. Lost/damaged PCC2. Lost/Damaged PIN3. Name exist in payroll but not in A/C

What are your suggestions for improving the pilot?

1. Incentive for Upazilla & Union members 2. Solution for lost Card/PIN


Findings from Field Trainers DashminaName

Muhitur Rahman Khan Designation

Program Co-ordinator, BCCPDescribe your typical Training session

On a particular day two training sessions took place one in the morning session and other in the afternoon. Beneficiaries were informed beforehand and they were given a time and a date to come. They usually came in half an before the session and stay till the end.

Number of Participants in each session?

40-45 beneficiaries

Duration of each session 3 HoursChallenges faced during training? 1. Illiterate people

2. New process3. Payment day problem


Findings from Field Trainers DashminaName

Jahidul IslamDesignation

BCCP field trainerDescribe your typical Training session

Number of Participants in each session?

30-40 beneficiaries

Duration of each session 3 HoursChallenges faced during training?


Findings from PO Official DashminaName Paresh Chandra Debnath Designation Post master Frequency of Transaction in PCC per week (other than EGPP Payments)

0

Average amount transaction in PCC (other than EGPP Payments)

0

Describe an EGPP payment day Payment start at 9:00 am and runs until 5:00pm, two employees from Patuakhali office come to assist. Lines are formed outside of the post office, then one by one each beneficiary is let in. Usually we use 1 POS machine but whenever there is huge line outside the second POS machine is also used. For regional deployment, 2 POS machines each are sent from Dashmina office to 5 other unions. The POS machines are accompanied by two representatives from the Patukhali office.

Maximum number of EGPP beneficiaries served in one day

623

How much time it takes 5 minutes for each beneficiaryAverage amount of cash out by EGPP beneficiaries 1,000 BDT

Total amount of cash needed upfront tfor EGPP payment

26,14,800 BDT

Was having so much cash in advance a challenge No

Run out of Cash No Actions after running out of casjh -Benefeciaries who can tell or enter PIN 0%

Charge per transaction to EGPP beneficiary 0

How do you handle beneficiaries who lose PIN Ask them to apply for replacement through proper channels

Challenges faced during EGPP wage disbursement -

Problems others around you have faced -Suggestion for the improvement of payment process -


Findings from PO Official BhurungamariName Md. Ashraf AliDesignation Post Master Frequency of Transaction in PCC per week (other than EGPP Payments)

0

Average amount transaction in PCC (other than EGPP Payments) 0

Describe an EGPP payment day

Maximum number of EGPP beneficiaries served in one day

200How much time it takes 5 minutes for each beneficiary

Average amount of cash out by EGPP beneficiaries 1,000 BDT

Total amount of cash needed upfront tfor EGPP payment -

Was having so much cash in advance a challenge No

Run out of Cash No Actions after running out of casjh -

Benefeciaries who can tell or enter PIN 0%

Charge per transaction to EGPP beneficiary 0

How do you handle beneficiaries who lose PIN -

Challenges faced during EGPP wage disbursement -

Problems others around you have faced -

Suggestion for the improvement of payment process -


Findings from Sardar DashminaName MD. NasirAge 50Gender MaleEducation Class 3Income 5,000Household size 5Household Income 10,000Profession of Hosehold Head FarmerLand ownership NoneOther Government Grant -Mobile -EGPP Mobile -Describe a day of EGPP attendance Gets at the site 8:45. starts taking attendance from

9:00 am. Takes the card and PIN from each beneficiary, then swipes the card and imputs the PIN.

How many workers work under your supervision 75Card based system allow for better monitoring YesReceive training on how to use POS machine YesSatisfaction level on the training progran It was really helpfulAverage attendence cards swiped with the POS machine each day

70

Average length of attendance procedure 1 hour Average length of attendance procedure with previous paper based system

3-4 hours

Average mistakes while entering attendance codes of beneficiaries in a single day

-

Experience of network failure Yes Frequency of network failure 2 out of 35 days Challenges you faced during EGPP pilotProblems others faced around you in th eprogram Suggestions for the improvement of the program '

Documents

Pi STRATEGY CONSULTING EGPP Card-based Pilot Jan – Mar 2014 Session Evaluation Report April 2014