Client Satisfaction Survey

what’s possible

Belgrade, July 2010

Report on the findings of the CSAI Success Grants Applicants’ Satisfaction with the Grant Application Process and Procedures

• Introduction

• Research Objectives

• Key Findings

• Analysis

Content

ISC/CSAI announced an RFA for its Success Grants Program on March 17, 2010. The purpose of this RFA was to solicit project proposals from CSOs operating in Serbia for the purpose of using mass media tools and new technologies to successfully promote their results and impacts in order to increase the overall visibility of CSOs.

In the second halh of May 2010, ISC/CSAI organized an on-line survey to find out what Success Grants Applicants think and feel about CSAI’s application process and procedures. The main purpose of the survey was to collect feedback in order to change and adjust future CSAI Requests for Applications.

The survey was anonymous; respondents did not have to leave any records regarding their organizations or themselves. Also - as it was noted to respondents - their participation in the survey, as well as their opinions and attitudes, did not in any way affected the evaluation of their project proposals.

Introduction/Background

• Method of data collection: Internet Based Questionnaire

• Population: 111 CSAI Success Grants Applicants

• Targeted Sample: All Applicants

• Sample Size: N=102 (92% of the Total Population)

• Data collection was conducted from May 10 – 27, 2010

Introduction/Methodology

• The objective of this research was to collect and

analyze applicants’ feedback regarding:

1. Manner of RFA announcement and channels

used;

2. Application Documents (including ISC

Application Form and ISC Budget Form);

3. RFA timelines;

4. Project Proposal method of submission to ISC;

5. General attitudes about the Application process

and procedures.

Research Objectives

How would you rate the overall application process?

Key Findings/Overall Satisfaction with the Application Process

It is fair to say that almost all respondents are satisfied with the Application Process in general, with 46% of respondents stating Absolutely satisfied, and 44% stating Satisfied.

Base: N=100

4.36

1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00

AverageScore

Average Score (on a scale of 1- 5)

46%

10%

44%

Very unsatisfiedUnsetisfiedIndifferentSatisfiedAbsolutely satisfied

Key Findings/Information (Re)sources

12%

13%

17%

21%

0% 5% 10% 15% 20% 25%

CRNPS’ mailinglist

ISC mailing list

Other CSOs

ISC website

8%

1%

2%

2%

3%

3%

3%

5%

6%

6%

0% 3% 6% 9%

Other way

Media Center website

ISC Facebook Page

Vojvodina Agency for Development ofNonprofit Sector website

BCIF mailing list

CDNPS website

Media Center mailing list

BCIF website

Civic Initiatives website

Vojvodina Agency for Development ofNonprofit Sector mailing list

Where did you first find out about ISC/CSAI’s Success Grants RFA?

Top Channels

Other Channels

ISC Serbia website, ISC mailing list and CDNPS mailing list seem to be the most valuable sources of information for CSO’s interested in ISC/CSAI RFAs. Receiving inputs from other CSO’s regarding funding opportunities seems to be another way of acquiring the information.

Base: N=102

Key Findings/RFA Narrative

3.44

1.00 1.50 2.00 2.50 3.00 3.50 4.00

Average Score

Did the Narrative Section of the RFA clearly articulated what kind of projects ISC/CSAI intends to support trough the Success Grants

Program?

Average Rate (on a 1-4 rating scale)

The Success Grants RFA Narrative was highly rated among all respondents, with average rate of 3.44, on a 1 -4 scale. Half of the respondents consider the RFA Narrative to be Absolutely Clear.

Base: N=101

45%50%

Not clear at allSomewhat clearMostly clearAbsolutely clear

6%

Key Findings/YouTube video

3.42

1.00 1.50 2.00 2.50 3.00 3.50 4.00

Average Score

How did you feel about having the option of using YouTube videos as part of the application?

Average Score (on a scale of 1- 4)

Submitting YouTube videos as part of the application for the Success Grants RFA was highly rated amongst a vast majority of respondents, with the average scores of 3.42 on a 1-4 point scale. More than half of the respondents Liked very much that option.

34%

11%

55%

1%

Did not liked it at allDid not liked itSomewhat liked itLiked it very much

Base: N=101

Key Findings/Application in English

How difficult was it to complete the application in English?

Completing the application in English was easy for the majority of respondents. Only 3% of respondents found it to be somewhat difficult.

56%

16%3%

25%

DifficultSomewhat difficult Not difficult, nor easySomewhat easyEasy

Base: N=101

Key Findings/Relevance of ISC Application Form Questions

Where questions in different sections of the ISC Application Form relevant?

More than 90% of respondents feel questions in the Application Form were Absolutely Relevant or Relevant. Seems that follow-up activities and Org’s experience in media campaigns caused some minor misunderstandings.

Base: N=101

1%

1%

1%

1%

1%

5%

2%

5%

4%

4%

4%

3%

1%

1%

30%

20%

34%

29%

24%

15%

22%

14%

21%

64%

78%

60%

66%

75%

80%

74%

85%

75%

0 10 20 30 40 50 60 70 80 90 100

Justification and Background

Project Goal and Objectives

Expected Results

Target Groups

Description of Activities

Management Team

Follow up Activities

Brief description of organization

Org’s experience in media campaigns

Not relevant at all Not relevant / nor / Relevant Absolutely relevant

96%

99%

96%

95%

99%

95%

94%

98%

94%

RELEVANCE

Key Findings/Budget Form I

How difficult was it to complete the ISC Budget Form?

More than two thirds (77%) of respondents didn’t find it difficult to complete the budget form, in general. However, some respondents had some difficulties regarding different features of the ISC Budget Form. Classification of the findings can be found on the next slide.

Base: N=101

39%18%

5%

38%

1%

Very DifficultDifficult/ nor /Not difficultNot difficult at all

Key Findings/Budget Form II

Total number of comments: 18

Positive: 2 comments

Negative:16 comments

Defining andUnderstandingBudget Lines 7 Comments

UnderstandingCost Share:2 Comments

Time Constraint:1 Comment

Language:1 Comment

General Looks of the Budget Form:1 Comment

Miscellaneous:4 Comments

Base: N=18

Was there anything particularly difficult about the ISC Budget Form?

Key Findings/Budget Instructions I

Did the Budget Instructions provided clear directions as to how ISC funds should be used by grantees?

Half of the respondents responded that the budget instruction were Mostly Clear. In general, 88% of the respondents feel that the budget instructions were clear. However, a number of respondents had some difficulties to clearly understand specific Budget Instructions. Classification of these difficulties are presented on the next slide.

Base: N=101

50%

17%33%

Not clear at allSomewhat clearMostly clearAbsolutely clear

Key Findings/Budget Instructions II

Total number of comments: 13

Positive: 1 comments

Negative:12 comments

Ratio between project and Other costs:6 Comments

Unclear Budget Categories:2 Comments

InstructionsNeed AdditionalClarifications:2 Comments

The Way Budget Categories are

Calculated:1 Comment

Definition of theIn Kind:

1 Comment

Base: N=13

Where there any specific budget instructions you found hard to understand?

Key Findings/Time

Was the 1-month timeframe sufficient to complete the application?

More than three quarters of respondents had enough time (6% had more than enough) to complete the application. However, a significant number – almost one quarter of all respondents - needed more than a month, mostly because it took time to reach agreements with media outlets (key partners in this program).

72%

22%6%

Not enough timeCould have been more timeQuite enough timeMore time than neede

Base: N=101

How does your organization prefer to submit applications?

Key Findings/Application Submission

Using e-mail is the most preferred way to submit project proposals.

Base: N=100

1.88

2.22

2.73

3.77

1 1.5 2 2.5 3 3.5 4

By electronicmail

On line

By land mail

Hand-delivering

Average Score (on a scale of 1- 4)

A high response rate to the survey is the first “result” that merits attention, and safely allows for conclusions to be drawn. The fact that 102 organizations, out of 111 (92%), took part in the survey, leads to the conclusion that CSOs are very interested and motivated to participate in these kinds of surveys and that they appreciate the fact that somebody (no other than donor organization) wants to hear their opinions and attitudes. Like one of the respondents noticed:

“It’s nice of you to ask about these kinds of things. I have a feeling that most of the donors are not doing this (client satisfaction surveys), which leaves their application procedures non-transparent for those using them.”

It can be argued that organizations felt pressured to take part in the survey, given that the survey was organized during the project proposal evaluation process, despite the fact that they were, on several occasions, assured that the survey was anonymous; and that their participation in the survey, as well as their opinions and attitudes expressed within, would in no way affect the evaluation of their proposal. This “pressure”, and the general type of relationship existing between donors and CSOs is something that should be taken in consideration when interpreting results.

Also, this high response rate allows us to safely make conclusions in regards to the entire population.

Results Analysis

Overall satisfaction:

Having 90% of all respondents satisfied with the overall Application Process (46% of respondents feeling Absolutely satisfied; and 44% feeling Satisfied) is an impressive achievement.

Application Form:

More than 90% of respondents felt that the questions in the Application Form sections (Justification and Background, Project Goal and Objectives, Expected Results, Target Groups, Description of Activities, Management Team, Follow up Activities, Brief description of organization, Organization’s experience in media campaigns) were Absolutely Relevant (score ranges from 64% to 85%, depending on the section) and/or Relevant (14% to 34%). It seems that only follow-up activities and organization’s experience in media campaigns caused some minor misunderstandings.

RFA Announcment:

The Success Grants RFA Narrative was highly rated by all respondents, with an average scores of 3.44 (on a scale of 1 – 4). One half of the respondents consider the RFA Narrative to be Absolutely Clear.

Results Analysis

Budget Form & Instructions:

The budget form did not cause too many difficulties to respondents, at least according to the survey results. However, compared to the Application form, some features caused misunderstandings or where not clear enough.

“It was not elaborated enough how detailed budget items should be, so we used our own judgment; It would be good to adjust budget form to the specific media production costs, in order to make it clearer; It was not clear which costs go under which budget line. We were not able to fully understand the form without additional consultations with ISC staff; We were not sure whether honoraria for consultants we assigned for the project activities, should be placed in budget line no.2 or no 9, etc.”

Exactly half of all respondents think that the budget instructions were Mostly Clear. In general, 88% of the respondents feel that the budget instruction were clear. However, some respondents had difficulties to clearly understand specific budget instructions:

“It was mostly clear in general, but it was not clear and precisely defined what is air time, does it refer to TV advertisement only, or to general advertisement trough official channels, the usual channels of media communication; In the budget instructions it is asked for a reasonable ratio between administrative and program costs, and the preference will be given to those proposals that do not exceed 40% of administrative costs compared to program costs. It is not logical to have personal salaries as administrative costs. Not clear!!!-Confusing!!!; The ratio between project and non-project costs is not in accordance with the needs of the project, etc.

Results Analysis

Getting the word out:

The ISC Serbia website, ISC mailing list and CDNPS mailing list seem to be the most valuable sources of information for CSOs interested in ISC/CSAI RFA, with 21%, 13% and 12% respectively. Receiving inputs from other CSO’s regarding funding opportunities seems to be another effective way of acquiring the appropriate information (17% of organizations stated that other organizations were their primary source). It is safe to assume that distribution channels amongst organizations which served as primary sources of information was similar to distribution amongst survey participants.

Results Analysis

Thank You