CMHDARN Evaluation-ReportFINALFeb2015

February 2015

www.cmhdaresearchnetwork.com.au

© 2015 The Mental Health Coordinating Council (MHCC) and the

Network of Alcohol and Other Drug Agencies (NADA)

Report author:

Aldo Spina - Evaluation Consultant

E: [email protected]

www.mhcc.org.au www.nada.org.au www.nswmentalhealthcommission.com.au

http://www.cmhdaresearchnetwork.com.au/

mailto:[email protected]

http://www.mhcc.org.au/

http://www.nada.org.au/

http://www.nswmentalhealthcommission.com.au/

CONTENTS

EXECUTIVE SUMMARY 1

1. INTRODUCTION 4

2. EVALUATION METHODS 6

3. EVALUATION FINDINGS 7

3.1 Overview of survey respondents 7

3.2 Research in respondents’ workplaces 8

3.3 CMHDARN membership and awareness 9

3.4 Project Reference Group 11

3.5 Research forums 12

3.6. Reflective practice webinars 14

3.7. Community Research Mentoring Project 15

3.8. Research Seeding Grants Program 16

3.9. CMHDARN website 17

3.10 E-Newsletter 18

4. IMPACT ON KNOWLEDGE, SELF-EFFICACY AND RESEARCH CAPACITY 19

4.1 Purpose and relevance 19

4.2 Impact on knowledge and self-efficacy 22

4.3 Impact on research capacity 24

4.4 Building future research capacity 25

4.5 Challenges in building research capacity 26

4.6 Discussion of Research Capacity Building Framework 27

5. CONCLUSIONS 29

5.1 Overall findings 29

5.2 Usefulness of CMHDARN activities 30

5.3 Planning, implementation and communication 31

5.4 Measuring impact over time 31

5.5 Next steps for CMHDARN 32

6. RECOMMENDATIONS 34

APPENDIX 36

| 1

Community Mental Health Drug and Alcohol Research Network Evaluation

EXECUTIVE SUMMARY

The Community Mental Health Drug and Alcohol Research Network (CMHDARN) was established

by the Mental Health Coordinating Council (MHCC) and the Network of Alcohol and other Drugs

Agency (NADA) in 2010. Its aim is to a) broaden involvement of the community mental health

and drug and alcohol sector in practice-based research, and b) promote the value of research

and the use of research evidence in practice, and to improve service delivery to people

affected by drug and alcohol and mental health issues.

This evaluation has found CMHDARN is effective in working towards its goals. It has led to

increased self-efficacy among participants of CMHDARN activities. For example, 66 per cent of

survey respondents report greater confidence to use research findings. CMHDARN has also led to

increased knowledge of drug and alcohol, mental health and co-existing issues. For example, 59

per cent of survey respondents indicated that it has improved their understanding of co-existing

mental health and drug and alcohol issues.

There was very good feedback on the variety of activities that have been implemented by

CMHDARN. All these activities evaluated well. This activity (period December 2010 - November

2014) has included:

o 11 research forums with a total of 493 participants;

o 7 webinars with 257 participants;

o 7 newsletters distributed;

o 240 CMHDARN members; and

o 1,437 NSW visitors to the website with 11,063 website page views.

There remains an ongoing need for CMHDARN. Evaluation participants believe that CMHDARN’s

goals are very relevant to individuals and organisations working in mental health and drug

alcohol.

One area requiring further exploration is to how to better engage senior management in the

project. Apart from a focus on individual skill development, management need to be supported

to identify how they can better prioritise and nurture engagement with research in their

organisations.

An important issue that arose during the evaluation is the funding required to support more

practice-based research in community managed organisations. The Mental Health Commission

of NSW has recognised in its Strategic Plan for Mental Health in NSW the importance of translating

research into practice. This evaluation, as also noted in the evaluation of the Research Seeding

| 2


Grants Program and NGO Mental Health and Drug and Alcohol Research Grants Program,

acknowledges that research funding is required if community managed organisation are to

better support practice-based research.

Finally, the extent to which research capacity building is achieved does depend on the level of

participation and engagement from community managed organisations. Among those who

participated in CMHDARN activities, the feedback is overwhelming that it has had a positive

impact on building capacity at an individual and organisation level. Future evaluations of

CMHDARN should use some of the measures that have been used within this report to monitor

changes over time in individual and organisation research capacity.

Recommendations

1. CMHDARN should continue to focus on developing the research capacity of community

managed organisations, promoting the value of research and the use of research evidence

in practice.

2. CMHDARN should continue the variety of activities it is currently undertaking as they are

useful and are having an impact on skills and self-efficacy, this includes webinars, research

forums, and mentoring.

3. CMHDARN should continue with the current format of research forums (presentations which

are followed by guided small group discussions) as they are working well to promote learning

and sharing of experiences.

4. Given the usefulness of the Research Seeding Grants, the identification of the importance of

funding among evaluation participants, recommendations from the Research Seeding

Grants Program and recommendation from the Final Report on the NGO Mental Health &

Drug and Alcohol Research Grants Program, it is recommended that CMHDARN approach

funders to discuss research funding for community managed organisations to support

practice-based research.

5. CMHDARN should continue to maintain and update the CMHDARN website, further

enhancing its technical functions by enabling online registration for research forums and

more prominently featuring the option to subscribe to the email list.

| 3


6. CMHDARN should move to a subscription email list (rather than the current member model)

as it makes it easier for interested individuals to subscribe and unsubscribe from receiving

emails from CMHDARN.

7. In addition to the focus on knowledge and skills development, and given most workplaces

do not appear to have dedicated research and evaluation teams or positions, CMHDARN

needs to consider how it can support and engage management in community managed

organisations to better promote, value and support research and an evaluation culture

within their organisations.

8. CMHDARN should consider the implications of this report and amend its draft Evaluation

Framework to reflect the following as priority indicators for its implementation, with a view to

evaluating the program in two years’ time:

i. understanding of drug and alcohol and mental health issues

ii. confidence to use research findings in your work

iii. confidence in undertaking practice-based research

iv. impact on individual research capacity

v. impact on organisational research capacity.

9. A future evaluation of CHMDARN should run focus groups with senior managers of

community managed organisations to understand how they perceive the role of research in

their organisations and how CMHDARN can support this role.

10. CMHDARN has promoted the role of consumers/clients/carers in research and should

continue to emphasise the importance of this role.

| 4


1. INTRODUCTION

The Community Mental Health Drug and Alcohol Research Network (CMHDARN) was established

in 2010 by the Mental Health Coordinating Council (MHCC) and the Network of Alcohol and

other Drugs Agency (NADA). The MHCC and NADA are the NSW state-wide peak bodies which

represent community managed organisations in the mental health and alcohol and other drugs

sectors, respectively.

The CMHDARN was established in 2010 to:

broaden involvement of the community mental health and drug and alcohol sector in

practice-based research, and

promote the value of research and the use of research evidence in practice, and to

improve service delivery to people affected by drug and alcohol and mental health

issues

Following three years of funding for the CMHDARN from the NSW Ministry of Health, the NSW

Mental Health Commission provided funding for an initial twelve month period from December

2013 to November 2014, and has continued this partnership into 2015.

CMHDARN has implemented a range of education activities to build research capacity

including research forums, reflective practice webinars, mentoring, research seeding grants,

newsletters and a website. These are further described within the report.

The release of Living Well: A Strategic Plan for Mental Health in NSW 2014-2024 by the Mental

Health Commission of NSW identifies the importance of developing a Research Framework for

Mental Health in NSW (at the time of the evaluation it was subject to consultation). The

framework confirms that the overarching aims of NSW-funded mental health research are to:

improve the lived experience of those with a mental illness and keep people well in the

community;

accelerate the translation of research into practice; and

support and attract a robust research capacity and infrastructure for NSW.

| 5


CMHDARN can make an important contribution to

achieving these aims, particularly given its focus on

supporting practice-based research and the use of

research evidence. In fact, CMHDARN has already

been recognised for its contribution to research and

work across the sectors. In 2014, the CMHDARN was a

joint winner of the inaugural Tom Trauer Research and

Evaluation Award in recognition of its contribution to

research and evaluation. In 2013, CMHDARN received

a NSW Mental Health Matters commendation award in

2013 in the category of ‘Cross Sector Collaboration’.

Informing the work of CMHDARN is a framework for research

capacity building that outlines six principles:*

building skills and confidence;

ensuring the research is 'close to practice';

developing linkages and partnerships;

developing appropriate dissemination;

building elements of sustainability and continuity; and

investments in infrastructure.

Figure 1: Research Capacity Building Framework adapted to CMHDARN from Cooke, J

The evaluation has attempted to explore progress across all six principles and identify where

further activity is required.

Since 2010, a steering committee, consisting of representatives from MHCC and NADA, has

overseen the implementation of the project with a focus on ensuring it meets its deliverables and

reporting requirements. A Project Reference Group, consisting of researchers, consumers and

practitioners from the community sector and academic researchers, was also established to

advise and strategically guide the work of the research network.

* Cooke J. A framework to evaluate research capacity building in health care. BMC Family Practice 2005,

6:44

Tina Smith and Bradley Foxlewin (a member of

the Project Reference Group, accepting the

Tom Trauer award on behalf of CMHDARN.

| 6


2. EVALUATION METHODS

The evaluation occurred throughout November-December 2014. An independent evaluation

consultant was contracted to undertake the evaluation. There are four main evaluation

methods:

Online survey: An invitation to complete an online survey was emailed to CMHDARN

members (240 members) and attendees of network events (201 attendees). A link was

posted on the CMHDARN website and promoted through both the MHCC and NADA

websites and communications. A reminder email was sent to members approximately

one week after the initial invitation email was sent. (See Appendix A for survey)

Analysis of project activity: An analysis of all project activity was undertaken. This

included reviewing project activity reports and prior evaluation reports. An analysis was

also undertaken of website statistics.

Focus group: Following the ‘Navigating Research Ethics’ Forum on 2 December 2014, a

focus group was held with interested participants from the forum. Nine people

participated in the focus group. The focus group ran for approximately 50 minutes. The

session was facilitated by the consultant.

Key informant interviews: All members of the Project Reference Group were invited to

contact the evaluation consultant if they wished to participate in a short interview. They

were also reminded that they could input into the evaluation by completing the online

survey. Ten current or former members of the reference group completed the survey. This

was supplemented by three telephone interviews. The interviews ran for between 15-30

minutes.

Limitations of the evaluation

The online survey was forwarded to CMHDARN members or attendees of CMHDARN events. This

means those who participated in the evaluation generally already have some degree of

familiarity and involvement with CMHDARN. There may have been a small number of other

people who accessed the survey through the websites where the link was placed. The

evaluation is unable to assess how MHCC or NADA members who have not participated in any

events or are not CMHDARN members value its purpose and view its work. It is important to bear

in mind that there were 65 survey respondents, based on the number of email invitations that

were sent out; this represents only a 15 per cent survey uptake rate.

| 7


3. EVALUATION FINDINGS

3.1 Overview of survey respondents

Sixty-five individuals responded to the survey. Among the 65 survey respondents, the

overwhelming majority (83 per cent) work for a non-government organisation. Forty-two per cent

of all survey respondents work for an MHCC or NADA member organisation. Of survey

respondents from another non-government organisation, four are also CMHDARN members.

Three respondents are consumers, clients or carers; all three respondents also indicated that they

work for a non-government organisation. Twenty per cent of respondents were academic

researchers. Their answers have been excluded from parts of the analysis given some questions

are not relevant to their work.

Approximately a third of respondents are working for small size organisations (25 employees or

less), a third for medium size organisations (26-100 employees), and a third are working for large

size organisations (over 100 employees).

Table 1. What type of organisation do you work for? (Tick all that apply) (n=65)

Answer Options % No.

MHCC member organisation 13.8% 9

NADA member organisation 27.7% 18

Other non-government organisation 41.5% 27

I work as an academic researcher 20.0% 13

I'm a consumer or client or carer 4.6% 3

Other 4.6% 3

Table 2. What is the approximate size of your organisation? (n=65)


10 or under employees 16.9% 11

11-25 employees 13.8% 9

26-100 employees 33.8% 22

Over 100 employees 33.8% 22

Not applicable 1.5% 1

| 8


3.2 Research in respondents’ workplaces

Survey respondents were asked a set of questions about research within their workplace to help

us better understand their research capacity. Respondents who are researchers are excluded

from the analysis below.

Using research

Eighty per cent of respondents indicated that they frequently or somewhat frequently use

research findings to inform their work. Only 12 per cent indicated that they somewhat

infrequently or infrequently use research findings to inform their work. Furthermore, when asked

how frequently they participate in the implementation of research, 67 per cent indicated

frequently or somewhat frequently. This time 17 per cent indicated that they somewhat

infrequently or infrequently participate in the implementation of research within their workplace.

It should be noted that those who are most likely to engage with CMHDARN are probably those

who are already interested in research or engaged with research in their workplace.

Table 3. How frequently or infrequently do you use research findings to inform your work?

(n=49)


Frequently 46.9% 23

Somewhat frequently 32.7% 16

Neither frequently nor infrequently 8.2% 4

Somewhat infrequently 10.2% 5

Infrequently 2.0% 1


Table 4. How frequently or infrequently do you participate in the implementation of research

within your workplace? (n=48)


Frequently 22.9% 11

Somewhat frequently 43.8% 21

Neither frequently nor infrequently 12.5% 6

Somewhat infrequently 8.3% 4

Infrequently 8.3% 4


Accessing research

The two most popular ways for respondents to access research is to either access the relevant

research themselves or to attend an event where research findings are presented, such as a

forum or conference. Among the 20 per cent who chose ‘other’, they generally commented

that they are studying or conducting their own research.

| 9


Table 5. How do you mostly access research related to your work? (Tick all that apply) (n=49)


My organisation provides me with information about

relevant research 42.9% 21

I access relevant research myself (such as via journals,

websites, e-newsletters) 83.7% 41

I attend forums, seminars, workshops, and conferences 83.7% 41

Other (please specify): 20.4% 10

Internal research and evaluation support

Within their organisations, almost half of all respondents indicated that there is no dedicated

research or evaluation team or even a dedicated position. While 18 per cent indicated that they

have a research or evaluation team, another 18 per cent indicated that they have one position

that undertakes research. Among those who selected ‘other’, a few commented that “various

staff do this on an ad hoc basis” and another commented that “we have a lot of people

interested in it”.

Table 6. How would you best characterise the research and evaluation support in your

organisation? (n=49)


We have a research or evaluation team 18.4% 9

We have one staff person who undertakes research or

evaluation 18.4% 9

We have no specific position which undertakes research

and evaluation work 44.9% 22


Other (please specify) 12.2% 6

3.3 CMHDARN membership and awareness

Eighty-six per cent of survey respondents had heard of CMHDARN prior to the survey. This is not

surprising given that respondents were recruited to the survey because they are members of the

network, had attended an event, visited the website or related MHCC/ NADA websites.

Respondents who had heard of CMHDARN were asked how familiar they are with the activities

of the network. Twenty-eight per cent indicated that they are familiar, while 54 per cent

indicated that they are somewhat familiar. However, this left approximately 19 per cent of survey

respondents who are more neutral in their responses, somewhat unfamiliar or unfamiliar with the

work of the network.

| 10


Table 7. You indicated that you have heard of the CMHDARN. How familiar or unfamiliar are

you with the activities of CMHDARN? (n=54)


Familiar 27.8% 15

Somewhat familiar 53.7% 29

Neither familiar nor unfamiliar 9.3% 5

Somewhat unfamiliar 7.4% 4

Unfamiliar 1.9% 1

CMHDARN Membership

Individuals can become members of CMHDARN by completing an online form or via a hard

copy membership form. There is no cost to become a member. By becoming a member you

receive notifications of CMHDARN activities and events. At the time of the evaluation, there

were 240 members. Approximately a third of survey respondents are members of CMHDARN,

although it may be higher as approximately another third were unsure if they are members.

Forty per cent of respondents indicated that they are not members.

Figure 2: Are you a CMHDARN member?

Two-thirds of CMHDARN members indicated that it was useful to be a member. While another 22

per cent indicated it was somewhat useful. Two respondents were more neutral in their

responses, one of these respondents indicated that they are new and have “not yet had a

chance to decide its usefulness”.

Table 8. You indicated that you are a member of the CMHDARN. Have you found it useful or

not useful to be a member of the network? (n=18)


Useful 66.6% 12

Somewhat useful 22.2% 4

Neither useful nor not useful 11.1% 2

Somewhat not useful 0.0% 0

Not useful 0.0% 0

30.20%

39.70%

30.20%

CMHDARN Member?

Yes, I am a member No, I am not a member Unsure

| 11


3.4 Project Reference Group

Ten survey respondents are or have been members of the Project Reference Group. Five of these

respondents have found the Project Reference Group useful and another four have found it

somewhat useful.

Key informants commented that the Project Reference Group was working well and was making

a useful contribution to the work of the network. Two survey respondents commented on how it

has been useful.

“The events and activities that have grown out of the PRG have been excellent

and a great example of the collaborative nature of CMHDARN.”

“Collective planning and prioritising”

A survey respondent wondered whether these types of committees are generally useful, though

they did believe the network itself is doing “useful stuff”.

“Always wonder about how much committees achieve, but the network is doing

useful stuff so maybe the committee is working.”

Table 9. You indicated that you are (or were) on the Project Reference Group. Do you feel

that the Project Reference Group has made a useful or not useful contribution to the network?

(n10)


Useful 50.0% 5




Not useful 0.0% 0

| 12


3.5 Research forums

Research forums aim to facilitate research capacity building and build understanding about

research in the community mental health and drug and alcohol sectors. Eleven forums have

been held, with a total of approximately 493 participants. Following each forum, the agenda,

presentations, the evaluation and other relevant resources are made available on the website.

Table 10. Research Forums

Year Focus No.

attendees

2014 Navigating research ethics 16

Understanding best practice research when working with Aboriginal

and Torres Strait Islander organisations and people

46

Strategies for building research capacity in your organisation (Held in

Ballina)

27

Fundamentals of research 43

2013 Building your evidence-base – Lessons from the field 68+

Realising research in rural areas (Held in Orange) 55+

Sharing our learning – A research forum for the recipients of Research

Seeding Grants

18+

2012 Ethics in Research – they’re more important than you think! 39+

Nothing About us, Without Us - We are the Evidence: Consumer

representation and participant in research

47+

Seeding Grants Program Launch: From ideas to action – developing

your research proposal

59+

2011 Research and evaluation in community managed mental health and

drug and alcohol sectors

75+

+ Number of registrations

Forty-two per cent of survey respondents have participated in a research forum. On average

respondents attended two forums. When asked if these research forums have been useful, 76 per

cent of respondents indicated that the forums have been useful. While another 20 per cent

indicated that they were somewhat useful (see figure 3).

“forums have been excellent”

| 13


Figure 3: Usefulness of CMHDARN membership

Forum ratings

At the end of each forum participants are asked to complete an evaluation form. Participants

could rate the forum on a five point scale. Participants rated the forums very highly with over 90

per cent (using a sample of four forums – see Table 13) rating the forum that they attended as

excellent or very good.

Table 11. Research forum evaluation feedback

No.

evaluations

Excellent or

Very good

Understanding best practice research when working with

Aboriginal and Torres Strait Islander organisations and

people

46 96%

Strategies for building research capacity in your

organisation

27 100%

Fundamentals of research 43 95%

Building your evidence base – Lessons from the field 45 90%

Focus group participants stated that they found the forum that they had just attended

(Navigating Research Ethics) informative and helpful.

“Presentations were very good”

“Getting a lot of ideas of things I can take back to my place”

“New information”

76

20

4

0

Useful Somewhat useful Neither useful or not useful Not useful

| 14


There was also agreement among focus group participants that the format of the forum,

presentations followed by small group work exploring a case study, worked really well to support

learning.

“You never learn anything if everyone is just sitting their listening to people”

They also appreciated that the forums were free to participants as they thought this most likely

aided individuals being able to attend the forum.

3.6. Reflective practice webinars

Reflective practice webinars aim to promote knowledge

and understanding about recent research. The webinars

commenced in 2012. One of the intentions of the webinars is to

make the work of the network more accessible to interested

parties in rural and regional areas. Each webinar runs for 60

minutes. The forums are facilitated by Associate Professor

Katherine Mills from the National Drug and Alcohol Research

Centre. Seven webinars have been held by the CMHDARN

with approximately 257 participants.

Table 12. Reflective Practice Forums and attendance

Year Title No.

participants

2014 Integrated psychological treatment addressing co-existing alcohol

misuse and depression

34

Improving organisational capacity and demonstrating efficacy 20

Analysing journal articles - key questions to ask 11

2013 Designing evidence-based treatments for young people with complex

needs

63

Incorporating research findings into service delivery – an introduction to

implementation science

42

2012 Substance use among consumers of a community managed mental

health service

43

Stigma and discrimination towards people with drug and alcohol and

mental health issues

44

Number of participants is based on registrations only – in some workplaces more people have watched

the webinars than registered.

http://www.cmhdaresearchnetwork.com.au/research-network-activities/reflective-practice-forums/

| 15


Sixteen survey respondents have participated (or later viewed or listened to) in a webinar. They

participated in an average of two webinars. Eighty per cent of respondents who attended a

webinar found them useful, 13 per cent found them somewhat useful. Only one respondent was

more neutral in their response.

Table 13. Overall, have the webinars been useful or not useful? (n=15)


Useful 80.0% 12




Not useful 0.0% 0

3.7. Community Research Mentoring Project

The Community Research Mentoring Project aims to support individuals and organisations who

have an interest in research with an academic mentor to support their knowledge and skills

development. Established initially in 2014, at the time of the evaluation, this program was still

underway.

The project is being implemented in collaboration with the Centre for Research in Excellence in

Mental Health and Substance Use (CREMS) at the National Drug and Alcohol Research Centre.

Ten mentees working in the community mental health and drug and alcohol sector have been

each matched up with one of the eight participating research mentors. Five survey respondents

are participating in the Mentoring Project as mentees. Another five participants are unsure if they

are participating. Four of the five who are participating in the program indicated it was useful.

One respondent indicated it was neither useful nor not useful (they provided no comment on

their rating).

Table 14. You indicated that you are participating in the Mentoring Project. Overall, is

participating in the program useful or not useful? (n=5)


Useful 80.0% 4




Not useful 0.0% 0

| 16


3.8. Research Seeding Grants Program

Research Seeding Grants Program provided 16 successful applicants with funds to conduct the

preliminary work needed to develop a research proposal, including partnering with an

academic researcher. In total, over $136,000 was disbursed.

This grant program was separately evaluated in 2014. The evaluation concluded that it had

achieved its key goal of increasing research capacity, building the confidence of participants

and improving relationships with academic research partners.*

Seven survey respondents indicated that their organisation had received a Researching Seeding

Grant. Eleven respondents were unsure if their organisation had received a grant. Five of the

seven respondents who said that their organisation had received a grant indicated that it was

useful, while the remaining two survey respondents indicated it was somewhat useful.

Table 15. You indicated that your organisation received a Research Seeding Grants. Overall,

was this grant useful or not useful? (n=7)


Useful 71.4% 5




Not useful 0.0% 0

Two respondents commented favourably on the grants.

“very very useful”

“We received a grant to run a pilot project. This enabled us to develop inclusive research

methodologies that facilitated the involvement of Consumers as co-researchers.”

A key informant commented that that “things wouldn’t have happened otherwise” if seeding

grants were not available. However, another two respondents outlined some of the challenges

that emerged from the implementation of the grants.

“it was brilliant, but again I was not supported by my organisation when trying to do the

research”

“I fell down a bit on the implementation, but we are still using the findings.”

* CMHDARN Research Seeding Grants Program: An Evaluation. Mental Health Coordinating Council and

Network of Alcohol and other Drugs Agencies, 2014.

| 17


3.9. CMHDARN website

The CMHDARN website was developed in 2012 and went live in late September 2012. It provides

an overview of the project, its activities and makes relevant resources available. For example,

the presentations and associated resources from each of the research forums and webinars are

on the site.

An analysis of the website statistics since September 2010

found:

1,437 NSW visitors

3,263 NSW sessions

11,063 page views.

Seventy-one per cent of survey respondents have visited the CMHDARN website. The majority of

survey respondents found the website informative, while 38 per cent found it somewhat

informative. Seven per cent of respondents were more neutral in their response.

Table 16. You indicated that you have visited the website. Overall, did you find the CMHDARN

website informative or uninformative? (n=42)


Informative 54.8% 23

Somewhat informative 38.1% 16

Neither informative nor uninformative 7.1% 3

Somewhat uninformative 0.0% 0

Uninformative 0.0% 0

Providing information on the activities of the network to interested individuals working in the

mental health and drug and alcohol community sectors is the primary purpose of the site. Visitor

statistics suggests it achieves this goal. Overall NSW visitor engagements are:

3.39 average page views per session;

3.36 minutes average session duration; and

47 per cent bounce rate – these are visitors who arrive on the site but leave right away

without clicking through to any other page.

The bounce rate is at the higher end, though still not outside of normal expectations for websites.

Almost a third of all page views (3,943 page views) are of the Research Network Activities pages

which feature information on the research forums, webinars, etc. Returning NSW visitors have

much better engagement with the site. Staying almost twice as long (4.34 vs. 2.21 minutes) and

have slightly better average number of page views per session (3.66 vs. 3.04).

http://www.cmhdaresearchnetwork.com.au/

| 18


Currently, few visitors make it through to the Newsletter page or Membership page. The

Newsletter page has achieved 261 page views from NSW visitors since the website commenced

(less than 2 per cent of total page views). Similarly, the Become a CMHDARN Member page has

received 215 page views from NSW visitors (less than 2 per cent of all page views). These are

important communication and relationship management tools for the project and so should

feature prominently on the home page.

3.10 E-Newsletter

The CMHDARN e-newsletter, CMHDARN-Yarn, is published three or four times a year to keep

recipients informed about the work of the network and to provide them with information on

research activities and resources. It is also posted to the website. Each edition provides updates

of the networks activities and other events that may be of interest to recipients. It also provides a

brief summary of sector news and useful resources.

Seven newsletter have been produced:

Issue 7, July 2014 (pictured)

Issue 6, February 2014

Issue 5, December 2013

Issue 4, September 2013

Issue 3, July 2013

Issue 2, April 2013

Issue 1, August 2012.

Forty-one per cent of survey respondents have read the newsletter. Of those who had read the

newsletter, half said it was informative, while the remaining half said it was somewhat informative.

One respondent commented on the relevance of the newsletter.

“I understand it's difficult to source material that's exactly relevant to the diversity of

organisations that form part of this network”

Table 17. You indicated that you have read the e-newsletter. Overall, do you find the e-

newsletter informative or uninformative? (n=24)


Informative 50.0% 12

Somewhat informative 50.0% 12

Neither informative nor uninformative 0.0% 0

Somewhat uninformative 0.0% 0

Uninformative 0.0% 0

http://www.cmhdaresearchnetwork.com.au/research-network-activities/newsletter/

| 19


4. IMPACT ON KNOWLEDGE, SELF-EFFICACY AND RESEARCH CAPACITY

4.1 Effectiveness and relevance

The aim of CMHDARN is to ‘broaden the involvement of drug and alcohol mental health

community organisations in practice-based research, to promote the value of research and to

increase the application of research evidence to every day practice’.

Effectiveness

Seventy-two per cent of survey respondents thought the CMHDARN was effective or somewhat

effective at achieving its aim. Nineteen per cent were more neutral and a further nine per cent

thought it was somewhat ineffective. One key informant commented that they thought the

CMHDARN had done “superbly well”.

Those respondents who were more neutral or critical are generally not CMHDARN members, are

less familiar with the work of CMHDARN and have had less involvement with the network than

other respondents. As one respondent notes below that when they selected the ‘neither

effective nor not effective’ option, they were indicating that they do not know.

“Sorry I am not familiar with the networks aims or goals so couldn't really comment.

Thanks (& I should inform myself!)”

“! would rather say -'I don’t know'

Table 18. How effective or ineffective do you think the CMHDARN has been at achieving its

aim? (n=58)


Effective 22.4% 13

Somewhat effective 50.0% 29

Neither effective nor not effective 19.0% 11

Somewhat ineffective 8.6% 5

Ineffective 0.0% 0

| 20


Relevance

Overall, the goals of the CMHDARN are seen as relevant to respondent’s own work and the work

of their organisation. Sixty per cent agree the goals are relevant to their own work and 59 per

cent agree that the goals are relevant to their organisation’s work.

One of the reasons key informants and focus group participants praised CMHDARN is that they

believe its goal of encouraging community managed organisations* to participate in practice-

based research will produce better results for consumers/clients†.

“what we most want: priorities for service users, not priorities for academic researchers”

“in your workplace you’re closer to the changes and things you want to research and the

changes you want to make”

More generally, survey respondents praised the work of CMHDARN.

“Effective health research must seek to answer the questions of consumers and service

providers I believe CMHDARN has made this its focus which fits well with my role in my

current position.”

* ‘Community managed organisations’ is the terminology used within the report to describe not-for-profit

community organisations and non-government organisations. † ‘Consumers/clients’ is used to reflect the different language used in the mental health and drug and

alcohol sectors.

0

20

40

60

80

100

To your ownwork

To yourorganisation

60 58.6

26.7 25.9

13.3 13.8

1.7

Relevance of goals

Relevant Somewhat relevant Neither relevant/ irrelevant N/A

| 21


“The emphasis on evidence based practice is high in the sector right now (as it should

be). Supporting CMOs to further their skills in and opportunities for the development of a

sound body of evidence on which to base their service delivery is vital. Capitalising on

the evidence and practice knowledge from the field is a great way to add to the body

of evidence on mental health and AOD.”

Some respondents commented on what they have found most useful from participation in

CMHDARN.

“CMHDARN is an excellent initiative. The opportunity for CMOs and research professionals

to come together, share information and develop new learning is fantastic.”

“CMHDARN research forums have been very informative and useful for contacts with

other NFP's/ NGO's:

They also commented on how valuable it has been for them and their organisation.

“My organisation's (& my own personal) involvement with CMHDARN has been extremely

profitable. I think it is a great network and a wonderful resource.”

“I have found CMHDARN an extremely useful forum and network - hope funding

continues to be delivered for further development of its operations”

While a few respondents commented on their inability to participate more fully in CMHDARN

activities.

“I only go through the emails I receive. I never got an opportunity to attend any

sessions/symposium of CMHDARN”

“"however I am not able to attend as many activities as I would like to .We are short

staffed in my area of work. Even though I enjoy my work despite the shortage"

| 22


4.2 Impact on knowledge and self-efficacy

Survey respondents (excluding researchers) have indicated that participating in CMHDARN

activities has had a positive impact on their understanding of the issues:

61% strongly agree or agree that it has improved their understanding of drug and alcohol

issues;

54% strongly agree or agree that it has improved their understanding of mental health issues;

and

59% strongly agree or agree that it has improved their understanding of co-existing mental

health and drug and alcohol issues.

Those who indicated that they are familiar with the activities of CMHDARN, rate its impact more

positively on all the three indicators above. For example, among those respondents who are

familiar with CMHDARN activities, 85 per cent strongly agree or agree that it has improved their

understanding of co-existing issues (vs. 59 per cent of all respondents).

Table 19. Has participating in CMHDARN activities improved your understanding of drug and

alcohol and mental health issues? (n=41)

Answer Options Strongly

agree Agree

Neither

agree nor

disagree

Disagree Strongly

disagree

Drug and alcohol issues 24% 37% 37% 2% 0%

Mental health issues 18% 36% 44% 3% 0%

Co-existing MH/DA 15% 44% 41% 0% 0%

Those who tend to neither agree nor disagree are not likely to be members and have had more

limited participation in CMHDARN network.

“I cannot comment properly as I have no recollection of any engagement”

“I haven’t really participated in the activities”

Interestingly, two participants commented that while they didn’t think their knowledge has

shifted it has still been valuable experience.

“I don't think my knowledge has changed, but it has been valuable to work with

community based clinicians - I would be interested in developing these relationships”

“Working in a non-clinical Mental Health support service means that my understanding of

MH issues is relatively sound; likewise re co-morbidity but networking with specialist AOD

services has been useful”

| 23


Confidence

Two-thirds of respondents indicated that participating in CMHDARN activities has made them

more confident or somewhat more confident to use research findings in their work. The remaining

third of respondents indicated that they were neither more confident nor less confident.

Excluded from the analysis are those who indicated that this question was not applicable (21 per

cent of survey respondents) and survey respondents who were researchers. Again, of those who

are more familiar with the CMHDARN activities, all have indicated that it has made them more

confident or somewhat more confident.

Table 20. Overall, has participating in CMHDARN activities increased or decreased your

confidence to use research findings in your work? (n=35)


More confident 25.7% 9

Somewhat more confident 40.0% 14

Neither more confident nor less confident 34.3% 12

Somewhat less confident 0.0% 0

Less confident 0.0% 0

Again, approximately two-thirds of respondents rated themselves as more confident or

somewhat more confident to undertake practice-based research as a result of participating in

CMHDARN activity. The remaining 37 per cent of respondents were neither more confident nor

less confident. (Excluded from the analysis are those who indicated that this question was not

applicable, 21 per cent of survey respondents, and survey respondents who were researchers).

Again, of those who are familiar with CMHDARN activities, all have indicated that it has made

them more confident or somewhat more confident.

Table 21. Overall, do you feel more or less confident in undertaking practice-based research

as a result of participating in a CMHDARN activity? (n=35)


More confident 20.0% 7

Somewhat more confident 42.9% 15

Neither more confident nor less confident 37.1% 13

Somewhat less confident 0.0% 0

Less confident 0.0% 0

| 24


4.3 Impact on research capacity

Respondents were asked to rate the impact, from high impact to no impact on a five point

scale, of the CMHDARN on their own research capacity as well as on their organisation’s

capacity.

Impact on individual

When assessing the impact on their own research capacity, respondents gave the highest

impact rating to ‘use of research evidence’ (3.24 rating average) and the lowest impact rating

was ‘undertaking practice-based research’ (2.86 rating average).

Table 22. How would you rate the impact, if at all, of the CMHDARN on your own research

capacity? (n=42)

Answer Options

1

No

impact

2 3 4 5

High

impact

n/a Rating

Average

Use of research evidence 5 3 6 10 5 12 3.24

Collaboration with

researchers 4 6 7 5 6 11 3.11

Involvement of consumers

and carers in research 5 4 9 6 4 10 3.00

Undertaking practice-

based research 4 8 9 2 5 11 2.86

Impact on organisation

When assessing the impact on their organisation’s research capacity, the highest impact rating

was for ‘collaboration with research’ (3.17 rating average) and the lowest impact rating was for

‘undertaking practice-based research’ (2.97 rating average).

Table 23. How would you rate the impact, if at all, of the CMHDARN on your organisation's

research capacity? (n=42)

Answer Options

1

No

impact

2 3 4 5

High

impact

n/a Rating

Average

Collaboration with

researchers 5 5 6 6 7 11 3.17

Use of research evidence 5 7 4 9 6 10 3.13

Involvement of consumers

and carers in research 5 4 9 9 4 8 3.10

Undertaking practice-

based research 5 7 8 6 5 8 2.97

| 25


Those respondents who chose ‘not applicable’ when rating its impact on their own research

capacity or their organisations’ are often researchers, consumers/clients or individuals who have

had less involvement with CMHDARN activities. Similarly, those who select ‘no impact’ have

often had more minimal involvement in CMHDARN activities (e.g. no participation in the

Mentoring Project, Research Seeding Grants Program and webinars). By contrast, those that

gave a ‘high impact’ rating have had involvement in a broad variety of activities such as the

research forums and seeding grants. For example, participants of the Research Seeding Grants

Program rate the CMHDARN as having had a much higher impact on their own research

capacity and their organisations, across all four indicators.

4.4 Building future research capacity

Survey respondents (excluding researchers) were provided with 12 options for future activities

and were asked to select those that are most valuable to help ensure sustainable research

capacity in their organisation. These options were based on examples used in the Research

Capacity Building Framework mentioned at the start of the report and addressed in the next

section. The six most popular choices (in order of popularity) were:

‘partnerships with research’ and ‘best practice resources’

‘applying skills in practice’

‘support and mentoring’

‘skills training’ and ‘funding opportunities’.

Table 24. To help us plan future activity, from the list below, what activities do you think are the

most valuable to help ensure sustainable research capacity in your organisation? (Tick those

you most value) (n=42)


Partnerships with researchers 57.1% 24

Best practice resources 57.1% 24

Applying skills in practice 54.8% 23

Support and mentoring 50.0% 21

Skills training 47.6% 20

Funding opportunities 47.6% 20

Dissemination of research findings 42.9% 18

Partnerships between MH and DA organisations 40.5% 17

Time allowance to undertake research 38.1% 16

Development of organisation research strategy or policy 28.6% 12

Management support for research 23.8% 10

Research included in job descriptions 11.9% 5

Other (please specify) 4.8% 2

| 26


Survey respondents and focus group participants commented on the need to identify funding

opportunities for research.

“how to get funding”

“Funding is the constant issue. It might be very useful to have regular info sessions on

where to go for funding, discussing new ways of raising funds such as "crowd funding"

and the pros & cons of such initiatives. Activities that foster a culture of collaboration

rather than competition would be good.”

“funding initiatives for projects”

4.5 Challenges in building research capacity

One of the key challenges identified by participants in the evaluation was that for CMHDARN to

achieve its goals it needs to address the priority accorded to research within community

managed organisations and the resources, both financial and human resources, allocated to

support research.

“CMHDARN can only be so effective without further resources (human and financial) in

the sector to support effective implementation and participation in research.”

Key informants and focus group participants commented that in their experience many

community managed organisations do not see research as core business.

“some see it as core business, research and evaluation, others it’s not part of their core

business”

“(my organisation) doesn’t have a culture of research and evaluation”

One participant noted that while they received a research seeding grant they did not feel

particularly supported by their organisation.

“it was brilliant (research seeding grant), but again I was not supported by my

organisation when trying to do the research”

| 27


More generally, focus group participants reported that the level of support for research within

their organisations varied. For some their organisations clearly support research and evaluation,

particularly if they work in a large organisation.

“(we have a) research and evaluation group in our organisation”

While some stated that their organisation “generally doesn’t” value research or have a “culture

of research and evaluation”, others noted that while it might be valued they “don’t budget for

research” which inhibits it occurring.

“I think CMHDARN do great work, I would like to see more support for funding research

and also trying to promote the benefits of it. I have been on my own in my organisation

trying to do research. When there is no funding no one cares”

“don’t budget for research”

Focus group participants discussed how their organisations generally sees themselves as working

at the “coalface” so the focus is on, as one participant described it, “doing the doing”.

In terms of where to next, key informants and focus group participants generally thought that the

network needed to keep building on the work it has commenced.

“continue with seeding grants – how can we build upon them?

“(on a) solid trajectory”

4.6 Discussion of Research Capacity Building Framework

As noted in the introduction, the CMHDARN has adopted a draft framework for research

capacity building.* To what extent is the project engaged in activities across the six principles

outlined in the framework?

Building skills and confidence: CMHDARN has had a significant focus on developing the skills and

confidence of the workforce to engage with research and conduct practice-based research.

Research forums, mentoring, research seeding grants and webinars are all designed to do this,

and the evaluation has found that they have been effective at achieving this.

* Cooke J. A framework to evaluate research capacity building in health care. BMC Family Practice 2005,

6:44

| 28


Ensuring the research is 'close to practice': The CMHDARN focus has been on practice research,

which inherently ensures research is close to practice. CMHDARN has encouraged community

managed organisations to conduct research relevant to their own practice, as well as facilitate

more collaboration with researchers who are interested in practice research. There has also

been an emphasis on the importance of consumer, client and carer involvement in research.

Developing linkages and partnerships: This is a strength of the CMHDARN. Firstly, the initiative is a

collaborative effort between MHCC and NADA. Secondly, there has been a significant focus on

nurturing collaboration and partnerships with researchers. Thirdly, the project has facilitated and

promoted the importance of consumers/clients/carers as partners in research.

Developing appropriate dissemination: Research findings dissemination has been supported by

the CMHDARN, particularly through the research forums and webinars. There were nine

presentations to CMHDARN Research Forum (November 2013) by research seeding grant

recipients. They have also disseminated their work through the publication of journal articles and

conference presentations. The Seeding Grants Program Evaluation identified that there were five

papers (published and submitted) and 12 external conference presentations up until January

2014.*

Building elements of sustainability and continuity: There has been a good attempt to ensure

acquired skills are sustained and extended through the Research Seeding Grant and Mentoring

Project. However, the extent to which others who participate in forums or webinars have applied

their skills to practice (or initiate collaborations or applied for research funding, etc.) in their

workplace is unknown. ‘Applying skills in practice’ is one of the top three priorities for further

CMHDARN activities identified by survey respondents.

Appropriate infrastructures: There have been some contributions to supporting infrastructure,

such as the Mentoring Project, however, CMHDARN had had limited influence on organisational

change (such as ensuring research is included in job descriptions or backfilling arrangements to

support research or organisational research dissemination strategies).

* CMHDARN. Research Seeing Grants Program: An Evaluation, 2014.

| 29


5. CONCLUSIONS

5.1 Overall findings

CMHDARN’s goals are relevant to individuals and organisations working in mental health and

drug alcohol sectors. CMHDARN is effective in working towards achieving its goals. There was

high level of awareness of the CMHDARN and familiarity with its work, although this is not

surprising given most survey respondents are either members of the network or have attended an

event.

CMHDARN has implemented activities (such as the Mentoring Project) cognisant of the often

limited research experience or skills of participants. This has been the right approach. According

to the survey, most organisations do not have a research team or even a research position to

undertake research (and even when they do their capacity may be limited). Individuals who

participate in CMHDARN activities report that it has had a positive impact on their knowledge,

skills and self-efficacy.

Key informants thought the CMHDARN has done a good job of trying to focus on both mental

health and drug and alcohol issues, although they observed that the mental health sector tends

to participate more in its activities. This was not the experience of the evaluation, where there

were twice as many survey respondents from NADA member organisations vs. MHCC member

organisations. Also a review of attendance at the Research Forums throughout 2014 indicated a

fairly even attendance between NADA and MHCC member organisations. CMHDARN is also

recognised as having worked to promote the importance of consumers/clients as partners in

research.

The evaluation has found that the impact the CMHDARN has on individuals and organisations

research capacity relates directly to the level of familiarity and involvement individuals have with

the CMHDARN, this suggest that an important component of future activity is to continue to

expand the reach and participation of community managed organisations in network activities.

| 30


Key impacts and achievements include:

CMHDARN has increased the self-efficacy of network participants:

– 66 per cent more confident or somewhat more confident to use research findings, and

– 63 per cent more confident or somewhat more confident in practice-based research.

CMHDARN has increased the knowledge of drug and alcohol, mental health and co-existing

issues among network participants:

– 61 per cent strongly agree or agree that it has improved their understanding of drug and

alcohol issues;

– 54 per cent strongly agree or agree that it has improved their understanding of mental

health issues; and

– 59 per cent strongly agree or agree that it has improved their understanding of co-

existing mental health and drug and alcohol issues.

Engagement with CMHDARN:

o 1,437 NSW visitors to the website;

o 11,063 website page views from NSW visitors;

o 11 research forums with a total of 493 participants;;

o 7 webinars with 257 participants;

o 7 newsletters distributed; and

o 240 CMHDARN members.

5.2 Usefulness of CMHDARN activities

There was very good feedback on the variety of activities (webinars, mentoring, research

seeding grants, research forums) that have or are being implemented by CMHDARN. For those

who have participated in these activities, they found them to be informative and useful.

Research forums and webinars are an important and useful activity, particularly given that these

types of events are one of the major ways survey respondents identify that they keep up to date

with research. The typical research forum format, presentations followed by workshops and/or

small group discussions, was well received. Focus group participants also commented that

research forums have been useful at facilitating collaboration with researchers, as well as

providing valuable opportunities to network with colleagues.

The CMHDARN website and e-newsletter are useful tools for providing information to interested

parties. A third of all page views related to finding more information on CMHDARN activities. The

| 31


usefulness of the website could be further enhanced by enabling participants to register for

forums online (via an embedded form).

CMHDARN has recognised that, due to the cost, obtaining journal articles is an issue for

individuals and organisations (this arose during the focus group and survey). CMHDARN offers a

service where it will source journal articles (two per request) for MHCC/NADA member

organisations and has promoted sites that offer ‘open access’ to research. The uptake of this

service to source journal articles has not occurred. Further consideration may need to be given

as to how the community sector can be supported to access journal articles. As one key

informant put it “library access would be good”.

5.3 Planning, implementation and communication

CMHDARN Project Reference Group is working well and makes a useful contribution to the work

of the network. There is also good ongoing monitoring of all project activities. All forums are

evaluated and reports posted online. This has made it easier to provide detailed reports to the

Project Reference Group, partners, funders and to inform this evaluation.

CMHDARN members are kept well informed of upcoming events. While a third of survey

respondents noted that they are members, there was some confusion, as nearly a third of

respondents were unsure if they are members. It would be easier and simpler to shift to an email

subscription service (managed online). The main advantage is that it makes it easier for people

to join the subscription (less information should be required, for example postal addresses would

not be required, and with one click you are subscribed).

In addition, it returns control to subscribers allowing them to easily withdraw from the subscription

when they chose which is more in keeping with current expectations for email recipients (as an

example, one respondent commented “I hope I will be removed from the address lists”).

5.4 Measuring impact over time

This evaluation has demonstrated the immediate impact CMHDARN has had on research

capacity. It will be valuable to assess the impact over the longer term. A number of impact

indicators used in this evaluation should be run again in approximately two years’ time to assess

how things may have changed. This includes:

understanding of drug and alcohol and mental health issues;

confidence to use research findings in your work;

confidence in undertaking practice-based research;

| 32


impact on your own research capacity; and

impact on organisational research capacity.

It is important to bear in mind that the level of involvement in CMHDARN activities will impact on

the performance as measured against the above indicators (i.e. those with greater involvement

n CMHDARN activities were more likely to rate its impact more highly). Furthermore, the overall

number of respondents to these questions was fairly small (approximately 40 respondents) so any

comparison between these findings and any future findings will be impacted by the number of

respondents recruited (i.e. a small increase in the number of respondents could significantly

change the outcomes).

This evaluation has focussed on interviewing those individuals who have participated in or

engaged with CMHDARN activities. A future evaluation should consider focus groups with senior

managers of community managed organisations to help us understand how they perceive their

organisations’ role in research and how CMHDARN can support this role.

5.5 Next steps for CMHDARN

There is strong support for CMHDARN to continue. Particularly survey respondents wanted

CMHDARN to:

develop the skills of the sector

support partnerships with researchers,

produce best practice resources,

support and mentoring

support the application of skills in practice.

It’s important that people not just learn but, as one respondent put it, “have a go at doing”

research. Within their workplaces, focus group participants and survey respondents noted that

they feel there is often a lack of support for research and evaluation within their workplace.

Furthermore, survey respondents generally indicated that there is no team or position that

focuses on research and evaluation (and even when there is they often have limited capacity).

This highlights that in addition to focussing on skills development, CMHDARN needs to give

consideration to how it can better support and engage management to build a workplace

culture that fosters research and evaluation. This would also align well with the research capacity

framework the project has adopted that emphasises the importance of sustainability and

continuity (principle five) and appropriate infrastructure (principle six).

| 33


However, to effectively do this discussions need to occur with funders about the funding required

to support practice-based research in community managed organisations. An evaluation of the

Research Seeding Grants Program Evaluation also made a similar recommendation.* Prior to the

formation of CMHDARN, the NGO Mental Health and Drug and Alcohol Research Grants

Program was administered by NADA and MHCC between 2007 and 2011. The program

evaluation† made a series of recommendations that are still relevant today (see Appendix B)

and are supported by the findings of this evaluation.

As noted in the introduction, the Mental Health Commission of NSW is developing a Research

Framework for Mental Health in NSW that identifies one of the overarching aims of NSW-funded

mental health services is to ‘accelerate the translation of research into practice’. This provides an

opportunity for CMHDARN, NADA and MHCC to advocate for greater funding support for the

research role of community managed organisations.

* CMHDARN. Research Seeing Grants Program: An Evaluation, 2014.

† EJD Consulting and Associates. NGO Mental Health & Drug and Alcohol Research Grants Program: Final

Report, 2012

| 34


6. RECOMMENDATIONS

1. CMHDARN should continue to focus on developing the research capacity of community

managed organisations, promoting the value of research and the use of research evidence

in practice.

2. CMHDARN should continue the variety of activities it is currently undertaking as they are

useful and are having an impact on skills and self-efficacy, this includes webinars, research

forums, and mentoring.

3. CMHDARN should continue with the current format of research forums (presentations which

are followed by guided small group discussions) as they are working well to promote learning

and sharing of experiences.

4. Given the usefulness of the Research Seeding Grants, the identification of the importance of

funding among evaluation participants, recommendations from the Research Seeding

Grants Program and recommendation from the Final Report on the NGO Mental Health &

Drug and Alcohol Research Grants Program, it is recommended that CMHDARN approach

funders to discuss research funding for community managed organisations to support

practice-based research.

5. CMHDARN should continue to maintain and update the CMHDARN website, further

enhancing its technical functions by enabling online registration for research forums and

more prominently featuring the option to subscribe to the email list.

6. CMHDARN should move to a subscription email list (rather than the current member model)

as it makes it easier for interested individuals to subscribe and unsubscribe from receiving

emails from CMHDARN.

7. In addition to the focus on knowledge and skills development, and given most workplaces

do not appear to have dedicated research and evaluation teams or positions, CMHDARN

needs to consider how it can support and engage management in community managed

organisations to better promote, value and support research and an evaluation culture

within their organisations.

| 35


8. CMHDARN should consider the implications of this report and amend its draft Evaluation

Framework to reflect the following as priority indicators for its implementation, with a view to

evaluating the program in two years’ time:

i. understanding of drug and alcohol and mental health issues

ii. confidence to use research findings in your work

iii. confidence in undertaking practice-based research

iv. impact on individual research capacity

v. impact on organisational research capacity.

9. A future evaluation of CHMDARN should run focus groups with senior managers of

community managed organisations to understand how they perceive the role of research in

their organisations and how CMHDARN can support this role.

10. CMHDARN has promoted the role of consumers/clients/carers in research and should

continue to emphasise the importance of this role.

| 36


APPENDIX

Appendix A: Survey Instrument

| 37


| 38


| 39


| 40


| 41


| 42


| 43


| 44


| 45


| 46


Appendix B: NGO Mental Health & Drug and Alcohol Research Grants Program: Final Report

The following recommendations are from the NGO Mental Health & Drug and Alcohol Research

Grants Program: Final Report (EJD Consulting and Associates, August 2011).

Research Culture within CMO

R5: Resource NADA and the MHCC as the peak bodies to support and promote the

development of evidence-based cultures within CMO, including with respect to:

a) the treatment and support of clients with co-existing mental health and alcohol and

drug issues, and

b) the inclusion of family and carers in treatment models.

R1: Provide AOD and mental health CMO, regardless of their size or location, with additional

opportunities and mechanisms to facilitate ongoing:

a) partnerships, collaborations and joint investigations with research organisations

b) capacity building, research training and skills transfer to enable CMO staff to lead and

manage research in their own right

c) promotion and dissemination of CMO focused research.

R4: Funding bodies allocate a proportion of their (core) grant funding to CMO to support

research, and enable them to:

a) collect and rigorously analyse client data and service delivery outcomes

b) undertake quality research using robust techniques

c) form partnerships with research organisations

d) disseminate their findings and share research outcomes with other CMO and research

organisations

e) in other ways implement a culture of continuous improvement and evidence-based

service delivery.

R13: NADA and the MHCC investigate further options to support CMO through:

a) commissioning joint research focused on co-existing mental health and drug and

alcohol issues relevant to CMO service delivery

b) promoting the findings of quality research relevant to CMO including via web-based

systems

c) (with appropriate approvals) promoting mechanisms to share client data that

supports service design, service delivery and/or research by other providers

d) expanding networking opportunities and information exchanges between CMO

interested in research

e) building partnerships between mental health and alcohol and drug sectors including

through supporting:

. i) joint research endeavours

. ii) other forms of collaboration and joint activity specifically related to clients with

co-existing mental health and drug and alcohol issues.

| 47


R14: Provide CMO with more training and information to specifically up-skill staff in:

a) relevant health and social sciences research techniques, methodologies and their

appropriate applications to CMO service settings

b) ethics approval processes and required ethics documentation

c) report writing and research dissemination.

R20:Decision makers at all levels better promote policy, funding and accountability systems that

are first and foremost client-driven, and facilitate effective, ‘joined-up’ responses to clients with

co-existing mental health and drug and alcohol issues.

Future Research Grant Administration

R2: Revise future CMO research grant funding processes to include a modest establishment grant

component to cover costs associated with ethics approval and staff recruitment.

R3: With legal input, draft a model partnership agreement for use by CMO and research

organisations covering clauses and options for the management of risks and other complex

issues including:

a) intellectual property

b) staff supervision and management

c) authorship and attributions related to research papers and publications

d) financial management and budget responsibilities

e) dispute and conflict resolution processes.

R8: Adopt similar reporting templates, and periodic feedback processes to that used for the

Research Grants Program in order to solicit formal reporting against key milestones and funding

accountabilities, together with qualitative feedback on issues and challenges arising.

R9: Amend future grant guidelines to include a provision that requires grant recipients to:

a) include the research grant as a separate line item within their organisation’s

annual financial statement and that this be:

i) reported against as part of their independently audited, end-of- year

financial statement

ii) submitted to the auspicing body as evidence of appropriate grant

expenditure

b) notify the auspicing body (on a quarterly basis for example) of any significant

over or under expenditure of grant funding based on the budget submitted,

together with either:

i) proposed measures for how to rectify the situation, and/or

ii) requests to modify or extend their research project, and allocate the

funding to complementary activities, particularly in cases where:

a) project savings have been made, or

b) grant under expenditure has occurred for other reasons.

| 48


R10: Include in future grant guidelines a provision for how the funding body will review and

respond to any request to vary grant funding expenditure. Criteria for approving grant

expenditure variations might include:

a) The proposal is consistent with the aims and objectives of the grants program

b) The proposal builds on or complements the initial research project’s purpose

c) The proposal can be delivered with the funding available, and within an

agreed timeframe

d) A final report (and where appropriate, also progress reports) on the revised

proposal outcome, using the standard reporting template, be submitted to the

funding body at the conclusion of the project

e) A short project plan is agreed to and signed that stipulates any requirements,

including obligations to report on project outcomes and expenditure.

R11: Future research guidelines include additional provisions for the management of unforeseen

delays in project delivery. This might involve the addition of a standardised ‘slippage’ clause in

the funding contract, that would allow, subject to appropriate notification, agreed project

milestones to be postponed for a specified number of months, without jeopardising the terms of

the grant.

R12: Future research guidelines (and related forms and processes) include a risk mitigation

section in relation to project plans. This might include risk management measures such as:

a) agreements and processes for managing the replacement or back-filling of

key personnel when extended sickness, leave or resignations occur mid-project.

b) communication protocols between partners for how significant problems or

changes in project plans or milestones will be managed.

One option might be to formally nominate ‘shadow’ or ‘back-up’ personnel within the

grant recipient organisation or research body who would be involved in key aspects of

the project and thus potentially available to assume responsibility should the need arise.

Research Partnerships

R6: Government and non-government agencies initiate new measures to extend collaboration

between the AOD and mental health sectors, at all levels, including through supporting

enhanced:

a) information sharing and referral pathways for clients

b) joint case management and service delivery options especially for clients with

co-existing mental health and drug and alcohol issues

c) partnerships agreements or memorandums of understanding defining joint

processes and procedures

d) cross-sectoral training and staff development opportunities

e) joint service delivery models, treatment and support options.

R7: Consider establishing a partnership development stream as part of core mental health and

AOD NGO funding to encourage and support formal collaboration and partnerships between

mental health and AOD CMO, as well as between CMO and government service providers.

| 49


Ethics Processes

R15: Establish a simplified and streamlined research ethics approval process in NSW suited to

NGO contexts and specifically including processes for managing:

a) working with vulnerable clients and other common though complex health

related issues

b) research involving multiple locations and potentially involving more than one

Local Health District.

R16: NADA and the MHCC investigate establishing a new peak ethics approval process (in

partnership with an established research body) specifically tailored to AOD and mental health

CMO in NSW.

Research Agenda

R17: Make sufficient funding available to enable the MHCC-NADA Research Network to be an

ongoing vehicle for the:

a) support, encouragement and dissemination of research relevant to CMO in NSW

b) establishment of partnerships with research bodies and individuals with an interest and

capacity for research relevant to CMO in NSW

R18: NADA and the MHCC enhance the current activities of the Research Network in order to

more intensively pursue activities listed at Recommendation 6, as well as:

a) enhancing inter-sectoral activities including via:

i) more face-to-face meetings and networking opportunities

ii) electronic exchange of members’ contact details and research interests

b) enhancing the web-based information available on current:

i) research relevant to AOD and mental health CMO

ii) research skills and development training opportunities

iii) research grant and research partnership opportunities

iv) upcoming conferences, forums and workshops.

Documents

CMHDARN Evaluation-ReportFINALFeb2015