Evaluation of Concurrent Disorders “Train the …...‘train the trainers’ program on concurrent disorders. The term concurrent disorder refers to the presence of both a s evere

Evaluation of Concurrent Disorders “Train the Trainers”

Program

Prepared for:

Canadian Mental Health Association: Ottawa-Carleton Branch 1355 Bank Street, 3rd Floor

Ottawa, Ontario K1H 8K7

Prepared by:

Gordon Josephson, Ph.D. Candidate

Tim Aubry, Ph. D. Brad Cousins, Ph. D.

Centre for Research on Community Services Faculty of Social Sciences, University of Ottawa

34 Stewart St. Ottawa, Ontario

K1N 6N5

July 2003

Evaluation of Concurrent Disorders Training 2

Centre for Research on Community Services

EXECUTIVE SUMMARY Context and Goals of the Evaluation In 2003 the Canadian Mental Health Association contracted the University of Ottawa’s Centre for Research on Community Services (CRCS) to evaluate their ‘train the trainers’ program on concurrent disorders. The term concurrent disorder refers to the presence of both a severe and persistent mental illness and a substance abuse problem. The intervention being evaluated was composed of two distinct phases. The first was an initial ‘train the trainer’ experience in which trainers were taught how to train one or more modules relating to concurrent disorders. The second was a pilot of the training program in which those trained as trainers, in the initial ‘train the trainer’, delivered their module to their fellow ‘classmates’ (i.e., those also trained to deliver a module). This second training, presented in February 2003, consisted of four topics spread over five days. These topics were: (1) mental health issues (day 1), (2) substance use (day 2), (3) concurrent disorders (day 3), and (4) intervention strategies (i.e., motivational interviewing) (days 4 and 5). In addition, the training modules include process information on launching (day 1) and wrapping up (day 5) the training. The specific goals of the present evaluation were as follows:

1) to examine participants’ satisfaction with the training, 2) to examine participants’ perceptions of knowledge and skills gained

through the training, 3) to examine the extent to which participants were able to transfer

knowledge from their training to others and to their day-to-day work, and, 4) to examine how the training process could be improved to better meet

these first three goals. Method Pre and post questionnaires were delivered by the course facilitator during the February pilot training for each of the 4 modules of training. In the questionnaires, participants quantitatively rated their perceived knowledge pre- and post-training, and their satisfaction post training. As well, qualitative questions about satisfaction with the training were solicited. Quantitative data were analyzed for significant pre- and post-training differences in perceived knowledge, level of satisfaction on seven aspects of the training, and significant differences among these aspects of satisfaction for a given module. Qualitative responses were examined and summarized. A member of the evaluation team conducted follow-up interviews in the period 2 to 5 months following the pilot. The interview consisted of open-ended questions



focusing on participants’ impressions of the training, how they are using it, and how it could be improved. Results Key findings include the following: - Participants reported an overwhelmingly positive view of the training sessions and the training manuals. One of the few exceptions to this is that, in the concurrent disorders module, satisfaction with the variety of tasks was significantly lower than the mean satisfaction for all items in the concurrent disorders module. Specific strengths within an overall strong program, include significantly higher satisfaction with: the instructor’s interaction in the substance use module, the instructor’s knowledge in the concurrent disorders module, and the duration of training in the motivational interviewing module. - Individual responses on satisfaction items indicate one participant was less satisfied with the quantity and quality of the substance use module, one was less satisfied with the variety of tasks in the motivational interview module, and 4 were less satisfied with the variety of tasks in the substance use module. - Qualitative comments from the questionnaire suggest strong satisfaction with the training including the content of the modules and the training style of the trainers. The comments do, however, point to some concerns with an excess of material in the substance use module and the need for a more varied way to presenting it. Comments also suggest participants felt positive about the material and trainers and that they learned a variety of new information. In particular, information related to assessment, conceptualization, and planning interventions. - The training strongly met trainer’s expectations and needs. - There are some concerns over the congruency and integration over the 5 days of training given the involvement of so many trainers. Similarly, there were some concerns with administrative details, such as the delivery of the evaluation forms at the appropriate time. However, it was felt the training remained largely effective - Overall there was a significant gain in perceived knowledge from the training in each of the 4 modules (mental health, substance use, concurrent disorders, and motivational interviewing). - All but 5 of the 32 items used to measure perceived knowledge indicated a gain in perceived knowledge. Of the 5 items, 1 did not indicate knowledge gain due to the fact that pre training knowledge levels were already at the top of the scale. Of the items that had room to improve after training, only 1 item showed no



improvement, that being item 7 of the mental health module, “I know many characteristics of people who are homeless in Ottawa”. - The participants reported pre-existing high levels of knowledge in the area. They described how the training helped them structure this knowledge in a framework that has made the information more readily accessible to them. - Assessment skills stood out as a particularly strong area of knowledge and skill gain. The participants also reported receiving exceptional training in adult education. They indicated this ‘how to train others’ aspect of their training was unique in its thoroughness and in the engaging style with which it was delivered. - Participants indicated they were comfortable offering training but see themselves as needing to become even more comfortable through offering more training sessions. As such, some concern was raised that there has not been sufficient opportunity to engage in training since the pilot session. - In discussing participants’ opportunity to share the information they gained in the training, there was some confusion over the freedom to use the training materials outside the context of a CMHA organized training session. Participants made reference to a signed agreement not to copy or disseminate material from the manual. Such an agreement seems to counteract the project objective of sharing the information as broadly as possible. - Almost all the trainers exhibited high levels of commitment and enthusiasm, the training appears to have stimulated their interest (or maintained an already high level of interest) in this area and they have many personal goals for increasing their knowledge in this topic. - There is a high opinion of CMHA by trainers from partner agencies, in particular participants noted CMHA’s commitment to the topic of concurrent disorders and their skills in working in partnerships.

- The goal of the training aligns with most but not all of the trainers’ home organizations’ goals. - Overall, all trainers received encouragement and enthusiasm from their colleagues. Trainers from partner agencies were more likely to mention a lack of encouragement from managers than were CMHA trainers. - It is unclear to many participants what recognition trainers will receive from their involvement in the program. - There is some indication that, despite high levels of enthusiasm, trainers are having some difficulty finding the time to be involved in offering training sessions.



- There are some concerns about the need for ongoing funding to ensure participant involvement and the adaptation and improvement of the training materials. - There is mixed opinion about the adequacy of the training facilities at CMHA, in particular there were concerns with parking and it was suggested that the size of training groups remain at or below 20 people. Limitations The findings of the evaluation are limited by the fact that, in the pilot training, the trainees were composed of individuals who participated in the original ‘train the trainer’ and thus were already trained to deliver components of the course. In addition, sample sizes for the evaluation were small. Recommendations On the basis of these findings we offer the following recommendations:

Recommendation 1: It is recommended that the variety of tasks in the concurrent disorders module and substance use module be reviewed.

Recommendation 2: In general, the training materials are solid and need not be changed at the present time. CMHA can feel confident that they have an interesting compilation of materials, an engaging means of conveying it, and have involved effective trainers in training this first team of trainers. The one consistent exception to this is the need for some changes to the substance use module. Several potential changes are suggested in the discussion of interview findings and these can serve as a starting place for further discussion on the appropriate content and delivery of this material.

Recommendation 3: It is recommended that the material on characteristics of people who are homeless in Ottawa in the mental health module be reviewed for the extent that it provides new knowledge to service providers who are already sophisticated in this area and then be revised based on this review. Recommendation 4: In order to offer consistency through the modules, it is recommended that a staff member from CMHA be assigned the role of attending the training for the first and last few minutes of every day as well as being available through the day for any questions that arise. This need not be a subject matter expert but should be someone familiar with the format and procedural elements of the training (e.g., evaluation forms).



Recommendation 5: It is recommended that the people trained to deliver this training be given frequent opportunities to use these new skills and this knowledge in order to continue to maintain and improve on what they have learned. Recommendation 6: It is recommended that CMHA’s position and directives to trainers regarding use of training materials outside of CMHA organized training sessions be clarified.

Recommendation 7: It is recommended that future ‘train the trainer’ activities consider explicitly discussing the following:

- using informal opportunities for sharing information from the training, and - finding a balance between trainers staying true to the content o f the material in the manual and developing their own presentation style.

Recommendation 8: It is recommended that CMHA provide a letter of recognition to the managers of the trainers from partner agencies subsequent to their offering training. As suggested by a participant, this letter could include an indication of the numbers of people trained and a summary of the outcome of this training (e.g., copies of the evaluation forms). Such a letter could also be sent to mangers of trainers staffed at CMHA, for notation in their annual review.

Recommendation 9: It is recommended that in using the training facility at CMHA, the size of training groups be limited to 20 people or less. Recommendations Related to Future Evaluations: Recommendation 10: It is recommended that the four point scales used by CMHA to assess participant’s opinions on the training be changed to a five-point scale in order to increase variability. It is also recommended that anchors be presented for all of the response alternatives. The items may want to explicitly ask about level of satisfaction with different aspects of the training (e.g., quite dissatisfied, mildly dissatisfied, mixed feelings, mostly satisfied and very satisfied). Recommendation 11: It is recommended that a 5-point response scale for evaluating perceived knowledge be used pre- and post-training in order to better distinguish between the level of knowledge on specific topics and to better capture knowledge gain.



Table of Contents

Page. Executive Summary……………………………………………………… ….. 2 Table of Contents…………………………………………………………….. 7 Introduction……………………………………………………………………. 8 Method…………………………………………………………………………. 9 Results…………………………………………………………………………. 11 Satisfaction with Training…………………………………………….. 11 Perceived Knowledge and Skills Gained…………………………… 20 Transfer of Knowledge and Skills……………………………………. 22 Limitations of Results…………………………………………………. 27 Key Findings and Recommendations……………………………………….. 28 Appendix A - Pre training questionnaire………………………………..… 33 Appendix B – Post training questionnaire……………………………….. 37

Appendix C - Interview consent form……………………………………. 49

Appendix D - Interview questions………………………………………… 50

Appendix E – Verbatim satisfaction comments ………………………… 53



INTRODUCTION Context The Ottawa Branch of the Canadian Mental Health Association (CMHA) has been providing outreach and case management services to persons with severe and persistent mental illness in the Ottawa area for over 15 years. Currently, CMHA provides direct services to over 500 clients. Over the course of providing these services, it has become evident to CMHA management and staff that many clients have a concurrent disorder. The term “concurrent disorder” refers to the presence of both a severe and persistent mental illness and a substance abuse problem. Based on data collected through the Community Mental Health Evaluation Initiative, the prevalence of concurrent disorders (i.e., either an alcohol abuse problem or substance abuse problem or both) among CMHA clients exceeds 60%. In this context, the Ottawa Branch of CMHA recently developed two pilot programs to address concurrent disorders: (1) a group treatment program for clients with concurrent disorders based on a harm reduction approach intended to supplement case management and outreach services, and (2) a “train the trainers” program on the diagnosis and treatment of concurrent disorders with the objective of providing this training to CMHA staff and staff from community agencies in Ottawa. In 2003 the Canadian Mental Health Association contracted the University of Ottawa’s Centre for Research on Community Services (CRCS) to evaluate their ‘train the trainers’ program on concurrent disorders. The evaluation team from CRCS was made up of Gordon Josephson, a doctoral candidate in the Clinical Psychology Training Program at the University of Ottawa, as well as Tim Aubry and Brad Cousins, faculty members at the University of Ottawa with expertise in program evaluation. For the purposes of the present evaluation, the intervention being evaluated was composed of two distinct phases. The first phase, conducted in fall 2002, was an initial ‘train the trainer’ experience in which trainers were taught how to train one or more modules relating to concurrent disorders. The second phase was a pilot of the training program in which those trained as trainers, in the initial ”train the trainer” experience, delivered their module to their fellow ‘classmates’ (i.e., those also trained to deliver a module). This second training activity, presented in February 2003, consisted of four topics spread over five days. These topics were:

(1) mental health issues (day 1), (2) substance use (day 2), (3) concurrent disorders (day 3), and (4) intervention strategies (i.e., motivational interviewing) (days 4

and 5).



In addition, the training modules included process information on launching (day 1) and wrapping up (day 5) the training. Purpose of Evaluation The purpose of the evaluation is to examine the effectiveness of the “train the trainers” program offered by the Canadian Mental Health Association during the winter of 2002/2003. The training focused on how to deliver one or more components of a 5-day course, developed for CMHA, on assisting persons with concurrent disorders. The specific goals of the evaluation were as follows:

(1) to examine participants’ satisfaction with the training, (2) to examine participants’ perceptions of knowledge and skills gained

through the training, (3) to examine the extent to which participants were able to transfer

knowledge and skills from their training, and, (4) to examine how the training process could be improved to better meet

these first three goals. The purpose of the present report is to summarize the findings of this evaluation. The report is composed of 3 additional sections, a description of the method used, a presentation of the findings (from both the questionnaire and follow-up interviews), and a description of key findings and recommendations.

METHOD

Evaluation Sample Seventeen participants in the training returned pre and/or post questionnaires collected during the pilot training in February 2003. Ten of the 17 participants were administered the follow-up interview. The education levels of the 17 participants included college diplomas (n=2), bachelor degrees (n=11), and master degrees (n=4). Eleven participants indicated their primary specialization was “mental health”, two indicated “addictions”, one “concurrent disorders”, one “generalized services”, and another one indicated a combination of “public education and management”. Similarly, twelve indicated their current professional role was in the mental health field, two in the addictions field, and one each in the fields of concurrent disorders, group therapy, and housing. The time spent in the current position ranged from 1 to 15 years with a mean of 7.12 years (n=17). There is no information on participants who did not complete the questionnaire however these statistics can be thought to be representative as the majority of participants handed in questionnaires.



Sources of Data

Three sources of data were examined to achieve the evaluation goals: (1) pre-training questionnaires in which participants recorded their perceptions of

the extent of their knowledge on key content topics covered in the training, (2) post-training questionnaires in which participants again recorded their

perceptions of the extent of the knowledge on the same key content topics and, in addition, rated their satisfaction with the training and provided demographic information and suggestions for improving the training, and,

(3) follow-up interviews in which participants described how successful they

have been in transferring knowledge and skills gained in the training program and offered suggestions for improving the training.

Pre- and post-training questionnaire. Items capturing perceived knowledge gain were developed by reviewing the content of the training manual and forming 8 questions about key material in each of the 4 main topics, mental health, substance use, concurrent disorders, and motivational interviewing. The consultant who designed the training material reviewed these items. An example of a perceived knowledge question from the mental health module is, “I can describe common characteristics of schizophrenia” to which the participant rated their agreement on a 4-point scale of: strongly disagree, disagree, agree and strongly agree. Ratings were conducted both before and after training. (See appendix A for the pre-training questionnaire.) Satisfaction was measured both quantitatively and qualitatively. Quantitative analysis involved participants rating their satisfaction with 7 aspects of the training (e.g., Duration of the training session) on a 4-point scale (where 1 = poor/completely inadequate and 4 = excellent). In the qualitative analysis, participants were asked their general comments, as well as what they liked, learned, wished for, and will do differently as a result of the training. (See appendix B for the post-training questionnaire.)

Follow-up interviews. The follow-up interview covered both the participants’ perceived knowledge and skill development and their comfort level and readiness to train other service providers. The interview focused on eight issues relating to satisfaction with the training, perceived knowledge and skills gained, and the extent participants had been able to use the training. More specifically, the interview consisted of eight questions asking about knowledge and skills gained, whether needs and expectations were met; if the



training was effective; what might be done differently in the future: if the interviewee had offered, or had plans to offer, training since the February pilot; how subsequent training had gone; how prepared and comfortable they were with offering training; and what organizational opportunities and barriers existed that might support or hinder their involvement in offering further training. (See appendix C for the complete interview format).

Procedures Pre and post questionnaires were provided to participants by the course facilitator the day of the training. This included a short script asking participants if they were interested in participating in a follow-up interview. Informed consent was obtained at that time and participants were informed of their right to refuse to participate (See Appendix D for informed consent form.) Follow-up interviews began two months after the training pilot in February 2003 to allow the participants some time to use the training in their work and offer subsequent trainings. Participants who indicated on the questionnaire they were open to learning more about the follow-up interview were contacted by phone or email and scheduled a time to be interviewed at their office. Interviews lasted an average of 40 minutes. Interviews were informal but centered on key questions identified previously. Interviews were taped to allow the evaluation team to type a summary of the discussion and record specific quotations. The participant’s responses were reviewed and a summary was written, quotations were used as examples of specific themes that emerged.

RESULTS

Satisfaction with Training. Quantitative Results. Participants were asked their views and opinions about the session through rating 7 items on a 4 point scale where 1 = poor/completely inadequate and 4 = excellent. They also responded to several open ended prompts as discussed below. Results of responses to these quantitative items are presented in Tables 1 to 4.



Table 1. Opinions on mental health module.

% of respondents1 (n)

Item 1 2 3 4

Mean

1. Duration of the training session 0 (0)

0 (0)

42.9 (3)

57.1 (4)

3.57

2. Quantity and quality of content 0 (0)

0 (0)

28.6 (2)

71.4 (5)

3.71

3. Handouts and overheads 0 (0)

0 (0)

28.6 (2)

71.4 (5)

3.71

4. The variety of tasks 0 (0)

0 (0)

57.1 (4)

42.9 (3)

3.43

5. Instructor’s knowledge of the subjects 0 (0)

0 (0)

42.9 (3)

57.1 (4)

3.57

6. Instructors responsiveness to participants’ needs

0 (0)

0 (0)

28.6 (2)

71.4 (5)

3.71

7. Instructors’ interaction with the participants 0 (0)

0 (0)

42.9 (3)

57.1 (4)

3.57

1 Percentage of respondents choosing a response alternative where alternatives are presented on continuum of 1 = poor/completely inadequate and 4 = excellent. Table 2. Opinion on substance use module.


Item 1 2 3 4

Mean


0 (0)

50 (5)

50 (5)

3.50


10 (1)

50 (5)

40 (4)

3.30


0 (0)

50 (5)

50 (5)

3.50


40 (4)

10 (1)

50 (5)

3.10


0 (0)

30 (3)

70 (7)

3.70


0 (0)

0 (0)

50 (5)

50 (5)

3.50


0 (0)

30 (3)

70 (7)

3.70

1 Percentage of respondents choosing a response alternative where alternatives are presented on continuum of 1 = poor/completely inadequate and 4 = excellent.



Table 3. Opinions on concurrent disorders module.


Item 1 2 3 4

Mean


0 (0)

21.1 (3)

76.9 (10)

3.77


0 (0)

15.4 (2)

84.6 (11)

3.85


0 (0)

15.4 (2)

84.6 (11)

3.85


0 (0)

38.5 (5)

61.5 (8)

3.62


0 (0)

0 (0)

100 (13)

4.00


0 (0)

0 (0)

7.7 (1)

92.3 (12)

3.92


0 (0)

7.7 (1)

92.3 (12)

3.92

1 Percentage of respondents choosing a response alternative where alternatives are presented on continuum of 1 = poor/completely inadequate and 4 = excellent. Table 4. Opinions on motivational interviewing module.


Item 1 2 3 4

Mean


0 (0)

0 (0)

100 (12)

4.00


0 (0)

16.7 (2)

83.3 (10)

3.83


0 (0)

25 (3)

75 (9)

3.75


8.3 (1)

8.3 (1)

83.3 (10)

3.75


0 (0)

25 (3)

75 (9)

3.75


0 (0)

0 (0)

16.7 (2)

83.3 (10)

3.83


0 (0)

25 (3)

75 (9)

3.75

1 Percentage of respondents choosing a response alternative where alternatives are presented on continuum of 1 = poor/completely inadequate and 4 = excellent.



Based on participants’ responses as shown in Tables 1 to 4, it is clear that the participants had a very positive view of the sessions. In fact, the only items that received any ratings less than a 3 on a scale of 1 to 4 were the “quality and quantity of content” and the “variety of tasks”, in the substance use module, and the “variety of tasks” in the motivational interviewing module. It can also be useful to determine which of the highly rated items are significantly different than the mean satisfaction level of a measure, particularly in the context of the current data in which participants are expressing high levels of satisfaction. In this way one can identify if there were items that represent relatively higher or lower levels of satisfaction. Such an analysis was conducted within each of the satisfaction results for the 4 modules with the following results:

- In the case of relatively higher levels of satisfaction, 3 items were

identified. In the concurrent disorders module, satisfaction with instructor’s knowledge was significantly higher than the mean satisfaction for all concurrent disorders items (t = 2.69, p < .02). In the substance use module satisfaction with instructor’s interaction was significantly higher than the mean satisfaction for all substance use items (t = 3.07, p. =.02). And, in the motivational interviewing module satisfaction with the duration of training was significantly higher than the mean satisfaction for all motivational interviewing items (t = 2.77, p. < .02).

- Only one item was significantly lower than the mean for all items in a

module. This was in the concurrent disorders module, where satisfaction with the “variety of tasks” item was significantly lower than the mean satisfaction for all concurrent disorders items (t = -2.21, p. < .05).

Qualitative Responses. In the post training satisfaction questionnaire, participants were given the opportunity to provide general comments and to indicate what they learned, liked, wished for, and would do in the future regarding the training. (For the actual comments by participants see appendix E.) In the mental health module, six participants provided general comments. These were overwhelmingly positive and referred to the module content, the organization and knowledge of the instructors, and the relaxed pace and fun learning environment. In contrast, one participant raised a concern with the clarity of the case studies and another indicated dislike for role -playing. In response to the prompt “what I learned… participants identified:

- the DSM IV (n =3), - the topic of denial (n=2),



- the grieving process (n=2), and - flip chart skills (n=1).

The participants liked a variety of aspects of the mental health training including, the presenters’ clarity and knowledge, the tone, pace and content of the training, the group interaction, and the use of small groups. The smooth transitions between presenters and the “combination of laughing and learning” were also mentioned. As well, learning about the grieving process was mentioned again. Only one participant in the mental health training module commented on what they wished for and that was to focus more on the practical approaches to specific symptoms such as what to say when dealing with clients’ delusions. Participants indicated they will do the following in the future:

- consider the grieving process, - focus more on hearing where a person is at in relation to their illness, - serve clients better using an increased knowledge base and new

insights, and - use the DSM IV criteria more often in discussions with medical

professionals. General comments in the substance use module suggested some difficulties with the amount of information and the way it was presented. Participants commented it was, “a heavy day” with a lot of information covered and a lack of variety in the means it was covered. Five participants commented that they had learned about human drives, one also added they learned about triggers and consequences. One participant commented that they learned that they needed to learn more about concurrent disorders. Aspects of the training that they liked included the following :

- the atmosphere created by facilitators (specifically their calm demeanor,

- their presence and the way they kept the information flowing and on time), that facilitators were well prepared,

- the case reviews, and - the “game style” to the exercise “contributors to substance use”. One participant wished there were more statistics on substance use in Ottawa. Another wished for more flow in the timing of the groups. Participants indicated that in the future they will:

- apply harm reduction,



- be more informed, - consider the relationship between personality disorders and

substance use, - do a more comprehensive and skilled assessment, and - have a greater appreciation of challenges and interventions in

substance abuse treatment. General comments on the post training satisfaction questionnaire for the concurrent disorders training were very positive. The training was described as: ‘interactive’, ‘creative’, ‘refined’, ‘well articulated’, ‘flowing well’, and composed of ‘good content’. Twelve participants commented on what they had learned including:

- integrated treatment, - linking the stages of change and the stages of treatment, and - assessment skills such as Hunter’s assessment.

Many participants mentioned the video as something they liked. Similar to the comments on what was learned, Hunter’s assessment was mentioned, as were the stages of treatment and change. The small group and debating exercises were also noted. Participants commented they wished for:

- fewer exercises and more information, - the chance to see more of the video, and - that their agency had the funding to provide treatment for concurrent

disorders on a larger scale.

Participants indicated that in the future they will: - be in a better position to help clients, - be more comfortable in assessing clients for potential concurrent

disorders, - be more confident in presenting information to others, - refine and redefine their practice through greater use of screening and

assessment tools, - focus more on empathy, - use Minkoff’s and Hunter’s approaches more, and - advocate for increased funding for concurrent disorders programming.

General comments on the motivational interviewing module were typically very positive as reflected in the use of words such as “excellent”, “loved” it, and



“good”. Comments indicated the training was well laid out and flowed nicely, the content was very relevant, in particular the information on best practices and the motivational interviewing book. One participant observed the need for more time in small-group exploration. Participants in the motivational interviewing module indicated they learned the following:

- implementing the techniques involved in motivational interviewing, - how to integrate motivational interviewing into substance use work, - to move clients along in change, - specific details such as the Payoff matrix, and - that it reinforced knowledge and skills they already had.

Participants liked: the delivery, flow, integration, variety of teaching approaches, group work, and “the whole day”. They wished for an interactive web site to continue sharing ideas, improvements to the addiction module, and more training on motivational interviewing. Participants indicated that in the future they were going to:

- be more effective with clients, - be more aware of their interviewing style, - include motivational interviewing in their work, - be more motivated to work with clients, - practice the motivational interviewing skills and strategies with clients, - use more open ended questions, and - rethink the ways they journey with clients.

Participants in the follow-up interview were also asked open-ended questions about their perceptions of the training, particularly whether or not it met their expectations and needs and the extent that they perceived the training to be effective. All 10 interviewees responded with a resounding yes when asked if the training met their needs and expectations. Several felt it did so because the information was practical, easily applicable to working with clients, and not too theoretical. In the words of the participants:

It gives you a sense of where to start with a client. I know it met my expectations because I am using the information that I got in my daily work and it pops up all over the place. I was doing something with a client the other day and I suddenly I thought, ‘oh I could do a little pay off matrix here’.



I was facilitating a concurrent disorders group and the training gave me more tools to use in the group itself.

Another reason the training was considered to have met their needs and expectations was that the trainees felt confident they could train others with the modules. They felt this way because the training process built confidence and was not too stressful. They also thought the information in the manuals was well presented and easily understood as demonstrated by the following observations:

The trainers’ guides to the modules explicitly addressed how to present the information. The manuals presented complex information with surprising clarity. The manuals condensed a great deal of information while still managing to effectively capture the problem.

One participant mentioned that they knew the training met their expectations by the fact that the organizers were open to suggestions to modify the training material. Another participant indicated that they knew it met their expectations as it showed them where their gaps in understanding were. Some specific needs were not met. The intensity of the original ‘train the trainer’ (conducted over 5-7 consecutive days) did not meet several participants’ need to be available at work. As well, a couple of participants mentioned that content and process difficulties in the addictions module interfered with their needs being met. Finally, some concerns were raised regarding a need to further condense and simplifying the information for less experienced audiences. A caution to interpreting the above information on participants’ expectations was highlighted in one particular response in which a participant indicated that, because they were involved in putting the material together, their expectations were already informed by the material. As noted in the discussion of limitations below, several of the participants (at least 4 of the 10) were involved in the development of the training process and content. Trainees were asked if they found their training effective, how they gauged this, and what they would do differently to make it more effective. They all indicated that they found it effective. There was a considerable amount of praise for Elva the primary trainer in the ‘train the trainer’. The participants indicated that she worked with their strengths and weaknesses and created a positive, supportive learning environment. As one participants noted:

The training increased your self-confidence, it was relaxed and comfortable. Elva was extremely helpful and very knowledgeable about how to train trainers.



Another participant indicated that they knew the training had been effective as, in unrelated situations, they found themselves critiquing other trainers and applying techniques they had learned. When asked what they might do differently participants offered the following ideas: arrange for the trainees to participate in the concurrent disorder groups at CMHA so that they can observe how the information and skills are used; and, create a website allowing trainees to communicate back and forth after training. There was some concern with the amount, and complexity, of the information for more novice audiences. For example,

At times the information comes very quickly and I had some concerns that the beginning learner might have a hard time keeping up. We have talked about how we will need to watch the content and to work with the range of trainees in the audience.

Similarly, a few participants raised concerns with the time demands of participating in the training. For example,

The biggest problem is the time commitment to just attending the sessions…When you’re a direct service provider and they are taking you out of the field for two days clients get stressed out, it’s a sensitive subject to bring up with management as they say the training will make you a better worker. The expectation is on you to prepare your clients so they can rely on others while you’re not available.

Although mention was made of the addictions module having undergone changes there appear to be some ongoing concerns. One participant thought that information vital to understanding addictions was missing. Information such as descriptions of blackouts, rebound effects, the withdrawal process, relapse, post acute withdrawal syndrome, defense mechanisms, population differences in the metabolism of substances, the nature of denial and repression, and warning symptoms for relapse. Concern was expressed that the addictions module was overly focused on human drive theory and, although alternative theories were presented, it lacked a discussion of the controversies between, and possible integrations of, the theories. In the words of one participant:

I have to believe in what I am teaching and I wasn’t comfortable with the relapse prevention information. It looks very theoretical. From a practical treatment perspective I don’t think it provides the information that you would need for treatment.



Perceived Knowledge and Skills Gained Quantitative Results. Participants were asked to rate their perception of their knowledge on various aspects of the training both prior to and just following the training. Analysis of data from the pre- and post- 8-item measures of perceived knowledge gain1 indicates that, in all four modules (i.e., mental health, addictions, concurrent disorders, and motivational interviewing), participants reported a significant gain in perceived knowledge.2

1. Prior to the examination of changes to perceived knowledge, an analysis of reliability was conducted for each module to determine if the 8 items assessing knowledge could be considered as measuring one construct “overall knowledge”. Cronbach alphas were calculated for the pre and post responses for each of the 4 modules and are as follows: pre-measure of mental health knowledge (.75), post-measure of mental health knowledge (.80), pre-measure of substance use (.64), post-measure of substance use (.80), pre-measure of concurrent disorders (.64), post-measure of concurrent disorders (.59), pre-measure of motivational interviewing (.86), and post-measure of motivational interviewing (.76). Although the reliability coefficients for the pre-measure of substance use and pre- and post-measures of concurrent disorders are somewhat low (less than .65), deleting items to improve their reliability (above .67) did not affect the finding of significant differences between pre and post perceived knowledge.

2. Results of the T-test for significant change are as follows: perceived knowledge

increased from pre to post training in the mental health module (t = 3.6, alpha = .008), problematic substance use module (t = 11.1, alpha = .000), concurrent disorders module (t = 8.1, alpha = .000), and motivational interviewing module (t = 3.8, alpha = .007). (In this analysis, an alpha less than .05 is considered significant).



A more detailed, item-by-item, analysis indicates a few specific subject areas in which participants did not indicate a significant change in perceived knowledge.

In the mental health module, these items were:

Item 2 - “I can describe the common characteristics of schizophrenia”,

Item 3 – “I can describe behaviours associated with schizophrenia”, Item 6 – “I know several coping strategies for caregivers working with

clients with personality illnesses (disorders)”, and Item 7 – “I know many characteristics of people who are homeless in

Ottawa”. In the concurrent disorders module, one item showed no change:

Item 1- “I understand what is meant by the term concurrent

disorders”. The lack of significant difference between pre and post perceived knowledge is at times due to a “ceiling effect” where pre-training levels of perceived knowledge is already high. For example, in the case of item 1 for concurrent disorders all but 1 of the 12 participants completing this item indicated they strongly agreed on the pre training measure that they understood what is meant by the term concurrent disorder. Qualitative Results. In addition to quantitative questions on the pre- and post-training evaluation forms, participants in the follow-up interview were also asked about knowledge and skills gained during the training. Several of these participants mentioned the training had provided a “format”, “template”, or “structure” that improved their ability to conceptualize, discuss and use the knowledge that they already had. Similarly, some mentioned the training “reinforced”, “reviewed”, or “refined” their previous understanding. For example, one participant mentioned the training served as a reminder of the stages of change and treatment. Interviewees also mentioned knowledge gains on specific components of the training such as the payoff matrix and the DSM IV. Several participants noted knowledge gain in the area of addiction assessment tools. Depending on the participants’ background, different modules were mentioned as providing new knowledge gain. Thus, those from a mental health background (the majority) cited the motivational interviewing and addictions modules as knowledge gain. Those from an addictions background (fewer in number) mentioned the mental health module. Other areas of training mentioned included the following: homelessness in Ottawa; the practice scenarios for motivational interviewing; and, information on the complexity of concurrent disorders.



When asked about skills gained a few participants mentioned general skills in working with concurrent disorders, including how to engage people who are homeless, general listening skills, and ways to get clients talking. However, the discussion of skills overwhelmingly centered on the training skills originally taught to them in the “train the trainer” program in the late fall of 2002, and augmented in the February pilot. For example, one participant stated:

The most important piece was the how to train, the initial module on how to deliver training. I thought that was really good. I’ve done other ‘train the trainer’ programs and that was never part of it. It was always you jump right into how to deliver this material … but they never backed up and ensured that the facilitators were going to have a general knowledge of how to deliver the training.

Participants spoke of learning how to “engage”, “energize”, and “empower” a group; how to manage your internal critic when you forget something or make an error; how to convey enthusiasm in your voice and body; how to train colleagues who are experts in their own right; how to maintain confidence; and, how to manage time. Many participants also listed specific training skills involving the presentation of information on flip charts. Examples of such comments include the following two quotations:

This training teaches a whole variety of adult education training techniques and I feel a lot more competent as an educator. The training was relaxed and comfortable. It was very positive training, it helped build self-confidence. Elva (the training consultant) was extremely helpful and very knowledgeable about how to train trainers.

Typically, participants indicated that they gained skills and knowledge in both the process of training and in the content of concurrent disorders. As one participant stated:

Explaining to others is a good way to both really understand the material and refine the knowledge I have.

Transfer of Knowledge and Skills

The follow-up interview focused especially on the extent that training participants were able to use the knowledge and skills gained in the training program. In particular, questions were asked about the following: whether the interviewee had offered, or had plans to offer, training since the February pilot; how subsequent training had gone; how prepared and comfortable they were with offering training; and what organizational opportunities and barriers existed that might support or hinder their involvement in offering further training



Involvement in Further Training. Only two participants indicated they had been involved in further training. This training had occurred at CMHA in June 2003. A third was involved in the June training but had been interviewed before this occurred. Of the seven who have not been involved in any training thus far, four indicated they had plans to be involved in training in the fall of 2003, likely in September and/or November, but they were not aware of the details of the training (for example, the exact date, their exact role, or the number or composition of the participants). In general there was some uncertainty as to the numbers of trainees that would be in upcoming training with some suggesting they will be groups of 15 –20 and others suggesting 23-30 (a potential concern given opinions about the adequacy of the facility for groups exceeding 20 people). Lack of involvement in training did not appear to imply lack of interest (indeed most interviewees were enthusiastic about being involved in training) but was due to issues such as maternity /paternity leave, changing job demands, and busy schedules. In the two cases where the participants had an opportunity to provide training, they were asked how effective they thought the training they offered was and what they might do differently the next time they offered it. They responded that they felt their training sessions were effective (based in part on their observation of the trainees evaluation forms). Their experiences training with the manual suggest it deserves the enthusiasm expressed for it above. However, with regards to the content of the training offered, one trainer thought changes in the manual were needed and noted “the evaluations were good but the piece I offered is repetitive, there is a lot of review and I think it could be condensed further”. The training also identified some differences in beliefs about how far trainers are authorized to move from the directions given in the manual. For example,

My co-trainer was uncomfortable with any change from the way it was worded in the manual and I thought it was important to make the words my own in order to deliver them with authenticity and energy.

The training process in June appeared to run well with only a few, minor, administrative difficulties. The participants’ manuals had sections that were intended for trainers only, the evaluation forms were not provided to the trainers, the snacks arrived after the scheduled break time, the air conditioning was not working, and a minority of CHMA trainees were distracted by their jobs (a hazard of training on-site). As well, the trainees were composed of all CMHA workers and it was felt that a mix of experiences and organizations would have enriched the training.



Trainers from June raised a concern with the coordination of the many trainers involved over the 5 modules. For example, goals outlined in the first session could not be reviewed in the last session, as the documentation of the goals had not been passed on to the final day trainers. This experience echoes concerns raised by several trainers who had not yet offered training beyond the February pilot. They wondered if the training might feel incongruent given the use of so many trainers. Some suggested that one of the two trainers be involved from start to finish. Concern was also raised about the matching of trainers for any given module, suggesting a need to balance high-energy trainers with low-energy trainers to maintain optimal energy levels. Those not able to be involved in a formal training organized by CMHA were asked if they had opportunities to share the information informally. Several mentioned they were able to raise issues addressed in the training in other situations. For example, in group supervisions, in dealing with other disciplines such as professionals in the court system, in debriefing their managers on their training, and in discussions with team members and other professionals. It appears, however, that those trainers who are members of partner agencies were told they were not to share actual manual content. They reported agreeing to this when they signed a contract respecting CMHA ownership of the training materials. In general there appeared to be a lack of clarity around the freedom for trainers from partner agencies to use the training materials independently of CMHA. In discussing the adequacy of training facilities at CMHA, it was mentioned by one participant that trainers from partner agencies could access CMHA facilities and offer the training if they wished. It was unclear if trainers from partner agencies were aware of this option and how it related to copyright of the training materials. Some clarification would be helpful in promoting the dissemination of the training information. Readiness to Train Others. Not surprising, given they felt it was effective, all 10 interview participants indicated they felt, for the most part, ready and comfortable to train others. When asked this question, one responded “definitely”, five indicated yes, and three indicated yes with some hesitancy. The majority qualified their response by indicating they still had some concerns about their ability to ensure the flow of the presentation due to general performance anxiety and a need to increase their familiarity with the material. However, they felt these concerns would diminish with practice. In the words of one participant:

With 10 or 12 practice runs I will be comfortable. With increased familiarity I will be more animated and with practice I will improve my presentation skills, such as my vocal control.



Indeed, when asked about continuing personal goals most mentioned improving familiarity and comfort level through repeated delivery. Trainers felt that experience in delivering the material was important to the trainings becoming more effective. Several trainers noted they hoped to train soon and often in order to improve their delivery of the materials. As one individual stated:

To keep things fresh people probably do need to deliver the training periodically, that may be a good recommendation – that to do a good job trainers need the practice and that should somehow be built in so that we do not go stale.

In explaining why they felt comfortable offering the training the trainers reiterated what they had found effective about the training, that it is well laid-out, straight forward, and easy to read. They also mentioned the feeling that the support and resources are there if any assistance is needed.

That is one thing I have found about working in partnership with CMHA is that they are very available. I feel very supported by them. I get the impression they have a lot of experience partnering with other agencies.

In addition to improving familiarity with content and reducing anxiety, several other personal goals were mentioned including the following: practicing motivational interviewing; helping people move through the growth stages by matching stages of treatment with stages of change; being involved in future training on addictions; applying new knowledge in the concurrent disorder groups; gaining a better understanding the role of trauma in concurrent disorders; and, thinking further about the different types of mental illnesses. Finally, related to readiness and comfort in training, it should be noted that the commitment and enthusiasm for the training was readily apparent in almost all interviewees. This was true even in cases where the individual’s involvement required their own, unpaid time. Organizational Support for Offering Further Training. Interviewees were asked about factors that could serve as either barriers or opportunities to their participating in offering further training sessions. In particular they were asked about the alignment between the goals of the training and the goals of their organization; the adequacy of facilities for training; the encouragement and enthusiasm from management and peers for the topic of concurrent disorders and also for new ideas in general; the organization’s ability to recognize their involvement in something new; and, the availability of time. The alignment between the goals of the training and the organizations involved was for the most part a strong asset. Six interviewees indicated a good alignment. This is not surprising as the training is an initiative of CMHA and many of the people interviewed are managers at CMHA. One participant



indicated that the idea for the training grew out of a bottom-up strategic planning process at CMHA and thus is well aligned with the agency’s goals. It was explained that the strategic plan has identified the importance of working with the hardest to serve in the system and people with concurrent disorders can often be found in this group. There were, however, also a small number of participants who alluded to some difficulties in the alignment of training and organizational goals. Concern was raised that the energies directed into concurrent disorders would detract from time and energy for clients who have mental illness without addictions. In other situations it was noted that the trainer’s direct job goals might be well aligned but their host organization’s goals were not as they were not fully in agreement with a harm reduction philosophy. This illustrates a potentially important point, that trainers from partner agencies may be engaged in activities atypical from those carried out in their organizations and their continued involvement as trainers may need support. With regards to whether training facilities represented an asset or barrier there was mixed opinion. Several interviewees thought the facilities at CMHA were good. Several others thought the facilities were good with the exception of the availability of parking due to a limit on the number of consecutive hours one can park. Still others saw the facilities as inadequate but this often depended on the number of trainees. As one participant indicated, “the room is small for the 26 trainees that were present, that would be an absolute maximum”. Another said, “the room is too small, it is not adequate for more than 20 people”. Two participants mention the temperature of the room and that it was either “too hot or too cold.” Positive characteristics sited included that there is plenty of places to eat in the area and that it is easy to find. When asked about encouragement and enthusiasm from management and peers for involvement in training related to concurrent disorders. Seven participants indicated their management and peers were enthusiastic and encouraging regarding their involvement in this training. Another three participants mentioned how enthusiasm was mixed. One suggested that the idea of concurrent disorders was at first implemented too fast and met with resistance but that members of the organization were becoming more accepting of the topic as they have had a chance to see how the information on concurrent disorders could lead to positive results with clients. A trainer from a partner agency noted enthusiasm for the topic but not necessarily for their involvement with the training due to increased demands on others in their organization during their absence. When the topic of recognition for involvement was brought up it was often met with surprise. Some participants believed that their involvement in the training would be noted in their annual reviews, others were not aware of whether it would be but did not seem overly concerned with recognition. A couple of



interviewees made mention of the existence of verbal recognition, some referred to some talk of a certificate but were unclear of its status. Perhaps more effective than a certificate of training is the following idea raised by one participant from a partner agency:

I just had my review and my participation with this training wasn’t in there so I will be putting it in myself. What I did was copy the evaluations from my training and give them to my manager… CMHA could have a standard letter, sent to our managers or administrative director, saying “thank you for allowing this person to participate, this person successfully trained x number of people”. They could include copies of the evaluation forms. It needs to be sent from management to management, and then it gets heard.

Participants were split of whether the availability of time was a barrier or opportunity to utilizing and sharing their training. One felt time was not a barrier as involvement in training would be included in job descriptions and case loads would thus be reduced. Not everyone shared this perception. As one interviewee mentioned:

Time availability is difficult; there are a lot of things that take us out of the field already. Although we may be told that its ok to reduce case loads and spend more time with one client, when year-end comes we are asked, ‘why have you not engaged more people’... Hopefully a lot of notice will be given in requests to be involved in training.

Trainers from external agencies were more likely to mention difficulties with time commitment but remained optimistic about their ability to be involved.

It is time consuming but time is not a barrier as others cover you off. To get on the training, to be allowed to do the training I really had to push. My manager had concerns about how much time commitment there would be.

Finally, in discussing organizational opportunities and challenges two participants spontaneously mentioned the need for greater security in funding for the training. One indicated that funding would be needed to adapt and improve the training modules. Another spoke of the need for funding for concurrent disorder groups. Study Limitations The findings of the evaluation are limited by the fact that trainees were composed of individuals who had already participated in the original ‘train the trainer’ and



thus were also trained to deliver components of the course. Indeed they may have been trainers for one of the other modules in the pilot. It is also possible that a few trainees in any given module may have been trained to deliver that very module. Another confound was in the administration of the pre and post questionnaires during the February pilot. For some modules questionnaires were completed not only by the trainees but by the trainers as well. For the questionnaires completed by trainers, one can assume pre and post knowledge does not measure the impact of receiving the training but of delivering it and satisfaction ratings of one’s own training could be considered biased. Similarly, in considering the interview responses, it is important to clarify that most of the comments on the knowledge and skills gained pertained not only the February pilot but to the original 7days of training on ‘train the trainer’ in the fall/ winter of 2002. Indeed it is not possible to distinguish between these two training experiences. During the interviews, participants referred to both training experiences to draw conclusions about their training. Thus this evaluation of satisfaction and perceptions of the training captures the effects of ‘train the trainer’ activities in both the fall/winter of 2002 and the February 2003 pilot training sessions. In fact, given many of the participants in the evaluation were involved in the development and management of the process and content of the training, it is not entirely accurate to say that their knowledge and experiences are reflective of only those two training events. Clearly, the multiple roles of many of the trainers make it somewhat difficult to be precise about the effects of the ‘train the trainer’ program or intervention. As noted earlier, it also adds a potential bias to the rating of their satisfaction with training. Finally, the evaluation findings are also limited by the relatively small sample size completing the pre- and post-training questionnaire (n=17) and participating in the follow-up interviews (n=10).

KEY FINDINGS AND RECOMMENDATIONS In summary, the evaluation has resulted in the following key findings. Recommendations are given following the findings on which they are based. Findings on Satisfaction - Participants reported having an overwhelmingly positive view of the training

sessions and the training manuals. One of the few exceptions to this is that, in the concurrent disorders module, satisfaction with the variety of tasks was



significantly lower than the mean satisfaction for all items in the concurrent disorders module.

- Individual responses on satisfaction items indicate one participant was less

satisfied with the quantity and quality of the substance use module, one was less satisfied with the variety of tasks in the motivational interview module, and 4 were less satisfied with the variety of tasks in the substance use module.

- Qualitative comments from the questionnaire suggest strong satisfaction with

the training including the content of the modules and the training style of the trainers. The comments do, however, point to some concerns with an excess of material in the substance use module and the need for a more varied way to presenting it. Comments also suggest that they learned a variety of new information, in particular, information related to assessment, conceptualization, and planning interventions.

- The training strongly met trainer’s expectations and needs. - There are some concerns over the congruency and integration over the five

days of training given the involvement of so many trainers. Similarly, there were some concerns with administrative details, such as the delivery of the evaluation forms at the appropriate time. However, it was felt the training remained largely effective.

Recommendation 1: It is recommended that the variety of tasks in the concurrent disorders module and substance use module be reviewed.

Recommendation 2: Overall, the training materials are solid and need not be changed at the present time. CMHA can feel confident that they have an interesting compilation of materials, an engaging means of conveying it, and have involved effective trainers in training this first team of trainers. The one consistent exception to this is the need for some changes to the substance use module. Several potential changes are suggested in the discussion of interview findings and these can serve as a starting place for further discussion on the appropriate content and delivery of this material.

Findings on Perceived Knowledge and Skills - Overall there was a significant gain in perceived knowledge from the training

in all four modules (mental health, substance use, concurrent disorders, and motivational interviewing).



- All but fi ve of the 32 items used to measure perceived knowledge indicated a gain in perceived knowledge. Of the five items, one did not indicate knowledge gain due to the fact that pre-training knowledge levels were already at the top of the scale.

- The participants reported pre-existing high levels of knowledge in the area.

They described how the training helped them structure this knowledge in a framework that has made the information more readily accessible to them.

- Assessment skills stood out as a particularly strong area of knowledge and

skill gain. The participants also reported receiving exceptional training in adult education. They indicated this ‘how to train others’ aspect of their training was unique in its thoroughness and in the engaging style with which it was delivered.

Recommendation 3: It is recommended that the material on characteristics of people who are homeless in Ottawa in the mental health module be reviewed for the extent that it provides new knowledge to service providers already sophisticated in this area and then be revised based on this review. Recommendation 4: In order to offer consistency through the modules, it is recommended that a staff member from CMHA be assigned the role of attending the training for the first and last few minutes of every day as well as being available through the day for any questions that arise. This need not be a subject matter expert but should be someone familiar with the format and procedural elements of the training (e.g., evaluation forms).

Findings on Transfer of Knowledge and Skills - Participants indicated they were comfortable offering training but see

themselves needing to become even more comfortable through offering more training sessions. As such, some concern was raised that there has not been sufficient opportunity to engage in training since the pilot session.

- In discussing participants’ opportunity to share the information they gained in

the training, there was some confusion over the freedom to use the training materials outside the context of a CMHA organized training session. Participants made reference to a signed agreement not to copy or disseminate manual material. The parameters of this agreement seemed unclear and had different interpretations for participants.

- Almost all the trainers exhibited high levels of commitment and enthusiasm,

the training appears to have stimulated their interest (or maintained an



already high level of interest) in this area and they have many personal goals for increasing their knowledge in this topic.

- There is a high opinion of CMHA by trainers from partner agencies, in

particular participants noted CMHA’s commitment to the topic of concurrent disorders and their skills in working in partnerships.

- The goal of the training aligns with most but not all of the trainers’ home

organizations’ goals. - Overall, all trainers received encouragement and enthusiasm from their

colleagues. Trainers from partner agencies were more likely to mention a lack of encouragement from managers than were CMHA trainers.

- It is unclear to many participants what recognition trainers will receive from

their involvement in the program. - There is some indication that, despite high levels of enthusiasm, trainers are

having some difficulty finding the time to be involved in offering training sessions.

- There are some concerns about the need for ongoing funding to ensure

participant involvement and adapt and improve the training materials. - There is mixed opinion about the adequacy of the training facilities at CMHA,

in particular there were concerns with parking and that the size of training groups remain at or below 20 people.

Recommendation 5: It is recommended that the people trained to deliver this training be given frequent opportunities to use these new skills and this knowledge in order to continue to maintain and improve what they have learned. Recommendation 6: It is recommended that CMHA’s position and directives to trainers regarding using training materials be clarified.

Recommendation 7: It is recommended that future ‘train the trainer’ activities consider explicitly discussing the following:

- using informal opportunities for sharing information from the training, and - finding a balance between trainers staying true to the content of the material in the manual and developing their own presentation style.



Recommendation 8: It is recommended that CMHA provide a letter of recognition to managers of the trainers from partner agencies subsequent to their offering training. As suggested by a participant, this letter could include an indication of the numbers of people trained and a summary of the outcome of this training (e.g., copies of the evaluation forms). Such a letter could also be sent to mangers of trainers staffed at CMHA, for notation in their annual review.

Recommendation 9: It is recommended that in using the training facility at CMHA, the size of training groups be limited to 20 people or less. Recommendations for Future Evaluations: Recommendation 10: It is recommended that the four point scales used by CMHA to assess participant’s opinions on the training be changed to a five-point scale in order to increase variability. It is also recommended that anchors be presented for all of the response alternatives. The items may want to explicitly ask about level of satisfaction with different aspects of the training (e.g., quite dissatisfied, mildly dissatisfied, mixed feelings, mostly satisfied and very satisfied).

Recommendation 11: It is recommended that a 5-point response scale for evaluating perceived knowledge be used pre- and post-training in order to better distinguish between the level of knowledge on specific topics and to better capture knowledge gain.



Appendix A – Pre-Training Questionnaires

Name: ______________________ Concurrent Disorders Training Program Concurrent Disorders Session: Pre-form

The purpose of this questionnaire is to gather information about your perceived level of knowledge about concurrent disorders. Please put your name on the questionnaire so that your responses can be matched to a questionnaire to be completed at the end of the module. This information will be used strictly for the purposes of evaluating the training program. No one other than the evaluation team members will see your answers. Your answers will be pooled with those of other participants for the purposes of the evaluation. Thank you for your participation. Indicate the extent to which you agree or disagree with the following statements describing your knowledge about concurrent disorders. For each of the following indicate by circling ONE option: SD=Strongly Disagree; D=Disagree; A=Agree; SA=Strongly Agree 1. I understand what is meant by the term concurrent disorder. 2. I am aware of the prevalence of concurrent disorders. 3. I can identify the elements of an integrated treatment system for concurrent disorders. 4. I understand why concurrent disorders are difficult to treat. 5. I am aware of several helpful responses if a client with a concurrent disorder were to use a substance after having abstained for some time. 6. I understand what is meant by a “harm reduction approach” in the context of people who are homeless and have concurrent disorders. 7. I am aware of the effects of combining substance use and medications used to treat mental illness. 8. I can differentiate among persuasion, treatment and relapse prevention stages of treatment for concurrent disorders.

SD D A SA SD D A SA SD D A SA SD D A SA SD D A SA SD D A SA SD D A SA SD D A SA



Name: ______________________

Concurrent Disorders Training Program

Motivational Interviewing Session: Pre-form The purpose of this questionnaire is to gather information about your perceived level of knowledge about motivational interviewing. Please put your name on the questionnaire so that your responses can be matched to a questionnaire to be completed at the end of the module. This information will be used strictly for the purposes of evaluating the training program. No one other than the evaluation team members will see your answers. Your answers will be pooled with those of other participants for the purposes of the evaluation. Thank you for your participation. Indicate the extent to which you agree or disagree with the following statements describing your knowledge about motivational interviewing. For each of the following indicate by circling ONE option: SD=Strongly Disagree; D=Disagree; A=Agree; SA=Strongly Agree 1. I understand what is involved in the intervention

strategy known as “motivational interviewing”. 2. I am aware of several techniques/approaches for

building a client’s motivation for change. 3. I am aware of traps to avoid when attempting to foster

motivation in others. 4. I know several strategies to handle client resistance. 5. I know how I would help clients set goals in a way that

they are more likely to achieve them. 6. I know how I would help a client make self-

motivational statements. 7. I know how to use a “Pay-Off Matrix Worksheet”. 8. I know several approaches to triggering a client’s

motivation to change.




Name: ______________________

Concurrent Disorders Training Program Problematic Substance Use Session: Pre-form

The purpose of this questionnaire is to gather information about your perceived level of knowledge about substance use. Please put your name on the questionnaire so that your responses can be matched to a questionnaire to be completed at the end of the module. This information will be used strictly for the purposes of evaluating the training program. No one other than the evaluation team members will see your answers. Your answers will be pooled with those of other participants for the purposes of the evaluation. Thank you for your participation. Indicate the extent to which you agree or disagree with the following statements describing your knowledge about substance use. For each of the following indicate by circling ONE option: SD=Strongly Disagree; D=Disagree; A=Agree; SA=Strongly Agree 1. I can recognize the symptoms of problematic substance

use. 2. I can describe the impact of substance use on an

individual’s physiology. 3. I can describe the impact of substance use on an

individual’s emotions. 4. I have a good understanding of the links between

substance use and mental illness. 5. I am aware of the four major drives in every human

and the importance of balance among them. 6. I can describe the effect of substance use on the human

drive for raw pleasure. 7. I have a working knowledge of Canadian trends in

substance use. 8. I am aware of the primary contributors to substance use

(psychological, physical, and social).




Name: ______________________

Concurrent Disorders Training Program Mental Health Issues Module: Pre-form

The purpose of this questionnaire is to gather information about your perceived level of knowledge about metal health issues. Please put your name on the questionnaire so that your responses can be matched to a questionnaire to be completed at the end of the module. This information will be used strictly for the purposes of the evaluation the training program. No one other than the evaluation team members will see your answers. Your answers will be pooled with those of other participants for the purposes of the evaluation. Thank you for your participation. Tim Aubry, University of Ottawa Principal Evaluator, Concurrent Disorders Program Indicate the extent to which you agree or disagree with the following statements describing your knowledge about mental health issues. For each of the following indicate by circling ONE option: SD=Strongly Disagree; D=Disagree; A=Agree; SA=Strongly Agree 1. I understand how diagnoses are made using the

Diagnostic and Statistical Manual of Mental Disorders. 2. I can describe common characteristics of

schizophrenia. 3. I can describe behaviours associated with

schizophrenia. 4. I am aware of the treatment options for affective

illnesses. 5. I know what the various personality disorders are. 6. I know care-giver coping strategies for clients

diagnosed with personality disorders. 7. I know what the characteristics of homeless people in

Ottawa are. 8. I understand the factors influencing the health of

homeless people in Ottawa.




Appendix B – Post-Training Questionnaires

Name: ______________________

Concurrent Disorders Training Program Concurrent Disorders Issues Module: Post-form

The purpose of this questionnaire is to gather information about your perceived level of knowledge about concurrent disorders now that you have completed the module. In addition, we are interested in your views and opinions about the module. Please put your name on the questionnaire so that your responses can be matched to the questionnaire you completed before the module. This information will be used strictly for the purposes of evaluating the training program. No one other than the evaluation team members will see your answers. Your answers will be pooled with those of other participants for the purposes of the evaluation. Thank you for your participation. A. Indicate the extent to which you agree or disagree with the following

statements describing your knowledge about concurrent disorders. For each of the following indicate by circling ONE option: SD=Strongly Disagree; D=Disagree; A=Agree; SA=Strongly Agree.

1. I understand what is meant by the term concurrent disorder.

2. I am aware of the prevalence of concurrent disorders.

3. I understand the elements of an integrated treatment system for concurrent disorders.

4. I understand why concurrent disorders are difficult to

treat.

5. I am aware of several helpful responses if a client with a concurrent disorder were to use a substance after having abstained for some time.

6. I understand what is meant by a “harm reduction

approach”.

7. I am aware of the effects of combining substance use and medications used to treat mental illness.

8. I can differentiate persuasion, treatment and relapse

prevention stages of treatment for concurrent disorders.




B. VIEWS AND OPINIONS ABOUT THE SESSION

1. Please indicate with a checkmark your opinion of the following items (where 1 = Poor/completely inadequate and 4 = Excellent):

1 2 3 4 Duration of the training session Quantity and quality of content Handouts and overheads The variety of tasks Instructor’s knowledge of the subjects Instructor’s responsiveness to participants’ needs Instructors’ interaction with the participants

2. Specific comments about any of the items above:

3. I learned … 4. I liked … 5. I wish… 6. As a result of what I’ve learned today, in the future I’m going to …



C. BACKGROUND INFORMATION

1. Highest level of education obtained (check one)

____ High School ____ Diploma _____ Bachelors _____ Masters ____ Doctorate ____Other (specify) ___________________________

2. Primary specialization (check one) ____ mental health ____ addictions _____ concurrent disorders

____ other (specify)____________

3. In what area is your current professional role? ____ mental health ____ addictions

_____ other (specify) _________________

4. How long have you been in this role? ________ (years) 5. In which sessions are you enrolled? (check as many as apply)

______ Mental Health Issues _____ Problematic Substance Use ______ Concurrent Disorders ______ Motivational Interviewing

In addition to the information you have provided us in the present survey, follow-up interviews are being conducted with a subgroup of participants from the training session. These interviews will be conducted in-person by a member of an evaluation team from the Centre for Research on Community Services at the University of Ottawa. They will take 45-60 minutes and will focus on your impressions of the training and the extent that you have been able to transfer the knowledge and skills coming out of the training into your current job. Are you interested in potentially participating in this follow-up interview by finding out more about it from a member of the research team at the University of Ottawa? Yes: ______ No: ______ If yes, please indicate where and how you can best be reached: Tel. no: ____________________ e-mail (if available): _____________________

THANK YOU FOR YOUR INPUT



Name: ______________________

Concurrent Disorders Training Program Motivational Interviewing Issues Module: Post-form

The purpose of this questionnaire is to gather information about your perceived level of knowledge about motivational interviewing now that you have completed the module. In addition, we are interested in your views and opinions about the module. Please put your name on the questionnaire so that your responses can be matched to the questionnaire you completed before the module. This information will be used strictly for the purposes of evaluating the training program. No one other than the evaluation team members will see your answers. Your answers will be pooled with those of other participants for the purposes of the evaluation. Thank you for your participation.

A. Indicate the extent to which you agree or disagree with the following statements describing your knowledge about motivational interviewing. For each of the following indicate by circling ONE option: SD=Strongly Disagree; D=Disagree; A=Agree; SA=Strongly Agree. 1. I understand what is involved in the intervention strategy

known as “motivational interviewing”. 2. I am aware of several techniques/approaches for building a

client’s motivation for change. 3. I am aware of traps to avoid when attempting to foster

motivation in others. 4. I know several strategies to handle client resistance. 5. I know how to help clients set goals in a way that they are

more likely to achieve them. 6. I know how I would help a client make self-motivational

statements. 7. I know how to use a “Pay-Off Matrix Worksheet”. 8. I know several ways, in a brief intervention, to trigger a

client’s motivation to change.















____ other (specify)____________


_____ other (specify) _________________







Name: ______________________

Concurrent Disorders Training Program Problematic Substance Use Session: Post-form

The purpose of this questionnaire is to gather information about your perceived level of knowledge about substance use now that you have completed the module. In addition, we are interested in your views and opinions about the module. Please put your name on the questionnaire so that your responses can be matched to the questionnaire you completed before the module. This information will be used strictly for the purposes of evaluating the training program. No one other than the evaluation team members will see your answers. Your answers will be pooled with those of other participants for the purposes of the evaluation. Thank you for your participation. A. Indicate the extent to which you agree or disagree with the following

statements describing your knowledge about substance use. For each of the following indicate by circling ONE option: SD=Strongly Disagree; D=Disagree; A=Agree; SA=Strongly Agree.

1. I am aware of the symptoms of problematic substance use.

2. I would recognize the symptoms of problematic substance use.

3. I can describe the impact of substance use on an individual’s physiology.

4. I can describe the impact of substance use on an individual’s emotions.

5. I have a good knowledge of the links between substance use and mental illness.

6. I can describe the effect of substance use on the human drive for raw pleasure.

7. I have a working knowledge of Canadian trends in substance use.

8. I am aware of the primary contributors to substance use (psychological, physical, and social).















____ other (specify)____________


_____ other (specify) _________________







Name: ______________________

Concurrent Disorders Training Program Mental Health Issues Module: Post-form

The purpose of this questionnaire is to gather information about your perceived level of knowledge about metal health issues now that you have completed the module. In addition, we are interested in your views and opinions about the module. Please put your name on the questionnaire so that your responses can be matched to the questionnaire you completed before the module. This information will be used strictly for the purposes of the evaluation the training program. No one other than the evaluation team members will see your answers. Your answers will be pooled with those of other participants for the purposes of the evaluation. Thank you for your participation. Tim Aubry, University of Ottawa Principal Evaluator, Concurrent Disorders Program

A. MENTAL HEALTH ISSUES Indicate the extent to which you agree or disagree with the following statements describing your knowledge about mental health issues. For each of the following indicate by circling ONE option: SD=Strongly Disagree; D=Disagree; A=Agree; SA=Strongly Agree. 1. I understand how diagnoses are made using the Diagnostic and Statistical Manual of Mental Disorders. 2. I can describe common characteristics of schizophrenia. 3. I can describe behaviours associated with schizophrenia. 4. I am aware of the treatment options for affective illnesses. 5. I can describe several personality illnesses (disorders). 6. I know several coping strategies for care-givers working with clients with personality illnesses (disorders). 7. I know many characteristics of people who are homeless in Ottawa. 8. I understand the factors influencing the health of people who are homeless in Ottawa.















____ other (specify)____________


_____ other (specify) _________________







Appendix C - Interview Consent Form

May 1, 2003

INFORMED CONSENT FORM FOLLOW-UP INTERVIEW WITH TRAINING PARTICIPANTS

Dear Training Program Participants: We would like to invite your voluntary participation in the evaluation of the pilot training program on concurrent disorders. The evaluation is being carried out by members of an evaluation team from the Centre for Research on Community Services, University of Ottawa. The general purpose of the evaluation is to determine how well the training program is being implemented and your perceptions of the extent it has prepared you to provide training in the area. Your participation would involve an in-person interview that would last about 1 hour. The interview will be private and with your permission, we would like to tape record it so as to ensure accuracy. The interview will ask questions about your impressions of the effectiveness of the training in which participated, how you are currently using it, your readiness for using the training to train others, and the conduciveness or receptivity of your agency to incorporate the training you received into its service delivery. All information collected will be treated as CONFIDENTIAL. No identifying names of persons or organizations will appear in any reports arising from this project. You are under no obligation to participate in the follow-up interviews and your decision to decline this invitation will result in no negative consequences related to your work nor will it be communicated to anyone. If you do decide to participate, you can also withdraw at any time or refuse to answer any questions without any negative consequences. The University of Ottawa requires its researchers to obtain formal consent from those participating in research. Your signature at the bottom of this letter would serve such a purpose. (A copy for your files is attached) You are free to withdraw from the study at any time. If you have any questions about the study these may be raised now or later by phoning at 562-5800 Tim Aubry (ext. 4815) or Brad Cousins (ext. 4036). Sincerely, Tim Aubry, Ph. D., Brad Cousins, Ph. D., Gordon Josephson, Ph.D. Candidate I agree to participate in the study. Signature: _____________________ / Date: _________________________ Researcher: Signature: _____________________ / Date: _________________________



Appendix D - Interview Questions

Provider of Service/Trainee Interview Guide Interviewer: Interviewee: Date: Location: MODULE(S) TAKEN: ________________________________________

This interview is a follow up to the training program that you took in February at the CMHA. We are interested to know about your knowledge and skill development in this area, and specifically about your comfort level in training other providers of service.

1. What were the most important things that you learned during the training module? What kinds of knowledge did you develop? What sorts of skills? Be as specific as you can.

• Knowledge

• Skills

2. Did the training program meet your needs and expectations? Why or why not



3. Since you took the module, have you trained other providers of service in this area? If yes, provide details (e.g., number and composition of trainee group, duration, materials, location).

4. How effective was this training? On what do you base your answer to this question?

5. What would you do differently next time?

6. Do you feel well prepared and comfortable about the prospect of training others in this area? Why or why not? Identify specific knowledge and/or skill areas that you would see as being personal growth areas (i.e., requiring further development)



7. Do you have plans for training other providers of service in this area? Why or

why not? If yes, describe the training that you have in mind (e.g., number and composition of trainee group, duration, materials, location).

8. Concerning the content of training that you took, comment on the extent to which each of the following may have (or would be likely to) assets or barriers to on ongoing training in these areas. Describe the nature of the influence.

• The extent to which goals of the training are aligned with your organization’s policies and mission

• The adequacy of the facilities and supports for running the training • Organizational and administrative encouragement for personnel to try new

practices and strategies here • Enthusiasm shown by colleagues/peers. • Extent to which leadership within the organization is supportive of change • Organizational provision of formal or informal mechanisms for

recognizing the success of individuals • Availability of time



Appendix E – Verbatim Satisfaction Comments from Questionnaires Mental Health General Comments:

- Great content, instructors were well organized and presented as confident and knowledgeable.

- Group size of 12-15 people is great! familiarity of group was good. - Not an overload, easy to digest, case studies could be a bit clearer - Relaxed and fun learning environment, the day seemed to go quickly. - Very well done, great preparation, great pace, relaxed - Well developed, two afternoon breaks a good idea, hated the role

playing, probably not as necessary as most agencies send staff in NCVI training anyhow

What I learned: - About the grieving process - Diagnostic criteria Axes I-V what doctors are looking for. - Expanded knowledge of DSM IV - Stages of grieving and DSM IV - The importance of flip chart recording and of displaying participants

work on them - To understand denial as a stage. - Well, I made some connections I hadn't made before, approaches to

denial and Prochaska and DiClement

What I liked: - All presenters were clear and knowledgeable - Group interaction - Learning and laughing in combination. - Section regarding grieving (this area is often overlooked in training

regarding mental illness). - Smooth transition between the presenters. - The tone, pace, content - The use of small groups to encourage participants participation and

involvement.

What I wish: - We could focus somewhat on practical approaches to specific

symptoms (how to deal with paranoia, delusions, what to say, what not to say).

In the future I am going to:

- Consider the grieving process - Focus more on hearing where a person is at in relation to their illness

before giving advice to workers.



- I have increased my knowledge base and insights which, will hopefully help me to better serve my clients.

- Re-enforced what I know, a bit more concrete information - Use DSM IV criteria more often in my discussions/contacts with

medical professionals.

Substance Use General Comments:

- A lot of information to take in, not enough interacting in morning although I don’t have any suggestions on what to do about it.

- I think more team tasks in the morning would be beneficial, sometimes difficult in morning to read alone.

- It is a heavy day - lots of information, information presupposes a basic level of knowledge, may or may not be a problem.

- Little too much time group work. - Overhead writing is too small (ok with handouts), overhead of cross

over effect would be helpful to follow along. - The tasks and activities (work groups and flip charts) are very similar. - Would like to focus on scenario of when a person can't get into existing

treatment - what can we do? more emphasis on assessment?

I learned: - About the 4 human drives. - I did learn a lot, thought presenters were knowledgeable, will probably

need less next time. - Nature of substance use, human drives, approaches - That I need to know more about interactions between drug addictions

and mental illness. - The concept of human drives. - Theory and model on human drives. - Triggers, actions consequences, about human drives.

I liked:

- Atmosphere created by facilitators. - Obviously lots of work and preparation put into presentation by all

facilitators. - The case reviews. - The exercise "contributors to substance use", the game style to it. - The trainers' presence, calm demeanor, flow. - The way they kept it on time.

I wish:

- More current Ottawa based stats were available (not your problem).



- Need more flow around time of groups.

In the future I am going to: - Apply harm reduction. - Be more informed. - Consider the relationship between PD and SA. - Do a more comprehensive multi-modal assessment. - Greater appreciation of challenges and interventions in substance

abuse treatment. - I liked the flow of the day. - Practice better assessment skills in field.

Conduct Disorder General Comments:

- A good use of best (better) practices. - Great job all round. - Session was interactive, well prepared, creative, it flows better and is

more refined. - Trainers were great, content was great. - Very knowledgeable presenters, good content, well articulated. - Very nicely done-flowed well.

I learned:

- About components of an integrated treatment plan. - About comprehensive integrated treatment. - About Minkoff's principles for treatment. - About stages, phases, treatment principles using the Hunter's

assessment. - Comprehensive information. - Integrated treatment, more knowledge of best practices manual,

assessment = Hunter. - Lots. - More about all the things listed on the first page. - More about substance abuse. - New assessments, screening tools – Hunter - Prevalence of concurrent disorders, connection between stages of

change and stages of treatment. - To link stages of change with stages of treatment.

I liked:

- All the presenters approach to concurrent disorders. - Hunter’s questionnaire. - I loved the video! - Stages of treatment/change, harm reduction.



- The debate was inspired. - The flow and diversity of material. - The inclusion of the best practices book. - The video, the group exercise. - The visual stimulation. - Video, small group work.

I wish:

- Fewer exercises, more information. - I could have seen more of the video. - I was in Jamaica learning this. - We had money to provide this kind of treatment on a large scale.


- Be in a better position to help my clients. - Be more comfortable in assessing clients for potential concurrent

disorders. - Be more confident in presenting information to others. - Change the way I work with clients (tools, instilling empathy etc.). - Implement Minkoff's approaches more. - Include in ongoing training. - Practice using empathy and assessing more frequently. - Refine and redefine my practice. - Screen more carefully for concurrent disorders, advocate for increased

money for concurrent disorders programming . - Use Hunter assessment, stages of change.

Motivational Interviewing General Comments:

- Excellent training, well laid out. - Excellent. - Good timing, important information. - Loved the introduction of Best Practices and Integration of motivational

interviewing book. - Nice flow, more time in small groups to explore things. - Tasks were ok considering the topic.

I learned:

- How to implement Motivational Interviewing. - How to move clients along in change. - More than I thought I would. - Payoff matrix, reinforced knowledge I already had. - Reinforced reflective listening skills.



- Strength, commitment to change, managing difficult situations - Techniques around motivational interviewing clarified approaches in

my daily practice. - Techniques strategies and objectives of interviewing. - To integrate motivational interviewing and substance use practice.

I liked:

- Almost everything. - The delivery method. - The flow, the focus, the way it all fits together. - The group work, it was very practical and helpful. - The overall flow and presentation. - The topic in general. - The whole day. - Variety of approaches to teaching.

I wish:

- An interactive web site to continue sharing ideas. - I could have been here for all 5 days. - The addiction piece needs work. - The tasks were a bit different. - Would be interested in more motivational interviewing training.


- Be able to help my clients more effectively. - Be more aware of my interviewing style. - Include motivational interviewing in my daily practice. - More motivated to work with my clients. - Practice the skills in my work. - Practice these techniques and strategies on my clients. - Rethink some of the ways I journey with clients. - Use more open ended questions. - Work with clients towards change more effectively.

Documents

Evaluation of Concurrent Disorders “Train the …...‘train the trainers’ program on concurrent disorders. The term concurrent disorder refers to the presence of both a s evere