Pilot Program Evaluation Ice Effectsresources-library.caraniche.com.au/system/files/RPP_011... · Ice Effects Pilot Program Evaluation 4 Executive Summary This report details the

Ice Effects Pilot Program Evaluation

W:\Research & Professional Practice\Projects\Current Projects\Ice Project\Ice Program Development\6 Hour Ice Effects\Program Evaluation

0 of 46 Issue Date: June 17

Pilot Program Evaluation Ice Effects 6 hour Drug and Alcohol Program for Prisoners with Methamphetamine Issues

Research and Professional Practice

Version: 2.0

Issued: 4th May 2017

RPP ID: RPP011


1

Contents

Executive Summary .......................................................................................................... 4

Key findings .................................................................................................................................... 4

Recommendations ......................................................................................................................... 5

1 Who is Caraniche? ..................................................................................................... 6

2 Program Objectives, Aims and Theory ....................................................................... 7

3 Evaluation Framework ............................................................................................. 10

3.1 Aim ..................................................................................................................................... 10

3.1.1 Appropriateness ................................................................................................................. 10

3.1.2 Efficiency ............................................................................................................................ 10

3.1.3 Effectiveness ...................................................................................................................... 10

3.1.4 Impacts ............................................................................................................................... 11

3.1.5 Sustainability ...................................................................................................................... 11

3.2 Evaluation Methodology .................................................................................................... 11

3.2.1 Target Population ............................................................................................................... 11

3.2.2 Measures ............................................................................................................................ 12

3.2.3 Data Analysis ...................................................................................................................... 13

4 Results .................................................................................................................... 14

4.1 Program Demographics...................................................................................................... 14

4.1.1 Pilot evaluation duration ................................................................................................... 14

4.1.2 Total programs run and prisoners participated ................................................................. 14

4.1.3 Program attendance .......................................................................................................... 14

4.1.4 Participant Charactertistics ................................................................................................ 14

4.1.5 Mental health ..................................................................................................................... 15

4.2 Participant Experience ....................................................................................................... 17

4.2.1 AOD Programs Feedback Survey: Short Form .................................................................... 17

4.2.2 Participant Focus Group ..................................................................................................... 21

4.2.3 Knowledge Survey .............................................................................................................. 23

4.3 Facilitator Feedback ........................................................................................................... 25

4.3.1 Preliminary analysis ........................................................................................................... 25

4.3.2 Results ................................................................................................................................ 25

5 Discussion ............................................................................................................... 28

5.1 Appropriateness ................................................................................................................. 28

5.2 Efficiency ............................................................................................................................ 28

5.3 Effectiveness ...................................................................................................................... 29


2

5.4 Impacts ............................................................................................................................... 29

5.5 Sustainability ...................................................................................................................... 30

6 Key findings ............................................................................................................. 30

7 Recommendations .................................................................................................. 31

8 References .............................................................................................................. 33

9 Appendix ................................................................................................................. 34

Appendix A: Facilitator Feedback Booklet.................................................................................... 35

Appendix B: Participant Focus Group ........................................................................................... 40

Appendix C: Participant Satisfaction Survey ................................................................................ 41

Appendix D: DASS-21 .................................................................................................................... 43

Appendix E: Six Hour Ice Effects Knowledge Survey .................................................................... 44


3

List of Tables

Table 1. Description of Psychometrics .............................................................................................................. 13

Table 2. Breakdown of Participation per Prison Location ................................................................................ 14

Table 3. Cut-off Scores for DASS-21 Severity Labels ......................................................................................... 15

Table 4. Participant Satisfaction Mean Scores ................................................................................................. 18

Table 5. What Participants Liked About the Program ...................................................................................... 19

Table 6. Participants Perception of How the Program Could be improved ...................................................... 20

Table 7. Knowledge Survey Questions and Correct Reponses .......................................................................... 24

Table 8. Percentage of Correct Reponses per Participant ................................................................................ 24

List of Figures

Figure 1. Program Theory with Assumptions ..................................................................................................... 9

Figure 2. Participant Level of Depression Prior to Program Commencement .................................................. 16

Figure 3. Participant Level of Anxiety prior to Program Commencement ........................................................ 16

Figure 4. Participant Level of Stress prior to Program Commencement .......................................................... 17

Figure 5. Willingness to recommend the program ........................................................................................... 20

Figure 6. Percentage of Correct Reponses per Question .................................................................................. 25


4

Executive Summary

This report details the evaluation of Caraniche’s Ice Effects program, a 6-hour (2 session) psychoeducation

group program developed specifically for prisoners with a history of crystalised methamphetamine (ice)

use. This program was developed in response to an identified need for tailored interventions targeting the

distinctive health and criminogenic needs of methamphetamine users, within the broader context of

increasing methamphetamine-related crime in the community. The Ice Effects Program is one component

of a suite of ice-focused prison programs, which also include two longer programs (of 24 and 44-hour

duration).

The long-term objective of the Ice Effects program is to reduce the prevalence of ice use in a prison-based

population. The Ice Effects program aims to achieve this by educating individuals about the risks and post

withdrawal effects of ice use, providing self-management strategies to assist them to live ice-free following

release from custody, and providing a pathway to more intensive supports.

The purpose of the current report is to evaluate the design and implementation of the Ice Effects pilot

program, which was conducted in three Victorian correctional centres (the Metropolitan Remand Centre,

Marngoneet, & Middleton) between January and December 2015. In particular, this evaluation aims to

assess the appropriateness, effectiveness and efficiency of the program, and where possible comment on

program impact and sustainability. In addition, this report identifies areas for improvement, and highlights

the ongoing funding needs of the program.

One hundred and sixty-two adult male prisoners with a history of ice use completed the Ice Effects pilot

program. The evaluation consisted of a range of both quantitative and qualitative methodologies including

feedback from participants and program facilitators, and a Test of Knowledge Survey completed by

participants at the end of the program. Participants also completed a brief screening measure of

psychological distress (the DASS-21) prior to commencing the program.

Key findings and recommendations are summarised below.

Key findings

1. Completion rates were high. Of the 183 prisoners who commenced the program, 162 (88.5%)

completed it. The reasons for non-completion included court attendances, health reasons, and safety

issues. Just nine prisoners (5%) refused to attend the second (final) session;

2. In addition to the strong completion rates, participant engagement with the program was also reflected

in feedback from facilitators and participants. Facilitators reported that participants were highly

engaged in group discussions and activities. In the feedback surveys, participants agreed that they were

satisfied with the program, that it suited their needs and that the facilitators did a good job. Ninety-

eight per cent (n=99) of respondents who completed the feedback survey indicated that they would

recommend the program to others;

3. Overall, facilitators reported that the manual provided a useful structure and content, but stated that,

at times, the content was too complex for the target group, many of whom appeared to be living with

acquired brain injuries, and as such extra time was spent explaining concepts to participants.


5

Interestingly, in their feedback, participants agreed, or strongly agreed, with the statement, “I

understood the information provided in this program”, however this may be due to the extra efforts

made by facilitators, rather than the content of the manual. Both facilitators and participants indicated

that additional visual and audio-visual learning aids would be beneficial;

4. The volume of content to be covered was considered to be too great for the scheduled program time (6

hours), and did not allow sufficient time for group discussions, which were viewed as a valuable and

engaging component of the program;

5. One of the key aims of the Ice Effects program was to increase participants’ knowledge about the

effects and risks of using ice. A Knowledge Survey was provided to participants at the end of the

program, of which all items were answered correctly by between 73% to 88% of those participants who

returned the Knowledge Survey. It is difficult to establish the extent to which this knowledge was new

knowledge, given that the survey wasn’t administered prior to the commencement of the program.

However, the results do indicate a reasonably good level of knowledge about the key aspects of the

program. It is unknown if participants’ decision to decline to complete the test (74 participants did not

return the survey) was in any way related to their actual level of knowledge (be this high or low). This

ambiguity gives reason for caution when drawing inferences about program participants’ true level of

knowledge. It is noteworthy, however, that one facilitator reported that the participants in her group

found the Knowledge Survey confronting, as it was too much like a “class-room test” which can be

highly off-putting to a forensic population. The knowledge gains indicated by the Knowledge Survey

were also confirmed by the feedback from facilitators and participants. Facilitators reported that as a

result of the program the participants were better able to understand the consequences of ice use and

related symptoms of withdrawal. In their feedback, participants either agreed or strongly agreed with

the statement “I have learned a lot from this program”;

6. Another key aim of the program was to increase participants’ motivation to change. In their responses

to the feedback survey, participants either agreed or strongly agreed that, “The program motivated me

to work on my problems”, and “As a result of the program I feel confident about tackling my

problems”. These responses suggest that the program was effective in enhancing motivation;

7. When asked how they thought the program could be improved, the most common response was that

they would like the program to go for longer, perhaps highlighting the need to provide a pathway to

additional supports for some participants;

8. Facilitators did not perceive that any resource constraints affected their ability to deliver the program,

stating that modifications to program content were made depending on the needs of the group. Only

one facilitator mentioned an operational disruption, with minimal negative effect on delivery of the

program.

Recommendations

1. The program manual be revised to ensure the content is simplified to suit the needs of an offender

population, incorporating additional visual/audio-visual learning aids. An additional focus on the

neurochemistry of ice use and coping strategies may also be beneficial and of interest to participants;


6

2. The volume of the program content be reduced to allow for additional time for group discussion, which

was perceived as a valuable and engaging component of the program by both facilitators and

participants. Consideration may also be given to moving the mental health component to the second

session, so as to reduce the volume of content to be covered in the first session;

3. Evaluation of outcomes may be further strengthened by incorporating a pre and post measure of

motivation or treatment readiness. The evaluation process should include a measure of participants’

motivation to engage in additional programs, such as evidence of referral to individual counselling or

higher dose treatment options post program completion. The Knowledge Survey in its current form

should be removed and alternative evaluation approaches developed that can effectively measure

changes in participant knowledge.

a. Update April 2017: It should be noted that the Knowledge Survey was removed from the

program test battery soon after the conclusion of the pilot testing phase. Caraniche has

recommended to Justice Health that the OQ Measures test suite provides a robust and

meaningful way of measuring program outcomes across a wide range of health and

criminogenic programs, but we are yet to receive a formal response to this proposal. It is

probable that the OQ Measures will provide a more suitable test battery for both clinical and

evaluation purposes. Further amendments to the Ice Effects program materials will be made as

part of a general review of programs planned for 2017-18, and following the completion of the

current evaluation process.

4. At the end of the Ice-Effects program, participants should be made aware that there are ice programs

of longer program dosage, and expressions of interest to participate in these programs should be

obtained. This is current practice and the findings of this evaluation reinforce the importance of this.

Participants in Ice Effects are encouraged to consider their subsequent treatment needs, including

participation in the 24-hour and 44-hour ice programs.

5. With the amendments indicated above, and in light of the positive results overall, it is recommended

that the 6-hour Ice Effects program be rolled-out for delivery within all Victorian prisons, as one

component of a suite of programs tailored to prisoners with a history of methamphetamine use.

a. Update April 2017: This recommendation has already been implemented and the Ice Effects

program is currently being offered, and run regularly, at relevant locations that have a KPI for

6-hour programs. It is noted that this program continues to be very highly subscribed, with

demand presumably driven by; the high rates of prisoners with ice-related drug issues, the

increasing number of remandees across numerous locations, and the continued increase in

churn and rapid movement between locations. This speaks to the need for the suite of short

health programs.

1 Who is Caraniche?

Caraniche Pty Ltd is a psychological consulting firm. Established in 1993, Caraniche has a long and well-

respected history as a provider of drug and alcohol services, prison treatment programs, research and


7

evaluation services, psychological counselling, Critical Incident Stress Management (CISM), Employee

Assistance Programs (EAP), and training to private, government and non-government organisations and the

community sector.

Caraniche specialises in offering high-quality, innovative and professional services in challenging and

complex environments, particularly within the criminal justice setting and has achieved this by

incorporating research, evaluation, consulting and quality improvement principles in its service provision.

Caraniche practices adhere to a scientist-practitioner model in which current and future trends in best

practice literature are integrated into all service delivery.

In the case of prison-based services, prisoners with a past or current history of substance dependence are

assisted in exploring their personal issues such as grief, loss, trauma, abuse and managing emotions and

relationship difficulties that may be underlying or associated with their substance dependence and

particularly their offending behaviour. This approach is thought to best address the issues underlying

substance dependence so as to promote lasting change and minimise the harm associated with continued

drug use and offending.

2 Program Objectives, Aims and Theory

A 2013-14 parliamentary inquiry into the supply and use of methamphetamines in the state of Victoria

revealed a 600 percent increase in ice-related offences from 2009 to 2013. This is consistent with outcomes

from Caraniche’s 2013-14 annual evaluation report for prison-based drug and alcohol services, which

identified methamphetamine as the most harmful drug for male prisoners. Almost half of all offenders

who engaged in prison-based drug and alcohol treatment reported methamphetamine as their primary

drug prior to incarceration, an increase of 34.2 percent from the previous year. Of that population,

approximately one third identified their most serious offence as being violent in nature.

In response to increases in methamphetamine-related crimes and at the request of Justice Health,

Caraniche has developed three therapeutic programs of varying intensities and application aimed at

targeting the health concerns and criminogenic needs of incarcerated men with a history of

methamphetamine use. These include the 6 hour Ice Effects, 24 hour Managing Ice Addiction, and 44 hour

Breaking the Ice programs. This report is focused on evaluating a pilot study of the 6 hour Ice Effects

psychoeducational group program. The outcomes of this pilot study will inform important changes to the

program’s design before the program is implemented for delivery in all public correctional centres across

the state of Victoria.

The objective of the Ice Effects program is to reduce the prevalence of ice use and its associated harms in a

prison-based population by educating users about the risks and post-withdrawal effects of ice use. To

achieve this, the Ice Effects program aims to develop motivation and initiate behaviour change around ice

use by providing individuals with information, support, and self-management strategies that promote

consideration of - and preparation for - living ice-free following release from custody.

The specific psychoeducational aims of the Ice Effects program are to assist participants to:

1. Gain an understanding of the effects of ice


8

2. Understand the neurobiology of ice, i.e. the effect of ice on the brain

3. Gain an understanding of the link between ice use and mental health problems

4. Understand the withdrawal process

5. Understand personal withdrawal experiences

6. Identify ways to assist the body and brain in recovering from ice use

7. Develop strategies for managing cravings and withdrawal

8. Understand the difference between lapse and relapse

9. Consider their own high risk situations and make plans to manage these

The Ice Effects program is founded on biopsychosocial theoretical perspectives that describe substance use

and behaviour change including social learning, self-determination, biological and offender-specific

theories. The program is primarily designed to reflect the principles of the Transtheoretical Model of

Change (TTM; Prochaska & DiClemente, 1983) in accordance with its primary objective to generate

motivation for making behavioural change.

The Ice Effects program recognises that in order for participants to transition toward recovery, certain

experiential and behavioural processes must occur during the course of treatment. Drawing from both TTM

and Self-Determination Theory (SDT; Deci & Ryan, 1991), the program acknowledges that when individuals

perceive themselves as competent or at least capable of managing the consequences of change they are

more likely to consider and/or prepare for health-related change. To achieve decisional balance and self-

efficacy in participants, the Ice Effects program incorporates the following as core components of the

program content:

(1) Psychoeducation about ice use,

(2) Psychoeducation about the skill sets needed for managing cravings and high-risk situations

associated with ice use, and the

(3) Teaching of effective strategies for minimising harmful ice use.

In addition, the Ice Effects program design encourages group processes and activities to support the natural

achievement of stage-related experiential learning as outlined in TTM. This includes having participants

engage in activities that promote cognitive appraisal and evoke intrinsic motivation. Throughout the

program, participants are encouraged to rely on their own ideas and reasons for change in an effort to

develop increased self-efficacy and motivation to make change. Figure 1 provides a detailed account of the

expected changes (or program assumptions) for transition to occur at each stage of the program.


9

↑ Motivation Program

Delivered

↑ Knowledge

base

Skill

Development

↑ Knowledge

about ice +

self

↑ Knowledge

about ice

↑ Knowledge

of available

skills

Enhanced

skills

Preconditions 1. Funding is obtained and maintained. 2. Effective operational strategy in place to deliver program, incl. CFS staff to facilitate delivery of program. 3. Suitable referral process is in place; prison staff & offenders are educated about referral process. 4. Demand exists for program at individual sites.

18. Participants have sufficient knowledge about the problem and feel competent/confident to begin to manage harmful ice use (through raising consciousness). 19. Participants identify problematic behaviour as problematic. 20. Participants are willing to engage in and have the capacity to apply critical reasoning to own behaviour (self-evaluation). 21. Participants have sufficient insight into personal contribution to problematic behaviour (self-evaluation). 22. Participants identify benefits of changing, believe in their ability to change and/or express optimism or hope for the future (self-liberation).

5. Program is delivered at the group’s level of risk and overall functioning. 6. Facilitators understand participant needs, including individual styles of learning and are equipped to assist where required. 7. Provisions are available and in place for literacy or language deficits. 8. Facilitators possess competent program delivery skills & good knowledge of underlying theories and research. 9. Program effectively manages interruptions. 10. Participants build sound rapport with facilitators. 11. Educational content positively influences participant knowledge. 12. Program appropriately teaches skills and provides opportunity to practice skills taught. 13. Participants are able to safely experience and express feelings, thoughts and solutions (dramatic relief).

23. Participants willing to seek/obtain further treatment 24. Participants consider drug-free options for future (social liberation). 25. Participants are willing to enlist or accept support from significant others and professional services. 26. Participants consider plan for dealing with high risk situations. 27. Participants are maintaining ongoing personal & professional support. 28. Ongoing opportunity exists for further MH/substance abuse counselling.

14. Participants develop an understanding of how their ice use is connected to their current circumstances (relatedness). 15. Participants understand that there is a choice to use and develop ways to respond to their ice use (autonomy support). 16. Participants understand their withdrawal experiences (relatedness). 17. Participants develop understanding and competence to manage withdrawal/harmful ice use (consciousness raising & competency).

Relatedness

Autonomy &

Competency

↑ Ability to

manage

harmful use

of ice

Engage in

further

treatment

Figure 1. Program Theory with Assumptions

Figure 1. Program Theory with Assumptions


10

3 Evaluation Framework

3.1 Aim

The primary goal in evaluating any pilot program is to examine design appropriateness and implementation

effectiveness in order to inform the refinement of program models and delivery approaches. Some

outcome data is provided (see section 3.2.3), however at this early stage, numbers are small and only

limited conclusions about program outcomes can be reached.

This evaluation examines three domains relevant to program design and implementation: appropriateness,

effectiveness, and efficiency. Where possible, the evaluation provides comment on the program impact and

sustainability. In addition, this evaluation will assist in determining the ongoing funding needs of the

program. Each core area of assessment is described in detail below.

3.1.1 Appropriateness

To what extent was the design of the program suitable in meeting the needs of key stakeholders?

Appropriateness refers to the measurement of the program’s suitability for achieving its desired effects

and working in its given context. This evaluation examines whether the Ice Effects program was appropriate

in meeting the needs of stakeholders through the design of a psychological intervention that adequately

addresses the rationale for the program and is appropriately suited to the needs of participants. This

evaluation also examines the extent to which the program manual content and resources were relevant

and useful.

3.1.2 Efficiency

To what extent was the program implemented in an efficient manner?

Efficiency measures the extent to which program resources are converted into results in a timely and

resource efficient manner. This evaluation will investigate how efficient the Ice Effects program was in

delivering agreed outputs within budgeted resources whist working within the constraints of the prison

environment and maintaining a high quality of service delivery. The degree to which the program

facilitators were able to efficiently manage the operational demands of the program location will also be

explored.

3.1.3 Effectiveness

To what degree was the program implemented as intended?

Effectiveness measures the degree to which the Ice Effects program objectives were achieved. This

evaluation will assess implementation effectiveness within two contexts. These include (1) the extent to

which the program was implemented as intended and (2) whether the program design achieved an

appropriate level of participant engagement. The first context includes the extent to which the program

manual allowed for intended timeframes to be achieved, as well as the extent to which it enabled


11

facilitators to promote guided learning and productive group process. The second context refers to the

degree to which the program engaged participants and enhanced their skill development.

3.1.4 Impacts

To what extent were the clinical outcomes of the program achieved?

Impacts refer to the positive and negative long-term effects produced by a program, either directly or

indirectly, intentional or not. This evaluation considers the degree to which the program contributed an

increase in participant’s knowledge about ice. Given that this program is within the pilot stage, only a

preliminary assessment of impacts is appropriate. Evaluation of the true impacts of the program requires

that the program design and delivery approaches are finalised and a sufficient sample of participant data is

collected. The evaluation of impacts will examine the extent to which the clinical outcomes of the Ice

Effects program were achieved through the delivery of psychometric measures.

3.1.5 Sustainability

To what extent did the program achieve its long-term objectives?

Sustainability refers to the continuation of benefits from an intervention in the medium and longer term.

Effective evaluation of the sustainability of the Ice Effects program would include an assessment of the

extent to which the program has resulted in engagement in further AOD treatment, sustained changes in

drug using attitudes and beliefs, and reduced ice use or abstinence from ice. This evaluation comments on

the value of such data in completing a comprehensive and thorough review of the Ice Effects program.

3.2 Evaluation Methodology

The Ice Effects program was developed in January 2015 and was piloted at the Metropolitan Remand

Centre (MRC), Marngoneet and Middleton Correctional Centres between January and December 2015.

3.2.1 Target Population

Participants were males aged 18 years of age and above who were incarcerated in a Victorian correctional

centre where the Ice Effects program was piloted. Individuals were considered eligible to participate in the

program if they met the following inclusion criteria;

1. Had a self-reported or documented history of ice use, and

2. Had completed the preliminary screening interview.

Individuals were not eligible to participate in the program if the screening interview identified them as:

Having a major mental illness or psychological syndrome that prevented them from engaging in the

program or if, engagement in the program would have detrimentally impacted their psychological

functioning (e.g., were actively psychotic),

Having a severe intellectual impairment or cognitive disability,

Likely to be disruptive to the group, or otherwise unable to engage in a group program,

Having insufficient time remaining on their sentence to complete the program.


12

Individuals not eligible to participate in the program were considered for one-on-one psychological

intervention on a case-by-case basis or alternatively, were referred to a more appropriate service to ensure

appropriate care was achieved in lieu of their participation in the Ice Effects program.

3.2.2 Measures

Evaluation packs comprising a set of measures representing each of the theoretically derived factors

considered important for assessing design appropriateness and implementation effectiveness were

completed by facilitators and participants of the program. Facilitators and participants each provided

written and verbal feedback in various forms.

3.2.2.1 Facilitator feedback booklet

The facilitator feedback booklet (see Appendix A) captured both qualitative and quantitative data. This

booklet was completed alongside the piloting of the program. Individual facilitators were responsible for

filling in the booklet, while the Research and Professional Practice (RPP) team within Caraniche were

responsible for collecting and collating the information reported. Quantitative data was collected through

an attendance record and captured program adherence. Qualitative data was collected through a feedback

form at each session and a disruption to services register. The feedback booklet included information on

operational demands, resourcing concerns, group dynamics and the ability of facilitators to manage

behaviour within the scope of the program. The booklet also collected information about the

appropriateness of the content and tasks proposed in the manual, times allocated for activities, and any

changes made to the program by facilitators (where applicable).

3.2.2.2 Participant focus group

The RPP team conducted a single focus group with Marngoneet participants, who successfully completed

the group program, on 11 September, 2015. The 11 participants in this group were invited to participate in

the focus group and all consented to participating. The focus group collected data (see Appendix B) on:

Appropriateness of session content and accompanying resources in supporting participant

understanding of program content and skill development

Participants knowledge about ice use and understanding of their personal reasons for using ice

Participant self-reported motivation to change following completion of the program

3.2.2.3 AOD Programs Feedback Survey: Short Form

The AOD Feedback Form (see Appendix C) is a questionnaire designed to provide Caraniche with

information about Drug and Alcohol Treatment Programs from the perspective of clients. The feedback

form provides information about which aspects of the pilot program were found to be effective and those

that require improvement, as identified by participants of the program.

The quantitative component of the questionnaire comprises ten structured questions that require a response

on a Likert-scale from 1 to 5, where 1 = strong disagreement and 5 = strong agreement, and one question

requiring a yes or no response.

The qualitative component of the questionnaire comprises four unstructured open-ended questions

requiring short answer responses designed to complement the structured questions. These questions ask


13

participants to comment on what they found most effective and useful about the program and any areas

where they felt the program could be enhanced.

3.2.2.4 Psychometrics

Psychometric measures were administered to obtain key information about participant functioning at the

point of program entry and to gain an understanding about level of ice-specific knowledge achieved at

program completion. Psychometric measures were not employed in the pilot evaluation of the Ice Effects

program as measures of pre-post change. This decision was based on the existing, well-developed empirical

knowledge base that has found pre-post psychometric change to be a poor indicator for measuring the

success of brief psychoeducational programs in using increased knowledge to achieve long-term

behavioural change. Table 1 below details the psychometric measures employed as part of this evaluation.

Table 1. Description of Psychometrics

Area of Assessment Description of Measure

Psychological distress The Depression-Anxiety-Stress scale (DASS-21; Lovibond & Lovibond,

1995) is a 21 item self-report measure that assesses emotional distress on

three scales: depression, anxiety and stress. The DASS-21 provides

information about the psychological wellbeing and presentation of the

individuals who are volunteering participation in the Ice-Effects Program.

The DASS-21 (see Appendix D) was administered as a pre-treatment

measure only.

Knowledge Survey The Ice 6 Hour Ice Effects Knowledge Survey (see Appendix E) is an 8-item

multiple choice measure that assesses participant learning post

completion of the program. The survey assesses core content areas

covered over the course of the program. Therefore, the knowledge survey

was administered to participants post treatment only.

3.2.3 Data Analysis

To assess whether the aims of the pilot program were achieved, a combination of qualitative and

quantitative research methods were employed. Information collected from facilitators and participants of

the program was subject to mixed-method statistical analyses. Drawing from grounded theory and

phenomenology, information obtained via survey method was examined for the emergence of conceptual

categories reflective of increased ice-related knowledge and motivation to change. Additionally, frequency

counts based on program adherence and participant psychopathology were analysed to test program

design and implementation effectiveness.


14

4 Results

4.1 Program Demographics

4.1.1 Pilot evaluation duration

The program pilot commenced in January 2015 and data collection ceased in early December 2015. Data

included within this report covers the 2014/2015 and 2015/2016 financial years.

4.1.2 Total programs run and prisoners participated

A total of 21 programs were facilitated across the three pilot sites. One-hundred and eighty three prisoners

commenced in the programs and of these 162 participants completed the Ice Effects program, yielding an

overall completion rate of 88.5%. Table 2 provides a breakdown of program delivery by location, in terms of

commencements and completions, as well as providing an overview of non-completion reasons.

Table 2. Breakdown of Participation per Prison Location

Location Commencements Completions Non-

completions Non-completion

reasons

MRC (11 programs) 106 93 13 3 x court; 6 x refusal 1 x health; 1 x other

2 x safety

Marngoneet (7 programs) 55 49 6 3 x health; 3 x refusal

Middleton (3 programs) 22 20 2 2 x other

TOTAL (all pilot locations) 183 162 21 3 x court; 9 x refusal 4 x health; 2 x safety;

3 x other

4.1.3 Program attendance

Participant attendance data was available for 21 programs. Attendance data demonstrated that program

adherence remained relatively stable, with seven out of the 21 programs delivered experiencing no

reduction in program attendance from session one to session two. Of the remaining programs, participants

dropped out for a range of reasons including health problems, safety incidents, court attendances and

refusals to participate.

4.1.4 Participant Characteristics

Psychometric data was collected from participants to determine their level of psychological distress at the

point of program entry. It is important to gain this information to be able to assess for whether the

program is attracting the participants it was designed to target, and to create a psychological profile for

individuals seeking this particular dose and type of treatment.


15

4.1.5 Mental health

4.1.5.1 Depression, Anxiety and Stress

The Depression-Anxiety-Stress scale (DASS-21; Lovibond & Lovibond, 1995) is a 21 item self-report measure

used to screen for symptoms of emotional distress on three scales: Depression, Anxiety and Stress. Higher

scores indicate higher levels of symptomology. A guide for interpreting severity is provided below in Table

3. A mild score, for example, demonstrates that a person is expressing levels of depression, stress, or

anxiety above the population mean, but still below the severity level of someone who would seek help for

such mental health concerns. The DASS-21 was administered as a pre-treatment measure only, with 63

participants volunteering to complete the measure. The DASS-21 was used for the purposes of

understanding the characteristics of the target population and for informing revisions to program design to

better accommodate participants current mental state. Participants were neither screened nor declined

entry to the program on the basis of their DASS-21 score. As is usual practice within the prison

environment, when a participant was identified as experiencing clinically significant levels of depression

and/or anxiety, they were referred to Medical for further assessment.

Table 3. Cut-off Scores for DASS-21 Severity Labels

Depression Anxiety Stress

Normal 0 - 4 0 - 3 0 - 7

Mild 5 - 6 4 - 5 8 - 9

Moderate 7 - 10 6 - 7 10 - 12

Severe 11 - 13 8 - 9 13 - 16

Extremely Severe 14 + 10 + 17 +


16

4.1.5.2 Results

Depression: As indicated in Figure 2 over half of the respondents commenced the program with normal

levels of depression, with 19% of respondents indicating severe to extremely severe levels of depression.

Figure 2. Participant Level of Depression Prior to Program Commencement

Anxiety: As indicated in Figure 3 just over 40% of the respondents commenced the program with

normal anxiety, with 29% of respondents indicating severe to extremely severe levels of anxiety.

Figure 3. Participant Level of Anxiety prior to Program Commencement

Stress: As indicated in Figure 4 over 60% of the respondents commenced the program with normal levels of

stress, with 13% of respondents indicating severe to extremely severe levels of stress.

51%

13%

17%

9%10%

normal

mild

moderate

severe

extremely severe

43%

17%

11%

14%

15%

normal

mild

moderate

severe

extremely severe


17

Figure 4. Participant Level of Stress prior to Program Commencement

In total, the majority of participants did not demonstrate levels of psychological distress equivalent to help-

seeking prior to the commencement of the program. However for the proportion of participants that did;

one third or just less reported significant, serious levels of anxiety and depressive symptomology. Anxiety

was the greatest concern, followed by depression and stress, respectively.

4.2 Participant Experience

4.2.1 AOD Programs Feedback Survey: Short Form

Of the 162 participants who completed the pilot Ice Effects programs, 153 AOD Feedback Forms were

returned. From the program’s outset all facilitators and program participants were aware that the program

was undergoing a pilot evaluation process. This foreknowledge may have created a clear set of normative

expectations regarding facilitators’ and participants’ involvement in the evaluation process. These

expectations when combined with the very short feedback forms used may help to explain why a high

percentage (92.6%) of feedback forms were returned. It is noted that not all feedback forms were complete

with some questions not answered, or no comments provided. Hence the ‘N’ values in the following section

(including table 4) will vary.

4.2.1.1 Quantitative Results

As outlined in Table 4 below, participants were satisfied with the Ice Effects program. This was indicated by

the reporting of mean scores well above the mid-range point for questions across the survey. Mean scores

falling above 4.5 out of 5 revealed that participants were very satisfied with facilitator ability to: answer

questions (Q.3), deliver the program content (Q.6) and manage the group (Q.8). This result is consistent with

participant satisfaction data for other Caraniche programs.

61%15%

11%

8% 5%

normal

mild

moderate

severe

extremely severe


18

With mean scores of 4.14 and 4.43 out of 5, participants reported having gained knowledge (Q.4) and an

understanding about the information provided within the program (Q.5), respectively. The majority of

participants also indicated that the information provided within the program was useful to them (Q.2; 4.28

out of 5), and that their confidence with tackling personal problems increased as a result of the program (Q.7;

4.16 out of 5). Mean responses at the high end of the scale demonstrated that the majority of participants

found the program to be both useful and effective.

Table 4. Participant Satisfaction Mean Scores

Participant Satisfaction N Mean SD

Q.1. I feel the program was suited to my needs 153 4.16 0.62

Q.2. The information provided in this program is useful to me 153 4.28 0.72

Q.3. The facilitator answered any questions I had 153 4.52 0.52

Q.4. I have learned a lot from this program 153 4.14 0.80

Q.5. I understood the information provided in this program 153 4.43 0.52

Q.6. The facilitator did a good job in delivering the program 153 4.62 0.50

Q.7. As a result of the program, I feel confident about tackling my problems 153 4.16 0.70

Q.8. I feel the facilitator did a good job in managing the group 153 4.62 0.54

Q.9. The program motivated me to work on my problems 153 4.25 0.72

Q.10. Overall, I am satisfied with this program 152 (one

missing

response) 4.44 0.59

4.2.1.2 Qualitative Results

“What did you like about the program?”

Table 5 details the five most common areas of program content that participants reported to be most

useful. These are presented in consecutive order, with the category “enhanced knowledge and

understanding” representing the aspect of the program that participants liked the most – as indicated by

the 44 respondents endorsing this category. Forty-four participants expressed an enhanced knowledge and

understanding as their most useful component of the program, whilst six participants identified facilitation

of the program to be the component they most enjoyed. Other aspects of the program that the participants

liked included the facilitator, and the coffee.


19

Table 5. What Participants Liked About the Program

Majority response Number of

respondents

Example participant response

Enhanced

knowledge and

understanding

44

“Information about the brain”

“Better understanding of ice use”

“Learnt more about addiction and ice”

“Learning about the medical effects of ice”

“Informed me of things I was unaware of”

Visual information

and content

15

“All information and charts”

“How specific it is”

“It was informative”

“The content was easy to relate to”

Group engagement

11

“Information about different peoples experiences”

“Talking about our problem of drug use”

“Open discussions”

“When we talked about the 3 chemicals”

Relapse

management and

goal planning

7

“Strategies and goals were a good reminder”

“Helped to acknowledge the negative things about ice”

“…Also suggestions from the group about relapse”

“Helped to manage drug use”

“Learnt ways of coping with triggers”

The facilitation

6

“The way it was delivered”

“How the information as explained”

“Well explained to us”

“She was straight to the point, I felt it would help me in learning

strategies”

“Facilitator answered questions”

Note: This table includes the five most common areas of program content that participants reported to be most useful. This table only includes the highest five responses, i.e., there were 70 other responses which are not reported within this table hence respondent numbers will not add to 153.

“How could the program be improved?”

There were only 44 responses to the question “how could the program be improved?” The four most

common responses for how the program could be improved are presented in Table 6. Over one third of

participants made suggestions that the program should be longer, with only a few commenting on group

engagement and size as a point of improvement. Six participants stated that nothing needed to be changed

and one participant indicated that the program had elicited motivation within them to engage in other

programs, by stating that they wanted “more programs”.


20

Table 6. Participants Perception of How the Program Could be improved

Improvement

theme

Number of

respondents

Example participant response

Longer Duration

15

“Go for longer”

“Longer program, maybe a 24”

“If it went for a bit longer!”

Enhance

information on ice

effects

13

“Talk more about ice effects”

“More visual aids of effects of ice 6-12-18-24 months”

“Video on effects”

“More information about ice effects on a more scientific level”

Follow-up support

and goal planning

5

“Information about following support if to be released soon”

“Provide information on strategy sharing program”

“Focussing on specific solutions”

“More on relapse prevention and making a plan”

Group engagement

and size

3

“More conversations”

“Maybe smaller groups, maybe someone-on-one”

“Everyone get more involved”

Note: This table includes the four most common areas of program content that participants reported to be most useful. This table only includes the highest five responses, i.e., there were 117 other responses which are not reported within this table hence respondent numbers will not add to 153.

“Would you recommend the program to others?”

Of the 111 participants who responded to this question, 98 percent indicated they would recommend the

program to others (see Figure 5).

Figure 5. Willingness to recommend the program

98%

2%

Would you recommend the program to others?

YES

NO


21

Of those who responded that they would recommend the program to others, the main reason provided for

doing so was that the program provided “a better understanding of ice”. For example, one participant

reported:

“Program is insightful, [it] gave me a better view on ice”, “because you get an idea

what it does” and “[I] learnt more than what I knew about ice”.

In addition to this, two participants stipulated that they would recommend the program to others only in

certain circumstances. Responses within this theme included:

“Only if they had an ice addiction” and “possibly”.

“Any other comments?”

When asked for any further comments, the majority of participants provided generally supportive

statements.

Example responses included:

“Thank you, very helpful” and “It was all good”.

With one participant (different to the abovementioned) indicating that the program elicited motivation

within them to engage in other programs by stating that:

“I was satisfied and looking forward to the next one”.

4.2.2 Participant Focus Group

A single participant focus group was held at Marngoneet Correctional Centre on September 11, 2015 (see

Appendix C) with 11 participants. Participants were aware from the outset that the program was

undergoing a pilot evaluation process and that their feedback would be sought upon the program’s

completion. At the conclusion of the Ice Effects program, participants at Marngoneet were invited to

attend a focus group to canvass their views. The focus group was facilitated by RPP staff who were

unknown to the participants. Focus group questions and participant feedback are presented below.

1. What did you think this program was about (or trying to achieve)?

The majority of participants in the focus group reported that the aim of the program was to provide

education about ice use, including the effects of using ice, harms associated with ice use and triggers for

use. Example statements included:

“Educate us on our experiences of using” and “The harms of ice use”’ and “The triggers of our

ice use”.

2. What did you learn about ice?

The majority of participants expressed increased knowledge with regards to damage to the body and brain

from ice use, recovery duration, and long-term effects of ice use. Example statements include:


22

“The damage that ice causes to the brain and body and that it can be irreversible” and

“Surprised by the recovery period form ice – 18 months” and “the long term effects of ice

use”.

Additionally, participants queried concern about using the knowledge acquired during the program to

facilitate long-term behavioural change. For example, one participant noted:

“We have the information that we need but practice is the hard bit. We will always have

problems. It’s easy to think just [expletive] it when things get bad”.

3. What did you learn about your ice use?

Participants expressed that the program elicited an enhanced awareness about their reasons for using ice

in the past. Particular reference was made to developing an understanding of how personal thoughts and

feelings contributed to using behaviour, and specific situational triggers for use. Example statements

included:

“Where you put yourself in high risk situations – skills to avoid and the ability to recognise

those feelings and act on it” and “It’s easy to get hold of when things aren’t going well”.

A developed knowledge about the longer term consequences of ice use and relapse management were also

expressed. For example, one participant stated:

“For somebody who suffers from depression, knowing that it can take up to 18 months to

build up dopamine again means that it’s just not worth using ice”.

4. How did the program help you manage your emotions?

Participants acknowledged that the content did not appropriately prepare them for learning new, general

regulatory skills attached to their ice use within the allocated time frame. For example, one participant

stated:

“The program was not long enough to understand emotions”

Participants also mentioned that a developed understanding of the negative consequences of ice use acted

as a motivator for attempting to not use. For example, one participant stated that the program contributed

to:

“A bit of understanding but more just thinking about how bad it is. It’s easy to talk yourself

into a lapse but knowing how long it takes to recover makes you think it isn’t worth it”.

5. How do you plan to keep working on you ice use?

Participants expressed awareness of losses from ice use as a motivator for not using in the future, as well as

the fear of returning to prison. Further, participants highlighted rehabilitation barriers when

communicating with corrections staff, due to perceived consequences of being returned to custody whilst

on community corrections orders. Example statements included:

“Focusing on the losses that ice has caused in our lives is helpful”.

and


23

“Losses are important – being in prison sucks and parole is a risky time. They are strict and if

you make one mistake you are back in prison. I would rather not be open about it- it’s hard

to do something about a lapse when you are scared of breaching parole”

However, one participant also expressed the importance of internal motivation contributing to the desire

to change:

“Nobody will stop until they want to make a change”.

6. How do you feel about your ability to make a change going forward?

Similar to recognising the importance of internal motivation, participants also expressed a low-level of self-

efficacy or confidence in their ability to implement changes. Example statements included:

“Saying we won’t use is unrealistic – saying no when it is in front of us is hard.”

and

“I feel the same.”

7. How did the group work together?

Participants expressed group cohesiveness, with regards to collaboration, trust and honesty. For example:

“Everyone participated and got involved”,

and

“We can be honest in these programs – we are already in jail so what is going to happen?”.

8. If anything could be done differently, what would you like to see changed?

Time constraints and perceived superficiality of the program were predominantly noted as areas for

improvement. Example statements include:

“It’s good for what it is” and “Can only do so much in 6 hours”.

4.2.3 Knowledge Survey

Eighty-eight participant knowledge surveys (58% of program completers) were received for evaluation

purposes. The low response rate for the knowledge survey likely reflects participants’ dislike of being tested

in a way that may have reminded them of school or other performance driven experiences. It is unknown if

participants’ decisions to decline to complete the test was in any way related to their actual level of

knowledge (be this high or low). This ambiguity gives reason for caution when drawing inferences about

program participants’ true levels of knowledge where these inferences are based solely on the 88

participants that completed the Knowledge Survey. Questions where respondents had neglected to provide

an answer were scored as an incorrect response. An example of the Knowledge Survey and correct

responses are provided in Table 7 below. The Knowledge Survey responses have been interpreted in two

ways. The first sub-section reports on the number of questions correct per respondent and the second sub-

section reports on the percentage of participants who answered each question correctly.


24

Table 7. Knowledge Survey Questions and Correct Reponses

Survey Questions Correct response

1 Which of the following statements are true

B You can become addicted to ice if you inject it or

smoke it

2 What effect does ice have on the brain B Ice increases the ‘feel good’ chemicals (like

dopamine) in the brain immediately after using

3 Experiences of psychosis that come from using ice

can last

D All of the above (i.e. 20 minutes, 2 hours, 2 days)

4 Withdrawal symptoms can last for D All of the above (i.e. 1 week, 1 month, longer

than one month)

5 The most common mental health disorders that are

associated with ice use are

B Anxiety, depression, and psychosis

6 Which of the following is not one of the 8D’s for

managing cravings

C Disagree

7 Which of the following best describes a relapse C Returning to old patterns of ice use after being

sober for a period of time

8 When trying not to use ice, which of the following

is an example of a high risk situation

B Having an argument with a family member

4.2.3.1 Percentage of Correct Reponses per Participant

Almost one third of the surveyed participants (30%) answered each of the eight questions correctly (See

Table 8). Figure 5 shows the correct response rates for each of the eight questions. The question

participants found the most ease in selecting the correct answer for was question one (88% of respondents)

which focused on assessing participant knowledge on the ways in which people can become addicted to

ice. The next question found to have accumulated the most correct responses (question 3; 85%) focussed

on assessing knowledge on the amount of time that ice-induced psychosis can last. Eighty-five percent of

respondents correctly answered question two and question four. Question two focused on assessing

participant knowledge of the neurobiology of ice, and question four, the amount of time withdrawal

symptoms can last for (see Table 7). As was also noted in the section discussing the participant satisfaction

survey, participants indicated program information such as ‘the effect of ice on the brain’ was most

satisfying to learn, and therefore, their interest in the content may have contributed to their ability to

correctly identify the accurate responses to these questions.

Table 8. Percentage of Correct Reponses per Participant

# of questions correct % of respondents to achieve this

8 30

7 29

6 12

5 10

4 10

3 1

2 7

1 0


25

Conversely, Figure 6 also indicates questions that participants had the most difficulty in selecting the

correct response. With a correct response rate of 73%, question eight proved to be the most difficult

question for participants. Although the majority of participants answered this question correctly, the lower

percentage of correct responses could perhaps be explained by the personal nature of the question. The

correct response, as identified by the test developer, may not have been personally relevant to all

participants making it less likely that some participants would select it as an example of a high-risk

situation. Lower response rates pertaining to question 5 and 6 also indicate that content on relapse

management skills and mental health may need to be enhanced.

Figure 6. Percentage of Correct Reponses per Question

4.3 Facilitator Feedback

4.3.1 Preliminary analysis

Seven Facilitator Feedback Booklets were received from three prison locations – MRC (4 group responses),

Loddon prison – Middleton (2 group responses), and MCC (1 group response). Qualitative data collected

from the Feedback Booklets and the post-program questionnaire have been thematically analysed, with the

major themes extracted presented below.

4.3.2 Results

1. Participants were able to better understand the consequences of ice use, and the related symptoms of

withdrawals

At the commencement of treatment, facilitators found that many participants lacked a real understanding

of the consequences of ice use, or were not aware of the symptoms of withdrawals. While few participants

appeared to possess a basic understanding of ice, it was evident to facilitators that these participants did

not think the effects of ice use applied to them, or did not note the symptoms of withdrawals in

themselves.

After the delivery of session content, facilitators found that participants were able to better identify how

some of their experiences resulted from personal ice use and/or withdrawal. Three out of seven facilitators,


26

for example, outlined that their participants were “able to identify ice abuse cycle”, and were “able to

recognise high risk situations” that related to their offending behaviour.

2. Participants responded well to the program, and were engaged with the materials provided

When asked to provide their perception on the strengths of the program, facilitators overwhelmingly

reported that participants in the program were responsive and receptive to the materials delivered to

them. Facilitators also reported that participants were interested in the content of the material offered,

were “active in their learning”, and were readily engaged in group discussions.

3. The manual used was clear and valuable in providing a foundation for discussions

While the majority of facilitators echoed the effectiveness of the manual in offering a clear structure and

basis for discussion, one facilitator suggested that the content of the manual was excessive and too

complex. This, they believed, hindered meaningful and extensive discussions due to additional time being

spent explaining the complex materials included.

4. The objectives of the sessions were met in the majority of cases

All but one facilitator confirmed having met the set objectives for individual sessions. In the case where the

objectives of a single session were not met, the facilitator reported time constraints as the reason for some

session content not having been delivered. This was reportedly due to session time having been interrupted

by a disturbance in the prison (MRC).

5. Participants required more time to thoroughly comprehend the concepts

Facilitators reported that the content for sessions was complex and difficult for many participants to

understand, and might not have been completely appropriate for the target audience – particularly with

the technical language used in the workbook. Facilitators suggested that the module “needs to take into

account brain impairments in men, and [needs to] be realistic about what they can absorb”. To correct this

disparity in understanding, many facilitators found that they had to spend more time explaining complex or

technical concepts to participants, which affected time allocated for group discussions and activities.

Additionally, many facilitators recorded that content for the first session was “too heavy”, and needed to

be modified. Three of seven facilitators felt that it was necessary to shift certain discussions from session

one to session two to allow for sufficient comprehension and engagement.

6. Participants required additional time to complete necessary workbooks/forms for the sessions

Due to the commonly reported issue of time restrictions, facilitators found that participants should have

been allocated additional time for form or workbook completion. The remaining time within sessions could

then be used for further exploration and discussions of topics.

7. Visual aid presentations (e.g. videos, images, etc.) would be helpful to assist participants in understanding

complex concepts

The question on suggested improvements for the program elicited requests from many facilitators to

provide audio visual aids to participants in order to assist them in understanding complex – especially

neurobiological or neurochemical – concepts relating to ice use. This could be in the form of a short video,

which facilitators found would be more “helpful for men as they are more visual”.


27

8. Facilitators had to be responsive to participants in the group, and had to make necessary modifications to

modules according to participants’ interests or needs

The intensity of the program in combination with the time constraints imposed on the delivery and

discussion meant that some facilitators chose to focus their efforts on certain topics for discussion. The

choice of topics was made primarily as a response to the interests or needs from participants. As a result,

the topics covered varied slightly from group to group. For example, many facilitators (3 out of 7) chose to

spend more time on the neurobiology and neurochemistry of ice use upon request and interest from

participants. Similarly, many participants were interested in learning about coping strategies in high-risk

situations because of their upcoming release. Thus, facilitators responded to this request, and spent

additional time during sessions on topics relevant to independent groups.

Moreover, facilitators reported that they needed to be flexible in their program delivery – particularly in

relation to the structure of the program – as it was necessary for them to first gauge the flow of the

discussion in the group in order to guide the direction of the session. It is evident from the responses that

facilitators found it challenging to follow the structure of the program prescriptively due to the differences

in interests and needs in each of their participant group.

9. Group discussions and activities engaged participants, and encouraged participants to share stories and

raise questions

Facilitators communicated that participants “enjoyed” the materials provided as evidenced by their “eager”

and “active” participation in group discussions. These group discussions also provided participants with a

space to “share [some] tips” on how best to abstain from ice use during their incarceration. Further, group

discussions and activities permitted participants to break down the complex concepts presented, and to

understand these concepts more thoroughly with the use of examples.

Based on the perspectives of facilitators, future program participants may benefit from additional time

allocation for extensive group discussions and activities.

10. Some participants confided that the extensive discussion on ice use during the program brought on

actual temptation for ice use

One facilitator noted that discussions of ice use during the program caused participants to feel the urge to

actually return to use. This is a common experience amongst participants in drug treatment programs and

Caraniche has strategies built into programs to manage such experiences.

11. Some evaluation surveys may not be appropriate for participants

One facilitator reported that her participants found the Knowledge Survey was confronting as it was

interpreted as a classroom test. Rather, she suggested that this evaluation measure should be revised, and

should reflect more on therapeutic models.


28

5 Discussion

This section discusses the results in relation to the appropriateness, efficiency and effectiveness of the pilot

program with regards to program design and implementation.

5.1 Appropriateness

To what extent was the design of the program suitable in meeting the needs of key stakeholders?

Overall, facilitators found the program manual to be useful in providing a clear structure for facilitation and

in eliciting group discussion, and reported minimal changes to content. Similarly, participants expressed

high levels of satisfaction with the program content, with some requesting to make it more scientific.

Despite these positive reviews, facilitators indicated that session content was complex at times and time-

consuming to elicit participant understanding. An additional reason for amending content occurred at times

when the content was informally covered during unintentional discussion elicited by the group. Therefore,

facilitators amended program content to simplify it for participants and focus more directly on the

participants’ interests.

The outcome that facilitators presented less content in session one than was originally intended resulted in

the majority of facilitators suggesting that the intensity of the program be reduced in session one and more

activities introduced. Program content found to be most interesting to participants, and therefore

requiring more facilitation time included neurobiology and neurochemistry of ice use and coping strategies

in high-risk situations. Participant interest in this content resulted in participants indicating that they would

like the program to include more of this information, but that understanding could be enhanced through

the introduction of more visual aids and videos. The ability of the facilitators to demonstrate flexibility in

their delivery of the program speaks to the effectiveness of the resources in providing the facilitators with

enough information to identify the program’s intent, including its focus on harm minimisation.

Participant’s interest and satisfaction in the program was demonstrated through their understanding of -

and expressed ability to relate to - the content as assessed via their keen engagement in group discussion

and post-program feedback. The ability of the majority of participants to identify correct responses to

content questions and articulate an increased understanding of their personal relationship with ice during

group discussions and on qualitative feedback measures, indicates that the pilot program manual and

resources did contribute to enhancing participants’ ice knowledge.

5.2 Efficiency

To what extent was the program implemented in an efficient manner?

The consistent facilitator response was that there were no resource problems experienced within the

program. This suggests that the budgeting constraints embedded within this program allowed for efficient

program delivery and the provision of adequate resources to effectively run the program was met.

Moreover, that overall participant satisfaction was high, especially with regards to program facilitation,

suggests that the budget constraints did not prevent a high level of service delivery from being achieved.


29

5.3 Effectiveness

To what degree was the program implemented as intended?

Although the pilot program was run within the intended timeline, facilitator and participant responses

suggest the need for more time. While participants would like the program to run for a longer duration,

facilitators would like some content to be reduced or re-distributed to ensure that the time constraint

posed on the program does not impact on the ability to deliver the required material in its entirety.

Changes should include simplifying the information, using more visual aids to explain complex content, and

moving the mental health component from session one to session two. Additionally, although participants

reported that they valued group discussion with engagement being high, the allocated time for this was

found to be insufficient both by participants and facilitators, with participants requesting a longer program

to achieve this, and facilitators reducing content in attempting to enhance content understanding.

In addition to the program eliciting participant engagement in group discussions and content, the program

had a high attendance rate, with the majority of participants who commenced the program, returning in

session two to complete the program, demonstrating that participants were highly engaged.

5.4 Impacts

With the evaluation being of the pilot program only, the ability of the program to achieve its long-term

program objectives cannot be assessed. Instead this section speaks to the impact of the design of the

program resources, particularly the psychometric assessment measures, in adequately capturing the

program data.

To what extent were the outcomes of the program achieved?

The AOD Programs Feedback Survey successfully captured participant program satisfaction, with almost all

participants expressing satisfaction with the program and stating they would recommend the program to

others. Qualitative feedback from both participants and facilitators is the most useful measure at this early

pilot stage and such feedback suggests that the program has had a positive impact on participants in

relation to their engagement in program content discussions and an enhanced self-awareness, including

knowledge of their relationship with ice.

No definitive comment can be made with regards to the program’s capacity to increase participants

knowledge and motivation to change, as pre-post data was not collected for the reasons stated previously.

Feedback from the facilitators about the usefulness of the Knowledge Survey indicated the need for this

measure to be removed or revised as an evaluation tool. Further, to assist facilitators with understanding

levels of motivation and readiness to engage in additional group programs, a measure of motivation

and/or treatment readiness may be useful to administer at pre and post treatment.


30

5.5 Sustainability

To what extent did the program achieve its long-term objectives?

The evaluation of the pilot program cannot speak to program sustainability at this time. Instead, it is

suggested that sustainability be measured through the ongoing funding and provision of group programs,

and through monitoring whether participant engagement in additional programs occurs. This would be

measured through identifying whether participants who still have time on their sentence are participating,

have participated or are waitlisted to participate in additional health or criminogenic programs, inclusive of

the 24 and 44 hour ice programs.

6 Key findings

1. Completion rates were high. Of the 183 prisoners who commenced the program, 162 (88.5%)

completed it. The reasons for non-completion included court attendances, health reasons, and

safety issues. Just nine prisoners (5%) refused to attend the second (final) session;

2. In addition to the strong completion rates, participant engagement with the program was also

reflected in feedback from facilitators and participants. Facilitators reported that participants were

highly engaged in group discussions and activities. In the feedback surveys, participants agreed that

they were satisfied with the program, that it suited their needs and that the facilitators did a good

job. Ninety-eight per cent (n=99) of respondents who completed the feedback survey indicated

that they would recommend the program to others;

3. Overall, facilitators reported that the manual provided a useful structure and content, but stated

that, at times, the content was too complex for the target group, many of whom appeared to be

living with acquired brain injuries, and as such extra time was spent explaining concepts to

participants. Interestingly, in their feedback, participants agreed, or strongly agreed, with the

statement, “I understood the information provided in this program”, however this may be due to

the extra efforts made by facilitators, rather than the content of the manual. Both facilitators and

participants indicated that additional visual and audio-visual learning aids would be beneficial;

4. The volume of content to be covered was considered to be too great for the scheduled program

time (6 hours), and did not allow sufficient time for group discussions, which were viewed as a

valuable and engaging component of the program;

5. One of the key aims of the Ice Effects program was to increase participants’ knowledge about the

effects and risks of using ice. A Knowledge Survey was provided to participants at the end of the

program, of which all items were answered correctly by between 73% to 88% of those participants

who returned the Knowledge Survey. It is difficult to establish the extent to which this knowledge

was new knowledge, given that the survey wasn’t administered prior to the commencement of the

program. However, the results do indicate a reasonably good level of knowledge about the key

aspects of the program. It is unknown if participants’ decision to decline to complete the test (74

participants did not return the survey) was in any way related to their actual level of knowledge (be

this high or low). This ambiguity gives reason for caution when drawing inferences about program


31

participants’ true level of knowledge. It is noteworthy, however, that one facilitator reported that

the participants in her group found the Knowledge Survey confronting, as it was too much like a

“class-room test” which can be highly off-putting to a forensic population. The knowledge gains

indicated by the Knowledge Survey were also confirmed by the feedback from facilitators and

participants. Facilitators reported that as a result of the program the participants were better able

to understand the consequences of ice use and related symptoms of withdrawal. In their feedback,

participants either agreed or strongly agreed with the statement “I have learned a lot from this

program”;

6. Another key aim of the program was to increase participants’ motivation to change. In their

responses to the feedback survey, participants either agreed or strongly agreed that, “The program

motivated me to work on my problems”, and “As a result of the program I feel confident about

tackling my problems”. These responses suggest that the program was effective in enhancing

motivation;

7. When asked how they thought the program could be improved, the most common response was

that they would like the program to go for longer, perhaps highlighting the need to provide a

pathway to additional supports for some participants;

8. Facilitators did not perceive that any resource constraints affected their ability to deliver the

program, stating that modifications to program content were made depending on the needs of the

group. Only one facilitator mentioned an operational disruption, with minimal negative effect on

delivery of the program.

7 Recommendations

Evaluation of the appropriateness, efficiency and effectiveness of the pilot program has identified various

areas in which the design and implementation of the program, inclusive of program evaluation could be

improved. These recommendations are outlined below.

1. The program manual be revised to ensure the content is simplified to suit the needs of an offender

population, incorporating additional visual/audio-visual learning aids. An additional focus on the

neurochemistry of ice use and coping strategies may also be beneficial and of interest to

participants;

2. The volume of the program content be reduced to allow for additional time for group discussion,

which was perceived as a valuable and engaging component of the program by both facilitators and

participants. Consideration may also be given to moving the mental health component to the

second session, so as to reduce the volume of content to be covered in the first session;

3. Evaluation of outcomes may be further strengthened by incorporating a pre and post measure of

motivation or treatment readiness. The evaluation process should include a measure of

participants’ motivation to engage in additional programs, such as evidence of referral to individual

counselling or higher dose treatment options post program completion. The Knowledge Survey in


32

its current form should be removed and alternative evaluation approaches developed that can

effectively measure changes in participant knowledge.

a. Update April 2017: It should be noted that the Knowledge Survey was removed from the

program test battery soon after the conclusion of the pilot testing phase. Caraniche has

recommended to Justice Health that the OQ Measures test suite provides a robust and

meaningful way of measuring program outcomes across a wide range of health and

criminogenic programs, but we are yet to receive a formal response to this proposal. It is

probable that the OQ Measures will provide a more suitable test battery for both clinical and

evaluation purposes. Further amendments to the Ice Effects program materials will be made as

part of a general review of programs planned for 2017-18, and following the completion of the

current evaluation process.

4. At the end of the Ice-Effects program, participants should be made aware that there are ice

programs of longer program dosage, and expressions of interest to participate in these programs

should be obtained. This is current practice and the findings of this evaluation reinforce the

importance of this. Participants in Ice Effects are encouraged to consider their subsequent

treatment needs, including participation in the 24-hour and 44-hour ice programs.

5. With the amendments indicated above, and in light of the positive results overall, it is

recommended that the 6-hour Ice Effects program be rolled-out for delivery within all Victorian

prisons, as one component of a suite of programs tailored to prisoners with a history of

methamphetamine use.

a. Update April 2017: This recommendation has already been implemented and the Ice Effects

program is currently being offered, and run regularly, at relevant locations that have a KPI for 6-

hour programs. It is noted that this program continues to be very highly subscribed, with

demand presumably driven by; the high rates of prisoners with ice-related drug issues, the

increasing number of remandees across numerous locations, and the continued increase in

churn and rapid movement between locations. This speaks to the need for the suite of short

health programs.


33

8 References

Berntson, G.G., & Cacioppo, J.T.(2009). The handbook of neuroscience for the behavioural sciences (Vol.2).

John Wiley & Sons, Inc. pp125-128.

Deci, E. L., & Ryan, R. M. (1991). A motivational approach to self: Integration in personality. In R. Dienstbier

(Ed.), Nebraska symposium on motivation: Perspectives on motivation, Vol. 38 (pp. 237-288).

Lincoln, NE: University Of Nebraska Press.

Fougnie, D. (2008). The relationship between attention and working memory. In N.B. Johansen, New

research on short-term memory. Nova Sience Publishers, Inc. pp. 1-45.

Kirk, R.E. (2004). Maturation effect. In M.S.Lewis-Beck, A. Bryman, & T.F. Liao, The SAGE encyclopedia of

social research methods (Vol. 1). http://dx.doi.org/10.4135/9781412950589

Lovibond, S. H., & Lovibond, P. F. (1995). Manual for the depression anxiety and stress scale. Sydney, NSW:

Psychological Foundation.

Prochaska, J. O. and DiClemente, C.C. ( 1984 ). The Transtheoretical Approach: Towards a Systematic

Eclectic Framework . Dow Jones Irwin , Homewood, IL, USA .

Robinson, L.J., Stevens, L.H., Christopher, J.D., Vainiute, J., McAllister-Williams, H., & Gallagher, P. (2012).

Effects of intrinsic and extrinsic motivation on attention and memory, Acta Psychologica, 141(2),

243-249.


34

9 Appendix

- This page has been left blank intentionally -


35

Appendix A: Facilitator Feedback Booklet

The purpose of this document

This Facilitator Feedback Booklet allows for the process of evaluation to occur at every level. The

evaluation will assess the overall effectiveness of the program and will inform changes that are needed

to improve the program.

For the purpose of this document, facilitators are asked to complete;

A feedback form following each session, and

The Disruption to Services Register in Appendix A.

All of the information provided is fundamental to the evaluation process and we appreciate careful

consideration of the responses provided. Examples of the information required have been provided on

the feedback form for Session One. We also understand time constraints for facilitators and ask you to

please;

Allow a maximum of 10 minutes to complete each form,

Only complete the relevant sections of the form, highlighting your main concerns.

It is important to note that the RPP team is seeking this information to assist in conducting a well-

informed program evaluation, and not for the purpose of reviewing performance. Honest, constructive

and critical feedback from all facilitators is encouraged.

At completion of the program;

Facilitators will be asked to complete a brief telephone interview with RPP.

Senior clinicians (and facilitators may) will be asked to participate in a focus group.

Should you require any assistance with completing this document or if you are interested in learning

more about the underlying program theory and logic, please speak to the Senior Clinician at your

location or contact the RPP team directly.

Attention: Melanie Kiehne, Team Leader

Research and Professional Practice

Ph: 03 8417 0500

Email: [email protected]

Thank you for your time and consideration in contributing to the evaluation process. Please record your

details below. Location: ____________________________________________ Name of Facilitator: ____________________________________________

mailto:[email protected]


36

Session One Date: ________________

Participants in attendance: ___ out of ___ Length of session ________minutes

(actual delivery time)

Reasons for not attending:

Please tick all boxes of the content that was delivered:

Group Discussion – Welcome and Introductions (10 minutes)

Activity – Ice Breake: Tallest Tower (15 minutes)

Group Discussion – Participants’ Group Rules (15 minutes)

Activity – What do you know about ice? (15 minutes)

Group Discussion – Consequences and benefits of using ice (15 minutes)

Activity – Identifying reasons for use (20 minutes)

Activity – Understanding the effects of ice (25 minutes)

Group Discussion Understanding the effects/symptoms of ice on the brain & body (25 minutes)

Group Discussion – Mental Health (20 minutes)

Group Discussion – Learning how to help the brain recover from ice use (20 minutes)

1. Were any changes made to the content or structure of the session? (Please justify reasons for these

changes)

This is particularly important as we would like to keep a record of what sections of the program are not effective

and require improvement. Please identify whether changes were made to the information presented or to the

order in which information was presented.

2. Was there content in this session that was inappropriate or difficult for participants to understand?

In evaluating the program we will endeavour to understand if the content is appropriate and make changes

where necessary. If you needed to reiterate information for clients, if they could not engage in particular

activities, or if the content was difficult to deliver please make note of this here.

3. Did participants show an understanding of the content delivered and actively participate in

activities? How was this demonstrated?

It is important to have examples of how clients were able to understand and apply the content to their everyday

lives. Examples of how clients showed their understanding of content will allow for evaluation of how the

program assists participants to develop knowledge and skills that relate to the overall outcomes of the program.

4. Was the allocated time sufficient for each component of the session? Which sections require

more/less time?

Below, provide information about sections of the session where the time allocated did not ensure for client

participation. If the content could not be delivered within time indicated, which sections were most time

consuming. If there were any sections that did not fill the time allocated, please make a note of those also.

5. Were the objectives of the session met? If not, what prevented the objectives from being met?

Please tick which of the learning objectives were achieved and identify reasons preventing achievement in those

not met.

To begin establishing rapport and group cohesion.


37

To introduce participants to the aims of the program.

To develop a participation agreement.

To identify reasons for ice use and patterns of use.

To identify and understand the effects ice has on the brain and body.

To understand the neurobiology of ice.

6. How could this session be improved? Session Two Date: ________________

Participants in attendance: ___ out of ___ Length of session ________minutes

Reasons for not attending:

Please tick all boxes of the content that was delivered:

Activity – Ice withdrawal (15 minutes)

Guided Learning – Symptoms and duration of withdrawal (15 minutes)

Group Discussion – The methamphetamine abuse cycle (15 minutes)

Activity – Your craving experience (15 minutes)

Group Discussion – Develop strategies for managing the withdrawal process/cravings (25 minutes)

Activity – Your withdrawal experience (35 minutes)

Group Discussion – Lapse/relapse (5 minutes)

Activity – Experience of lapse and relapse (10 minutes)

Activity – Managing high risk situations (15 minutes)

Closing the program (30 minutes)

1. Were any changes made to the content or structure of the session? (Please justify reasons for these

changes).

2. Was there content in this session that was inappropriate or difficult for participants to understand?

3. Did participants show an understanding of the content delivered and actively participate in

activities? How was this demonstrated?

4. Was the allocated time sufficient for each component of the session? Which sections require

more/less time?

5. Were the objectives of the session met? If not, what prevented the objectives from being met?

To identify personal withdrawal symptoms.

To develop a withdrawal management plan.

To understand lapse and relapse.

To identify high risk situations.

To develop strategies for managing high risk situations.

6. How could this session be improved?


38

Interview component of booklet

Participants who completed the program: ____ out of ____ Date: _____________

Name of Facilitator: ________________

1. How did you find the program overall?

Please identify specific strengths and challenges of the program.

2. Did the manual allow for effective facilitation of the program?

2a. Do you have any suggestions for how this program could be improved? If so, please provide details

in the space below.

3. Were there any ongoing operational or resource problems, specific to the “Ice Effects” program that

need to be addressed?

For example: Difficulties with ensuring sufficient participant numbers

3a. What support do you require to manage these concerns in the future?

If you answered yes to the above question, please provide us with suggestions of how to address the problems

experienced with the “Ice Effects” program specifically.

4. How long did it take to complete the feedback booklet after each session and was this manageable?

5. Do you have any further comments about your experience of the piloting process, including delivery

of the program and participation in the evaluation process?


39

Disruption to Services Register

Session Description of Disruption

Please tick which disruption applied, and make comment where necessary.

1 ○ Lockdown ○ Program Clash ○ Facilitator

Availability ○ Lack of Space

Comments:

2 ○ Lockdown ○ Program Clash ○ Facilitator

Availability ○ Lack of Space

Comments:


40

Appendix B: Participant Focus Group

General

1. What did you think this program was about (or trying to achieve)? (i.e., methamphetamine,

harm minimisation)

Knowledge

2. What did you learn about ice? (i.e., elicit examples of knowledge learnt)

3. What did you learn about your ice use? (i.e., elicit examples of knowledge learnt)

Skill Development

4. How did the program help you manage your emotions?

Motivation

5. How do you plan to keep working on your ice use? (i.e., further treatment)

Competency

6. How do you feel about your ability to make change going forward?

Process

7. How did the group work together? (i.e., roles and participation)

Suggestions

8. If anything could be done differently, what would you like to see changed?

6 Hour Male Ice Effects Focus Group Schedule


41

Appendix C: Participant Satisfaction Survey

Drug and Alcohol Treatment Programs Feedback Survey: Short Form

Program Name:

Date: Prison:

This questionnaire is designed to provide Caraniche with information about their Drug and Alcohol

Treatment Programs from the perspective of clients. Please be as honest as possible in answering these

questions. Both positive and negative feedback will help us to ensure that programs are run at a high

standard.

Strongly

Disagree

Disagree Neither

Agree or

Disagree

Agree Strongly

Agree

1. I feel the program was suited to my

needs

1 2 3 4 5

2. The information provided in this

program is useful to me

1 2 3 4 5

3. The facilitator answered any questions

I had

1 2 3 4 5

4. I have learned a lot from this program 1 2 3 4 5

5. I understood the information provided

in this program

1 2 3 4 5

6. The facilitator did a good job in

delivering the program

1 2 3 4 5

7. As a result of the program, I feel more

confident about tackling my problems

1 2 3 4 5

8. I feel the facilitator did a good job in

managing the group

1 2 3 4 5

9. The program motivated me to work

on my problems

1 2 3 4 5

10. Overall, I am satisfied with this

program

1 2 3 4 5


42

Please provide some feedback

11. What did you like about the program?

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

12. How could the program be improved?

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

13. Would you recommend the program to others? Yes No

________________________________________________________________________________________

________________________________________________________________________________________

_______________________________________________________________________________________

14. Any other comments?

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

________________________________________________________________________________________

Thank you for your time.


43

Appendix D: DASS-21


44

Appendix E: Six Hour Ice Effects Knowledge Survey

1. Which of the following statements are true

a. You can only become addicted to ice if you inject it

b. You can become addicted to ice if you inject it or smoke it

c. It is rare to become addicted to ice

d. None of the above are true

2. What effect does ice have on the brain

a. Ice does not change the chemicals in the brain

b. Ice increases the ‘feel good’ chemicals (like dopamine) in the brain immediately

after using

c. Ice reduces the ‘feel good’ chemicals in the brain immediately after using

d. The ‘feel good’ chemicals in the brain return to normal during withdrawal from ice

3. Experiences of psychosis that come from using ice can last

a. 20 minutes

b. 2 hours

c. 2 days

d. All of the above

4. Withdrawal symptoms can last for

a. One week

b. One month

c. Longer than one month

d. All of the above

5. The most common mental health disorders that are associated with ice use are

a. Eating disorders, anxiety, and depression

b. Anxiety, depression, and psychosis

c. Personality disorders, depression, and ADHD

d. Ice does not affect mental health

6. Which of the following is not one of the 8D’s for managing cravings

a. Distract

b. Drink Water

c. Disagree


45

d. Deep Breath

7. Which of the following best describes a relapse

a. Using ice once after being sober for a long period of time

b. Getting high after being sober for a period of time

c. Returning to old patterns of ice use after being sober for a period of time

d. Using ice at a party after being sober for a period of time

8. When trying not to use ice, which of the following is an example of a high risk situation

a. Running a red light when driving

b. Having an argument with a family member

c. Going to work

d. Riding a bike without a helmet

Documents

Pilot Program Evaluation Ice Effectsresources-library.caraniche.com.au/system/files/RPP_011... · Ice Effects Pilot Program Evaluation 4 Executive Summary This report details the