Evaluation Methodology - womansencore.comwomansencore.com/wordpress/.../01/Across-Ages-Eva… · Web viewEvaluation efforts such as these are invaluable also to replication site staff

Evaluation Methodology

The Across Ages (AA) evaluation instrument measures the impact of the program on middle school students at high risk for substance abuse, as reflected in increased resiliency and protective factors. Published and adapted measures have been included in the evaluation instrument, which allows staff the opportunity to obtain valuable feedback, as well as measure participants’ progress. Evaluation efforts such as these are invaluable also to replication site staff who are interested in determining the value of the program and for anticipating how the program will function in their unique settings.

Evaluation begins with a plan for an overall strategy, based on the kinds of information you want to know. Keep in mind that a balance, between the kind of information program staff will consider useful and the kind of information granting agencies require, is crucial. Both groups of stakeholders can gain immeasurably from an evaluation that includes both “process” and “outcome” data, which take into account both the development and implementation aspects of a program, as well as the immediate results and long-term impact.

In selecting instruments for the outcome evaluation instrument, issues of reliability and validity are important.** Validity refers to whether a scale measures what it is supposed to measure, and reliability refers to whether a scale measures a construct accurately and consistently. Assessment of the reliability and validity of the AA youth survey began with the selection of several extant measures, including measures developed by the Positive Youth Development curriculum (Caplan, Jacoby, Weissburg, & Grady, 1988), as well as other published measures (Harter, 1985; Palmore, 1977). When necessary, scales were adapted for the AA program and modified for content and language. In addition, some original scales were included that addressed some issues of particular interest to program staff. Survey instruments and modifications were reviewed with focus groups conducted with teachers, students, and program staff. Reliability coefficients, as measured by the Cronbach’s Alpha statistical test, were recalculated for each modified measure. Only measures that demonstrated adequate reliability (.60 or higher on a scale of 0 to 1, with 1 indicating total accuracy) and validity have been used in the self-administered survey.

The survey is administered twice during the academic year, once in the fall and once in the spring. The pre-test contains 117 questions (including demographic household composition) and the post-test contains 105 questions (excluding demographic and household composition items). The instrument is presented to participants in paper-and-pencil format and is read aloud as students complete them. The construct definitions for the Across Ages Youth Survey used in outcome evaluations during the Across Ages I program are presented in Table 1. The psychometric (e.g., reliability) properties of all scales are presented in Table 2.

**While in most cases the scales are not restricted in their use, the authors or publishers should be contacted for permission.

Construct Definitions

Table 1. Youth Survey Construct Definitions

Youth Survey

Risk and Resiliency Outcomes

Attitudes Toward School, Future and Elders

Scale items focus on youths’ perceptions regarding academic achievement, their hopes for the future, and impressions of adults in their lives.

Attitudes Toward Older People Scale items focus on youths’ perceptions regarding people over age 50.

Harter’s (1985) Self-Perception Scale Two sub-scales of Self-Perception Profile for Children. Scale items assess children’s perceptions of personal competency and how much a child likes him/herself as a person.

Rand Well-Being Scale (Veit & Ware, 1983)

Scale items focus on general mood and emotional state and how often youth experienced feelings of loneliness, restlessness, and sadness in the past month.

Reactions to Stress or Anxiety(Caplan et al., 1990)

Scale items focus on the frequency of using eight behaviors to calm down when stressed or anxious. Additional question asks youth to indicate five things (of 10) they do to calm themselves down when they feel stressed or anxious.

Knowledge About Older People Scale items focus on facts about older people, aging, and health issues related to getting older.

Problem-Solving Efficacy(Caplan et al., 1990)

Scale items focus on the helpfulness of four pro-social techniques for dealing with difficult tasks.

Knowledge About Substance Abuse(Caplan et al., 1988)

Scale items focus on facts regarding use of various drugs, including alcohol, cigarettes, marijuana, and others.

Reactions to Situations Involving Drug Use (Caplan et al., 1986)

Two hypothetical dilemmas are presented and youth are asked to indicate five ways they might respond to peer pressure to drink beer and smoke cigarettes.

Attitudes Toward Community Service Scale items focus on youths’ perceptions of their ability to help others and sympathize with individuals with problems.

AOD Outcomes

Frequency of Substance Abuse(Caplan et al., 1990)

A measure of usage of cigarettes, beer, wine, hard liquor, cocaine, depressants, and stimulants in the past month. Scale includes an item (“donovites”) that checks for over-reporting of substance use.

Demographics

Gender Gender of youth

Date of birth Date and year used to calculate age of youth

Ethnic Background Race or ethnic background of youth

Language Language spoken at home of youth

Living Situation Number of adults living in home and family members youth lives with most of the time

Psychometric Properties

Table 2. Number of items and reliabilities for Across Ages Scales of Constructs

Youth Survey

Risk and Resiliency Outcomes Number of items Alpha reliability coefficient

Attitudes Toward School, Future and Elders

k = 11 N/A due to multidimensionalnature of scale

Attitudes Toward Older People k = 6 alpha = .46

Harter’s (1985) Self-Perception Scale k = 15 alpha = .78

Rand Well-Being Scale (Veit & Ware, 1983)

k = 12 alpha = .86

Reactions to Stress or Anxiety(Caplan et al., 1990)

k = 1 N/A due to single item scale

Knowledge About Older People k = 10 alpha = .70

Problem-Solving Efficacy(Caplan et al., 1990)

k = 4 alpha = .69

Knowledge About Substance Abuse(Caplan et al., 1988)

k = 14 alpha = .87

Reactions to Situations Involving Drug Use (Caplan et al., 1986)

N/A due to nature of scale (responses to two scenarios)

Attitudes Toward Community Service k = 8 alpha = .92

AOD Outcomes

Frequency of Substance Abuse(Caplan et al., 1990)

k = 8 alpha = .72

References

Caplan, M.Z., Jacoby, C., Weissberg, R.P., & Grady, K. (1988). The Positive Youth Development Program: A substance use prevention program for young adolescents. New Haven, CT: Yale University.

Caplan, M.Z., Weissburg, R.P., Bersoff, D.M., Ezekowitz, W., & Wells, M.L. (1986). The Middle School Alternative Solutions Test (AST): Scoring Manual. Unpublished manuscript. New Haven, CT: Yale University.

Caplan, M.Z., Weissburg, R.P., Grober, J.S., & Sivo, P.J. (1990). Social competence promotion with inner-city and suburban young adolescents: Effects of social adjustment and alcohol use. Unpublished manuscript. New Haven, CT: Yale University.

Harter, S. (1985). Manual for the Self-Perception Profile for Children. Unpublished manuscript. Denver, CO: University of Denver.

Palmore, E. (1977). Facts on aging: A short quiz. The Gerontologist, 17, 315-320.

Veit, C.T., & Ware, J.E. (1983). The structural of psychological distress and general well-being in the general population. Journal of Consulting and Clinical Psychology, 51, 730-742.

Implementation Guide

Survey Administration Procedures

General Instructions: The Across Ages evaluation instrument is designed to be administered by program staff to all youth participating in the Across Ages program. In the Across Ages program, each survey administration has been conducted in a classroom setting after consultation with school officials and classroom teachers. Program staff should be present during survey instrument administration and should read the instrument as students follow along. The evaluation instrument should also be administered to control group participants. Control group participants are similar in demographic composition and education level to the program participants, but do not participate in the program. The results from program participants are compared to those from the control participants to determine whether those youth who participated in the program have improved significantly over those who did not participate.

Prior to the beginning of the Across Ages program, students and teachers should be contacted and informed that they will be asked to participate in the program evaluation twice during the academic year, once in the fall and once in the spring. Students should be informed that their participation in the evaluation is voluntary and that they will not be identified individually. If incentives are to be given for participating in the evaluation (typically, $5 is given to each participant), students should be informed of this incentive.

Time Requirements Adequate staff time must be allocated for carrying out the evaluation. The written questionnaire can be administered in a classroom setting in about 45 minutes. The pre-test must be administered prior to the beginning of the program, and the post-test should be administered within two weeks of the end of the program.

Participants’ Permission/ConfidentialityChildren and their parents can often be wary of participating in a highly structured program evaluation. For this reason, and particularly if an evaluation team hopes to publish or present program findings, active parental consent is recommended. Written permission from each child’s (including children in the control group) parent/caregiver must be obtained prior to the administration of any evaluation instrument. Letters describing the evaluation procedure can be sent home with each student. Letters should emphasize that each child’s participation is voluntary and that all responses will be kept confidential. Permission slips should be included with the letter that parents should return to the child’s homeroom teacher for staff collection. Permission slips that are not returned can be followed up with a phone call from a staff member in the event that more explanation is required. If parental permission is not granted, the child should be taken from the classroom during survey administration, but should still be included in program activities. In cases where the parent gives permission and the child decides that he or she does not want to participate in the evaluation, the child is not forced to participate and may still take part in all project activities.

Administration Protocol Confidentiality is extremely important during evaluation procedures. Prior to the survey administrations, each participant should be assigned a unique identification number that should be placed on both the pre-tests and post-tests. In doing so, evaluation staff are able to link pre-test and post-test responses to determine whether participants have exhibited improvement over the course of the year. Participants are instructed during administration not to place their names on the surveys.

In assigning identification numbers to students, care should be taken that the numbers are unique to individuals. One method is to assign a prefix number that indicates whether the student is part of the participant group (for example, “01”) or a part of the control group (for example, “02”). Including these identifiers will facilitate data analysis. After assigning a prefix, each student can be given a number, for example, beginning with 001, and going as high as is needed for the particular sample size. An example of an identification number for a student in the control group would be 02-120, which indicates that the student is part of the control group and is number 120 in that group. No other student who is participating in the evaluation will have this same identification number.

After creating a coding scheme for assigning identification numbers, evaluation staff should create master lists that contain students’ names and identification numbers. These lists should remain the sole property of the evaluation team and should only be used to ensure that students are matched with the appropriate, labeled surveys during administration. Teachers, administrators, and program staff should not be allowed to view these lists, and they should be kept in a locked cabinet for the purposes of evaluation only.

After creating the master list, evaluation staff should label surveys with the identification numbers which have been carefully matched to student names. It should be brought to the attention of the participants during survey administration that their names do not appear on the survey and that their confidentiality is ensured.

Survey Administration

Survey administration should be conducted by members of the evaluation team whenever possible. Administration by program staff may make participants wary of providing honest answers and should be done only when necessary. In addition to bringing the labeled surveys and the master lists for matching participants to surveys, administrators should bring plenty of pencils for participants to use.

Prior to beginning survey administration, evaluation staff should consult with the classroom teacher to determine the accuracy of the master list and find out which students have been transferred, are absent, or have failed to returned a signed consent form. Those students who have been transferred should be removed from the list. Students who are absent on the day of administration should be noted and contacted on a “make-up” administration date, and arrangements should be made for them to complete the survey

instrument on this date. Several administration dates may be necessary during each administration period in order to minimize absentee effects and maximize the number of participants in the final sample.

Before beginning the survey, administrators should read aloud to participants the introduction and instructions. Participants should be informed that the purpose of the survey is to find out what they think about things and to improve school programs. Confidentiality of responses should be emphasized and participants should be informed that the survey is not a test. They should realize that there are no right or wrong answers, just honest answers, and that they should say what is true for them. Participants should be instructed that there should be no talking or discussing answers during the survey, and that this is important to keeping their answers confidential.

Preparing for Analyses

After participants have completed the survey instruments, they should be collected and organized by an evaluation team member. The completed surveys should be checked against the master lists to determine which students have completed them and which should be contacted for a make-up administration session. Completed surveys should then be given to the evaluation staff for coding and analysis, and for storage in a locked cabinet, separate from the master lists.

Scoring the Survey Instruments

The following coding scheme is recommended for the Across Ages Youth Survey.

ATTITUDES TOWARD SCHOOL, FUTURE, AND ELDERS

1. I like school.

2. I have learned much of value in school.

3. It is important to me to get good grades.

4. People who graduate from high school are better off than people who do not.

5. I often think about what I will do later on in life.

6. I think it is important to plan for the future.

7. I find it easy to be close to adults.

8. Adults like kids.

9. I like being with people.

10. *Things bother me a lot of the time.

11. On the whole I am satisfied with myself.

Scoring KeyAll items have the same response options, as presented below.

SCALE SCORE0 = NO! (Strongly Disagree)1 = no (Disagree) 2 = 0 (Neither Agree nor Disagree)3 = yes (Agree)4 = YES! (Strongly Agree)Phrases in parentheses should be written above each corresponding response option.

NOTE: * Item reverse-coded, that is, a higher score (e.g., a “4”) indicates a poor attitude as the item is presented originally. Prior to conducting analyses, this item should be recoded (the scale scores reversed) so that a higher number indicates a pro-social attitude, as do the other items on the scale. This allows the items to be combined to obtain a single score for each individual on this scale.

ATTITUDES TOWARD OLDER PEOPLE

1. There’s a lot I can learn from people over 50.

2. *People over 50 can’t understand kids.

3. I have a lot in common with people over 50.

4. *I am uncomfortable around people who are over 50.

5. *Most people over 50 don’t like kids.

6. It’s possible to have fun with people over 50.

Scoring KeyAll items have the same response options, as presented below.

SCALE SCORE0 = NO! (Strongly Disagree)1 = no (Disagree) 2 = 0 (Neither Agree nor Disagree)3 = yes (Agree)4 = YES! (Strongly Agree)Phrases in parentheses should be written above each corresponding response option.

NOTE: * Items are reverse-coded, that is, a higher score (e.g., a “4”) indicates a poor attitude as these items are presented originally. Prior to conducting analyses, these items should be recoded (the scale scores reversed) so that a higher number indicates a pro-social attitude, as do the other items on the scale. This allows the items to be combined to obtain a single score for each individual on this scale.

HARTER’S SELF-PERCEPTION SCALE

1. Some kids feel that they are very good at their schoolwork.

2. *Some kids find it hard to make friends.

3. *Some kids are often unhappy with themselves.

4. Some kids have a lot of friends.

5. *Some kids do things they know they should not do.

6. *Some kids have trouble figuring out the answers in school.

7. Some kids think that they are good looking.

8. Some kids behave themselves very well.

9. *Some kids don’t like the way they are leading their lives.

10. *Some kids wish their bodies were different.

11. Some kids are happy with themselves as people.

12. *Some kids often forget what they learn.

13. *Some kids wish that more people their age liked them.

14. *Some kids usually get into trouble because of the things they do.

15. Some kids like the kind of people they are.

Scoring Key All items have the same response options, as presented below.

SCALE SCORE0 = Not at all true for me1 = Not really true for me2 = Sort of true for me3 = Really true for me

NOTE: * Items are reverse-coded, that is, a higher score (e.g., a “3”) indicates a poor attitude as these items are presented originally. Prior to conducting analyses, these items should be recoded (the scale scores reversed) so that a higher number indicates a pro-social attitude, as do the other items on the scale. This procedure allows the items to be combined to obtain a single score for each individual on this scale.

RAND WELL-BEING SCALE

1. *I felt lonely.

2. I felt relaxed and free of tension.

3. I generally enjoyed the things I did.

4. *I felt downhearted and sad.

5. I could relax without difficulty.

6. *I was nervous or anxious.

7. *I felt restless, fidgety, or impatient.

8. *I felt moody and brooded about things.

9. I felt cheerful and lighthearted.

10. *I felt anxious or worried.

11. I felt that I was a happy person.

12. I woke up feeling fresh and rested.


SCALE SCORE0 = None of the time1 = Once in a while2 = Some of the time3 = Pretty much of the time4 = Most of the time


REACTIONS TO STRESS OR ANXIETY

When I feel nervous or anxious I:

1. Sit quietly and relax all the muscles in my body.

2. Imagine myself being in a peaceful place.

3. Focus on my breathing.

4. *Eat.

5. Try to make myself be calm and at ease.

6. Use a special method for relaxing.

7. *Go to my room and cry.

8. *Punch something really hard.


SCALE SCORE0 = Never1 = Once in a while2 = Sometimes 3 = Most of the time4 = Always


Additional item for reactions to stress or anxietyParticipants indicate by circling five different things that they do to calm themselves down when they feel anxious or stressed.

a. punch something really hardb. have a cigarette or a beerc. *do something helpful around the housed. *take deep breaths, then think about the probleme. *talk to my parents or a counselor about itf. go to my room and cryg. start a fight with someoneh. eat or bite my nailsi. *exercise or play sportsj. *mediate (sit in a quiet place and think)

Scoring KeyThe five choices marked with an “*” are acceptable ways to deal with anxiety or stress. This item may be coded on a scale from 0 to 5, with 1 point assigned for each “*” item circled. In doing so, a higher number (e.g. , a “5”) indicates that the participant has chosen all healthy ways of dealing with a problem, and a lower number (e.g., “0”) indicates that a participant has not chosen any healthy alternatives.

KNOWLEDGE ABOUT OLDER PEOPLE

1. *Intelligence goes down with age.

2. If it is very hot or very cold it may be dangerous for people over 50.

3. Most people over 50 can take care of themselves.

4. *Everyone becomes absent-minded sooner or later, if he or she lives long enough.

5. The number of people over 50 is growing.

6. *Depression is a serious problem for people over 50.

7. Deaths from stroke and heart disease are going down.

8. Many people over 50 are hurt in accidents that could have been prevented.

9. *More men than women survive to old age.

10. *If a person has been smoking for 30 or 40 years, it does no good to quit.

Scoring Key The response options for each statement are “True,” “False,” and “Don’t Know.” Participants should circle one response for each statement.

For items 2, 3, 5, 7, and 8, the answer is “True.” For items 1, 4, 6, 9, and 10 (items marked with an asterisk), the answer is “False.” Each item should be coded for correctness of the circled response. A response of “Don’t Know” is given zero points, as is an incorrect answer. The range for this scale, with higher numbers indicating greater knowledge, is 0 to 10.

SCALE SCORE1 = Correct answer0 = Incorrect answer; a response of “Don’t Know”

PROBLEM SOLVING EFFICACY

When you have a problem:

1. How helpful is it to you to stop and think before you act?

2. How helpful is it to think of lots of different things you could do to solve the problem?

3. How helpful is it to think ahead to the consequences of your actions before you do something?

4. How helpful is it to make a plan to solve the problem?


SCALE SCORE0 = Not at all helpful1 = A little helpful2 = Halfway helpful3 = Helpful4 = Very helpful

For this scale, all four items are socially desirable ways to solve a problem. The lowest possible score is 0, and the highest is 16.

KNOWLEDGE OF SUBSTANCE ABUSE

1. *Children under age 12 can not become addicted to drugs.

2. Sometimes smoking marijuana (“pot,” “grass”) causes a feeling of panic

3. *A person who has smoked marijuana (“pot,” “grass”) can drive a car as well as a person who has not smoked marijuana.

4. *You can not get addicted to crack.

5. *You can not get AIDS from sharing drug needles.

6. Cocaine can cause heart failure.

7. Marijuana destroys brain cells.

8. *Alcohol is a stimulant. It usually makes you more awake and alert.

9. Cigarette smoking causes tar to build in the lungs.

10. *Marijuana is also called “meth.”

11. Using a lot of cocaine (“coke” or “crack”) over a short period of time can make you feel really down.

12. *Amphetamines are also called “downers.”

13. *More people use marijuana than use alcohol.

14. *Alcohol has no effect is you smoke marijuana at the same time.


For items marked with an asterisk, the answer is “False.” Each item should be coded for correctness of the circled response. A response of “Don’t Know” is given zero points, as is an incorrect answer. The range for this scale, with higher numbers indicating greater knowledge, is 0 to 14.


REACTIONS TO SITUATIONS INVOLVING DRUG USE

SCENARIO 1

Here is a situation that we want you to think about carefully.

You are at a friend’s house when his/her parents are out. After watching T.V. for a while, your friend pulls out a pack of cigarettes and asks you to try one.

Circle only FIVE things you might do or say in this situation.

1. I would say “O.K.”

2. I would not say anything, but just sit and watch him/her smoke.

3. *I would tell him/her “No,” and ask him/her to give them to me.

4. I would say, “Well, smoking isn’t that bad. Let’s smoke!”

5. *I would say, “You shouldn’t do that. It’s not right.”

6. *I would grab the cigarettes and throw them away.

7. *I would tell him/her to stop, because if he/she doesn’t they will lose my friendship.

8. I would get angry, yell at him/her, and leave.

9. I would say that smoking is bad for your health.

10. *I would say, “No way. You should stop for your parents and for yourself.”

SCENARIO 2

Here is another situation that we want you to think about carefully.

You are hanging around with some of your friends in your neighborhood. A couple of kids start drinking a beer, and they ask you to try some.

Circle only FIVE things you might do or say in this situation.

1. I would say “O.K.”

2. I would not say anything, but just sit and watch them drink.

3. *I would tell them “No,” and ask them to give the beer to me to throw away.

4. I would say, “Well, drinking isn’t that bad. Give me some!”

5. *I would say, “You shouldn’t do that. It’s not right.”

6. *I would grab the beer and pour it out.

7. *I would tell them to stop, because if they don’t they will lose my friendship.

8. I would get angry, yell at them, and leave.

9. I would say that drinking is bad for your health.

10. *I would say, “No way. You should stop for your parents and for yourself.”

Scoring KeyThe five choices marked with an “*” on each scale (they are the same choices but differ in wording for the two situations) are acceptable ways to deal with these situations. These two situations may be coded together on a scale from 0 to 10, with 1 point assigned for each “*” item circled. In doing so, a higher number (e.g. , a “10”) indicates that the participant has chosen all healthy ways of dealing with the situations, and a lower number (e.g., “0”) indicates that a participant has not chosen any healthy alternatives.

ATTITUDES TOWARD COMMUNITY SERVICE

1. *Most people in nursing homes are crazy.

2. *At my age, I can’t do very much for someone who has problems.

3. I spend more time helping other people than most of my friends do.

4. I make a special effort to help people with problems.

5. *I don’t think I can do anything to solve the problems in my neighborhood.

6. I have talents and skills that can be helpful to others.

7. I have done something important for others.

8. *I get scared or disgusted when I see someone who is handicapped in some way.


For items marked with an asterisk, the socially desirable answer is “False.” Each item should be coded for correctness of the circled response. A response of “Don’t Know” is given zero points, as is an incorrect answer. The range for this scale, with higher numbers indicating more prosocial attitudes, is 0 to 8.


FREQUENCY OF SUBSTANCE ABUSE

Circle how often you have tried any of the following drugs in the past 30 days.

1. Tobacco (cigarettes, pipe, cigar)

2. Marijuana (grass, weed, hash)

3. Alcohol (beer, wine)

4. Hard liquor

5. Cocaine (coke)

6. Crack

7. Donovites (vinnies, phos)

8. Depressants (downers, seccy, seconal)

9. Stimulants (speed, uppers)


SCALE SCORE5 = Never4 = Less than once a month3 = Once or twice a month2 = Once a week1 = 2 or 3 times a week0 = Almost every day

NOTE: Item 7 is a fictitious drug. It is included to detect over-reporting of substance abuse. If a respondent indicates that he or she has used this drug, the substance abuse scale and the rest of the respondents’ survey data should be examined for possible response bias.

To remain consistent with the other scales in this instrument, the responses have been coded so that higher scores indicate less substance abuse and therefore are more socially desirable. If desired, it is possible to recode the response options so that higher scores indicate more substance abuse. The decision regarding the coding of this scale should be made by the evaluation and program teams to ensure that the findings are easily interpretable.

Documents

Evaluation Methodology - womansencore.comwomansencore.com/wordpress/.../01/Across-Ages-Eva… · Web viewEvaluation efforts such as these are invaluable also to replication site staff