Workbook 2 - Request Rejectedwhqlibdoc.who.int/hq/2000/WHO_MSD_MSB_00.2c.pdf · Workbook 2 • Implementing Evaluations 5 WHO/MSD/MSB 00.2c Table of contents Overview of workbook

1Workbook 2 · Implementing Evaluations

WHO/MSD/MSB 00.2c

Workbook 2

ImplementingEvaluations

2 Evaluation of Psychoactive Substance Use Disorder Treatment

WHO/MSD/MSB 00.2c

WHOWorld Health Organization

UNDCPUnited Nations International Drug Control Programme

EMCDDA European Monitoring Center on Drugs and Drug Addiction

World Health Organization, 2000c

This document is not a formal publication of the World Health Organization (WHO) and all rights are reserved by theOrganization. The document may, however, be freely reviewed, abstracted, reproduced and translated, in part or inwhole but not for sale nor for use in conjunction with commercial purposes. The views expressed in documents bynamed authors are solely the responsibility of those authors.


WHO/MSD/MSB 00.2c

ited the workbook series in laterstages. Munira Lalji (WHO, Sub-stance Abuse Department) and Jen-nifer Hillebrand (WHO, SubstanceAbuse Department) also edited theworkbook series in later stages.Maristela Monteiro (WHO, Sub-stance Abuse Department) pro-vided editorial input throughout thedevelopment of this workbook.

Some of the material in this work-book was adapted from a NIDApublication entitled “How Good isYour Drug Abuse Treatment Pro-gram? A Guide to Evaluation.” Con-tributions drawing from this reportare gratefully acknowledged.

The World Health Organizationgratefully acknowledges the contri-butions of the numerous individu-als involved in the preparation ofthis workbook series, including theexperts who provided useful com-ments throughout its preparation forthe Substance Abuse Department,directed by Dr. Mary Jansen. Finan-cial assistance was provided byUNDCP/EMCDDA/Swiss FederalOffice of Public Health. Cam Wild(Canada) wrote the original text forthis workbook and Brian Rush(Canada) edited the workbook se-ries in earlier stages. JoAnneEpping-Jordan (Switzerland) wrotefurther text modifications and ed-

Acknowledgements


WHO/MSD/MSB 00.2c


WHO/MSD/MSB 00.2c

Table of contents

Overview of workbook series 6

Introduction 7

The six steps of implementing evaluations 8

Step 1. Prepare for data collection 10

Step 2. Colect data 22

Step 3. Analyse data 24

Step 4. Report results 40

Step 5. Make use of what was learned 44

Step 6. Start again 46


WHO/MSD/MSB 00.2c

Overview ofworkbook seriesThis workbook is part of a series in-tended to educate programme plan-ners, managers, staff and other deci-sion-makers about the evaluation ofservices and systems for the treatmentof psychoactive substance use disor-ders. The objective of this series is toenhance their capacity for carrying outevaluation activities. The broader goalof the workbooks is to enhance treat-ment efficiency and cost-effectiveness

using the information that comes fromthese evaluation activities.

This workbook (Workbook 2) de-scribes step-by-step methods forimplementing evaluations. Thesesteps span from starting the study,to collecting, analysing, and report-ing the data, to putting the resultsinto action in your treatmentprogramme.

Introductory WorkbookFramework Workbook

Foundation WorkbooksWorkbook 1: Planning EvaluationsWorkbook 2: Implementing Evaluations

Specialised WorkbooksWorkbook 3: Needs Assessment EvaluationsWorkbook 4: Process EvaluationsWorkbook 5: Cost EvaluationsWorkbook 6: Client Satisfaction EvaluationsWorkbook 7: Outcome EvaluationsWorkbook 8: Economic Evaluations


WHO/MSD/MSB 00.2c

Introduction

You should use the appropriatespecialised workbook with thisworkbook as you move through thesteps of evaluation implementation.

This workbook describes the generalsteps involved in conducting treatmentevaluation, including selecting/prepar-ing your data collection measure(s),deciding how to administer themeasure(s), and developing a data col-lection and management plan. Theworkbook then provides advice andsome technical details regardinganalysing and reporting the results.Some sections should be read by allevaluation partners, whereas others aremost appropriate for those trained asresearchers and/or functioning asevaluation consultants. Each sectionis marked appropriately.

As the skills and resources that areavailable for evaluation vary widely,you may not be able to undertake allof the steps and procedures that areoutlined here. However, you should try

to adhere to the basic principles andsteps that are presented. You alsoshould try to acquire additional re-sources you may need.

Workbook 1 (on evaluation planning),and this workbook (on implementa-tion), will give you foundation toolsand information to plan and imple-ment programme evaluation. With anunderstanding of the material in thesetwo workbooks, and some practiceapplying it, you will be prepared toundertake evaluations of differenttypes. The remaining specialisedworkbooks in this series present theprinciples and practices of these dif-ferent types of evaluation. You shoulduse the appropriate specialised work-book with this workbook as you movethrough the steps of evaluation imple-mentation.


WHO/MSD/MSB 00.2c

The six steps ofimplementing evaluations

1. Prepare for data collection.

2. Collect data.

3. Analyse data.

4. Report results.

5. Make use of what was learned.

6. Start again.

As described in the framework manual,there are six steps that you need toaccomplish to implement evaluations:

Each of these steps is presented below. Thediscussion of these steps includes someexercises for you to follow that will help

you learn and apply the material to yourown situation.


WHO/MSD/MSB 00.2c

If you are conducting a... Then you should review...

Needs Assessment Evaluation Workbook 3

Process Evaluation Workbook 4

Cost Evaluation Workbook 5

Client Satisfaction Evaluation Workbook 6

Outcome Evaluation Workbook 7

Economic Evaluation Workbook 8

Use the foundation and specialisedworkbooks together, to help youmake the most of the informationthat is presented.

Workbooks 1 and 2 provide a solid foun-dation of general information about con-ducting evaluations, whereas thespecialised workbooks (Workbooks 3through 8) present detailed information fordifferent types of evaluation. If you alreadyknow what type of evaluation you are go-ing to conduct, you should consult the

workbook that is applicable. Use the foun-dation and specialised workbooks to-gether, to help you make the most of theinformation that is presented. If you donot know what type of evaluation you aregoing to conduct, wait until you have de-veloped your evaluation questions (Step5) to consult a specialised workbook.


WHO/MSD/MSB 00.2c

Step 1

Prepare fordata collectionThere are a few important tasks thatneed to be accomplished when startingyour study. These include:

1 managing ethical issues

2 developing a data management plan

3 conducting a pilot test

4 writing an evaluation plan.

These tasks are explained in detail below.

1. Manage ethical issues

People being studied need to be protectedfrom undue burdens on their time, inva-sion of privacy, and risks or other harms.You need to pay attention to these ethi-cal issues!

For all studies involving people as partici-pants (sometimes called “human subjects”studies), you must first ask approval froman ethics committee, and/or consult theWHO-CIOMS publication, InternationalEthical Guidelines for Biomedical Re-search Involving Human Subjects(1993). These guidelines are intended toprotect the rights of participants. Some ofthe basic principles are outlined below.

Informed Consent. According to this prin-ciple, researchers must ask permission ofindividuals before enrolling them in theresearch study. When you do this, you

The standard practicae is to have a Con-sent Form that informs clients aboutthese procedures and that asks for theirwritten consent for data to be collectedfrom them or about them. A consentform typically:

• describes the purposes and methods ofthe study explains study requirements,for example, completion of a question-naire, the kinds of information re-quested, the time to complete thequestionnaire, the amount of time in-volved

No personshould beforced orcoerced toparticipatein the study.

should explain to them the purpose, na-ture, and time of their participation. Givethem, or have available upon request, de-tailed information about the evaluation inwriting. No person should be forced orcoerced to participate in the study.


WHO/MSD/MSB 00.2c

• describes the purposes and methods ofthe study explains study requirements,for example, completion of a question-naire, the kinds of information re-quested, the time to complete the ques-tionnaire, the amount of time involved

• explains that participation is voluntaryand individuals can withdraw from thestudy at any time without fear of re-prisal such as withholding of treatmentthat the person would otherwise beentitled to receive

• explains the extent to which confiden-tiality will be maintained

• explains all foreseeable benefits andrisks that the study poses

• explains any monetary or other benefits (ifapplicable) that accompany participation

The investigator must communicate allnecessary information, give the prospec-tive subject full opportunity to ask ques-tions, and exclude the possibility of un-justified deception or intimidation in theconsent process. The participant mustsign the consent form before the infor-mation is collected, and must receive acopy of the signed form. If a participantis illiterate, you must read the consentform and obtain verbal consent. Note thatconsent forms also may be needed forevaluations that use archival recordsabout individuals.

Take necessaryprecautions toprotect thesafety of yourparticipants.

Risks of participation

Before beginning a study, researchersshould thoroughly evaluate risks that mightbe present for participants. The typical risksin evaluating treatment programmes forPSU disorders are accidental disclosure ofthe information obtained about patients’PSU and illegal activities. In addition, evalu-ations involving biomedical interventions orconfrontational tactics could harm clientsphysically or psychologically. Take neces-sary precautions to protect the safety ofyour participants.

Participants may be paid for their inconve-nience and time spent, and may receive freemedical services in the course of participa-tion. However, payments should not be solarge, or medical services so extensive, sothat they would induce people to consent toparticipate against their better judgement.

Confidentiality

All researchers have an obligation to pro-tect the confidentiality of their participants.

Researchers must never reveal the name,address, or any identifying information abouta participant unless they have written per-mission to do so. If researchers want topublish a case report of a participant or agroup of participants, they must changeenough details so that no one could discoverthe identity of the person or group.

You should develop procedures to ensurethat all information collected about anyperson will remain anonymous. No namesor other data should be kept that couldidentify an individual with the informationgathered. If the data collection plan in-cludes follow-up with clients over time tomeasure services received or individualoutcomes, each client can receive a uniqueidentification (ID) number. This numbershould be the only identifier used in dataprocessing and analysis. Only authorisedpersonnel should have access to the namesand other information linked to the IDnumbers for use during the evaluation. Inaddition, everyone who collects or pre-pares data should sign a pledge signifyingthey will adhere to all confidentiality pro-cedures.

You shoulddevelopprocedures toensure that allinformationcollected aboutany person willremainanonymous orconfidential.

• uses simple words and concepts thatcan be easily understood by partici-pants

• explains that they are entitled to freemedical care for any harm that resultsfrom participation


WHO/MSD/MSB 00.2c

Into action

Our evaluation team discussed all possiblerisk that might be present for participantsand how they are going to ensure that allparticipant information will remain confi-dential. They also consulted the WHO-

CIOMS publication, International EthicalGuidelines for Biomedical Research In-volving Human Subjects (1993) in orderto write a consent form for their evalua-tion project.

They created the following consent form:

Our inpatient opiate detoxification programme is conducting a study theeffectiveness of our AIDS education intervention.Participants will completea questionnaire at their initial assessment interview at arrival into ourprogramme. The questionnaires will ask about attitudes, beliefs, HIV-riskbehaviour and psychoactive substance use. Completion of the questionnaireswill take about 15 minutes. Participants are free to skip any questions thatthey find troubling or overly personal.

The purpose of this research is to find out if our AIDS education interven-tion is useful for people with opioid dependence. This information will behelpful in providing better information on HIV for opioid dependent people.

All questionnaires and information collected from your initial assessment willbe kept confidential; only a code number for each participant will appear on thequestionnaires and initial assessment sheet. All information about clients in thisservice will be kept in a safe, locked location. The names of clients will be keptseparate from the information itself.

There is no risk involved in this research, other than the unpleasant feelingsthat might arise when thinking about problems associated HIV and your opioiduse. Your drug worker is available to talk to if you experience unpleasantfeelings as a result of participating in this study. There are not likely to be anydirect benefits to participants.

Participation in this research is voluntary. The decision to participate will notaffect the nature of the treatment services participants receive. Your are free towithdraw from the study at any time without giving any reason and withoutprejudice to your continued care.

I understand all the above information, and agree to participate in this study. Iunderstand that I may withdraw from the study at any time.

Participants Signature Date


WHO/MSD/MSB 00.2c

Establish well-defined objectives for yourdata collection system, and make a strongcommitment to achieving the objectives.Your data management plan must also ad-dress:

• designing your record keeping system

• acquiring and processing the data

• training the people who will collect data

Your evaluation planning group shouldconsider how to best manage ethical is-sues in your study. Use the examples pro-vided below as a guide, then adapt themto your own evaluation project.

1 Each partner should individually writedown possible safety issues (harm thatcould come to your participants) as aresult of participating in your research.After each partner has finished, youshould discuss these issues as a group.Write down a specific plan for how youwill minimise or limit this harm.

Example: In a questionnaire surveystudy of individuals being treated foropioid dependence, possible risks in-clude a) unpleasant feelings that mightarise from thinking about problems as-sociated with opioid use, and b) pos-sible breaches of confidentiality. Tominimise this potential harm: a) all par-ticipants will be instructed at the timeof enrolling that they may have accessto a trained mental health professional

It�s your turn: (1A)

to discuss any unpleasant feelings thatmight arise as a result of participation,and b) use of confidentiality policies andprocedures (see below).

2 Working separately, write down a planfor how you will ensure that all partici-pant information will remain confiden-tial. Discuss your ideas as a group andwrite down a specific plan for how youwill maintain confidentiality.

Example: The data will be kept in a safe,locked location; the names of participantswill be kept separate from the data itself;all research staff will sign a pledge of con-fidentiality.

3 Working as a group, develop a consentform for participants to sign. This formshould outline the study itself, as well asall risks of participation. Use the informa-tion provided in the preceding section andthe example below as a guide. Also con-sult your institution’s ethical guidelines onconsent of participants, if applicable.

2. Develop a data management plan

Another important step of starting a studyis to develop a data management plan.Ideally, a programme administrator shouldmake specific staff responsible for theimportant task of data management - col-lecting and editing the data. Those whouse and interpret the data should under-stand and appreciate the strengths andweaknesses of the information collected.Assignment of clear responsibilities fordata quality is an important part of the pro-cess.


WHO/MSD/MSB 00.2c

Development ofa recordkeeping systemor databasedesign is animportant partof a successfulevaluation.

Development of a record keeping systemor database design is an important part ofa successful evaluation. The following sug-gestions may be helpful in guiding youthrough this process:

l It is a good idea to delegate one per-son to monitor the data collection pro-cess. This person will ideally takecharge of administering theinstrument(s), to make sure that thedata are collected properly, monitor theforms and questionnaires that are used,oversee the storage and retrieval ofdata, and supervise the data analyses.

l Keep all your records in one place.This includes any questionnaires, in-terview schedules, tapes, data collec-tion forms, and archived materials. Bystoring all information associated withthe evaluation in one place you canavoid problems of having to searchmultiple locations to find the informa-tion you require. Develop a system forhow you will organise the records.(Organisation by ID number is usuallya good idea.)

l Remember to use ID numbers insteadof names, and to remove any identify-

ing information from the centralrecords.

l Assign a person (or group of people)for regular transferral of information onthe records to a central “database.” Thiscould be a computer file (see below), orcould be a notebook containing ID num-bers and summary data for each partici-pant. This transfer of information shouldbe completed routinely (once a week isa good guideline).

l Keep the database in a safe location.Make a “back up” copy if possible, incase the original data base is damagedin some way. You can back up com-puter data onto a separate diskette, orphotocopy written records. Back upcopies should be stored in a separatelocation from the central database, incase of fire, flood, theft, etc.

l Make a habit of routinely picking a fewcases from the database to ensure thatthe information in the database corre-sponds with the information from theoriginal records (e.g., questionnaires).This will help to spot errors and/ormake corrections to your data trans-ferring procedures.

This will require extensive pilot testing toidentify errors and shortcomings that al-most always exist in the initial stages. Italso requires careful design to make thebest use of limited storage, and to keepdata processing time to a minimum. Forthese reasons, it is important to use askilled computer programmer to designand pilot test the database. It is equallyimportant that the programmer have agood understanding of what you do, andwhat you need from the database.

Database software for microcomput-ers includes dBASE, Paradox, Rbase,Dataease, Access and Excel. Becom-ing proficient with any database soft-ware requires many hours of use.Consequently, databases should be de-veloped only by computer program-mers with extensive training or expe-rience with the selected software. Aspreadsheet programme might be a bet-ter choice for a treatment programme.Many of the latest spreadsheet pro-

Designing a record keeping system

Designing and setting up an automated(computer) database


WHO/MSD/MSB 00.2c

grams, including LOTUS 123 and Ex-cel, are easier to use. Database man-agers are capable of filling most of thedata processing needs you are likelyto have. They have data entry screensthat make them easy to use, and theycan be used for analysis.

If you are using a computer database, youmay want to consider obtaining specialisedsoftware to conduct statistical analyses ofquantitative data. A variety of options areavailable, ranging from general programscapable of performing a wide range ofanalyses (SPSS, SAS), to specialised pro-grams that handle specific types of analy-ses or provide simple statistical calcula-tions (Minitab, EPI-Info). EPI-Info is aspecialised piece of software designed fororganising and analysing data from evalu-ations, surveys and other studies of healthand social services. It is shareware, whichmeans it is available free of charge. Youmay order a copy from: The Division ofSurveillance and Epidemiology, Centresfor Disease Control (CDC) Atlanta, Geor-

gia 30333., USA: Phone 404-728-0545;Fax 404-315-6440; Internet www.cdc.gov.

If you are using a computer system, eachcompleted form should be edited for miss-ing data and other problems before the dataare entered in the computer. Dependingon the number of forms involved and de-lays due to data problems, this phase cantake up to a month to complete. After in-formation from a batch of forms is ed-ited, the data can be entered and processed.Routine processing should be done everytwo weeks, to allow enough data to buildup for efficient operations. Exceptions canbe made when the flow of forms warrantsmore or less frequent processing. Auto-mated procedures for data entry should befully explored before the task is delegatedto someone to enter the information intothe computer by hand.

If you are not using a computer, an inde-pendent person can be assigned to ran-domly spot check for data entry mistakesfrom the original data entry form to thecentral data notebook.

Selectingknowledgeable,competent datacollection staffis a key tosuccess forhigh qualityprogrammeevaluation.

Data collection staff

Selecting knowledgeable, competent datacollection staff is a key to success for highquality programme evaluation. Your datacollection staff should be trained in rou-tine or standardised procedures for inter-viewing and/or administering question-naires. Training should include anoverview of the purpose of the study, thepurpose of each research measure and howit is used, and instructions for recordingor filling in responses to each item. Thetraining should also include some practicesessions to ensure that the staff membersare comfortable and competent in usingthe measures. In addition, the staff shouldhave a thorough understanding of partici-pants’ rights to confidentiality, and a planfor handling unusual or emergency situa-tions (e.g., suicide attempts, “overdose,”

crises at home or work) that may arise inthe course of data collection.

Data collectors should be trained in theimportance of standardisation of proce-dures. This means that they should col-lect data the same way with each partici-pant. Why? Standardisation helps to ensurethat differences in participants’ responsesare not due to differences in how the dataare collected. For example, if a data col-lector interviews some participants in agroup situation, but others alone, thenit is possible that differences in responsesare due solely to this different interviewformat.

The same principle is true between datacollector: if one data collector is very


WHO/MSD/MSB 00.2c

Standardisationhelps toensure thatdifferences inparticipants�responses arenot due todifferences inhow the dataare collected.

Into action

friendly to participants, while another isdistant, then participants may give differ-ent types of responses. Clear training instandardised procedures helps to preventthese possibilities.

When considering standardisation, thinkabout the following issues:

l Will data be collected individually, orin the presence of others?

l Will participants complete question-naires on their own, or receive guid-ance from the data collector?

l How “friendly” should data collectorsbe with participants?

l How should data collectors respond toquestions from participants?

There is no right or wrong answer to thesequestions: the key to success is ensuring thateveryone does it the same way every time.This is the essence of standardization.

Data collection also should involve exten-sive communication between those collect-ing the data and those managing it. Issuesinevitably arise in the data reporting prac-tices, and these must be understood by thoseusing the data. Ideally, select a data co-ordinator who has enough time to obtain,edit, and co-ordinate collection procedures,and who can communicate effectively aboutthe complexity of your data collection.

Chris C. was selected as the data co-ordinator. He would supervise the datacollection and will train all other drugworkers at the service in standardisedprocedures for administering the ques-tionnaire. All data would be stored inlocked file cabinets. Chris C. would havethe keys for these cabinets. He wouldalso collect questionnaire and file themin a “data to be entered” file within thecabinet. Once a week, Chris C. wouldenter these records into the main recorddatabase using the database softwareSPSS. Once every two weeks, Sue R.would double check Adam’s work toensure that errors are not being made.A separate list, linking data code num-bers to names, would be stored in theSue R. office in a locked file cabinet. Shewould be the only one to have the keysto this file cabinet.

The team decides to conduct a 3 hour-workshop with all data collector (drugworkers at the service) in order to trainthem in standardised procedures of ad-

ministering the questionnaire. This train-ing would include information about pur-pose of the study, about the chosen mea-surements and instructions for filling inresponses to each item as well as somepractice session. Further, the data collec-tion staff would be informed about thestandardised procedures:

1 The questionnaire will be administeredindividually

2 Participants will complete question-naires by themselves

3 In case of difficulties in completing thequestionnaire the client will receiveguidance from the data collector

4 The data collector should respond toquestions as discussed in the trainingworkshop

5 The data collector provides time forquestions about the study, after the cli-ents filled in the questionnaire.


WHO/MSD/MSB 00.2c

It�s your turn (1B)

Working as a group, complete the follow-ing tasks:

1 Write down where you will store yourdata, how it will be organised to ensureclarity and protect participants’ confi-dentiality.

Example: Data will be stored in lockedfile cabinets in the main storage area. Onlythe data co-ordinator will have the keys.A separate list, linking data code numbersto names, will be stored in the investi-gator’s office in a locked file cabinet. Onlythe investigator will have keys to this file.

2 Write down how you will maintain cen-tral records of the data, and who will beresponsible for maintaining the records.

Example: A data co-ordinator (Jon)will collect questionnaires and file themin a “data to be entered” file within thecabinet. Once a week, Jon will enterthese records into the main record da-tabase. Once every two weeks, Serenewill double check Jon’s work to ensurethat errors are not being made.

3 Write down standard procedures foryour data collectors, and a plan for howyou will train them in these procedures.

Example: All data collectors will re-ceive training in how to obtain consent,administer the questionnaires in astandardised fashion, and deal withemergency situations. Michel (seniorclinician) will conduct this training. Be-fore being allowed to collect data, datacollectors will be tested and approvedby the study investigator.

4 If you plan to use clinicians as data col-lectors, consult with them to ensure thatyour data collection plan is feasible fromtheir perspective. (Because clinicianssometimes feel too busy to use standard-ised forms or participate in research,gaining their co-operation at the outsetis essential.) Work with them to achievea mutually agreeable plan.


WHO/MSD/MSB 00.2c

3. Conduct a pilot test

(Note: This section is most applicable forresearchers. If you have a researcheramong your evaluation partners, have thatperson read this section and then explainits contents briefly to the others.)

Once you have reached this point, you areready to try out your data collectioninstrument(s) in a small pilot test. The ba-sic idea behind conducting a pilot test isto try out your data collectioninstrument(s) in a small, preliminarysample in order to answer the followingquestions:

l Does the data collection instrument pro-vide useful information? For example, arevariables being measured accurately?

l Can the instrument be administeredproperly, given constraints on time andstaff? For example, is the instrument toolong or too complicated to be com-pleted?

l Can the information be easily man-aged? For example, can the form(s) beintegrated with the data managementsystem easily?

l Does other information need to be col-lected? How did people react to com-pleting the instrument? What is miss-ing? How should the questions bechanged, and why?

If a formal pilot test is not possible, havestaff, community members, or patientscomplete the draft data collection formsto see which parts may be unclear, con-fusing, or otherwise misleading. If a moreformal pilot test is feasible, consider thefollowing suggestions in testing differenttypes of forms:

The reason fora pilot test isto identify anyflaws in thedata collectionplan so theycan becorrectedbefore a full-blownimplementationof theevaluation getsunderway.

1 Self-administered questionnaires:Select 10-15 people who are similarto your clients; ask them to completethe form; and analyse the data. Re-view the completed forms carefully,looking for potential problems: Isthere a particular question severalrespondents did not answer? Are re-spondents answering questions theyshould have skipped? Are somechecking or circling more than oneresponse category when instructionssay to give only one answer? If suchproblems appear (and they usuallydo), talk to some respondents whofailed to answer properly. Ask themwhy they answered as they did. Thewording may be unclear, or the for-mat may need improvement. Usingspecial type face features might help,such as italics, underlines, or boldtype for instructions.

2 Interviews: Conduct several pilot in-terviews to test the interview instru-ment. Consider having a group discus-sion with pilot test respondents afterthey have completed their individualinterviews. A moderator would leadthe people being interviewed throughthe form as a group, encouraging themto talk about any problems they hadwith the questions or the interviewer’sstyle, and to suggest changes. Groupdiscussions can be an effective way toidentify problems and gather sugges-tions. They also may point out differ-ences in item interpretation that wouldremain hidden in individual interviews.

3 Record Reviews: Pilot testing yourrecord review forms can be done effi-ciently as part of staff training for thisactivity. Problems in wording or for-mat should become clear after a fewtrials or exercises.


WHO/MSD/MSB 00.2c

The reason for a pilot test is to identifyany flaws in the data collection plan sothey can be corrected before a full-blown implementation of the evaluationgets underway. Thus, the last step, inthe pilot test phase is to summarise find-ings and to revise the data collectionplan as needed. If the changes are sig-nificant, it may be advisable to conductanother pilot test of the revised proce-dures before proceeding with the evalu-ation.

It�s your turn (1C)

1 The researcher in the group should ex-plain the key points of this section tohis/her evaluation partners. For ex-ample, explain the important functionsthat pilot tests serve, and how to con-duct them properly.

2 As a group, write down when you willconduct a pilot test, and how manypeople will participate in each portionof it. Use the information in the pre-ceding section as a guideline for choos-ing the number and type of people foryour pilot test.

You may want to have a staff meeting tointroduce the evaluation and the data col-lection plans to everyone else who will beinvolved. This meeting would explain thereasons for the evaluation, the questionsto be addressed, and the importance of eachperson’s role in answering the questions.

After you have pilot-tested your data col-lection instrument and have incorporatedany changes that are needed, you are readyto formally start collecting data.

Into action

The next task was to conduct a pilot test.Each of the drug workers administeredthe baseline questionnaire to two clients.Afterwards the collectors and the evalu-ation team discussed identified flaws.The data collectors reported that giventhe emotional and physical state of someclients, it seemed not wise to administerthe questionnaire at the first face- to-face interview with the client. They sug-gested that the questionnaire should be

given to the client during the second dayof his stay at the unit during his secondindividual contact with his drug worker.The evaluation team agreed to adminis-ter the questionnaire during the sug-gested second interview with the drugworkers. Apart from some wordingproblems, no other difficulties arosefrom administering the questionnaire. Toadminister the questionnaire took onaverage 15 minutes.


WHO/MSD/MSB 00.2c

4. Write an evaluation plan

The final step in preparing for data col-lection involves the preparation of a writ-ten evaluation plan. Writing a structuredevaluation plan will help you to organizeyour thoughts clearly. It also serves as areminder to everyone involved about thepurpose of the evaluation and what ques-tions and decisions the results are in-tended to address.

The basic elements of an evaluationplan are:

1 Background and General PurposeThis is a summary of your evaluationplan. It should include a brief descrip-tion of the purpose of the evaluation,how you plan to conduct the evalua-tion, who will receive the results, andhow results will be used.

2 Programme Logic Model (refer toStep 4 of Workbook 1 for more infor-mation). The model should describe:

l what it is that you plan to evaluate (e.g.services, network of services);

l your process or implementation ob-jectives — the short-term and long-term goals of various parts of yourprogramme/service/system.

3 Evaluation TeamThis section lists the people who willbe involved with your evaluationproject, along with a brief descriptionof their experience and anticipated rolesin the evaluation.

4 Evaluation Questions to be addressedYour questions should follow logicallyfrom your programme logic model andshould be linked to one of yourprogramme’s implementation objec-tives or goals in a specific way.

5 Data Collection StrategyThis section is very important, as itexplains how you will get from yourevaluation questions to results. Re-fer to Steps 6 and 7 of Workbook 1for information and suggestions aboutcreating your data collection strategy.

For each evaluation question you listedabove, you should indicate:

l whether you will use a standardizeddata collection or develop your own,and what the specific variables youplan to use are how you will collectthe data;

l who will administer the data collectioninstrument(s);

l how many participants you will as-sess-your sampling strategy (how youwill choose participants)-your time linefor data collection.

6 Data Management Plan (refer toStep 1-B of Workbook 2 for moreinformation)In this section, explain how you willstore your data and maintain centralrecords.

7 Staff Training (refer to Step 1-B ofWorkbook 2 for more information)Describe how you will train all evalua-tion staff to conduct their jobs in a stan-dardized way, and whether additionalstaff or outside consultants will beneeded.

8 Pilot Test (refer to Step 1-C of Work-book 2 for more information)What type of pilot test will you con-duct, and how many people will par-ticipate in each portion of it? If youare using a newly developed data


WHO/MSD/MSB 00.2c

It�s your turn (1D)

1 Compose a written evaluation plan.Use the exercises that you have com-pleted so far to help you. Componentsshould include:

l Background and general purpose of theevaluation (Step 3, Workbook 1)

l Programme logic model (Step 4,Workbook 1)

l Evaluation questions to be addressed(Step 5, Workbook 1)

l Data collection strategy (Steps 6 and7, Workbook 1)

l Data management plan (Step 1B,Workbook 2)

l Staff training (Step 1B, Workbook 2)

l Pilot plan (Step 1C, Workbook 2)

l Strategy for using results

collection instrument, you should in-clude it in your pilot test and thenmake changes as necessary after thepilot test.

9 Strategy for Using ResultsExplain how your evaluation results canbe applied to the long-term goal of im-proving substance use treatment. Ex-plain who will receive results, and howthis information could be used to makechanges to improve treatment.

2 Review your written evaluation planwith the expected user(s) of the results.Modify as needed.


WHO/MSD/MSB 00.2c

Step 2

In resolving datacollectionproblems, keepthe instrumentsand proceduresconsistentunless you mustmake changesto solve fatalflaws in theproject.

(Note: This step is most applicable for re-searchers. If you have a researcher amongyour evaluation partners, have that per-son read this section and then explain itscontents briefly to the others.)

Collect data

If you have planned carefully during thepreceding stages, data collection shouldproceed smoothly and with little need forreadjustment. Nonetheless, a few guide-lines may assist you:

Periodic meetings with staff involved inthe data collection can be very helpful. Thetopics can include questions about datacollection. Write down how these ques-tions are resolved. You also may need todiscuss results of reliability and validity

checks on the indicators, and preliminaryfindings.

In resolving data collection problems,keep the instruments and proceduresconsistent unless you must make changesto solve fatal flaws in the project. Evenminor changes in forms and procedurescan cause more problems than theysolve. For this reason, once you havestarted your data collection, attempt tomake it as systematic and as free frombias as possible.

Over time, data collectors’ skills tendto erode. Periodic retraining of data col-lectors, even every few months, canhelp ensure quality control during yourstudy.


WHO/MSD/MSB 00.2c

Into action

Our evaluation team decided to meet twicea month to discuss problems with datacollection and entry. Sue will be respon-

sible for organising them. She will also takeresponsibility for recording the decisionsabout the evaluation project.

It�s your turn

The researcher in the group should makethe following decisions:

1 Write down how often you will haveresearch team meetings, and who willbe responsible for organising them.

2 Nominate a person to take responsi-bility for recording the decisionsabout research procedures. Have thisperson maintain a notebook of allthese decisions, so they can be re-ferred to later if questions arise.

Discuss your plan with your evaluationpartners. Make changes based on their in-put as appropriate.


WHO/MSD/MSB 00.2c

Step 3

Analyse data

(Note: This step describes technical infor-mation about data analysis procedures. Itis most applicable for researchers and ordata consultants. If you have a researcheramong your evaluation partners, have thatperson read this step and then explain itscontents briefly to the others.)

Data analysis is the reward for all of yourplanning and data collection efforts. If youcarefully plan and carry out data collec-tion, the analyses will help you provideanswers to questions about what effortsyour programme is making and what ef-fects those efforts are having.

Data analysis is not simple, and requires cal-culation, interpretation, and judgement. Do

not hesitate to obtain technical assistance,if available, during this phase of the evalua-tion.

A general guideline is that the methodsof analysis you choose should be tai-lored to the specific evaluation ques-tions with which you started. Thesechoices also depend on the size of thesample you collect, the quality of thedata, the resources available to you(e.g., transcribers, interpreters, comput-ers with statistical programs installed),and the effort you are willing to put intothe analysis. This section will introduceyou to a number of general concepts andprocedures in data analysis and presen-tation.

Decide whether you are addressingdescriptive or explanatory questions

Before you begin your analysis, it is veryimportant to be clear about what generaltype of analysis you wish to conduct.Descriptive analysis summarises themeasurements you have made on the vari-ables included in the data collection in-

Descriptiveanalysissummarises themeasurementsyou have madeon the variablesincluded in thedata collectioninstrument.Explanatoryanalysisattempts toexplainrelationshipsamong thevariables inwhich you areinterested.

strument. Explanatory analysis attemptsto explain relationships among the vari-ables in which you are interested. Thefigure below summarises the differencesbetween these two different approachesto data analysis.


WHO/MSD/MSB 00.2c

As the figure shows, descriptive analysessummarise the data collected for each indi-vidual variable. If your indicators are quan-titative in nature, descriptive analyses canbe used to provide information for ques-tions such as “how many clients in theprogramme are over the age of 50?”, or“how much time did staff spend complet-ing administrative forms in the assessmentprocess?”, or “how much variability wasthere in the education levels of our clients?”.If your indicators are qualitative in nature,descriptive analyses can be used to provideinformation for questions such as: “whattypes of responses were reported when cli-ents were asked about the most beneficialaspect of the programme?”

The figure also shows that explanatoryanalyses describe relationships between

or among variables included in your datacollection instrument(s). Explanatoryanalyses can also use either quantitativeor qualitative data. For example, if yourindicators are quantitative in nature, ex-planatory analyses can be used to pro-vide information for questions such as“do male and female clients differ in theirrelapse rates in our programme”, or“compared to clients who didn’t receivea motivational interview, were clients inthe interview programme more likely toshow up for group therapy sessions?”.If your indicators are qualitative in na-ture, explanatory analyses can be usedto address questions such as “did clientswho reported liking the harm reductionorientation of our programme reportmore positive benefits in our inter-view?”.

Descriptive analysis

Goal:

Explanatory analysis

to summarize the mea-surements for each vari-able

Goal: to explain relationshipsbetween variables andgroups

Examples:How many clients re-ported?How much time didstaff spend?How much variabilitywas there in ...?

Examples:Do male and female cli-ents differ?Compared to clients as-signed to a controlgroup. were clients ex-posed to ... more likelyto...?

Different types of data analysis


WHO/MSD/MSB 00.2c

1 Has the number of referrals increasedfrom the previous year?

2 Did the characteristics of the clientschange in comparison to last years cli-ents?

3 Has the number of self-discharges de-creased from the previous year?

4 Has the number of clients who were re-ferred to long-term treatment increasedfrom the previous year?

Descriptive analysis with quantitativeindicators




DescriptionQuestions

Into action

The research questions regarding the process evaluation would be analysed with thefollowing types of analyses





DescriptionQuestions

5 Did attitudes in favour of HIV low-riskpractices improve among clients?

6 Did client’s HIV-risk behaviour changein favour of low risk practices aftertreatment?

7 Did clients knowledge about AIDS in-creased after the AIDS education inter-vention?

8 Did clients self-efficacy regarding theirability to use skills maintaining AIDSharm reduction behaviours increase?

For the outcome evaluation they decide on the following analyses:


WHO/MSD/MSB 00.2c

1 Write down 4 research questions re-lated to your evaluation project.

2 Decide what type of analysis (or analy-ses) you will conduct for each ques-tion:

Descriptive analysis with quantitativeindicators (Section A)

It�s your turn

Descriptive analysis with qualitative in-dicators (Section B)

Explanatory analysis with quantitativeor qualitative indicators (Section C)

3 Review the appropriate sections below.

Recall from Workbook 1 that quantitativeindicators can be arranged into differenttypes, based on the measurement scalethat they use (nominal, ordinal, interval,ratio). Each type of scale can be analysedin different ways, and lend itself to differ-ent presentation strategies, as indicated in

A. Descriptive analysis withquantitative indicators

the following subsections. The subsectionsbelow (frequencies, measures of centraltendency, measures of variability) provideinformation on how to calculate statisticsthat summarise the variables in your evalu-ation.

If you counthow manyobservationsfall into eachcategory, youobtain astatistic knownas a frequency.

Frequencies1

Simple frequency tables are used to seepatterns in groups of nominal and ordi-nal data. A frequency distribution pro-vides a record of the number of indi-viduals located in each category of thequantitative indicator . If you counthow many observations fall into eachcategory, you obtain a statistic knownas a frequency. If you know the totalnumber of observations that were mea-sured for that variable, you can calcu-late the percentage of observations ineach category by simply dividing the

number of cases in each category by thetotal number.

A frequency table displays these results.As the next figure shows, a frequency tablelists the number of observations in eachcategory. The table also presents the totalnumber of cases and the percentages.

In the example presented on the nextpage, measurements have been taken toassess severity of PSU problems for 125hypothetical clients.

1 Note that all of themajor statisticalpackages forcomputer-aided dataanalysis have routinesthat will calculatefrequencies


WHO/MSD/MSB 00.2c

Title: PSU problem severity for aSample of 125 hypothetical clients

Labels

Totals

Category Frequency Percentage

None reported

Mild

Moderate

Severe

Very severe

42

13

50

15

5

125

33.6

10.4

40.0

12.0

4.0

100.0

In some cases, especially with small samplesizes, you might want to show the fre-quency of every measurement value. Inmost cases, however, (and especially forinterval and ratio data) you will want to

group these measurements into a smallernumber of categories called a groupedfrequency table. The next figure showsan example, this time using client age asthe variable.

Age (years) Frequency Percentage

5

5

20

30

60

160

3.1

3.1

12.5

18.8

37.5

60-69

30-39

20-29

Totals

40 25.010-19

A grouped frequency table describing age of clients

100.0

50-59

40-49


WHO/MSD/MSB 00.2c

Measure Definition Data-types Examples

Score whichoccurs most

Moderate

Average

NominalOrdinal

Mode

Median

Mean

Ordinal

Interval

GenderClient Satisfaction

Client Satisfaction

Numbers ofCounseling

Each grouping is called an interval , andthe number of scale values in each in-terval is called the interval width .There are no strict rules for decidingon the number of intervals or theirwidth. But in general, there should notbe so many groupings that you end up

with only one or two observations ineach group. On the other hand, thereshould not be so few groupings that thehighest and lowest observations are notdistinct from each other. For conve-nience, the interval width should be awhole number.

Measures of central tendency

Another useful way to summarise the mea-surements you have made for a variable isto report the average or typical measure-ment in your data for that variable. We

call this the data set’s central tendency.There are three ways to describe centraltendency: mode, median, mean. These areshown in the next figure.

Measures, definitions, data types, and example of central tendency

The mode isthe score thatoccurs mostfrequently.

The mode is the score that occurs most fre-quently. This measure of central tendencycan be used for nominal, ordinal, or inter-val scale data. Not that a data set can havemore than one mode if the maximum fre-quency occurs equally for two or more ob-servations for a variable.

The median score is the one that separatesthe upper half of the observations from thelower half of the observations. In otherwords, 50% of the scores fall above themedian, and 50% of the scores fall below

the median. To calculate a median, you mustbe able to rank order the scores from high-est to lowest. This measure of central ten-dency can be used with both ordinal andinterval scale data. One important propertyof the median is that, unlike the mean, ex-treme scores do not affect it.

The mean is the arithmetic average for theobservations on that variable. It is themeasure of central tendency that is usedmost often to describe a set of interval-scale observations. To calculate the mean

The medianscore is the onethat separatesthe upper half ofthe observationsfrom the lowerhalf of theobservations.


WHO/MSD/MSB 00.2c

for a variable, simply add up all of thescores and divide that sum by the totalnumber of scores. The next figuresummarises these measures of central ten-dency, using a hypothetical example of 11observations made on a client satisfactionvariable, which uses a 5-point intervalscale. A mean score can be influenced sig-nificantly by one or two extreme scores inthe set of observations.

The mean is the arithmetic average for theobservations on that variable. It is the mea-

sure of central tendency that is used mostoften to describe a set of interval-scaleobservations. To calculate the mean for avariable, simply add up all of the scoresand divide that sum by the total number ofscores. The next figure summarises thesemeasures of central tendency, using a hy-pothetical example of 11 observationsmade on a client satisfaction variable,which uses a 5-point interval scale. A meanscore can be influenced significantly by oneor two extreme scores in the set of obser-vations.

The mean is thearithmeticaverage for theobservations onthat variable.

Client satisfaction data(5 point interval scale)

Rank-ordereddata

5

4

4

3

2

1

5

1

1

2

3

1

24

5

4

4

3

5

1

1

2

Note: 1=hated program; 2=disliked program; 3=neutral; 4=liked program; 5=loved program

Mean

Median

Mode

=

=

=

Sum of scores

# of scores

2

1

(divides observations into upper & lower halves)

(most frequently occuring observation)

24

11= 2.18=


WHO/MSD/MSB 00.2c

A measure of central tendency is not enoughto summarise or describe a quantitative vari-able. You also need to know how widely aset of observations varies around its aver-age. Another way to think about variabilityis that measures of variability provide auseful summary of how different the ob-servations are from each other. There areseveral measures that can be calculated.

The range determines the extreme obser-vations in your data set and the differencebetween them. For example, a series ofobservations made on a 5-point client sat-isfaction interval scale might look like this:2, 1, 4, 3, 3. The range of scores here is 1-4 (lowest observation to highest observa-tion). The range is useful because it tellsus about minimum and maximum obser-vations for a variable. Unfortunately, itonly tells us about extreme scores anddoesn’t take into account all of the obser-vations at once.

The standard deviation describes how farthe observations deviate from the meanin a data set. It is the most important andmost widely used measure of variability.It is always reported in conjunction withthe mean for a variable (e.g., “mean cli-ent satisfaction in the sample was 3.2,with a standard deviation of 1.7”). Howdo we calculate a standard deviation?There are two important properties of thismeasure:

l if all of the observations are equal tothe mean, then there is no variability in theobservations;

l the more the observations differ from eachother, the greater the standard deviation forthat variable.

To calculate standard deviations, we needto introduce the concept of deviationscores. Here is the way we represent de-viation scores:

X - µ,

where any single observation (X) is sub-tracted from the overall mean of the ob-servations for that variable (µ). To calcu-late a standard deviation, we first have tosquare the summed deviation scores for allobservations:

sum (X - µ)2 = SS = sum of squares

This value is called the “sum of squares” be-cause it simply reflects the sum of the squareddeviation scores for our data.

The variance of our variable gives us an “av-erage” of the variability of our observations,and is calculated as follows:

S2 = variance = SS (sum of squares)/Nwhere N = the number of observations inour data set for that variable.

Finally, the standard deviation of ourvariable is simply the square root of thevariance:

S = standard deviation = square root ofvariance

You can see on the next page an example,again using client satisfaction data, but thistime for only 7 observations:

Measures of variability2

The rangedeterminesthe mostextremeobservationsin your dataset and thedifferencebetweenthem.

The standarddeviationdescribeshow far theobservationsdeviate fromthe mean in adata set.

2 All of the majorstatistical packagesfor computer-aideddata analysis haveroutines that willcalculate measures ofvariability for you


WHO/MSD/MSB 00.2c

X(Observations)

X - µ(Deviation Scores)

(X - µ) 2

(Squared deviation scores)

-1

-1

0

0

0

1

1

1

0

0

0

1

3

3

4

4

4

5

1 15

sum = 28 sum = 0 sum = 4 = sum of squares

mean = 28/7 = 4 = µ

Variance = SS/N = 4/7 = .57

Standard deviation = square root (.57) = .76

Remember the following steps when cal-culating standard deviations:

l Calculate the mean for the variable;

l Subtract the mean from each score; putthese values in a separate column;

l Square each deviation score; put thesevalues in a separate column;

l Calculate the variance and standarddeviation, after summing the columns.


WHO/MSD/MSB 00.2c

B. Descriptive analysis withqualitative indicators

Decide whetheryour analysis willtry to findexamples ofwhat interestsyou already.This is referredto as a top-down,deductivemethod. Youralternative willbe to constructtypes based onwhat you find.This is a bottom-up, inductivemethod.

Whenever you have qualitative (detailedobservation-based or language-based) in-dicators in your data set, instead of statis-tics, descriptive analyses provide a seriesof responses that summarise the indicatorsand describe their characteristics.

Several preliminary steps are required toconduct descriptive analyses of qualitativedata. The steps include the requirements:

l Make sure that each of the indicatorshas received an accurate transcrip-tion. In other words, you must trans-fer taped interviews to computer filescontaining the qualitative information.Further, checks should be made in or-der to ensure that whoever was del-egated to transcribe interviews accu-rately recorded the information. If youare using open-ended written descrip-tions, transfer them to computer files

so that they can be easily sorted andmanipulated.

l Make sure that you have allocated suffi-cient time and staff resources to con-duct the descriptive analyses of the quali-tative data. Descriptive analyses ofquantitative data, can be accomplishedeasily using widely-available computerprograms. However, qualitative analysesrequire people to interpret the indicator.This takes extensive time and effort. Do notundertake these analyses unless you are pre-pared to devote the resources to them.

l Decide whether your analysis will tryto find examples of what interests youalready. This is referred to as a top-down, deductive method. Your alter-native will be to construct types basedon what you find. This is a bottom-up, inductive method.

One prominent approach to descriptivequalitative analysis is to use the tran-scripts to find examples of types thatalready interest you. Consider, again,the variable used earlier in this workbookto illustrate qualitative data: “of the pro-gramme most liked.” You might approachthis qualitative analysis in a top-downmanner if you have preconceived ideasabout the types of responses that mightbe given (e.g., if you think, in advance,that clients will report that three aspectsof the programme are most liked: (1)harm reduction philosophy, (2) staff (suchas case therapists), and (3) follow-up sup-port groups. If you adopted this ap-proach, the following steps could be fol-lowed:

l assign a reader who reviews and inter-prets each of the observations

l instruct the reader to read through thetranscripts three times. The first read-ing will identify all responses that men-tion the harm reduction philosophy. Thesecond reading will identify all responsesthat mention staff, and the third readingwill identify all responses that mentionthe follow-up support groups

l After each reading, the interpreter collectsthe responses for each type and collatesthem into one document. The result willbe three documents: one describing allof the interview responses for each typeof interest to the programme evaluator(harm reduction, staff, follow-up groups)

l A second reader will duplicate the firstreader’s interpretations, in order to de-termine whether instances of the threetypes were missing

Top-down (deductive) analysis


WHO/MSD/MSB 00.2c

Another approach to describing qualita-tive data is used when you don’t have apreconceived idea of the types of responsesthat are given when the indicator is used.This approach attempts to describe the

Bottom-up (inductive) analysis

variety of responses that are given. Yourgoal is to develop the types from the infor-mation you have, rather than confirm ordisconfirm their existence. The figure onthe next page provides an example:

This figure presents a rudimentaryanalysis of three clients’ responses tothe open-ended qualitative question.In the bottom-up approach, the readeror interpreter attempts to develop arecord of the types expressed in theobservations. In this example, twomain types were reported: grouptherapy (by clients 43 & 89) and forms(by client 112). Note that within thegroup therapy type, the interpreteridentified three different types. Thesesubtypes reflect different aspects ofgroup therapy that were viewed posi-tively; the fact that it gave good in-formation (client 43), the flexibility ofgoals (client 89), and support fromother clients (client 43). As you cansee, the inductive approach yields newtypes of responses, as opposed tof inding examples of pre-exist ingtypes.

In general, it is better to use both induc-tive and deductive approaches to controlfor one another. The following guidelinesshould be kept in mind:

l The interpreter must systematically readall of the transcripts, not just interest-ing ones or ones that show the data ina particular light.

l The interpreter must systematically docu-ment the analysis by recording wherethe examples can be found, and whataspects of the transcript support the in-terpretation.

l The interpreter’s work should bechecked by another reader, whodoesn’t know the results of the firstanalysis. This will allow you to deter-mine whether there are any biases inthe interpretative process.

Indicator

Observations

Types

What is the aspect of the programme you likedbest?

�For me, it has to be the amount of good information I gotand the support from other members of the group� (Client#43)

I liked the fact that I could set my own goals regardingdrinking and work in group therapy to achieve them.� (Client#89)

�I didn�t have to fill out too many forms� (Client #112)group therapy [43, 89] + group therapy (information) [43]

+ group therapy (own goals) [89]+ group therapy (other clients) [43]

forms [112]

(Hypothetical Data)


WHO/MSD/MSB 00.2c

C. Explanatory analysis withquantitative or qualitative indicatorsDescriptive analyses cannot fully answerall types of questions, such as: Do clientsabusing alcohol and cocaine differ in thenumber of times they have been arrestedfor PS-related problems; or compared toclients assigned to a comparison group,were clients who received a motivationalinterview more likely to attend grouptherapy sessions? To make these determi-nations, you need to go beyond simplydescribing the frequencies, central ten-

dency, and variability of the variables youcollected in your programme evaluation.Infer ential statistics go further to exam-ine relationships among variables in yourevaluation. The following subsections out-line two of the basic inferential techniquesthat are available for you. However, thereare many more techniques that can be used,and you should seek technical help, if avail-able, to conduct more advanced types ofexplanatory analysis.

The X2 test is based on the idea of find-ing the difference between the observedfrequencies, displayed in the figure, and

the frequencies that you’d expect, if thetwo variables are independent of eachother.

Chi-square (X2) test

The 2 test is a technique that can be usedto test the strength of the relationshipbetween two variables. This test is usedwith nominal data (e.g., gender, types ofprogrammes), but also can be used withordinal or grouped interval data. For ex-ample, you might want to compare theproportion of clients who relapsed fol-lowing motivational interviewing with theproportion of clients who relapsed in yourcomparison group who received no mo-tivational interviewing.

To begin, cross-classify each observa-tion across the two variables: e.g., mo-

tivational interviewing versus no moti-vational interviewing, and relapse ver-sus no relapse. Then, count the num-ber of observations that fall into eachof the categories created. For example,maybe you have a total of 200 obser-vations (clients): 100 received motiva-tional interviewing and 100 did not.Within the “motivation group,” 68 didnot relapse, while 32 did relapse. Withinthe no-intervention group, perhaps 45did not relapse while 55 did relapse.You can arrange this information in acontingency table, as shown in the fig-ure below.

Motivationalinterview

No interview

ColumnTotals

X2 Test of association between two variables

68 32 100

68 32 100

Observed frequencies

Didn�t relapse Relapsed (Row totals)

45 55 100


WHO/MSD/MSB 00.2c

Here is how to calculate the expected values of the frequencies for each cell:

Now that we have calculated the ex-pected values, we will test whether ornot these expected frequencies differfrom the actual, observed frequencies inour data.

The chi-square (X2) statistic providesa way for us to compare the actualand expected frequencies, and is cal-culated as:

X2 = sum (of - fe)2/f

e, where

of = observed frequencies,

and fe = expected frequencies

If there is very little difference between ourobserved and expected frequencies, X2

will be a small value. If there is a greatdifference between our observed and ex-pected frequencies chi-square will be alarge value.

Hence, our observed value of X2 = 10.76.Our question is now: is this observed valueof chi-square sufficiently large to reject thehypothesis that motivational interviewingand relapse are independent? If there isno association between the two variables,what is the probability of obtaining an X2

value this large just by chance? Remem-ber, motivational interviewing may not beassociated with decreased relapse rates. Ifthis is so, you may have happened by

chance to test a sample of clients for whomthis is true.

To answer these questions, you must con-sult a table of critical values for the X2

statistic. This table reports the likelihoodof obtaining various X2 values by chancealone. Any statistical textbook (see refer-ences) will provide a table of critical val-ues. To use the table, you need to knowthe appropriate “degrees of freedom” and

Category of fe (of - fe) (of - fe)2 (of - fe)2/fe

Interview/no relapse 45 56.5 -11.5 132.25 2.34

Interview/relapse 55 43.5 11.5 132.25 3.04

No interview/no 68 56.5 -11.5 132.25 2.34

No interview/relapse 32 43.5 11.5 132.25 3.04

sum (of - fe)2/fe = 10.76

fe (expected value for the cell) = (row total) X (column total)/N

Therefore,f

e(interview + no relapse cell) = (100)(113)/200 = 56.5

fe (interview + relapse cell) = (100)(87)/200 = 43.5

fe (no interview + no relapse cell) = (100)(113)/200 = 56.5

fe (no interview + relapse cell) = (100)(87)/200 = 43.5


WHO/MSD/MSB 00.2c

“level of significance”. Most of the time,a level of significance of .05 is set, whichdeclares that the difference between thegroups is probably real if the likelihood ofobtaining an X2 that large by chance isless than 5 percent. With two variables, aone degree of freedom test is indicated.At the .05 level of significance with onedegree of freedom, the critical value of

Independent samples t-test

The t-test is used to test whether twoindependent* groups have truly differentmean scores on some variable of inter-est. For example, you might want toknow whether methadone clients whoreceive take home privileges use heroinfewer days per month than clients whodo not have these privileges.

The t-test examines the difference be-tween the two group means on a vari-able. It takes into account the variabil-ity of the scores and the number ofobservations in each group. A test sta-tistic is computed (>t’) that showswhether the difference in the meansacross the groups reflects a true differ-ence or random fluctuations. As in thechi-square statistic, a large t value indi-cates that the groups are quite differentfrom each other (e.g., if you repeatedthe evaluation again, you’d find the samedifference between the two groups). Asmall t value indicates that the differencebetween the means is likely due tochance.

* The values of the observations in one group areindependent of the values of the observations inthe second group. For example, we may be com-paring a group of clients treated with behaviouraltherapy vs. drug therapy. These two groups areclearly different from each other. There are differ-

X2 is 3.84. That is, any observed X2 valuegreater than 3.84 would be considered sta-tistically significant, while any observedvalue below 3.84 would be consideredchance. In the present example, since theobserved value of X2 is 10.76, we con-clude that there is a statistically significantrelationship between motivational inter-viewing and relapse rates.

A conceptual formula for t is as follows:

t = (mean for group 1 - mean for group2)/(standard error)

or …

As you can see, if the numerator of thisequation is zero, the overall value of t willbe zero, indicating that there are no dif-ferences between the groups. But if thedifference is large, relative to the variabil-ity within the groups (the standard error),then t will also be large, suggesting thatthere is a true difference between the groupmeans.

The following example presents data onthe number of abstinent days reportedduring a 20 day methadone treatmentprogramme, for two different client groups(there were 9 clients in each group):

ent types of t-tests for situations where the groupsare not independent (e.g., when making compari-sons for the same group of clients at two differentpoints in time). Consult a statistical text and/or aresearch professional to decide whar types of testare appropriate for your evaluation question(s).


WHO/MSD/MSB 00.2c

Group 1 Group 2

(clients without privileges) (clients with privileges)

13 11

13 9

10 12

14 10

14 14

10 7

12 9

12 10

10 8

—- —-

Means: 12 10

Standarddeviations: 1.66 2.12

The question becomes: Are the mean num-ber of abstinent days reported in the twogroups significantly different from eachother? (Or, are clients with privileges ac-tually more abstinent than clients withoutprivileges?).

This formula can be expanded as follows:

= (12 - 10)/[(8 X (2.76) + 8 X (4.49))/16]X [1/9 + 1/9] = 2.23

Our observed value of t = 2.23. Ourquestion is now: is this observed t valuesufficiently large to reject the hypoth-esis that there is no difference in thenumber of abstinent days between thetwo groups? If there is no differencebetween the two groups, what is theprobability of obtaining a t value thislarge just by chance?

To answer these questions, you must againconsult a table of critical values for the tstatistic, which reports the likelihood ofobtaining various t values by chance alone.Any statistical textbook (see references)will provide a table of critical values. Asin the case of chi-square, you need toknow the appropriate degrees of freedomand level of significance. Most of thetime, a level of significance of .05 is used,which declares that the difference betweenthe groups is probably real if the likeli-hood of obtaining a t that large by chanceis less than 5 percent. In the present ex-ample, the degrees of freedom are 16 (seea statistical text to determine degrees offreedom for a t-test). At the .05 level ofsignificance with 16 degrees of freedom,the critical value of t is 2.12. That is, anyobserved t value greater than 2.12 wouldbe considered statistically significant,while any observed value below 2.12would be considered chance. In thepresent example, since the observed valueof t is 2.23, we conclude that there is astatistically significant difference in thenumber of abstinent days reported by thetwo client groups.


WHO/MSD/MSB 00.2c

Into action

After describing the type of analysis foreach research question, Adam S. gave abrief overview about the most appropri-ate data analysis for each evaluation (pro-cess and outcome evaluation).

Process evaluation

For research question investigating theservice coverage at the activity level num-ber of clients would be calculated and com-pared to last years figure.

Outcome evaluation

Data analyses procedures for the outcomeevaluation were proposed as follows:

In order to compare pre- and post-inter-vention knowledge about AIDS, degreeof engaging in high risk behaviours andattitudes towards AIDS a t-test for de-pendent samples would be carried out,because variables taken into accountwere all measured at an interval scale.The level of significance of .05 wouldbe used, which declared an observed dif-ference between pre- and post-interven-tion could have arisen by chance in 5%of cases.

It�s your turn (3C)

1. The partner who read this section shouldbriefly describe the most appropriate dataanalysis procedures to his/her evaluation

partners. Remember, this is highly techni-cal information, so a brief overview is allthat is needed for your partners.


WHO/MSD/MSB 00.2c

Step 4

Report resultsYour evaluation is not complete until theresults are communicated to the user(s)of the research. You can present the re-sults in writing, with pictures or graphs,or in oral presentations.

If your results are favourable to your cur-rent programme or treatment network,then you should plan carefully for how you

will present them, in order to maximisetheir impact. On the other hand, if yourresults indicate that your programme is notperforming as expected or desired, thenyou have an opportunity and a challenge.You can decide what changes are neededin your programme, then present the re-sults with the proposed changes at thesame time.

A detailed, written report of the evalua-tion is a good place to start. This reportcan be distributed within your evaluationteam, and/or to the users of the results.Components should include:

Title page

The title page should include the nameof the report, the name and address ofthe programme, the time period of theevaluation, and the date the report wascompleted.

Summary

The summary is very important becausemany people do not have time to read thefull report and may read this only. Sum-maries are usually 1-2 pages long, and

contain the purpose of the evaluation, avery brief description of the programme,how the evaluation was conducted, impor-tant results and conclusions, and recom-mendations.

Table of contents

The table of contents will help readers tofind information quickly.

Description of programme

This section should briefly describe theprogramme that was evaluated.

Evaluation questions

Start by presenting your evaluation ques-tions to the reader.

Written report

(Note: This sectionshould be reviewed byall evaluationpartners.)


WHO/MSD/MSB 00.2c

Description of evaluation

This section explains the methods usedfor the evaluation. It should describe whoconducted the evaluation, and where, andwhat measures were used to collect data.

Results

This section presents the results of the dataanalysis.

Conclusions andrecommendations

This section is very important, because itis where you can make recommendationsabout what to do in response to the evalu-ation findings.

Oral reportIf you are presenting information orally, itis important to remember that the averagelistener will retain only 3 or 4 main mes-sages from your presentation. Be selec-tive. Use visual aids to help orient listen-ers and keep their attention. Keep visualaids simple and to the point.

Think about highlighting certain resultsfor different audiences. For example, ad-ministrators may want to know about ef-ficiency or cost issues, while cliniciansmay be interested in hearing about theviews of patients. Some audiences maybe familiar with your programme, whileothers will not be familiar, and will need

background information about yourorganisation. In each case, it is impor-tant to tailor your presentation to youraudience.

Consider giving oral presentations (andmaking written presentations available) inthe following settings:

l A programme staff meeting

l A board meeting

l A meeting for funder(s)

l A meeting for patients/clients

Visual aids

Bar graphs and pie charts are useful forsummarising data in written and oral reports.A good bar graph has the following features:

l Both the categories and the frequencylabels are clearly indicated.

l The length of each bar in the graphshows the frequency for that particularcategory

l The corresponding percentage of ob-servations are presented at the top ofeach bar in the graph.

l All of the bars are of the same width.

l Adjacent bars are separated by a space.


WHO/MSD/MSB 00.2c

These same data can be displayed in a pie chart:

Grouped frequency data can be presentedin two ways: histograms and line graphs.Histograms are like ordinary bar graphs,except that they (a) use interval-scale data,and (b) the data are grouped in intervals.

An alternative to the histogram is a linegraph. Instead of drawing a bar for eachinterval, put a point above the centre ofeach interval marker and connect the ad-jacent points by straight lines.

Preparation40%

Contemplation33.6%

Contemplation10.4%

Maintenance4%

Action12%


WHO/MSD/MSB 00.2c

It�s your turn (4)

1 Among your partners, nominate a per-son to take responsibility for preparingthe written report. Use the format de-scribed above to create it.

2 Nominate another person to take re-sponsibility for preparing oral report(s).Remember to tailor your reports to theintended audience. If you have mul-tiple audiences, you may want to as-sign different people to prepare differ-ent reports:

• Funder(s)

• Expected users of the research

• Clinicians

• Patients


WHO/MSD/MSB 00.2c

Step 5

Make use of whatwas learnedAfter months or even years of time in-vested in conducting your evaluationproject, you want to ensure that your re-sults are put to good use. One way toaccomplish this is to report your resultsin written or oral form (Step 4). It isequally important, however, to explorewhat the results mean for yourprogramme and its service to patients/clients. Discuss your findings with ex-pected user(s), community members,funders, and other important groups. Gettheir point of view on what should lieahead for your programme. Do changesneed to happen? If so, what is the bestway to accomplish these new goals?

Before making changes in yourprogramme, it is important to consider thelimitations of your evaluation. If you havefollowed the steps outlined in this work-book series, you can feel reasonably con-fident in your results. Nonetheless, it maybe prudent to attempt to replicate thestudy’s results before making large-scalechanges.

Following discussions, return to the ex-pected user(s) of the research with spe-cific recommendations based on your re-sults. You should have done some of thiswork when preparing the written report,but now it is even more important for youto be clear and precise. List your recom-

mendations, link them logically to yourresults and mutual objectives, and sug-gest a period for implementation ofchanges. The examples below illustratethis technique.

Based on the finding that clients have towait an average of 3,5 months before re-ceiving an appointment, we recommendthat the programme re-examine its assess-ment processes, with the goal of short-ening assessments and/or moving to agroup assessment format. The reviewcould happen in January, and changes tothe assessment process could happen bymid-February.

The results indicate that treatment B issuperior to treatment A in terms of reduc-ing PSU among patients. Therefore, werecommend that treatment B is offered toall clients, and that treatment A is reservedfor those who do not show improvementwith treatment B. A follow-up replicationstudy, starting in April with a new sampleof patients, is also recommended to con-firm results.

In addition, consider taking your researchresults to the public. This can be accom-plished through the local media, or throughcommunity groups. If approaching themedia, prepare a one page summary state-ment of your study and its main findings.



WHO/MSD/MSB 00.2c

If presenting at a community meeting,highlight the main findings and their im-plications using an oral presentation. Ei-ther way, be sure to explain yourprogramme, the research, and the resultsusing simple, non-technical language.Don’t forget to make recommendations,if appropriate. For example:

Our findings indicate that most peoplewith PSU problems do better in treat-ment when supported by a communitymember or family member. A trialprogramme that included family membersin certain parts of treatment showedfavourable results when compared tostandard, individual PSU treatment.

Into action


The following key recommendationswere formulated by the evaluation team.

1 Based on our findings that the numberof female drug user who enter treat-ment increased, the service needs to re-examine if the service meets the needof female clients.

2 The results indicate that less clients werereferred to long term treatment in com-parison to the previous year. It is recom-mended that the service re-examines theirreferral process and analyse possible bar-riers for clients to seek further treatmentand how the service could provide helpto overcome these barriers.

3 The findings show that the number ofself-discharges decreased in compari-son to the previous year. This repre-sents a confirmation that the serviceprovides good withdrawal manage-ment. accountability requirements, safewithdrawal from opiates are met.

4 The results support the sufficiencyof the AIDS education intervention.It is therefore recommended to con-tinue the intervention and thus pre-vent further introduction and spreadof HIV among intravenous opiateusers and to prevent HIV transmis-sion to their sexual partners. Arandomised control trial, startingnext September is recommended toconfirm results.

1 Working individually, think about yourrecommendations, based on your treat-ment research results. Write down therecommendations you want to make foreach target audience (e.g., funders, cli-nicians, patients).

2 Discuss your ideas with your evalu-ation partners. Arrive at 1-3 keyrecommendations for each targetaudience.


WHO/MSD/MSB 00.2c

Step 6

Start again

In the introduction to the FrameworkWorkbook, you learned about the impor-tance of creating a “healthy culture forevaluation” within your programme.You learned that a healthy culture forevaluation is one in which feedback loopsare woven into the fabric of the treatmentservice or system. Some feedback loopsserve the purpose of providing basic ac-countability data to programme or sys-tem funders, clients and the general pub-lic. Other feedback loops are betterviewed as the means by which aprogramme or treatment system seeks tocontinuously improve their services andoutcomes. The steps of planning andimplementing evaluations provide youwith the tools to make this happen in yourown setting.

Remember,evaluation mustbe an attitudeof continuallyquestioningand gaininginformation.

This final step, Step 6, serves as a reminderthat feedback also should be directed to-ward new research and ongoing evalua-tion of your programme. Remember,evaluation must be an attitude of continu-ally questioning and gaining information.Accordingly, you may want to considerreplicating or extending this evaluation, orplanning a new evaluation to answer a dif-ferent question. Results from your currentstudy will help guide you to your nextevaluation activities. The steps outlined inWorkbooks 1 and 2 can help guide futureresearch efforts. Most important, now thatyou have established a culture for evalua-tion within your setting, you should takethe necessary action to ensure that thisculture remains healthy and strong for thefuture.



WHO/MSD/MSB 00.2c

Into action

Based on their research experience andtheir results the evaluation team had thefollowing ideas for future evaluationprojects:

1 to conduct a randomised control trialin order to confirm the effectiveness ofthe AIDS education intervention.

2 to plan and implement a process evalu-ation regarding coverage at the activ-ity level in order strengthen the cul-ture for evaluation within the service.

3 to plan an outcome evaluation regard-ing illicit substance use, retention,transfer to further treatment.

1 Based on your overall research ex-perience and results, what can youdo next? Working individually, writedown one new idea for your nextevaluation research project. Discussthese ideas with your evaluationpartners and make a plan for how toproceed.


Documents

Workbook 2 - Request Rejectedwhqlibdoc.who.int/hq/2000/WHO_MSD_MSB_00.2c.pdf · Workbook 2 • Implementing Evaluations 5 WHO/MSD/MSB 00.2c Table of contents Overview of workbook