43 (2005) 99–115

Practice Guidelines in school psychology: Issues and

directions for evidence-based interventions in

practice and training

Jennifer L. WhiteT, Thomas R. Kratochwill

University of Wisconsin–Madison, United States

Received 2 April 2004; received in revised form 25 January 2005; accepted 31 January 2005

Abstract

Practice Guidelines have become increasingly popular at the national and international level.

Practice Guidelines are a natural extension of the bevidence-based interventionQ movement, and

could be a mechanism to promote the use of evidence-based interventions within the field of

school psychology practice and training. In this paper we review the use of Practice Guidelines

in other fields and the promise and potential pitfalls associated with the development and

application of Practice Guidelines within the field of school psychology. Recommendations for

how to avert some of the difficulties faced by other disciplines in the development and

application of Practice Guidelines are discussed. Suggestions for next steps in the profession and

future research are noted.

D 2005 Society for the Study of School Psychology. Published by Elsevier Ltd. All rights reserved.

Keywords: Practice Guidelines; Evidence-based intervention; School Psychology

The development and implementation of evidence-based interventions (EBIs) is

well underway in psychology and education. This initiative towards an empirical

basis for the use of interventions is an extension of the scientist–practitioner

0022-4405/$ -

All rights rese

doi:10.1016/j.

T Correspon

Johnson St., U

E-mail add

Journal of School Psychology

see front matter D 2005 Society for the Study of School Psychology. Published by Elsevier Ltd.

rved.

jsp.2005.01.001

ding author. Jennifer L. White is to be contacted at Educational Science Bldg, WCER 1025 West

niversity of Wisconsin–Madison, Madison 53706, United States. Tel.: 608 212 7304.

ress: JLWhite2@wisc.edu (J.L. White).

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115100

model1 (Kratochwill & Shernoff, 2003; Kratochwill & Stoiber, 2000, 2002; Stoiber &

Kratochwill, 2000, 2002). The development of Practice Guidelines is a natural extension

of the international movement towards bevidence-basedQ practice (Rowland & Gross,

2000) and has been recommended as a bnext stepQ in the use of EBIs in practice (see

Kratochwill & Shernoff, 2003; Kratochwill & Stoiber, 2002). Practice Guidelines have a

long history of application in medicine and psychiatry but, have received less attention in

the field of psychology and education. The application of Practice Guidelines is an

important innovation in that it has implications for how EBIs are implemented in schools

and other applied settings. The purpose of this paper is to review the development of

Practice Guidelines among related disciplines of professional practice, with a special

emphasis on the issues surrounding the development and application of Practice

Guidelines for the profession of school psychology.

Clarification of terminology

Within the EBI initiative, a number of different terms have been used to refer to

overlapping constructs and phenomena. In this section we briefly outline the meaning of

relevant terms to assist the reader in understanding the context for references to Practice

Guidelines in this paper and in the professional literature.

Empirically validated treatment/intervention (EVT)

EVT was an early term used by the Task Force on Promotion and Dissemination of

Psychological Procedures (1995) to refer to a treatment validated by experimental research

(see Chambless & Ollendick, 2001). The term has fallen out of favor because it is

increasingly recognized that a treatment is never completely validated. Therefore, the term

evidence-based or scientifically supported in research has been used more recently in

psychology and education (see below).

Evidence-based treatment/intervention (EBI)

The term EBI is current and used in both the clinical and school psychology literatures

to refer to an intervention that meets criteria of a task force for support on a wide range of

methodological and statistical features (e.g., Kratochwill & Stoiber, 2002; Weisz &

Hawley, 2002). Typically, experimental (group and single-participant) methodologies are

the corner-stone of this approach to designation of an intervention as evidence-based. The

1 However, not all have embraced the Boulder Model, and some have argued that the time has come for

psychologists to babandon the monolithic scientist–practitioner modelQ and instead focus attention on training

psychologists to function in its three discrete sub-components: (a) The clinical scientist, responsible for generating

exploratory research on efficacy, (b) The evaluative scientist, conducting practical research on the development

and implementation of issues, and (c) the empirical clinician, who consumes and applies research (Milne &

Paxton, 1998). This breakdown of roles is based on the assumption that all psychologists will function as

empirical clinicians, a large proportion of psychologists will participate at least occasionally as evaluative

scientists, but only a small number of psychologists would be expected to participate in exploratory research.

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115 101

major feature of this designation is that a program or procedure has experimental research

support.

Evidence-based practice (EBP)

To understand Practice Guidelines, it is useful to understand current definitions offered

about the term bevidence-based practice.Q Generally, EBP is referred to as integration of

the best research with clinical expertise and preferences of the client for treatment (see

Institute of Medicine, 2001; Sackett, Straus, Richardson, Rosenberg, & Haynes, 2000).

Hoagwood and Johnson (2003) define EBP as:

ba body of scientific knowledge, defined usually by reference to research methods or

designs, about a range of service practices (e.g., referral, assessment, case

management, therapies, or support services). The knowledge base is usually

generated through application of particular inclusions criteria (e.g., type of design,

types of outcome assessments) and it generally describes the impact of particular

service practices on child, adolescent, or family outcomes. dEvidence-based practice’or EBP is a shorthand term denoting the quality, robustness, or validity of scientific

evidence as it is brought to bear on these issues.Q (p. 5)

Cournoyer and Powers (2002) offer the following definition:

bEvidence-based practice. . . dictates that professional judgments and behavior

should be guided by two distinct but interdependent principles. First, whenever

possible, practice should be grounded on prior findings that demonstrate empirically

that certain actions performed with a particular type of client or client system are

likely to produce predictable, beneficial, and effective results. . . Secondly, everyclient system, over time, should be individually evaluated to determine the extent to

which the predicted results have been attained as a direct consequence of the

practitioner’s actions.Q (p. 15)

Generally, EBP designates the application of a psychological intervention that has

previously been documented to have empirical support and be designated as an EBI. We

embrace the proposal that the terms bpsychological treatmentsQ or interventions is the

language of choice for these procedures (see Barlow, 2004) but would also include

academic and social/emotional treatments in this category. An extension of the concept is

also that EBP involves an evaluation of an intervention in a practice context to determine if

the intervention is effective in its application.

Methods to document intervention decision-making parameters

Documenting intervention decision-making parameters can be an important first step in

promoting the use of EBIs among practitioners. Traditionally, the field of school

psychology has primarily relied on four sources to support claims that a given intervention

or strategy is bevidence-basedQ: (a) basic intervention research literature published in

professional journals, (b) consensus or expert panel recommendations in which identified

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115102

experts in a given profession review and reach agreement on the nature of what is the best

practice in treatment; (c) reviews of single interventions or programs undertaken by

professional groups (e.g., Task Force on Evidence-Based Interventions in School

Psychology) or government funded agencies (e.g., What Works Clearinghouse), and, (d)

literature reviews and synthesis documents such as the Best Practices in School

Psychology Series (Thomas & Grimes, 2002), with topical chapters by single or multiple

authors. Given the increasingly diverse and broad literature base relevant to the field of

school psychology, the primary reliance on these four mechanisms for information

dissemination may become unsustainable over time.

Indeed, few school-based practitioners have sufficient time available to keep up with

the burgeoning literature relevant to the practice of school psychology, yet synthesis

documents and book publications are often difficult to update and correct in a timely

fashion. Although expert panel and independent intervention reviews may provide more

up-to-date and user friendly ratings of various interventions, the utilization of different

intervention rating criteria and failure to integrate multiple intervention options in a

comprehensive way can create a system of fragmented (sometimes contradictory) clinical

and applied recommendations.

A variety of alternate strategies to document and disseminate intervention decision-

making parameters have been developed by other health care professions to promote the

use of EBIs within their respective fields. The main purpose of these strategies and

techniques has been to encourage the use of EBIs by synthesizing an intervention literature

into a useable form. In the following section, we review the most commonly used

strategies to document intervention decision-making parameters, and highlight some of the

most commonly cited pros and cons associated with their use.

Manualized interventions

As EBIs become more widely used, the development and use of intervention

manuals has increased greatly. Today, most EBIs are accompanied by an intervention

manual that provides specific guidelines for how to implement and monitor the

effectiveness of the intervention in question. As an illustration, in the area of childhood

anxiety disorders, several manualized treatment protocols have been developed to guide

the administration of individual cognitive-behavioral therapies (e.g., Giebenhain &

O’Dell, 1984; Kendall, Kane, Howard, & Siqueland, 1992; March & Mulle, 1996),

family-based therapies (e.g., Howard & Kendall, 1996), and group therapies (e.g.,

Albano, Marten, & Holt, 1991; Flannery-Schroeder & Kendall, 1996). Intervention

manuals such as these typically spell out in great detail the materials needed, steps in

implementation of the intervention, as well as special considerations in implementation

of the intervention (see Kratochwill & Stoiber, 2000 for a brief review of some issues

surrounding manualized interventions).

Kendall and Flannery-Schroeder (1995) cite a number of potential advantages

associated with intervention manualization including enhanced internal validity, treatment

integrity, and the ability for intervention providers to compare differences in the number of

sessions, training requirements, and time commitments of various interventions prior to

intervention selection. The use of intervention manuals has also been found to help

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115 103

facilitate practitioner training and the replication of intervention outcome studies (Dobson

& Shaw, 1988).

Although intervention manuals have been criticized on the basis of their potential to

limit practitioner creativity (Davidson & Lazarus, 1995) or restrict practitioners’ ability to

adapt interventions to fit specific clinical circumstances (Wilson, 1995, 1996), it is

important to note that intervention manuals are typically intended to facilitate the

implementation of a single intervention or treatment program. As such, intervention

manuals do not necessarily provide practitioners with a bbig pictureQ overview of

clinically-relevant variables that should be accounted for during intervention selection or

what interventions practitioners should consider if no response to intervention was

observed. To be considered a useful tool in guiding practitioners’ intervention decision-

making parameters, this information is vitally important—but often missing from—

manualized interventions.

Treatment Guidelines and Algorithms

Treatment Guidelines2 refer to specific procedural protocols and related materials that

assist therapists, mediators, or other professionals in implementation of a particular EBI

(Hatcher, Butler, & Oakley-Browne, 2005). Treatment guidelines are often accompanied

by Treatment Algorithms which provide a step-by-step protocol for making clinical

decisions. In this respect, Treatment Algorithms can be thought of as a means of

operationalizing Treatment Guidelines by providing a suggested course of action to treat a

particular problem under various clinically-relevant circumstances.

Treatment Guidelines and Algorithms have been used extensively in the field of

medicine to guide clinical decision-making in the treatment of cancer, asthma, arthritis,

diabetes and a variety of other chronic medical conditions where first and second-line

treatments are clearly defined. More recently, Treatment Algorithms have been

developed within the field of psychiatry in an effort to codify recommended treatment

sequences for the use of medication in treating mental illnesses such as Depression

(Crismon et al., 1999), Bipolar Disorder (Dantzler & Osser, 1999), Schizophrenia

(Buscema, Abbasi, Barry, & Lauve, 2000), Panic Disorder (Coplan & Gorman, 1999),

Social Anxiety Disorder (Marshall & Schneier, 1996), Obsessive–Compulsive Disorder

(Greist & Jefferson, 1998), Posttraumatic Stress Disorder (Alarcon, Glover, Boyer, &

Balon, 2000), Personality Disorders (Soloff, 1998), and in the treatment of behavioral

disorders among adults with Developmental Disabilities (Mikkelsen & McKenna,

1999).

Treatment Algorithms for the use of psychopharmacological interventions with children

and adolescents have been developed as well. Specific examples include Algorithms for

the treatment of childhood Depression (Hughes et al., 1999), Attention-Deficit Hyper-

activity Disorder (Pliszka et al., 2000), and problem behaviors among children with

Autism (Stigler, Posey, McDougle, & Whitcomb, 2003).

2 In the present paper we use the term bTreatment GuidelineQ even though the term bIntervention GuidelineQ ispreferred. The term bTreatment GuidelineQ is consistent with a large body of writing in medicine and clinical

psychology.

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115104

Although the clarity of Treatment Guidelines and Algorithms’ step-by-step approach to

clinical decision-making is attractive to some, recommending a single bif-thenQ course of

action to guide intervention decision-making may oversimplify the multitude of

considerations influence the intervention selection process. Indeed, many stand alone

Treatment Algorithms assume a common knowledge base, level of training, philosophical

orientation, and professional context (e.g., office practice) that may be inappropriate for

the field of school psychology at this time.

Expert Consensus Guidelines

Expert Consensus Guidelines are intervention recommendations based on the results of

surveying a broad range of experts on a given topic, form of treatment, or disorder area

(Atezaz-Saeed, 2004). Expert Consensus Guidelines are typically commissioned by

professional organizations, and often focus on developing best practice recommendations

for topics or intervention issues not well covered (or clear) from the research literature

alone. For example, some Expert Consensus Guidelines have focused on recommended

practices in the treatment of behavioral emergencies (Allen, Currier, Hughes, Reyes-

Harde, & Docherty, 2001), psychiatric and behavioral problems among individuals with

Mental Retardation (Rush & Frances, 2000), and the treatment of depression in women

(Altshuler et al., 2001). A variety of other Expert Consensus Guidelines has been

developed to guide the treatment of specific psychiatric disorders, and most can be

downloaded from the Internet at no cost (http://www.psychguides.com).

The methodology used to develop Expert Consensus Guidelines was originally

designed by the Rand Corporation to aid decision-making in the defense industry

(Linstone & Turnoff, 1975). Originally called the bDelphi methodQ it involves a systematic

process for engaging a panel of experts in building scientifically-based consensus around

important and possibly controversial questions. A modified version of this method has

been used by the American Psychological Association to develop their Expert Consensus

Guidelines (American Psychological Association, APA, 2001) for the treatment of various

psychological disorders.

Advocates of Expert Consensus Guidelines have argued that this form of intervention

decision-making parameter can play an important role in guiding practice when existing

research is insufficient to support the development of formal treatment or Practice

Guidelines. Indeed, advocates of this approach argue that the current intervention research

literature often fails to address critical issues practitioners face when making intervention

decisions (Frances, Kahn, Carpernter, Frances, & Docherty, C. 1998; Frances, Kahn,

Carpernter, Frances, & Docherty, J. 1998). For example, although randomized-clinical

trials are considered the bgold standardQ for most EBI research, generalizing from clinical

trial research to practice may be difficult because these studies often include homogeneous

samples free from common co-morbid disorders, or may not include diverse representation

among clinically-relevant demographic groups. It is in these bgrey areasQ of practice whereExpert Consensus Guideline recommendations can be most useful in bridging the gap

between research and practice (Atezaz-Saeed, 2004).

Although a very practical method for documenting expert consensus on a diverse array

of topics within a given field or intervention area, Expert Consensus Guidelines suffer

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115 105

from a number of limitations that should be noted. First, Expert Consensus Guideline

recommendations are not necessarily based on empirical data or systematic literature

review. Therefore, the quality of recommendations produced is intimately tied to the

quality of knowledge and experience of expert panel members. Second, since these

guidelines are based on a synthesis of opinions derived from a large group of experts,

some individual experts may differ with the consensus view. How these dissenting

viewpoints are handled is a matter of methodology, and may vary from guideline to

guideline. Third, sponsorship and selection of expert panel members is far from an exact

science. Few guidelines exist for how expert panel members should be selected, and what

constitutes a representative expert panel. Fourth, because of the nature of the method used

to develop Expert Consensus Guidelines, some important topics and intervention issues

may be omitted from the expert panel query (e.g., issues of acceptability or diversity). And

finally, as the developers of the Expert Consensus Guidelines astutely note, because these

guidelines are generated to provide direction on issues and questions not well answered by

the existing scientific literature base bexpert opinion at any given time can be very wrongQ(http://www.psychguides.com/methodology.htm. Accessed November 1, 2004).

Although Expert Consensus Guidelines can be an important first step in providing

practitioners with guidance on providing interventions to special populations or under

circumstances in which empirical evidence is limited or unclear, these guidelines can

suffer from many methodological limitations that make them a less than ideal intervention

decision-making parameter among professions with a mature intervention literature base.

Practice Guidelines

Practice Guidelines are documents that provide applied and clinical practice

recommendations based on comprehensive literature reviews undertaken by content

experts (Atezaz-Saeed, 2004). These reviews typically evaluate treatment efficacy, known

mechanisms of action, required training, safety, and intensity or dosage of various

treatment options. The reviews and recommendations of content experts are then evaluated

by an independent panel of experts, and submitted to a sponsoring professional

organization for final approval.

Practice Guidelines have been used in medicine for more than 50 years (Woolf, Grol,

Hutinson, Eccles, & Grimshaw, 1990), and their application in other fields has increased

exponentially. To date, more than 40 professional organizations have developed Practice

Guidelines (Stuart, Rush, & Morris, 2002). Examples of sponsoring organizations include

the American Psychiatric Association, the American Academy of Child and Adolescent

Psychiatry, the Veterans Health Administration, the Department of Defense, the U.S.

Substance Abuse and Mental Health Services Administration, the Institute for Clinical

Systems Improvement, the National Institutes of Health, and the American Society of

Addiction Medicine, just to name a few.

Unlike Manualized Interventions or Treatment Guidelines with Algorithms, Practice

Guidelines provide practitioners with a wide range of criteria for decisions and services for

a particular problem or disorder. Typically, these guidelines include information on critical

issues related to diagnosis, assessment, intervention integrity, and outcome evaluation.

Therefore, they are typically more comprehensive than most Treatment Guidelines or

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115106

Algorithms, and the identification of first or second-line interventions is typically made on

a case-by-case basis. Although the development of Practice Guidelines does rely on the

expertise of content experts, unlike Expert Consensus Guidelines, final recommendations

must be solidly supported by a systematic literature review and subject to evaluation by

external reviewers and sponsoring professional organizations.

In their review of the validity of guidelines, Grimshaw and Hutchinson (1995) found

that those guidelines with the highest degree of validity used a formal meta-analysis or

graded systematic review to synthesize evidence, had multidisciplinary development

panels, and explicitly linked guidelines to evidence. Guidelines of low validity were those

that relied on expert opinion to synthesize evidence, included representatives of only one

discipline, and relied on informal consensus development procedures. More detailed

information on the issues surrounding the application of Practice Guidelines is presented

later in this paper.

Strategies for locating Practice Guidelines

Ultimately, the ability to conduct evidence-based practice requires the school

psychologist and other professionals to locate and appraise the scientific evidence in the

context of a particular academic and/or social/emotional need (Rosenberg & Donald,

1995). Estimates of the number of Practice Guidelines ranges from 1200 to 20,000

(Grimshaw & Hutchinson, 1995; Hayward & Laupacis, 1993) and these numbers continue

to grow. Given the diversity in the quality and content among the thousands of practice

parameters currently available, the search for reliable, valid Practice Guidelines can be a

time consuming and confusing process.

To help facilitate the location of guidelines, a number of private and federal agencies

have developed electronic databases that catalog established and emerging guidelines and

expert consensus recommendations in a variety of domains. For example, the National

Guideline Clearinghouse (NGC) (http://www.guideline.gov) is an on-line database of

clinical Practice Guidelines, which can be used to locate and compare guidelines. The

database is sponsored by the Agency for Healthcare Research and Quality (AHRQ), the

American Medical Association, and the American Association of Health Plans. To be

indexed in the database, a set of guidelines or recommendations must have been

authorized by approved organizations, associations, agencies, or societies and the

organization sponsoring the development of the guideline must be able to provide

documentation demonstrating that a systematic search and review of empirical evidence

was performed during guideline development. Moreover, all indexed guidelines must have

been developed, reviewed, or revised recently.

The NGC database allows practitioners to search for Practice Guidelines by disorder,

treatment/intervention, or sponsoring organization. However, the NGC is only one, of

many on-line resources cataloguing reviews of Practice Guidelines. Table 1 outlines some

of the many electronic sources of Practice Guideline information. To date, more than 487

Practice Guidelines relevant to children’s educational and mental health needs have been

officially registered with the National Guideline Clearinghouse. The extent to which

school psychologists are aware of existing Practice Guidelines, are able to find these

Table 1

On-line resources for practice guidelines

Organization Relevance Website

Agency for Healthcare Research and Quality

National Guideline Clearinghouse

Information on screening, prevention, diagnosis and intervention http://www.guidelines.gov

Centers for Disease Control and Prevention Guidelines relevant to screening and prevention http://wonder.cdc.gov/wonder/prevguid/prevguid.shtml

Cochrane Database of Systematic Reviews Screening, diagnosis, prevention, and intervention http://cochrane.org/cochrane/hbook.htm

Medscape Multispecialty Practice Guidelines Diagnosis and treatment of children with medical conditions http://www.medscape.com

EBM Reviews on OVID Results of empirical trials for behavioral and medical conditions http://www.ovid.com/products/clincial/ebmr.cfm

American Academy of Child and Adolescent

Psychiatry

Guidelines on the treatment and evaluation of a variety of

childhood behavioral disorders

http://www.aacap.org/clinical/Summ-1.htm

MEDLINE Results of empirical trials for behavioral and medical conditions http://www.ncbi.nim.hig.gov80/Literature/index.html

Centre for Evidence-Based Medicine Information on evidence-based practice, updates on new

guidelines

http://cebm.jr2.ox.ac.uk/docs/otherebmgen.html

Evidence-Based Medicine Listing of evidence-based practice resources http://www.herts.ac.uk/lis/subjects/health/ebm.htm

Health Services/Technology Assessment Texts Government guidelines, protocols, consensus statements and reports http://text.nim.nih.gov

New York Academy of Medicine Evidence-Based

Medicine Resource Center

Resources on evidence-based practice, methods for establishing and

appraising evidence

http://www.ebmny.org

American Psychiatric Association Practice Guidelines for treatment of common childhood behavioral

disorders

http://www.psych.org/clin_res/prac_guide.cfm

Interdisciplinary Council on Developmental and

Learning Disorders

Practice Guidelines for the treatment of childhood behavioral and

learning disorders

http://icdl.com/ICDLguidelines/toc.htm

National Center on Educational Outcomes Guidelines on the assessment and inclusion of students with

disabilities

http://www.education.umn.edu/nceo/overview/overview.html

Primary Care Practice Guidelines Guidelines on the treatment, prevention and assessment of child

behavioral, learning, and medical disorders

http://medicine.ucsf.edu/resources/guidelines/index.html

American Academy of Pediatrics Guidelines on the assessment and treatment of behavioral and

medical disorders

http://www.aap.org/policy/paramtoc.html

Buros Institute Standards for teacher competence in student assessment. Code of

professional responsibilities in educational measurement

http://www.unl.edu/buros/article3.html

http://www.unl.edu/buros/article2.html

Council for Exceptional Children Code of Ethics and Standards of Practice in the practice and training

of practitioners to work with students with exceptionalities

http://www.cec.sped.org/ps/code.html

National Institute of Health On-line Library Gateway to research updates and empirical trails http://nihlibrary.nih.gov/resourceindex.htm

National Institute of Mental Health Updates on clinical research trials and consensus group reports http://www.nimh.nih.gov/home.cfm

National Association of School Psychologists Consensus based position papers regarding the treatment and

assessment of a variety of school-based learning and behavioral

disorders

http://www.nasponline.org/information/position_paper.html

American Psychological Association Society of

Clinical Psychology

Overview of disorder based empirically supported treatments http://www.apa.org/divisions/div12/rev_est/index.shtml

J.L.White,

T.R.Kratochwill

/JournalofSchoolPsych

ology43(2005)99–115

107

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115108

guidelines, judge the relative quality of information presented, and apply guideline

recommendations to their practice is largely unknown. Preliminary studies have found that

school psychologists have a generally low level of familiarity with Practice Guidelines

relevant to the practice of school psychology, and rarely apply guideline recommendations

to actual cases (White & Kratochwill, submitted for publication). Given the absence of

professionally sanctioned school psychology-specific Practice Guidelines, this finding is

not all that surprising. However, it does raise the question of how school-based

practitioners can (or should) use Practice Guidelines developed by non-school psychology

related professions? As the number and diversity of Practice Guidelines increase, and

practitioners’ exposure to competing guideline recommendations grows, guidance on this

issue from professional bodies within the field of school psychology or development of a

process for guideline endorsement may become necessary and is discussed later in the

paper.

Judging the quality of Practice Guidelines

There is an increasing awareness that not all Practice Guidelines are created equal. In

fact, Practice Guidelines developed by different organizations, to remediate the same

problem or disorder, can be dramatically different. Some guidelines may be heavily

evidence-based, quantitative, and derivative, whereas another guideline may only

summarize the risks and benefits associated with alternative treatments and offer context

specific caveats regarding implementation. Given the wide diversity in the nature of

Practice Guidelines, some organizations have developed guidelines whose sole purpose is

to guide the development of future Practice Guidelines.

The most widely cited bguidelines for guidelinesQ are those developed by the Institute

of Medicine (1990, 1992). The Institute of Medicine (IOM) identified eight important

attributes of Practice Guidelines: validity, reproducibility/reliability, clinical applicability,

clinical flexibility, clarity, multidisciplinary process, scheduled review, and documentation

(Institute of Medicine, 1992). The IOM also suggested that developers should consider the

following additional criteria in developing guidelines: (a) From whose perspective are the

outcomes considered—patients, practitioners, administrators, society, or a combination of

these? (b) How are competing values arbitrated? (c) How have costs been conceptualized

and quantified (Indeed, only until more rigorous economic evaluations of school-based

interventions are developed will the inclusion of costs or cost-effectiveness be appropriate

to include in most Practice Guidelines).

Should school psychologists use Practice Guidelines?

In this final section of the paper we address the issue of whether the profession of

school psychology should use and/or develop Practice Guidelines. Currently, intervention-

relevant research findings are widely scattered throughout a poorly organized and ever-

burgeoning literature base in school psychology and related psychological treatment

specialties. Given the sheer bulk of the professionally-relevant literature, many

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115 109

practitioners and trainers may feel overwhelmed in their efforts to find an authoritative

source of practice-relevant intervention information. The movement toward identification

of EBIs such as through school and clinical psychology task forces as well as the What

Works Clearinghouse will be helpful to the school psychology field in identifying these

programs and procedures but may not be enough to promote their adoption, use, and

sustainability in school practice. The utilization of Practice Guidelines is an alternate,

arguably more practical, strategy practitioners could use to integrate evidence-based

information into their assessment and intervention decisions for several reasons which can

be advanced at the conceptual level and can ultimately in some areas be addressed at the

empirical level.

First, Practice Guidelines may promote adoption, use, and sustainability of a

package of strategies relevant to solving or addressing a problem or disorder. Practice

Guidelines represent tools that involve the full spectrum of issues surrounding

intervention strategies such as diagnosis, assessment, treatment, and outcome

evaluation. Few sources in our profession provide the detail needed to implement a

treatment in the way that Practice Guidelines accomplish this process. Nevertheless,

several studies have found that the publication and distribution of Practice Guidelines

alone is not enough to change clinical practice (Cabana et al., 1999; Kosecoff et al.,

1987; Lomas, 1991). For change to be sustained, Practice Guidelines must be

combined with the necessary professional development, and environmental context

information (e.g., school climate, school leadership, other programs in place, cultural

issues) to facilitate implementation (Fishbein, 1995). Other professional discipline level

variables, such as the establishment of administrative rules and regulations governing

the use of EBIs, the provision of ongoing supervision and feedback, and increasing

consumer awareness and demand for evidence-based services may also be necessary

for changes in practice to be maintained (Davis, Thomson, Oxman, & Haynes, 1995;

Greco & Eisenberg, 1993).

Second, unlike efforts to promote empirical practice that require practitioners to

evaluate their own practices and procedures, Practice Guidelines could be used to

support scientifically based decision-making in a timely and cost-efficient manner.

Traditionally, evidence-based practice wherein practitioners are urged to use, for

example, single-participant research methods to evaluate interventions and increase

their knowledge of effective treatments (e.g., Barlow, Hayes, & Nelson, 1984) has

contributed little to improving the integration of research and practice for most school-

based practitioners. In contrast, Practice Guidelines often and can include reference to

tailor-made evaluation tools to determine whether an intervention is effective (e.g.,

problem or disorder specific scales and checklists). These customized tools can help

focus practice on specific dimensions of problems that need to be considered and

implemented in outcome evaluation. In this regard the development of Practice

Guidelines may also be a more practical and attractive option for documenting

school-based intervention parameters. The development of Practice Guidelines for

school-based interventions could provide practitioners with clear documentation of

intervention effects, in addition to important information about diagnostic, assessment,

integrity, and outcome evaluation issues that must also be considered during the

problem-solving process. If constructed properly, Practice Guidelines could provide

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115110

practitioners with a more reliable and comprehensive level of information than

Manualized Interventions, Treatment Guidelines/Algorithms, or Expert Consensus

Guidelines can provide. The empirical literature base is growing to support such an

endeavor, and the development of Practice Guidelines for school-based interventions

should be considered.

Third, although little is known regarding the potential effects of Practice Guidelines

on the processes or outcomes of school psychology interventions, effectiveness data

emerging from the medical literature suggests there may be reason to be optimistic

about the potential positive benefits of Practice Guidelines on intervention outcomes.

Indeed, Practice Guidelines have been demonstrated to increase the utilization of EBI

information among practitioners in related professional fields, and may be a means to

promote the use of EBIs among school-based practitioners. Ultimately, this issue can be

addressed empirically by designing studies that measure the treatment utility of Practice

Guidelines as compared to more conventional practice formats (see Braden &

Kratochwill, 1997; Kratochwill & McGivern, 1996 for further discussion of the merits

of using a treatment utility model to evaluate assessment–treatment links). In terms of

effects on client outcomes in other areas of health care practice, Grimshaw and Russell

(1993) found that of the 50 investigations reviewed, only 4 failed to find any

statistically significant improvement in the process of case, and 9 of 11 investigations

examining the impact of guidelines on clients’ outcomes reported statistically

significant findings. On the basis of their synthesis of the available evidence,

Grimshaw and Russel (1993) concluded that bexplicit guidelines do improve clinical

practice (p.1321).QFourth, Practice Guidelines may have profession-wide benefits that extend beyond

changes in practitioner behavior and improving student outcomes to their effect on

professional preservice and inservice training. Published guidelines could be potentially

beneficial teaching tools, and be used for practicum and internship education by guiding

students to use strategies consistent with the best evidence available. Adoption of Practice

Guidelines also allow us to embrace a competency-based model of intervention training

because specific practice criteria can be established ranging from diagnosis to outcome

evaluation. The process of guideline development and subsequent training (preservice and

inservice) may also help uncover areas of practice uncertainty, which could help direct

research activities.

Recommended next steps

Readers may well ask by whom and how will Practice Guidelines be developed and

used in the profession of school psychology. Although a number of options are available

for this agenda to be enacted, it may be cost and time efficient to link these efforts to the

Task Force on Evidence-Based Interventions in School Psychology as it is already

considering the research-to-practice agenda supported by our three major professional

organizations.

First, a careful review of current Practice Guidelines relevant to school psychology

practice needs to be initiated. As noted in this paper, there are numerous guidelines

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115 111

already developed and these guidelines should be reviewed for consideration for

adoption. In addition to the necessary attention to duplication of effort, some specific

templates for review need to be considered for the review process. It is possible, for

example, to review the Practice Guidelines with consideration of school contextual

variables in mind. This template could be based on existing efforts of the Task Force

on Evidence-Based Interventions in School Psychology. Unfortunately, because few

Practice Guidelines (that meet NGC criteria) currently exist for the treatment of school-

based problems most practitioners commonly see, the NGC database and other

resources like it may currently be of limited relevance for many practicing school

psychologists. However, should the field of school psychology decide to pursue the

development of Practice Guidelines, we may be able to take advantage of an emerging

infrastructure and technology to support evidence-based practice where development is

already well underway. Funding of this process could be provided through either

federal or foundational sources. This funding would support the review and

development process and allow summaries to be created and ultimately shared with

the profession.

Second, a structure to guide the development of Practice Guidelines in school

psychology should be developed. The IOM framework may provide a useful starting

point, however, additional criteria may be advantageous to consider during the

development process. For example, although taking client preferences into account is

considered an important component of evidence-based practice (Sackett et al., 2000), in

their review of guideline content, Shaneyfelt, Mayo-Smith, and Rothwangl (1999)

found that only 21.5% of guidelines reviewed discussed the role of client preferences

in choosing among available options. The important role teacher, student, and parental

treatment acceptability plays in the success of school-based interventions has also been

widely cited within the school psychology literature (e.g., Elliott, 1988). Therefore, it

may be appropriate for school psychology-based Practice Guidelines to clearly

delineate issues of acceptability, and specify the extent to which deviation is tolerated

when (a) teacher, (b) parent, and/or (c) student treatment acceptability is low.

Third, a system for state, regional, and national dissemination of guideline information

needs to be developed. This system should take into account issues of dissemination for

both school psychologists in practice, faculty in graduate training programs, and student

trainees. Technological solutions such as cross program web-based course options and

creation of university partnerships (called the Virtual University) could be helpful in the

process of sharing information on Practice Guidelines (see Kratochwill, Steele-Shernoff, &

Sanetti, 2004).

Conclusions

Practice Guidelines are no panacea for improving the quality of mental health and

educational services to our schools. Yet, they hold promise for improving the quality of

services provided to our children and schools in that they can convey critical

information about a full range of issues in problem solving and implementing

prevention and intervention programs. They will be especially useful to trainers and

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115112

practitioners if they include contextual information necessary to effectively implement

prevention and intervention programs and services. Still, important research issues

surrounding their effectiveness in improving services needs to be addressed in our

profession for guidelines adopted and/or developed. Moreover, ethical issues will need

to be addressed as Practice Guidelines could become a new standard for the quality of

mental health and education services provided by school psychologists in schools.

Ultimately, our ability to disseminate, adopt, use, and sustain Practice Guidelines is up

to our profession and those who are willing to take leadership in this area.

Acknowledgement

The authors thank Karen O’Connell for word processing the manuscript and Lois

Triemstra for her assistance with the figure.

Requests for reprints should be sent to Jennifer L. White or Thomas R. Kratochwill,

School Psychology Program, University of Wisconsin–Madison, 1025 West Johnson St.,

Madison, WI 53706-1796.

References

Alarcon, R. D., Glover, S., Boyer, W., & Balon, R. (2000). Proposing an algorithm for the pharmacological

treatment of posttraumatic stress disorder. Annals of Clinical Psychiatry, 12(4), 239–246.

Albano, A. M., Marten, P., & Holt, C. S. (1991). Therapist’s manual for cognitive-behavioral group therapy for

adolescent social phobia. Albany7 State University of New York Unpublished manuscript.

Allen, M. H., Currier, G. W., Hughes, D. H., Reyes-Harde, M., & Docherty, J. P. (2001). The expert consensus

guideline series: Treatment of behavioral emergencies. New York7 McGraw-Hill (A postgraduate medicine

special report).

Altshuler, L. L., Cohen, L. S., Moline, M. L., Kahn, D. A., Carpenter, D., & Docherty, J. P. (2001). The expert

consensus guideline series: Treatment of depression in women. New York7 McGraw-Hill (A postgraduate

medicine special report).

American Psychological Association (2001). Expert consensus guidelines. http://www.psychguides.com,

Retrieved 10/10/02.

Atezaz-Saeed, S. (2004). An overview of clinical decision-making parameters. In C. E. Stout, & R. A. Hayes

(Eds.), The evidence-based practice: Methods, models, and tools for mental health professionals. Hoboken,

NJ7 Wiley.

Barlow, D. H. (2004). Psychological treatments. American Psychologist, 59, 869–878.

Barlow, D. H, Hayes, S. N., & Nelson, R. O. (1984). The scientist–practitioner. Elmsford, NY7 Pergamon Press.

Braden, J. P., & Kratochwill, T. R. (1997). Treatment utility of assessment: Myths and realities. School

Psychology Review, 26, 475–485.

Buscema, C. A., Abbasi, Q. A., Barry, D. J., & Lauve, T. H. (2000). An algorithm for the treatment of

schizophrenia in the correctional setting: The forensic algorithm project. Journal of Clinical Psychiatry,

61(10), 767–783.

Cabana, M. D., Rand, C. S., Powe, N. R., Wu, A. W., Wilsoff, M. H., Abboud, P. C., et al. (1999). Why don’t

physicians follow clinical practice guidelines? A framework for improvement. Journal of American Medical

Association, 282, 1458–1465.

Chambless, D. L., & Ollendick, T. H. (2001). Empirically supported psychological interventions: Controversies

and evidence. Annual Review of Psychology, 52, 685–716.

Coplan, J. D., & Gorman, J. M. (1999). Panic disorder. In J. Fawcett, D. J. Stein, & K. O. Jobson (Eds.), Textbook

of treatment algorithms in psychopharmacology (pp. 88–89). New York7 Wiley.

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115 113

Cournoyer, B. R., & Powers, G. T. (2002). Evidence-based social work: The quiet revolution continues. In A. R. Roberts,

& G. Greene (Eds.), The social work desk references (pp. 798–806). New York7 Oxford University Press.

Crismon, M. L., Trivedi, M. H., Pigott, T. A., Rush, A. J., Hirschfeld, R. M., Kahn, D. A., et al. (1999). The Texas

Medication Algorithm Project: Report of the Texas Consensus Conference Panel on Medication Treatment of

Major Depressive Disorder. Journal of Clinical Psychiatry, 60, 142–156.

Dantzler, A., & Osser, D. N. (1999). Algorithms for the pharmacotherapy of acute depression in patients with

bipolar disorder. Psychiatric Annals, 29, 270–284.

Davidson, G. C., & Lazarus, A. A. (1995). The dialectics of science and practice. In S. Hayes, V. Follette,

R. Dawes, & K. Grady (Eds.), Scientific standards of psychological practice: Issues and recommendations

(pp. 95–120). Reno, NV7 Context Press.

Davis, D. A., Thomson, M. A., Oxman, A. D., & Haynes, R. B. (1995). Changing physician performance: A

systematic review of the effect of continuing medical education strategies. Journal of American Medical

Association, 274, 700–705.

Dobson, K., & Shaw, B. (1988). The use of treatment manuals in cognitive therapy: Experience and issues.

Journal of Consulting and Clinical Psychology, 56, 673–682.

Elliott, S. N. (1988). Acceptability of behavioral treatments: Review of variables that influence treatment

selection. Professional Psychology, 19, 68–80.

Fishbein, M. (1995). Developing effective behavioral change interventions: Some lessons learned from behavioral

research. Reviewing the behavioral science knowledge base on technology transfer (NIDA Research

Monograph No. 155). Rockville, MD7 National Institute on Drug Abuse.

Flannery-Schroeder, E. C., & Kendall, P. C. (1996). Cognitive-behavioral therapy for anxious children: Therapist

manual for group treatment. Ardmore, PA7 Workbook Publishing.

Frances, A., Kahn, D., Carpenter, D., Frances, C., & Docherty, C. (1998). A new method of developing expert

consensus practice guidelines. The American Journal of Managed Care, 4, 1023–1029.

Frances, A., Kahn, D., Carpenter, D., Frances, C., & Docherty, J. (1998, July). A new method of developing

expert consensus practice guidelines. American Journal of Managed Care (7)1023–1029.

Giebenhain, J., & O’Dell, S. (1984). Evaluation of a parent—training manual for reducing children’s fear of the

dark. Journal of Applied Behavior Analysis, 17, 121–125.

Greco, P. J., & Eisenberg, J. M. (1993). Changing physician practices. New England Journal of Medicine, 329,

1271–1274.

Greist, J. H., & Jefferson, J. W. (1998). Pharmacotherapy for obsessive compulsive disorder. British Journal of

Psychiatry, 173(Suppl. 35), 64–70.

Grimshaw, J. M., & Hutchinson, A. (1995). Clinical practice guidelines—do they enhance value for money in

health care? British Medical Bulletin, 51, 927–940.

Grimshaw, J. M., & Russell, I. T. (1993). Effect of clinical guidelines on medical practice: A systematic review of

rigorous evaluations. Lancet, 342, 1317–1322.

Hatcher, S., Butler, R., & Oakley-Browne, M. (2005). Evidence-based mental health care. Edinburgh7 Elsevier.

Hayward, R. S., & Laupacis, A. (1993). Initiating, conducting and maintaining guidelines development programs.

Canadian Medical Association Journal, 148(4), 507–512.

Hoagwood, K., & Johnson, J. (2003). School psychology: A public health framework: I. From evidence-based

practices to evidence-based policies. Journal of School Psychology, 41, 3–21.

Howard, B., & Kendall, P. C. (1996). Family cognitive-behavioral therapy for anxiety-disordered children: A

multiple-baseline evaluation. Cognitive Therapy and Research, 20, 423–443.

Hughes, C. W., Emslie, G. J., Crismon, M. L., Wagner, K. D., Birmaher, B., & Geller, B. (1999). The Texas

Children’s Medication Algorithm Project: Report of the Texas Consensus Conference Panel on Medication

Treatment of Childhood Major Depressive Disorder. Journal of the American Academy of Child and

Adolescent Psychiatry, 38, 1442–1454.

Institute of Medicine (1990). Clinical practice guidelines: Directions for a new program. Washington DC7

National Academy Press.

Institute of Medicine (1992). Guidelines for clinical practice: From development to use. Washington DC7

National Academy Press.

Institute of Medicine (2001). Crossing the quality chasm: A new health system for the 21st century. Washington,

DC7 National Academy Press.

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115114

Kendall, P. C., & Flannery-Schroeder, E. C. (1995). Rigor, but not rigor mortis, in depression research

methodology. Journal of Personality and Social Psychology, 68, 892–894.

Kendall, P. C., Kane, M., Howard, B., & Siqueland, L. (1992). Cognitive-behavioral therapy for anxious

children: Therapist manual. Ardmore, PA7 Workbook Publishing.

Kosecoff, J., Kanouse, D. E., Rogers, W. H., McCloskey, L., Winslow, C. M., & Brook, R. H. (1987). Effects of

the National Institutes of Health Consensus Development Program on physician practice. Journal of

American Medical Association, 258, 2708–2713.

Kratochwill, T. R., & McGivern, J. E. (1996). Clinical diagnosis, behavioral assessment, and functional analysis:

Examining the connection between assessment and intervention. School Psychology Review, 25, 342–355.

Kratochwill, T. R., & Shernoff, E. S. (2003). Evidence-based practice: Promoting evidence-based interventions in

school psychology. School Psychology Quarterly, 18(4), 389–408.

Kratochwill, T. R., & Stoiber, K. C. (2000). Empirically supported interventions and school psychology:

Conceptual and practice issues—Part II. School Psychology Quarterly, 15, 75–105.

Kratochwill, T. R., & Stoiber, K. C. (2002). Evidence-based interventions in school psychology: Conceptual

foundations of the procedural and coding manual of Division 16 and the Society for the Study of School

Psychology Task Force. School Psychology Quarterly, 17(4), 341–389.

Kratochwill, T. R., Shernoff, L. S., & Sanetti, L. (2004). Promotion of academic careers in school psychology: A

conceptual framework of impact points, recommended strategies, and hopeful outcomes. School Psychology

Quarterly, 19, 342–364.

Linstone, H. A., & Turnoff, M. (1975). The Delphi method: Techniques and applications. Reading Massachusetts7

Addison-Wesley Publishing Company.

Lomas, J. (1991). Words without action? The production, dissemination, and impact of consensus

recommendations. Annual Review of Public Health, 12, 41–65.

March, J., & Mulle, K. (1996). Cognitive-behavioral psychotherapy for obsessive–compulsive disorder. Durham,

NC7 Duke University Medical Center Unpublished manuscript.

Marshall, R. D., & Schneier, F. R. (1996). An algorithm for the pharmacotherapy of social phobia. Psychiatric

Annals, 26(4), 210–216.

Mikkelsen, E. J., & McKenna, L. (1999). Psychopharmacologic algorithms for adults with developmental

disabilities and difficult-to-diagnose behavioral disorders. Psychiatric Annals, 29(5), 302–314.

Milne, D., & Paxton, R. (1998). A psychological re-analysis of the scientist-practitioner model. Clinical

Psychology & Psychotherapy, 5(4), 216–230.

Pliszka, S. R., Greenhill, L. L., Crismon, M. L., Sedillo, A., Carlson, C., Conners, C. K., et al. (2000). The Texas

Children’s Medication Algorithm Project: Report of the Texas Consensus Conference Panel on Medication

Treatment of Childhood Attention-Deficit/Hyperactivity Disorder: I. Attention-deficit/hyperactivity disorder.

Journal of American Academy of Child and Adolescent Psychiatry, 39(7), 908–919.

Rosenberg, W., & Donald, A. (1995). Evidence based medicine: An approach to clinical problem-solving. British

Medical Journal, 310, 1122–1126.

Rowland, N., & Gross, S., (Eds.) (2000). Evidence-based counselling and psychological therapies: Research and

Applications. London7 Routledge.

Rush, A. J., & Frances, A. (2000). The expert consensus guideline series: Treatment of psychiatric and behavioral

problems in mental retardation. American Journal on Mental Retardation, 105, 159–228.

Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (2000). Evidence-based

medicine: How to practice and teach EBM. (2nd ed.). New York7 Churchill Livingstone.

Shaneyfelt, T. M., Mayo-Smith, M. F., & Rothwangl, J. (1999). Are guidelines following guidelines? The

methodological quality of clinical practice guidelines in the peer-reviewed medical literature. Journal of the

American Medical Association, 281, 1900–1905.

Soloff, P. H. (1998). Algorithms for pharmacological treatment of personality dimensions: Symptom-specific

treatment for cognitive-perceptual, affective, and impulsive-behavioral dysregulation. Bulletin of the

Menninger Clinic, 62(2), 195–214.

Stigler, K. A., Posey, D., McDougle, C. J., & Whitcomb (2003). Advances in the pharmacotherapy of pervasive

developmental disorders (PDD). Child and Adolescent Psychopharmacology News, 8(7), 6–7.

Stoiber, K. C., & Kratochwill, T. R. (2000). Evidence-based intervention programs: Rethinking, refining, and

reframing the new standing section of School Psychology Quarterly. School Psychology Quarterly, 16, 1–8.

J.L. White, T.R. Kratochwill / Journal of School Psychology 43 (2005) 99–115 115

Stoiber, K. C., & Kratochwill, T. R. (2002). Outcomes PME: Planning, monitoring, and evaluating. The

Psychological Corporation.

Stuart, G. W., Rush, A. J., & Morris, J. A. (2002). Practice guidelines in mental health and addiction services:

Contributions from the American College of Mental Health Administration. Administration and Policy in

Mental Health, 30(1), 21–33.

Task Force on Promotion and Dissemination of Psychological Procedures (1995). Training in and dissemination

of empirically validated psychological treatments: Report and recommendations. Clinical Psychologist, 48,

3–23.

Thomas, A., & Grimes, J. (Eds.). (2002). Best practices in school psychology IV. Bethesda, MD7 National

Association of School Psychologists.

Weisz, J. R., & Hawley, K. M. (2002). Procedural and coding manual for identification of beneficial treatments

(Draft #4). Washington DC7 American Psychological Association, Society for Clinical Psychology Division

12 Committee on Science and Practice.

White, J.L., & Kratochwill, T.R. (2004). A survey of school psychologists’ awareness and use of treatment

guidelines: Present and future (Manuscript submitted for publication).

Wilson, G. T. (1995). Empirically validated treatments as a basis for clinical practice: Problems and prospects. In

S. C. Hayes, V. M. Follette, R. D. Dawes, & K. Grady (Eds.), Scientific standards of psychological practice:

Issues and recommendations (pp. 163–196). Reno, NV7 Context Press.

Wilson, G. T. (1996). Treatment of bulimia nervosa: When CBT fails. Behaviour Research and Therapy, 34,

197–212.

Woolf, S. H., Grol, R., Hutinson, A., Eccles, M., & Grimshaw, J. (1999). Clinical guidelines: Potential benefits,

limitations, and harms of clinical guidelines. British Medical Journal, 318, 527–530.