Chapter 14

©2011, Cengage Learning, Brooks/ Col©2011, Cengage Learning, Brooks/ Cole Publishinge Publishing

Becoming an Effective Becoming an Effective Policy AdvocatePolicy Advocate

PowerPoint created byPowerPoint created by

Gretchen Heidemann, MSW, PhD CandidateGretchen Heidemann, MSW, PhD Candidate

University of Southern CaliforniaUniversity of Southern California

School of Social WorkSchool of Social Work


Chapter 14Chapter 14Assessing Policy: Towards Assessing Policy: Towards

Evidence-Based Policy Evidence-Based Policy


Assessing PolicyAssessing Policy

Policy assessment is Task #8 of the Policy Policy assessment is Task #8 of the Policy Practice and Policy Advocacy ModelPractice and Policy Advocacy Model

We depart from the belief that social We depart from the belief that social welfare policy can and should be based on welfare policy can and should be based on the notion of the notion of what workswhat works


Evidence-Based PolicyEvidence-Based Policy

Evidence-based policyEvidence-based policy is the utilization of is the utilization of assessment and evaluation to guide policy assessment and evaluation to guide policy advocacy through its many tasksadvocacy through its many tasksIs drawn from the concept of Is drawn from the concept of evidence-evidence-based practicebased practice in social work, which in turn in social work, which in turn is drawn from the concept of is drawn from the concept of evidence-evidence-based medicinebased medicine::– ““the conscientious, explicit and judicious use the conscientious, explicit and judicious use

of current best evidence in making decisions of current best evidence in making decisions about the care of individual patients”about the care of individual patients”


Evidence-Based PolicyEvidence-Based Policy

A symbiotic relationship exists between A symbiotic relationship exists between evidence-based practice and evidence-based evidence-based practice and evidence-based policy policy – Evidence-based policyEvidence-based policy consists of policies that allow consists of policies that allow

evidence-based practices to be implemented in evidence-based practices to be implemented in specific settings, such as by: specific settings, such as by:

funding themfunding them

hiring staff to provide themhiring staff to provide them

requiring staff in specific settings to use themrequiring staff in specific settings to use them

– Evidence-based practice comes to naught if Evidence-based practice comes to naught if evidence-based policies are not enacted to support itevidence-based policies are not enacted to support it


Purposes of Evidence-based Purposes of Evidence-based PolicyPolicy

Evidence-based policy involves utilizing Evidence-based policy involves utilizing evidence (i.e. the results of assessments and evidence (i.e. the results of assessments and evaluations) throughout the policy advocacy evaluations) throughout the policy advocacy process, such as during:process, such as during:– agenda-settingagenda-setting– problem-analyzingproblem-analyzing– proposal-writingproposal-writing– policy-enactingpolicy-enacting– policy-implementingpolicy-implementing– policy-assessingpolicy-assessing



Evidence-based Policy for Agenda SettingEvidence-based Policy for Agenda Setting Policy advocates can use the results of rigorous Policy advocates can use the results of rigorous research to establish a policy agenda by:research to establish a policy agenda by:– Assessing existing programs and services and using the Assessing existing programs and services and using the

evidence of their effectiveness (or ineffectiveness) to evidence of their effectiveness (or ineffectiveness) to formulate a policy agendaformulate a policy agenda

– Comparing several different policy options that address a Comparing several different policy options that address a social problem to find the best one, then use the evidence to social problem to find the best one, then use the evidence to advocate for it over the othersadvocate for it over the others

– Assessing the impact of a Assessing the impact of a lacklack of a coherent or of a coherent or comprehensive policy to address a social problem, and using comprehensive policy to address a social problem, and using the evidence tothe evidence to formulate a policy agendaformulate a policy agenda

– Using evidence to contend that an existing policy actually Using evidence to contend that an existing policy actually works against a specific policy objectiveworks against a specific policy objective



Evidence-based Policy for Policy-Analyzing Evidence-based Policy for Policy-Analyzing and Proposal-Writingand Proposal-WritingPolicy advocates can use the results of rigorous Policy advocates can use the results of rigorous research to analyze problems and to write research to analyze problems and to write proposals to address them by:proposals to address them by:– Conducting research to assess the impact of Conducting research to assess the impact of

previously enacted social welfare policies so that they previously enacted social welfare policies so that they can be expanded, altered, or completely done away can be expanded, altered, or completely done away with, as appropriatewith, as appropriate

– Using evidence of the negative impacts of an existing Using evidence of the negative impacts of an existing policy to analyze a social problem and to develop policy to analyze a social problem and to develop fresh solutionsfresh solutions



Evidence-based policy for policy-enacting Evidence-based policy for policy-enacting

Policy advocates can use evidence that a policy Policy advocates can use evidence that a policy is cost-effective in order to gain support for it is cost-effective in order to gain support for it from legislatorsfrom legislators

Although some legislators still rely on their Although some legislators still rely on their ideology, cronyism, and patronage, the use of ideology, cronyism, and patronage, the use of empirical data is often a critical power resource empirical data is often a critical power resource during the political processduring the political process



Evidence-based policy for implementation Evidence-based policy for implementation and assessmentand assessment

Evidence-based policy often shapes the Evidence-based policy often shapes the implementation process because it mandates implementation process because it mandates the use of specific evidence-based practicesthe use of specific evidence-based practices

Advocates often need fresh evidence to be Advocates often need fresh evidence to be certain the policies that they enact are effectivecertain the policies that they enact are effective– If the policy reform isn’t as cost effective as they had If the policy reform isn’t as cost effective as they had

hoped, they can propose modifications of it during hoped, they can propose modifications of it during implementation to seek a more cost effective policyimplementation to seek a more cost effective policy


The Fundamental Logic of The Fundamental Logic of ResearchResearch

The policy assessing task represents both the The policy assessing task represents both the ending and the beginning of social policy practiceending and the beginning of social policy practice– When our assessments of existing policies suggest When our assessments of existing policies suggest

that they are flawed, we are motivated to develop, that they are flawed, we are motivated to develop, enact and implement policies to change themenact and implement policies to change them

Policy assessment requires examining relationships Policy assessment requires examining relationships between implemented policies and their effectsbetween implemented policies and their effects

It forces us to ask:It forces us to ask: – How, if at all, the world is different because a specific How, if at all, the world is different because a specific

policy existspolicy exists– What, if any, difference it would make if we removed What, if any, difference it would make if we removed

or modified the policyor modified the policy


Similarities between Assessing Similarities between Assessing and Analyzing Policyand Analyzing Policy

Policy assessment and analysis are strikingly Policy assessment and analysis are strikingly similarsimilar

A policy assessment matrix allows us A policy assessment matrix allows us examine a policy innovation and one or more examine a policy innovation and one or more alternativesalternatives– Comparing an enacted policy with something Comparing an enacted policy with something

else provides a standard for evaluationelse provides a standard for evaluation


Similarities between Assessing Similarities between Assessing and Analyzing Policyand Analyzing Policy

The evaluator must identify what criteria or The evaluator must identify what criteria or outcomes will be used to compare the policy outcomes will be used to compare the policy alternativesalternatives– Each outcomes requires dataEach outcomes requires data– In some cases, it is beneficial to select more than oneIn some cases, it is beneficial to select more than one

Legislators often want to know about costs, in addition Legislators often want to know about costs, in addition to behavioral or other social outcome indicatorsto behavioral or other social outcome indicators

To protect against To protect against goal displacementgoal displacement, which is , which is mistaking secondary goals or objectives for the most mistaking secondary goals or objectives for the most important ones (Ex: family reunification over child well-important ones (Ex: family reunification over child well-being)being)


Similarities between Policy Similarities between Policy Assessment and Policy DebatesAssessment and Policy Debates

When subjected to rigorous, quantitative When subjected to rigorous, quantitative evaluations, many policies reflect only modest gains evaluations, many policies reflect only modest gains on some outcomeson some outcomes

Why do many policy innovations not produce the Why do many policy innovations not produce the changes that their framers intended?changes that their framers intended? – Many factors shape people’s behavior and development, Many factors shape people’s behavior and development,

such as their prior experiences, their economic condition, such as their prior experiences, their economic condition, and the persons with whom they associateand the persons with whom they associate

– Programs in the human services cannot be expected to Programs in the human services cannot be expected to transform the lives of clients, patients, and consumers transform the lives of clients, patients, and consumers dramatically and quicklydramatically and quickly

– The instruments that researchers use may also fail to The instruments that researchers use may also fail to capture some important dimensions of human behaviorcapture some important dimensions of human behavior



Policy assessments often fail to provide Policy assessments often fail to provide definitive evidence about a program’s meritsdefinitive evidence about a program’s merits

Assessments are often controversial because Assessments are often controversial because the evaluator and others disagree in their the evaluator and others disagree in their interpretations of the findingsinterpretations of the findings– Program evaluation can be regarded as a form of Program evaluation can be regarded as a form of

“a“arrgument” gument” – Policy evaluators become like debaters, who must Policy evaluators become like debaters, who must

defend their argumentsdefend their arguments



The debate could involve several dimensions:The debate could involve several dimensions:– Is the glass half full or half empty?Is the glass half full or half empty?

I.e. Are the cost reductions and social improvements enough I.e. Are the cost reductions and social improvements enough to justify continuing and enlarging the program? to justify continuing and enlarging the program?

– The time frame The time frame I.e. Is 2 years enough reveal whether the program is actually I.e. Is 2 years enough reveal whether the program is actually effective?effective?

– Weight of the outcome criteriaWeight of the outcome criteriaI.e. Isn’t it more important to save lives than a few dollars?I.e. Isn’t it more important to save lives than a few dollars?

– Accuracy of dataAccuracy of data

Individual values and perspectives influence Individual values and perspectives influence responses to all of theseresponses to all of these


Tools for Countering CriticismTools for Countering Criticism

Evaluation tools used to minimize threats:– A design

Forward-looking research (research that analyzes current phenomena) is less subject to external criticism than correlational research (research analyzing projects that have already been implemented)

– A sampling and assignment strategyA random sample decreases the likelihood that the control and experimental groups have different characteristicsRandom assignment decreases the likelihood that characteristics of the participants, rather than the effects of the program, cause any differences in the outcome



Evaluation tools used to minimize threats (cont.):– Instruments or measures

Using instruments with high validity and reliability (i.e. have been empirically tested for the accuracy of the data they yield) will be less subject to external criticism

– Findings (often in the form of statistics)

All these tools can be used to reduce the likelihood of rival explanations that critics can use to question conclusions



Why are rigorous evaluations not routinely used Why are rigorous evaluations not routinely used to assess programs’ effectiveness?to assess programs’ effectiveness?– As the rigor of evaluations increases, so do their costs As the rigor of evaluations increases, so do their costs

and the amount of time it takes to conduct themand the amount of time it takes to conduct them– It is often not practical to implement as many controls It is often not practical to implement as many controls

as are required for a truly rigorous evaluationas are required for a truly rigorous evaluation– Some evaluation techniques raise ethical questionsSome evaluation techniques raise ethical questions– The role of politics, ideology and vested interestsThe role of politics, ideology and vested interests– The risk to the policy innovator and the implementing The risk to the policy innovator and the implementing

organization if findings are not positiveorganization if findings are not positive


Overcoming Barriers to the Use Overcoming Barriers to the Use of Policy Evaluationof Policy Evaluation

Policy advocates face 2 challenges: Policy advocates face 2 challenges: – To ensure that good policy evaluation occurs To ensure that good policy evaluation occurs – To ensure that findings will actually be usedTo ensure that findings will actually be used

Michael Patton suggests that evaluation focus Michael Patton suggests that evaluation focus on on “i“inntended use by intended users”tended use by intended users”It is crucial to get people involved in the process It is crucial to get people involved in the process of evaluation in order to: of evaluation in order to: – Increase their stake in the results Increase their stake in the results – Increase the likelihood that they will find the results Increase the likelihood that they will find the results

relevantrelevant– Increase the likelihood that they will use the resultsIncrease the likelihood that they will use the results


Qualitative EvaluationsQualitative Evaluations

Criticisms of traditional, quantitative approaches:Criticisms of traditional, quantitative approaches:– Failure gather data from several sources, including

direct observations and open-ended interviews– Failure to consult consumers, clients, or providers to

determine their perspectives– Failure to capture consumers true attitudes toward the

quality or nature of the services they receive – Failure to examine “sensitive” areas like child abuse,

drug use, AIDS, sexual deviance, or mental illness– The “myth” of value-free scientific inquiry: Evaluators

are not truly unbiased and objective– Failure to give attention to the context of a program


Qualitative EvaluationsQualitative Evaluations

Feminist and qualitative researchers argue for:Feminist and qualitative researchers argue for:– Openly acknowledging biases and beliefs related to Openly acknowledging biases and beliefs related to

gender, race, sexual orientation, and classgender, race, sexual orientation, and class– Greater dialogue between researchers and subjectsGreater dialogue between researchers and subjects– New devices for collecting data, such as: New devices for collecting data, such as:

Visual imageryVisual imageryGroup diaries and associative writingGroup diaries and associative writingDramaDramaTextual analysis Textual analysis Network tracingNetwork tracing

– DataData ttrriangulationiangulation in which data are collected from in which data are collected from more than two sourcesmore than two sources


Why All Social Workers Should Why All Social Workers Should Assess PoliciesAssess Policies

ALL social workers can assess policy, even ALL social workers can assess policy, even when they lack training in researchwhen they lack training in researchThey can:They can:– Participate in arguments about research findingsParticipate in arguments about research findings– Cite researchCite research– Use theory based on their practice wisdomUse theory based on their practice wisdom– Draw on their moral standardsDraw on their moral standards– Generate relevant and realistic questions and Generate relevant and realistic questions and

hypotheses based on the intimate understanding of hypotheses based on the intimate understanding of problems they have gained through their practice problems they have gained through their practice experienceexperience


Why All Social Workers Should Why All Social Workers Should Assess PoliciesAssess Policies

Even if they don’t produce research evaluation Even if they don’t produce research evaluation findings themselves, they can:findings themselves, they can:– Participate in critical discussions of the evaluation of Participate in critical discussions of the evaluation of

specific projectsspecific projects– Critically examine evaluators’ choice of criteria, Critically examine evaluators’ choice of criteria,

methodological choices, instruments, and methodological choices, instruments, and interpretations of findingsinterpretations of findings

– Vigorously challenge evaluations that appear ill advisedVigorously challenge evaluations that appear ill advised– Draw on their professional wisdom, as well as on theory Draw on their professional wisdom, as well as on theory

drawn from the social and human sciences to criticize drawn from the social and human sciences to criticize policiespolicies

Some programs can be defended on moral grounds when Some programs can be defended on moral grounds when there is no empirical evidence of their effectivenessthere is no empirical evidence of their effectiveness

Education

Chapter 14