Factors affecting utilization of evaluation findings by direct service staff

Factors Affecting Utilization of Evaluation Findings by Direct

Service Staff

Rosemary Holland

Simulated evaluation report abstracts were used to determine whether factors previously identified as enhancing utilization of evaluation findings by policy level decisionmakers and agency directors also enhance utilization by direct service staff. Respondents were 238 direct service staff members in mental health agencies. Dependent variables were perceived usefulness and perceived truthfulness. The predictor variables were relevance of the evaluation, staff participation, quality of research design and specificity of recommendations. All predictor variables except the last were significant but relevance explained almost all the variance. The two dependent variables were highly correlated and virtually could not be distinguished from each other.

Evaluators and other persons conce rned with the ut i l izat ion of social service knowledge have long been conce rned with the ut i l izat ion of evaluat ion findings. As factors are identified that enhance ut i l izat ion of evaluations, they are t rans la ted into prescr ipt ions for evaluators (Patton, 1978; Rothman, 1980). These factors have been identified primarily from studies in which the r e sponden t s are pol icymakers , executive directors, or evaluators. However, a substant ia l a m o u n t of evaluat ion occurs at the local level in service providing agencies; and changes in these agencies require at least the cooperat ion, if not the en thus iasm, of direct service staff. Factors that inf luence direct service staff in accept ing f indings and adopt ing r e c o m m e n d a t i o n s con ta ined in evaluat ions have not yet been identified. The purpose of this s tudy was to explore whe the r certain factors that have been identified as likely to enhance the ut i l izat ion of evaluat ion f indings by pol icymakers and agency executive directors also enhance uti l ization by direct service staff of local agencies.

Rosemary Holland is currently working as a private consultant to mental health agencies for program development and program evaluation. She has extensive experience in the administration and planning of mental health services. Dr. Holland's primary interest is in user-initiated evaluations. She is on leave this year from the social work department of the University of Detroit. Address correspondence to: 470 N. Cranbrook, Birmingham, MI 48009.

Knowledge in Society: The International Journal of Knowledge Transfer, Fall 1988, Vol. 1, No. 3, 45-56.

46 Knowledge in Society/Fall 1988

From previous research on utilization of social science research and program evaluations, several variables that influence utilization by pol- icymakers and agency executive directors have emerged. Although ter- minology varies, these variables may be described as: 1) relevance of the problem, 2) staff participation in the evaluation process, 3) quality of research design, and 4) inclusion of specific recommendations for action.

Some studies have found that the utilization of findings is likely to be positively influenced by the perceived relevance of the research (Roth- man, 1980; Weiss & Bucuvalas, 1980a, 1980b). Other studies (Dunn, 1980; Patton et al., 1977) have found this factor less than crucial. Par- ticipation by program personnel in the evaluation process has been found to be a positive factor affecting utilization (Dunn, 1980; Rothman, 1980; Weeks, 1979), but these findings are not consistent with those of Patton et al. (1977). The quality of research design has been considered in most studies on utilization with results varying from positive (Weiss & Bucuvalas, 1980a) to negative correlations (van de Vail & Bolas, 1980; Weeks, 1979) as well as the conclusion that the respondents could not distinguish research on the basis of the quality of the design (Sechrest et al., no date). In both research studies and program evaluations, the inclusion of recommendations that provide a direction for action has been found to have a positive effect on utilization (Rothman, 1980; van de Vail & Bolas, 1980; Weiss & Bucuvalas, 1980a).

This exploratory study attempts to determine the extent to which direct service staff in mentat health agencies are likely to be influenced by the following variables: l) relevance of the evaluation, 2) staff participation in the evaluation process, 3) the quality of research design, and 4) the specificity of the recommendations in assessing the truthfulness and usefulness of evaluation reports.

The findings of Weiss and Bucuvalas (1980a, 1980b) distinguishing two separate "tests" applied to research findings--a test for usefulness and a test for truthfulness--influenced the design of this research. This study explores the presence of two such "tests" applied by direct service staff to program evaluation findings.

Methodology

Research Design

In this study, staff members providing direct service to clients in mental hea l th centers were shown s imula ted abst racts descr ib ing evaluations of intervention programs. The abstracts were prepared so as to systematically vary three independent variables: staff participation in the evaluation process, the quality of the research design, and the specificity of the recommendations. A high and low level for each variable was used. Three different evaluations were used, and each evaluation was presented with all possible combinations of the three variables.

Each respondent was asked to read two abstracts and answer questions about the dependability of the information (truthfulness) and the

Holland 47

l ikel ihood of its use (usefulness). Responses were expected to vary in relation to the high or low level of the variables in each abstract.

The relevance variable was not man ipu la ted in the abstracts. Three ques t ions in the ques t ionnai re (concerned with the value of doing the evaluat ion and the l ikel ihood that the f indings and r e c o m m e n d a t i o n s would have an effect on the respondent ) were des igned to de te rmine the perceived relevance of the evaluat ion s tudy to the respondent . These three ques t ions were used to de te rmine a relevance score for each case.

Al though no t involved in specific hypotheses , the fol lowing variables were included in ques t ions in the quest ionnaire:

Perceived role in decision making--the respondent's view of his/her ability to participate in decisions in the agency. Perceived climate for change--the respondent's perception of openness to change in his/her agency. Demographic variables--the respondent's length of time in the agency and in the field, and his/her academic degree. Ratings of knowledge of and experience with program evaluation--the respondent's knowledge of research design, the extent of the respondent's experience with evaluations and whether that experience was positive or negative. The dependent variables were measured as follows: Truthfulness--Five questions were asked to measure the perceived truthfulness of the evaluation findings. Three referred to the truthfulness of the evaluation itself, and two referred to truthfulness based on consistency with knowledge and values. Scores on these five questions constituted the truthfulness score for each case. Usefulness--Four ques t ions were designed to solicit responses on usefulness (usefulness of the recommendations, willingness to consider the findings, and support for implementation of the recommendations). Scores on these four questions constituted the usefulness score for each case .

The ins t ruments were pre tes ted in two ways: first, by a panel of menta l hea l th adminis t ra tors for plausibility, and second, by menta l hea l th cen- ter staff mem ber s to de te rmine that the task was unders tandab le and reasonable.

Sampling and Data Collection

The r e s p o n d e n t s were direct service staff and supervisors who re- por ted spending at least thirty percent of their t ime in direct service. They came from menta l hea l th programs opera ted or funded by count ies in sou theas t e rn Michigan. Since C o m m u n i t y Mental Health Programs were manda t ed to do program evaluat ion between 1975 and 1982, these agencies appeared to be good locales for conduc t ing the research in 1983. In fact, n ine ty percent of the r e sponden t s indicated they had expe- r ience in evaluated programs. There were 238 responden t s , each of w h o m reviewed two abstracts, providing a total of 476 cases.


Findings

The th ree h y p o t h e s e s for Perceived Tru thfu lness and Perceived Usefulness as dependent t variables were:

HI: Staffwill rate evaluations with high staff involvement in the evaluation process as more truthful and more useful than evaluations without staff involvement.

H2: Staffwill rate evaluations with a higher quality of research design as more truthful and more useful than evaluations with a lower quality of research design.

H3: Evaluations that are rated as relevant by staff will have higher truthfulness and usefulness scores than evaluations that are rated not relevant.

An additional hypothesis was: H4: Staffwill rate evaluations with specific recommendat ions as more

useful than evaluations with general recommendat ions . Multiple regression was used to predict Truthfulness and Usefulness

with quality of design, staff participation, specificity of recommendations and relevance entered as predictors.

Relevance accounted for a lmost all the variance in both equations, a l though the other variables, except for specificity of recommendat ions , were significant. For the four specific predictions, the null hypothesis can be rejected in three cases. The specificity of recommenda t ions in the evaluation was expected to influence the perceived usefulness of an evaluation but that variable was not significant, at least not as it was presented in this study.

Clearly, relevance explains most of the variance. The next step was

Table I Multiple Regression of Truthfulness

On Hypothesized Variables

Multiple R .52

R Square .27

F P

57.90 0.00

Variables in the Equation

Multiple R Simple R F

Relevance .48 .48 147.42.*

Research Design .50 .14 12.92"*

Staff Participation .52 .14 13.29"*

**p < .01

Holland 49

Table 2 Multiple Regression of Usefulness

On Hypothesized Variables

Multiple R .52

R Square .27

F P

43.94 0.00



Relevance .49 .49 154.38"*

Research Design .49 .08 4.34*

Staff Participation .52 .52 16.08"*

Specificity of Recommendation .52 .52 .04

*p < .05 **p < .01

to use the available data to explore more fully variables which influence Truthfulness and Usefulness (See Tables 3 and 4).

In these multiple regressions, truthfulness and usefulness explain most of the variance in each other. Truthfulness, explaining a significant amount of the variance is usefulness, would be expected for judgments of professionals. However, the fact that usefulness explains significant variance in truthfulness leaves us with the question, "Does it have to appear useful, in order to also appear truthful?" If the answer is yes, then direct service staff do not apply two separate tests to evaluation findings.

The other variables in this study were explored in detail elsewhere to determine the extent to which they contributed to an understanding of truthfulness or usefulness (Holland, 1984). These explorations were not fruitful.

Conclusions and Implications

The purpose of this study was to determine if direct service staff of local agencies are influenced by the same factors as policy makers, executive directors and evaluators in assessing the usefulness of an evaluation. This study was exploratory, with hypotheses formulated to ex- tend knowledge about one pool of respondents (policy makers, directors, and evaluators) to another pool of respondents (direct service staff in mental health agencies).

The most striking finding of this study is the high correlation (r = .85) between Truthfulness and Usefulness, the two dependent variables. This high correlation raises the question of whether potential knowledge-


Table 3 Multiple Regression of Usefulness

On All Variables

Multiple R

R Square

.86

.74

Truthfulness Score

Relevance Score

Climate for Change Score

Quality of Research Design

Staff Participation

Specificity of Reconm~endations

Months Worked in Agency

Months Worked in MH and Social Services

Perceived Influence on Agency Decisions

Perceived Influence on Clinical Decisions

Self-assessed Knowledge of Research Design

Experience Working in Programs Being Evaluated

Quality of Experience (positive-negative) with Program Evaluated

F P

83.17 0.00



.85 .85 601.70"*

.50 .50 9.59**

.85 .12 .91

.85 .07 .95

.85 .18 4.51"

.86 .06 1.66

.86 .00 3.48

.86 .15 3.13

.86 .16 6.113"

.86 .06 .28

.86 .04 .07

�9 86 .04 .35

.86 .20 1.26

*p < .05 **p < .01

Holland 51

Table 4 Multiple Regression of Truthfulness

On All Variables

Multiple R .86

R Square .74

Usefulness Score

Relevance Score

Climate for Change Score

Quality of Research Design

Staff Participation

Specificity of Recommendations

Months Worked in Agency

Months Worked in MH and Social Services

Perceived Influence on Agency Decisions

Perceived Influence on Clinical Decisions

Self-assessed Knowledge of Research Design

Experience Working in Programs Being Evaluated

Quality of Experience (positive-negative) with Program Evaluated

F P

80.97 0.00


Multiple R

85

85

85

85

85

86

86

Simple R

.85

.50

.ii

.12

.15

.02

.03

F

601.70"*

12.50"*

1.28

5.34*

.18

2.21

.83

.86 .15 .00

.86 .I0 2.70

.86 .03 .02

.86 .01 .05

.~ .03 .02

.86 .18 .30

*p < .05 **p < .01


users at the staff level make a distinction between T~uthfulness and Usefulness.

The independent variable that was most strongly associated with perceived Truthfulness and perceived Usefulness was Relevance. It was sig- nif icant for both d e p e n d e n t variables and expla ined a subs tant ia l amount of variance in all of the regression equations. Relevance was found to be clearly an important aspect of an evaluation study, agreeing with previous findings (Glaser, 1978; Mann & Likert, 1952; Patton, I978; Rothman, 1980; Zaltman, 1979). Weiss and Bucuvalas (1980b) found relevance to be a significant predictor of usefulness.

Utilization literature has not previously tied relevance to truthfulness. The results of this research indicate that such a relationship may exist and warrant further exploration. If direct service staff are inf luenced by the relevance to them of an evaluation in assessing its truthfulness, an essential first step in the evaluation process may be to convince potential users that the findings may affect them. In the evaluation problems presented in the abstracts, the agencies were faced with a need to address a problem caused by decreased funding or client need. Such situations are highly relevant to staff. Many other problems addressed by evaluations may be relevant to evaluators and directors, but not to direct service staff. If convincing direct service staff of relevance is essential for acceptance and implementa t ion of the evaluation findings, evaluation of many problems may be unsuccessful. The role of relevance, especially as a prdictor of truthfulness, is the most interesting finding of this study because it indicates a need for involvement and commi tment by users of the evaluation at a very early stage. Perhaps the Mann and Likert (1952) technique of making the findings relevant after the fact is possible; but their time commi tment to bringing this about was greater than that available to most evaluators (they conducted 200 meet ings in one or- ganization).

The finding that Staff Participation is a significant influence on the dependent variables is expected and is consis tent and nonstat is t ical studies that have explored utilization in depth (Patton, 1978; Rothman, 1980) and with Dunn's (1980) statistical findings.

Staff Participation was ~ significant predictor, but it accounted for only 2 percent of the variance in the Truthfulness or Usefulness scores. That Staff Participation did not explain more of the variance for either of the two dependent variables was disappointing. At least two explanations for the failure of this variable to emerge as more important exist: 1) direct service staff members may not see evaluations as activities in which they wish to be involved; and /o r 2) direct service staff members may not see themselves as a group with similar values and concerns. Whatever the reasons, staff participation probably ought not be ignored in planning an evaluation study but high staff participation may not greatly enhance utilization.

The finding that Quality of Research Design was significant with both dependent variables was not so clearly expected from past findings (Pat- ton, 1978; van de Vall &Bolas, 1980; Weeks, 1979). Of particular interest, this variable is significant for both Usefulness and Truthfulness with

Holland 53

direct service staff. Such respondents might be expected to place less value on Quality of Research Design than higher level administrators or policy makers, who have given mixed responses on this variable in other utilization studies.

Line level staff are concerned with the quality of research design, and this concern will need to be addressed by evaluators. This study does not give any indication of the extent of knowledge that direct service staff have of the quality of research design; it only suggests that they need to believe it is good.

The last independent variable is the Specificity of Recommendations, which was hypothesized only to influence Usefulness. The findings were not significant for that hypothesis, thus failing to confirm the extension of previous findings by Rothman (1980), van de Vail and Bolas (1980), and Weiss and Bucuvalas (1980a, 1980b) to this population. The use of abstracts may not be the way to determine the importance of the Specif- icity of Recommendations variable. The extent to which specific recommendations influence staff's willingness to support the evaluation's findings has probably not been adequately tested in this study.

An assumption made in this discussion is that perceived Truthfulness influences perceived Usefulness but, in fact, the causal effect could go either way. Truthfulness and Relevance are the two major predictors of Usefulness, with Staff Participation and Quality of Research Design also having significant effects.

Two additional variables were significant for both dependent variables: 1) months worked in mental health and social services agencies; and 2) respondent's overall experiences with program evaluation rated from positive to negative. The latter variable makes sense; and it provides hope that as evaluation processes improve a larger number of people will have favorable experiences and therefore be more open to seeing truth in evaluation findings. The former variable is not easy to explain. The expectation might well be that the longer a person is in the field, the more difficult it would be to convince her/him that evaluation findings are true. Because the present finding appears to confirm the opposite result, additional scrutiny is required to ascertain the effects of length of service with openness to evaluation findings. It may be that the longer people are in the field, the less confident they feel in dogma and the more open they are to "what works." Knowing more about this variable could give evaluators some valuable knowledge about their audiences.

One other variable was significant for Usefulness, the respondent's rating of himself/herself as having influence on overall agency function- ing. This variable negatively correlated with Staff Participation (r ----- . 10), which may indicate that this variable is not indicative of Usefulness as it relates to any generalized belief in staff involvement in agency function- ing, but rather that staff members are more likely to see an evaluation as useful if they believe they personally, and not jointly with other staff, will have. some power in its implementation.

The analysis was designed to explore both dependent variables to the extent possible. The picture which emerges is not greatly different for these two variables. The variable which appears to need further explora-


tion is Relevance. Greater knowledge about Relevance would lay a foun- dat ion to begin exploring the complexi t ies of Truthfulness and Usefulness to determine if, in fact, they are two distinct "tests" used to judge evaluations.

Limitation of the Study

The abstracts used to present the evaluations to the respondents were simulations and had to be written to allow for the manipulation of three variables. Therefore, they may be lacking in the degree of verisimilitude desirable in all research.

The Questionnaire, which was pretested for clarity and conceptual soundness, also has its measurement limitations. The Relevance, Truth- fulness, and Usefulness scores are measured by more than one question, and the alpha coefficients for these scales were high. However, all the other variables, which are measured by a single question, may not be measured adequately. These single measures have, at best, modest reliability and were only intended as exploratory measures.

The extent to which these findings may be generalized is not clear. The respondents were direct service staff members in mental health agencies located in a section of Michigan composed of urban, rural, and suburban areas. Although participation was voluntary, the respondents are probably representative of direct service staff in this field. However, they may not be representative of staff in social service agencies even though both types of agencies employ a large number of MSWs. Further exploration needs to be done with a variety of line workers to achieve a complete picture of the factors influencing staff at that level.

Areas for Further Study

In this study, Usefulness was primarily influenced by both Relevance and Truthfulness scores. Together they accounted for 73 percent of the variance in the Usefulness scores. The perceptions of Truthfulness and Relevance appear to be the areas of greatest interest for further research.

The role of Relevance in influencing perceived Truthfulness of an evaluation is a potentially interesting area of exploration. We need to know if direct service staff or any other group of users of evaluations make judgments about the truthfulness of findings based on the potential of the evaluation to affect them. We need to know more about those factors that account for the perceived Relevance of an evaluation in the various groups of users.

Perceived Truthfulness needs much greater exploration beyond what can be learned about it through Relevance. The factors that influence audiences in making their judgments of Truthfulness need to be identified more accurately than was accomplished in this study. Also the potential influence of Usefulness on Truthfulness needs to be explored. Basically, we need to know what judgment the user makes first and how it influences other judgments. From this study one may conclude that three basic judgments are made: 1) This evaluation could affect "me"

Holland 55

(Relevance); 2) The findings of this evaluation appear truthful (Truth- fulness); and 3) The recommendations from this evaluation appear useful (Usefulness). We need to know more about the order in which these judgments are made.

Ideally, these variables should be studied in real situations, as evaluations are: performed; but such opportunities are not likely to be plentiful. The use of abstracts may be the closest approximation to the real situa- tion; and although not the optimal means of studying utilization, they may be a very suitable means for studying factors that influence perceptions of lrelevance and truthfulness. However, if abstracts are used in this way, determining how independent variables may be presented to assure recognition of the variables by the respondents will be essential.

The study of factors affecting the utilization of evaluation findings is still in a very early develomental phase. There must be movement beyond 1this phase in order to establish the value of evaluation as an endeavor and to apply the knowledge to the ultimate goal, which is more effective intervention programs.

N o t e

The author wishes to express appreciation to the following people for their assis- tance with this research: Lee Sechrest, University of Arizona; Jack Rothman, UCLA; Tony 'lripodi, University of Pittsburgh; and Jesse Gordon, University of Michigan.

R e f e r e n c e s

Broskowski, A., White, S. L., & Spector, E E. (1979). A management perspective on program evaluation. In H. C. Schulberg & J. M. Jerrell (Eds.), The evaluator and management (pp. 105-118). Beverly Hills, CA: Sage.

Calan, N., Morrison, A., & Stambaugh, R. (1975). The use of social science knowledge in policy decisions at the national level. Ann Arbor: University of Michigan, In- stitute for Social Research.

Carmines, E.G., & Zeller, A. (1979). Reliability and validity assessment. Beverly Hills, CA: Sage.

Dunn, W. E. (1980). The two communities metaphor and models of knowledge use: An exploratory case survey. Knowledge: Creation, Diffusion, Utilization, 1 (4), 515-536.

Glasel~, E. (1978). If Mohammed Won't Come to the Mountain . . . [Special Issue] Evaluation and Change.

Holland, R. (1984). Perceived truthfulness and perceived usefulness of program evaluations by direct service staff. Unpublished doctoral dissertation, University of Michigan, Ann Arbor.

Mann, E, & Likert, R. (1952). The need for research on the communication of research results. In E Caro (Ed.), Readings in evaluation research (2nd ed. 1977) (pp. 125-136). New York: Russell Sage.

Patton, M. Q. (1978). Utilization--focused evaluation. Beverly Hills, CA: Sage. Patton, M. Q., Grimes, P. S., Guthrie, K. M., Brenna, N. J., French, A. O., & Blyth, D. H.

(1977). In search of impact: An analysis of the utilization of federal health evaluation research. In C. Weiss (Ed.), Using social research in public policy-making (pp. 141-163). Lexington, MA: Lexington Books.

Rothman, J. (1980). Using research in organizations. Beverly Hills, CA: Sage. Sechrest, L., Bloom, L., Cohen, R., Cohen, L. & Olbrisch, M. (No date). Use of

research in policy decisions. Unpublished manuscript.


Stevenson, J. E ( 1981 ). Assessing evaluation utilization in human services agencies. In J. A. Ciarlo (Ed.), Utilizing evaluation: Concepts and measurement techniques (pp. 35-58). Beverly Hills, CA: Sage.

van de Vail, M., & Bolas, C. (1980). Applied social discipline research on social policy research: The emergence of a professional paradigm in sociological research. The American Sociologist, 15, 128-137.

Weeks, E.. C. (1979). The managerial use of evaluation findings. In H. C. Schulberg & J. M. Jerrell (Eds.), The evaluatorand management (pp. 137-156). Beverly Hills, CA: Sage.

Weiss, C. & Bucuvalas, M. (1980a). Social science research and decision-making. New York: Columbia University Press.

Weiss, C. & Bucuvalas, M. (1980b). Truth tests and utility tests: Decision-makers ' frames of reference for social science research. American Sociological Review, 45, 302-313.

Weiss, J. (I 979). Access to influence. American Behavioral Scientist, 22, 437-58. Zaltman, G. (1979). Knowledge utilization as planned social change. Knowledge:

Creation, Diffusion, Utilization, 1, 82-105.

Documents

Factors affecting utilization of evaluation findings by direct service staff