AMOB Evaluation Paper

Running header: AMOB OUTCOME EVALUATION

A Matter of Balance Outcome Evaluation:

The Psychological Components of a Fall Prevention Program

Kathleen Francis

University of North Texas Health Science Center

School of Public Health

BACH 5316

Dr. Spence-Almagur

April 21, 2014


The Intervention

Among community dwelling older adults (persons living in their own homes), the

evidence based falls prevention program, “A Matter of Balance: Volunteer Lay Leader”

(AMOB: VLL), is a popular and highly effective program used by the older adult community

(Lindemare, 2014). The group based program is a multi-factorial intervention created by

Boston University that utilizes the constructs of the Social Cognitive Theory, with emphasis

on learning processes with the intention of reducing the fear of falling, to promote physical

fitness, and to enhance the social and emotional dimensions of health in older adults (Smith

et al., 2012). The core constructs of AMOB/VLL are (a) cognitive restructuring behavioral

activation activities that promote the belief that falls and fear of falling are controllable (b)

enhancement of falls self-efficacy and falls management by helping participants set realistic

goals for increasing activity (c) promotion of changes in modifiable risk factors (like the

removal of rugs in the home and lighting dark hallways) and (d) teaching exercises known to

reduce the risk of falling by increasing strength and balance (Tennstedt et al., 1998).

Currently, the Senior Citizens Services of Tarrant County performs process evaluations

on the AMOB: VLL in order to look at the degree in which the program components were

delivered and if the program is being implemented as planned so that its effects reach the

target population (older adults). After speaking with Lindsay Lindemare, the AMOB: VLL

coordinator of the Senior Citizen Services of Tarrant County, I have decided to conduct an

outcome evaluation for Texas Health Harris Methodist’s Trauma Department to assess the

potential secondary effects of AMOB: perceived general health, self-rated overall life

satisfaction, perceived performance of ADL tasks, symptoms of depression, and social

support and social connectedness.


Target Population

In my evaluation, an older adult will be defined as a person aged 65 or older who are still

living within the community, are mentally sound (able to solve basic problems/perform tasks)

and who show interest in improving their flexibility, overall balance and muscular endurance and

strength. According to the United Way of Tarrant County 2007 data, older adults make up 11%

of Tarrant County’s population, with the older adult population projected to grow from 11.9% in

2007 to almost 17% in 2020 (TAMHSC, 2009).

Within this critical population, falls are a common accident, with one in three older adults

experiencing a fall in their lifetime (Lach, 2002). The National Council on Aging states that falls

are not an inevitable part of aging, but one of the most preventable causes of injury and trauma

(Arfken, 1994; Aging, 2014). The cost of a single fall can range from roughly $3,700-26,000 per

year depending on the severity of the fall episode (Heinrich, 2010). From those older adults who

do experience a fall, along with the incredible the financial toll, comes a profound influence on

their overall quality of life. Following a fall episode, negative psychological outcomes along

with the loss of confidence to perform daily physical activity grows, which in turn severely

restricts daily activities (Smith M. , 2012). The restriction of daily activity results in the decrease

in opportunities for social interaction/ participation, puts older adults at risk for emotional

disorders such as depression and a poor outlook on life satisfaction, due to an isolated

environment (Lach, 2002; Albert & Freedman, 2010). Regular contact with other individuals

(either through daily activities or AMOB: VLL classes) promotes a sense of social integration

and belonging, reducing feelings of isolation, encouraging good health and longevity for an

individual (Visser, 2000). These key concepts have been associated with increased falls risk

(Tinetti, 1994), making them my targets for my outcome evaluation for AMOB: VLL. Comment [EA1]: Very good target population description!


Program Justification

The primary goal of the “A Matter of Balance: Volunteer Lay Leader” program (AMOB:

VLL) is to empower people who have fallen or who fear falling with a sense of control over falls

(Tennstedt et al., 1998). Using cognitive strategies to address the factors associated with falls in

older adults, the expected outcomes of the program upon completion of the classes are to have

increased activity levels along improved mobility (Aging, 2014).

The primary goal of my outcome evaluation is to assess the potential secondary effects of

AMOB: VLL of perceived general health, self-rated overall life satisfaction, perceived

performance of ADL tasks, symptoms of depression, and social support and social connectedness

upon completion of the falls prevention program.

EXAMPLE PERCEIVED GENERAL HEALTH SMART GOALS

Process Objective: Between August 2014 and August 2015, there will be an 80% pre and post

intervention completion rate of the MOS SF-20 short form health survey to assess overall

participant health status in order to evaluate program effect on participant

wellness/medical outcomes.

Outcome Objective: By the completion of one iteration of A Matter of Balance: Volunteer Lay

Leader, there will be a 10% increase in self-reported perceived general health as indicated

by the MOS SF-20 short form health survey.

Evaluation questions

1. What were the observable short-term outcomes relating to perceived general health?

2. Were there any unintended outcomes related to perceived general health?

3. To what extent have the intended outcomes been achieved/measured?

4. Should a different measurement tool be used to capture the intended target?

Comment [EA2]: objectives

Comment [EA3]: Think about re-wording this to make it more flexible (i.e., what are the strengths and weaknesses of the measurement tools utilized…)


Proposed Design and Methods

The proposed study design for the outcome evaluation of AMOB: VLL is a pre-posttest

strategy that will assess the program participants on the same designated variables i.e. perceived

general health, self-rated overall life satisfaction, perceived performance of ADL tasks,

symptoms of depression, and social support and social connectedness over the course of one

AMOB: VLL class cycle both before the participants begin the classes and eight weeks later

right after completing the graduation ceremony. Although the pre-posttest design is not as

rigorous as other potential outcome evaluation designs, I feel that it is appropriate for the goal of

the evaluation and will produce useful results not only for program improvement, but also to

show the extended effects of the program not originally outlined in the program rationale.

Through a pre-posttest design, we will be able to see the stability of the outcomes being

measured, we can determine if the evaluation and program objectives are being met, and can see

the type of client who was most impacted by the intervention through repeated measures.

Participants will be recruited or self-selected through advertisements at host centers

(church, senior community center, Texas Health Burleson etc.) and will be notified of the

outcome evaluation when they sign up for the class. All the participants will be informed that the

program will be extended to ten weeks to all the first and last week to be the designated data

collection sessions. A consent form for completion of all ten sessions will be sent via mail three

weeks prior to the first session that will be brought to the first session. Additional consent forms

will be on site at the first session for those participants who do not return the form.

The evaluation will utilize five different sites with data collected from each during the

respected scheduled site delivery periods. There will be only one control delivery site used


during the evaluation period for the sake of comparison of the program delivery effect on the

target secondary outcome measures.

The data collection methods will be a mixed-methods design with the majority of the data

quantitatively collected through various pre- and posttest survey tools. Each variable will be

measured through its own survey tool; each survey was developed to measure a specific variable.

The time needed to complete all five survey assessments will be approximately an hour and a

half and the second half hour of the session period will be available for those who need

additional time to complete the assessments. A free comment form will accompany the final

assessment surveys to allow the participants the space to write down any comments regarding the

evaluation process, class comments, or any other phrases without having to come in for an

additional focus group session or interview via telephone.

ASSESSMENT SURVEY TOOLS

Perceived General Health- This variable will be measured using the Medical Outcomes Study-

Short Form Health Survey (MOS SF-36) [appendix 1]. This is a 36 question assessment

tool used to assess a patient’s overall health and wellness by asking questions relating to

the two summary measures of physical and mental health. Those two measures are

evaluated through eight scales to gain a quick and succinct overview of a person’s health

and wellness. The eight scales are as follows: physical functioning (PF), role-physical

(RP), bodily pain (BP), general health (GH), vitality (VT), social functioning (SF), role-

emotional (RE), and mental health (MH) (Ware, 2002). The questions are mixed mode,

utilizing both Y/N questions and variations of Likert scales.

Self-Rated Overall Life Satisfaction- This variable will be measured using Ryff’s Psychological

Well-Being Scales (PWB) [appendix 2], 42 Item version, that asks questions relating to


self-acceptance, personal growth, purpose in life, environmental mastery, positive

relations with others, and autonomy through a 1-6 Likert scale, (1-strongly disagree, 6-

strongly agree) (Seifert, 2005). The Ryff PWB was developed by Carol Ryff, and is

based off of several theories to include the theory of Individuation by Carl Jung and the

theory of Self-Actualization by Abraham Maslow (Ryff, 1989).

Perceived Performance of ADL Tasks- This variable will be measured using the Groningen

Activity Restriction Scale (GARS) [appendix 3], a simple non-disease-specific 18

question survey used to measure disability and assess a person’s ability to perform

activities of daily living (ADL) (Suurmeijer, 1994). ADLs are basic daily tasks such as

get in and out of bed, dress oneself completely, prepare lunch, and go up and down stairs.

The questions are rated on a scale of 1-3, with 3 indicating the highest ability to perform

the task.

Symptoms of Depression- This variable will be measured using the Hospital Anxiety and

Depression Scale (HADS) [appendix 4] to evaluate the participant’s level of anxiety and

depression prior to the start of the intervention. The HADS scale has fourteen items that

evenly ask about anxiety and depression by presenting a situation to respond to through

phrases like “occasionally, not at all, definitely, hardly at all” that are coded to be

represented by a numeric code 0-1-2-3 for easier data analysis (Zigmond & Snaith, 1983).

Social Support and Social Connectedness- This variable will be measured using the SSL12-I, the

shortened version of the Social Support List--Interaction survey tool, which has been

developed to be used specifically with the older adult population. The scale was primarily

developed to assess the extent of which an individual receives social support through a

variety of social interactions (Kempen & Eijk, 1995). The SSL 12-I has three subscales:


everyday social support, social support in problem situations, and esteem support,

measured using an abbreviated four choice Likert scale (Kempen & Eijk, 1995).

All five of the proposed measurement tools have been assessed and found that they are

satisfactory instruments for evaluating different constructs of health in older adults: perceived

general health (Brzyski & Knurowski, 2003), self-rated overall life satisfaction (Springer, 2006),

perceived performance of ADL tasks (Suurmeijer, 1994), symptoms of depression (Bjelland,

Dahl, Haug, & Neckelmann, 2002), and social support and social connectedness (Brzyski &

Knurowski, 2005).

Being that all five assessment tools are quantitative in nature, the data will be assessed

using the Statistical Analysis System (SAS). The data derived from all the measurement tools

will be analyzed through summary statistics (mean, standard deviation, kurtosis) and ANOVA

regression analyses to predict whether the A Matter of Balance: Volunteer Lay Leader

intervention sessions promote psychological/psycho-social wellness. The data will be prepared

for analysis by coding the responses, for example: Strongly agree, agree, neutral, disagree,

strongly disagree. Strongly disagree = 1, disagree = 2, neutral = 3, agree = 4, strongly agree = 5.

For the responses like “only sometimes, and hardly never”, they too will be coded respectful to

the 1-5 scale. Some of the data from the Likert scales will be reduced to the nominal level by

combining all agree and disagree responses into two categories of "accept" and "reject" and

analyzed using the Chi-Square test.

Comment [EA4]: Excellent details of tools


Evaluation Logic Model

Comment [EA5]: Need to flip the formatting on this page (add section breaks)


Dissemination and Stakeholder Utilization Plan

I have identified Texas Health Harris Methodist Hospital Fort Worth (THR) as my main

stakeholder, but have also recognized the Area Agency on Aging, the Senior Citizen Services of

Tarrant County and the United Way of Tarrant County as additional stakeholders due to the

nature of their organizations and their services targeted towards the older adult community. The

organizations work in conjunction with each other and support each other’s efforts to penetrate

the entire county with amenities and services for the 65 and older population. The stakeholders

that will be involved in the outcome evaluation will be doing so with the understanding that there

will be minimal involvement on their part for the duration of the evaluation. I have decided to

perform this outcome evaluation from the stance of the external evaluator, with no-long term or

on-going position with the AMOB: VLL program offered through THR. Being that I am

interested in the secondary effects of the program, the actual program components will not be

changed or altered during my evaluation time frame, allowing the stakeholders to continue

providing the AMOB: VLL classes like normal.

Prior to the start of the evaluation, the aforementioned stakeholders will be arraigned in a

meeting to outline their involvement during the one year evaluation period and will sign an

evaluation contract agreeing to allow my evaluation team to be a part of their AMOB: VLL

programs. They will also agree to a formal follow-up meeting to discuss the results of the

evaluation. It will be a “lunch and learn” type of gathering with the findings being reported in a

way that promotes knowledge creation and awareness of the probable additional benefits of the

falls prevention program. The information can be used by the stakeholders, if viable, to combat

social isolation and promote social connectedness through participation while also combating

falls in older adults, essentially getting two interventions in one class room setting.

Comment [EA6]: I find that organizations are more responsive to participating in evaluation using letters of commitment rather than legal contracts


Data Visualizations

Figure 2- An illustrative example of the HADS evaluation tool in assessing depression and anxiety in older adults prior to initiating the AMOB: VLL intervention program. Created in Tableau.

Figure 1- Example data visualization of real Tarrant County population growth trends showing the increased need for awareness of health outcomes of older adults in Tarrant County. Created in Tableau.

Comment [EA7]: Tiny fonts on graph are hard to read. I’d like to see graphs with the pre/post data that you’ve proposed in your evaluation. These appear to be all needs assessment oriented.


Figure 3 - An illustrative example of real falls frequency data from THR's Trauma department to target at risk areas for delivery of AMOB: VLL program. Created in Tableau


Evaluation Reflection

My personal tactic to the evaluation mostly mimics the approach to research as Donald

Campbell. Campbell’s alternative method to the “true experiment” gave me the perfect way to

accomplish my outcome evaluation goals without having to follow the rigor of the experimental

design (Alkin, 2013). The quasi-experimental gives me the opportunity to explore a more fluid

approach to the evaluation and not focus so much on ensuring randomization and methodological

rigor (Patton, 2012). I am able to utilize method appropriateness and exploit a comparison group

to analyze my target variables instead of a control group (Patton, 2012). I appreciate Campbell’s

approach to decision making through the inclusion of various stakeholders to engage in a

collaborative effort that employed data and experimentation, while not restricted by the true

experimental design (Alkin, 2013).

As for my research paradigm, I have found that I most closely align with the post-

positivist model in the way that I believe that the researcher conducting the evaluation should be

“distant and objective”, provide a service and not be swayed by the organization being evaluated

(Alkin, 2013). I like to follow a scientific method when conducting my research, but like

mentioned above, I am like Campbell, a scientific realist (McKelvey, 1999). I employed surveys

as my main measurement tool because I relate to the post-positivist ideal that reality can be

measured and should be measured only if the validity is proven (Alkin, 2013).

Overall, evaluation is a critical piece in maintaining the efficacy and effectiveness of any

health intervention. With the movement towards evidence-based programs as the gold standard,

evaluation will take the front seat in ensuring that those programs are achieving their desired

goals. As public health professionals, we need to be aware of the power of evaluation in attaining

said goals.


References Aging, N. C. (2014). Falls Prevention Fact Sheet. Washington DC: NCOA.

Albert, S., & Freedman, V. (2010). Public Health and Aging. New York: Springer Publishing

Complany, LLC.

Alkin, M. (2013). Evaluation Roots A Wider Perspective of Theorists' Views and Influences.

SAGE Publications, Inc.

Arfken, C. (1994). The prevalence and correlates of fear in falling in elderly persons living in the

community. American Journal of Public Health, 84:4 (565-570).

Bjelland, I., Dahl, A., Haug, T., & Neckelmann, D. (2002). The validity of the Hospital Anxiety

and Depression Scale. An updated literature review. Journal of Psychosomatic Research,

52(2): 69-77.

Brzyski, P., & Knurowski, T. (2003). Validity and reliability of Short Form General Health

Survey (SF-20) in population of elderly people. Przegl Epidemiol, 57(4): 693-702.

Brzyski, P., & Knurowski, T. (2005). Validity and reliability of Social Support Interactions Scale

SSL12-I in a population of elderly people in Poland. Przegl Epidemiol, 59(1): 135-145.

Heinrich. (2010). Cost of falls in old age: systematic review. Osteoporosis International, 21:891-

902.

Kempen, G. (1995). The psychometric properties of the SSL12-I, a short scale for measuring

social support in the elderly. Social Indicators Research, 35(3): 303-312.

Kempen, G., & Eijk, L. V. (1995). The psychometric properties of the SSL12-I, a short scale for

measuring social support in the elderly. Social Indicators Research, 35(3): 303-312.

Lach, H. (2002). Fear of Falling: An Emerging Public Health Problem. Generations, 3:33-37.

Lindemare, L. (2014). Senior Citizen Services of Greater Tarrant County, Inc. A Matter of

Balance.

McKelvey, B. (1999). Toward a Campbellian Realist Organization Science. Variations in

Organization Science: In Honor of Donald T. Campbell, 383-411.

Patton, M. Q. (2012). Essentials of Utilization Focused Evaluation. SAGE Publications, Inc.

Ryff, C. (1989). Happiness is everything, or is it? Explorations on the meaning of psychological

well-being. Journal of Personality and Social Psychology, 57(6): 1069-1081.

Seifert, T. A. (2005). The Ryff Scales of Psychological Well-Being. University of Iowa.


Smith et al., M. L. (2012). Personal and delivery site characteristics associated with intervention

dosage in an evidence based fall risk reduction program for older adults. Translational

Behavioral Medicine, 188-198.

Smith, M. (2012). Falls efficacy among older adults enrolled in an evidence-based program to

reduce fall-related risk. Family and Community Health, 256-262.

Smith, M. L. (2010). Successful falls prevention programming for older adults in Texas: rural

urban variations. Journal of Applied Gerontology, 1-27.

Springer, K. (2006). An assessment of the construct validity of Ryff’s Scales of Psychological

Well-Being: Method, mode, and measurement effects. Social Science Research, 35(4):

1080-1102.

Suurmeijer, T. (1994). The Groningen Activity Restriction Scale for measuring disability: Its

utility in international comparisons. American Journal of Public Health, 84: 1270-1273.

TAMHSC. (2009). Addressing the Needs of Older Adults in Tarrant County. Fort Worth TX:

United Way of Tarrant County. Retrieved from United Way of Tarrant County:

http://www.unitedwaytarrant.org/sites/default/files/UWTC_OlderAdults_FINAL.pdf

Tennstedt et al., S. (1998). A randomized control trial of a group intervention to reduce fear of

falling and associated activity restriction in older adults. The Journals of Gerontology

Series B, Psychological Sciences and Social Sciences, 53(6), 384-392.

Tinetti, M. (1994). a mutlifactorial intervention to reduce the risk of falling among elderly people

living in the community. The New England Journal of Medicine, 331:921-827.

Visser, G. (2000). Lesser status, less aid? Socio-economic differences in networking support for

the elderly in the Dutch society. Aging in the Netherlands, 137-146.

Ware, J. E. (2002). SF Tools. Retrieved from SF-36 Health Survey Update: http://www.sf-

36.org/tools/sf36.shtml

Zigmond, A., & Snaith, R. (1983). The hospital anxiety and depression scale. Acta Psychiatrica

Scandinavica, 67(6): 361-370.


Appendix

Appendix 1

First page of the MOS-SF 36 question assessment tool for health status


Appendix 2

First page of the Ryff Psychological Well-Being Scale assessment tool for the life satisfaction

rating


Appendix 3

First page of GARS assessment tool for the disability and ADL performance measure.


Appendix 4

HADS assessment tool for the depression and anxiety measure.

Appendix 5

SSL-12 I assessment tool for the Social Support and Social Connectedness measure.



Grading Criteria (Points may be deducted in general for failure to follow assignment instructions.)

Your Grade

The intervention is clearly described such that the average person can understand how and why it was chosen to address a community concern. Why was the intervention chosen? Why is this evaluation needed?

10/10

The intended results of the program are clearly described and evidenced by the goal statement, objectives and evaluation questions and/or hypotheses.

10/10

Baseline data demonstrates a community need and a strong understanding of the target population.

10/10

The proposed methods are detailed and specific. 10/10

The proposed methods are feasible and reasonable for a community program.

8.5/10

The proposed methods are sufficient to answer research questions, follow the assignment instructions, and are appropriate for the overall evaluation design.

10/10

The logic model displays the entire program and evaluation design in a user-friendly and visually pleasing manner with logical links between components.

9/10

The dissemination and utilization plan identifies key stakeholder groups and describes their utilization needs and dissemination methods that will best meet those needs.

4/5

The sample charts and graphs are easy to read and understand and use an appropriate format for the type of data analyzed.

8.5/10

The evaluator statement demonstrates a clear understanding of differences in evaluation models and critical reflection on matching the evaluation model to the service-learning situation. The statement also demonstrates self-reflection regarding skills and design choices.

5/5

TOTAL __85_/90

Writing Adjustment: Writing Quality Adjustment ___ Excellent, free of errors, concisely written, well organized 0 Satisfactory ___ Contains some errors or awkward wording

___ Needs significant editing and restructuring

Final Score: Grade: 94% A

Comment [EA8]: It may be challenging to engage community stakeholders around the additional data collection sessions, contracts, etc…

Comment [EA9]: Good charts, but as I discussed in class, it is more important to demonstrate your ability to work with the design of your evaluation- which in your case is a pre/post analysis

Documents

AMOB Evaluation Paper