View
218
Download
2
Category
Tags:
Preview:
Citation preview
Conciliating scientific rigor & pragmatics in Outcome Evaluation Research for human
service professionalsTheoretical underpinnings and implementation techniques
Network for Health & Welfare Studies
Dr. Charles C. ChanDr. Amy P. Y. HoMr. Kevin Chan
February 25, 2004
ConvenorNetwork for Health & Welfare Studies
Associate ProfessorDepartment of Applied Social Sciences
Dr. Charles C. Chan
Conciliating scientific rigor & pragmatics in OER
Preamble
• Applied research can be of better service to the human service industry, especially in a quality conscious era, if it promises to pay balanced attention to the dialectical relationship between the professional / researcher and the service recipient.
Conciliating scientific rigor & pragmatics in OER
Quality and Effectiveness :
• Strange bed-fellows or estranged couples?
Conciliating scientific rigor & pragmatics in OER
• The overall all goal of placing priority on both quality and effectiveness of health programs is to document the synergistic contribution of its component parts to a holistic improvement in health and welfare of the population.
WHO-EURO Working Group on Health Promotion Evaluation (1998)
Quality and Effectiveness :
Conciliating scientific rigor & pragmatics in OER
• The focus should not be on documenting quality or evaluating effectiveness of isolated interventions as end points, but rather on the relationship of a given intervention to the other components of the health promotion strategy. Such an analysis may indeed provide a fresh approach to the very issue of quality.
Quality and Effectiveness :
Conciliating scientific rigor & pragmatics in OER
• BARD is a tested approach in applied research to address the demands of scholarship of application and community benefits
• It recognizes problems central to applied research
• It contributes by directing decision making in the research process
Definition of the Balanced Applied Research Development Model (BARD):
Conciliating scientific rigor & pragmatics in OER
1. Placing value on a balanced approach to the scientific requirements and the pragmatics of outcome evaluation in human services
Main features of BARD:
Conciliating scientific rigor & pragmatics in OER
2. Emphasizing integrating research into practice by transferring skills in data capturing to professionals or lay persons in service delivery
Main features of BARD:
Conciliating scientific rigor & pragmatics in OER
3. Analyzing as much as possible, the triadic relationship between the professional, lay person, and service recipient and sustaining intervention effects through supportive supervision
Main features of BARD:
Conciliating scientific rigor & pragmatics in OER
Efficacy Vs Effectiveness
• Efficacy – Intervention does more good than harm under optimum conditions
• Effectiveness - Intervention does more good than harm under real-world conditions
(Flay, 1986)
Conciliating scientific rigor & pragmatics in OER
The RE-AIM framework (Glasgow, 1999)
• Reach
• Efficacy or Effectiveness
• Adoption
• Implementation
• Maintenance and cost
Conciliating scientific rigor & pragmatics in OER
RE-AIM issue Efficacy studies Effectiveness studies
Reach Homogenous Heterogeneous
Efficacy or Effectiveness Very specialized & standardized; randomized
Broad intervention; not necessarily standardized nor randomized
Adoption Run in controlled experimental setting
Run in real-world setting with little control
Implementation Run rigorously by well-trained research staff
Run by staffs in service setting with competing demands
Maintenance and cost Less interest on maintenance and cost
Major concerns in real settings
The RE-AIM framework (Glasgow, 1999) - Cont’d
Conciliating scientific rigor & pragmatics in OER
Outcome Evaluation Research in Hong Kong – Financial Landscape
• Health Services Research Fund (1995 - 2003)
• Health and Health Services Research Fund (2003 - present)
• Research Fund on Control for Infectious Diseases (2003 – present)
• Community Investment & Inclusion Fund (2003 - present)
Conciliating scientific rigor & pragmatics in OER
• Capital amount of $55M
• Total number of submission: 1096
• Total number of approved HSRC projects=224
Health and Services Research Fund (1995-2003)
Conciliating scientific rigor & pragmatics in OER
Health and Health Services Research Fund (2003 - present )
•Capital amount of $10M
•Total number of submission: 187
•Total number of approved projects=4 (ESGAA meeting in July 2003)
Conciliating scientific rigor & pragmatics in OER
Research Fund on Control for Infectious Diseases (2003 – present)
• Capital amount of $ 500M (50M given to PRC’s Ministry of Health for SARS research)
• Open call and commissioned research activities: tentative 30M to HKU in basic and epidemiological modeling research & 25 M to CUHK on public health and emerging infectious diseases drug and treatment development research
• Leaving sums for the establishment of the Centre for Health Protection
Conciliating scientific rigor & pragmatics in OER
Community Investment & Inclusion Fund - Funding information
Project No. Name of Project Name of Applicant OrganizationGrant Approved
0015-01 Helping Others - Self Growing ProjectChristian Concern for the Homeless Association $579,600
0023-01Project to Promote Family Cohesion & More Colourful Family Life
Pok Oi Hospital Madam Chu Kok King Child Care Centre $189,630
0069-01Collaborating People in Tsuen Wan Rural Areas The Yuen Yuen Institute $750,910
0070-01'Healthy Mothers - to-be' - A Women and Community Networking Project
Kwun Tong Methodist Social Service Grassroots Occupational Service Unit ' Healthy Mothers - to-be' Club $232,040
0079-01 Sunny Community Program
Hong Kong Outlying Islands Women's Association, Jockey Club Women's Integrated Service Centre $1,045,800
0091-01It's All Because of You: Community Cohesion Project Richmond Fellowship of Hong Kong $209,160
Conciliating scientific rigor & pragmatics in OER
Community Investment & Inclusion Fund - Funding information -Cont’d
Project No. Name of Project Name of Applicant OrganizationGrant Approved
0092-01Space of Dream - Youth Employment Project in the North District
The Evangelical Lutheran Church of Hong Kong, North District Integrated Youth Service Centre $1,107,700
0100/0101-01 "Caring Estates" in Southern District
HK Southern District Women's Association Ltd and Aberdeen Kaifong Welfare Association Social Service Centre $2,042,800
0113-01Community Care & Networking Programme for "Po Tin" Residents Urban Peacemaker Evangelistic Fellowship $1,005,300
0127-01Cross Generational Community Integration Programme
SAGE Eastern District Multi-service Centre for the Elderly $264,600
0133/0134-01
Retired Senior Volunteers' Association - Community Mutual Help Network Project St James' Settlement $954,740
0151-01Family Service Ambassador Scheme, Caritas Family Service Caritas - Hong Kong Family Service $599,700
Total $8,981,980
Dr. Amy P. Y. Ho
Member
Network for Health & Welfare Studies
Senior Lecturer
Department of Applied Social Sciences
Conciliating scientific rigor & pragmatics in OER
OER in human service setting – An example
• “Promoting health and well-being of elderly patients with chronic illness: A coordinated medical and social service program”
• Funded by the Health Care & Promotion Fund
Conciliating scientific rigor & pragmatics in OER
Recruitment of participants in participating hospital and obtain their consent to join the programme
Experimental group will receive 1) Comprehensive Discharge Planning and need assessment by Medical
Officers of participating hospital and Social Workers of NGO 2) Baseline Assessment and re-assessment at 3 months interval by
researcher of HKPU 3) Coordinated package of medical and social services provided by the
project
Participating NGO
-Provide tailor-made psychosocial
service using case management
approach
Regular Meetings and Case-conferences to
exchange information on participants
Participating hospital -Provide medical care
Case Manager to
coordinate all
social services
Home-based
services
Monthly home
visit by trained
volunteers
Community-based
services
Medical follow up at SOPD by the same Consultant Geriatrician & referrals to other medical services
Final Assessment / Evaluation of Project Effectiveness by The Hong Kong Polytechnic University
Randomize the participants into experimental group and control group
Control group will receive
1) Baseline assessment and subsequent 3 months interval
re-assessment by researcher of HKPU
2) Routine medical and social services offered by HA &
NGOs in the community
Hong Kong Polytechnic
University
- Data collection and data
analysis
Referrals to
other social
services
Intervention flowchart
Conciliating scientific rigor & pragmatics in OER
Standardized Interventions Intervention Potent elements Indicators for treatment integrity 1. Comprehensive hospital
discharge planning (carry out by both the medical officers and social workers)
To ensure that the participants receive coordinated and comprehensive medical and social services
Home visit to the participants should be carried out within 2 weeks after discharge
Both the hospital and welfare agency have the same set of patient records
2. Special outpatient clinic service
Participants are seen by the same consultant geriatrican & the participant is accompanied by the same social worker to attend appointments
Progress reports of each participants are keep by the case-mangers (filled out by service providers of the welfare agency)
The consultant geriatricans will inform the case manager of any changes in medical services to the participants
3. Monthly home visit Carry out by a group of relatively stable, trained volunteers to ensure continuity and quality of service
Standardized ongoing training is provided to the volunteers (with training manuals)
Home visit report is submitted to the case manager after each visit
Bi-monthly meeting will be held to maintain the motivation and commitment of the volunteers
4. Progress monitoring by the same case manager
The progress and needs of the participants are closely monitored by the same case manager
The case manager will call monthly case conference (attended by the consultant geriatricans) to discuss the progress of the participants.
Individualized Interventions Additional, tailor-made medical and social services are provided by the hospital and the welfare agency
To provide additional support to participants who have changing needs
The additional services are recorded and participants’ situations are closely monitored
Intervention components
Conciliating scientific rigor & pragmatics in OER
Issues in unblinded study
• Feasibility of blinding in human service research
• Artifacts introduced by confounders in unblinded study
Conciliating scientific rigor & pragmatics in OER
Standardization of intervention
• Contrasting the notion of “experimental group” in the context of medical research and human services research
• “Black Box” phenomenon in health promotion research (Moore, 2003)
Conciliating scientific rigor & pragmatics in OER
Control on extraneous variables
• Between-group imbalance in the control of extraneous variables
• Incorporating extraneous variables in the analysis
Conciliating scientific rigor & pragmatics in OER
Data quality and reliability
• Staff turnover
• Artifact due to unblinded group assignment
• Reliability of self-reported outcomes
Conciliating scientific rigor & pragmatics in OER
Issues in data analysis
• Loss to follow-up
• Causes of participants attrition in human services OER
• Effect of attrition on data analysis
Mr. Kevin Chan
Research AssociateNetwork for Health & Welfare StudiesDepartment of Applied Social Sciences
Honorary Scientific OfficerThe Hong Kong Childhood Injury Prevention & Research Association
Conciliating scientific rigor & pragmatics in OER
Presumption of the experimentalist ideal (Smith, 1985)
• OER in human services are more than often violating the assumptions required in traditional RCT convention
• Such difference should not be viewed as a deviation from the conventional approach, but rather a calling for applied research with stronger ecological validity
Conciliating scientific rigor & pragmatics in OER
In fulfilling requirements set for RCT, we often undermine the following (Rolls, 1999):
• The difficulties associated with the random assignment• The ethical and administrative objections to
randomization (de_Raeve, 1994)• That inputs are rarely stable• That awareness of difference introduces bias• That it is almost impossible to exclude extraneous
variables• The research is unable to say why changes detected
have occurred (Newell, 1992)
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – OER in human service setting
• Dealing with less-than-perfect random assignment• Adjusting for unstable input in human services OER• Statistical control for unblinded studies• The inclusion of extraneous variables as covariates• Narrowing down to a closer approximation of causal
relationship
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Group assignment in OER for human service setting
• Balancing incentives for follow-up of control group• Intention-to-treat analysis• As-treated analysis• Complier-Average Causal Effect (CACE) analysis
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Group assignment in OER for human service setting – Cont’d
Balancing incentives for follow-up of control group
• Despite the lack of intervention, control group participants could be prescribed with “non-specific” intervention or substantial support to maintain their interests to comply with the data collection process
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Group assignment in OER for human service setting – Cont’d
Balancing incentives for follow-up of control group
“Non-specific” intervention
-E.g. Delivery of printed health promotion materials
Substantial support
-E.g. Safety device, health supplements
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Group assignment in OER for human service setting – Cont’d
Intention-to-treat analysis As-treated analysis
•Treating non-compliers / drop-outs with compliers as same
•Adhere to original group assignment but lead to bias estimate of treatment effect (i.e. magnitude of intervention “dragged down” by non-compliers / drop-outs
•Realigning participants group assignments according to actual intervention received or sought
•Used as a secondary analysis to contrast the IIT results
•Subject to selection bias in case of noncompliance (i.e. those longing for an integrated medico-social care package might be more active in maintaining physical and mental health)
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Group assignment in OER for human service setting – Cont’d
Complier-Average Causal Effect (CACE) analysis
(Angrist, 1996; Little, 1998)
• Expanding the conventional regression equation with a new term – compliance with intervention
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Group assignment in OER for human service setting – Cont’d
Complier-Average Causal Effect (CACE) analysis
(Angrist, 1996; Little, 1998)• Difference of evaluation outcome (e.g. score on SF-36,
probability of sustaining home injury) between compliers and non-compliers, with other missing information (e.g. covariate variables such as age, gender, health status, attitude toward the research question) adjusted and stratified by compliance category.
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Group assignment in OER for human service setting – Cont’d
Complier-Average Causal Effect (CACE) analysis
(Angrist, 1996; Little, 1998)• CACE accounts for potential interaction between
randomized group assignment & compliance and allows more accurate estimate of program effectiveness by:
i. Increasing number of “usable” cases
ii. “Subtract” the artifact of compliance that moderates the outcome variable
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Adjusting for unstable input in human services OER
• Explicit definition of intervention protocol
i. Behavioral interventionii. Substantial inputiii. Personnel inputiv. Temporal data (Date, time, duration, frequency)v. Define boundary of intervention
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Adjusting for unstable input in human services OER
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Adjusting for unstable input in human services OER
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Adjusting for unstable input in human services OER
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Adjusting for unstable input in human services OER
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Adjusting for unstable input in human services OER – Cont’d
Process analysis
• Process evaluation fills up the “Black Box” (Moore, 2003) left void in health promotion research and strength internal validity of the intervention under investigation.
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Adjusting for unstable input in human services OER – Cont’d
Process analysis• Process analysis identifies variance in protocol
implementation & participants’ receptiveness to the prescribed intervention. A process analysis covers
i. Count of activities attendedii. Perceived effectiveness of the interventioniii. Compliance to intervention protocoliv. Met and unmet needs related to the intervention
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Adjusting for unstable input in human services OER – Cont’d
Process analysis (Cont’d)• The information gathered from the process research
should be integrated into the outcome research, rather than setting aside as auxiliary document in the discussion of study results
Conciliating scientific rigor & pragmatics in OER
Troubleshooting to barriers – Quality Assurance in OER for human service setting
Quality assurance exercise
Organizational requirement
Staff involvement
Conciliating scientific rigor & pragmatics in OER
Maintenance and cost in OER
• Unanticipated effect of health promotion on social capital and cohesion (Raphael, 2000)
• Economic analysis (including Cost-effectiveness Analysis or Cost Benefit Analysis)
Conciliating scientific rigor & pragmatics in OER
Decision questions generated from the BARD Model
Q1. How can the range of problems be considered as part of public service (health, welfare, and education) priority?
Conciliating scientific rigor & pragmatics in OER
Decision questions generated from the BARD Model
Q2. What is the status of the efficacy studies on similar intervention programs in the international literature?
Conciliating scientific rigor & pragmatics in OER
Decision questions generated from the BARD Model
Q3. Who should do what and how much for whom before one can consider the intervention effective?
Conciliating scientific rigor & pragmatics in OER
Decision questions generated from the BARD Model
Q4. Should we vary the intensity of intervention (e.g. tiered multi-level program) to suit targeted service recipients?
Conciliating scientific rigor & pragmatics in OER
Decision questions generated from the BARD Model
Q5. How can staff training and documentation of quality manual help differentiate role and skills for professional and lay participation to ensure intervention integrity?
Conciliating scientific rigor & pragmatics in OER
Decision questions generated from the BARD Model
Q6. Is the technology transferred in the process of conducting OER sufficient to ensure human and financial sustainability?
Conciliating scientific rigor & pragmatics in OER
WHO-EURO Working Group on Health Promotion Evaluation (1998) pointed out this :
Evaluation help them find out what works and how; make decisions about future health promotion policy strategies and programs,….modify and improve current programs; justify policy choices; and increase impact and effectiveness (including cost-effectiveness) of health promotion initiatives.
Some final words for the Policy-makers…
Conciliating scientific rigor & pragmatics in OER
Major NHWS OER publicationsChan, C. C. (2000). Workbook on measurement & use of
outcomes in social services. Hong Kong: Hong Kong Polytechnic University.
Chan, C. C., Chow, C. B., & Chan, K. (2003). SARS and social cohesion in Hong Kong: A research agenda. Asian Pacific Journal of Social Work, manuscript in review.
Chan, C. C., Lui, W. S., Wan, D., & Yau, S. (2002). Evaluating Service Recipient Outcomes in Psychiatric Residential Services in Hong Kong. Research on Social Work Practice, 12(4), 570-581.
Chan, C. C., Lui, W. S., Wan, D., & Yau, S. (2002). A multilevel approach to supported employment outcomes in Schizophrenia. American Journal of Orthopsychiatry, manuscript submitted for publication.
Conciliating scientific rigor & pragmatics in OER
Major NHWS OER publicationsChan, C. C., Luis, B. P., Chow, C., Cheng, J. C. Y., Wong,
T. W., Chan, K., & Tsui, S. (2001). Validating narrative data on residential child injury. Journal of Safety Research, 37(4), 377-389.
Chan, C. C., Luis, B. P., Chow, C. B., Cheng, J. C. Y., Wong, T. W., & Chan, K. (2003). Unintentional residential child injury surveillance in Hong Kong. Journal of Paediatrics and Child Health, 39(6), 420-426.
Chan, C. C., Tsui, M. S., Chan, M. Y. C., & Hong, J. H. (2002). Applying the Structure of the Observed Learning Outcomes (SOLO) Taxonomy on Student's Learning Outcomes: An empirical study. Assessment & Evaluation in Higher Education, 27(6), 511-528.
Conciliating scientific rigor & pragmatics in OER
Major NHWS OER publicationsChan, C. C., Wan, D., & Lui, W. S. (2002). Determining
performance standards for supported employment service in Hong Kong. Hong Kong Journal of Social Work, 36(1 & 2), 95-111.
Cheung, F., Yau, A., & Wong, K. (2001, February 22, 2001). The Costs of Cost-Effectiveness: An initial review of the competitive bidding experience in contracting out home care service. Paper presented at the Seminar on Efficiency & Effectiveness of Social Care, Part III, The Hong Kong Polytechnic University, Hong Kong.
Chow, C. B., Cheung, W. L., & Chan, C. C. (2002, April 28-30, 2002). Transforming AEIS data for the prevention of unintentional childhood injuries in Hong Kong. Paper presented at the Hospital Authority Convention 2002, Hong Kong.
Conciliating scientific rigor & pragmatics in OER
Major NHWS OER publicationsHo, A. P. Y. (2001, May 19, 2001). Creating a supportive
environment for elderly with chronic illness. Paper presented at the Healthy Ageing Convention, Hong Kong, Hong Kong Convention and Exhibition Centre.
Chan, C. C., & Chan, K. (2003). Competence-based evaluation of volunteer progress. Evaluation and Program Planning, manuscript in review.
Network for Health & Welfare Studies, the Hong Kong Childhood Injury Prevention & Research Association. (2003). The Volunteer Home Visitation Programme – Training Manual
Network for Health & Welfare Studies, the Hong Kong Childhood Injury Prevention & Research Association. (2003). The Volunteer Home Visitation Programme – Quality Assurance Manual
Conciliating scientific rigor & pragmatics in OER
Key ReferencesAngrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of
Causal Effects Using Instrumental Variables. Journal of the American Statistical Association, 91(434), 444-455.
Flay, B. R. (1986). Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Preventive Medicine, 15(5), 451-474.
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why Don't We See More Translation of Health Promotion Research to Practice? Rethinking the Efficacy-to-Effectiveness Transition. American Journal of Public Health, 93(8), 1261-1267.
Little, R. J., & Yau, L. H. Y. (1998). Statistical techniques for analyzing data from prevention trials: Treatment of no-shows using Rubin's causal model. Psychological Methods, 3(2), 147-159.
McPherson, K., & Britton, A. (2001). Preferences and understanding their effects on health. Quality & Safety in Health Care, 10(Supplement 1), 61i-66.
Moore, L., Graham, A., & Diamond, I. (2003). On the feasibility of conducting randomised trials in education: case study of a sex education intervention. British Educational Research Journal, 29(5), 673-689.
Conciliating scientific rigor & pragmatics in OER
Key References
Newell, D. J. (1992). Randomised controlled trials in health care research. In J. Daly & I. G. McDonald & E. Willis (Eds.), Researching health care : designs, dilemmas, disciplines (pp. 47-61). New York: Routledge.
Raphael, D. (2000). The question of evidence in health promotion. Health Promotion International, 15(4), 355-367.
Rolls, L. (1999). The Challenge of Evidence-based Practice. In E. R. Perkins & I. Simnett & L. Wright (Eds.), Evidence-based health promotion (pp. 47-50). Chichester, New York: John Wiley.
Smith, G. (1985). Assessing health care : a study in organisational evaluation / Gilbert Smith and Caroline Cantley. Milton Keynes: Open University Press.
Thompson, B., Coronado, G., Snipes, S. A., & Puschel, K. (2003). Methodologic advances and ongoing challenges in designing community-based health promotion programs. Annual Review of Public Health, 24(1), 315-340.
Recommended