Upload
malcolm-wood
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
davidr:davidr:
C-8 Moving Toward Excellence:C-8 Moving Toward Excellence:Integrating Performance and Integrating Performance and
Outcome MeasurementsOutcome Measurements
Jill Pfitzenmayer, Ph.D.Child & Family Services, Newport, RI David Robinson, Ed.D.Center for Evaluation and Research with Children and Adolescents, Massachusetts Society for the Prevention of Cruelty to Children, Boston, MA
Presentation ObjectivesPresentation Objectives
11. . Participants will learn how to identify program goals, objectives and specific program activities toward the accomplishment of program mission, vision and values.
2. 2. Participants will obtain information about establishing an organizational structure that supports critical analysis of program performance and quality improvement.
3. 3. Participants will identify specific outcome indicators, measurement tools and data collection methods relevant to their program plans.
Presentation ObjectivesPresentation Objectives
4. 4. Participants will learn how to link staff training, employee retention, program performance and outcome measurement efforts.
5. 5. Participants will be able to understand when internal outcome assessment can be enhanced by the introduction of an outside evaluator as consultant or primary investigator.
• Conceptual framework
•Description of the identified needs and problems
• Discussion of the methodology and approaches used to address the systems issues,
• Review of specific agency initiatives and reporting structures developed to enhance organizational quality, and
• Discussion by an outside evaluator familiar with the agency who was brought in for consultation
AgendaAgenda
DefinitionsDefinitions
Performance MeasurementPerformance Measurement – “Ongoing monitoring and reporting of program accomplishments, particularly progress towards preestablished goals.”*
• type or level of program activities (process)
• direct products and services delivered (outputs)
• results of products and services (outcomes)
Program EvaluationProgram Evaluation – “Individual systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working.”*
• internal or external experts and program managers
• learn benefits of program or how to improve it
* GAO, April 1998
Integrating Performance Measurement and Integrating Performance Measurement and Program Evaluation: A Conceptual FrameworkProgram Evaluation: A Conceptual Framework
• LinkedLinked to institutional mission, goals, objectives, strategic plan, process and outcome
evaluation
• SMARTSMART - SSimple, MMinimum paper, AActively used, RRevised, and TTransferable to new programs
• Best PracticesBest Practices – continuously updated incorporating new findings, technology and resources
• ComprehensiveComprehensive - Quantity, Quality, Effort and Effect
Outcome Evaluation* Needs Assessment Formative and Process* *An understanding of Evaluation Mission, Problem/Needs and Goals is also needed
Logic Model, Evaluation Questions and Types of Evaluation
Goals Objectives Inputs Program Services + Activities
Outputs
Outcomes
Mission Problem Needs Immediate
Short-term
Long-term
Program EvaluationProgram Evaluation
Performance Measurement SystemPerformance Measurement System
1.1. Gilbert Performance Engineering Approach
• Develop a MODELMODEL of process, department, program
• Define INDICATORSINDICATORS of valued results and expectations
• Take MEASURESMEASURES to see how well process is working
• Implement METHODSMETHODS of improvement and evaluate results
Performance Measurement SystemPerformance Measurement System
2.2. Friedman Four Quadrant Approach
HOW MUCH SERVICE DID WE DELIVER?
HOW WELL DID WE DELIVER SERVICE?
HOW MUCH EFFECT/CHANGE DID WE
PRODUCE?
WHAT QUALITY OF EFFECT/CHANGE DID WE
PRODUCE?
INPUTINPUT
OUTPUTOUTPUT
QUANTITYQUANTITY QUALITYQUALITY
Effective Organizational Performance Effective Organizational Performance Measurement SystemsMeasurement Systems
• Commitment Commitment – Leadership team committed to measurement
• ClarityClarity – Degree of clear strategies and metrics
• MetricsMetrics – Indicators measure valuable processes and expectations
• AlignmentAlignment – Performance measures aligned with key people processes and structures
• InvolvementInvolvement – Key stakeholders involved in defining, tracking, and ongoing review of quality performance improvement system
• Multi-service organization over 136 years old. CEO been in office for 22 years; powerful Board of Directors
•Primary services:
•Nine child & adolescent residential sites•Two child care centers •Family counseling program•Home-based program•School-based programs •Transitional housing program
Information about CFSInformation about CFS
•Minimal COA requirements:
Chart reviewCustomer satisfactionUtilization review in select areasIndividual outcome dataIndication of systematic review of agency
Implementation of Continuous Quality Implementation of Continuous Quality Improvement (CQI) ProgramImprovement (CQI) Program
Problem 1:Problem 1: disconnect between accrediting body and agency leadership perceptions of need
Problem 2:Problem 2: global discomfort with/ suspicion of “science” and “data”
Problem 3:Problem 3: Continuous Quality Improvement is supposed to involve all levels of staff, not be a top-down process
Implementation Model: ChallengesImplementation Model: Challenges
1.1. Assemble committee representing key constituents of the agency:
Agency leadershipInformation systemsHuman ResourcesBusiness OfficeProgram staffParentConsumer
Steps Required to Develop an Outcome System Steps Required to Develop an Outcome System Year 1: Year 1:
2. Develop a conceptual model to help staff review program activities and outcomes (e.g., logic models)
3. Map out organizational structure and evaluation activities (e.g., department teams)
4. Identify strategy that makes sense to staff to collect, analyze and use information
Steps Required to Develop an Outcome System Steps Required to Develop an Outcome System Year 1: Year 1:
5. Develop protocol and process for all levels of staff participation
6. Develop tools for reporting program activities at least quarterly and identify survey instruments
7. Train staff on recommended model and process
Steps Required to Develop an Outcome System Steps Required to Develop an Outcome System
Year 1:Year 1:
Identify a Solution-Focused ProcessIdentify a Solution-Focused Process
• PLAN
• Decide what is the problem
• Brainstorm solutions
• DO
• Enact solutions
• Collect data on how solutions are working
• ACT• Revise action plan if needed• Discontinue plan• Continue to monitor
• CHECK
• Analyze data
• Discuss findings
CQI Activities Related to COA CQI Activities Related to COA StandardsStandards
C&FS develops long-term strategic plan (4 yrs)
(G2.3)Identified stakeholders
participate in all appropriate
activities (G2.2)
Each dept. develops annual plan of goals and objectives (G2.4)
Internal & externalaudits &
reviews (G2.10, G2.5)
Chart Reviews
(G2.6, G2.1)
Customer satisfaction
surveys(G2.8, G2.10)
OutcomesMeasures
(G2.7, G2.10)
Quarterly review of each dept.achievements
by agency CQI Team(G2.9)
Annual reportsof year-end progress(G2.9)
Reassessment,as necessary (G2.11)
Annual plan Quarterly Monitoring summary Summaries of client/referral satisfaction
forms Summaries of incidents that occurred
over the quarter Summary of external review activities Review department-specific and
aggregated findings
Flow of CQI ActivitiesFlow of CQI Activities
Year 2: Challenges to CQI SystemYear 2: Challenges to CQI System
• Many staff did not understand logic models
• Program Directors didn’t understand basic concepts (e.g., difference between benchmark and baseline data)
• Challenges existed around grasping conceptual material and managing task of putting information on computer!
Year 2: Challenges to CQI SystemYear 2: Challenges to CQI System
• Some programs never tracked outcome data at all
• State-funded programs required their own outcome tools that were either cumbersome or lacked validity
• Staff lacked buy-in, experienced turnover or couldn’t make use of findings
• Several programs had a hard time identifying individual outcome indicators
Year 2: Challenges to an Outcome SystemYear 2: Challenges to an Outcome System
• Mechanically tracking data and outcomes became a challenge—no standardized data base in the agency
• Changes implemented as a result of outcome data were often not documented
• Staff were “busy” but couldn’t document real client change (i.e., conflating outputs and outcomes)
Year 2: Small Successes for an Outcome SystemYear 2: Small Successes for an Outcome System
• Some programs used the process to study and implement real change
Residential Network: system for getting paperwork accomplished
Family Counseling: saw change in patient population via use of SCL-90-R
School-based: outcomes supported pilot program around prevention education
Creating Organizational ChangeCreating Organizational Change
• Year 2:Year 2: Agency/Board leadership in strategic planning meetings, which required a SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis of programs
• Lack of careful investigation of program outcomes forced leadership to rely more on fiscal and output information than on outcome data
• Frustration Innovation
Creating Organizational ChangeCreating Organizational Change
• Continuous Quality Improvement
• Reactive• Minimal Board
involvement• Driven by COA
requirements• Voluntary
management participation
• Organizational Quality & Performance
• Proactive
• Board subcommittee
• Driven by strategic planning process
• Sr VP assigned to job
Coordination of OQP ActivitiesCoordination of OQP Activities
Agency Strategic Plan
Organizational Quality & PerformanceAssess performance and outcomes
to align agency goals and program activities
Learning InstituteLink staff needs with training and
learning activities
Employer of Choice InitiativesMaintain focus on performance
to attract and retain highest quality staff
Moving Toward Excellence: Moving Toward Excellence: Changes in Culture and SystemsChanges in Culture and Systems
• Staff are now required to identify steps in developing programs reflecting excellence and best practices
• Management are held accountable for process and program outcomes
• Systematic data collection and analysis requires new MIS system
• Agency-wide committees and initiatives brought into oversight process
Moving Toward Excellence: Moving Toward Excellence: Changes in Culture and SystemsChanges in Culture and Systems
• New OQP committee members include Corporate Communications and Development in order to ensure inter-departmental coordination
• Methodology for tracking process and outcomes in place
• OQP SVP meets provides technical assistance in program plan development
Moving Toward Excellence: Moving Toward Excellence: Goals of OQP ProcessGoals of OQP Process
• Department-specific process and outcome data will be collected, analyzed and acted upon
• Inter-departmental coordination will improve
– Corporate Com, MIS & Development included in program planning
– Programs work together on mutual or complimentary projects
– Inter-program conflicts will be quickly identified and resolved
Moving Toward Excellence: Moving Toward Excellence: Goals of OQP ProcessGoals of OQP Process
• Communication across departments and up, down chain of command will improve
• Agency leadership will have improved and speedier methods of determining where there are problems and strengths in programs
• Agency and program leaders have tools to continuously plan for the future
CFS Program Performance PlanCFS Program Performance Plan
Initial
Program Goals
(1)
Accomplishments
(2)
Current Status of Activities
(3)
Additional Activity Required
(4)
In order to accomplish our Program and
Performance Goals, we need to have the following
in place
This is how we define the components of Initial
Program Goals
These are the program activities we are currently
engaged in or have completed to reflect our
intended Accomplishments
These are the program activities that must be
completed to reflect our intended
Accomplishments with expected date (s) of
completion
•CFS has developed a framework for enabling staff to see patterns and interrelationships and we are able to learn from our own experiences by reviewing successes and failures systematically.
•Staff are taught and encouraged to employ the ability to deconstruct complex tasks while maintaining awareness of the connection to a larger whole.
•Staff meetings are held regularly to communicate to and training staff on current policies and practices.
•Staff are encouraged by supervisors to voice opinions about policies and procedures.
•Interdepartmental staff meetings should be held with regularity to ensure smooth operations and mutual problem-solving.
•Use of critical thinking skills should be modeled by supervisors.
CFS Program Outcome PlanCFS Program Outcome Plan
Outcomes
(5)
Outcomes Quantified
(6)
Date Sources
(7)
Measures
(8)
Baseline/
Benchmark Data
(9)
These are the observable,
measurable and quantifiable
changes we can expect if we meet
our Program Goals
This is how we will quantify our
expected program outcomes
Where we will obtain
information to measure Program
Outcomes
Instruments or measurement
tools we will use to assess Outcomes
Information to help assess where we are starting (baseline) or comparative markers
(benchmark)
•Staff will be skilled at acquiring knowledge
•75% of staff will report that they participated in at least one activity in which they acquired knowledge.
•Staff self-reports •Learning self-report
•None to report.
CFS Quarterly Program Performance ReportCFS Quarterly Program Performance Report
Initial
Program Goals
(10)
Additional Activity Required
(11)
Progress Made on Additional Activities
(12)
Comment/Analysis on Progress
(13)
As stated in the Program Performance plan from
column (1)
As stated in the Program Performance plan from
column (4)
Status of progress toward Additional Activities
Required as stated in the Performance Plan
Note any difficulties or challenges that are interfering with the
progress being made toward completion of
each task
•CFS has developed a framework for enabling staff to see patterns and interrelationships and we are able to learn from our own experiences by reviewing successes and failures systematically.
•Interdepartmental staff meetings should be held with regularity to ensure smooth operations and mutual problem-solving.
•Use of critical thinking skills should be modeled by supervisors.
•Five interdepartmental staff meetings have been held since Sept 03.
•Supervisors received critical thinking training in Aug 03 and have started modeling skills.
•Supervisors find challenges integrating multidisciplinary staff issues in meetings. Additional training required and will be completed by 1/1/04
CFS Quarterly Program Outcome ReportCFS Quarterly Program Outcome Report
Expected Outcomes
(14)
Baseline or Benchmark
Data
(15)
Results:
•Sample Size
•Findings
(16)
Analysis/
Conclusions
(17)
Future Actions:
•By Whom
•Activity
•Time Frame
(18)
As stated in the Program Outcome
plan from
Column (5)
As stated in Program Outcome plan from column
(9)
Sample size and data results from
measures and data sources
Discussion of findings;
comparison of this quarter’s findings to
baseline/
benchmarks
State what further action will be taken,
by whom, and expected date of
completion
•75% of staff will report that they participated in at least one activity in which they acquired knowledge.
None to report •100 staff surveyed•66% reported participation
•Some progress made toward staff participation in knowledge acquisition; staff report challenges in finding time to learn outside of job.
•By Director: provide two hours/month for staff development activities. Implement no later than 12/03.
Moving Toward Excellence: Moving Toward Excellence: How Do Program Staff Use OQP System?How Do Program Staff Use OQP System?
• Program Directors use process to bring their vision into program planning
• Staff are oriented to OQP performance and outcome plans upon hire as a learning tool
• Program leaders use plans to maintain an eye on program improvement activities
Moving Toward Excellence: Moving Toward Excellence: Lessons LearnedLessons Learned
• Appreciate what you bring to the process from your own discipline and theoretical orientation and how they inform your outcome assessment work
• Ask for help. Use volunteers, experts, etc.
• Get organized and don’t panic
• Don’t underestimate the value of communication
• Maintain a stance of being inquisitive, not judgmental
• Remind yourself and your staff that the process is developmental
• Aim for direction, not perfection
“I wanted to share with you my excitement of having completed the baseline ECERS and ITERS for child care today!…With this data we can now create our baseline and use it to establish our movement forward to improved overall quality in our Centers for the children, families and our staff! This is so exciting. Good things are in our future, along with much hard work…I am confident that I will lead the way to our success!”
Patty Shelley, Child Care Program Director
Genuine Unsolicited Testimony from Staff:Genuine Unsolicited Testimony from Staff:
• Commitment Commitment – Leadership team committed to measurement
• ClarityClarity – Degree of clear strategies and metrics
• MetricsMetrics – Indicators measure valuable processes and expectations
• AlignmentAlignment – Performance measures aligned with key people processes and structures
• InvolvementInvolvement – Key stakeholders involved in defining, tracking, and ongoing review of quality performance improvement system
ASSESSMENT: ASSESSMENT:
Case Study of CQI & Performance Measurement System Case Study of CQI & Performance Measurement System
SCORE = SCORE =
Rating Scale: 1 = A Little; 2 = Just Enough; 3 = SuccessRating Scale: 1 = A Little; 2 = Just Enough; 3 = Success
Primary Presenter: Jill Pfitzenmayer, Ph.D.Title: Senior Vice President, Organizational Quality and PerformanceAgency: Child & Family ServicesAddress: 24 School Street, Newport, RI 02840Phone: 401-848-4186Fax: 401-841-8841Email: [email protected] Co-Presenter: David Robinson, Ed.D.Title: Director for the Center for Evaluation and Research with Children and Adolescents (CERCA)Agency: Massachusetts Society for the Prevention of Cruelty to ChildrenAddress: 399 Boylston Street, Boston, MA 02116Phone: 617-587-1594Fax: 617-587-1582Email: [email protected] or [email protected]: www.mspcc.org
CONTACT INFORMATIONCONTACT INFORMATION