Performance Evaluation:From Grant Application and Federal Regulations to Program Performance Results

Carol Bergquist, Ph.D.Hannahville Indian CommunityProject VISIONS906-723-2580Carol.Bergquist@HICVisions.org

Michael Shoemaker, M.A., CRC, LVRC, CPMUtah State Office of Rehabilitation

CANAR Annual ConferenceSalt Lake City, Utah

November 2015

What is Program Evaluation?

• A Reflective and Systematic Process…• that focuses on questions of interest or

concern regarding activities, characteristics or outcomes of a program…

• where information is collected, analyzed and interpreted…• to assist in making data informed decisions.

Program Evaluation

• Program Evaluation is everybody’s business.• It is inquiry: an act of asking for information.• It should lead to ways to change and improve the

program.• It is likely something you are already doing and are not

aware of it. (improving client showing to appointments). The Truth: You may be the best evaluator because nobody knows your program better than you.There is a time and place for both internal and external evaluation.

Purposes of Program Evaluation

• Demonstrate program effectiveness to funders • Improve the implementation and effectiveness of programs• Better manage limited resources• Document program accomplishments• Justify program funding• Satisfy ethical responsibility to clients to demonstrate

positive and negative effects of program participation (accountability and transparency)

• Document program development and activities to help ensure successful replication

Program Evaluation is Part of Continuous Quality Improvement

Stewhart’s PDCA Cycle For Change

PROGAM EVALUATION MODEL

1. EVALUATION PLAN AND DESIGN

2. DATA COLLECTION

3. DATA ANALYSIS

4. RECOMMENDATIONS 5. IMPLEMENTATION

6. LEARN AND ADJ UST

PROGRAM EVALUATION

MODEL

Program Evaluation Activities

• Customer Satisfaction Surveys• Case File Reviews• Needs Assessments• Economic Impact Return on Investment• Process Evaluation• Authorizations and Payments Reviews• Impact and Outcome Evaluation

Performance Measures of Quality WIA WIOA

Entered Employment Rate

Employment Rate (Q2 post-exit)

Now Employed Q-2 after exit

Employment Retention Rate

Employment Rate (Q4 post-exit)

Now Employed Q-4 after exit

Average Earnings Median Earnings

Median earnings Q-2 after exit only

Credential Rate Will require follow-up for 1 yr after exit

Measurable Skills Gain

Real Time Measure, note exit based. How to document skills gain?

Performance Measures of Quality

• Rehabilitation Rates (percent rehabilitated vs. those that were not rehabilitated)

• Employment• Serious Vocational Impediment• Timeliness of Determination• Timeliness of Evaluation and Planning Phase• Accuracy of Determination, Service, Program

Compliance and Fiscal Transactions

Common Areas for Performance Measures

CUSTOMER SERVICE FINANCIAL

POLICY AND PROCEDURE EMPLOYEES

What Program Evaluation Activities are Required by RSA for Tribal VR Projects?

• Funded Grant Application • Financial Draw Downs • 6 Month Report to RSA Year 1• Annual Reports to RSA Years 1 – 5• 5 Year Report to RSA at End of Grant • MOU with State VR Agencies• New Requirements under WIOA Optional but Best Practice• Participation in Monthly RSA Conference Calls• Communication with Project Officer• Participation in Annual and Mid-Year CANAR Conferences• Participation in Annual Project Directors’ Conference

A Detailed Look at Project Visions

AIVRS Grant Application – Identify program evaluation activities as described in your application and others you are currently doing• Project VISIONS Program Evaluation

Description• Logic Model• Management Work Plan – excerpt

Client Outcome Annual Reports

• DataOps Electronic Client Database• DataOps Ad Hoc Reporting - Disability Report

Example

Satisfaction Surveys

• Client Satisfaction Survey• Client Satisfaction Survey Instrument• Client Satisfaction Report Excerpt

• Partner Satisfaction Survey• Partner Email Invitation• Partner Satisfaction Report Excerpt

Success Stories

• Bryan – Client Success Story• Betty and Al – Client Success Story• Joe – Client Success Story• 3 Fires Supportive Housing – Project Initiative

Success Story

Components to Inform Planning for 2016-20 Grant Application – External Evaluator

• Client In-person Interviews – Power of VR Services Abstract• Invitation to Stakeholders for Phone Interview• Interview Questions for Stakeholders• Stakeholder Interview Abstract• Stakeholder Interview Results & Recommendations• Evaluation of Operational Procedures and Staff Roles

Abstract

Additional Methods• Instrumented Group Interview (IGI)• Case File Review – RSA Corbridge Model• Project VISIONS Abstract• CANAR Conference Presentation Description• Coffee Café Model• Focus Groups and Talking Circles• Client Long Term Follow-up “Where have all our

clients gone?”

Guiding thoughts about Program Evaluation

• “Data are our friends”• “What gets measured get’s done”• “ Innovation or New Idea - Pilot test and

measure results”• “Given little or no information - people make

up the answers”

Resources

• Suzanne Malson – New Beginnings Brochure• DataOps – Information – Catherine and

George Laguna

SMALL GROUP DISCUSSION

How will you apply what you learned today?Did you think about a program evaluation method or

measure you hadn’t considered before?Will you look more deeply at a particular project

component? If so, what component? Will you develop a plan of how you can use

something from today’s presentation?

“Megwetch”

Thank You for Participating in our Presentation

A Community of Practice

• The Summit Group http://vocational-rehab.com/• Free Email Listserv with 360 VR professionals• The 9th Annual Summit on Vocational Rehabilitation

Program Evaluation, Fall 2016, Richmond, Virginia• The Rehabilitation Program Evaluation Network

http://vocational-rehab.com/rpen/what-is-rpen/