Planning for Performance Measurement: Reviewing the Performance Measures

Planning for Performance Planning for Performance Measurement: Measurement:

Reviewing the Performance Reviewing the Performance MeasuresMeasures

Presenters:

Gregory Thompson, OJJDPHeidi Hsia, OJJDP

Marcia Cohen, DSG, Inc.

Reporting RequirementsReporting Requirements

Output: All mandatory (bold) and two non-mandatory

output measures.

Outcome: All mandatory (bold) and two non-mandatory

measures.*

*Except Core Requirements and JJ System Improvement require only one non-mandatory measure.

Key to Logic Models and Grid ChartsKey to Logic Models and Grid Charts

Bold = Mandatory measure

* = Mandatory only for intervention programs

** = Mandatory only for prevention programs

Output Objective CategoriesOutput Objective Categories

Six types of output objectives:• Increase organizational/system capacity• Improve planning and development• Improve program quality• Improve program activities• Improve program efficiency • Improve the monitoring of compliance

Outcome Objective CategoriesOutcome Objective Categories

Nine types of outcome objectives:• Reduce delinquency• Increase accountability• Improve system effectiveness• Increase prosocial behaviors• Increase program support• Increase compliance with Core Requirements• Reduce disproportionate minority contact• Improve school climate• Improve the management of the State Juvenile Justice and Delinquency Prevention program

Prevention and Intervention Prevention and Intervention Program Areas Program Areas

Mandatory Mandatory OutputOutput Measures for Measures for Prevention and InterventionPrevention and Intervention Programs Programs

Title V or FG $ allocated for services:

• The amount of Formula Grants and Title V funds in whole dollars that are allocated for delinquency prevention services during the reporting period. Program records are the preferred data source.

Mandatory Mandatory OutputOutput Measures for Measures for Prevention Prevention

and Interventionand Intervention Programs (cont’d) Programs (cont’d)

Number of program youths served:

• An unduplicated count of the number of youths served by the program during the reporting period. Number served should be determined by subtracting out youths who dropped out/completed program/or were discharged/ terminated during the reporting period and adding in new admissions during the reporting. Include all youths who received at least one program service and met the program’s minimum criteria for participation. Program records are the preferred data source.

Mandatory Mandatory OutcomeOutcome Measures Measures for for PreventionPrevention** Programs** Programs

Number and percent of program youths exhibiting desired change in targeted behaviors: • Substance use • School attendance • Antisocial behavior • Family relationships • Pregnancies

Note: Double ** denotes mandatory for prevention programs

Mandatory Mandatory OutcomeOutcome Measures Measures for for InterventionIntervention* Programs* Programs

Number and percent of program youths who offend or reoffend:

• The number of program youths who were rearrested or seen at juvenile court for a new delinquent offense. Appropriate for any youth-serving program.

Note: Single * denotes mandatory for intervention programs

Mandatory Mandatory OutcomeOutcome Measures for Measures for Prevention and InterventionPrevention and Intervention Programs Programs

Number and percent of youth completing program requirements:

• The number and percent of program youth who have successfully fulfilled all program obligations and requirements. Program obligations will vary by program but should be a predefined list of requirements or obligations that clients must meet prior to program completion. Program records are the preferred data source.

Example: Example: Probation Program Area Probation Program Area

Outcome Measure DefinitionsShort Term:Occurs during the program or by the end of the programLong Term:Occurs 6 months to 1 year after program completion

Key = System-level indicator

= Individual-level indicator = Objectives

Bold= Mandatory measure

Performance measures should report on those activities funded by Title II (Formula Grants) funds.

Long-Term Outcome ObjectivesReduce delinquencyIncrease accountability

Short-Term Outcome Objectives

Reduce delinquencyImprove prosocial behaviorsIncrease accountabilityIncrease program support

#/% of program staff with increased knowledge of program area

#/% of program youths exhibiting desired change in targeted behaviors (e.g., substance use, school attendance, antisocial behavior, family relationships, social competencies)

#/% of program youths completing program requirements

#/% of program youths charged with formal probation violation

#/% of program youths who offend or reoffend

#/% of program youths committed to correctional facility

OUTCOME MEASURES

SHORT-TERM

#/% of program youths committed to correctional facility

#/% of program youths who offend or reoffend

LONG-TERM

Delinquency

To improve JJ systems by increasing compliance with the Core Requirements and increasing the availability and types of prevention and intervention programs

Sample Logic Model: PA 24—Probation

PROBLEM

Goals

INTERVENTION PROGRAM

(Deficiency or lack of adequate probation

programs)

To support both State and local

intervention efforts and JJ

system improvements

SUBPROBLEM(S)

Objectives

Output ObjectivesIncrease organizational capacityImprove planning & developmentImprove program qualityImprove program activitiesImprove program efficiency

OUTPUT MEASURES

Conduct PlanningActivities

andImplement Program(s)

Monitor Program(s)

ACTIVITIES

FG $ allocated for services

# of planning activities conducted

# of MOUs developed

# of program slots available

# of FTEs funded with FG $

#/% of program staff trained

# of hours of program staff training provided

Use of best practice model (Y/N)

# of program materials developed

# of program/agency policies or procedures, created, amended, or rescinded

# of program youths served

Average length of stay in program

# of service hours completed

PROGRAM AREA 24. ProbationPROGRAM AREA 24. ProbationOUTPUTOUTPUT PERFORMANCE MEASURES PERFORMANCE MEASURES

Sample Grid Chart: PA 24—ProbationOutput Performance Measures

Sample Grid Chart: PA 24—ProbationOutcome Performance Measures

Example: Example: Delinquency Prevention Delinquency Prevention

Program Area Program Area

Outcome Measure DefinitionsShort Term:Occurs during the program or by the end of the programLong Term:Occurs 6 months to 1 year after program completion





Delinquency


Sample Logic Model: PA 09—Delinquency Prevention

PROBLEM

Goals

PREVENTION PROGRAM

(Deficiency or lack of adequate delinquency prevention programs)

To support both State and local


system improvements

SUBPROBLEM(S)

Objectives

Output ObjectivesIncrease organizational capacityImprove planning & developmentImprove program qualityImprove program activitiesImprove program efficiency

OUTPUT MEASURES


andImplement Program(s)

Monitor Program(s)

ACTIVITIES

FG $ allocated for services

# of MOUs developed

# of planning activities conducted

# of program slots available

# of FTEs funded with FG or TV $



Use of best practice model (Y/N)



# of program youths served

#/% of parents served



Long-Term Outcome ObjectivesImprove prosocial behaviors

Short-Term Outcome Objectives

Improve prosocial behaviorsIncrease program supportIncrease accountability


#/% of program youths exhibiting desired change in targeted behaviors (e.g., substance use, school attendance, antisocial behavior, family relationships, pregnancies)

#/% of program youths completing program requirements

#/% of program families satisfied with program

OUTCOME MEASURES

SHORT-TERM

#/% of program youths exhibiting desired change in targeted behaviors (e.g., substance use, school attendance, antisocial behavior, family relationships, pregnancies)

LONG-TERM

#/% of youths satisfied with program

Sample Grid Chart: PA 09—Delinquency PreventionOutput Performance Measures

Sample Grid Chart: PA 09—Delinquency PreventionOutcomes Performance Measures

Core Requirements Core Requirements Program Areas Program Areas

Mandatory Mandatory OutputOutput Measures for Measures forCore RequirementsCore Requirements Programs Programs

FG funds allocated:

• The amount of Formula Grants money in whole dollars that is allocated for the separation requirement during the reporting period. Program records are the preferred data source.

Mandatory Mandatory OutputOutput Measures for Measures for Core RequirementsCore Requirements Programs (cont’d) Programs (cont’d)

Number of programs implemented:

• The number of programs created and/or implemented during the reporting period. Program records are the preferred data source.

Mandatory Mandatory OutcomeOutcome Measure for Measure for Core RequirementsCore Requirements Programs Programs

Change in the number of violations

The change in the number of violations of the sight and sound requirement from the previous reporting period compared with the current reporting period. Sight and sound compliance is determined according to the definition in the OJJDP Guidance Manual for Monitoring Facilities Under the JJDP Act of 2002. The Annual Compliance Monitoring Report is the preferred data source.

Example: Example: Separation Program AreaSeparation Program Area

Outcome Measure DefinitionsReported Annually.





Delinquency

To improve JJ systems by increasing compliance with the Core Requirements and increasing the availability and types of prevention and intervention programs.

Sample Logic Model: PA 28—Separation

PROBLEM

Goals

Outcome ObjectivesIncrease compliance with core requirementsIncrease program support

OUTCOME MEASURES

#/% of staff with increased knowledge of program area (e.g., separation regulations)

Change in the number of violations of sight and sound separation

OUTPUT MEASURESACTIVITIES

Output Objectives•Improve monitoring of compliance•Increase organizational capacity•Improve planning & development

Conduct Planning and

Oversight Activities

Implement Processes

and/or Programs

FG $ allocated for separation

# of MOUs developed

Needs assessment completed (Y/N)

#/% of programs implemented

# of transportation plans developed

# of shelter beds contracted

#/% of staff trained

# of hours of staff training provided

# of materials developed

# of program policies/procedures created, amended, or rescinded

Monitor Processes

and/or Programs

#/% of site visits conducted

# of facilities receiving TA

CORE REQUIREMENT(Ensuring/monitoring compliance with the

Separation Core Requirement)

To support both State and local prevention and


system improvements

SUBPROBLEM(S)

Objectives

Sample Grid Chart: PA 28—SeparationOutput Performance Measures

Sample Grid Chart: PA 28—SeparationOutcome Performance Measures

Example: Example: Compliance Monitoring Compliance Monitoring

Program AreaProgram Area






Delinquency


Sample Logic Model: PA 06—Compliance Monitoring

PROBLEM

Goals

CORE REQUIREMENT(Ensuring/monitoring

compliance with Section 223 (a)(14) of the JJDP Act of 2002)



system improvements

SUBPROBLEM(S)

Objectives

Output Objectives•Improve monitoring of compliance•Increase organizational capacity•Improve planning & development

OUTPUT MEASURES



ACTIVITIES

Funds allocated to adhere to Section 223 (a)(14) of the JJDP Act of 2002

# of activities that address compliance with Section 223 (a)(14) of the JJDP Act of 2002

# of MOUs developed




# of program policies/procedures created, amended, or rescinded

# of facilities receiving technical assistance

Outcome ObjectivesImprove the management of the State JJDP ProgramIncrease program support

OUTCOME MEASURES

#/% of staff with increased knowledge of program area (e.g., compliance monitoring)

Submission of complete Annual Monitoring Report to OJJDP

Sample Grid Chart: PA 06—Compliance MonitoringOutput Performance Measures

Sample Grid Chart: PA 06—Compliance MonitoringOutcome Performance Measures

Outcome Measure DefinitionsShort-Term:Occurs during the program or by the end of the programLong-Term:Occurs 6 months to 1 year after program completion

Key = system-level indicator

= individual-level indicator = objectives

Bold= mandatory measure

Performance Measures should report on those activities funded by Title II (Formula Grants) funds.

Delinquency


Sample Logic Model:PA 10 – Disproportionate Minority Contact

PROBLEM

Goals

Output ObjectivesImprove monitoring of complianceIncrease system capacityImprove program qualityImprove planning and development

OUTPUT MEASURES


&Implement Program(s)

and Processes

Monitor Program(s)

and/or Programs

ACTIVITIES

FG $ allocated for DMC at the State and local levels

# of FTEs funded with FG or TV $

# of activities conducted

# of programs implemented



# of materials developed


# of objective decision-making tools developed

# of program youth served



#/% of site/TA visits conducted

Outcome ObjectivesReduce DMCImprove system effectivenessIncrease program supportReduce delinquency

Number of minority staff hired

#/% of recommendations implemented from assessment studies

OUTCOME MEASURES

Short-Term

% of contact points reporting reduction in diproportionality at state level

Long-Term

Number of agencies with improved data collection systems


#/% of program youth who offend or reoffend

#/% of program youth exhibiting a desired change in targeted behaviors (e.g., substance use, school attendance, family relationships, and antisocial behavior)

#/% of program youth completing program

#/% of program youth satisfied with program

#/% of program families satisfied with program

% of contact points reporting reduction in diproportionality at local level

* #/% of program youth who offend or reoffend

#/% of state agencies reporting improved data collection systems

# of local agencies reporting improved data collection systems

SUBPROBLEM(S)

CORE REQUIREMENT(Ensuring compliance

with the DMC Core Requirement)


intervention efforts and JJ system

improvements to reduce DMC

Objectives

INTERVENTION PROGRAM

(Deficiency or lack of adequate intervention programs to reduce

DMC)

PREVENTION PROGRAM

(Deficiency or lack of adequate delinquency prevention programs)

Sample Grid Chart: PA 10—Disproportionate Minority ContactOutput Performance Measures

Sample Grid Chart: PA 10—Disproportionate Minority ContactOutcome Performance Measures

Juvenile Justice System Juvenile Justice System Improvement Program AreasImprovement Program Areas

Mandatory Mandatory OutputOutput Measure for Measure for JJ System ImprovementJJ System Improvement Programs Programs

Number of grants funded with Formula Grants funds:

• The number of grants funded with Formula Grants funds during the reporting period.

Mandatory* Mandatory* OutputOutput Measures for Measures for JJ System ImprovementJJ System Improvement Programs Programs

* If applicable (if not applicable, select another non-mandatory measure)

• Number of programs implemented• Number of program youth served• Number of subgrants awarded• Number of grant applications reviewed

and commented upon

Mandatory* Mandatory* OutcomeOutcome Measures for Measures for JJ System ImprovementJJ System Improvement Programs Programs

* If applicable (if not applicable, select another non-mandatory measure)

• Number/percent of youth completing program • Number/percent of youth exhibiting desired

changed in targeted behaviors • Average length of time between initial appearance

and disposition• Number of programs funded directly in line with 3-

year Plan• Number of Plan recommendations implemented • Number of allegations of civil rights violations filed• Number/percent of court hearings attended by an

advocate

Example: Example: State Advisory Group (SAG) State Advisory Group (SAG)

Program AreaProgram Area

Mandatory Mandatory OutputOutput Measures for Measures for SAGSAG Program Area Program Area

Number of grants funded with Formula Grants funds:

• The number of grants funded with Formula Grants funds during the reporting period.

Mandatory Mandatory OutputOutput Measures for Measures for (SAG) Program Area(SAG) Program Area

Number of grant applications reviewed and commented on:

• The number of grant applications reviewed and commented on to guide the development of juvenile justice programming in the State.

Mandatory Mandatory OutcomeOutcome Measure for Measure forSAGSAG Program Area Program Area

Number and percent of Plan recommendations implemented:

• Number and percent of SAG recommendations for the State Plan implemented during the reporting period. The preferred data source is

committee meeting minutes.






Delinquency


Sample Logic Model: PA 31—State Advisory Group (SAG) Allocation

PROBLEM

Goals

JUVENILE JUSTICE SYSTEMS

IMPROVEMENT(Need for effective

SAG activities)



system improvements

SUBPROBLEM(S)

Objectives

Output Objectives•Improve planning and development•Improve program efficiency•Improve program quality

OUTPUT MEASURES



ACTIVITIES

# of grants funded with Formula Grants funds

# of SAG committee and subcommittee meetings held

Annual Report submitted to the Governor

#/% of SAG members trained

# of grant applications commented on

#/% of activities/meetings that involve youth

# of units of local government consulted during implementation of the 3-Year Plan

#/% of programs using best practice model

% of SAG allocation used

# of SAG members contributing to plan or plan update

Outcome ObjectivesIncrease compliance with Core RequirementsIncrease program support

#/% of SAG members who show increased knowledge of the program areas (for which they have oversight)

OUTCOME MEASURES

# of FG-funded programs sustained after 3 years

#/% Plan recommendations implemented

Sample Grid Chart: PA 31—State Advisory Group Output Performance Measures

Sample Grid Chart: PA 31—State Advisory Group Outcome Performance Measures

Formula Grants Formula Grants

Performance Measurement Performance Measurement Implementation TimelineImplementation Timeline

(First year) (First year)

April 1, 2005 Reporting period begins

Sept. 30, 2005 Reporting period ends

Dec. 31, 2005 Data report due to OJJDP

Formula Grants Formula Grants

Performance Measurement Performance Measurement Implementation TimelineImplementation Timeline

(Second and Subsequent Years) (Second and Subsequent Years)

Oct. 1, 2005 Reporting period begins


Dec. 31, 2006 Data report due to OJJDP

Title V Performance MeasurementTitle V Performance MeasurementImplementation Timeline Implementation Timeline

(First year)(First year)

January 1, 2005 Reporting period begins


Nov. 30, 2005 Data reports due to OJJDP

March 31, 2006 FY 2005 OJJDP Report to Congress due

Title V Performance MeasurementTitle V Performance MeasurementImplementation TimelineImplementation Timeline

(Second and Subsequent Years)(Second and Subsequent Years)

October 1, 2005 Reporting period begins


Nov. 30, 2006 Data reports due to OJJDP

March 31, 2007 FY 2006 OJJDP Report to Congress due

Performance MeasurementPerformance MeasurementTools and SupportTools and Support

• Web site:

www.dsgonline.com/performance_measures.htmwww.dsgonline.com/performance_measures.htm

• Training

• Technical assistance

Documents

Planning for Performance Measurement: Reviewing the Performance Measures